---
title: "AI Crawler Log Analysis: Measure Your Crawl-to-Referral Ratio"
description: Your server logs show which AI bots take your content and how little traffic they send back. Build a crawl-to-referral ratio in BigQuery and use it to decide what to open, block, and fix.
author: LETSGROW Dev Team
date: 2026-10-28
category: Analytics
tags: ["AI Crawlers", "Log Analysis", "BigQuery", "AI Search", "GEO"]
url: "https://letsgrow.dev/blog/ai-crawler-log-analysis-crawl-to-referral-ratio"
---
Most teams measure AI search visibility by typing prompts into chatbots and screenshotting the answers. That is anecdote, not measurement. Your server logs already hold the real evidence: which AI crawlers fetch your pages, how often, and how little traffic they send back. If you are not reading them, you are guessing about the most important new distribution channel you have.

::stat-block
- value: 3
  label: Distinct bot classes with different business meanings: training, search index, user fetch
- value: 1 ratio
  label: Crawl-to-referral, the number that tells you what each bot actually costs and returns
- value: 0 vendors
  label: Needed beyond the logs you already store
::

## Not All AI Bots Are the Same Decision

Lumping every AI user agent into one bucket is the first mistake. Each class answers a different question.

Training crawlers such as GPTBot, ClaudeBot, and CCBot collect content to build models. Blocking them is a licensing and brand decision, and it has no direct effect on whether you are cited in live answers. Search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot build the indexes that answer engines retrieve from. Block these and you opt out of being a source. User-fetch agents such as ChatGPT-User, Claude-User, and Perplexity-User fire when a real person asks a question that needs your page. Every one of those hits is a human with intent, which makes it the closest thing to an AI impression you can count.

Treat the three classes as three separate metrics. A single "AI bot traffic" number hides the only signal worth acting on.

## Build the Crawl Table in BigQuery

You do not need a log platform. Ship your CDN or edge logs to BigQuery (Cloudflare Logpush, Fastly, and CloudFront all support this) and classify by user agent. Start with this query and extend the pattern lists as new agents appear.

```sql
SELECT
  DATE(timestamp) AS day,
  CASE
    WHEN REGEXP_CONTAINS(user_agent, r'GPTBot|ClaudeBot|CCBot') THEN 'training'
    WHEN REGEXP_CONTAINS(user_agent, r'OAI-SearchBot|Claude-SearchBot|PerplexityBot') THEN 'search_index'
    WHEN REGEXP_CONTAINS(user_agent, r'ChatGPT-User|Claude-User|Perplexity-User') THEN 'user_fetch'
  END AS bot_class,
  REGEXP_EXTRACT(path, r'^/[^/]*') AS site_section,
  COUNT(*) AS hits
FROM `project.logs.edge_requests`
WHERE status = 200
  AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY ALL
HAVING bot_class IS NOT NULL
```

One warning before you trust the output: user agents are self-declared. Anyone can claim to be GPTBot. Verify against each vendor's published IP ranges or reverse DNS before you count a hit, and drop anything that fails. Unverified rows inflate your numbers and send you chasing a channel that is partly scrapers.

## Read the Crawl-to-Referral Ratio Like a P&L

Now join the other side. In GA4's BigQuery export, filter sessions whose source is chatgpt.com, perplexity.ai, claude.ai, gemini.google.com, or copilot.microsoft.com. Divide verified crawler hits by referred sessions per site section. That is your crawl-to-referral ratio.

The absolute number matters less than the pattern across your sections. A section that gets crawled heavily and never referred is content being consumed without credit. A section with modest crawling and strong referrals is content that answer engines trust. Decide per section, not per site.

::compare-table
| Pattern in the data | What it usually means | Action |
| --- | --- | --- |
| Heavy training crawl, near zero referrals | Content is feeding models with no return | Decide on blocking or licensing for that section |
| Search-index crawl, no user fetches | Indexed but never retrieved | Rewrite for direct answers, tighten headings |
| User fetches, no referral sessions | Cited but not clicked | Add a clear next step and a reason to visit |
| Crawl errors above 5 percent | Bots hit 404s and redirects | Fix routes, update the sitemap |
| Bots skip a section entirely | Blocked or buried | Check robots.txt and internal links |
::

User fetches deserve the most attention. They are demand you can see, and the pages that attract them tell you which questions your market is actually asking AI tools about you. Rank your top 20 fetched URLs weekly and compare them to the pages you are promoting. The mismatch is usually uncomfortable and always useful.

## Turn the Numbers Into Decisions

Logs are only worth the pipeline if they change what you ship. Run this loop monthly and keep the changes small and reversible.

::checklist
title: AI crawler log analysis rollout
items:
  - Ship edge or CDN logs to BigQuery with user agent, path, status, and IP
  - Classify bots into training, search index, and user fetch
  - Verify bot identity by IP range or reverse DNS before counting
  - Join verified hits to GA4 referral sessions by site section
  - Alert when crawl errors or 4xx responses to bots rise week over week
  - Review robots.txt per bot class, not with one blanket rule
  - Rank the top 20 user-fetched URLs and refresh the weakest ones
  - Log every change with a date so ratio shifts can be attributed
::

The uncomfortable truth is that crawl volume is not a win. A bot that takes thousands of pages and sends back ten visits is a cost, not an audience. The teams that pull ahead will be the ones who can say exactly which sections earn their crawl and which ones do not, and they will say it with a query, not a screenshot.

Start with thirty days of logs and the one query above. By the end of the week you will know more about your AI search footprint than any prompt-testing tool can tell you.
