Server logs and AI crawlers, seeing who really visits your site

Read your server logs. Every request writes one line with the client IP, the timestamp, the requested URL, the HTTP status and the user agent. Filter that file on known AI agent tokens, count lines per agent and per URL, and you get crawl frequency, the pages that are fetched, and the ones that never are.

No other tool gives you that view. Search Console only reports Googlebot, GA4 only measures a human visitor whose browser runs JavaScript, and AI crawlers generally do not run any. Logs record the request at the server level, before rendering and before any analytics consent. They are also the only source that can tell you an important page has never been requested at all, which no citation dashboard can do.

The work splits into four steps, and each answers a different question.

• Collect. Confirm your entry layer actually writes the user agent and the HTTP status, which is not guaranteed by default on every platform.
• Filter. Isolate lines whose user agent contains a known AI agent token, case insensitive.
• Aggregate. Count visits per agent, per URL and per status code, then record the last seen date for every page.
• Verify. Match client IPs against the ranges the vendor publishes before drawing any conclusion about an agent identity.

Before you open the log file. AI crawler access checker | AI crawler simulator

Three very different things land in the same log file, and mixing them up ruins the analysis. Scheduled crawling is planned by the vendor, it walks the site without any person asking for it, and it feeds either a search index or a training dataset. A user-triggered fetch starts from a question asked inside an assistant, it touches only a handful of URLs, and it is usually immediate. Referred human traffic is not a robot at all, it is a person who clicked a citation link in an AI answer, and it shows up in the referer field rather than the user agent.

Vendor documentation owns that split. OpenAI separates OAI-SearchBot, used to surface websites in ChatGPT search features, GPTBot, which serves the foundation models, and ChatGPT-User, used for certain user actions. Anthropic splits ClaudeBot, Claude-SearchBot and Claude-User the same way. Perplexity contrasts PerplexityBot, which powers search, with Perplexity-User, which answers an action inside the product.

That nuance has a direct consequence for robots.txt. OpenAI writes that robots.txt rules may not apply because the actions are initiated by a user, Perplexity states that Perplexity-User generally ignores robots.txt rules, and Google says the same about its user-triggered fetchers. Seeing one of those agents despite a blocking rule is therefore not necessarily an incident, it is documented behaviour.

OpenAI, crawlers and bots | Anthropic, web crawling | Perplexity, crawlers | Google, user-triggered fetchers

The combined format is enough to start, as long as you add response time. Nginx defines it out of the box with the client address, the user, the local time, the request line, the status, the body bytes, the referer and the user agent. Only one genuinely useful field is missing for crawl diagnosis, processing time, available through the request_time variable that the documentation describes as request processing time in seconds with a milliseconds resolution.

Apache exposes the same set of fields through its LogFormat directive. Its combined format carries the host, the timestamp, the request, the final status and both the Referer and User-Agent headers. Two additions are worth making, duration in microseconds with the D specifier, and bytes actually sent including headers with the O specifier, which requires the mod_logio module.

# nginx.conf, le format combined enrichi de la latence de réponse
log_format ai_agents '$remote_addr - $remote_user [$time_local] '
                     '"$request" $status $body_bytes_sent '
                     '"$http_referer" "$http_user_agent" $request_time';

access_log /var/log/nginx/access.log ai_agents;

nginx, ngx_http_log_module | Apache, log files | Apache, mod_log_config

Under Apache the logic is identical and only the syntax changes. Declare a named format, then point CustomLog at it. Keep the final status with the greater-than modifier, otherwise an internal redirect makes you record the status of the original request rather than the one actually served.

# httpd.conf, combined enrichi des octets réels et de la durée
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %D" ai_agents
CustomLog logs/access_log ai_agents

These commands carry no customer data and work on any combined format file. Field 9 is the HTTP status and field 7 is the requested URL in that format, which makes aggregation direct. Adapt the token list to your market, an English language site will mostly see OpenAI, Google, Perplexity, Anthropic and Microsoft.

The sixth command is the most useful and the least often run. It compares the URLs actually fetched against your list of priority pages and returns the difference, in other words the pages no AI agent has ever come for. Content that is never crawled cannot be indexed or cited, whatever its editorial quality.

AI='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Googlebot|GoogleOther|bingbot|meta-externalagent'

# 1. Every AI agent visit, all statuses
grep -Ei "$AI" access.log

# 2. Volume per agent, busiest first
grep -Eio "$AI" access.log | sort | uniq -c | sort -rn

# 3. Only the error responses served to an agent
grep -Ei "$AI" access.log | awk '$9 ~ /^(403|404|429|500|503)$/'

# 4. Top 20 URLs crawled by GPTBot
grep -i 'GPTBot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# 5. Last recorded visit for one agent
grep -i 'ClaudeBot' access.log | tail -1

# 6. Priority pages never visited, by diffing against the sitemap
grep -Ei "$AI" access.log | awk '{print $7}' | sort -u > seen.txt
comm -23 priority-urls.txt seen.txt

To build the priority page list. llms.txt generator | Hikoo Analyzer

A user agent header is a free text string chosen by the client. Anyone can send a request declaring itself GPTBot, and your log will record it as such. A log line is therefore never proof that a vendor visited, it only proves a request claimed that identity. As long as you stay at the level of trend counting, the approximation is acceptable. The moment you decide to block, to charge, or to serve different content, it no longer is.

Two documented methods settle the question. The first is matching the client IP against the ranges the vendor publishes. OpenAI publishes one file per bot, Anthropic publishes a single list, Perplexity publishes one file per crawler. The second is a reverse DNS lookup followed by a forward one, the method Google documents for Googlebot, where the address must resolve to a name under googlebot.com, google.com or googleusercontent.com, and that name must then resolve back to the original address.

A third path is emerging. Web Bot Auth has the bot sign its requests with the Signature, Signature-Input and Signature-Agent headers, built on HTTP Message Signatures from RFC 9421 and on Ed25519 keys. Verification becomes cryptographic and no longer depends on an address list you have to keep current. Cloudflare already implements it for its verified bots, and the matching specifications are still in progress at the IETF, so adoption remains partial today.

# 1. Reverse DNS on the address found in the log
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

# 2. Forward DNS on the name returned, it must resolve back
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

# 3. Vendors that publish their ranges in CIDR form
curl -s https://openai.com/gptbot.json
curl -s https://claude.com/crawling/bots.json
curl -sL https://www.perplexity.com/perplexitybot.json

Google, verifying Googlebot | Cloudflare, Web Bot Auth | IETF, Web Bot Auth architecture

Before you interpret a log file, check that your chain writes what you think it writes. Send a request to your own site with a test user agent, then read back the line it produced. If the user agent is truncated, if the status belongs to an intermediate redirect, or if the request never appears because an upstream cache served it, you find out now rather than halfway through an analysis.

The capture below shows exactly that on a local server. It is a test request we sent ourselves with a user agent string imitating GPTBot, not a visit from an official crawler. Its only purpose is to prove that the log format captures the user agent, the status and the duration, and that errors show up too.

# A TEST request to our own local server, sent with a test user agent.
# This is not a visit from an official crawler.

$ UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
$ curl -s -o /dev/null -w "%{http_code} %{size_download} %{time_total}\n" -A "$UA" \
    http://localhost:3010/fr/blog/guides/logs-serveur-crawlers-ia-guide-2026/
200 460948 0.016284

$ curl -s -o /dev/null -w "%{http_code} %{size_download} %{time_total}\n" -A "$UA" \
    http://localhost:3010/fr/blog/guides/page-qui-nexiste-pas/
404 8376 0.023425

# The two lines actually written to access.log by the combined format + $request_time

$ tail -2 access.log
::1 - - [31/Aug/2026:18:55:31 +0200] "GET /fr/blog/guides/logs-serveur-crawlers-ia-guide-2026/ HTTP/1.1" 200 460948 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" 0.014
::1 - - [31/Aug/2026:18:55:31 +0200] "GET /fr/blog/guides/page-qui-nexiste-pas/ HTTP/1.1" 404 8376 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" 0.020

The status code is the highest yield field in the file, because it says what the agent actually received. A 200 on an empty page is no better than a 404, and a repeated 403 across a whole section almost always points to a web application firewall rule or an anti-bot protection blocking an agent you believed you were allowing.

The 429 deserves particular attention. RFC 6585 defines it as the response to a client that has sent too many requests in a given amount of time, with an optional Retry-After header telling it how long to wait. Served in volume to a crawler, it slows exploration and delays discovery of your new pages. Anthropic documents support for the non-standard Crawl-delay extension in robots.txt, a cleaner way to pace a crawler than rejecting it with a 429.

RFC 6585, status 429 | Anthropic, Crawl-delay

A request crosses several layers and each one logs only part of the story. The CDN sees all traffic, including what it serves from cache and what will therefore never reach your origin. The origin server only sees requests that made it through. If you analyse origin logs alone behind an aggressive CDN, you systematically underestimate crawler activity.

On Vercel, runtime logs expose the method, the requested path, the status, the region, the cache state and the request user agent. Two limits matter. First, static requests only appear when they serve cache, and the documentation points to Log Drains to get all static requests. Second, retention is short, one hour on Hobby, one day on Pro, three days on Enterprise, and thirty days with Observability Plus. A thirty day crawl analysis therefore implies an export, not a dashboard lookup.

On Cloudflare, the HTTP requests dataset covers exactly what a crawl analysis needs, notably ClientRequestUserAgent, ClientIP, ClientRequestPath, EdgeResponseStatus, EdgeResponseBytes and EdgeTimeToFirstByteMs. Bot Management customers also get BotScore and VerifiedBotCategory. Cloudflare additionally offers AI Crawl Control, formerly AI Audit, which shows which AI services access your content and works on all plans.

Vercel, runtime logs | Cloudflare, HTTP log fields | Cloudflare, AI Crawl Control

A useful AI crawl report fits in nine columns. They answer the only questions that matter, namely who comes, where, how often, with what result, and what stays unvisited. Anything beyond that list is comfort, not diagnosis.

Two indicators deserve to be tracked over time rather than read as a single value. Priority page coverage, meaning the share of your strategic URLs fetched at least once in the period, measures your real exposure. The error rate served to agents tends to degrade quietly after a firewall update or a CDN plan change, and a sudden rise almost always explains a drop in visits the following week.

A 200 in the logs does not tell you what was served. A page can respond correctly and still contain only an HTML shell whose text arrives after JavaScript runs, which most AI crawlers do not do. The log gives you the status, the simulator gives you the content.

In the capture below we submitted a public page from our own blog to the Hikoo AI crawler simulator. The result shows the number of words actually readable, the count of headings and links, and the raw text as an agent receives it. Always cross the two, a 200 status in the log and a sane word count in the simulator.

Tools used. AI crawler simulator | AI crawler access checker | Free AI audit

An access.log contains IP addresses, therefore personal data as soon as human visitors appear in it. A crawl analysis, however, needs no identifying data once identity verification is done. Treat verification and analysis as two separate stages, the first on raw short lived data, the second on an aggregated set you can keep longer.

• Set an explicit retention period for raw logs and automate the purge, your host default retention is not a policy.
• Anonymise the IP after verification, for example by truncating the last octet in IPv4 and the trailing segments in IPv6.
• Keep only the normalised agent, the URL, the timestamp, the status, the bytes and the latency for crawl analysis.
• Strip query parameters from aggregated URLs, they often carry session or campaign identifiers.
• Document the purpose and the duration in your processing record, a technical analysis is still processing.
• Restrict raw log access to people with an operational need, and log those accesses.

Manual analysis answers a one off question. It holds up badly over time, because you have to collect, deduplicate, verify identities, keep history beyond the platform retention window, and redo the exercise after every infrastructure change.

Hikoo ships AI agent tracking inside the dashboard. It records AI crawler visits to your site, when they come, how often and which pages they read, agent by agent. That view sits alongside Spotlight, which measures how your brand is cited in AI answers. The two answer different questions, one tells you what agents read on your site, the other tells you what they say about you elsewhere.

Putting them together is where the value is. A heavily crawled page that is never cited is a content problem, a page that is never crawled is a technical problem, and only reading both together lets you decide. Hikoo Analyzer completes the picture by scoring how readable each page is for retrieval.

Keep going. Hikoo Spotlight | Hikoo Analyzer | robots.txt and AI crawlers | Prepare your site for AI agents | Measure AI traffic in GA4 | AI visibility

Server logs remain the only exhaustive measurement of AI agent activity on your site. They tell you which pages are read, how often, with which HTTP status, and which ones stay unvisited, where classic analytics tools see nothing at all. The work comes down to enriching the log format, filtering on the right tokens, verifying identities by IP, then tracking nine columns over time.

The natural next step is to cross that technical reading with your citation data. Start with a free audit of your site to place your readability for agents, then track both crawler visits and brand citations in Hikoo. Putting the two side by side is what tells you what to fix first.