Verify AI crawler requests to your pages

Which records do you need?

Choose the pages whose access matters: a product fact page, a newsroom statement or an original report. Ask the site operator for the server or CDN logs covering an agreed period. Include timestamp, requested URL, request method, response status, user-agent string and enough network evidence to verify identity. Handle raw IP data according to the organization’s access and retention rules.

Check where logging happens. A request served at the CDN may never reach the origin server, and a sampled log may omit requests. Record those coverage limits. A missing row is a missing observation until you establish what the logging system captures.

How do you distinguish real crawlers?

Treat the user-agent string as a starting label. For Google, use the provider’s published IP ranges or perform reverse DNS followed by a matching forward lookup. The steps are documented in Google’s request-verification guide. Keep the verification result beside the request, including “unverified” when evidence is insufficient.

OpenAI publishes different agents and IP lists for search, possible training use and user-triggered fetching. Its crawler documentation distinguishes OAI-SearchBot, GPTBot and ChatGPT-User. A GPTBot hit should not be labeled a buyer search. Mark a deliberate test that triggers ChatGPT-User as your own test and exclude it from independent audience activity.

Google-Extended requires special care: it is a robots.txt control token and has no separate HTTP request user agent. Google’s documentation describes its Gemini-related uses and separates it from Google Search controls. Do not create a “Google-Extended visits” metric from a guessed log label.

What does a request actually establish?

A verified successful request establishes that a particular resource was served to that requester at that time. Record status codes and distinguish requests for the article from requests for robots.txt, images or other assets. Preserve redirects and failures so a raw request total does not conceal an access problem.

Use a simple ledger:

The example illustrates the fields, not a real campaign result. None of these fields measures whether an answer quoted the page or a customer saw it.

Which stage does the evidence describe?

Chapter 12 of Become the Answer separates the checks that can otherwise disappear inside a crawler total. Keep evidence rows for requesting a URL, delivering useful content, index inclusion where observable, source selection for a query and support in the resulting answer. Add an action check when an agent is expected to operate a page function.

Mark stages without evidence as unobserved. These are diagnostic distinctions; assistants can implement them differently. A successful request establishes none of the later stages by itself.

How should access and citations be compared?

For a configured Rankfor publishing surface, Publishing > Statistics provides recorded requests and available user-agent classifications. Keep the serving surface and date range with the count. Those classifications use declared user-agent signatures, so independently verified identity requires the additional log checks described here.

Separately collect relevant saved answers and their visible citation links. Match normalized URLs, retaining the original links and dates. A fetched-and-cited page has two observations; it does not prove that this particular fetch caused that answer. A fetched-but-uncited page can still be useful to readers or appear outside the sampled questions.

For newsroom pages, investigate a factual correction and its technical availability on their own merits. Avoid concluding that an absent citation means the newsroom is globally invisible. The citation-verifiability research also shows why a visible link needs a claim-level support check before it is used as evidence.