AI crawler access check

Which AI crawlers your robots.txt lets in

We read one file — your robots.txt — and report whether each of 19 AI crawlers may fetch your front page and the pages under it. When an AI answer names a source, that is a citation.

  • Can they get in
  • Free
  • ~5s

16 of 19 crawlers www.nytimes.com keeps out of the pages under its front page Measured 2026-09-18

One URL. No sign-in needed.

Add one and we put both in the same table, then list the bots that reach them but not you.

Sign in and you get 10 runs a day. Sign in or create an account

First, a screen we measured ourselves

16 of 19 crawlers www.nytimes.com keeps out of the pages under its front page

  • citeangle.com blocked, 0 of 19
  • www.nytimes.com blocked, 16 of 19
  • www.wikipedia.org blocked, 0 of 19

A checker with two columns prints Korea's largest portal as open. The front page is open; nothing below it is. What an answer quotes usually sits below.

19 AI bots (6 retrieval · 4 user-fetch · 9 training) · read straight from robots.txt · Measured 2026-09-18 · See the source

What you saw here / what the paid tier adds

Free ends here On the paid tierRe-measured every month Which product opens it
  • A verdict for all fifteen AI bots
  • One rival on the same screen
  • Nothing held back
  • Locked — More rivals on one screen
  • Locked — We re-check every month and tell you when a verdict changes
  • Compass$1,199
  • Competitive Signal Program$10,470 a quarter · billed $3,490 monthly, three times

What splits them — How many sites, and how often

Here is what the paid readout looks like — The numbers in the locked rows go in once we have measured them.

Where free ends

Every verdict is on screen. What we sell is deciding what to open, what to keep shut, in what order — and doing it.

The first paid rung is $399. See the diagnostic ladder →

What this check misses

Where this comes from — the crawler operators' own documentation: OpenAI's crawler docs · Google's list of crawlers

See the audit ladder → · Other free tools

Start the free check here

One minute, no card, no meeting — that is the free AI visibility check. Three questions are on us — write the ones your buyers would put to an AI, and we run them exactly as you wrote them and mail you the results page.

Each question goes to 6 AI answer surfaces. Send 3 and that is 18 observations in all, and we count how many of them came back with your name.

One question is enough to start.

We start from what a marketing chief asks first.

Ready to turn US visibility evidence into a market plan? Tell us the US buyer, target state and decision date.

Request a written proposal