AI crawler access check

Which AI crawlers your robots.txt lets in

We read one file — your robots.txt — and report whether each of 15 AI crawlers may fetch your front page and the pages under it. When an AI answer names a source, that is a citation.

  • Can they get in
  • Free
  • ~5s

15 of 15 crawlers www.naver.com keeps out of the pages under its front page Measured 2026-08-09

One URL. No sign-in needed.

Add one and we put both in the same table, then list the bots that reach them but not you.

Sign in and you get 10 runs a day. Sign in or create an account

First, a screen we measured ourselves

15 of 15 crawlers www.naver.com keeps out of the pages under its front page

  • citeangle.com blocked, 0 of 15
  • www.naver.com front door only, 15 of 15
  • blog.naver.com blocked, 9 of 15

A checker with two columns prints Korea's largest portal as open. The front page is open; nothing below it is. What an answer quotes usually sits below.

15 AI bots (3 retrieval · 3 user-fetch · 9 training) · read straight from robots.txt · Measured 2026-08-09 · See the source

What you saw here / what the paid tier adds

Free ends here On the paid tier
  • A verdict for all fifteen AI bots
  • One rival on the same screen
  • Nothing held back
  • More rivals on one screen
  • We re-check every month and tell you when a verdict changes

What splits them — How many sites, and how often

Where free ends

Every verdict is on screen. What we sell is deciding what to open, what to keep shut, in what order — and doing it.

The first paid rung is $399. See the diagnostic ladder →

What this check misses

Where this comes from — the crawler operators' own documentation: OpenAI's crawler docs · Google's list of crawlers

See the audit ladder → · Other free tools

Start the free check here

One minute, no card, no meeting. Three questions are on us — write the ones your buyers would put to an AI, and we run them exactly as you wrote them and mail you the results page.

Each question goes to 6 AI answer surfaces. Send 3 and that is 18 observations in all, and we count how many of them came back with your name.

One question is enough to start.

We start from what a marketing chief asks first.

Ready to turn US visibility evidence into a market plan? Tell us the US buyer, target state and decision date.

Request a written proposal