Can AI Crawlers Read Your Website? How to Check
You can have the best page in town on a topic and still never be mentioned by ChatGPT. One common reason has nothing to do with content: the AI tool was never allowed in. Many sites block AI crawlers without anyone deciding to.
Crawler accessThe gist
- AI tools can only cite pages their crawlers can reach and read.
- Each AI company uses its own crawlers, and robots.txt can block any of them by name.
- Security and bot-blocking settings at your host can turn crawlers away even when robots.txt allows them.
- Pages that only show content after scripts run can look empty to some crawlers.
Who the AI crawlers are
A crawler is a program that visits web pages and reads them. Search engines have used crawlers for decades. AI companies now run their own, and most run more than one: one that collects pages to train their models, one that finds pages to cite in search answers, and one that fetches a page when a user asks about it.
- OpenAI (ChatGPT): GPTBot, OAI-SearchBot, and ChatGPT-User.
- Anthropic (Claude): ClaudeBot, Claude-SearchBot, and Claude-User.
- Perplexity: PerplexityBot and Perplexity-User.
- Google: Googlebot reads pages for Search, including AI Overviews and AI Mode. Google-Extended is a separate setting that controls whether your pages can be used for Gemini.
- Microsoft: Bingbot reads pages for Bing, which also feeds Copilot.
How robots.txt can block them
Every site can have a small file at /robots.txt that tells crawlers what they may visit. A rule like User-agent: GPTBot followed by Disallow: / tells OpenAI's training crawler to stay out of the whole site. Some website builders, plugins, and themes add rules like this by default. Others were added years ago by someone who has since left.
Open your own robots.txt in a browser by adding /robots.txt to your domain. Look for Disallow: / under any of the names above, or under User-agent: *, which applies to everyone.
A single Disallow: / under User-agent: * blocks every crawler that follows the rules, including Google. It is the same mistake that makes sites vanish from search after a redesign.
Security settings that block them
robots.txt is a polite request. Your hosting company or security service can go further and refuse a crawler's visit outright. Many hosting and security services now offer a one-click "block AI bots" setting, and some turn it on for new sites by default. When that happens, robots.txt can say "welcome" while the door is actually locked.
Firewalls and bot protection can also mistake a legitimate crawler for an attack, especially if it visits many pages quickly. If your site uses a security service, check its bot settings, not only your robots.txt.
When the page looks empty to a crawler
Some websites send a nearly blank page and then build the content in the browser with scripts. Googlebot runs those scripts. Many AI crawlers do not, or not reliably. To them, a page like that can look empty. Sites built so the content is in the page from the start avoid the problem. Read more about how your site is built.
How to check your site
- Read your robots.txt. Make sure none of the crawlers above are blocked, unless you chose to block them.
- Check your host and security settings. Look for any setting that blocks AI bots, scrapers, or crawlers, and confirm it is set the way you want.
- View the page without scripts. In your browser, view the page source. If your main text is not in it, some crawlers will not see it either.
- Ask the AI tools. Ask ChatGPT or Perplexity to summarize a specific page on your site. If they cannot, something is in the way. Our guide to tracking AI citations goes further.
- Add an llms.txt. It does not unlock access, but it gives AI tools a clean summary of your site. See what llms.txt is.
Search versus training
Some businesses block AI training crawlers on purpose, and that is a fair choice. Just know what each rule does. Blocking a training crawler does not always block the search crawler from the same company, and blocking the search crawler means that tool cannot cite you in its answers. If you want to be recommended by AI, the search crawlers need to get in.
How we check it
For every site we manage, we allow search engines and AI crawlers in by default, and the Lodestar Dashboard checks crawler access as part of our AI search work, alongside monthly checks of what ChatGPT, Gemini, and Google AI Mode say about you. Not sure whether AI can read your site? Start with a free audit.
Questions we get a lot
Does blocking GPTBot keep my business out of ChatGPT?
It keeps your pages out of OpenAI's model training. ChatGPT search uses a different crawler, OAI-SearchBot, so check both rules to make sure they say what you intend.
Will allowing AI crawlers slow down my website?
For most small business sites, no. Crawlers visit a limited number of pages. A well-built site handles them without any noticeable change.
Does Google-Extended affect my Google rankings?
No. Google-Extended only controls whether your pages can be used for Gemini. Googlebot handles Search, including AI Overviews.
Can my host block AI crawlers without me knowing?
Yes. Some hosting and security services offer a setting that blocks AI bots, and some turn it on by default. Check your host's bot settings as well as robots.txt.
Is robots.txt enough to keep AI out?
Not always. It is a request that well-behaved crawlers follow. To refuse visits outright, it has to be done at your host or security service.
Make sure AI can read you, then get it to recommend you.
We open the doors for AI crawlers and track what ChatGPT, Gemini, and Google AI Mode say about your business every month.