Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot in your robots.txt file.
That lets ChatGPT, Claude and Perplexity find your pages and cite your business.
Block them, and you cannot appear in their search answers.
Many UK trade websites block them by accident.
A security plugin, a hosting default or an old template adds the block, and nobody notices.
This guide shows how to check yours in five minutes.
It also gives you a file you can copy.
The Three Jobs AI Crawlers Do
Not every AI bot does the same thing.
Each AI company runs bots for up to three different jobs.
Training bots collect pages to teach future AI models.
Search bots build the index behind AI search answers.
User bots fetch a page live when a person asks a question.
Blocking a training bot is a different decision from blocking a search bot.
Mixing them up is the most common mistake.
| Company | Training | Search | User fetch |
|---|---|---|---|
| OpenAI (ChatGPT) | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic (Claude) | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None listed | PerplexityBot | Perplexity-User |
Google works differently.
Googlebot crawls for Google Search, and AI Overviews appear inside Google Search.
Google-Extended is a robots.txt token that controls whether Google uses your content for Gemini.
It does not affect your Search rankings.
Blocking Googlebot would remove you from Google, so never do that.
What the Bot Makers Say
OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
It recommends allowing OAI-SearchBot in your robots.txt file.
It says ChatGPT-User fetches start with a person, so robots.txt rules may not apply to them.
Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude search results.
It says blocking Claude-User stops Claude retrieving your pages when someone asks a question.
It also says its bots honour robots.txt.
Read the originals: OpenAI crawler overview and Anthropic bot guide.
The Five Minute Check
1. Open your robots.txt file
Type your domain into a browser, followed by /robots.txt.
For example, yourdomain.co.uk/robots.txt.
If you see a page not found message, you have no file, and bots can read everything.
That is fine.
2. Look for a blanket block
These two lines block every crawler, including every AI bot.
User-agent: *
Disallow: /
Developers often add them while a site is being built.
Sometimes nobody removes them at launch.
If you see them, your site is hidden from search as well as AI.
Ask whoever built the site to remove them today.
3. Look for named AI bots
Search the file for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot.
A Disallow line under any of those names blocks that bot.
Blocking the training bots is a choice you can make.
Blocking the search bots costs you visibility.
4. Check your host, plugin and CDN
Some security plugins, firewalls and CDNs block AI bots before they ever read robots.txt.
Cloudflare, for example, offers a setting to block AI crawlers.
If your robots.txt looks clean but AI tools still cannot see you, ask your host or web developer to check.
A robots.txt File You Can Copy
This version allows every AI search and user bot.
Replace yourdomain.co.uk with your own domain.
A bot follows the group that names it, so these rules apply even if another group blocks everyone else.
# Everyone is welcome by default
User-agent: *
Allow: /
# AI search bots: allow, so you can be cited
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# AI user bots: allow, so live questions can fetch your pages
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
Sitemap: https://yourdomain.co.uk/sitemap.xml
Do you want to block the training bots?
That is your call.
Training bots teach AI models that your business exists.
For most trades, we leave them allowed.
If you want your content kept out of training, add these lines.
OpenAI treats its bots as separate settings, so blocking GPTBot does not remove you from ChatGPT search.
# Optional: keep your content out of AI model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
robots.txt Is Only the First Gate
Allowing a bot does not guarantee you appear in answers.
It only means the bot may read your pages.
Whether it cites you depends on what it finds.
AI tools cross-check your claims against other sources, such as your Google Business Profile and review sites.
Clear schema markup, direct answers and consistent business details all help.
Our guide to getting listed on ChatGPT search covers the next steps.
And this post explains why some businesses get recommended while others are ignored.
Common Mistakes
Leaving a staging file live.
A Disallow: / line from the build stage is the most common cause.
Blocking every AI bot because one article said to.
That removes you from search answers along with training.
Forgetting your host or CDN.
The block may sit outside robots.txt entirely.
Expecting instant results.
OpenAI has said search systems can take about 24 hours to adjust, and citations take longer.
How Potter Gold Digital Handles This
Every AI Search Visibility plan starts with a crawler access check.
Standalone AI Search Visibility plans start from £250 a month for businesses that already have a website.
Local SEO + GEO adds AI visibility to your Google rankings, from £500 a month.
Local SEO + GEO has a 3-month minimum term.
We take one trade per town, so your competitors cannot buy the same slot.
Frequently Asked Questions
Which AI crawlers should a trade business allow in robots.txt?
Allow the search and user bots first.
That means OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.
These bots find your pages and cite your business in AI answers.
Training bots such as GPTBot and ClaudeBot are a separate choice.
Does blocking GPTBot remove my business from ChatGPT search?
Blocking GPTBot does not remove you from ChatGPT search, because OpenAI treats its bots as separate settings.
OAI-SearchBot controls whether you appear in ChatGPT search answers.
GPTBot relates to training, so you can block it and still be found in search.
Does ChatGPT-User obey my robots.txt file?
Not always.
OpenAI says ChatGPT-User fetches start with a person asking a question, so robots.txt rules may not apply.
For search opt-outs, OpenAI points site owners to OAI-SearchBot instead.
If you want to be found, allow both.
How do I check whether my website blocks AI crawlers?
Open yourdomain.co.uk/robots.txt in a browser.
Look for a line reading Disallow: / under User-agent: * or under any AI bot name.
Then check your security plugin, host and CDN settings, because some block AI bots before robots.txt is read.
How long after I change robots.txt will AI tools notice?
OpenAI has said its search systems can take about 24 hours to adjust.
Being cited takes longer.
Schema and crawler access fixes can influence AI answers within 4 to 8 weeks.
Steady citation across ChatGPT, Claude and Perplexity usually takes 3 to 6 months.
Is it safe for a small business to allow AI crawlers?
Yes, for public pages.
Your website is already public, and robots.txt only controls which bots may read it.
Keep private pages behind a login, because robots.txt is a request, not a lock.
Find out if AI tools can read your website
A free audit checks your robots.txt, your schema and your AI visibility signals in one go.
Get My Free Audit & Growth Plan →📞 01902 961078 · kp@pottergold.digital
