OpenAI's crawler that collects web content which may be used to train its models. Blocking it doesn't block ChatGPT search, which uses OAI-SearchBot.
Glossary
GPTBot
Related terms
OAI-SearchBot
OpenAI's crawler for ChatGPT search. Allowing it lets your pages be found and cited in ChatGPT's answers.
AI crawler
A bot that fetches web pages for an AI company, either to answer questions in real time or to collect training data.…
Content signals
A robots.txt extension, proposed by Cloudflare in 2025, that states how a site's content may be used: for search, as input to…
ClaudeBot
Anthropic's crawler that collects web content for training Claude models. Anthropic uses separate agents for search results and for pages people ask…
Google-Extended
A robots.txt token that lets sites opt out of their content being used to train Google's Gemini models, without affecting Google Search.
Large language model (LLM)
An AI model trained on huge amounts of text to understand and write language, such as GPT, Claude or Gemini. It powers…
Work with us
Ready for your next stage of growth?
Tell us where you want to take your brand. Our team will come back within one working day with next steps.