Glossary

robots.txt

A file at the root of a website that tells crawlers which areas they may visit. It can also allow or block individual AI crawlers such as OAI-SearchBot or PerplexityBot.

GEO servicesAll terms

Related terms

  • OAI-SearchBot

    OpenAI's crawler for ChatGPT search. Allowing it lets your pages be found and cited in ChatGPT's answers.

  • PerplexityBot

    Perplexity's crawler, which indexes pages so they can be found and cited in Perplexity's answers.

  • Content signals

    A robots.txt extension, proposed by Cloudflare in 2025, that states how a site's content may be used: for search, as input to…

  • Google-Extended

    A robots.txt token that lets sites opt out of their content being used to train Google's Gemini models, without affecting Google Search.

  • GPTBot

    OpenAI's crawler that collects web content which may be used to train its models. Blocking it doesn't block ChatGPT search, which uses…

  • llms.txt

    A proposed plain-text file that gives large language models a short guide to a website’s most useful pages. It is quick to…

Work with us

Ready for your next stage of growth?

Tell us where you want to take your brand. Our team will come back within one working day with next steps.

Your privacy

We use cookies to run this site and, with your consent, to understand how it's used and to measure our marketing. Read our cookie policy.

↑↓ to moveEnter to openEsc to closeFuzzy search across every page

Region

We work with clients across the UK and beyond. Prices are in pounds (GBP) and exclude VAT. The site is in English (UK).