A robots.txt extension, proposed by Cloudflare in 2025, that states how a site's content may be used: for search, as input to AI answers, or for AI training.
Glossary
Content signals
Related terms
robots.txt
A file at the root of a website that tells crawlers which areas they may visit. It can also allow or block…
Cloudflare
A company providing CDN, DNS and security services in front of websites, protecting and speeding up a large share of the web.
Duplicate content
The same or very similar content at more than one address. Search engines choose one version to show, which may not be…
Answer engine optimisation (AEO)
Making content easy for answer engines and voice assistants to quote: direct answers, clear questions and structured data. Often used to mean…
Google-Extended
A robots.txt token that lets sites opt out of their content being used to train Google's Gemini models, without affecting Google Search.
llms.txt
A proposed plain-text file that gives large language models a short guide to a website’s most useful pages. It is quick to…
Work with us
Ready for your next stage of growth?
Tell us where you want to take your brand. Our team will come back within one working day with next steps.