Update robots.txt to control AI training while keeping SEO intact

Google’s Gary Illyes explained in a Search Central video that robots.txt governs which URLs a crawler may request, but it does not control what happens to the content after it’s fetched. The discussion focused on whether a site can let Google’s search crawler index pages while preventing that same data from being used to train generative models.

Understanding this distinction matters because you want your pages to stay visible in Google Search while you may wish to stop other AI crawlers—or even Google’s own training pipelines—from ingesting your proprietary content.

What the current standard allows

Robots.txt, as defined in RFC 9309, supports four directives: User‑Agent, Allow, Disallow and Sitemap. Anything beyond those lines is ignored by Google’s parser. That means a custom token like “No‑AI‑Chicken‑Training” shown in the video has no effect.

Google‑Extended token for training control

Google’s documentation lists a special token called “Google-Extended.” It is not a separate crawler; it works with Google’s existing user agents. When present, it tells Google whether the crawled content may be used for training future Gemini models and for grounding in Gemini Apps. The token does not influence search inclusion or ranking.

No standard way to signal “train‑ai=n” yet

Illyes and Mozilla’s Martin Thomson are co‑authoring an IETF draft that would add a “Content‑Usage” field (e.g., train‑ai=n) to robots.txt. The draft is still in Working Group status and has not been approved. Until it becomes a standard, any such line will be ignored by most crawlers.

Practical steps for your site

• Identify the specific AI crawlers you want to block (for example, non‑Google models) and add a Disallow rule targeting their exact user‑agent token.
• If you want to keep Google’s search bots but stop Google’s training pipelines, add the Google-Extended token with the appropriate value as described in Google’s own docs.
• For URLs containing non‑English characters or emojis, use percent‑encoded paths in robots.txt. Google treats raw and encoded versions the same, but other crawlers may misinterpret raw characters.
• Keep the file simple: only the four supported directives. Remove any custom lines that are not recognised, as they will be ignored and could cause confusion.
• Test the updated robots.txt with Google’s robots.txt tester in Search Console to confirm that the intended bots are blocked and that Google Search can still crawl the pages you want indexed.

Additional considerations

Because the Content‑Usage draft is not yet a standard, you cannot rely on it to prevent training while still allowing crawling. If you need to protect highly sensitive content, consider serving it behind authentication or using noindex meta tags where appropriate. These methods operate outside robots.txt but give you tighter control over what Google indexes.

Monitor your crawl stats regularly. A sudden drop in indexed pages after adding Disallow rules may indicate you unintentionally blocked a Google search bot. Adjust the User‑Agent strings accordingly.

What to do now

Review your current robots.txt file. Delete any non‑standard directives. Add Disallow lines for any third‑party AI crawler you know is accessing your site, using the exact user‑agent string. If you wish to opt‑out of Google’s training use, insert the Google-Extended token as per Google’s guidance. Run the file through the Search Console tester and monitor crawl stats for any unexpected changes.

Sources

• Search Engine Journal — Google’s Illyes: Robots.txt Covers Crawling, Not AI Training