01
Google and Microsoft will honour a single robots.txt signal for opting out of AI training
On 30 October Google and Microsoft announced support for a shared policy token in robots.txt that lets a website state one AI training preference for all participating providers instead of addressing each crawler separately. The proposal is an individual Internet-Draft submitted to the IETF by an Anthropic engineer and has no formal standing yet. Google says it aims to support the token in 2027 and will treat an opt-out as a block on Google-Extended unless a site names Google-Extended explicitly. Microsoft says it will start honouring the new preferences on 1 February 2027, adds a separate page-level control for use of content in AI answers, and states that neither choice affects eligibility for normal Bing search results.
What it means for you
If you run a website, decide now whether you want your content used for AI training and whether you still want it cited in AI answers, because these are becoming separate choices. Nothing changes until 2027, and the signal is voluntary, so existing per-company controls remain worth keeping.
Sources
- Supporting a shared AI training control for websites | Google Search Central Blog (developers.google.com) (opens in a new tab)
- Your content, your choice: clearer preferences for training and grounding (blogs.bing.com) (opens in a new tab)
- Machine-Readable Training Preference Signal (datatracker.ietf.org) (opens in a new tab)