Robots.txt for AI: decide if models train on your site

What changes with robots.txt for AI

On 30 October, Google explained on its blog for website owners that it plans to support a shared AI training control in robots.txt, the text file at the root of every website that gives instructions to the bots that crawl it. The proposal builds on work from the IETF's AIPREF working group; the IETF is the body behind many of the internet's standards. Google says it aims to support it alongside other AI providers in 2027, and its crawler documentation narrows that down to early 2027.

Today, anyone who wants to opt out of training has to write a rule for each company, as heise points out: GPTBot for OpenAI, ClaudeBot for Anthropic or Google-Extended for Google, among others. The shared signal uses a single name, __usage_ai-training, defined in a draft submitted to the IETF the same day by Kevin Kelley of Anthropic. It adds no new syntax: it is written like any other robots.txt rule and lets you, for example, say no to training across the whole site except the press area.

Three caveats apply. The draft is an individual proposal which, by its own account, has no formal standing at the IETF yet. Nor does it bind anyone: it expresses a preference, and the draft itself accepts that a private agreement may override it. And it does not replace each company's own rules: according to Google, those will keep working, so you can set a general choice and fine-tune it for a particular provider.

Free recipes, paid courses and a nagging doubt

Imagine Teresa runs a cookery school with two chefs. Every week she posts a free recipe on her website, with photos she takes herself on her phone, and she sells video courses in a members' area. The recipes bring in visitors; the courses pay the bills.

One Tuesday, a student tells her that an AI assistant gave her a rice recipe that looks a lot like Teresa's. Teresa has no idea whether it is a coincidence. Her nephew tells her to block everything and forget about it. But she also wants her school to come up when someone asks an assistant about cookery courses in her city. And she does not know whether one rules out the other.

Why have so few businesses decided what AI does with their site?

Plenty of businesses are in Teresa's position. Someone built the website years ago, nobody has ever opened the robots.txt file, and decisions about AI have been made by default. Until now, deciding was a chore, too: you had to keep track of every company's bots and write a rule for each one.

And three different things get mixed up: your site showing up on Google, an assistant using your page as a source for an answer with a link, and your text and photos being used to train a model. Each is governed by its own rules, and confusing them leads to decisions nobody meant to make.

What is at stake if you do not decide?

If you say nothing, other people decide how your content is used. For a business whose product is that very content, such as recipes, courses, guides or photos, that is a lot to leave to chance.

Going too far the other way has a cost as well. Google's documentation says Google-Extended does not affect a site's inclusion in its search engine and is not used for ranking, so opting out of training will not take you off Google. But in its announcement, Google adds that sites which specifically allow Google-Extended can appear, with a link, in Gemini app answers, and that using the shared signal to opt out of training, without a separate Google-Extended rule, will count as blocking it. Shutting everything off without thinking may keep you out of those answers.

One more warning: robots.txt works on good faith. As heise notes, it is no substitute for a technical access restriction, and only the providers that choose to respect it will do so.

How can you get your site ready before 2027?

There is no need to wait for the shared signal to go live before sorting out your decision. These steps work now:

• Take stock of your content and split it into two piles: your shop window (services, prices, contact details, press) and your product (courses, downloadable guides, paid photos).
• For each part, decide whether you accept it being used to train models and whether you want to appear as a source in assistants. They are two separate questions.
• Open your current robots.txt at yourdomain.com/robots.txt and check which bots it mentions and what it allows them to do.
• Put anything that is genuinely paid for behind a username and password: robots.txt is a request, not a lock.
• Make a note to review it once Google and the other providers confirm they read the shared signal, because the proposal is still under discussion.

For Teresa, the outcome could look like this: the courses page and contact page open to everyone; the free recipes with her training decision written down; the members' area behind its password, as it already is. Three decisions, each with its reason.

What we do at digizone with your website content

At digizone we help businesses make technology decisions with a clear rationale through our technology consulting: we review what your website publishes and what it tells search engines and assistants, and we bring it into line with what you want to protect and what you want to show.

We see it often in projects where content is the heart of the business. For CulturApp we built a web app that turns places of interest into tours with text, images, audio, video and even 3D elements, with a control panel where managers and collaborators create their own content. Managing free and paid access was part of the brief from the start, and separating what is open from what is reserved is the same question that now needs asking about AI.

Start with three rows

You do not need a twenty-page AI policy. At Teresa's school, the first version would be a table with three rows, recipes, courses and members' area, with what she allows and what she does not next to each. With that, whoever looks after her site can write today's provider-by-provider rules and have the shared rule worked out for when it arrives. If she adds a downloadable recipe book next month, she adds a row.

If you would like to see how it would work for you, our free workshop, in Spanish, walks through it step by step.