Brave’s index sale to AI firms forces a rethink of page visibility
Brave’s crawler lacks a name, so robots.txt blocks won’t work. Here’s how to manage your pages and protect SEO.
Brave announced that it is selling access to its search index to AI companies such as Cohere, Mistral, Perplexity and You.com, via the Brave Search API. The index holds over 30 billion pages and is refreshed with millions of new pages each day. Because Brave’s crawler does not identify itself with a unique user‑agent, the usual robots.txt method of blocking a specific bot no longer works. The New York Times lawsuit, reported by Search Engine Journal, highlighted that site owners cannot block Brave without also blocking Google, which Brave says it follows the same rules as.
Why this matters: AI assistants that rely on Brave’s index can pull excerpts from your pages and use them to answer user queries. Those answers may appear without a link back to your site, meaning you lose the click‑through that traditional search results provide.
What the Brave index sale means for SEO
* Because the crawler lacks a name, you cannot add a line to robots.txt to block it alone. The only way to keep a page out of Brave’s index is to use a noindex meta tag, which also removes the page from Google’s results.
* AI providers receive “extracted page chunks and source metadata,” allowing them to craft answers without sending traffic back to you. The content they use comes directly from the indexed copy, not a live fetch of your site.
Practical steps to protect your organic traffic
- Run a site: query on search.brave.com to see which of your pages are currently in Brave’s index.
- For pages you do not want AI assistants to cite, add a <meta name=\"robots\" content=\"noindex\"> tag. Remember this also removes them from Google.
- Review your robots.txt file to ensure you are still allowing Googlebot and other legitimate crawlers the access you want.
- Keep your most important pages well‑structured with clear headings and concise meta descriptions; this helps both Google and AI providers surface the right content.
- Use canonical tags to signal the preferred version of a page, reducing the chance that duplicate copies are indexed and used in AI answers.
What to do now
Start with a quick scan of your domain on Brave’s search. Flag any high‑value pages that appear and decide whether you’re comfortable with them being used in AI answers. Apply noindex tags only where you can accept the loss of Google visibility. For the rest, focus on strong on‑page SEO—clear headings, concise meta descriptions and well‑implemented schema—so that when AI does pull content, it reflects the most accurate brand message.
If you need help auditing your site, updating meta tags, or implementing a scalable SEO strategy that accounts for this new AI‑driven distribution channel, our SEO services are designed to keep your pages indexed where it matters and protect the traffic you earn.
The services behind this post.
Frequently asked questions.
No. Brave’s crawler does not use a distinct user‑agent, so robots.txt cannot target it alone. Using a noindex tag will block the page for both Brave and Google.
Yes, a noindex tag tells all major search engines, including Google, not to index the page. Consider whether the page is essential for organic traffic before applying it.
Talk to the team that does this every day.
Tell us what you're trying to grow. You'll get a plan, not a pitch.
Contact Us