SEO Rules Are Being Rewritten: Cloudflare, OpenAI and Google Move at Once

Several major shifts are unfolding at once in search marketing — touching who gets to crawl your site, what an AI index actually retains about your pages, and how Google benchmarks your campaigns. A recent roundup from Search Engine Journal lays out these changes in detail.

Starting September 15, Cloudflare will classify Google's and Bing's bots under the same category as "crawling for search and gathering data to train AI." Any site with AI-training-data blocking turned on will now automatically block these bots too — including sites that flipped the older "block AI bots" toggle. Site owners can opt out of the setting before September; several have already reported Googlebot getting blocked, though the exact cause hasn't been confirmed yet.

A study by French consulting firm Resoneo found that sites with no content agreement with OpenAI show up in ChatGPT's internal search index just as readily as licensed partners do. In other words, a content deal isn't a prerequisite for your page appearing in ChatGPT's answers — most responses shown to free-tier users are pulled straight from that internal index, which stores only a page's title and roughly the first 200 characters.

Google, for its part, is rolling out an AI agent in Google Analytics called Ask Advisor, which will compare your campaign performance against anonymized average metrics from similar companies in your industry. Analytics already places your site into a "peer group" based on industry and other signals, and that grouping can be changed — but it's still unclear whether Ask Advisor will draw on that same group or build its own.

At the same time, Google refiled its DMCA (Digital Millennium Copyright Act) lawsuit against SerpApi, bolstering it with clauses drawn from content-licensing agreements — a federal judge had previously dismissed both claims because Google couldn't prove it had rolled out its anti-scraping system with copyright holders' consent. The case will effectively determine who can scrape Google's search results, and at what scale — data that rank trackers, SERP monitoring services, and AI-visibility tools all depend on.

All four developments are connected: Cloudflare now controls which bots can reach your pages, OpenAI decides how much of a page stays in its index, and the courts are settling who gets to scrape Google's results. All of this used to be governed by a robots.txt file or a simple agreement — now those rules are being rewritten from scratch.