Hundreds of local government websites block ChatGPT search while letting Google in

I checked the robots.txt file of every state and local .gov domain on the federal government's official list.

15,331 domains, crawled on October 24, 2026.

For each one I asked two questions. Can Googlebot read the homepage? And can OAI-SearchBot, the crawler OpenAI uses for ChatGPT search?

The Numbers

• 353 sites block ChatGPT search while allowing Google. That's 2.9% of the 12,298 government sites that responded, about 1 in 35. Count only the 6,765 sites that publish a robots.txt and it's 5.2%, about 1 in 19.
• Almost none of them do it by name. OAI-SearchBot is blocked on 423 government sites. Only 11 name it. The other 412 catch it with a single catch-all rule (User-agent: * /Disallow: /) at the bottom of a list of crawlers they let in.
• Google and Bing get in. The independent AI search engines don't. 283 of the 353 sites (80%) let in both Googlebot and Bingbot but block all three of OAI-SearchBot, Claude-SearchBot and PerplexityBot.
• Election offices are no exception. 20 election-office websites do it. That's 5.4% of the 369 with a robots.txt, about the same as every other government site.
• Google almost never gets blocked. Just 72 of the 6,765 robots.txt files (1.1%) block Googlebot from the homepage.

OpenAI is blunt about what blocking its search crawler means:

"Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." (OpenAI, Overview of OpenAI Crawlers)

So when a resident asks ChatGPT search about trash pickup, permits or the next council meeting, the city's own website can't be a cited source in the answer.

Google keeps indexing it the whole time.

The Allowlist Problem

Most of these files never mention AI search at all. They're allowlists.

An allowlist names the crawlers it lets in, then ends with Disallow: / for everyone else. Any crawler not on the list is blocked, including every crawler that didn't exist when the file was written.

When I grouped the 353 sites by the rules in their file (ignoring the lines that change per site, like the sitemap address), two files stood out.

File #1: The Platform Template (183 Sites)

183 of the 353 sites, 52%, serve the exact same file. It shows up in 27 states.

It also explains Minnesota. 63 of Minnesota's 72 cases use it, which is why 22% of Minnesota sites with a robots.txt block ChatGPT search. That's four times the national rate (5.2%).

It appears to be the default file on a website platform widely used by small local governments (Catalis's GovOffice). I checked 13 sites from the platform's own client list: all 13 serve it, identical apart from the sitemap address. The template sites I spot-checked credit the platform in their footers.

Here's who it lets in, and who it doesn't:

Look at CCBot. The file lets it in.

CCBot is Common Crawl's crawler, and Common Crawl's archive is widely used to train AI models. So these sites are open to a crawler that collects AI training data, and closed to the AI search crawlers that would send residents back to them.

File #2: The 2000s Template (59 Sites)

A second, older file covers another 59 sites in 26 states. It lets in Googlebot, Slurp (Yahoo), MSNBot (Microsoft's crawler from before Bing) and ia_archiver (Alexa).

Everything else is blocked. It was written for the search engines of the 2000s.

Together: Two-Thirds of the Cases

Those two files account for 242 of the 353 sites, about two-thirds. Neither one mentions an AI search crawler.

They block AI search the same way they block DuckDuckGo: by NOT listing it.

And because File #1 appears to be a platform default, it also shows up on town websites with .com, .org and .us addresses. Those weren't part of this crawl, so the total number of local government websites affected is higher than 183.

Where It Happens

The highest state rates (states with at least 10 sites publishing a robots.txt): Minnesota 21.9%, North Dakota 19.4%, Maine 13.4%, Vermont 13.2% and Iowa 12.7%.

Several states with dozens of sites have none at all.

Download the state-by-state results (CSV)Every state, with denominators. States with fewer than 10 sites publishing a robots.txt are marked suppressed.

Compared with the Top 10,000 Websites

I ran the same check on the 10,000 most popular websites (Tranco list 262N9). The rate is almost identical: 301 of the 5,820 sites with a robots.txt (5.2%) block ChatGPT search while allowing Google.

The reason isn't.

Big commercial sites mostly block AI crawlers by name. 16.3% block GPTBot, and 78% of those name it directly.

Government sites mostly block AI search because their file only lets in the crawlers it already knows.

Blocking Beyond robots.txt (Indicative)

robots.txt is only half the picture. Plenty of sites sit behind a CDN or firewall that can turn a crawler away no matter what robots.txt says.

So I requested every homepage twice: once with each AI crawler's User-Agent, once with a normal browser.

On 1,166 government sites (17.2% of those with a robots.txt), the GPTBot request was refused while the browser loaded fine. For OAI-SearchBot it happened on 281 sites, on top of the robots.txt blocks above.

Treat those numbers as indicative only. My requests came from my own server, not OpenAI's, and firewalls that verify crawlers by IP address may treat the real ones differently.

How to Fix It (If You Run a Government Website)

1. Open yourdomain.gov/robots.txt.
2. Scroll to the bottom. If it ends with User-agent: * and Disallow: /, every crawler not named above that line is blocked: AI search, plus Apple's and DuckDuckGo's search crawlers.
3. Add this group anywhere in the file. Every other crawler keeps following your existing rules.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

One catch: a crawler named in its own group ignores the * rules. If your file blocks private folders (like Disallow: /admin/), copy those lines into the new group too.

OpenAI says ChatGPT search takes about 24 hours to pick up a robots.txt change.

If a platform vendor hosts your site, the file may be managed for you. Ask them to change it.

Letting AI search in doesn't mean letting AI training in. GPTBot, ClaudeBot, Google-Extended and CCBot are separate crawlers, and you can keep blocking them.

Your robots.txt was written for the search engines of its day. Make sure it knows about the ones your residents use now.

Method

• Domains: all 15,331 state, county, city, special district, school district and interstate domains in CISA's .gov list (cisagov/dotgov-data, current-full.csv, October 24, 2026). Federal and tribal domains are excluded. For comparison: the Tranco top 10,000, list 262N9.
• What I fetched: /robots.txt, following up to five redirects, plus the homepage for the CDN check. Details in how the crawler works.
• How files were read: RFC 9309, the robots.txt standard, with the same group-matching rules asGoogle's parser. A crawler named in its own group follows only that group. Otherwise the * group applies. Only the homepage (/) was checked, so a site that blocks just a subfolder counts as open.
• Templates: files were grouped after removing comments and site-specific Sitemap and Host lines. The platform attribution rests on footer credits on template sites, plus the robots.txt of 13 sites from the platform's client list (13 of 13 identical apart from the sitemap line), both checked October 24, 2026.
• Limits: robots.txt is a request, not a lock. It says nothing about why a site is set up the way it is. Blocks on ChatGPT-User, Claude-User and Perplexity-User (fetches made when a person asks) are stated preferences. Of ChatGPT-User, OpenAI says "because these actions are initiated by a user, robots.txt rules may not apply." Files change. This is a snapshot of one day.
• Data: state-by-state results (CSV). Other aggregate tables are available on request. Individual sites aren't named, and state figures are left out where fewer than 10 sites publish a robots.txt.

How the Crawler Works

If you found this page in your server logs, you're in the right place. The crawler identifies itself with this User-Agent:

AICrawlAccessStudy/0.1.0 (+https://joshblackburn.co/studies/ai-crawler-access; research crawler, reads robots.txt and homepages only)

• It requests /robots.txt, following up to five redirects.
• It requests the homepage a few times: once with a normal browser User-Agent, and once with the User-Agent of each AI crawler your robots.txt allows. This checks whether a firewall turns those crawlers away even though robots.txt lets them in. So you may see requests labeled GPTBot, ClaudeBot or PerplexityBot that came from this study's server, not from those companies.
• It reads nothing else: no other pages, forms, images or logins.
• It sends at most one request at a time to your server, at least a second apart. It stops reading after the first 500 KB of robots.txt and the first 16 KB of the homepage.
• Requests come from 5.161.203.146 (reverse DNS ai-crawler-study.joshblackburn.co).

Opting Out

Email josh@joshblackburn.co with the domains you'd like left out, and I'll exclude them from every future run.

Questions about the study are welcome at the same address.