devrives

robots.txt Per-Bot Groups Silently Drop Your Shared Rules (Plus Naver and Daum)

9 min readdevrives

TL;DR

  • A crawler obeys only the groups naming it. The User-agent: * group applies only when no group matches, and it is never merged with a bot's own group.
  • Per Google's open-source parser, which follows RFC 9309, our old robots.txt let Googlebot, Bingbot and GPTBot into /api/ because each had its own Allow: / group, while Naver's Yeti and Daum, which the template forgot, were blocked.
  • Any per-bot group you keep, for example one that only adds Crawl-delay for ClaudeBot, must repeat every shared Disallow line.
  • Python's urllib.robotparser applies the first matching rule, not the longest, so it can report the opposite of what Googlebot does. Test with Google's parser instead.

The moment your robots.txt contains a User-agent: Googlebot group, Googlebot stops reading the User-agent: * group. Every Disallow you wrote for "all crawlers" no longer applies to it. A bot's own group is never merged with the * group: the crawler uses the group that names it and ignores the wildcard one.

The file still validates, nothing errors, and the rules look right when you read them. That is why this mistake survives for months. Ours did.

The file we shipped

The robots.txt on devrives.com from December 2025 came from one of those "SEO-optimized robots.txt" templates. It was 94 lines with 18 User-agent lines: one wildcard group and 17 per-bot groups. Condensed:

User-agent: *
Allow: /
Disallow: /api/
Disallow: /unsubscribe/*
Disallow: /verify/*
Disallow: /reset/*
# ...plus /.env, /.git/, /admin/ and others

User-agent: Googlebot
Allow: /
Crawl-delay: 0

User-agent: Bingbot
Allow: /
Crawl-delay: 0

User-agent: GPTBot
Allow: /
# ...14 more: Slurp, Twitterbot, anthropic-ai, Claude-Web, PerplexityBot...

The intent is obvious: allow everything except the API and account links, and make extra sure the big crawlers are welcome. On October 13, 2026 we built google/robotstxt, the C++ library Google open-sourced from its production robots.txt parser, and asked it about each crawler.

Crawler tokenGroup it obeys/api/x/unsubscribe/abc/reset/abc
Googlebotits ownallowedallowedallowed
Bingbotits ownallowedallowedallowed
GPTBotits ownallowedallowedallowed
Yeti (Naver)*blockedblockedblocked
Daum*blockedblockedblocked

These verdicts come from Google's parser. Bing, OpenAI, Naver and Daum run their own parsers, so details can differ, but group selection is the core of RFC 9309 and Naver states that it follows the standard. The raw output looks like this:

user-agent 'Googlebot' with URI 'https://devrives.com/api/x': ALLOWED
user-agent 'Yeti' with URI 'https://devrives.com/api/x': DISALLOWED

Every crawler the template named, to "help" it, got a free pass past the rules. The only bots that behaved as intended were the two Korean search crawlers the template did not mention. Our current file has a single * group, and the same parser blocks /api/x for all five.

How a crawler picks its group

RFC 9309, the standard behind robots.txt, spells it out in section 2.2.1:

  • The crawler looks for groups whose User-agent value matches its product token, compared case-insensitively.
  • If several groups match, their rules are combined into one.
  • Only if no group matches does the crawler fall back to the * group.

Google's documentation says the same thing in plainer words: "Only one group is valid for a particular crawler," and "User agent specific groups and global groups (*) are not combined" (Google Search Central).

Within the chosen group, the longest matching path wins. When an Allow and a Disallow for the same path conflict, RFC 9309 says Allow should be used, and Google applies the least restrictive rule (section 2.2.2). That tie-break matters below.

The merging rule is the part people half-remember. Two User-agent: Googlebot groups in one file do merge. A * group never joins them. In our file, Googlebot's entire rule set was Allow: / and a Crawl-delay line Google does not support.

Doing per-bot rules correctly

There are two safe shapes.

One wildcard group. If every crawler should get the same rules, write only User-agent: *. A group that says nothing but Allow: / adds no permission a crawler did not already have, and it deletes every shared rule for that bot.

Per-bot groups that restate the shared rules. Say you want to block GPTBot but let ClaudeBot crawl at a slower pace:

User-agent: *
Disallow: /api/

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /api/

The GPTBot group blocks the whole site, so restating /api/ adds nothing. The ClaudeBot group exists only to add Crawl-delay, which Anthropic says it honors (Anthropic's crawler page), so it has to carry Disallow: /api/ again. Leave that line out and ClaudeBot can crawl your API. Googlebot ignores Crawl-delay entirely: Google supports only user-agent, allow, disallow and sitemap.

Check tokens against the operator's own docs. Anthropic currently lists ClaudeBot, Claude-User and Claude-SearchBot. The anthropic-ai and Claude-Web groups in our old file are not on that list, so they did nothing for the crawlers Anthropic documents.

If your audience includes Korea: Naver and Daum

English-language robots.txt guides rarely mention Korea's search engines. If your site targets Korean users, you register separately with Naver Search Advisor and Daum Webmaster Tools, and Naver's crawler reads robots.txt with a few rules of its own.

Naver (Yeti). The Naver Search Advisor robots.txt guide (Korean) says Yeti interprets rules according to the robots exclusion standard, so the group logic above applies. It does not describe group selection in its own words. Its "allow only Naver" example, * with Disallow: / plus Yeti with Allow: /, works whether groups merge or not, because Allow wins a tie on the same path. Do not read it as proof either way.

Where Naver clearly differs from Google is in how it treats the response:

Response for /robots.txtNaver YetiGooglebot
4xxallow everythingno restrictions (except 429)
5xxdisallow everything (may briefly reuse earlier rules)stops crawling the site for the first 12 hours and keeps retrying
HTML instead of plain textmay treat the file as missing, so allow everythingtries to extract rules from it
Redirectsfollows up to 5, then allows everythingfollows at least 5, then treats it as a 404

Two practical consequences. A short 5xx outage on robots.txt can pause Naver crawling for the whole site. And a single-page app server that answers unknown paths with index.html can hand Yeti an HTML robots.txt that it ignores. Serve it as text/plain with a 200.

Naver also notes that rules are per host, protocol and port: http://www.example.com/robots.txt does not cover https://example.com/. Every subdomain needs its own file.

Daum (Kakao). We could not find official Daum documentation on group handling. In the test above we used Daum, the product name in its user-agent string Daum/4.1. Daum Webmaster Tools verifies ownership with a single comment line in robots.txt, #DaumWebMasterTool:.... Lines starting with # belong to no group and change no rule, so append it to the end of the file and leave everything else alone. All four of our sites carry one (devrives.com/robots.txt, cctv.devrives.com/robots.txt, tickmark.devrives.com/robots.txt, safelife.devrives.com/robots.txt).

Testing a robots.txt before you ship it

Check the response first. On Naver, status code and content type can override everything inside the file.

curl -sI https://example.com/robots.txt | grep -iE '^(HTTP|content-type|location)'
curl -s -o /dev/null -w '%{num_redirects} %{http_code}\n' -L http://example.com/robots.txt

Our five hosts (devrives.com, www.devrives.com, cctv.devrives.com, tickmark.devrives.com, safelife.devrives.com) all ended in a 200 with text/plain on October 13, with one redirect from plain http (two for www).

Ask Google's parser, path by path. This is how the table above was made:

git clone https://github.com/google/robotstxt.git && cd robotstxt
mkdir c-build && cd c-build
cmake .. -DROBOTS_BUILD_TESTS=OFF && make robots-main
./robots /path/to/robots.txt Googlebot https://example.com/api/x

Don't trust urllib.robotparser for this. Python's standard-library parser walks a group's rules in file order and returns the first match (Entry.allowance in CPython 3.12). It does not apply longest-match precedence. Our old * group started with Allow: /, so Python said an unnamed bot could fetch /api/x. Google's parser says it is blocked. Same file, opposite answers.

Then confirm with the search engines. The robots.txt report in Search Console settings shows which robots.txt files Google found, when it last fetched each and any parse issues, and lets you request a recrawl. Naver Search Advisor's robots.txt tool can request a fetch and test whether a given URL is crawlable.

What else we removed from that file

  • Crawl-delay: 0 and Host: lines. Neither is a field Google supports, so they did nothing for Googlebot.
  • Disallow: /.env, /.git/, /admin/. robots.txt is public, so listing sensitive paths advertises them. Google states robots.txt "is not a mechanism for keeping a web page out of Google" (introduction to robots.txt). Block those paths at the server.
  • Blocking pages we only wanted out of the index. A blocked page's noindex tag is never seen, and the bare URL can still be listed if other sites link to it. Our privacy policy and terms pages now use noindex and stay crawlable.

Sources

FAQ

How long until a robots.txt change takes effect?

Google generally caches robots.txt for up to 24 hours, and RFC 9309 says crawlers should not use a cached copy longer than that unless the file is unreachable. To speed it up, request a recrawl from the Search Console robots.txt report, or use the fetch request in Naver's robots.txt tool.

Are paths case-sensitive too?

Yes. User-agent names match case-insensitively, but Google compares Allow and Disallow paths case-sensitively, and RFC 9309 recommends the same. Disallow: /API/ does not block /api/.

Share

Back to Blog