Skip to content
All notes

The site was blocked from Google twice, and fixing one did nothing

Note 12 of 12
  • #seo
  • #deployment
  • #debugging

robots.txt had Disallow: / and vercel.json set X-Robots-Tag: noindex on every route. Removing either one on its own changes nothing.

Went looking for why almost nothing on this domain was in Google’s index. Found Disallow: / in robots.txt. Removed it, redeployed, requested indexing, waited a week. Still nothing.

The second block was in vercel.json. A headers rule matched the source pattern /(.*) and sent X-Robots-Tag: noindex on every response the domain served.

The two blocks fail at different stages, which is why removing one produced no visible change. robots.txt stops the crawl. The header stops indexing of a page that has already been crawled. While robots.txt was in place, Googlebot never fetched a page, so it never saw the header. Fixing robots.txt did not unblock the site. It made the second block reachable for the first time.

From the outside both states look identical: pages not in the index, no error anywhere, deploys green.

The check I should have run first:

curl -sI https://samuelhuang.org/ | grep -i x-robots-tag

That is one line and it would have shown the header on day 1. I spent the week assuming the fix had not propagated yet. It had propagated. There was just a second door.

Back to all 12 notes