How to Use 503s and Updated robots.txt Rules to Keep Your Site Crawl‑Friendly
Google reminded webmasters to serve 503 during PHP patch downtime and clarified that robots.txt only controls crawling, not post‑crawl use. Here’s how to apply those rules without hurting SEO.
Google Search Central recently reminded site owners that a 503 status code is the correct response when taking a site offline for emergency PHP patches. The guidance, shared in a LinkedIn post on Oct. 21, says a 503 “ensures already indexed URLs are preserved in Search while you work” and warns against returning generic 404 or 200 pages.
At the same time, Gary Illyes explained that the standard robots.txt directives only tell compliant crawlers what they may fetch; they do not dictate how crawlers use the content after it’s retrieved. This distinction matters now that Google‑Extended tokens are being used to signal training preferences for Gemini models.
Why the 503 Matters for SEO
A 503 tells Google the page is temporarily unavailable, so the existing index entry is retained. Google’s crawl‑rate docs advise limiting 503s to 1‑2 days, otherwise the URL could be dropped from the index.
What Robots.txt Actually Controls
Illyes highlighted that robots.txt is an “access preference protocol.” It can block or allow crawlers, but it does not stop Google from using already‑crawled content for training. The only way to express a training‑preference is via the draft Content‑Usage rule, which is not yet approved. For now, you can use the Google‑Extended token to keep that specific crawler out of certain paths, but it won’t affect whether the content appears in Search.
Practical Steps to Implement the Guidance
- Set up a CDN or DNS‑level rule that returns a 503 for all requests when you deploy a PHP patch. The rule must stay active even if your PHP engine is down, because a 503 triggered by PHP itself would never fire.
- Limit the 503 window to 1‑2 days. Monitor the response with a tool like Google Search Console’s URL Inspection to confirm Google still sees the 503.
- If you need to block a specific AI crawler (e.g., Google‑Extended), add a Disallow line for its user‑agent token in robots.txt. Remember, this only stops crawling, not post‑crawl use.
- Keep your robots.txt file live during the 503 window. Google notes that serving a 503 in response to robots.txt requests blocks all crawling, which could unintentionally hide your entire site.
- After the patch, revert to a normal 200 response and verify that no 404 chains were introduced during the downtime.
What to Do Now
Audit your current patch‑deployment workflow. If you rely on PHP to return the 503, move that logic to a CDN edge rule or a static server that stays up during the patch. Update your robots.txt to include any necessary Disallow entries for AI tokens, but keep the file accessible at all times.
Finally, run a quick crawl check in Search Console after the patch to ensure Google still sees your important pages and that no unexpected 404s appeared. If you notice indexing issues, submit a re‑index request for the affected URLs.
The services behind this post.
Frequently asked questions.
Google’s own crawl‑rate documentation advises limiting 503 responses to 1‑2 days. Extending beyond that window increases the chance that Google will drop the URL from its index.
Talk to the team that does this every day.
Tell us what you're trying to grow. You'll get a plan, not a pitch.
Contact Us