RenderCache Now Checks That a Crawler Is Who It Says It Is
Anyone can send a request that says it’s GPTBot. Until now RenderCache took that at its word: a user agent naming a crawler got the saved copy. That’s mostly harmless, since the saved copy is public anyway, but it means a scraper can pull your pre-rendered pages just by borrowing a name, and your delivery log can’t tell real crawler traffic from imitations.
Version 0.3.0 adds a check, and settings you can change without a restart.
Where a crawler really comes from
Most crawler operators now say where their crawlers come from. Google, Bing, DuckDuckGo, OpenAI, Anthropic, Perplexity, Apple and Common Crawl publish their addresses as JSON lists. Google, Bing, Apple, Common Crawl, Yandex and Baidu also document a reverse DNS check: the address’s name has to end in their domain, .googlebot.com for Googlebot, and that name has to point back to the same address.
Turn on verify_crawlers (or start with --verify-crawlers) and RenderCache uses both. It downloads the lists, refreshes them daily and keeps the last good copy on disk for the next start. A request naming one of these crawlers gets the saved copy only when its address is on the list or passes the DNS check. Otherwise it gets the live app, the same thing any visitor gets, marked BYPASS with the reason unverified.
That last part is deliberate. A failed check never costs anyone a wrong page, only a slower one. The live app is always correct. And a crawler still never waits for Chrome: reverse DNS lookups take two seconds at most, sixteen at a time, and their answers are kept for up to six hours.
Before this release, all eleven lists were downloaded from the operators’ own addresses by a test run on GitHub, from Google’s 317 ranges down to Perplexity’s ten for its user-triggered agent.
Crawlers nobody can check
Some crawlers come with no published way to check them: Meta’s, Bytespider, Amazonbot (Amazon publishes addresses as web pages, not lists), and the link-preview bots of Slack, Discord, LinkedIn and X. For those, unverifiable_crawlers decides. snapshot, the default, keeps today’s behaviour, so link previews keep working. live sends them to the app.
Verification is off by default. Upgrading to 0.3.0 changes nothing until you switch it on.
Behind a proxy
Behind nginx or the Cloudflare Worker, RenderCache checks the visitor’s address from X-Forwarded-For, never the proxy’s. It trusts that header only from peers in trusted_proxies or with the edge token, and takes the last address that isn’t a proxy, so a client can’t put an address of its own choosing in front. Both deployment examples already send the right header.
Change settings without a restart
Send SIGHUP, or POST /reload to the admin listener, and RenderCache reads its config file again. Bot patterns, bypass paths, ignored query parameters, trusted proxies, tokens, the snapshot age and the verification settings apply at once. Requests already in flight finish with the settings they started with. Anything that only takes effect at start, such as the listeners or Chrome, is listed back to you as needing a restart, and a broken file changes nothing.
Reviewed twice, and one older bug
An independent review went through the new code twice before release. It found a reload that could have removed the admin token while the admin listener was still public, and broad bot patterns such as bot that let a fake GPTBot through. Both are fixed, with tests.
Testing also turned up a bug from before this release. If a refresh saved a page at the exact moment a crawler was opening it, the read could fail and the crawler got a cold miss instead of the page. That’s fixed too. The test results show all fourteen promise tests passing on Linux, macOS and Windows, two of them new for this release.
Archives for all three systems are on the download page, and the reference lists the new settings.