Over the past few days you've been sending us the same kind of messages: "My Otto tracker has been stuck on the same price for days," "Adidas shows €6 even though the item costs €70," "Why isn't this price moving?" The answer is the same in all three cases — and it has nothing to do with our scraper, it has to do with the fact that the shop deliberately doesn't let us in. Here's what's actually happening, and what we shipped a few days ago to handle it properly.
How a price gets refreshed in the first place
When your tracker is due for a check, this little pipeline runs in the background:
- HTTP request — we fetch the product page like a regular browser and read the source
- Price detection — our ML model finds the price in the page, compares it against the reference price, confirms the match
- Headless-browser fallback — if the fast HTTP path comes up empty (too much JavaScript, dynamic pricing), we render the page in a real browser on a separate server
For the vast majority of shops — Amazon, IKEA, MediaMarkt, Cyberport, Zalando, hsnstore, and a few thousand others — this works fine. You never notice the pipeline, the price is just current.
What's different about Otto and Adidas
Otto and Adidas (and a handful of other big brands) have added a particular kind of protection in recent months: commercial bot detection. The specific vendors are Kasada (Otto) and DataDome (Adidas). These aren't simple captchas — they're sophisticated fingerprinting systems that evaluate hundreds of browser signals (canvas rendering, WebGL, audio context, timing patterns, mouse movement) and decide in milliseconds: human or machine?
Our answer is always the same: machine. Which is true. And then this happens:
Instead of the real product page (typically 200–400 KB of HTML with images, descriptions, prices), we get back a tiny challenge page — at Otto that's exactly 912 bytes. Content: a bit of JavaScript initialisation, a hint that a browser should solve a challenge. No price, no product name, nothing useful.
The real problem wasn't the wall — it was our reaction to it
Until recently our scraper treated these 912-byte pages the same as any other "empty" response: no price found, try something else. In practice that meant: after three such empty attempts, the system assumed the headless browser was broken, switched to a different strategy, found nothing there either, switched back, and so on — endless back-and-forth between strategies that all have zero chance of working.
From your perspective: price stuck for days, occasional "tracker is having issues" emails, sometimes a false mismatch alert when a stray digit got extracted from the 912 bytes (which was exactly the Adidas case: from a 403 error page our ML model fished out "6.0" and reported it — for a shoe that costs €70).
What we changed
Instead of blindly retrying, the scraper now recognises the typical signatures of these bot walls:
- For Kasada: characteristic
KPSDKinitialisation,KP_UIDztokens in script URLs, hidden iframe - For DataDome: references to
js.datadome.co,dd-captcha-containerelements,ddjskeyconfiguration
As soon as one of these signatures shows up in a short response (< 10 KB), it's clear: this is a challenge page, not a product page. Three direct consequences:
- No more failure counters — the shop doesn't get bounced between scraping strategies because no strategy switch will fix the problem
- No more false mismatch alerts — we know there's no real price, so we don't alert on one
- Visibility in the daily status — our daily health report now shows a line
🛡️ Bot protection blocks N shops, broken down by vendor
What this means for your tracker
If you have a tracker for otto.de or adidas.de that hasn't updated since May 24th: it's not the tracker, it's that the shop systematically locks us out. We mark that cleanly internally now, you won't get "tracker broken" alerts, and your last stored price stays put.
But it also means honestly: as long as these bot walls are active, we can't actively track the price. We don't attempt technical bypasses — that would be an endless arms race, and the bot-protection vendors ship updates weekly.
What we plan to do instead
The clean way to get Otto and Adidas prices is via official product feeds through affiliate networks like Awin. Both shops offer daily XML feeds with all their products and prices — no scraping, no bot walls, just reliable data. We're applying for inclusion; once approved (typically 1–4 weeks), trackers for those shops will be served via the feeds.
Until then: a bit of patience. And if you've been wondering whether you did something wrong — you haven't. Neither have we. Some doors are just closed.
While we were in there: backend cleanup
While looking at the bot walls we also noticed our backend container never quite cleaned up its embedded Chrome pool — over long uptimes it accumulated memory until the dashboard would hang. So we did two things: a safety net that restarts the container before it hits its memory cap, and (the cleaner step) took Chrome out of the backend entirely — the running browser now lives only in the scheduler container, where it belongs. If you experienced dashboard hangs in recent days: those should be gone now.
What's next
Early June: affiliate applications for the most important shops. In parallel: persistent browser sessions in the scheduler (for more stable detection on dynamic shops) and the Microsoft Edge version of our browser extension. If you have topics or shops that matter especially to you — a quick email to [email protected].
