Some websites simply do not want to be scraped. They deploy sophisticated bot detection, serve JavaScript-only content, or block automated access entirely. For months, we fought these defenses with increasingly clever browser automation. With v0.11, we are taking a different approach: when scraping fails, we ask politely via APIs instead.
The Etsy Problem
Etsy is one of the most popular marketplaces for handmade and vintage goods. It is also one of the hardest to scrape. Etsy uses DataDome, a particularly aggressive bot detection service that blocks virtually all automated requests, including headless browsers with stealth plugins.
Our users kept asking us to support Etsy, and we kept running into DataDome walls. Traditional scraping was a dead end. Then we discovered that Etsy offers an Open API v3 that provides listing data including prices, and it is free for applications that meet their terms of use.
When you add an Etsy URL to DealMonitor, we now extract the listing ID from the URL and query the Etsy API directly. The response includes the price as a structured object with amount, divisor, and currency code. No browser needed, no bot detection to bypass, and the data is always accurate because it comes straight from Etsy's database.
This is a pattern we want to apply wherever possible: use the official data source instead of fighting the website.
IsThereAnyDeal: Game Prices Across All Stores
Video game prices are another tricky domain. Games are sold across dozens of digital storefronts: Steam, GOG, Humble Bundle, Epic Games Store, Green Man Gaming, and many more. Each has its own website, its own bot protection, and its own way of displaying prices.
IsThereAnyDeal (ITAD) is a service that aggregates game prices across all major stores. We integrated their API in a two-step process: first, we look up the game's unique identifier from its title or store URL. Then we query the price overview, which returns the best current price across all stores.
This means when you track a game on DealMonitor, you are not just tracking one store. You are effectively tracking every store that sells that game. If the price drops anywhere, you will know about it.
Multi-Price Per URL
A single product page often contains multiple prices: the main product, different sizes or colors, bundle offers, or subscription tiers. Until v0.11, DealMonitor tracked one price per URL. That was a limitation.
We introduced a concept called candidate keys. Each price candidate on a page now has a unique fingerprint based on its HTML context: the tag type, CSS classes, and selector path. When our browser extension detects multiple distinct prices on a page, you can choose which one to track. Each tracked price gets its own history, its own target price, and its own notifications.
This is particularly useful for shops that show a product in multiple configurations. Track the 500g bag and the 1kg bag separately, each with its own target price. Or track both the monthly and annual subscription price of a software product.
Chrome Pool with Health Monitoring
For shops that still require a full browser session, we overhauled our Selenium infrastructure. Instead of launching a new Chrome instance for every scrape (expensive and slow), we now maintain a pool of persistent Chrome instances that are reused across scrapes.
The pool includes health monitoring: each Chrome instance is regularly checked for responsiveness. Instances that stop responding or consume too much memory are automatically killed and replaced. Between scrape jobs, idle instances perform light browsing activity to maintain realistic session histories.
The auto-restart mechanism ensures that even if a Chrome instance crashes mid-scrape, the pool recovers automatically. The health status of the Chrome pool is visible in our internal monitoring dashboard, giving us early warning of systemic issues.
Shop Scrape Mode Auto-Learning
Not every shop needs a full browser. Some serve their prices in plain HTML or structured data that a simple HTTP request can retrieve. But until now, we had no systematic way to know which shops needed Selenium and which did not.
v0.11 introduces scrape mode auto-learning. Every shop domain starts in "auto" mode: we try HTTP first, and fall back to Selenium if needed. After five consecutive successful HTTP-only scrapes, the shop is automatically upgraded to "HTTP-only" mode, skipping Selenium entirely for future checks.
If an HTTP-only shop later starts returning challenge pages or empty results, the system resets to "auto" mode and tries Selenium again. This adaptive approach means our scraping infrastructure learns from experience and allocates resources where they are actually needed.
Inline JavaScript Price Extraction
Some e-commerce platforms, particularly Magento and Hyva shops, embed product configuration data directly in JavaScript objects on the page. The visible price is rendered client-side from these objects, which means a simple HTML parse misses it entirely.
We added a new extraction stage that scans inline JavaScript for known patterns like optionPrices objects. When found, we parse the JavaScript data structure and extract variant prices directly. This gives us access to prices that were previously invisible to our pipeline.
JSON-LD Improvements
Structured data on the web is messy. The JSON-LD specification allows for AggregateOffer types with price ranges, @graph wrappers that nest multiple entities, and @type arrays like ["Product", "VideoGame"]. We added support for all of these edge cases, which significantly improves price detection on game stores like Humble Bundle and large retailers with complex product catalogs.
We also changed the priority order of our detection pipeline: JSON-LD candidates now take precedence over ML-scored candidates from XHR network traffic. Structured data is intentionally machine-readable and almost always more reliable than prices extracted from dynamic page content.
What This Means for You
- Etsy tracking works reliably thanks to the official API, no more DataDome blocks.
- Game price tracking covers all major stores through IsThereAnyDeal.
- Track multiple prices per product page for different variants or configurations.
- Faster and more reliable scraping with the Chrome pool and auto-learning.
We believe the future of price tracking is a hybrid approach: APIs where available, intelligent HTTP scraping where possible, and browser automation only as a last resort. v0.11 is a big step in that direction.
Start tracking prices from Etsy, game stores, and anywhere else at dealmonitor.app. For the full list of changes, visit our changelog.
