ArtificialGuyBR

Home / Blog / Fixing things

Agent Scrape Kit: discover a site once, extract from it forever

Two LLM-driven skills for agents: one reverse-engineers a site into a blueprint with endpoints, auth and pagination; the other runs that blueprint determinis…

2 sources cited Fixing things

On this page

  1. TL;DR
  2. What it is and who it is for
  3. Two skills, two jobs
  4. The blueprint
  5. How to use it
  6. When to use it, when not to
  7. Sources
  8. FAQ

Agent Scrape Kit: discover a site once, extract from it forever

TL;DR

  • The kit is two skills instead of one scraper: scraper-discover spends LLM time once to reverse-engineer a site into a blueprint.json, and scraper-run executes that blueprint deterministically.
  • The result is the pitch in one line: you pay for exploration once and get reliable extraction forever — no browser in the hot path, no LLM tokens on repeat runs.
  • Discovery classifies the site's architecture, probes API endpoints directly, fingerprints anti-bot protection and maps the pagination pattern, then validates the blueprint by extracting at least five records.
  • The run skill handles auth refresh, rate limits with throttling and backoff, and checkpoints every 500 records, so a long extraction resumes instead of restarting.

What it is and who it is for

Agent Scrape Kit is a pair of LLM-driven skills for agents doing systematic web scraping, and the README's diagnosis is the familiar one: most scraping is ad hoc — open a browser, find the element, extract, repeat for the next site — which works and does not scale, because every run pays the same discovery cost again.

It is for the person who scrapes the same five sites every week and has written five scripts that break every quarter, and for the agent that needs data from a site whose API nobody documented.

Two skills, two jobs

scraper-discover is reconnaissance. It classifies the site's architecture by archetype — the README names SSR, SPA REST, SPA GraphQL, static commerce and anti-bot shielded — probes API endpoints directly, capturing network traffic through Playwright only when needed, detects auth mechanisms with CSRF tokens and session cookies, fingerprints anti-bot protection across Cloudflare, Datadome, PerimeterX and Akamai, and identifies the pagination pattern: cursor, offset, page number or infinite scroll.

Then it validates the blueprint by extracting at least five records and saves a reusable blueprint JSON. That validation step is the honest part: a blueprint that has never extracted a record is a hypothesis, not a specification.

scraper-run is execution. It loads and validates the blueprint, executes pagination automatically, handles auth refresh for session cookies, JWT and CSRF, respects rate limits with built-in throttling and backoff, checkpoints every 500 records, and self-heals when it detects stale endpoints, a changed response shape or anti-bot escalation. Output is clean JSON plus CSV.

The self-healing is the design decision worth naming: a scraper that fails loudly on a response shape change is honest, and one that silently writes half the fields is not.

The blueprint

Blueprints are saved to ~/.hermes/scraper/blueprints/<domain>.json and describe the site rather than the script:

How to use it

Discovery is conversational: give the skill a URL and what you want — for example all products with name, price and SKU — and it produces the blueprint. Then the run skill takes the blueprint name and extracts everything. A cron entry turns that into a recurring job with change notification.

# 1. discover a site and produce a blueprint
# 2. run the blueprint to extract everything
# 3. optional: schedule scraper-run every 24h and notify on changes

The scripts behind discovery are scripts/recon_capture.py for Playwright network capture and scripts/detect_antibot.py for anti-bot fingerprinting.

When to use it, when not to

Use it when you scrape the same site repeatedly, when you want the discovery cost paid once instead of every run, or when the site has a JSON endpoint behind a browser page and you want to skip the browser.

Do not use it on a site you will scrape once: the discovery pass is the expensive half, and a one-off extraction does not amortize it. Do not use it on a site whose data requires a browser-only interaction such as a multi-step form either — the run skill is HTTP and extraction, and the README frames the browser as discovery-time.

Sources

  1. Agent Scrape Kit — repository — the two skills, discovery and run behaviors, blueprint format and usage quoted above.
  2. README.md — skills, scripts, usage commands, blueprint example.

Project page: Agent Scrape Kit

For the SEO side of what a crawled site should do afterwards, see the SEO, GEO and ASO skill.

FAQ

Does the run skill use a browser? No. scraper-run executes the blueprint with HTTP calls and extraction only; the browser appears in discovery, when network capture is needed.

Does it cost LLM tokens on every run? No, that is the point: discovery spends LLM time once, and the run is deterministic afterwards.

How is a blueprint validated? By extracting at least five records before the blueprint is saved as reusable JSON.

What does the run skill do at scale? It handles pagination, auth refresh, rate limits with throttling and backoff, and checkpoints progress every 500 records.