A Field Guide to
Web Crawling

Four instruments for collecting, cleaning, and cataloguing what lives on the open web - built on Crawl4AI. No two specimens behave alike; each one is suited to different terrain.

View on GitHub Examine the specimens

Catalogued Specimens4 of 4 documented

PLATE I news_digest.py

The Digest Crawler

Habitat: scattered news and blog sources, read once a day

Visits many sources at once, concurrently
Strips nav, ads, and boilerplate automatically
Optionally filters by topic relevance (BM25)
Produces one clean markdown digest

$ python news_digest.py \ --urls urls.txt --query "AI regulation" \ --out digest.md ✅ Wrote digest.md (5/5 sources)

PLATE II price_tracker.py

The Price Sentinel

Habitat: product pages with a repeating template, watched over time

One LLM call teaches it the page's shape
Every run after that is fast and free
Keeps history and flags only what changed
Scales to hundreds of product URLs

$ python price_tracker.py track \ --urls products.txt --schema schema.json ⚠ 2 changes detected - see latest.md

PLATE III docs_to_kb.py

The Docs Cartographer

Habitat: documentation sites, mapped in full and brought home

Follows only in-scope internal links
Saves each page as clean, standalone markdown
Builds an index of everything it found
Ready to feed into another LLM or search index

$ python docs_to_kb.py \ --start-url docs.example.com --max-pages 100 ✅ Saved 87 pages to ./kb (see index.md)

PLATE IV competitive_snapshot.py

The Rival Scout

Habitat: competitor pages, no two laid out the same way

Reads pricing, features, and positioning by meaning, not markup
Works across pages with completely different structures
Same fields extracted from every rival, side by side
Outputs one plain comparison table

$ python competitive_snapshot.py \ --urls competitors.txt \ --fields "pricing_tiers,key_features" ✅ comparison.md written

Before you head out

All four instruments share the same field kit. Set it up once.