Case study
Keyword exhaustive database scraper
Reconstructs a site's entire database by exhausting its keyword space, working around a hard cap of 225 results per response and the absence of any listing endpoint.
Overview
The client wanted a complete copy of a site’s database. The site offered no listing, no index and no export. It had a search box that returned two hundred and twenty five results at a time and loaded the next batch when you scrolled.
The deliverable was the script and a SQLite database with everything in it.
What I own
The collection strategy, the script and the resulting database.
Complexities tackled
With no index, coverage has to be constructed. You cannot ask a site like this for everything it has, so completeness comes from querying it enough different ways that every record surfaces at least once. That turns the job into a search-space problem: choose keywords, track what each one returns, and keep going until new queries stop producing new records.
A result cap is a signal, not just a limit. A query that comes back at exactly the cap almost certainly has more behind it and needs narrowing; one that comes back under the cap has been exhausted. Treating the boundary that way is what makes the sweep terminate with a defensible claim of completeness rather than an optimistic one.
Deduplication belongs in the database, not the script. Records surface repeatedly across overlapping queries. Letting the database enforce uniqueness keeps the collection simple and makes reruns safe.
Stack
Python with Requests for collection, SQLite as the deliverable store.
Working on something similar?
Tell me what you are building and what is in the way. I will tell you honestly whether I am the right person for it.