Case study
City scale business data scraper
Collects business listings across an entire city from map data, walking the area by region and category rather than by search term, so the coverage is complete.
Overview
Business listings for a whole city, collected rather than compiled. The client needed coverage, not a sample, which changes the problem from “search for the thing” to “cover the area”.
The fourth and smallest of the systems built for this direct client, and a direct descendant of the collection work in the AI tour guide, which walks map regions by coordinate polygon for the same reason.
What I own
The collection strategy, the runner and the output format.
Complexities tackled
Searching gives you a sample; walking the map gives you coverage. Map search returns what it thinks is relevant to a query, ranked and truncated. Ask it for restaurants in a city and you get the ones it wants to show, not all of them. Getting completeness means dividing the area into regions and iterating categories within each, so the boundary of the collection is one you set rather than one the platform decides.
Category lists are the actual index. The category taxonomy is what makes the sweep exhaustive, so it is maintained as its own input rather than hardcoded into the runner. Adding a category is a data change, not a code change.
Long runs need to be resumable. A sweep at this size does not complete in one sitting and will be interrupted. Links are collected first and resolved second, so an interruption costs the current batch rather than the whole run.
Stack
Python for the collection and resolution passes, map platform data as the source, spreadsheet and JSON output for the client’s own tooling.
Working on something similar?
Tell me what you are building and what is in the way. I will tell you honestly whether I am the right person for it.