Case study
Quran.com collector
Collects verses, translations and metadata into a structured, queryable local set, with a command line interface over it and markdown conversion for the output.
Overview
The text is freely readable online and awkward to work with in bulk. This produces a structured local copy: verses, their translations and the metadata around them, in a form that can be queried rather than only browsed.
Complexities tackled
Structure is the whole deliverable. Anyone can save pages. The value is in the shape: verse identity, the relationship between a verse and each of its translations, and metadata that survives being pulled out of the page it was rendered in. The collector validates that shape as it goes rather than trusting it.
Two collection paths for two kinds of page. Content that is present in the response is fetched directly, which is fast and cheap. Content that only exists after the page runs goes through a browser. Using the browser for everything would be simple and needlessly slow.
A command line interface makes it reusable. Wrapping the collectors in a proper CLI, with structured logging behind it, is the difference between a script you ran once and a tool you can point at a different section a year later.
Stack
Python, Requests and BeautifulSoup for the direct path, Playwright for pages that need a browser, Typer for the interface and Rich and loguru for output and logging.
Working on something similar?
Tell me what you are building and what is in the way. I will tell you honestly whether I am the right person for it.