PermitParser exists because a roofing permit filed in Branford and one filed in Wallingford describe the exact same kind of event — a homeowner spending real money on their house — but live in two completely different systems, in two completely different formats, neither of which is built to be queried by anyone who isn't standing at a town hall counter.
Municipal data is the purest possible test of resilient ingestion: there is no clean API, no consistent schema, and no vendor who coordinates with any other vendor. One town exports a PDF with a real text layer; the next scans theirs from paper. One publishes a table; another hides everything behind a JavaScript single-page app that explicitly asks not to be crawled.
Every piece of this system exists to survive that without a rewrite every time a new town looks slightly different: a raw/normalized table split so a parser bug never means re-scraping, a content-hash ledger so a re-run costs nothing, a portal-type interface so a new town on an already-supported vendor is a config row, not new code.
A newly filed roofing permit, solar installation, or commercial addition is one of the warmest signals in local real estate — a homeowner who is, right now, already spending money on their property. Contractors, installers, and investors currently get this by checking fifteen town websites by hand, or not at all.
PermitParser exposes the same normalized data through a public, filterable API — by municipality, category, valuation range, date range, or a straight-line radius from an address — so that signal is a query, not a weekly ritual of clicking through town portals.
| Layer | Choice | Why |
|---|---|---|
| Database | PostgreSQL 16 + PostGIS | A radius search is a first-class query, not a bolt-on — ST_DWithin against a real spatial index, not a bounding-box hack. |
| ORM & migrations | SQLAlchemy 2.0 (async) + Alembic | Every migration in this repo was generated against a live database and verified with zero drift before being trusted — not hand-written and hoped. |
| Validation | Pydantic v2 | PermitRecord is the hard boundary between whatever a source's HTML or PDF actually says and what the schema is allowed to store. |
| Scraping | httpx + BeautifulSoup, portal-type dispatched | Static-first. A JavaScript-only portal gets a real browser step; a plain HTML table doesn't pay that cost for nothing. |
| PDF / OCR | pdfplumber, PyMuPDF + Tesseract fallback | Most municipal PDF exports have a real text layer. Some scanned agendas don't — both paths are real, tested against synthesized fixtures, not stubbed. |
| Task queue | arq (Redis-backed, async) | The whole stack is already asyncio-native; arq avoids a sync/async seam at the one place — the scheduler — where it would otherwise show up. |
| Logging | structlog | Every line carries bound context — municipality, document hash — so a fleet of scrapers is debugged by grep, not by guessing which town failed at 3am. |
Branford's own building permits are not in this system.
Branford's real permit system is OpenGov Permitting & Licensing, a
JavaScript application with no server-rendered data to parse — and its
robots.txt explicitly disallows automated access. What's
live today is Branford's Planning & Zoning Commission agenda and
minutes archive, a separate, legitimately public, statically scrapable
source. It's a real pipeline running against a real Branford data
source — just not the specific one a JavaScript portal declined to
allow.
Fourteen more towns are named in the roadmap and none of them are built yet. Each one needs its actual portal found and verified before a scraper is written for it — this project does not fabricate coverage it hasn't earned.
Docstrings that explain the "why," not just the "what" — the same standard as every other real engineering decision on this page.