directory-pipeline
A crawl → extract → enrich → resolve → search pipeline on Temporal, where an eval decides how much of the extraction the model is allowed to do. It allowed two fields out of 234.
- result
- Selectors and regex recovered 232 of 234 recoverable fields with zero incorrect values across 36 fixture pages. Only two fields needed the model.
- my contribution
- Built the extraction cascade, evaluated an agentic alternative, and fixed worker startup, retry, and reindexing failures.
- 2026
- ai
how it works
A directory of companies is built by crawling listing pages, pulling a record out of each one, enriching it, deciding which records are the same company, and indexing the survivors behind an OpenSearch alias. Temporal owns the orchestration, so a crawl that dies halfway resumes rather than restarts: one batch of twenty-five exhausts its own retries and fails alone while the rest keep indexing.
Extraction runs in three layers, cheapest first (CSS selectors, then regex over the page's prose, then a model call), and each layer fills only what the one above it missed. The fixtures are the ground truth, since the mock site renders its pages from the same records the extractor is trying to recover, so every company can be rendered under rich markup, prose-only, and near-empty templates and scored per field. Facts a page never prints are excluded from recall, because a stub listing genuinely has no phone number and no layer could recover one; counting those as misses had overstated the case for the model by nearly three times.
The deterministic layers recover 232 of 234 recoverable fields with zero incorrect values, and the model is needed for two, both headcounts written as bare numbers, where "around 500" could as easily be revenue. An agentic extractor was built first and rejected on its own numbers. The eval is also what caught a regression I introduced while closing an address gap: a single pattern accepting both "TX" and "Texas" matched the street as the city on "1 Rockefeller Plaza, New York, NY".
engineering details
- Measured the extraction cascade: 232 of 234 recoverable fields from selectors and regex, zero incorrect values across 36 pages, and the model needed for two.
- Built the agentic extractor and rejected it on the numbers: $2.56 for one company, six tool calls including three 404s, and it read an unrelated company's page, which in a pipeline that then resolves duplicates is a correctness problem rather than waste.
- Found the Dockerised worker had never once started: it crash-looped on a TypeError while 144 tests, an in-process smoke run and the image build all stayed green, because none of the three ever ran the container that ships. Added a CI job that boots the stack, and confirmed it against the bug by reverting the fix.
- Fixed a retry policy that named an exception class nothing raises, so terminal 404s retried six times over ten minutes; the verdict now travels with the exception, and a test asserts every entry in that list resolves to a real class.
- Stopped a reindex retry from starting a second concurrent copy of a multi-hour index build. The index name is unique by construction, so a retry minted a new target and left the first copy filling an index nothing would swap to.
- Exported latency as histogram buckets rather than percentiles so worker replicas aggregate correctly, since averaging two workers' p95 is not the fleet's p95.
in the product
Why This Ranked
Expanding a result shows the stored document and the BM25 breakdown behind its score (boost, idf, tf and the parameters), so anyone can check why a result ranked where it did.
Scored Per Layer, Per Field
An eval scores each extraction layer per field against the records the fixture pages are rendered from, and separates facts a page never printed from facts the parser missed.
The Agent That Did Not Pay
The agentic version of the extractor cost $2.56 per company, wandered onto 404s, and returned a state name where the index wanted a code. The write-up states what the single spike does and does not prove.
Green Tests, Dead Worker
Unit tests drove workflows through a time-skipping environment and never started a worker; CI built the image and stopped. A job now boots the real containers and asserts the worker logged a start, holds no traceback, and has not restarted.