Domain pipeline
1. Import directly into PostgreSQL
The ingest object is one provider file per feed run, not one S3 object and not one API call per domain.
- Download the daily CSV, TSV, ZIP, or gzip file to ephemeral disk.
- Calculate SHA-256 and create a
feed_runsrow. - Decompress as a stream when possible.
- Use PostgreSQL
COPYinto an unlogged staging table. - Validate row count, required columns, encoding, and malformed-row rate.
- Merge new observations and update canonical domains with set-based SQL.
- Commit the run summary, then remove the temporary file.
Preserve the source row in jsonb if replay or audit matters. PostgreSQL is the system of record; S3 can be added later only if retaining the original vendor files becomes useful.
Minimum source fields:
domain
reported_expiration_at
registrar
tld
source
source_row_id
raw_payload
2. Early normalization and filtering
Create lexical, service, and geographic features without external paid calls. Reject unsupported TLDs, poor names, excluded industries, obvious trademarks, irrelevant geographies, and previous rejects.
The output is a lifecycle watchlist, not a registration list.
3. Lifecycle monitoring
Poll registry status on a schedule that increases in frequency as the expected transition approaches.
| State | Suggested check rate | Action |
|---|---|---|
| Expiring or grace period | Daily | Keep or close based on renewal evidence |
| Redemption period | Every 6–12 hours | Refresh cheap evidence and capacity forecast |
pendingDelete | Every 15–60 minutes | Confirm drop window and start full intelligence |
| Available | Immediate | Attempt only if pre-approved |
| Renewed, transferred, or auction-only | Stop | Close with reason |
Rate limits and registry terms override these starting values.
4. Full intelligence
Run expensive calls only on confirmed pendingDelete candidates:
- complete archive and abuse review
- referring domains, anchors, linked pages, and lost links
- predicted service and geo keyword clusters
- city, county, metro, and state search demand
- CPC, commercial intent, local SERPs, and rankability
- trademark and reputation review
- portfolio overlap and expected lead value
5. Rank by drop event
Create a separate queue for each drop window. Rank candidates by adjusted opportunity, evidence confidence, expected lead value, registration feasibility, and portfolio constraints. Keep several approved fallbacks because competitors may capture the first choices.
6. Attempt and record
Immediately before the window, refresh status and disqualifiers. At the predicted time, submit idempotent registration attempts through configured registrars. Stop further attempts after confirmed success. Record every request, response, price class, and outcome.