Real CTI prose defangs IOCs (1[.]2[.]3[.]4, hxxp://, evil[dot]com) so they
don't auto-link in email/chat. A model trained only on canonical inputs
will fail to extract them.
New lines/defang.py: defang_ip, defang_domain, defang_url, defang_text —
four dot-styles ([.], (.), [dot], {.}) plus protocol defanging
(http→hxxp, https→hxxps). Each occurrence picks its style independently
since real advisories don't keep one style across paragraphs.
train.BuildOptions adds defang_frac (default 0.0) and seed; build()
threads options + a seeded Random through the example builders so
the augmentation is reproducible. Only _ex_ioc_extraction reads it
today — output stays canonical so the model learns messy→canonical.
CLI: train-build and train-build-all gain --defang-frac and --seed.
8 new tests including a frac=1.0 / output-canonical integration check.
The pipeline runs but is dormant at defang_frac=0.0 — psyc-v5 dataset
build will set 0.5 once OTX cases land.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Live test against abuse.ch revealed two issues with the stage-19 wiring:
- ThreatFox returns `ioc` (not `ioc_value`) and `first_seen` (not
`first_seen_utc`) — older field names from stale docs. Parser now reads
the real names and falls back to the old aliases defensively. Also
captures `malware_malpedia` (per-family writeup URL) and
`threat_type_desc` for richer downstream prose.
- MalwareBazaar's API expects form-encoded bodies, unlike ThreatFox's
JSON. Extended _http with form_body=; MB fetcher switched to it.
Verified live: 10 ThreatFox cases landed with mixed botnet/malware
classification (4/6 split from threat_type signal — first real
incident-type diversity from a single feed). 10 MalwareBazaar cases
landed with sha256+sha1 hash observables and exe/file_type metadata.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CERT-Bund (authority) requires_approval by default; PSYC_REQUIRE_APPROVAL=1
forces every routable submission through the queue. Courier branches at
execute_routes: approval-required → freeze payload + enqueue, no HTTP; else
submit directly as before. Approve dispatches the frozen payload to mock-cert
and writes the ledger row (detail=approved_by=…); reject writes a ledger row
with the reviewer's reason. CLI: queue / approve / reject. Cockpit /queue
page with POST approve / reject and counts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
38 tests covering the pure worker-line logic: Classifyline rules, Routeline
TLP/country/incident-type gates, Sealine seal/unseal round-trip, Proofline
confidence scoring, Mapline CVEResolver escalation, Trainline dataset
well-posedness (the v1/v3 input-signal bugs are now regression-guarded), and
the Scoutline feed parsers. pytest added as a dev extra.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
