v5 trained on 598 ex/task (20× v4's 30), with --defang-frac 0.5 over
the new ThreatFox + MalwareBazaar + OTX corpus. Final train_loss 0.3225
vs v4's 0.7397 (56% reduction), 60m20s wall clock on a 3090.
Live eval before swap:
- severity (botnet, ONLINE): v4 high / v5 high — tied, both correct
- ioc_extraction with defanged input (hxxps://, [.], (.), [dot]):
v4 kept hxxps:// in output (failed canonicalization)
v5 returned canonical https:// — defang training paid off
- ioc_extraction on real OTX-style prose (never trained on this shape):
v5 cleanly extracted 2 domains + 1 IP + 1 SHA256 + 1 CVE
Cockpit /api/inference-status confirms the swap:
{"online":true,"adapter":"/data/adapters/psyc-v5/final"}
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The /pulses/subscribed endpoint enumerates every curated feed a fresh
account is auto-subscribed to. On its own that's enough to 504 from
OTX's backend regardless of client timeout. Narrowing by
modified_since=now-7d brings the response back to a single-second fetch.
Also: _http now accepts params + per-call timeout overrides (OTX uses
120s). The CLI --limit still slices post-fetch.
Verified live: 10 OTX pulse-cases ingested, each carrying real
paragraph-form descriptions (Mirai, macOS Stealer, FlowerStorm PhaaS,
Vidar v1.5, manufacturing intrusion) — exactly the real-prose source
the IOC extractor's been missing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Real CTI prose defangs IOCs (1[.]2[.]3[.]4, hxxp://, evil[dot]com) so they
don't auto-link in email/chat. A model trained only on canonical inputs
will fail to extract them.
New lines/defang.py: defang_ip, defang_domain, defang_url, defang_text —
four dot-styles ([.], (.), [dot], {.}) plus protocol defanging
(http→hxxp, https→hxxps). Each occurrence picks its style independently
since real advisories don't keep one style across paragraphs.
train.BuildOptions adds defang_frac (default 0.0) and seed; build()
threads options + a seeded Random through the example builders so
the augmentation is reproducible. Only _ex_ioc_extraction reads it
today — output stays canonical so the model learns messy→canonical.
CLI: train-build and train-build-all gain --defang-frac and --seed.
8 new tests including a frac=1.0 / output-canonical integration check.
The pipeline runs but is dormant at defang_frac=0.0 — psyc-v5 dataset
build will set 0.5 once OTX cases land.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Live test against abuse.ch revealed two issues with the stage-19 wiring:
- ThreatFox returns `ioc` (not `ioc_value`) and `first_seen` (not
`first_seen_utc`) — older field names from stale docs. Parser now reads
the real names and falls back to the old aliases defensively. Also
captures `malware_malpedia` (per-family writeup URL) and
`threat_type_desc` for richer downstream prose.
- MalwareBazaar's API expects form-encoded bodies, unlike ThreatFox's
JSON. Extended _http with form_body=; MB fetcher switched to it.
Verified live: 10 ThreatFox cases landed with mixed botnet/malware
classification (4/6 split from threat_type signal — first real
incident-type diversity from a single feed). 10 MalwareBazaar cases
landed with sha256+sha1 hash observables and exe/file_type metadata.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CERT-Bund (authority) requires_approval by default; PSYC_REQUIRE_APPROVAL=1
forces every routable submission through the queue. Courier branches at
execute_routes: approval-required → freeze payload + enqueue, no HTTP; else
submit directly as before. Approve dispatches the frozen payload to mock-cert
and writes the ledger row (detail=approved_by=…); reject writes a ledger row
with the reviewer's reason. CLI: queue / approve / reject. Cockpit /queue
page with POST approve / reject and counts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three load-bearing operational pieces before any new features:
* .env.example committed, .env gitignored — per-developer API keys
(THREATFOX_AUTH_KEY, OTX_API_KEY, NVD_API_KEY) ready for the registrations
ahead; python-dotenv loads it in the venv CLI; compose picks it up via
env_file: .env on the cockpit service.
* Cockpit /api/inference-status endpoint + a topbar status chip that polls it
on page load — "model · live" green when up, "model · offline" amber when
the inference server is unreachable. No more manual checking. Compose also
gains a healthcheck on the inference service (applies on next recreate).
* New `psyc backup` command — tars the audit trail (db + sealed packages +
recipient keys + ledger + datasets) to data/backups/psyc-data-<ts>.tar.gz.
Excludes the HF model cache, mock-cert receipts, and the re-trainable
adapters — the goal is the irrecoverable evidence, not bulk artifacts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removed the neuronetz.ai hub link from the topbar; the footer now reads
"powered by neuronetz.ai". Topbar keeps the NN-sc app icon only.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bumped from 32px to 44px so it reads at the scale of the brand logo beside it,
with a faint cyan glow to seat it in the cockpit chrome.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
psyc carries its neuronetz application identity — the NN-sc (Security/Control)
icon replaces the plain text family tag in the topbar. The icon was cropped
from the design-kit NN-* grid and its baked-in checkerboard flood-filled to
transparent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
38 tests covering the pure worker-line logic: Classifyline rules, Routeline
TLP/country/incident-type gates, Sealine seal/unseal round-trip, Proofline
confidence scoring, Mapline CVEResolver escalation, Trainline dataset
well-posedness (the v1/v3 input-signal bugs are now regression-guarded), and
the Scoutline feed parsers. pytest added as a dev extra.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mapline gains kev_cve_set() (the known-exploited CVE set, derived from the
already-ingested KEV cases) and resolve_cves() — flags any of a case's CVEs
that are known-exploited and escalates a non-KEV case's severity to HIGH when
one surfaces. Folded into map-case / map-all / demo.
Honest limit: only KEV-sourced cases carry CVEs today, so the cross-check is
largely self-referential until a CVE-bearing source or model extraction feeds
CVEs into other cases — the escalation path is verified against a synthetic case.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New lines/proof.py: IOCChecker validates observables are well-formed,
FreshnessChecker ages observed_at into new/recent/stale/resurfaced,
ConfidenceScorer sets the Admiralty source-reliability code and an overall
confidence level. Fills case.confidence (previously left at defaults).
CLI prove-case / prove-all; folded into psyc demo. Logo glow strengthened
to a solid-cyan drop-shadow so it reads against the dark topbar.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cyber feel, kept restrained: a faint cyan grid on the app background, HUD
corner-brackets on every panel, a soft glow behind page titles. Each cockpit
view gains a collapsible help block — how to use it, what you're seeing, and
why it matters — alongside the one-line purpose intro.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Logo background stripped to transparent (it was a flattened export with the
checkerboard baked in) so the cyan drop-shadow glow now hugs the shield.
Space Grotesk for headings + chrome (the neuronetz family typeface); data
stays monospace. Topbar gains the family framing — an "NN-sc · Security"
tag and a link to the neuronetz.ai hub. Every cockpit view now carries a
one-line page-intro explaining its purpose.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
psyc now runs as a single docker compose stack — cockpit + mock-cert +
(gpu-profile) inference — on the shared external `backend` network, fronted
by nginx-proxy as psyc.neuronetz.ai. Replaces the venv processes + one-off
docker run. MOCK_CERT_BASE and INFERENCE_URL are now env-configurable
(PSYC_MOCK_CERT_URL / PSYC_INFERENCE_URL) so the cockpit reaches the other
services by compose service name. Restart policies + healthchecks. deploy.md
rewritten to match.
Verified: cockpit serves directly and via the proxy; the full
scout→…→courier→ledger chain runs over the compose network.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Lean python:3.12-slim platform image (cockpit + CLI + workers, 214 MB — no GPU,
no model). docker-compose.yml runs cockpit + mock-cert on a persistent
psyc-data volume. DATA_DIR is now overridable via PSYC_DATA_DIR so the
container's data path is explicit. docs/deploy.md covers Proxmox hosting,
first-run ingestion, and the honest caveats — no built-in auth (deploy behind
the perimeter), the GPU model server is separate, egress-proxy config.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
psyc demo now closes with cockpit links pointing at the Worker Mesh and
reports whether the live model server is up. README rewritten to current
state — Worker Mesh, inference server, model-in-operation, the three
services, accurate code layout. Adds docs/demo.md, a one-page run-sheet.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Classifier bot in the Worker Mesh now shows the real fine-tuned model's
severity verdict beside the rule's. cockpit/inference.py calls serve_model.py
over HTTP; if the server is down it returns None and the bot silently falls
back to rules — the mesh never breaks. SEVERITY_INSTRUCTION + severity_features
are shared from lines/train.py so the live prompt matches what the model
trained on. The model is now genuinely in operation, not animation over rules.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
scripts/serve_model.py — FastAPI in the CUDA container, loads base Qwen3.5-4B
+ a psyc adapter once and serves POST /infer. Lets the cockpit (no torch in
its venv) put a real fine-tuned model behind a Worker Mesh bot over HTTP.
Dockerfile.train gains a fastapi + uvicorn layer.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
_ex_severity_classification copied only URLhaus's `url_status` into the task
input, so Feodo botnet cases lost the online/offline signal their label
depends on — v3 severity eval stuck at 7/8 with one unlearnable example.
The input now carries a normalized `status` (url_status or status), matching
the field classify.py already uses for the label.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the passive journey timeline with an active worker mesh: seven robot
agents (Scout, Classifier, Mapper, Sealer, Router, Courier, Ledger), each with
a geometric SVG body, glowing antenna + reactor core in its own accent colour,
expressive awake/asleep faces, and an idle float. A case token travels the
conduit; as it reaches each bot the bot wakes (activation ring + work-flash),
performs its action, and speaks its real answer in a speech bubble. Asleep
bots are steps that did not occur for this case. Replay button re-runs it.
Every answer is real persisted data — the bots animate, they do not fake.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New /cases/{id}/journey view tells a case's story as it moved through psyc:
Detected → Classified → Located → Sealed → Routed → Submitted → Recorded.
Each beat is reconstructed from real persisted state (classification, sealed
package, planned routes, ledger rows) — a replay of recorded events, not a
script; beats that did not happen render as "pending". CSS-staggered reveal
with pulsing timeline nodes, on-brand cyan/navy, replay button.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ExampleBuilder guard checked urls/domains/ips/hashes but not cves, so
CISA KEV cases (CVE is their only observable) were silently dropped from the
ioc_extraction dataset. Now they produce CVE-extraction examples.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Scoutline is now a source registry: urlhaus, cisa-kev, feodo. CISA KEV brings
exploit/CVE cases, Feodo Tracker brings botnet C2 cases — real incident-type
variety beyond URLhaus's malware monotone. Classifyline is source-aware
(feed tag → incident type; ransomware-flagged KEV → critical). CLI gains
fetch-cisa-kev, fetch-feodo, fetch-all. Both new feeds are keyless public
download feeds (verified).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ioc_extraction ExampleBuilder now embeds every IOC into the advisory text so
the extraction task is answerable from the input (v1 asked the model to
"extract" a URL that was never given). /train page distinguishes trained /
training… / not-started, and renders a per-step loss bar chart. Dockerfile no
longer bakes the training script — scripts/ is mounted at run time so edits
take effect without a 21 GB rebuild (this is why psyc-v2's loss capture was
silently skipped on its first run).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New /train route lists built JSONL datasets (examples, size) and trained
adapters with their base model, hyperparameters, dataset provenance, and
loss history. train_qlora.py now records train_loss + per-step loss_history
into training_meta.json so future runs surface a loss curve in the cockpit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Training and eval now run clean on the unsloth 2026.5.2 / transformers v5 /
torch 2.10 stack. Fixes: pytorch/pytorch base image (sidesteps the nvidia/cuda
apt-signature failure and the torch download), correct base-model slug
unsloth/Qwen3.5-4B, TRL SFTConfig API. Adds scripts/eval_adapter.py — runs
dataset rows through base+adapter with structured (transformers-v5) message
content and Qwen3.5 thinking-mode stripping.
First v1 adapter: loss 2.10 -> 0.32 over 3 epochs. Eval surfaced an ill-posed
ioc_extraction dataset (output URL not present in input) — to be fixed in the
ExampleBuilder before the next training run.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Dockerfile.train builds a CUDA 12.4 + unsloth container that consumes the
Trainline JSONL datasets and emits a LoRA adapter at data/adapters/<run>/final.
Defaults target a 24 GB GPU (Qwen3.5-4B-Instruct-bnb-4bit, r=16, bf16, 3 epochs,
effective batch 8). README documents the build + run workflow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cases now carry a resolved hosting country, which feeds the country-scoped
destination policy. CN-hosted URLhaus malware correctly stays gated off
CERT-Bund (only DE) while still firing MISP-Community + URLhaus.
psyc demo runs the map step between classify and seal.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the end-to-end demo chain. PyNaCl sealed boxes implement the dossier's Model A
authority public-key encryption; SQLAlchemy ledger records every submission and every
policy-blocked route. Cockpit gains /ledger and an enriched case detail (sealed-package
card, routes panel, per-case audit). Mock CERT FastAPI app on :8770 stands in for the
real authority endpoints. `psyc demo` runs the whole chain on a fresh URLhaus row.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
