Compare commits

...

103 Commits

Author SHA1 Message Date
m17hr1l
4f12e344a8 xss fixes from audit:
- F1 case_detail.html: scheme-check source_ref href (block javascript: URLs)
- F2 admin.html / F3 admin_federation.html: replace inline onsubmit confirm()
  with data-attr + global handler in base.html (no more label/domain
  interpolation into onsubmit attribute string)
- federation.register_peer: validate hostname + fingerprint regex at ingest
- federation_explore.html: window.PSYC_EXPLORE via | tojson
- federation_network.js: DOMAIN_RE guard on peer-supplied domain before
  building cross-origin fetch URL (also closes open-redirect via 'open
  their explorer' button)
- app.py: nosniff + Referrer-Policy: no-referrer + X-Frame-Options: DENY
- sw.js: psyc-v11 cache bump

CSP deferred — needs inline scripts moved to external files first.
Tests: +2 cases, 245/245 green.
2026-06-07 14:23:55 +02:00
m17hr1l
00cd8ca252 db: NullPool + WAL + busy_timeout — fixes QueuePool exhaustion under federation+classify load 2026-06-07 11:57:07 +02:00
m17hr1l
77e4cb6ab9 deploy-all: redirect deploy.sh stdin from /dev/null so loop doesn't drop hosts 2026-06-07 02:06:45 +02:00
m17hr1l
9ba4cd2189 merge topology: per-peer container view in federation network detail panel 2026-06-07 01:59:23 +02:00
m17hr1l
155d6eaaf9 stage-topo-d topology-export: CLI fed-topology + SW v10 2026-06-07 01:58:27 +02:00
m17hr1l
d998be276b stage-topo-c topology-export: federation network panel renders peer containers 2026-06-07 01:57:29 +02:00
m17hr1l
367f17a013 stage-topo-b topology-export: /federation/topology endpoint + CORS cache 2026-06-07 01:56:09 +02:00
m17hr1l
a8216d00ef stage-topo-a topology-export: sanitized public docker snapshot module + tests 2026-06-07 01:55:49 +02:00
m17hr1l
8587e079bb federation/network: fetch peer's own /federation/explore/data on click + render their self-view inline (their peers, vouches in/out, transitive, translog head) 2026-06-07 01:21:58 +02:00
m17hr1l
cef3bcb1ed merge explore: public transparent federation explorer with cross-jump 2026-06-07 01:19:56 +02:00
m17hr1l
9ab3271bc8 stage-exp-f explore: tests 2026-06-07 01:17:11 +02:00
m17hr1l
c2bd68e246 stage-exp-e explore: link from home + info endpoint 2026-06-07 01:16:32 +02:00
m17hr1l
587fd07d38 stage-exp-d explore: JS — cross-jump navigation + verify button 2026-06-07 01:16:02 +02:00
m17hr1l
ca6ba83950 stage-exp-c explore: HTML template + landing layout 2026-06-07 01:13:51 +02:00
m17hr1l
a10203d8f1 stage-exp-b explore: public routes + CORS on existing public endpoints 2026-06-07 01:12:25 +02:00
m17hr1l
56466c334d stage-exp-a explore: public payload builder + tests 2026-06-07 01:11:17 +02:00
m17hr1l
351e16c3ce inference: openai-compatible mode + bearer auth (for api.neuronetz.ai etc.) 2026-06-07 01:09:19 +02:00
m17hr1l
2c7f71eff8 deploy: scripts/deploy-all.sh + hosts.example for multi-node federation rollouts 2026-06-07 01:03:31 +02:00
m17hr1l
925bf76a0b merge network-detail: rich detail panel, corroboration edges, 24h timeline, search 2026-06-07 01:01:42 +02:00
m17hr1l
0d9baef4c8 stage-netd-f network detail: tests for admin enrichment (stats/corroboration/timeline) 2026-06-07 01:00:39 +02:00
m17hr1l
980cf74b76 stage-netd-d cockpit SW: bump CACHE_VERSION to psyc-v7 for network detail CSS+JS 2026-06-07 00:57:53 +02:00
m17hr1l
70b6af6a35 stage-netd-c network detail: rich detail panel + hover tooltips + search/intensity + timeline JS 2026-06-07 00:57:49 +02:00
m17hr1l
15749e050e stage-netd-b network detail: corroboration edges + timeline strip (CSS + template) 2026-06-07 00:57:44 +02:00
m17hr1l
c6c5d3b2ea stage-netd-a network detail: enrich peer stats (signals/severity/vouches/quorum) 2026-06-07 00:52:41 +02:00
m17hr1l
e33c5b41f5 merge network-view: federation graph per node with vouches + signal flow 2026-06-07 00:43:13 +02:00
m17hr1l
865be2e239 stage-net-f network view: tests 2026-06-07 00:42:11 +02:00
m17hr1l
ff44e9e450 stage-net-e network view: CLI fed-network command 2026-06-07 00:40:43 +02:00
m17hr1l
5950d34deb stage-net-d network view: cockpit page + JS force-directed graph 2026-06-07 00:40:13 +02:00
m17hr1l
5ff6d80333 stage-net-c network view: transitive fetcher + admin data endpoint 2026-06-07 00:37:32 +02:00
m17hr1l
6dcaae39c3 stage-net-b network view: public endpoint + signed payload 2026-06-07 00:37:12 +02:00
m17hr1l
fbad78a611 stage-net-a network view: data model + local view builder 2026-06-07 00:36:29 +02:00
m17hr1l
77533eccb1 ui: widen content to 1600px + force long URLs/hashes to wrap (fixes 4K/1920 overflow) 2026-06-06 21:29:16 +02:00
m17hr1l
3e737d61b3 stage-34 wire admin nav: federation hub links to discovery + SW v4 2026-06-06 21:17:57 +02:00
m17hr1l
a53aacfdd8 merge auto-response: severity/quorum/local-only gated execution
# Conflicts:
#	src/psyc/db.py
2026-06-06 21:17:20 +02:00
m17hr1l
53ba537ce8 merge vouching+translog: web-of-trust + signed merkle audit log
# Conflicts:
#	src/psyc/_federation_cli.py
#	src/psyc/cockpit/federation_routes.py
2026-06-06 21:15:11 +02:00
m17hr1l
726117b19b merge discovery: DNS-SD walker + public peers endpoint 2026-06-06 21:13:29 +02:00
m17hr1l
c5472b3134 stage-auto-e pulse: tests for auto-response gating
Cover the auto-fire decision matrix:
- _severity_rank ordering
- mode != auto-execute → never fires (auto-propose, manual)
- below-threshold action is skipped + audited
- federation case + no quorum → skipped + audited "no quorum"
- federation case + quorum met → fires
- local case + quorum required + local-only on → still fires
- local case + quorum required + local-only off → still fires
- quorum gating disabled → federation cases fire too
- kill switch armed → tick() skips everything
- pulse_audit records both auto-fire and skip rows
- audit_count_since returns the per-action counts the cockpit needs
- config round-trips through pulse_settings

Tests patch federation.is_quorum_met (raising=False so the sibling
agent can ship the real function later without breaking these), and
swap respond.execute_action for a counter so no SOAR sink call escapes.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:12:02 +02:00
m17hr1l
f5ca928f92 stage-auto-d pulse: cockpit auto-response state panel + CLI
Cockpit:
- /admin/pulse now renders an "AUTO-RESPONSE STATE" panel above the
  pipeline table — mode badge (traffic-light colored), threshold,
  quorum on/off, local-only on/off, auto-fired-in-24h count, last 5
  audit entries, and a one-form save for threshold + gates.
- POST /admin/pulse/respond-config writes the new gates.

CLI:
- pulse-respond-config [--threshold …] [--quorum/--no-quorum]
                       [--local-only/--no-local-only]
  Args left unset are unchanged; echoes the post-state.
- pulse-respond-status prints mode, gates, and the last 10 audit
  entries.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:11:52 +02:00
m17hr1l
e66c3d3359 stage-auto-c pulse: respond runner with gates + auto-fire path
Replace the propose-only respond runner with a two-phase runner: phase 1
always proposes actions for fresh high-severity cases (unchanged); phase 2
fires when pipeline mode is auto-execute and the action clears all gates.

Gates:
- severity ≥ configured threshold
- if require_quorum is on, federation-sourced cases must hit
  federation.is_quorum_met (wrapped in try/except so we tolerate the
  sibling agent not having shipped that function yet — fallback is
  "no quorum metric → don't auto-fire", the safe default)
- locally-generated cases (no row in federation_signals for their case_id)
  bypass the quorum check
- when local-only is armed, federation-sourced cases never auto-fire
  even with quorum

Every decision (auto-fire, skipped, error) records a pulse_audit row so
the cockpit and CLI can show history. Per-action try/except keeps one
bad action from aborting the whole batch.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:11:39 +02:00
m17hr1l
f4148d86a6 stage-vouch-e federation: tests for vouching + quorum gate
test_vouching covers the contract auto-response and other agents will
gate on:

- issue_vouch round-trip (sign + verify under our own pubkey)
- accept_vouch rejects expired vouches
- accept_vouch rejects mismatched signatures
- accept_vouch rejects vouchers whose peers.status != "trusted"
- accept_vouch happy path
- is_vouched needs DISTINCT vouchers (two upserts from one peer == 1)
- is_vouched clears threshold with two distinct trusted vouchers
- is_quorum_met counts only listening-eligible peers (untrusted +
  duplicate rows don't count)
- quorum_config defaults + pulse_settings persistence
- import_signed_feed rejects unknown peer ("not trusted")
- import_signed_feed accepts directly-trusted peer
- import_signed_feed accepts a peer made eligible via two vouches
- import_signed_feed stores vouches embedded in a trusted peer's feed

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:11:18 +02:00
m17hr1l
0e56fa70af stage-vouch-d federation: cockpit pages + CLI + public endpoints
Public JSON endpoints (no auth):
- GET /federation/vouches — our_vouches() so peers can pull our trust
- GET /federation/log — last 100 transparency-log entries
- GET /federation/log/verify — re-walks the chain, returns
  {verified, head_hash} or 409 with {error, head_hash}

Admin pages (TOTP-gated):
- /admin/federation/vouches — issued list, issue form, revoke
  buttons, per-peer quorum-met table
- /admin/federation/log — chain verification status + last 200 entries
- /admin/federation/quorum — config form + per-peer eligibility +
  per-signal-hash distinct-eligible-peer counts

CLI: fed-vouch / fed-unvouch / fed-vouches / fed-quorum-set / fed-log /
fed-log-verify, plus existing fed-* commands untouched.

The base /admin/federation page now links to the three new sub-pages.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:11:03 +02:00
m17hr1l
31ec1557ec stage-auto-b pulse: pulse_audit table + history
Add a per-decision audit log so the cockpit + CLI can show what the
auto-response runner did each tick:
- pulse_audit table: id, pipeline, action ('auto-fire'|'skipped'|'error'),
  action_id, case_id, detail, timestamp
- helpers: pulse_audit_record, pulse_audit_recent, pulse_audit_count_since
- indexes on (pipeline, timestamp desc) and on action_id

Also add db.signals_for_case(case_id) — checks the federation_signals
buffer to tell whether a case was peer-sourced. Used by the runner to
decide if a quorum check is required.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:10:38 +02:00
m17hr1l
eadd1aea3b stage-vouch-c federation: import gate + translog hook (stage-trans-b)
import_signed_feed now refuses any feed whose declared fingerprint isn't
peer_is_listening_eligible (directly trusted OR vouched in), returning
Err("peer not trusted: …") before any signal lands.

For every case/IOC it does record, it also appends a "signal" entry to
the transparency log (best-effort — logger warns but doesn't abort
ingest if the append fails). This is the stage-trans-b hook: the
import path is the chokepoint, so attaching the chain there gives
us coverage of every peer-originated signal we've ever accepted.

build_signed_feed now includes our_vouches() in the feed body so vouches
propagate. On import we accept_vouch each one — but only if the embedded
voucher_fingerprint matches the peer we just authenticated, so a peer
can't forge vouches "from" someone else through us.

test_federation: the long-standing round-trip test now first registers
the synthetic peer as trusted so the gate lets it through.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:10:36 +02:00
m17hr1l
234e6d98ba stage-vouch-b federation: vouch sign/verify + quorum API
Add the web-of-trust primitives, all keyed off the existing node keypair:

- Vouch + QuorumConfig Pydantic models
- vouch_payload_bytes — canonical-JSON body that the voucher signs
- issue_vouch / accept_vouch / revoke_vouch / our_vouches / vouches_for
- is_vouched(target, min_vouchers) — counts DISTINCT trusted vouchers,
  ignoring expired vouches and re-using QuorumConfig defaults
- peer_is_listening_eligible(fp) — direct-trust OR vouched-in
- is_quorum_met(signal_hash, k) — distinct listening-eligible peers
  reporting the same hash
- quorum_evidence(signal_hash) — (peer_fp, received_at) tuples for UI
- quorum_config / set_quorum_config — persisted in pulse_settings

accept_vouch is paranoid: rejects expired vouches, vouchers that aren't
currently "trusted" in our peers table, mismatched pubkey-fingerprint
pairs, malformed base64, and Ed25519 verification failures — each with
a short Err reason.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:10:03 +02:00
m17hr1l
0dbeb056c5 stage-auto-a pulse: respond config (threshold/quorum/local-only)
Persisted-key/value helpers for the respond-pipeline auto-fire gates:
- respond_auto_threshold / set_… (Severity, default HIGH)
- respond_require_quorum / set_… (bool, default True)
- respond_local_only / set_…     (bool, default False)
Plus a _severity_rank helper for threshold comparison.

Backing store is the existing pulse_settings table; this commit also adds
generic pulse_setting_get / pulse_setting_set helpers in db.py so future
pulse settings don't each need their own column-pair helper.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:09:46 +02:00
m17hr1l
7a510c7acf stage-trans-a translog: append-only signed merkle chain + tests
translog.append computes
sha256(canonical({prev_hash, entry_type, entry_data, timestamp})) and
writes one row per call; the first entry uses prev_hash = "0"*64.
verify_chain walks rows in id order, re-hashes each, and returns
Err("broken at id=X expected=... got=...") on the first mismatch — so
tampering with either entry_data or prev_hash invalidates every
downstream row. recent / entries_after / head support peer sync and UI.

Tests cover: genesis prev_hash, chained prev_hash, full-chain verify,
tampered-data detection, tampered-prev_hash detection, slicing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:09:32 +02:00
m17hr1l
4a9f6ceb7f stage-vouch-a federation: vouches table + DB helpers
Add `vouches` (voucher_fp, target_fp, issued_at, expires_at, signature)
with unique (voucher, target) and target index, plus `translog` for the
append-only signed merkle chain (id, prev_hash, entry_type, entry_data,
timestamp, entry_hash). Also surface `setting_get` / `setting_set`
helpers on pulse_settings so the quorum config has a place to live.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-06 21:09:25 +02:00
m17hr1l
ff88aba569 stage-disc-e discovery: tests 2026-06-06 21:08:15 +02:00
m17hr1l
9b49f768ca stage-disc-d discovery: cockpit + CLI 2026-06-06 21:06:39 +02:00
m17hr1l
ddb40ff92c stage-disc-c discovery: pulse pipeline wiring + seeds settings 2026-06-06 21:04:54 +02:00
m17hr1l
6241a21af5 stage-disc-b discovery: peer walker (BFS) 2026-06-06 21:04:17 +02:00
m17hr1l
de6204819b stage-disc-a discovery: dnssd resolver + public peers endpoint 2026-06-06 21:03:33 +02:00
m17hr1l
1675a2326e stage-33 wire pulse + federation: cockpit routes, CLI, nav links, SW bump 2026-06-06 16:15:48 +02:00
m17hr1l
de5ff09815 merge federation: ed25519 identity + signed feeds
# Conflicts:
#	src/psyc/db.py
2026-06-06 16:13:36 +02:00
m17hr1l
02ce6d791c merge pulse: scheduler line + autonomy dial 2026-06-06 16:11:17 +02:00
m17hr1l
d4229dd264 stage-fed-g federation: tests 2026-06-06 16:10:31 +02:00
m17hr1l
2ef0448165 stage-fed-f federation: CLI commands 2026-06-06 16:10:26 +02:00
m17hr1l
17b94acf6b stage-fed-e federation: cockpit admin page + public feed routes 2026-06-06 16:10:19 +02:00
m17hr1l
55ffd9da3d stage-fed-d federation: signed feed export + verified import 2026-06-06 16:09:53 +02:00
m17hr1l
63e3ff2777 stage-fed-c federation: db tables for peers + signal buffer 2026-06-06 16:08:36 +02:00
m17hr1l
50158f7fa8 stage-fed-b federation: dns record format 2026-06-06 16:08:31 +02:00
m17hr1l
4c35aad2bb stage-fed-a federation: ed25519 keypair + fingerprint 2026-06-06 16:08:03 +02:00
m17hr1l
a7c59c9faa stage-33e pulse: tests 2026-06-06 16:06:54 +02:00
m17hr1l
e071f289f2 stage-33d pulse: CLI commands 2026-06-06 16:05:14 +02:00
m17hr1l
26fbe08b65 stage-33c pulse: admin cockpit page 2026-06-06 16:04:39 +02:00
m17hr1l
4d67605371 stage-33b pulse: db tables + persistence 2026-06-06 16:03:30 +02:00
m17hr1l
e710be6ebd stage-33a pulse: scheduler module with pipeline registry 2026-06-06 16:03:22 +02:00
m17hr1l
6356c5535b stage-32: add Umami analytics tracking script in base.html
Inserted the analytics.neuronetz.ai tracking script tag in the <head>
of base.html with the project's website-id. defer attribute means it
never blocks page render. Cookieless / privacy-respecting (Umami).
Applies to every page that extends base — the admin gate's standalone
template intentionally skips it (no point tracking the login screen).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 20:09:54 +02:00
m17hr1l
43c7c199c3 stage-31 polish: featured hero — particles sync with sweep, stat chips, radar emblem
1. Particles now blink in time with the scan column.
   Each <circle>/<rect> in the case hero SVG carries class="hero-particle"
   and a CSS animation-delay equal to -(arrival-time-of-sweep-at-its-x),
   so as the sweep marches left-to-right the particles light up in
   sequence — a real "the scanner just touched this point" effect. The
   sweep is now linear / non-alternating on a 12s cycle so the phase
   stays predictable. Drop-shadow + opacity flash at the peak.

2. Stat chips added under the featured title.
   Up to five compact chips showing incident type, total IOC count with
   per-type breakdown (U/D/I/H/C), victim country if mapped, confidence
   level, malware family. Each chip is color-coded.

3. Radar emblem in the top-right of the hero.
   Pure SVG: three concentric range rings, cross-hairs, a translucent
   sweep arm that spins every 4.5s, a center dot. The whole emblem
   inherits the cycling-hero hue cycle so it changes color with the
   grid. Smaller + dimmer on mobile; honors prefers-reduced-motion.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 19:38:31 +02:00
m17hr1l
977c3670f3 stage-31 polish: featured hero — full-bleed animated grid cycling colors
The hero is now layered:
  base   — radial accent + dark gradient (truly full-width, min 240px /
           up to 320px tall, no aspect-ratio cap)
  grid   — full-bleed CSS grid (38px cells), line color cycles through
           cyan → red → gold → green → violet over 18s, plus a slow
           grid-drift so the lattice subtly slides; a glowing column
           sweeps left-to-right across the hero every 12s for depth
  particles — the existing case-specific SVG sits on top at lower
           opacity (mix-blend-mode: screen) so the per-case identity
           is preserved as a constellation glittering over the grid
  overlay — title + Open-case CTA pinned to the bottom

Severity just nudges the grid opacity (CRITICAL/HIGH get a bit denser);
the color cycle stays on for all severities. Honors prefers-reduced-motion.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 19:32:27 +02:00
m17hr1l
3f1f7cc420 stage-31 polish: featured-threat banner — uniform tint, full width
The banner gradient was a horizontal accent → dark fade, so visually
only the left ~30% read as a banner; the right looked unstyled. Swapped
for a uniform severity-tinted background (subtle top-to-bottom gradient
for depth, but no horizontal falloff). Border + inset shadow on the
bottom edge give it a real divider between the banner and the hero
beneath. Per-severity tints (CRITICAL red, HIGH amber) get the same
treatment — full-width, uniform.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 19:28:26 +02:00
m17hr1l
04e0d3323f stage-31 fix: featured card — display:block on the new <a> wrapper
When I refactored the featured card from <section>→<a> for full-card
clicking, I forgot the <a> defaults to display:inline. That collapsed
the hero+overlay layout — the title rendered word-per-line down the
left edge instead of spanning the hero. display:block (+ inherit
color, no text-decoration) restores the layout.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 19:15:48 +02:00
m17hr1l
5cf7cb5655 stage-31 polish: featured banner header + clickable news cards w/ hover
Two enhancements from the screenshot annotations:

1. The featured card now has a proper banner header above the hero —
   gradient strip with "⌖ Featured Threat" label, a pulsing severity
   dot, severity/TLP badges, source + ingest time. The header is
   sev-themed (red border + glow for CRITICAL, amber for HIGH), and an
   "Open case →" CTA at the bottom of the hero with an arrow that
   nudges right on hover.

2. News items are now full-card clickable (transparent overlay link)
   with rich severity-tinted hover transitions:
   - Lifts (translateY -1px), tinted background + border that matches
     severity (red CRITICAL, amber HIGH, pale MEDIUM/LOW), per-kind
     accents for enforced/submitted/rejected/failed items.
   - A right-side arrow (→) slides in on hover signaling "click to open".
   - Case glyph SVG gets a subtle drop-shadow + 1.04x scale on hover.
   - 180ms ease transitions across background/border/transform/shadow.
   - Respects prefers-reduced-motion (turns animations off).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 19:12:55 +02:00
m17hr1l
f51e672ad3 stage-31 fix: home page 500 — replace bad jinja sum with precomputed count
Template had `{{ buckets|sum(attribute='items')|length }}` which tried
to sum lists starting at 0 → TypeError. Switched to a Python-side
`total_items = sum(len(b.items) for b in buckets)` passed through the
context and rendered directly.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 19:05:25 +02:00
m17hr1l
76a0b0b636 stage-31: Newsline polish — featured case, time buckets, severity accents, generated visuals
The home page goes from a flat event stream to something that reads like
a news blog:

- Featured-case hero card at the top, picked as the highest-severity
  case (CRITICAL > HIGH, tie-break recency) from the last 7 days. Wide,
  with a procedurally generated SVG hero behind a gradient overlay that
  carries title + severity + TLP + feed + ingest time.
- Recent activity is now grouped under Today / Yesterday / Earlier this
  week / Older bucket headers.
- Each item gets a left-border severity accent (red CRITICAL, amber
  HIGH, muted MEDIUM/LOW) so the page is scannable at a glance.

Images: new cockpit/case_visuals.py generates SVGs from case data —
zero external image gen, zero curated assets. Every visual is
deterministic from case_id (so a case keeps its identity across
sessions) and themed to its severity:

- case_hero_svg() — 880x220 hero with severity radial glow, a faint
  scan grid, a particle constellation with auto-connecting lines, HUD
  corner brackets, and the case id whispered in the bottom-right.
- case_glyph_svg() — small mirror-symmetric identicon (5-grid),
  severity-colored, shown beside each case news item in place of an
  emoji icon. Two case_ids → two distinct glyphs; same id → same glyph.

7 news tests pass; visual sanity print confirms hero is deterministic
and uses the right severity accent.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 19:02:22 +02:00
m17hr1l
4d36db90f1 stage-30 fix: SW cache strategy — bump version + stale-while-revalidate
User reported mobile nav still looked unresponsive. Verified the new
nav-toggle CSS and markup are deployed; the browser's service worker
was serving the prior cockpit.css cache-first from psyc-v1.

Two fixes so this can't recur:
- CACHE_VERSION bumped to psyc-v2 → the activate handler now wipes
  the v1 cache on the user's next visit.
- Static-asset strategy changed from cache-first to
  stale-while-revalidate: serve from cache for instant render, then
  refresh in the background so the *next* load picks up the new CSS/JS
  with no manual cache-busting. Falls back to network if cold.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 17:36:08 +02:00
m17hr1l
88e4fb1dcd stage-30 fix: proper responsive nav (hamburger drawer) + cases-pipeline fix
Replaced the horizontal-scroll hack with a real mobile pattern:
- Hamburger button (☰) appears below 760px width, animates into an X when
  open. Tap to slide the full nav into a drawer below the topbar, each
  link a tap-target-sized row.
- Nav links auto-close the drawer on click so navigating dismisses it.
- aria-expanded / aria-controls wired up so screen readers track state.
- Below 420px the model status chip + family icon hide to keep the
  topbar compact; brand-sub already hidden by the earlier mobile pass.

Discovered while debugging: prod had 377 unclassified cases (only
fetch-all was run, not classify-all/prove-all/reindex), so the journey
page's Classifier beat skipped the live model verdict entirely. Ran
the full pipeline on prod — the psyc-v5 model chip now renders on
case journeys. Worth noting for the deploy runbook: after psyc fetch-all
on a new env, always also run classify-all + prove-all + reindex to
prime the pipeline state.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 17:27:11 +02:00
m17hr1l
16cf873044 stage-30: home page (Newsline digest) + PWA + mobile pass
NEW / start page replaces the redirect-to-/cases:
- KPI strip (cases, IOCs, +24h, high/critical,  enforced 24h, ledger total) —
  clickable, responsive grid (2 cols mobile, 3 mid, 6 desktop).
- Recent activity feed: ledger events (enforced/submitted/rejected/failed) +
  newest case ingests, interleaved newest-first, with severity badges, icons,
  case links. Sources via lines/news.py.
- Feed health sidebar: per-feed count + last ingest time.

PWA:
- /static/manifest.json declares a standalone install with theme colors.
- /static/sw.js — cache-first for static, network-first for HTML/API, with a
  graceful offline page. Registered from / scope via a dedicated /sw.js route
  that sets Service-Worker-Allowed: /.
- viewport + apple-touch-icon + theme-color meta tags in base.html.

Mobile pass on the chrome:
- Topbar wraps; nav horizontally scrolls instead of crowding; brand-sub hides.
- Tables (cases, ledger) scroll horizontally on narrow screens instead of
  exploding the layout.
- Hero / KPI / news-list layouts collapse cleanly at < 720px.

4 news tests; verified locally — home page renders, /sw.js serves with
Service-Worker-Allowed: /, manifest is valid JSON.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 17:18:40 +02:00
m17hr1l
7a57a7390a stage-29 fix: inference service — wire build: directive in compose
The inference service declared 'image: psyc-trainer' but no build: stanza,
so 'docker compose build inference' was a silent no-op and 'compose up
inference' tried to pull from a registry that doesn't exist (denied).
Added build context pointing at Dockerfile.train so future production
deploys can rebuild it via the compose lifecycle instead of needing a
manual 'docker build' on the side.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 17:08:50 +02:00
m17hr1l
d7999150b3 stage-29: fetch-all resilience + Mozilla-compatible UA for CISA
Two production-discovered fixes after first deploy:

- CISA's CDN was 403'ing the "psyc/0.1 (defensive CTI; hackathon
  prototype)" User-Agent from the cloud.neuronetz.ai exit IP. Switched
  to a Mozilla-compatible UA that identifies us honestly while passing
  the CDN's UA filters. Overridable via PSYC_HTTP_USER_AGENT.
- fetch-all aborted on the first HTTPStatusError, so a CISA hiccup
  killed the threatfox/malware-bazaar/otx legs that come after. The
  outer loop now catches any exception per-source, logs a skip, and
  moves on. Single-source failures no longer poison the rest of the
  pull.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 16:56:27 +02:00
m17hr1l
fad7ad0d49 stage-28: make the proxy docker network name configurable per environment
On AnD0R the reverse-proxy lives on the 'backend' docker network; on
cloud.neuronetz.ai it's 'neuronetz_default'. With a hardcoded name the
cockpit ended up on a network the prod proxy couldn't see and routing
silently dropped. Network is now overridable via PSYC_PROXY_NETWORK in
.env (default 'backend' keeps dev working).

On prod, set PSYC_PROXY_NETWORK=neuronetz_default in .env before the
next compose up.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 16:45:47 +02:00
m17hr1l
92f754e012 stage-28: wire LETSENCRYPT_HOST + LETSENCRYPT_EMAIL on the cockpit service
Adds the two env vars nginxproxy/acme-companion looks for to issue +
auto-renew the TLS cert for psyc.neuronetz.ai. LETSENCRYPT_EMAIL is
interpolated from the prod .env (LETSENCRYPT_EMAIL=...) with a sensible
fallback so dev / local deploys don't fail on the variable being unset.
.env.example documents the var.

Requires the proxy stack to (a) have acme-companion alongside
nginx-proxy with shared certs/vhost.d/html volumes and (b) publish :443.
psyc-side change only — no app code touched.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 16:42:46 +02:00
m17hr1l
9c3447723a stage-28 fix: deploy.sh — auto-trust Gitea host (TOFU), never touch identity keys
Reinstating the auto known_hosts entry on first deploy. Clear scope:
host trust (TOFU known_hosts entry) is automated — same as
'ssh -o StrictHostKeyChecking=accept-new' would do; identity keypairs
(~/.ssh/id_*) are never generated/copied/modified by deploy.sh.
PSYC_SKIP_HOST_TRUST=1 disables the auto-trust step if you'd rather
verify fingerprints manually.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 16:36:18 +02:00
m17hr1l
9edd56e28b stage-28 fix: deploy.sh — read-only SSH preflight, no key/known_hosts edits
User asked the script not to touch their SSH config. Reverted the
auto-ssh-keyscan; the script now only READS ~/.ssh/known_hosts (via
ssh-keygen -F) and, when the entry is missing, exits with explicit
manual instructions for verifying the host key and registering an
identity key in Gitea. Identical behavior on the happy path; clearer
diagnostics on the unhappy path; zero modification of ~/.ssh anywhere.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 15:39:06 +02:00
m17hr1l
2c2ead6149 stage-28 fix: deploy.sh pre-trusts the Gitea SSH host key (first-clone)
A fresh prod box has never SSH'd to gitea.neuronetz.ai before, so the
first 'git clone' failed with 'Host key verification failed'. The
script now parses the git remote URL to extract host+port, and on the
prod box does an ssh-keyscan into ~/.ssh/known_hosts before the clone
when the entry is missing. TOFU — if you want to verify the fingerprint
out-of-band, pre-populate known_hosts manually and the script will see
the entry and skip the scan.

Also: if the clone still fails after the host key is trusted (likely a
missing SSH key on Gitea side), the script now prints a clear hint
pointing at where to register it. Supports both ssh://user@host:port/
and user@host: URL forms.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 15:32:44 +02:00
m17hr1l
61b7b8ef20 stage-28: deploy.sh — idempotent remote deploy + health probe
scripts/deploy.sh pushes the current branch to origin, ssh's into the
prod box (neuronetz@cloud.neuronetz.ai:/home/neuronetz/docker-public/
neuro-psyc by default — overridable via env vars), clones-or-pulls,
ensures the external 'backend' docker network exists, runs docker
compose up -d --build (+ --profile gpu if PSYC_PROD_GPU=1), and then
verifies the cockpit is healthy both on prod-internal :8767 and at the
public URL — so the script ends knowing whether the page is up.

Refuses to touch prod's .env (warns + copies .env.example if missing,
so you can edit it manually). Never transfers data/ or adapters
(gitignored; prod fetches its own corpus). Color output, idempotent,
safe to re-run.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 14:51:47 +02:00
m17hr1l
494755ec4f stage-26d: click any topology node → structured spec panel below
Click a container, switch, or the host in the graph: the node gets a
pulsing accent highlight and a detail panel below renders its full
logical overview — for a container, image/status/per-network IPv4+MAC+
gateway + published ports + all ports; for a switch, driver/scope/
subnet/gateway/internal + attached-container table; for the host, OS/
CPUs/container counts. Cross-jumps in the spec tables click through to
the related node (the graph re-selects too). Click the × button or any
empty graph area to deselect.

Drag is distinguished from click by a 4px move threshold so dragging
nodes doesn't accidentally open the panel. HTML escaped end-to-end.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 12:25:15 +02:00
m17hr1l
ef88cd9d5d stage-26c: topology layout views, traffic flow, full-width page
Three layout modes (Force/Hierarchical/Radial) switch the topology graph
between organic force-directed, host→switches→containers tiers, and a
radial host-in-center wheel. Animated traffic flow on edges via
stroke-dashoffset marching: wires to running containers flow cyan,
host↔switch uplinks pulse slower in violet, container→host publish
edges flash fastest in amber-dashed; dead edges (exited containers) fade.
"traffic flow" checkbox kills the animation globally; layout choice
auto-fixes nodes (drag still overrides). Respects prefers-reduced-motion.

/admin/docker now opts into a wide layout via a new body_class block on
base.html — content area uncaps, SVG sized min(74vh, 880px), network
cards get wider min columns. Other pages keep the 1280px reading cap.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 12:20:18 +02:00
m17hr1l
b51a88d502 stage-26b: Docker topology in /admin — read-only socket-proxy + graph
New tecnativa/docker-socket-proxy sidecar exposes only GET on
containers/networks/info/ping; POST and DELETE are blocked. The cockpit
queries it over the backend network — /var/run/docker.sock is never
mounted into a web-facing container.

cockpit/docker_view.py normalizes the daemon view: containers carry
per-network IP/MAC + published_ports; networks carry subnet/gateway from
IPAM; host_info pulls /info (degrades gracefully). topology() returns
the combined snapshot.

/admin/docker (admin-gated): a force-directed graph (pure SVG +
vanilla JS, ~280 lines) renders the complete setup — a host node,
switch nodes with subnet labels colored by driver, container nodes
colored by state, member wires labeled with the container's IP on that
network, uplinks from non-internal switches to the host labeled with
the gateway, and dashed publish-edges from containers to the host for
their published ports. Drag to rearrange, scroll to zoom, re-settle
kicks the physics. Below the graph: containers table + grouped network
cards as a textual mirror. 12 docker_view tests; verified live (32
containers, 11 switches, real subnets + gateways).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 03:08:39 +02:00
m17hr1l
eaca27be26 stage-27 polish: admin presence announces itself in the topbar
Once you're signed in, the topbar gains:
- An "Admin" nav link (cyan-accented, with a small shield-lock SVG)
  appearing only when an admin session exists.
- A glowing chip on the right showing "● ADMIN · <who>" with a
  separated lock button (⏻) for sign-out. Pops in with a scale-bounce
  entrance, then a slow box-shadow pulse so it stays noticeable without
  being annoying. Respects prefers-reduced-motion.
- Both rendered via request.session in base.html, so every page gets
  them automatically with no per-route plumbing.

Removed the now-redundant in-panel "lock ⏻" from admin.html — the
topbar owns logout, the admin page header shows enrolled count instead.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 01:14:58 +02:00
m17hr1l
cb7bef4e40 stage-27: per-member TOTP enrollment + individual revocation
Replaces the single shared admin secret with named per-member
enrollments — no more "one key for everyone". First visit bootstraps an
'owner' slot; further members are added from inside the admin panel,
each scanning their own QR. Login accepts a code matching any active
member and records who got in. Offboarding is a per-member revoke: that
person's codes stop immediately, everyone else is unaffected, nobody
re-enrolls. Old single-secret state migrates to an 'owner' member.

Admin panel gains an Access Control table (member, enrolled, last used,
revoke) + add-member form that shows the new QR once. 7 tests including
revocation isolation; verified the full lifecycle live (bootstrap → add
→ authenticate → revoke → rejected while owner persists).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 00:46:45 +02:00
m17hr1l
4a832964a3 stage-26 polish: restyle the /admin gate as a secure console
Replaced the plain login box with a proper restricted-zone screen:
ambient breathing radial backdrop, frosted glass card with a glowing
top accent + animated HUD corner brackets, an SVG shield-lock emblem in
a pulsing glow ring, a SECURE CHANNEL status line with a blinking dot,
the QR in a white frame with a sweeping cyan scanline, a large
OTP-style monospace code field, and a gradient UNLOCK button. Honors
prefers-reduced-motion.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 00:38:47 +02:00
m17hr1l
abdf5e7747 stage-26: hidden /admin gated by TOTP (authenticator-app 2FA)
A hidden /admin path (not in nav) protected by a TOTP secret you enroll
by scanning a QR into Google Authenticator / Authy, then entering the
rotating 6-digit code. adminauth.py persists the secret + session key
under DATA_DIR (gitignored); the QR only renders until first successful
verification so the provisioning secret isn't perpetually exposed.
SessionMiddleware carries a 60-min admin session. This becomes the
secured control center the rest of the system gets built into.

Verified end-to-end: gate renders QR, the live code authenticates and
sets the session, the dashboard renders only with the session cookie,
a wrong code is rejected, and an unauthenticated request never leaks
the dashboard. Deps: pyotp, qrcode[pil], itsdangerous.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 00:35:02 +02:00
m17hr1l
73a932d8be stage-25: response actions — human-gated enforcement + the disco
Closes the loop: intel -> decision -> enforcement -> audit. High/critical
cases propose response actions (alert SOC, push IOCs to perimeter
firewall+DNS). Nothing fires automatically — each sits PROPOSED until a
human approves, then it's POSTed to the enforcement sink (PSYC_SOAR_URL,
default mock-cert /soar/enforce) and written to the ledger as ACTIONED.

- models: ActionType / ActionStatus / ResponseAction
- db: response_actions table
- lines/respond.py: propose_for_case (idempotent, sev-gated), execute_action
  (fire + ledger + mark), reject_action; mock SOAR endpoint in mock_cert
- cockpit /response page: proposed/enforced/declined tabs,  Enforce +
  decline, and the disco — a full-screen strobe + "ENFORCED" + IOC-scatter
  animation that fires on approval (respects prefers-reduced-motion)
- cli: respond / actions / act-approve / act-reject
- 8 tests; verified the full loop live (propose -> enforce -> disco ->
  SOAR receipt -> ledger ACTIONED row)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 00:24:31 +02:00
m17hr1l
d0a71d0226 stage-24: indicator lookup page + blocklist download in cockpit
Surfaces the stage-23 index in the UI. New /lookup page: paste any
indicator (IP/domain/URL/hash/CVE) → red KNOWN-BAD banner with the
matching cases/feeds/severities, or green clean banner. New
/export/blocklist endpoint returns deduplicated plain-text indicator
lists (all or high+ severity) for firewall/DNS/SIEM ingestion, linked
from a download table on the lookup page. Lookup added to topbar nav.

Verified live: lookup of a real corpus IP returns the OTX case;
8.8.8.8 returns clean; blocklist endpoint emits 26 high-severity IPs
with a descriptive header line.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 23:55:50 +02:00
m17hr1l
9a2a31ec9a stage-23: IOC index + lookup — the actionable keystone
New iocs table (value, type, case_id, feed, severity, first_seen) +
lines/lookup.py: normalize() (CVE upper, rest lower), reindex() to
rebuild from the corpus, lookup() (normalization-insensitive, scans all
types), export_blocklist() (deduped, min-severity filter).

CLI: psyc reindex / lookup <indicator> / export-blocklist --type --min-severity.

Verified on the live corpus: 1288 IOCs from 598 cases; lookup of a real
IP/CVE resolves to its case+feed+severity; 8.8.8.8 correctly misses;
blocklist export yields 148 IPs / 289 domains / 150 URLs / 514 hashes /
108 CVEs. This primitive backs the upcoming search UI, asset matching,
and watchlist alerting.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 23:39:05 +02:00
m17hr1l
f88db2fdf7 stage-22: cockpit reflects the live adapter + all six feeds
The Worker Mesh Classifier badge was hardcoded "psyc-v4" — it kept
claiming the old model even after the v5 swap, making it look like
nothing had changed. Now derived dynamically from the inference
server's reported adapter via inference.adapter_name(); shows the real
version (psyc-v5) or "rules" when the server is down, so it can't go
stale on the next swap.

Also refreshed the cases page intro/help text to name all six feeds
(URLhaus, CISA KEV, Feodo, ThreatFox, MalwareBazaar, OTX) instead of
the original three.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 23:33:19 +02:00
m17hr1l
ee387abcd4 stage-21: swap inference server to psyc-v5 adapter
v5 trained on 598 ex/task (20× v4's 30), with --defang-frac 0.5 over
the new ThreatFox + MalwareBazaar + OTX corpus. Final train_loss 0.3225
vs v4's 0.7397 (56% reduction), 60m20s wall clock on a 3090.

Live eval before swap:
- severity (botnet, ONLINE): v4 high / v5 high — tied, both correct
- ioc_extraction with defanged input (hxxps://, [.], (.), [dot]):
    v4 kept hxxps:// in output (failed canonicalization)
    v5 returned canonical https:// — defang training paid off
- ioc_extraction on real OTX-style prose (never trained on this shape):
    v5 cleanly extracted 2 domains + 1 IP + 1 SHA256 + 1 CVE

Cockpit /api/inference-status confirms the swap:
  {"online":true,"adapter":"/data/adapters/psyc-v5/final"}

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 23:55:47 +02:00
m17hr1l
376c5b6f4a stage-19-fix2: OTX — narrow by modified_since, longer timeout
The /pulses/subscribed endpoint enumerates every curated feed a fresh
account is auto-subscribed to. On its own that's enough to 504 from
OTX's backend regardless of client timeout. Narrowing by
modified_since=now-7d brings the response back to a single-second fetch.

Also: _http now accepts params + per-call timeout overrides (OTX uses
120s). The CLI --limit still slices post-fetch.

Verified live: 10 OTX pulse-cases ingested, each carrying real
paragraph-form descriptions (Mirai, macOS Stealer, FlowerStorm PhaaS,
Vidar v1.5, manufacturing intrusion) — exactly the real-prose source
the IOC extractor's been missing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 22:39:24 +02:00
m17hr1l
f6fa52839f stage-20: defanging pipeline for IOC-extraction augmentation
Real CTI prose defangs IOCs (1[.]2[.]3[.]4, hxxp://, evil[dot]com) so they
don't auto-link in email/chat. A model trained only on canonical inputs
will fail to extract them.

New lines/defang.py: defang_ip, defang_domain, defang_url, defang_text —
four dot-styles ([.], (.), [dot], {.}) plus protocol defanging
(http→hxxp, https→hxxps). Each occurrence picks its style independently
since real advisories don't keep one style across paragraphs.

train.BuildOptions adds defang_frac (default 0.0) and seed; build()
threads options + a seeded Random through the example builders so
the augmentation is reproducible. Only _ex_ioc_extraction reads it
today — output stays canonical so the model learns messy→canonical.

CLI: train-build and train-build-all gain --defang-frac and --seed.
8 new tests including a frac=1.0 / output-canonical integration check.
The pipeline runs but is dormant at defang_frac=0.0 — psyc-v5 dataset
build will set 0.5 once OTX cases land.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 22:33:52 +02:00
m17hr1l
85830be9fa stage-19-fix: ThreatFox + MalwareBazaar — real API shape
Live test against abuse.ch revealed two issues with the stage-19 wiring:

- ThreatFox returns `ioc` (not `ioc_value`) and `first_seen` (not
  `first_seen_utc`) — older field names from stale docs. Parser now reads
  the real names and falls back to the old aliases defensively. Also
  captures `malware_malpedia` (per-family writeup URL) and
  `threat_type_desc` for richer downstream prose.
- MalwareBazaar's API expects form-encoded bodies, unlike ThreatFox's
  JSON. Extended _http with form_body=; MB fetcher switched to it.

Verified live: 10 ThreatFox cases landed with mixed botnet/malware
classification (4/6 split from threat_type signal — first real
incident-type diversity from a single feed). 10 MalwareBazaar cases
landed with sha256+sha1 hash observables and exe/file_type metadata.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 22:25:56 +02:00
m17hr1l
d87bd710bb stage-19: ThreatFox + MalwareBazaar + OTX Scoutline sources
Three new feeds — biggest near-term data-diversity win. ThreatFox brings
multi-malware IOCs with threat_type signal (botnet_cc → BOTNET,
payload_delivery → MALWARE, phishing → PHISHING). MalwareBazaar brings
file-hash samples with signatures. OTX brings curated multi-source pulses
with paragraph-form descriptions — by far the richest real-prose source.

Auth: THREATFOX_AUTH_KEY (one abuse.ch key covers ThreatFox + MalwareBazaar)
and OTX_API_KEY. fetch-all skips keyed feeds cleanly with where-to-get-it
guidance instead of tracebacking. Proofline reliability table extended;
abuse.ch sources rated B/2, OTX rated C/3 (community-driven).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 22:14:18 +02:00
m17hr1l
994a5c642f stage-18: approval queue — human gate before evidence leaves
CERT-Bund (authority) requires_approval by default; PSYC_REQUIRE_APPROVAL=1
forces every routable submission through the queue. Courier branches at
execute_routes: approval-required → freeze payload + enqueue, no HTTP; else
submit directly as before. Approve dispatches the frozen payload to mock-cert
and writes the ledger row (detail=approved_by=…); reject writes a ledger row
with the reviewer's reason. CLI: queue / approve / reject. Cockpit /queue
page with POST approve / reject and counts.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 21:42:08 +02:00
80 changed files with 17028 additions and 57 deletions

View File

@@ -14,6 +14,19 @@ OTX_API_KEY=
# (raises throttling from ~5 to ~50 requests / 30s)
NVD_API_KEY=
# --- Production-only: Let's Encrypt email for the acme-companion sidecar ---
# Used as the contact address for the TLS cert acme-companion issues for
# psyc.neuronetz.ai. Safe to leave the default in dev (cert isn't issued
# without a reachable acme-companion + public DNS + :443).
# LETSENCRYPT_EMAIL=admin@neuronetz.ai
# --- Production-only: the docker network the reverse-proxy is on ---
# Cockpit must share this network with nginx-proxy + acme-companion so the
# proxy can route to it. Default 'backend' matches AnD0R dev; on cloud
# production this is typically 'neuronetz_default' (whatever the proxy stack
# declares — check with `docker network ls`).
# PSYC_PROXY_NETWORK=neuronetz_default
# --- Internal service URLs — overridden in docker compose; defaults for venv CLI ---
# PSYC_MOCK_CERT_URL=http://127.0.0.1:8770
# PSYC_INFERENCE_URL=http://127.0.0.1:8771

3
.gitignore vendored
View File

@@ -14,6 +14,9 @@ data/
.env
.env.local
# per-operator federation host list (SSH targets are sensitive)
scripts/hosts
# editors
.vscode/
.idea/

View File

@@ -20,8 +20,16 @@ services:
environment:
VIRTUAL_HOST: psyc.neuronetz.ai
VIRTUAL_PORT: "8767"
# Triggers nginxproxy/acme-companion (which must be running alongside
# nginx-proxy on the host) to issue + auto-renew a Let's Encrypt cert
# for psyc.neuronetz.ai. LETSENCRYPT_EMAIL comes from .env so per-env
# configurable — falls back to the default if unset.
LETSENCRYPT_HOST: psyc.neuronetz.ai
LETSENCRYPT_EMAIL: ${LETSENCRYPT_EMAIL:-admin@neuronetz.ai}
PSYC_MOCK_CERT_URL: http://mock-cert:8770
PSYC_SOAR_URL: http://mock-cert:8770
PSYC_INFERENCE_URL: http://inference:8771
PSYC_DOCKER_PROXY: http://docker-socket-proxy:2375
ports:
- "8767:8767" # direct/debug access; the proxy serves psyc.neuronetz.ai on :80
volumes:
@@ -47,11 +55,34 @@ services:
timeout: 5s
retries: 3
# Read-only Docker daemon proxy. The cockpit's /admin/docker view queries this
# over the backend network instead of touching /var/run/docker.sock directly,
# so a compromise of the web app can't drive the daemon. Only GET on
# containers/networks/ping is enabled — POST/DELETE/EXEC stay blocked.
docker-socket-proxy:
image: tecnativa/docker-socket-proxy:0.3
environment:
CONTAINERS: "1"
NETWORKS: "1"
PING: "1"
INFO: "1"
POST: "0"
DELETE: "0"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
networks: [backend]
restart: unless-stopped
# The live fine-tuned model behind the Classifier bot. GPU-only — opt in with
# `--profile gpu`. Uses the psyc-trainer image (built from Dockerfile.train).
# The build context is local so `docker compose --profile gpu build inference`
# actually builds it (without this, compose silently skips the build).
inference:
image: psyc-trainer
command: ["/scripts/serve_model.py", "--adapter", "/data/adapters/psyc-v4/final", "--host", "0.0.0.0", "--port", "8771"]
build:
context: .
dockerfile: Dockerfile.train
command: ["/scripts/serve_model.py", "--adapter", "/data/adapters/psyc-v5/final", "--host", "0.0.0.0", "--port", "8771"]
volumes:
- ./data:/data
- ./scripts:/scripts
@@ -73,6 +104,10 @@ services:
capabilities: [gpu]
networks:
# The reverse-proxy + acme-companion need to share a docker network with the
# cockpit so they can see each other. The actual network name differs by
# environment (e.g. 'backend' in dev, 'neuronetz_default' in production), so
# it's overridable via PSYC_PROXY_NETWORK in .env. Default keeps dev working.
backend:
name: backend
name: ${PSYC_PROXY_NETWORK:-backend}
external: true

View File

@@ -16,9 +16,14 @@ dependencies = [
"httpx>=0.27",
"typer>=0.12",
"pynacl>=1.5",
"cryptography>=42.0",
"structlog>=24.1",
"sqlalchemy>=2.0",
"python-dotenv>=1.0",
"pyotp>=2.9",
"qrcode[pil]>=7.4",
"itsdangerous>=2.1",
"dnspython>=2.4",
]
[project.optional-dependencies]

65
scripts/deploy-all.sh Executable file
View File

@@ -0,0 +1,65 @@
#!/usr/bin/env bash
# Deploy the current main commit to every federation host listed in
# scripts/hosts (one node per line: LABEL SSH_TARGET REMOTE_PATH PUBLIC_URL).
# Loops scripts/deploy.sh against each. Bails on first failure unless --keep-going.
set -euo pipefail
HOSTS_FILE="${PSYC_HOSTS_FILE:-$(dirname "$0")/hosts}"
KEEP_GOING=0
for arg in "$@"; do
case "$arg" in
--keep-going) KEEP_GOING=1 ;;
-h|--help)
echo "usage: $0 [--keep-going]"
echo " reads $HOSTS_FILE (override with PSYC_HOSTS_FILE=...)"
exit 0
;;
esac
done
if [[ ! -f "$HOSTS_FILE" ]]; then
echo "no hosts file at $HOSTS_FILE — copy scripts/hosts.example to scripts/hosts and edit" >&2
exit 2
fi
declare -a OK=() FAIL=()
while IFS= read -r line; do
# skip blanks + comments
[[ -z "${line// /}" || "${line# }" == \#* ]] && continue
# shellcheck disable=SC2206
parts=($line)
if [[ ${#parts[@]} -lt 4 ]]; then
echo "[deploy-all] skipping malformed line: $line" >&2
continue
fi
LABEL="${parts[0]}"
SSH_TARGET="${parts[1]}"
REMOTE_PATH="${parts[2]}"
PUBLIC_URL="${parts[3]}"
echo
echo "════════════════════════════════════════════════════════════════"
echo " deploying → $LABEL ($SSH_TARGET:$REMOTE_PATH$PUBLIC_URL)"
echo "════════════════════════════════════════════════════════════════"
if PSYC_PROD_HOST="$SSH_TARGET" \
PSYC_PROD_PATH="$REMOTE_PATH" \
PSYC_PROD_URL="$PUBLIC_URL" \
bash "$(dirname "$0")/deploy.sh" < /dev/null; then
OK+=("$LABEL")
else
FAIL+=("$LABEL")
if [[ $KEEP_GOING -ne 1 ]]; then
echo "[deploy-all] $LABEL failed — stopping. pass --keep-going to continue past failures." >&2
break
fi
fi
done < "$HOSTS_FILE"
echo
echo "════════════════════════════════════════════════════════════════"
echo " summary"
echo "════════════════════════════════════════════════════════════════"
echo " ok: ${OK[*]:-(none)}"
echo " failed: ${FAIL[*]:-(none)}"
[[ ${#FAIL[@]} -eq 0 ]]

191
scripts/deploy.sh Executable file
View File

@@ -0,0 +1,191 @@
#!/usr/bin/env bash
# deploy.sh — sync this branch to the prod box and verify the cockpit is serving.
#
# Usage: scripts/deploy.sh
#
# Env vars (all have defaults — override only if your setup differs):
# PSYC_PROD_HOST default: neuronetz@cloud.neuronetz.ai
# PSYC_PROD_PATH default: /home/neuronetz/docker-public/neuro-psyc
# PSYC_PROD_URL default: https://psyc.neuronetz.ai
# PSYC_PROD_GPU set to 1 to also bring up the inference (GPU) service
# PSYC_GIT_REMOTE default: origin
# PSYC_BRANCH default: the currently checked-out branch
#
# What it does (idempotent — safe to re-run):
# 1. push the current branch to origin
# 2. ssh into prod, clone the repo if missing, pull the branch
# 3. docker compose up -d --build (+ gpu profile if PSYC_PROD_GPU=1)
# 4. probe :8767/healthz on the prod box + the public URL; report state
#
# What it does NOT do:
# • touch .env on the prod box (set keys there once, manually — gitignored)
# • transfer data/ or model artifacts (gitignored; prod fetches its own)
# • configure DNS or TLS (that's the reverse-proxy + acme-companion side)
set -euo pipefail
HOST="${PSYC_PROD_HOST:-neuronetz@cloud.neuronetz.ai}"
REMOTE_PATH="${PSYC_PROD_PATH:-/home/neuronetz/docker-public/neuro-psyc}"
PUBLIC_URL="${PSYC_PROD_URL:-https://psyc.neuronetz.ai}"
GIT_REMOTE="${PSYC_GIT_REMOTE:-origin}"
BRANCH="${PSYC_BRANCH:-$(git rev-parse --abbrev-ref HEAD)}"
WITH_GPU="${PSYC_PROD_GPU:-}"
# ── tty styling ─────────────────────────────────────────────────────────
if [[ -t 1 ]]; then
B=$'\e[1m'; D=$'\e[2m'; R=$'\e[31m'; G=$'\e[32m'; Y=$'\e[33m'; C=$'\e[36m'; Z=$'\e[0m'
else B=; D=; R=; G=; Y=; C=; Z=; fi
say() { printf "%s[deploy]%s %s\n" "$C" "$Z" "$*"; }
ok() { printf "%s[deploy]%s %s%s%s\n" "$G" "$Z" "$G" "$*" "$Z"; }
warn() { printf "%s[deploy]%s %s%s%s\n" "$Y" "$Z" "$Y" "$*" "$Z"; }
fail() { printf "%s[deploy]%s %s%s%s\n" "$R" "$Z" "$R" "$*" "$Z" >&2; exit 1; }
# ── 0. preflight ────────────────────────────────────────────────────────
command -v ssh >/dev/null || fail "ssh not installed locally"
command -v git >/dev/null || fail "git not installed locally"
command -v curl >/dev/null || fail "curl not installed locally"
[[ -d .git ]] || fail "run from the psyc repo root (no .git here)"
if ! git diff --quiet HEAD -- 2>/dev/null || ! git diff --cached --quiet 2>/dev/null; then
warn "local working tree has uncommitted changes — they won't be deployed (git push only sends commits)."
fi
GIT_URL=$(git config --get "remote.${GIT_REMOTE}.url") \
|| fail "no remote '${GIT_REMOTE}' configured locally"
# Parse the git URL to pull out the SSH host + port so the prod box can
# pre-trust the Gitea host key before its first clone. Supports both
# ssh://user@host:port/path and user@host:path
GIT_HOST=""; GIT_PORT="22"
if [[ "$GIT_URL" =~ ^ssh://[^@]+@([^:/]+)(:([0-9]+))?/ ]]; then
GIT_HOST="${BASH_REMATCH[1]}"
[[ -n "${BASH_REMATCH[3]:-}" ]] && GIT_PORT="${BASH_REMATCH[3]}"
elif [[ "$GIT_URL" =~ ^[^@]+@([^:]+): ]]; then
GIT_HOST="${BASH_REMATCH[1]}"
fi
# ── 1. local push ───────────────────────────────────────────────────────
say "pushing ${B}${BRANCH}${Z} to ${B}${GIT_REMOTE}${Z}"
git push "${GIT_REMOTE}" "${BRANCH}" || fail "git push failed — fix and retry"
LOCAL_REV=$(git rev-parse --short HEAD)
ok "pushed ${BRANCH} @ ${LOCAL_REV}"
# ── 2. remote bring-up ──────────────────────────────────────────────────
say "deploying to ${B}${HOST}:${REMOTE_PATH}${Z}"
COMPOSE_PROFILES=""
[[ -n "$WITH_GPU" ]] && COMPOSE_PROFILES="--profile gpu"
# heredoc runs on the prod box. Local vars are interpolated by THIS shell;
# remote vars start with \$ so they're set on the remote side.
ssh -o StrictHostKeyChecking=accept-new -T "${HOST}" bash -s <<REMOTE
set -euo pipefail
HOST_PATH="${REMOTE_PATH}"
BRANCH="${BRANCH}"
GIT_URL="${GIT_URL}"
GIT_HOST="${GIT_HOST}"
GIT_PORT="${GIT_PORT}"
COMPOSE_PROFILES="${COMPOSE_PROFILES}"
prn() { printf ' · %s\n' "\$*"; }
# 2a. trust the Gitea SSH host on first deploy.
#
# Boundary, intentional and narrow:
# • host trust (~/.ssh/known_hosts entry) → AUTO on first run. This is TOFU,
# same as what 'ssh -o StrictHostKeyChecking=accept-new' would do.
# • identity keys (~/.ssh/id_*) → NEVER touched. We won't
# generate, copy, or modify your private/public keypairs.
# Skip the auto-trust by setting PSYC_SKIP_HOST_TRUST=1 on your laptop.
if [[ -n "\$GIT_HOST" && -z "${PSYC_SKIP_HOST_TRUST:-}" ]]; then
mkdir -p ~/.ssh && chmod 700 ~/.ssh
KH_ENTRY="[\$GIT_HOST]:\$GIT_PORT"
if ! ssh-keygen -F "\$KH_ENTRY" -f ~/.ssh/known_hosts >/dev/null 2>&1; then
prn "adding \$KH_ENTRY to ~/.ssh/known_hosts (TOFU on first deploy)"
ssh-keyscan -T 5 -p "\$GIT_PORT" "\$GIT_HOST" 2>/dev/null >> ~/.ssh/known_hosts \
|| { echo "[deploy] couldn't reach \$GIT_HOST:\$GIT_PORT to fetch host key" >&2; exit 1; }
chmod 600 ~/.ssh/known_hosts
fi
fi
# 2b. ensure dir + working tree
if [[ ! -d "\$HOST_PATH/.git" ]]; then
prn "no working tree at \$HOST_PATH — cloning \$GIT_URL"
mkdir -p "\$(dirname "\$HOST_PATH")"
if ! git clone "\$GIT_URL" "\$HOST_PATH"; then
cat >&2 <<HINT
[deploy] git clone failed. Likely causes (check in order):
• Host key wasn't trusted → ssh -p \$GIT_PORT -T git@\$GIT_HOST to accept it once.
• No SSH identity key here, or its pubkey isn't in Gitea for this user.
ls ~/.ssh/id_* 2>/dev/null
(none?) → ssh-keygen -t ed25519
then: cat ~/.ssh/id_ed25519.pub # paste into Gitea → Settings → SSH Keys
• Repo URL wrong or you're not a collaborator on m17hr1l/psyc.
deploy.sh will NOT modify ~/.ssh — fix it once and re-run.
HINT
exit 1
fi
fi
cd "\$HOST_PATH"
# 2b. fetch + checkout + pull
prn "git fetch origin"
git fetch --quiet origin
prn "git checkout \$BRANCH"
git checkout --quiet "\$BRANCH" 2>/dev/null || git checkout --quiet -b "\$BRANCH" "origin/\$BRANCH"
prn "git pull --ff-only origin \$BRANCH"
git pull --quiet --ff-only origin "\$BRANCH"
REMOTE_REV=\$(git rev-parse --short HEAD)
prn "now at \$REMOTE_REV"
# 2c. .env sanity
if [[ ! -f .env ]]; then
prn "WARNING: .env missing — copying .env.example. Edit it before psyc fetch-all will work."
cp .env.example .env
fi
# 2d. external 'backend' network for nginx-proxy
if ! docker network ls --format '{{.Name}}' | grep -qx backend; then
prn "creating external docker network 'backend'"
docker network create backend
fi
# 2e. compose up
prn "docker compose up -d --build \$COMPOSE_PROFILES"
docker compose up -d --build \$COMPOSE_PROFILES
prn "container status:"
docker compose ps --format "table {{.Name}}\t{{.Status}}" | sed 's/^/ /'
REMOTE
ok "remote bring-up complete"
# ── 3. internal health probe (on the prod box localhost) ───────────────
say "probing ${B}127.0.0.1:8767/healthz${Z} on prod (up to 90s)…"
REMOTE_HEALTH=$(ssh -o StrictHostKeyChecking=accept-new "${HOST}" '
for i in $(seq 1 45); do
if curl -fs http://127.0.0.1:8767/healthz >/dev/null 2>&1; then echo OK; exit 0; fi
sleep 2
done
echo TIMEOUT')
if [[ "${REMOTE_HEALTH}" != *OK* ]]; then
fail "cockpit unhealthy on prod after 90s — ssh ${HOST}, cd ${REMOTE_PATH}, run 'docker compose logs cockpit' to debug"
fi
ok "cockpit healthy on prod"
# ── 4. external probe via the public URL ────────────────────────────────
say "probing ${B}${PUBLIC_URL}/healthz${Z} from here…"
if curl --max-time 8 -fs "${PUBLIC_URL}/healthz" >/dev/null 2>&1; then
INF=$(curl --max-time 5 -s "${PUBLIC_URL}/api/inference-status" || printf '%s' '{}')
ok "${PUBLIC_URL} is LIVE"
printf " inference: %s\n" "${INF}"
else
warn "public URL not reachable from here — most likely DNS or TLS isn't finished"
warn " diag:"
warn " dig +short psyc.neuronetz.ai → expect A record to prod IP"
warn " on the prod-host: docker logs acme-companion --tail 30"
warn " cockpit IS healthy on prod-internal :8767 — the app is fine, the front isn't there yet"
fi
ok "done — deployed ${BRANCH} @ ${LOCAL_REV}"

6
scripts/hosts.example Normal file
View File

@@ -0,0 +1,6 @@
# Federation hosts — one node per line.
# Format: LABEL SSH_TARGET REMOTE_PATH PUBLIC_URL
# Lines starting with # are ignored. Copy to scripts/hosts and edit.
prod neuronetz@cloud.neuronetz.ai /home/neuronetz/docker-public/neuro-psyc https://psyc.neuronetz.ai
sto user@sto-host.example /path/to/neuro-psyc https://psyc.maschinen-stockert.de
bittomine user@bittomine-host.example /path/to/neuro-psyc https://psyc.bittomine.com

353
src/psyc/_federation_cli.py Normal file
View File

@@ -0,0 +1,353 @@
"""Federation CLI — keygen, DNS records, feed export, peer registry, verify.
Registered onto the top-level Typer app from cli.py so the surface stays flat.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import List, Optional
import httpx
import typer
from psyc import db, log
from psyc.lines import discovery, federation, network_view, pulse, topology_export, translog
from psyc.result import Err, Ok
_log = log.get(__name__)
def register(typer_app: typer.Typer) -> None:
"""Mount all `fed-*` commands onto `typer_app`."""
@typer_app.command("fed-keygen")
def fed_keygen() -> None:
"""Generate the node's Ed25519 keypair (or load existing). Prints fingerprint."""
federation.node_keypair() # creates the files if missing
typer.echo(federation.node_fingerprint())
@typer_app.command("fed-dns")
def fed_dns(
domain: str = typer.Argument(..., help="public domain to advertise this node on"),
port: int = typer.Option(443, "--port", help="port psyc is reachable on"),
) -> None:
"""Print the DNS SRV + TXT records to publish under `domain`."""
rec = federation.dns_record(domain, port=port)
typer.echo(rec.human_instructions)
@typer_app.command("fed-feed")
def fed_feed(
window_hours: int = typer.Option(24, "--hours", help="lookback window (hours)"),
) -> None:
"""Build + print the signed feed JSON."""
db.init_db()
payload = federation.build_signed_feed(window_hours=window_hours)
typer.echo(json.dumps(payload, indent=2))
@typer_app.command("fed-verify")
def fed_verify(
peer_url: str = typer.Argument(..., help="peer base URL, e.g. https://peer.example"),
) -> None:
"""Fetch a peer's /federation/{info,key,feed} and verify the signature."""
peer_url = peer_url.rstrip("/")
try:
with httpx.Client(timeout=10.0) as client:
info = client.get(f"{peer_url}/federation/info").json()
key_text = client.get(f"{peer_url}/federation/key").text
feed = client.get(f"{peer_url}/federation/feed").json()
except Exception as exc:
typer.echo(f"error: fetch failed: {exc}", err=True)
raise typer.Exit(1)
# If the peer is already in the registry, prefer the stored pubkey
# (TOFU pin); otherwise warn and use the freshly fetched one.
declared_fp = info.get("fingerprint", "")
pubkey_pem = key_text
pinned = None
for p in federation.list_peers():
if p.fingerprint == declared_fp:
pinned = p
break
if pinned:
pubkey_pem = pinned.pubkey_pem
typer.echo(f" · using pinned pubkey for {pinned.domain}")
else:
typer.echo(" · WARNING: no pinned pubkey for this peer — trusting fetched key (TOFU)")
db.init_db()
result = federation.import_signed_feed(feed, pubkey_pem)
if isinstance(result, Err):
typer.echo(f" ✗ verification failed: {result.reason}", err=True)
raise typer.Exit(1)
s = result.value
typer.echo(f" ✓ verified peer {s.peer_fingerprint}")
typer.echo(f" cases: {s.cases_seen} iocs: {s.iocs_seen} signals buffered: {len(s.signal_ids)}")
@typer_app.command("fed-peer-add")
def fed_peer_add(
domain: str = typer.Argument(..., help="peer's public domain"),
fingerprint: str = typer.Argument(..., help="peer's 32-hex fingerprint"),
pubkey_file: Path = typer.Option(..., "--pubkey-file", help="path to peer's PEM public key"),
status: str = typer.Option("unknown", "--status", help="unknown | trusted | blocked"),
) -> None:
"""Register a peer's identity in the local registry."""
db.init_db()
pem = pubkey_file.read_text(encoding="utf-8")
federation.register_peer(domain, fingerprint, pem, status=status)
typer.echo(f"registered peer {domain} ({fingerprint[:8]}…) status={status}")
@typer_app.command("fed-peer-list")
def fed_peer_list() -> None:
"""List all registered peers."""
db.init_db()
rows = federation.list_peers()
if not rows:
typer.echo("(no peers registered)")
return
for p in rows:
typer.echo(
f" {p.status:8s} {p.domain:30s} {p.fingerprint[:8]}{p.fingerprint[-8:]}"
f" last_seen={(p.last_seen or '')[:16]}"
)
@typer_app.command("fed-peer-trust")
def fed_peer_trust(domain: str = typer.Argument(...)) -> None:
"""Mark a peer as trusted — their signals count toward quorum."""
db.init_db()
federation.set_peer_status(domain, "trusted")
typer.echo(f"{domain} → trusted")
@typer_app.command("fed-peer-block")
def fed_peer_block(domain: str = typer.Argument(...)) -> None:
"""Block a peer — ignore their feeds."""
db.init_db()
federation.set_peer_status(domain, "blocked")
typer.echo(f"{domain} → blocked")
@typer_app.command("fed-peer-remove")
def fed_peer_remove(domain: str = typer.Argument(...)) -> None:
"""Drop a peer from the registry."""
db.init_db()
federation.remove_peer(domain)
typer.echo(f"removed {domain}")
# ---------- discovery (DNS-SD walker) ----------------------------------
@typer_app.command("fed-resolve")
def fed_resolve(
domain: str = typer.Argument(..., help="domain to look up via _psyc._tcp.<domain>"),
timeout: float = typer.Option(5.0, "--timeout", help="DNS lookup timeout, seconds"),
) -> None:
"""Resolve a domain's psyc DNS-SD record. Prints fingerprint + port."""
result = discovery.resolve_psyc(domain, timeout=timeout)
if isinstance(result, Err):
typer.echo(f"error: {result.reason}", err=True)
raise typer.Exit(1)
c = result.value
typer.echo(f" domain {c.domain}")
typer.echo(f" fingerprint {c.fingerprint}")
typer.echo(f" port {c.port}")
typer.echo(f" source {c.source}")
@typer_app.command("fed-walk")
def fed_walk(
seeds: List[str] = typer.Argument(..., help="one or more seed domains"),
depth: int = typer.Option(2, "--depth", help="max BFS depth"),
max_peers: int = typer.Option(200, "--max-peers", help="cap discovered candidates"),
record: bool = typer.Option(False, "--record", help="persist candidates as status=unknown"),
) -> None:
"""Walk DNS-SD + peer-public from `seeds`. Prints discovered table."""
db.init_db()
cands = discovery.walk(seeds, max_depth=depth, max_peers=max_peers)
if not cands:
typer.echo("(no candidates discovered)")
return
typer.echo(f"{'domain':<32} {'fingerprint':<24} {'port':>5} source")
for c in cands:
fp = f"{c.fingerprint[:8]}{c.fingerprint[-8:]}"
typer.echo(f"{c.domain:<32} {fp:<24} {c.port:>5} {c.source}")
if record:
for c in cands:
discovery.record_candidate(c)
typer.echo(f"recorded {len(cands)} candidate(s) into peers table")
@typer_app.command("fed-seeds-list")
def fed_seeds_list() -> None:
"""Print the operator-curated discovery seed list."""
db.init_db()
seeds = pulse.get_discovery_seeds()
if not seeds:
typer.echo("(no seeds configured)")
return
for s in seeds:
typer.echo(s)
@typer_app.command("fed-seeds-add")
def fed_seeds_add(domain: str = typer.Argument(...)) -> None:
"""Append a seed domain (no-op if already present)."""
db.init_db()
seeds = pulse.get_discovery_seeds()
d = domain.strip()
if d in seeds:
typer.echo(f"{d} already a seed")
return
seeds.append(d)
pulse.set_discovery_seeds(seeds)
typer.echo(f"added seed {d}")
@typer_app.command("fed-seeds-remove")
def fed_seeds_remove(domain: str = typer.Argument(...)) -> None:
"""Remove a seed domain (no-op if absent)."""
db.init_db()
seeds = pulse.get_discovery_seeds()
d = domain.strip()
if d not in seeds:
typer.echo(f"{d} not in seeds")
return
seeds = [s for s in seeds if s != d]
pulse.set_discovery_seeds(seeds)
typer.echo(f"removed seed {d}")
# ---------- vouching + quorum --------------------------------------
@typer_app.command("fed-vouch")
def fed_vouch(
target_fp: str = typer.Argument(..., help="target peer fingerprint (32 hex)"),
ttl_days: int = typer.Option(90, "--ttl-days", help="vouch lifetime in days"),
) -> None:
"""Issue a signed vouch for `target_fp`. Persists locally + rides our feed."""
db.init_db()
v = federation.issue_vouch(target_fp.strip(), ttl_days=ttl_days)
typer.echo(f"vouched: {v.target_fingerprint} (expires {v.expires_at})")
@typer_app.command("fed-unvouch")
def fed_unvouch(target_fp: str = typer.Argument(...)) -> None:
"""Revoke OUR vouch for `target_fp`."""
db.init_db()
federation.revoke_vouch(target_fp.strip())
typer.echo(f"revoked vouch for {target_fp}")
@typer_app.command("fed-vouches")
def fed_vouches() -> None:
"""List vouches WE have issued."""
db.init_db()
rows = federation.our_vouches()
if not rows:
typer.echo("(no vouches issued)")
return
for v in rows:
exp = v.expires_at.isoformat() if v.expires_at else ""
typer.echo(f" {v.target_fingerprint} issued={v.issued_at.isoformat()[:16]} expires={exp[:16]}")
@typer_app.command("fed-quorum-set")
def fed_quorum_set(
trust: Optional[int] = typer.Option(None, "--trust", help="trust_min_vouchers"),
k: Optional[int] = typer.Option(None, "--k", help="signal_quorum_k"),
) -> None:
"""Update quorum thresholds. Either flag is optional — only changed values overwrite."""
db.init_db()
cfg = federation.quorum_config()
if trust is not None:
cfg.trust_min_vouchers = max(1, int(trust))
if k is not None:
cfg.signal_quorum_k = max(1, int(k))
federation.set_quorum_config(cfg)
typer.echo(f"quorum: trust_min_vouchers={cfg.trust_min_vouchers} signal_quorum_k={cfg.signal_quorum_k}")
# ---------- transparency log --------------------------------------
@typer_app.command("fed-log")
def fed_log(
limit: int = typer.Option(20, "--limit", help="number of entries to show"),
) -> None:
"""Print recent transparency-log entries (newest first)."""
db.init_db()
rows = translog.recent(limit=limit)
if not rows:
typer.echo("(transparency log empty)")
return
for e in rows:
typer.echo(
f" id={e.id:5d} {e.entry_type:6s} {e.timestamp[:19]} hash={e.entry_hash[:16]}"
)
# ---------- network view ------------------------------------------
@typer_app.command("fed-network")
def fed_network() -> None:
"""Print the local federation network view — nodes, vouches, stats."""
db.init_db()
view = network_view.build_local_view()
# Nodes table.
typer.echo("NODES")
typer.echo(f" {'fingerprint':<34} {'label':<32} {'status':<9} dist")
for n in view.nodes:
fp = f"{n.fingerprint[:8]}{n.fingerprint[-8:]}" if len(n.fingerprint) >= 16 else n.fingerprint
label = (n.label or "")[:30]
typer.echo(f" {fp:<34} {label:<32} {n.status:<9} {n.distance}")
# Vouches breakdown.
our_fp = view.nodes[0].fingerprint
vouch_out = [e for e in view.edges if e.kind == "vouch" and e.source_fingerprint == our_fp]
vouch_in = [e for e in view.edges if e.kind == "vouch" and e.target_fingerprint == our_fp]
bidir = [e for e in vouch_out if e.bidirectional]
typer.echo("")
typer.echo("VOUCHES")
if not vouch_out and not vouch_in and not bidir:
typer.echo(" (no vouches)")
else:
for e in vouch_out:
arrow = "" if e.bidirectional else ""
fp = f"{e.target_fingerprint[:8]}{e.target_fingerprint[-8:]}"
typer.echo(f" us {arrow} {fp}")
for e in vouch_in:
fp = f"{e.source_fingerprint[:8]}{e.source_fingerprint[-8:]}"
typer.echo(f" {fp} → us")
# Signal edges.
sig_edges = [e for e in view.edges if e.kind == "signal"]
typer.echo("")
typer.echo("SIGNALS (24h)")
if not sig_edges:
typer.echo(" (no signals)")
else:
for e in sig_edges:
fp = f"{e.source_fingerprint[:8]}{e.source_fingerprint[-8:]}"
typer.echo(f" from {fp}: {int(e.weight)}")
# Stats footer.
typer.echo("")
typer.echo("STATS")
for k, v in view.stats.items():
typer.echo(f" {k:<32} {v}")
@typer_app.command("fed-topology")
def fed_topology() -> None:
"""Print the sanitized docker topology JSON published at /federation/topology.
Useful for auditing what gets exposed to peers — pipe through `jq` to
confirm no env vars / volume mounts / IPs leak. On a dev box where
the docker-socket-proxy isn't running the export will be empty.
"""
db.init_db()
export = topology_export.build_export()
typer.echo(json.dumps(export.model_dump(mode="json"), indent=2))
@typer_app.command("fed-log-verify")
def fed_log_verify() -> None:
"""Re-walk the chain locally and report verification status."""
db.init_db()
result = translog.verify_chain()
head = translog.head()
head_hash = head.entry_hash if head else "(empty)"
if isinstance(result, Err):
typer.echo(f" ✗ broken: {result.reason}", err=True)
typer.echo(f" head_hash: {head_hash}")
raise typer.Exit(1)
typer.echo(f" ✓ verified {result.value} entries")
typer.echo(f" head_hash: {head_hash}")

182
src/psyc/_pulse_cli.py Normal file
View File

@@ -0,0 +1,182 @@
"""Typer commands for the Pulse scheduler.
Imported and wired by `cli.py` via `register(app)`. Kept as a separate module
so the main CLI surface stays grep-friendly and the scheduler can grow its own
verbs without bloating cli.py.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Optional
import typer
from psyc import db
from psyc.lines import pulse
from psyc.models import Severity
def _relative(dt: Optional[datetime]) -> str:
if dt is None:
return ""
now = datetime.now(timezone.utc)
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
delta = int((dt - now).total_seconds())
past = delta < 0
secs = abs(delta)
if secs < 5:
return "now"
if secs < 60:
unit = f"{secs}s"
elif secs < 3600:
unit = f"{secs // 60}m"
elif secs < 86400:
unit = f"{secs // 3600}h"
else:
unit = f"{secs // 86400}d"
return f"{unit} ago" if past else f"in {unit}"
def register(typer_app: typer.Typer) -> None:
"""Add pulse-* commands to the given Typer app."""
@typer_app.command("pulse-status")
def pulse_status() -> None:
"""Print the pipeline table (mode · cadence · last-fired · next-fire · last-result)."""
db.init_db()
ks = pulse.kill_switch_state()
typer.echo(f"kill switch: {'ARMED' if ks else 'OFF'}")
rows = pulse.state()
if not rows:
typer.echo("(no pipelines registered)")
return
typer.echo(f"{'name':<16} {'mode':<14} {'cadence':>8} {'enabled':<7} {'last':<10} {'next':<10} result")
for p in rows:
typer.echo(
f"{p.name:<16} {p.mode.value:<14} {p.cadence_seconds:>6}s "
f"{'yes' if p.enabled else 'no':<7} "
f"{_relative(p.last_fired):<10} {_relative(p.next_fire):<10} "
f"{(p.last_result or '')[:60]}"
)
@typer_app.command("pulse-tick")
def pulse_tick() -> None:
"""Run one scheduler heartbeat and print the per-pipeline outcomes."""
db.init_db()
results = pulse.tick()
for name, outcome, result in results:
marker = {"ok": "", "err": "", "skipped": ""}.get(outcome, "·")
typer.echo(f" {marker} {name:<16} {outcome:<8} {result[:120]}")
@typer_app.command("pulse-set-mode")
def pulse_set_mode(
name: str = typer.Argument(..., help="pipeline name"),
mode: str = typer.Argument(..., help="manual | auto-propose | auto-execute"),
) -> None:
db.init_db()
try:
pulse.set_mode(name, pulse.PulseMode(mode))
except ValueError as exc:
typer.echo(f"error: {exc}", err=True)
raise typer.Exit(1)
typer.echo(f"{name} mode → {mode}")
@typer_app.command("pulse-set-cadence")
def pulse_set_cadence(
name: str = typer.Argument(..., help="pipeline name"),
seconds: int = typer.Argument(..., help="cadence in seconds (>0)"),
) -> None:
db.init_db()
try:
pulse.set_cadence(name, seconds)
except ValueError as exc:
typer.echo(f"error: {exc}", err=True)
raise typer.Exit(1)
typer.echo(f"{name} cadence → {seconds}s")
@typer_app.command("pulse-run")
def pulse_run(name: str = typer.Argument(..., help="pipeline name")) -> None:
"""Manually fire one pipeline (bypasses cadence; honors kill switch)."""
db.init_db()
try:
outcome, result = pulse.run_now(name)
except ValueError as exc:
typer.echo(f"error: {exc}", err=True)
raise typer.Exit(1)
marker = {"ok": "", "err": "", "skipped": ""}.get(outcome, "·")
typer.echo(f" {marker} {name}: {outcome}{result}")
@typer_app.command("pulse-kill")
def pulse_kill() -> None:
"""Arm the kill switch — every pipeline halts on the next tick."""
db.init_db()
pulse.set_kill_switch(True)
typer.echo("kill switch ARMED — all pipelines halted")
@typer_app.command("pulse-unkill")
def pulse_unkill() -> None:
"""Disarm the kill switch — pulse resumes on the next tick."""
db.init_db()
pulse.set_kill_switch(False)
typer.echo("kill switch disarmed — pulse resumes")
@typer_app.command("pulse-respond-config")
def pulse_respond_config(
threshold: Optional[str] = typer.Option(
None, "--threshold", help="min severity: low | medium | high | critical"
),
quorum: Optional[bool] = typer.Option(
None, "--quorum/--no-quorum", help="require quorum on federation-sourced cases"
),
local_only: Optional[bool] = typer.Option(
None, "--local-only/--no-local-only",
help="when armed, auto-execute defers federation cases until quorum"
),
) -> None:
"""Update the respond-pipeline auto-fire gates. Args left unset are unchanged."""
db.init_db()
if threshold is not None:
try:
pulse.set_respond_auto_threshold(Severity(threshold))
except ValueError:
typer.echo(f"error: unknown severity {threshold!r}", err=True)
raise typer.Exit(1)
if quorum is not None:
pulse.set_respond_require_quorum(quorum)
if local_only is not None:
pulse.set_respond_local_only(local_only)
typer.echo(
f"threshold={pulse.respond_auto_threshold().value} "
f"quorum={'on' if pulse.respond_require_quorum() else 'off'} "
f"local-only={'on' if pulse.respond_local_only() else 'off'}"
)
@typer_app.command("pulse-respond-status")
def pulse_respond_status() -> None:
"""Print the respond-pipeline gates + the last 10 audit entries."""
db.init_db()
mode = "manual"
for p in pulse.state():
if p.name == "respond":
mode = p.mode.value
break
typer.echo(f"respond mode : {mode}")
typer.echo(f"threshold : {pulse.respond_auto_threshold().value}")
typer.echo(f"require quorum : {'yes' if pulse.respond_require_quorum() else 'no'}")
typer.echo(f"local-only : {'yes' if pulse.respond_local_only() else 'no'}")
audit = db.pulse_audit_recent("respond", limit=10)
if not audit:
typer.echo("(no audit entries yet)")
return
typer.echo("")
typer.echo(f"{'timestamp':<28} {'action':<11} {'case_id':<22} detail")
for row in audit:
typer.echo(
f"{(row['timestamp'] or '')[:27]:<28} "
f"{(row['action'] or ''):<11} "
f"{(row['case_id'] or ''):<22} "
f"{(row['detail'] or '')[:80]}"
)

View File

@@ -2,7 +2,7 @@
from __future__ import annotations
from typing import List
from typing import List, Optional
import typer
import uvicorn
@@ -13,15 +13,19 @@ from psyc import db, log
load_dotenv() # per-dev .env (API keys) is loaded into os.environ for venv CLI
from psyc.cockpit import inference
from psyc.lines import classify, courier, proof, route, scout, seal, train
from psyc.lines import classify, courier, lookup, proof, respond, route, scout, seal, train
from psyc.lines import map as map_line
from psyc.models import Outcome
from psyc.result import Err, Ok
from psyc._federation_cli import register as _register_federation_cli
from psyc._pulse_cli import register as _register_pulse_cli
app = typer.Typer(add_completion=False, help="psyc — defensive CTI routing & sealing")
log.configure()
_log = log.get(__name__)
_register_pulse_cli(app)
_register_federation_cli(app)
@app.command("init")
@@ -88,10 +92,34 @@ def fetch_feodo(limit: int = typer.Option(50, help="max C2 records to ingest"))
_ingest("feodo", limit)
@app.command("fetch-threatfox")
def fetch_threatfox(limit: int = typer.Option(200, help="max IOCs to ingest")) -> None:
"""ThreatFox (abuse.ch) — needs THREATFOX_AUTH_KEY in .env."""
_ingest("threatfox", limit)
@app.command("fetch-malware-bazaar")
def fetch_malware_bazaar(limit: int = typer.Option(100, help="max samples to ingest")) -> None:
"""MalwareBazaar (abuse.ch) — also uses THREATFOX_AUTH_KEY."""
_ingest("malware-bazaar", limit)
@app.command("fetch-otx")
def fetch_otx(limit: int = typer.Option(100, help="max pulse-cases to ingest")) -> None:
"""AlienVault OTX — needs OTX_API_KEY in .env."""
_ingest("otx", limit)
@app.command("fetch-all")
def fetch_all() -> None:
for source, limit in (("urlhaus", 50), ("cisa-kev", 100), ("feodo", 50)):
_ingest(source, limit)
"""Fetch every configured source. Keyed feeds skip cleanly when the key is missing."""
plan = (("urlhaus", 50), ("cisa-kev", 100), ("feodo", 50),
("threatfox", 200), ("malware-bazaar", 100), ("otx", 100))
for source, limit in plan:
try:
_ingest(source, limit)
except Exception as exc: # noqa: BLE001 — keep going if one feed misbehaves
typer.echo(f" skip {source}: {exc}", err=True)
@app.command("classify-case")
@@ -284,6 +312,151 @@ def submit_case(case_id: str) -> None:
typer.echo(f"{b.destination_name}: {b.reason} (logged)")
@app.command("queue")
def queue_list(
status: str = typer.Option("pending", help="pending | approved | rejected | all"),
limit: int = typer.Option(50, help="max rows"),
) -> None:
"""List the approval queue."""
from psyc.models import ApprovalStatus
status_filter = None if status == "all" else ApprovalStatus(status)
rows = courier.list_pending(status=status_filter, limit=limit)
if not rows:
typer.echo(f"(no submissions with status={status})")
return
for p in rows:
rev = f" by {p.reviewer}" if p.reviewer else ""
typer.echo(
f" #{p.id} {p.status.value:9s} {p.destination_name:16s} {p.case_id} "
f"({p.payload_kind}, tlp={p.tlp.value}){rev}"
)
@app.command("approve")
def approve(
pending_id: int = typer.Argument(..., help="pending submission id"),
reviewer: str = typer.Option("operator", "--by", help="reviewer identity"),
) -> None:
"""Approve a pending submission — dispatches to its destination."""
result = courier.dispatch_pending(pending_id, reviewer=reviewer)
if isinstance(result, Err):
typer.echo(f"error: {result.reason}", err=True)
raise typer.Exit(1)
r = result.value
rcpt = f"{r.receipt_id}" if r.receipt_id else ""
typer.echo(f"approved #{pending_id} · {r.destination_name}: {r.outcome.value}{rcpt}")
@app.command("reject")
def reject(
pending_id: int = typer.Argument(..., help="pending submission id"),
reviewer: str = typer.Option("operator", "--by", help="reviewer identity"),
reason: str = typer.Option("", "--reason", help="rejection reason"),
) -> None:
"""Reject a pending submission — nothing leaves; ledger row written."""
result = courier.reject_pending(pending_id, reviewer=reviewer, reason=reason)
if isinstance(result, Err):
typer.echo(f"error: {result.reason}", err=True)
raise typer.Exit(1)
typer.echo(f"rejected #{pending_id}{(': ' + reason) if reason else ''}")
@app.command("reindex")
def reindex() -> None:
"""Rebuild the IOC index from all cases."""
db.init_db() # ensure the iocs table exists (idempotent)
cases = db.list_cases(limit=1_000_000)
n = lookup.reindex(cases)
typer.echo(f"indexed {n} IOC(s) from {len(cases)} case(s). total: {db.ioc_count()}")
@app.command("lookup")
def lookup_ioc(value: str = typer.Argument(..., help="indicator: IP, domain, URL, hash, or CVE")) -> None:
"""Look up an indicator across the case corpus."""
rows = lookup.lookup(value)
if not rows:
typer.echo(f"'{value}' — not found in the corpus (no known-bad match)")
return
typer.echo(f"'{value}'{len(rows)} match(es):")
for r in rows:
sev = r["severity"] or "?"
typer.echo(f" [{r['ioc_type']}] {r['case_id']} feed={r['feed'] or '?'} severity={sev} seen={(r['first_seen'] or '')[:10]}")
@app.command("export-blocklist")
def export_blocklist(
ioc_type: str = typer.Option("ip", "--type", "-t", help=f"one of: {', '.join(lookup.IOC_TYPES)}"),
min_severity: str = typer.Option("", "--min-severity", help="low | medium | high | critical"),
out: str = typer.Option("", "--out", help="write to file instead of stdout"),
) -> None:
"""Emit a deduplicated blocklist of indicators (firewall/DNS/SIEM ingestion)."""
values = lookup.export_blocklist(ioc_type, min_severity or None)
text = "\n".join(values)
if out:
from pathlib import Path as _Path
_Path(out).write_text(text + "\n", encoding="utf-8")
typer.echo(f"wrote {len(values)} {ioc_type}(s) → {out}")
else:
typer.echo(text)
@app.command("respond")
def respond_propose(case_id: str = typer.Argument(..., help="case to propose response actions for")) -> None:
"""Propose human-gated response actions for a high-severity case."""
result = db.get_case(case_id)
if isinstance(result, Err):
typer.echo(f"error: {result.reason}", err=True)
raise typer.Exit(1)
ids = respond.propose_for_case(result.value)
if not ids:
typer.echo(f"{case_id}: no actions proposed (not high-severity, or already has actions)")
return
typer.echo(f"{case_id}: proposed {len(ids)} action(s) → ids {', '.join(map(str, ids))}")
@app.command("actions")
def actions_list(status: str = typer.Option("proposed", help="proposed | executed | rejected | failed | all")) -> None:
"""List response actions."""
from psyc.models import ActionStatus
sf = None if status == "all" else ActionStatus(status)
rows = respond.list_actions(status=sf)
if not rows:
typer.echo(f"(no actions with status={status})")
return
for a in rows:
appr = f" by {a.approver}" if a.approver else ""
typer.echo(f" #{a.id} {a.status.value:9s} [{a.action_type.value:9s}] {a.case_id} sev={a.severity or '?'}{appr}")
typer.echo(f" {a.summary}")
@app.command("act-approve")
def act_approve(
action_id: int = typer.Argument(..., help="response action id"),
approver: str = typer.Option("operator", "--by", help="approver identity"),
) -> None:
"""Approve + fire a response action (pushes to the enforcement sink)."""
result = respond.execute_action(action_id, approver=approver)
if isinstance(result, Err):
typer.echo(f"error: {result.reason}", err=True)
raise typer.Exit(1)
a = result.value
typer.echo(f"⚡ enforced #{action_id} [{a.action_type.value}] → {a.detail}")
@app.command("act-reject")
def act_reject(
action_id: int = typer.Argument(..., help="response action id"),
approver: str = typer.Option("operator", "--by", help="reviewer identity"),
reason: str = typer.Option("", "--reason", help="why declined"),
) -> None:
"""Decline a proposed response action — nothing fires."""
result = respond.reject_action(action_id, approver=approver, reason=reason)
if isinstance(result, Err):
typer.echo(f"error: {result.reason}", err=True)
raise typer.Exit(1)
typer.echo(f"declined #{action_id}{(': ' + reason) if reason else ''}")
@app.command("mock-cert")
def mock_cert_serve(host: str = "127.0.0.1", port: int = 8770) -> None:
uvicorn.run("psyc.mock_cert:app", host=host, port=port)
@@ -293,12 +466,15 @@ def mock_cert_serve(host: str = "127.0.0.1", port: int = 8770) -> None:
def train_build(
task: str = typer.Option(..., "--task", "-t", help=f"one of: {', '.join(train.TASKS)}"),
limit: int = typer.Option(10_000, help="max cases to process"),
defang_frac: float = typer.Option(0.0, "--defang-frac", help="fraction of ioc_extraction inputs to defang ([0.0, 1.0])"),
seed: Optional[int] = typer.Option(None, "--seed", help="rng seed for reproducible defanging"),
) -> None:
if task not in train.TASKS:
typer.echo(f"unknown task: {task}; choices: {', '.join(train.TASKS)}", err=True)
raise typer.Exit(1)
cases = db.list_cases(limit=limit)
report = train.build(task, cases)
options = train.BuildOptions(defang_frac=defang_frac, seed=seed)
report = train.build(task, cases, options=options)
typer.echo(f"task: {report.task}")
typer.echo(f"path: {report.path}")
typer.echo(f" written: {report.written}")
@@ -309,10 +485,15 @@ def train_build(
@app.command("train-build-all")
def train_build_all(limit: int = typer.Option(10_000, help="max cases per task")) -> None:
def train_build_all(
limit: int = typer.Option(10_000, help="max cases per task"),
defang_frac: float = typer.Option(0.0, "--defang-frac", help="fraction of ioc_extraction inputs to defang ([0.0, 1.0])"),
seed: Optional[int] = typer.Option(None, "--seed", help="rng seed for reproducible defanging"),
) -> None:
cases = db.list_cases(limit=limit)
options = train.BuildOptions(defang_frac=defang_frac, seed=seed)
for task in train.TASKS:
report = train.build(task, cases)
report = train.build(task, cases, options=options)
typer.echo(f" {task}: wrote {report.written}{report.path.name}")

View File

@@ -0,0 +1,179 @@
"""Admin gate — per-member TOTP auth for the hidden /admin zone.
Each project member enrolls their own authenticator (own secret, own QR) under
a named slot. Login accepts a code matching ANY active member, so offboarding
is a per-member revoke — no shared secret, no re-enrolling everyone when one
person leaves. The first visit bootstraps an "owner" slot; further members are
added from inside the authenticated admin panel. State persists under DATA_DIR
(gitignored).
"""
from __future__ import annotations
import base64
import io
import json
import secrets
from datetime import datetime, timezone
from typing import List, Optional, Tuple
import pyotp
import qrcode
from psyc import DATA_DIR, log
_log = log.get(__name__)
_STATE_PATH = DATA_DIR / "admin_auth.json"
_ISSUER = "psyc"
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _load() -> dict:
if _STATE_PATH.exists():
data = json.loads(_STATE_PATH.read_text())
# Migrate the old single-secret format → member list.
if "members" not in data:
members = []
pending = data.get("totp_secret")
if data.get("provisioned") and data.get("totp_secret"):
members = [_new_member("owner", data["totp_secret"])]
pending = None
data = {
"session_secret": data.get("session_secret", secrets.token_urlsafe(32)),
"pending_secret": pending,
"members": members,
}
_save(data)
return data
data = {
"session_secret": secrets.token_urlsafe(32),
"pending_secret": pyotp.random_base32(),
"members": [],
}
_save(data)
_log.info("adminauth.initialized", path=str(_STATE_PATH))
return data
def _save(state: dict) -> None:
_STATE_PATH.parent.mkdir(parents=True, exist_ok=True)
_STATE_PATH.write_text(json.dumps(state, indent=2))
def _new_member(label: str, secret: str) -> dict:
return {
"id": secrets.token_hex(4),
"label": label,
"secret": secret,
"created_at": _now(),
"active": True,
"last_used": None,
}
def session_secret() -> str:
return _load()["session_secret"]
def members() -> List[dict]:
"""Active members, without exposing their secrets."""
return [
{k: m[k] for k in ("id", "label", "created_at", "last_used")}
for m in _load()["members"] if m.get("active")
]
def is_bootstrapped() -> bool:
return any(m.get("active") for m in _load()["members"])
def verify(code: str) -> Optional[str]:
"""Check a code against every active member (and the bootstrap slot).
Returns the matched member label, or None. Updates last_used; promotes the
pending bootstrap secret into the first 'owner' member on first success.
"""
code = code.strip()
state = _load()
for m in state["members"]:
if m.get("active") and pyotp.TOTP(m["secret"]).verify(code, valid_window=1):
m["last_used"] = _now()
_save(state)
_log.info("adminauth.verify.ok", member=m["label"])
return m["label"]
# Bootstrap: no active members yet → accept the pending secret as owner.
if not any(m.get("active") for m in state["members"]) and state.get("pending_secret"):
if pyotp.TOTP(state["pending_secret"]).verify(code, valid_window=1):
owner = _new_member("owner", state["pending_secret"])
owner["last_used"] = _now()
state["members"].append(owner)
state["pending_secret"] = None
_save(state)
_log.info("adminauth.bootstrapped")
return "owner"
_log.info("adminauth.verify.fail")
return None
def add_member(label: str) -> Tuple[str, str]:
"""Enroll a new member. Returns (member_id, QR data-uri) to hand to them."""
label = (label or "member").strip()[:40]
secret = pyotp.random_base32()
state = _load()
m = _new_member(label, secret)
state["members"].append(m)
_save(state)
_log.info("adminauth.member.added", member=label, id=m["id"])
return m["id"], _qr_for(secret, label)
def revoke_member(member_id: str) -> bool:
state = _load()
for m in state["members"]:
if m["id"] == member_id and m.get("active"):
m["active"] = False
m["revoked_at"] = _now()
_save(state)
_log.info("adminauth.member.revoked", id=member_id, label=m["label"])
return True
return False
def member_qr(member_id: str) -> Optional[str]:
"""One-time QR for a just-created member (admin-only surface)."""
for m in _load()["members"]:
if m["id"] == member_id and m.get("active"):
return _qr_for(m["secret"], m["label"])
return None
def bootstrap_qr() -> str:
"""QR for the initial owner enrollment (only meaningful before bootstrap)."""
state = _load()
if not state.get("pending_secret"):
state["pending_secret"] = pyotp.random_base32()
_save(state)
return _qr_for(state["pending_secret"], "owner")
def _qr_for(secret: str, label: str) -> str:
uri = pyotp.TOTP(secret).provisioning_uri(name=label, issuer_name=_ISSUER)
img = qrcode.make(uri)
buf = io.BytesIO()
img.save(buf, format="PNG")
return "data:image/png;base64," + base64.b64encode(buf.getvalue()).decode()
def current_code(member_id: Optional[str] = None) -> str:
"""Live code for tests/local verification — never shown in the UI."""
state = _load()
if member_id:
for m in state["members"]:
if m["id"] == member_id:
return pyotp.TOTP(m["secret"]).now()
return pyotp.TOTP(state["pending_secret"]).now()

View File

@@ -5,14 +5,19 @@ from __future__ import annotations
from pathlib import Path
from typing import List
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import HTMLResponse
from fastapi import FastAPI, Form, HTTPException, Request
from fastapi.responses import FileResponse, HTMLResponse, PlainTextResponse, RedirectResponse
from fastapi.staticfiles import StaticFiles
from fastapi.templating import Jinja2Templates
from starlette.middleware.sessions import SessionMiddleware
from psyc import db, log
from psyc.cockpit import inference, journey as journey_view
from psyc.cockpit import adminauth, case_visuals, docker_view, federation_routes, inference, journey as journey_view, pulse_routes
from psyc.lines import courier as courier_line
from psyc.lines import ledger as ledger_line
from psyc.lines import lookup as lookup_line
from psyc.lines import news as news_line
from psyc.lines import respond as respond_line
from psyc.lines import route as route_line
from psyc.lines import seal as seal_line
from psyc.lines import train as train_line
@@ -27,14 +32,54 @@ log.configure()
_log = log.get(__name__)
app = FastAPI(title="psyc Operations Cockpit", version="0.1.0")
app.add_middleware(SessionMiddleware, secret_key=adminauth.session_secret(), max_age=3600)
app.mount("/static", StaticFiles(directory=str(HERE / "static")), name="static")
@app.middleware("http")
async def _security_headers(request: Request, call_next):
"""Defense-in-depth headers. CSP is intentionally NOT set yet — the
cockpit currently uses inline scripts in base.html / journey.html /
federation_explore.html which would need nonces or extraction first."""
resp = await call_next(request)
resp.headers.setdefault("X-Content-Type-Options", "nosniff")
resp.headers.setdefault("Referrer-Policy", "no-referrer")
resp.headers.setdefault("X-Frame-Options", "DENY")
return resp
pulse_routes.register(app, TEMPLATES)
federation_routes.register(app, TEMPLATES)
def _admin_ok(request: Request) -> bool:
return bool(request.session.get("admin_ok"))
@app.get("/", response_class=HTMLResponse)
def index(request: Request) -> HTMLResponse:
cases = db.list_cases(limit=200)
total = db.case_count()
return TEMPLATES.TemplateResponse(request, "cases.html", {"cases": cases, "total": total})
items = news_line.recent_items(limit=40)
featured = news_line.featured_case()
case_index: dict = {}
for i in items:
if i.kind == "case" and i.case_id and i.case_id not in case_index:
got = db.get_case(i.case_id)
if not isinstance(got, Err):
case_index[i.case_id] = case_visuals.case_glyph_svg(got.value)
buckets = news_line.bucket_items(items)
return TEMPLATES.TemplateResponse(
request,
"home.html",
{
"kpis": news_line.kpis(),
"buckets": buckets,
"total_items": sum(len(b.items) for b in buckets),
"feeds": news_line.feed_health(),
"featured": featured,
"featured_hero": case_visuals.case_hero_svg(featured) if featured else "",
"case_glyphs": case_index,
},
)
@app.get("/cases", response_class=HTMLResponse)
@@ -76,7 +121,12 @@ def case_journey(request: Request, case_id: str) -> HTMLResponse:
if isinstance(result, Err):
raise HTTPException(status_code=404, detail=result.reason)
beats = journey_view.build_journey(result.value)
return TEMPLATES.TemplateResponse(request, "journey.html", {"case": result.value, "beats": beats})
model_label = inference.adapter_name() or "rules"
return TEMPLATES.TemplateResponse(
request,
"journey.html",
{"case": result.value, "beats": beats, "model_label": model_label},
)
@app.get("/ledger", response_class=HTMLResponse)
@@ -98,7 +148,173 @@ def healthz() -> dict:
return {"status": "ok"}
# PWA service worker — must live at the root so its scope is the whole site.
# Static file is on disk under /static/sw.js; this route just serves it from /.
@app.get("/sw.js", include_in_schema=False)
def service_worker() -> FileResponse:
return FileResponse(
HERE / "static" / "sw.js",
media_type="application/javascript",
headers={"Service-Worker-Allowed": "/", "Cache-Control": "no-cache"},
)
@app.get("/api/inference-status")
def inference_status() -> dict:
adapter = inference.server_adapter()
return {"online": adapter is not None, "adapter": adapter}
@app.get("/lookup", response_class=HTMLResponse)
def lookup_view(request: Request, q: str = "") -> HTMLResponse:
query = q.strip()
matches = lookup_line.lookup(query) if query else []
counts = {t: len(lookup_line.export_blocklist(t)) for t in lookup_line.IOC_TYPES}
return TEMPLATES.TemplateResponse(
request,
"lookup.html",
{
"query": query,
"matches": matches,
"searched": bool(query),
"total_iocs": db.ioc_count(),
"counts": counts,
},
)
@app.get("/export/blocklist", response_class=PlainTextResponse)
def export_blocklist(type: str = "ip", min_severity: str = "") -> PlainTextResponse:
if type not in lookup_line.IOC_TYPES:
raise HTTPException(status_code=400, detail=f"unknown type: {type}")
values = lookup_line.export_blocklist(type, min_severity or None)
header = f"# psyc blocklist — type={type} min_severity={min_severity or 'any'} count={len(values)}\n"
return PlainTextResponse(header + "\n".join(values) + "\n")
@app.get("/response", response_class=HTMLResponse)
def response_view(request: Request, status: str = "proposed", fired: int = 0, kind: str = "") -> HTMLResponse:
from psyc.models import ActionStatus
sf = None if status == "all" else ActionStatus(status)
actions = respond_line.list_actions(status=sf, limit=200)
counts = {
"proposed": respond_line.action_count(ActionStatus.PROPOSED),
"executed": respond_line.action_count(ActionStatus.EXECUTED),
"rejected": respond_line.action_count(ActionStatus.REJECTED),
"failed": respond_line.action_count(ActionStatus.FAILED),
}
return TEMPLATES.TemplateResponse(
request,
"response.html",
{"actions": actions, "counts": counts, "current_status": status, "fired": fired, "fired_kind": kind},
)
@app.post("/response/approve/{action_id}")
def response_approve(action_id: int, approver: str = Form("operator")) -> RedirectResponse:
result = respond_line.execute_action(action_id, approver=approver)
if isinstance(result, Err):
_log.warning("cockpit.response.approve.error", action_id=action_id, reason=result.reason)
return RedirectResponse("/response", status_code=303)
# Carry the fired action id + type so the page can set off the disco.
kind = result.value.action_type.value
return RedirectResponse(f"/response?fired={action_id}&kind={kind}", status_code=303)
@app.post("/response/reject/{action_id}")
def response_reject(action_id: int, approver: str = Form("operator"), reason: str = Form("")) -> RedirectResponse:
respond_line.reject_action(action_id, approver=approver, reason=reason)
return RedirectResponse("/response", status_code=303)
# ---------- hidden admin zone (TOTP-gated) -------------------------------
@app.get("/admin", response_class=HTMLResponse)
def admin_home(request: Request) -> HTMLResponse:
if _admin_ok(request):
enrolled = request.query_params.get("enrolled", "")
new_qr = adminauth.member_qr(enrolled) if enrolled else None
return TEMPLATES.TemplateResponse(
request, "admin.html",
{"members": adminauth.members(), "who": request.session.get("admin_who", ""),
"new_qr": new_qr, "new_label": request.query_params.get("label", "")},
)
# Not authenticated — show the gate. Bootstrap QR only until first member exists.
ctx = {"provisioned": adminauth.is_bootstrapped(), "error": request.query_params.get("error", "")}
if not ctx["provisioned"]:
ctx["qr"] = adminauth.bootstrap_qr()
return TEMPLATES.TemplateResponse(request, "admin_gate.html", ctx)
@app.post("/admin/verify")
def admin_verify(request: Request, code: str = Form(...)) -> RedirectResponse:
who = adminauth.verify(code)
if who:
request.session["admin_ok"] = True
request.session["admin_who"] = who
return RedirectResponse("/admin", status_code=303)
return RedirectResponse("/admin?error=1", status_code=303)
@app.post("/admin/members")
def admin_add_member(request: Request, label: str = Form("member")) -> RedirectResponse:
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
member_id, _ = adminauth.add_member(label)
return RedirectResponse(f"/admin?enrolled={member_id}&label={label}", status_code=303)
@app.post("/admin/members/{member_id}/revoke")
def admin_revoke_member(request: Request, member_id: str) -> RedirectResponse:
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
adminauth.revoke_member(member_id)
return RedirectResponse("/admin", status_code=303)
@app.get("/admin/logout")
def admin_logout(request: Request) -> RedirectResponse:
request.session.pop("admin_ok", None)
request.session.pop("admin_who", None)
return RedirectResponse("/admin", status_code=303)
@app.get("/admin/docker", response_class=HTMLResponse)
def admin_docker(request: Request) -> HTMLResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
topo = docker_view.topology()
return TEMPLATES.TemplateResponse(request, "admin_docker.html", {"topo": topo})
@app.get("/queue", response_class=HTMLResponse)
def queue_view(request: Request, status: str = "pending") -> HTMLResponse:
from psyc.models import ApprovalStatus
status_filter = None if status == "all" else ApprovalStatus(status)
rows = courier_line.list_pending(status=status_filter, limit=200)
counts = {
"pending": courier_line.pending_count(ApprovalStatus.PENDING),
"approved": courier_line.pending_count(ApprovalStatus.APPROVED),
"rejected": courier_line.pending_count(ApprovalStatus.REJECTED),
}
return TEMPLATES.TemplateResponse(
request,
"queue.html",
{"rows": rows, "counts": counts, "current_status": status},
)
@app.post("/queue/approve/{pid}")
def queue_approve(pid: int, reviewer: str = Form("operator")) -> RedirectResponse:
result = courier_line.dispatch_pending(pid, reviewer=reviewer)
if isinstance(result, Err):
_log.warning("cockpit.queue.approve.error", pending_id=pid, reason=result.reason)
return RedirectResponse("/queue", status_code=303)
@app.post("/queue/reject/{pid}")
def queue_reject(pid: int, reviewer: str = Form("operator"), reason: str = Form("")) -> RedirectResponse:
result = courier_line.reject_pending(pid, reviewer=reviewer, reason=reason)
if isinstance(result, Err):
_log.warning("cockpit.queue.reject.error", pending_id=pid, reason=result.reason)
return RedirectResponse("/queue", status_code=303)

View File

@@ -0,0 +1,182 @@
"""Procedural SVG visuals derived from case data.
Zero external image-gen, zero curated assets — every visual is generated
server-side from the case_id (deterministic per case) and severity. Cyber-HUD
aesthetic, theme-coordinated with cockpit.css.
"""
from __future__ import annotations
import hashlib
import math
from typing import Optional
from psyc.models import Case, Severity
_SEV_ACCENT = {
Severity.CRITICAL: "#f87171",
Severity.HIGH: "#fbbf24",
Severity.MEDIUM: "#1ec8ff",
Severity.LOW: "#7d8597",
}
_DEFAULT_ACCENT = "#1ec8ff"
def _seed(case_id: str) -> int:
"""Stable 32-bit int seed from a string — deterministic per case."""
return int.from_bytes(hashlib.sha256(case_id.encode()).digest()[:4], "big")
def _prng(seed: int):
"""Mulberry32 — small fast deterministic PRNG. Yields floats in [0,1)."""
state = [seed & 0xFFFFFFFF]
def nxt():
state[0] = (state[0] + 0x6D2B79F5) & 0xFFFFFFFF
t = state[0]
t = ((t ^ (t >> 15)) * (t | 1)) & 0xFFFFFFFF
t ^= (t + ((t ^ (t >> 7)) * (t | 61))) & 0xFFFFFFFF
return ((t ^ (t >> 14)) & 0xFFFFFFFF) / 4294967296.0
return nxt
def _accent(case: Optional[Case]) -> str:
if case and case.classification.severity:
return _SEV_ACCENT.get(case.classification.severity, _DEFAULT_ACCENT)
return _DEFAULT_ACCENT
# ---------- hero SVG (featured card) -----------------------------------
def case_hero_svg(case: Case, width: int = 880, height: int = 220) -> str:
"""Wide SVG for the featured-case hero. Particle constellation + severity glow."""
seed = _seed(case.case_id)
rng = _prng(seed)
accent = _accent(case)
sid = f"h{seed:x}"
parts = []
parts.append(
f'<defs>'
f'<linearGradient id="bg-{sid}" x1="0" y1="0" x2="1" y2="1">'
f'<stop offset="0%" stop-color="#0f1115"/><stop offset="100%" stop-color="#1c2230"/>'
f'</linearGradient>'
f'<radialGradient id="glow-{sid}" cx="78%" cy="50%" r="60%">'
f'<stop offset="0%" stop-color="{accent}" stop-opacity="0.42"/>'
f'<stop offset="55%" stop-color="{accent}" stop-opacity="0.08"/>'
f'<stop offset="100%" stop-color="{accent}" stop-opacity="0"/>'
f'</radialGradient>'
f'<radialGradient id="grain-{sid}" cx="20%" cy="35%" r="60%">'
f'<stop offset="0%" stop-color="#1ec8ff" stop-opacity="0.10"/>'
f'<stop offset="100%" stop-color="#1ec8ff" stop-opacity="0"/>'
f'</radialGradient>'
f'</defs>'
)
parts.append(f'<rect width="{width}" height="{height}" fill="url(#bg-{sid})"/>')
parts.append(f'<rect width="{width}" height="{height}" fill="url(#grain-{sid})"/>')
parts.append(f'<rect width="{width}" height="{height}" fill="url(#glow-{sid})"/>')
# Faint scan grid
for x in range(0, width, 44):
parts.append(f'<line x1="{x}" y1="0" x2="{x}" y2="{height}" stroke="{accent}" stroke-opacity="0.04"/>')
for y in range(0, height, 44):
parts.append(f'<line x1="0" y1="{y}" x2="{width}" y2="{y}" stroke="{accent}" stroke-opacity="0.04"/>')
# Particle field — each particle gets a CSS animation-delay so it blinks
# exactly when the sweep column passes over its x-position. Sweep cycle is
# 12s left-to-right; arrival time at x = 12 * (px*100 + 20) / 140 sec.
n = 42
pts = []
for _ in range(n):
x = rng() * width
y = rng() * height
s = 1.2 + rng() * 3
op = 0.3 + rng() * 0.55
# negative delay so the cycle is already at the right phase when the
# page loads (otherwise every particle would flash in unison at t=0).
delay = -round(12.0 * ((x / width) * 100 + 20) / 140, 2)
pts.append((x, y, s, op, delay))
# Connect close particles
for i in range(n):
ax, ay, *_ = pts[i]
for j in range(i + 1, n):
bx, by, *_ = pts[j]
d = math.hypot(bx - ax, by - ay)
if d < 90:
op = (1 - d / 90) * 0.20
parts.append(
f'<line x1="{ax:.1f}" y1="{ay:.1f}" x2="{bx:.1f}" y2="{by:.1f}" '
f'stroke="{accent}" stroke-width="0.6" stroke-opacity="{op:.2f}"/>'
)
for x, y, s, op, d in pts:
style = f'animation-delay: {d}s;'
if rng() < 0.7:
parts.append(
f'<circle class="hero-particle" cx="{x:.1f}" cy="{y:.1f}" r="{s:.1f}" '
f'fill="{accent}" fill-opacity="{op:.2f}" style="{style}"/>'
)
else:
parts.append(
f'<rect class="hero-particle" x="{x - s:.1f}" y="{y - s:.1f}" '
f'width="{s * 2:.1f}" height="{s * 2:.1f}" fill="{accent}" fill-opacity="{op:.2f}" '
f'transform="rotate(45 {x:.1f} {y:.1f})" style="{style}"/>'
)
# HUD corner brackets
for cx, cy, dx, dy in [
(12, 12, 1, 1), (width - 12, 12, -1, 1),
(12, height - 12, 1, -1), (width - 12, height - 12, -1, -1),
]:
parts.append(
f'<polyline points="{cx + dx*18},{cy} {cx},{cy} {cx},{cy + dy*18}" '
f'fill="none" stroke="{accent}" stroke-width="1.6" stroke-opacity="0.55"/>'
)
# Ornamental case id, very faint
parts.append(
f'<text x="{width - 14}" y="{height - 12}" text-anchor="end" '
f'fill="{accent}" fill-opacity="0.22" font-family="ui-monospace,Menlo,monospace" font-size="11">'
f'{case.case_id}</text>'
)
return (
f'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 {width} {height}" '
f'preserveAspectRatio="xMidYMid slice" class="case-hero-svg" aria-hidden="true">'
+ "".join(parts) + "</svg>"
)
# ---------- glyph SVG (news list items) ---------------------------------
def case_glyph_svg(case: Case, size: int = 36) -> str:
"""A small identicon-like SVG glyph for a case — symmetric, severity-colored."""
seed = _seed(case.case_id)
rng = _prng(seed)
accent = _accent(case)
grid = 5
cell = size / grid
parts = [f'<rect width="{size}" height="{size}" fill="#1c2230" rx="7"/>']
pad = 3
# mirror-symmetric pattern
for cy in range(grid):
for cx in range((grid + 1) // 2):
if rng() < 0.55:
op = 0.45 + rng() * 0.45
x = cx * cell + pad / 2
y = cy * cell + pad / 2
w = cell - pad
h = cell - pad
parts.append(f'<rect x="{x:.1f}" y="{y:.1f}" width="{w:.1f}" height="{h:.1f}" '
f'fill="{accent}" fill-opacity="{op:.2f}" rx="1"/>')
# mirror
mx = (grid - 1 - cx) * cell + pad / 2
if cx != grid - 1 - cx:
parts.append(f'<rect x="{mx:.1f}" y="{y:.1f}" width="{w:.1f}" height="{h:.1f}" '
f'fill="{accent}" fill-opacity="{op:.2f}" rx="1"/>')
return (
f'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 {size} {size}" '
f'width="{size}" height="{size}" class="case-glyph-svg" aria-hidden="true">'
+ "".join(parts) + "</svg>"
)

View File

@@ -0,0 +1,144 @@
"""Docker topology — read-only daemon view via socket-proxy.
The cockpit never touches /var/run/docker.sock directly. It talks to a
tecnativa/docker-socket-proxy sidecar over the backend network. The proxy is
configured GET-only (CONTAINERS, NETWORKS, PING) so a web-app compromise
can't drive the daemon. Returned data is normalized for templates: a flat
list of containers and a network-grouped topology.
"""
from __future__ import annotations
import os
from typing import Any, Dict, List, Optional
import httpx
from psyc import log
_log = log.get(__name__)
PROXY_URL = os.environ.get("PSYC_DOCKER_PROXY", "http://docker-socket-proxy:2375")
HTTP_TIMEOUT = 5.0
class DockerProxyError(RuntimeError):
"""Proxy unreachable or returned an error."""
def _get(path: str) -> Any:
try:
with httpx.Client(timeout=HTTP_TIMEOUT) as client:
resp = client.get(f"{PROXY_URL}{path}")
resp.raise_for_status()
return resp.json()
except httpx.HTTPError as exc:
_log.warning("docker_view.proxy.error", path=path, error=str(exc))
raise DockerProxyError(str(exc)) from exc
def list_containers() -> List[Dict[str, Any]]:
raw = _get("/containers/json?all=1")
out: List[Dict[str, Any]] = []
for c in raw:
names = c.get("Names") or []
name = (names[0] if names else "?").lstrip("/")
nets = []
for net_name, net_data in (c.get("NetworkSettings", {}).get("Networks", {}) or {}).items():
nets.append({
"name": net_name,
"ip": net_data.get("IPAddress") or "",
"mac": net_data.get("MacAddress") or "",
"gateway": net_data.get("Gateway") or "",
})
ports = []
published_ports = [] # outer ports only — the ones reachable from the host
for p in c.get("Ports") or []:
inner = p.get("PrivatePort")
outer = p.get("PublicPort")
proto = p.get("Type") or "tcp"
if outer:
ports.append(f"{p.get('IP', '0.0.0.0')}:{outer}->{inner}/{proto}")
published_ports.append(f"{outer}/{proto}")
elif inner:
ports.append(f"{inner}/{proto}")
# Dedupe published_ports while keeping order.
seen: set = set()
published_ports = [x for x in published_ports if not (x in seen or seen.add(x))]
out.append({
"id": (c.get("Id") or "")[:12],
"name": name,
"image": c.get("Image", ""),
"state": c.get("State", ""),
"status": c.get("Status", ""),
"networks": nets,
"ports": ports,
"published_ports": published_ports,
})
out.sort(key=lambda c: (c["state"] != "running", c["name"]))
return out
def list_networks() -> List[Dict[str, Any]]:
raw = _get("/networks")
out: List[Dict[str, Any]] = []
for n in raw:
attached = []
for cid, info in (n.get("Containers") or {}).items():
ip = (info.get("IPv4Address") or "").split("/")[0]
attached.append({
"id": (cid or "")[:12], "name": info.get("Name", ""),
"ip": ip, "mac": info.get("MacAddress") or "",
})
attached.sort(key=lambda x: x["name"])
ipam_cfgs = (n.get("IPAM") or {}).get("Config") or []
subnet = ipam_cfgs[0].get("Subnet") if ipam_cfgs else ""
gateway = ipam_cfgs[0].get("Gateway") if ipam_cfgs else ""
out.append({
"id": (n.get("Id") or "")[:12],
"name": n.get("Name", ""),
"driver": n.get("Driver", ""),
"scope": n.get("Scope", ""),
"internal": bool(n.get("Internal")),
"subnet": subnet or "",
"gateway": gateway or "",
"containers": attached,
})
_DEFAULTS = {"bridge", "host", "none"}
out.sort(key=lambda n: (n["name"] in _DEFAULTS, n["name"]))
return out
def host_info() -> Dict[str, Any]:
"""Daemon-side info for the synthetic host node. Best-effort."""
try:
info = _get("/info")
except DockerProxyError:
return {"name": "docker host", "os": "", "ncpu": None}
return {
"name": info.get("Name") or "docker host",
"os": info.get("OperatingSystem") or info.get("OSType") or "",
"ncpu": info.get("NCPU"),
"containers": info.get("Containers"),
"containers_running": info.get("ContainersRunning"),
}
def topology() -> Dict[str, Any]:
"""Combined snapshot. Either field may be [] with an 'error' key set."""
state: Dict[str, Any] = {
"containers": [], "networks": [], "host": {"name": "docker host"},
"error": None, "proxy": PROXY_URL,
}
try:
state["containers"] = list_containers()
except DockerProxyError as exc:
state["error"] = f"containers: {exc}"
return state
try:
state["networks"] = list_networks()
except DockerProxyError as exc:
state["error"] = f"networks: {exc}"
state["host"] = host_info()
return state

View File

@@ -0,0 +1,530 @@
"""Federation cockpit routes — admin page, public feed/key/info endpoints.
Wired into the FastAPI app by app.py via a single `register(app, TEMPLATES)`
call so the federation surface stays self-contained.
"""
from __future__ import annotations
import json
import time
from typing import Any, Dict, Optional, Tuple
from fastapi import FastAPI, Form, HTTPException, Request
from fastapi.responses import HTMLResponse, JSONResponse, PlainTextResponse, RedirectResponse
from fastapi.templating import Jinja2Templates
from psyc import db, log
from psyc.lines import discovery, federation, network_view, pulse, topology_export, translog
from psyc.result import Err
_log = log.get(__name__)
# Tiny in-memory cache for the signed feed — peers may poll, recomputing
# canonical JSON + signature on every hit would be wasteful.
_FEED_CACHE: Dict[str, Any] = {"ts": 0.0, "payload": None}
_FEED_TTL = 60.0
# Mirror the feed cache for the public peers list — same poll-load pattern.
_PUBLIC_PEERS_CACHE: Dict[str, Any] = {"ts": 0.0, "payload": None}
_PUBLIC_PEERS_TTL = 60.0
# And again for the public federation-network payload (signed JSON view).
_PUBLIC_NETWORK_CACHE: Dict[str, Any] = {"ts": 0.0, "payload": None}
_PUBLIC_NETWORK_TTL = 60.0
# Explore-view cache. The builder fans out to trusted peers' explore feeds
# for the distance-2 snapshot, so a polled hit must NEVER trigger that walk.
_EXPLORE_CACHE: Dict[str, Any] = {"ts": 0.0, "payload": None, "domain": None}
_EXPLORE_TTL = 60.0
# Sanitized docker topology cache. The build call hits the docker-socket-proxy
# sidecar; polled peer admin pages mustn't re-trigger that on every poke.
_TOPOLOGY_CACHE: Dict[str, Any] = {"ts": 0.0, "payload": None}
_TOPOLOGY_TTL = 60.0
# Headers we slap on every public endpoint so other psyc nodes' explore
# pages can fetch them cross-origin from the browser.
_CORS_HEADERS: Dict[str, str] = {
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Methods": "GET, OPTIONS",
"Access-Control-Allow-Headers": "Content-Type",
}
def _admin_ok(request: Request) -> bool:
return bool(request.session.get("admin_ok"))
def _cached_feed() -> Dict[str, Any]:
now = time.time()
if _FEED_CACHE["payload"] is None or (now - _FEED_CACHE["ts"]) > _FEED_TTL:
_FEED_CACHE["payload"] = federation.build_signed_feed()
_FEED_CACHE["ts"] = now
return _FEED_CACHE["payload"]
def _cached_public_peers() -> Any:
now = time.time()
if _PUBLIC_PEERS_CACHE["payload"] is None or (now - _PUBLIC_PEERS_CACHE["ts"]) > _PUBLIC_PEERS_TTL:
_PUBLIC_PEERS_CACHE["payload"] = discovery.public_peer_attestation()
_PUBLIC_PEERS_CACHE["ts"] = now
return _PUBLIC_PEERS_CACHE["payload"]
def _cached_public_network() -> Dict[str, Any]:
now = time.time()
if _PUBLIC_NETWORK_CACHE["payload"] is None or (now - _PUBLIC_NETWORK_CACHE["ts"]) > _PUBLIC_NETWORK_TTL:
_PUBLIC_NETWORK_CACHE["payload"] = network_view.build_public_view()
_PUBLIC_NETWORK_CACHE["ts"] = now
return _PUBLIC_NETWORK_CACHE["payload"]
def _cached_topology() -> Dict[str, Any]:
"""Cached sanitized docker topology — same poll-load pattern as the feed."""
now = time.time()
if _TOPOLOGY_CACHE["payload"] is None or (now - _TOPOLOGY_CACHE["ts"]) > _TOPOLOGY_TTL:
export = topology_export.build_export()
_TOPOLOGY_CACHE["payload"] = export.model_dump(mode="json")
_TOPOLOGY_CACHE["ts"] = now
return _TOPOLOGY_CACHE["payload"]
def _cached_explore(domain: Optional[str]) -> Dict[str, Any]:
"""Cached explore payload. Re-uses the cache when the host domain matches.
Domain is recorded into the payload's `node.domain` field, so a fresh
cache slot per host avoids serving the wrong reflected name.
"""
now = time.time()
cached_domain = _EXPLORE_CACHE.get("domain")
if (
_EXPLORE_CACHE["payload"] is None
or (now - _EXPLORE_CACHE["ts"]) > _EXPLORE_TTL
or cached_domain != domain
):
_EXPLORE_CACHE["payload"] = network_view.build_explore_view(node_domain=domain)
_EXPLORE_CACHE["ts"] = now
_EXPLORE_CACHE["domain"] = domain
return _EXPLORE_CACHE["payload"]
def _public_json(payload: Any) -> JSONResponse:
"""JSONResponse with the public-CORS header set."""
return JSONResponse(payload, headers=_CORS_HEADERS)
def register(app: FastAPI, TEMPLATES: Jinja2Templates) -> None:
"""Mount all federation routes onto `app`."""
@app.get("/admin/federation", response_class=HTMLResponse)
def admin_federation(request: Request) -> HTMLResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
host = request.url.hostname or "your-node.example"
suggested = request.query_params.get("domain", host)
rec = federation.dns_record(suggested)
peers = federation.list_peers()
signals = db.recent_signals(limit=20)
return TEMPLATES.TemplateResponse(
request,
"admin_federation.html",
{
"fingerprint": federation.node_fingerprint(),
"pubkey_pem": federation.public_key_pem(),
"suggested_domain": suggested,
"dns": rec,
"peers": peers,
"signals": signals,
},
)
@app.post("/admin/federation/peers/add")
def admin_federation_add_peer(
request: Request,
domain: str = Form(...),
fingerprint: str = Form(...),
pubkey_pem: str = Form(...),
status: str = Form("unknown"),
) -> RedirectResponse:
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
try:
federation.register_peer(domain.strip(), fingerprint.strip(), pubkey_pem.strip(), status=status)
except Exception as exc:
_log.warning("federation.peer.add.error", domain=domain, error=str(exc))
return RedirectResponse("/admin/federation", status_code=303)
@app.post("/admin/federation/peers/{domain}/status")
def admin_federation_set_status(
request: Request,
domain: str,
status: str = Form(...),
) -> RedirectResponse:
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
try:
federation.set_peer_status(domain, status)
except ValueError as exc:
_log.warning("federation.peer.status.bad", domain=domain, status=status, error=str(exc))
return RedirectResponse("/admin/federation", status_code=303)
@app.post("/admin/federation/peers/{domain}/remove")
def admin_federation_remove(
request: Request,
domain: str,
) -> RedirectResponse:
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
federation.remove_peer(domain)
return RedirectResponse("/admin/federation", status_code=303)
# ---------- discovery (DNS-SD walker) ----------------------------
@app.get("/admin/federation/discovery", response_class=HTMLResponse)
def admin_discovery(request: Request) -> HTMLResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
seeds = pulse.get_discovery_seeds()
candidates = federation.list_peers()
flash = request.query_params.get("flash") or ""
return TEMPLATES.TemplateResponse(
request,
"admin_discovery.html",
{
"seeds": seeds,
"seeds_text": "\n".join(seeds),
"candidates": candidates,
"flash": flash,
},
)
@app.post("/admin/federation/discovery/seeds")
def admin_discovery_seeds(
request: Request,
seeds: str = Form(""),
) -> RedirectResponse:
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
lines = [line for line in seeds.splitlines()]
pulse.set_discovery_seeds(lines)
return RedirectResponse("/admin/federation/discovery?flash=seeds+saved", status_code=303)
@app.post("/admin/federation/discovery/walk")
def admin_discovery_walk(request: Request) -> RedirectResponse:
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
seeds = pulse.get_discovery_seeds()
if not seeds:
return RedirectResponse("/admin/federation/discovery?flash=no+seeds+configured", status_code=303)
try:
cands = discovery.walk(seeds)
for c in cands:
discovery.record_candidate(c)
msg = f"discovered+{len(cands)}+candidates+from+{len(seeds)}+seed(s)"
except Exception as exc: # noqa: BLE001 — surface the error to the operator
_log.warning("federation.discovery.walk.error", error=str(exc))
msg = f"walk+failed:+{str(exc)[:80]}"
return RedirectResponse(f"/admin/federation/discovery?flash={msg}", status_code=303)
# ---------- public endpoints --------------------------------------
@app.get("/federation/info")
def federation_info() -> JSONResponse:
return _public_json({
"fingerprint": federation.node_fingerprint(),
"version": federation.FEED_VERSION,
"feed": federation.FEED_PATH,
"key": "/federation/key",
"explore": "/federation/explore",
})
@app.get("/federation/key", response_class=PlainTextResponse)
def federation_key() -> PlainTextResponse:
return PlainTextResponse(
federation.public_key_pem(),
media_type="text/plain",
headers=_CORS_HEADERS,
)
@app.get("/federation/feed")
def federation_feed() -> JSONResponse:
return _public_json(_cached_feed())
@app.get("/federation/peers/public")
def federation_peers_public() -> JSONResponse:
"""Publicly attested peer list — what other psyc walkers discover us through.
Only trusted peers leak; unknown + blocked are internal state and must
never appear here.
"""
return _public_json(_cached_public_peers())
@app.get("/federation/network")
def federation_network_public() -> JSONResponse:
"""Signed federation-network attestation — for transitive-view fetchers.
Mirrors /federation/peers/public in spirit but adds our outbound vouches
so a fetcher can reconstruct the local web-of-trust shape. Trusted peers
only — never unknown or blocked. Signal hashes are deliberately omitted.
"""
return _public_json(_cached_public_network())
@app.get("/federation/topology")
def federation_topology_public() -> JSONResponse:
"""Sanitized docker topology — public, for peer-side display.
Whitelist-only: container names + images + state + network names. No
env vars, no volume mounts, no IPs/MACs/gateways, no labels. CORS open
so a peer's `/admin/federation/network` page can fetch it from the
browser and render every node's containers alongside its own.
"""
return _public_json(_cached_topology())
# ---------- public vouches + transparency log --------------------
@app.get("/federation/vouches")
def federation_vouches() -> JSONResponse:
"""Vouches WE have issued. Peers fetch this to learn who we trust."""
return _public_json({
"fingerprint": federation.node_fingerprint(),
"vouches": [v.model_dump(mode="json") for v in federation.our_vouches()],
})
@app.get("/federation/log")
def federation_log() -> JSONResponse:
"""Last 100 transparency-log entries, newest first."""
entries = translog.recent(limit=100)
return _public_json({
"count": len(entries),
"entries": [e.model_dump(mode="json") for e in entries],
})
@app.get("/federation/log/verify")
def federation_log_verify() -> JSONResponse:
"""Re-walk the chain locally and report status. Auditors poll this."""
result = translog.verify_chain()
head = translog.head()
head_hash = head.entry_hash if head else None
if isinstance(result, Err):
return JSONResponse(
{"error": result.reason, "head_hash": head_hash},
status_code=409,
headers=_CORS_HEADERS,
)
return _public_json({"verified": result.value, "head_hash": head_hash})
# ---------- public explore page + data ---------------------------
@app.get("/federation/explore", response_class=HTMLResponse)
def federation_explore_page(request: Request) -> HTMLResponse:
"""Public transparency view — anyone can verify this network."""
host = request.url.hostname or ""
peer_param = request.query_params.get("peer", "").strip()
return TEMPLATES.TemplateResponse(
request,
"federation_explore.html",
{
"fingerprint": federation.node_fingerprint(),
"domain": host,
"peer": peer_param,
},
)
@app.get("/federation/explore/data")
def federation_explore_data(request: Request) -> JSONResponse:
"""Signed public explorer payload — peer counts, vouches, transitives.
Public, no auth, CORS-enabled. Cached so a polled hit never triggers
the distance-2 fan-out. See `network_view.build_explore_view` for the
no-leak contract.
"""
host = request.url.hostname or None
payload = _cached_explore(host)
return _public_json(payload)
# ---------- admin: vouches page ---------------------------------
@app.get("/admin/federation/vouches", response_class=HTMLResponse)
def admin_federation_vouches(request: Request) -> HTMLResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
peers = federation.list_peers()
cfg = federation.quorum_config()
ours = federation.our_vouches()
# Per-peer view: vouches naming each peer and whether quorum is met.
peer_rows = []
for p in peers:
vouches = federation.vouches_for(p.fingerprint)
peer_rows.append({
"peer": p,
"vouches": vouches,
"vouched": federation.is_vouched(p.fingerprint),
"eligible": federation.peer_is_listening_eligible(p.fingerprint),
})
return TEMPLATES.TemplateResponse(
request,
"admin_federation_vouches.html",
{
"fingerprint": federation.node_fingerprint(),
"our_vouches": ours,
"peer_rows": peer_rows,
"cfg": cfg,
},
)
@app.post("/admin/federation/vouches/issue")
def admin_federation_vouch_issue(
request: Request,
target_fingerprint: str = Form(...),
ttl_days: int = Form(90),
) -> RedirectResponse:
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
try:
federation.issue_vouch(target_fingerprint.strip(), ttl_days=ttl_days)
except Exception as exc:
_log.warning("federation.vouch.issue.error", error=str(exc))
return RedirectResponse("/admin/federation/vouches", status_code=303)
@app.post("/admin/federation/vouches/revoke")
def admin_federation_vouch_revoke(
request: Request,
target_fingerprint: str = Form(...),
) -> RedirectResponse:
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
federation.revoke_vouch(target_fingerprint.strip())
return RedirectResponse("/admin/federation/vouches", status_code=303)
# ---------- admin: transparency log page ------------------------
@app.get("/admin/federation/log", response_class=HTMLResponse)
def admin_federation_log(request: Request) -> HTMLResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
result = translog.verify_chain()
head = translog.head()
entries = translog.recent(limit=200)
verify_status: Dict[str, Any]
if isinstance(result, Err):
verify_status = {"ok": False, "reason": result.reason}
else:
verify_status = {"ok": True, "verified": result.value}
return TEMPLATES.TemplateResponse(
request,
"admin_federation_log.html",
{
"verify_status": verify_status,
"head_hash": head.entry_hash if head else "",
"head_id": head.id if head else 0,
"entries": entries,
},
)
# ---------- admin: federation network view ----------------------
@app.get("/admin/federation/network", response_class=HTMLResponse)
def admin_federation_network(request: Request) -> HTMLResponse:
"""Cockpit page — force-directed federation map. Data lives at /data."""
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
# Build local stats up front so the header card renders even if the
# JS data-endpoint fetch fails (defensive — never give the operator a
# blank page).
view = network_view.build_local_view()
return TEMPLATES.TemplateResponse(
request,
"admin_federation_network.html",
{
"fingerprint": federation.node_fingerprint(),
"stats": view.stats,
},
)
@app.get("/admin/federation/network/data")
def admin_federation_network_data(request: Request) -> JSONResponse:
"""Full admin view — includes unknown/blocked peers + transitive peers.
Public /federation/network filters those out; this surface does not,
because it sits behind the admin gate and the operator needs to see
the real shape of the federation including the parts being ignored.
"""
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
return JSONResponse(network_view.build_admin_view(include_transitive=True))
# ---------- admin: quorum config + per-peer/per-hash view -------
@app.get("/admin/federation/quorum", response_class=HTMLResponse)
def admin_federation_quorum(request: Request) -> HTMLResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
cfg = federation.quorum_config()
peers = federation.list_peers()
peer_rows = [
{
"peer": p,
"vouched": federation.is_vouched(p.fingerprint),
"eligible": federation.peer_is_listening_eligible(p.fingerprint),
}
for p in peers
]
# Group buffered signals by signal_hash and count distinct eligible peers.
signal_rows: Dict[str, Dict[str, Any]] = {}
for s in db.recent_signals(limit=500):
h = s.get("signal_hash") or ""
entry = signal_rows.setdefault(h, {
"signal_hash": h,
"signal_type": s.get("signal_type") or "",
"signal_id": s.get("signal_id") or "",
"peers": set(),
"latest": s.get("received_at") or "",
})
entry["peers"].add(s.get("peer_fingerprint") or "")
hash_summary = []
for h, row in signal_rows.items():
distinct_eligible = sum(
1 for fp in row["peers"] if federation.peer_is_listening_eligible(fp)
)
hash_summary.append({
"signal_hash": h,
"signal_type": row["signal_type"],
"signal_id": row["signal_id"],
"distinct_peers": len(row["peers"]),
"distinct_eligible": distinct_eligible,
"quorum_met": distinct_eligible >= cfg.signal_quorum_k,
"latest": row["latest"],
})
hash_summary.sort(key=lambda r: r["latest"], reverse=True)
return TEMPLATES.TemplateResponse(
request,
"admin_federation_quorum.html",
{
"cfg": cfg,
"peer_rows": peer_rows,
"hash_summary": hash_summary,
},
)
@app.post("/admin/federation/quorum/save")
def admin_federation_quorum_save(
request: Request,
trust_min_vouchers: int = Form(...),
signal_quorum_k: int = Form(...),
) -> RedirectResponse:
if not _admin_ok(request):
raise HTTPException(status_code=403, detail="admin session required")
try:
cfg = federation.QuorumConfig(
trust_min_vouchers=max(1, int(trust_min_vouchers)),
signal_quorum_k=max(1, int(signal_quorum_k)),
)
federation.set_quorum_config(cfg)
except Exception as exc:
_log.warning("federation.quorum.save.error", error=str(exc))
return RedirectResponse("/admin/federation/quorum", status_code=303)
_log.info("federation.routes.registered")

View File

@@ -3,13 +3,19 @@
The cockpit venv has no torch; the fine-tuned model only runs inside the CUDA
container behind serve_model.py. This client reaches it over HTTP and degrades
gracefully — if the server is down, callers get None and fall back to rules.
Two backends are supported via PSYC_INFERENCE_MODE:
- "psyc" (default) — native serve_model.py, POST /infer
- "openai" — OpenAI-compatible / Ollama, POST /v1/chat/completions
A bearer token can be set via PSYC_INFERENCE_TOKEN; it is sent on every request
when present (psyc-native ignores it; api.neuronetz.ai requires it).
"""
from __future__ import annotations
import json
import os
from typing import Optional
from typing import Dict, Optional
import httpx
@@ -18,15 +24,31 @@ from psyc.lines.train import SEVERITY_INSTRUCTION, severity_features
from psyc.models import Case
INFERENCE_URL = os.environ.get("PSYC_INFERENCE_URL", "http://127.0.0.1:8771")
INFERENCE_URL = os.environ.get("PSYC_INFERENCE_URL", "http://127.0.0.1:8771")
INFERENCE_TOKEN = os.environ.get("PSYC_INFERENCE_TOKEN", "")
INFERENCE_MODE = os.environ.get("PSYC_INFERENCE_MODE", "psyc").lower()
INFERENCE_MODEL = os.environ.get("PSYC_INFERENCE_MODEL", "psyc-v5")
_log = log.get(__name__)
def _auth_headers() -> Dict[str, str]:
"""Bearer header when a token is set, empty dict otherwise."""
return {"Authorization": f"Bearer {INFERENCE_TOKEN}"} if INFERENCE_TOKEN else {}
def server_adapter(timeout: float = 2.0) -> Optional[str]:
"""Return the adapter the server is running, or None if it is unreachable."""
try:
with httpx.Client(timeout=timeout) as client:
if INFERENCE_MODE == "openai":
# OpenAI/Ollama exposes GET /v1/models — first available id wins.
resp = client.get(f"{INFERENCE_URL}/v1/models", headers=_auth_headers())
resp.raise_for_status()
data = resp.json().get("data") or []
if data:
return str(data[0].get("id") or INFERENCE_MODEL)
return INFERENCE_MODEL
resp = client.get(f"{INFERENCE_URL}/healthz")
resp.raise_for_status()
return resp.json().get("adapter")
@@ -34,20 +56,55 @@ def server_adapter(timeout: float = 2.0) -> Optional[str]:
return None
def adapter_name(timeout: float = 2.0) -> Optional[str]:
"""Short name of the live adapter, e.g. 'psyc-v5' from '/data/adapters/psyc-v5/final'."""
path = server_adapter(timeout=timeout)
if not path:
return None
parts = [p for p in path.split("/") if p and p != "final"]
return parts[-1] if parts else None
def model_severity(case: Case, timeout: float = 15.0) -> Optional[str]:
"""Ask the live model to classify case severity. None if the server is down."""
payload = {
"instruction": SEVERITY_INSTRUCTION,
"input": json.dumps(severity_features(case), ensure_ascii=False),
"max_new_tokens": 16,
}
features_json = json.dumps(severity_features(case), ensure_ascii=False)
try:
with httpx.Client(timeout=timeout) as client:
resp = client.post(f"{INFERENCE_URL}/infer", json=payload)
resp.raise_for_status()
output = str(resp.json().get("output", "")).strip().lower()
if INFERENCE_MODE == "openai":
payload = {
"model": INFERENCE_MODEL,
"messages": [
{"role": "system", "content": SEVERITY_INSTRUCTION},
{"role": "user", "content": features_json},
],
"max_tokens": 16,
"temperature": 0.0,
}
resp = client.post(
f"{INFERENCE_URL}/v1/chat/completions",
json=payload,
headers=_auth_headers(),
)
resp.raise_for_status()
choices = resp.json().get("choices") or []
if not choices:
return None
output = str(choices[0].get("message", {}).get("content", "")).strip().lower()
else:
payload = {
"instruction": SEVERITY_INSTRUCTION,
"input": features_json,
"max_new_tokens": 16,
}
resp = client.post(
f"{INFERENCE_URL}/infer",
json=payload,
headers=_auth_headers(),
)
resp.raise_for_status()
output = str(resp.json().get("output", "")).strip().lower()
except httpx.HTTPError as exc:
_log.info("inference.unavailable", error=str(exc))
return None
_log.info("inference.severity", case_id=case.case_id, model_answer=output)
_log.info("inference.severity", case_id=case.case_id, model_answer=output, mode=INFERENCE_MODE)
return output

View File

@@ -0,0 +1,155 @@
"""Cockpit routes for the Pulse scheduler — admin-gated.
The integration is intentionally single-call: `register(app, TEMPLATES)` adds
the routes AND wires the FastAPI startup hook that launches the background
scheduler loop. Caller in app.py just imports + invokes register().
"""
from __future__ import annotations
import asyncio
from datetime import datetime, timedelta, timezone
from typing import List, Optional
from fastapi import FastAPI, Form, Request
from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates
from psyc import db, log
from psyc.lines import pulse
from psyc.models import Severity
_log = log.get(__name__)
TICK_INTERVAL_SECONDS = 30
def _admin_ok(request: Request) -> bool:
"""Mirror of the local helper in app.py — admin session is just session['admin_ok']."""
return bool(request.session.get("admin_ok"))
def _relative(dt: Optional[datetime]) -> str:
"""Human-friendly "3m ago" / "in 12m" / "now". None → ''."""
if dt is None:
return ""
now = datetime.now(timezone.utc)
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
delta = (dt - now).total_seconds()
past = delta < 0
secs = abs(int(delta))
if secs < 5:
return "now"
if secs < 60:
unit = f"{secs}s"
elif secs < 3600:
unit = f"{secs // 60}m"
elif secs < 86400:
unit = f"{secs // 3600}h"
else:
unit = f"{secs // 86400}d"
return f"{unit} ago" if past else f"in {unit}"
def register(app: FastAPI, templates: Jinja2Templates) -> None:
"""Attach the /admin/pulse routes and the background scheduler loop to `app`."""
@app.get("/admin/pulse", response_class=HTMLResponse)
def pulse_view(request: Request) -> HTMLResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
flash = request.query_params.get("flash", "")
pipelines = pulse.state()
respond_mode = next((p.mode.value for p in pipelines if p.name == "respond"), "manual")
since = (datetime.now(timezone.utc) - timedelta(hours=24)).isoformat()
auto_fired_24h = db.pulse_audit_count_since("respond", "auto-fire", since)
audit_recent = db.pulse_audit_recent("respond", limit=5)
return templates.TemplateResponse(
request,
"admin_pulse.html",
{
"pipelines": pipelines,
"kill_switch": pulse.kill_switch_state(),
"tick_interval": TICK_INTERVAL_SECONDS,
"relative": _relative,
"flash": flash,
"respond_mode": respond_mode,
"respond_threshold": pulse.respond_auto_threshold().value,
"respond_require_quorum": pulse.respond_require_quorum(),
"respond_local_only": pulse.respond_local_only(),
"respond_auto_fired_24h": auto_fired_24h,
"respond_audit_recent": audit_recent,
"severity_choices": [s.value for s in Severity],
},
)
@app.post("/admin/pulse/kill")
def pulse_toggle_kill(request: Request) -> RedirectResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
new = not pulse.kill_switch_state()
pulse.set_kill_switch(new)
flash = "kill switch ARMED — all pipelines halted" if new else "kill switch disarmed — pulse resumes"
return RedirectResponse(f"/admin/pulse?flash={flash}", status_code=303)
@app.post("/admin/pulse/{name}/update")
def pulse_update(
request: Request,
name: str,
mode: str = Form(...),
cadence_seconds: int = Form(...),
enabled: Optional[str] = Form(None),
) -> RedirectResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
try:
pulse.set_mode(name, pulse.PulseMode(mode))
pulse.set_cadence(name, int(cadence_seconds))
pulse.set_enabled(name, enabled is not None)
flash = f"updated {name}: mode={mode}, cadence={cadence_seconds}s, enabled={enabled is not None}"
except (ValueError, KeyError) as exc:
_log.warning("pulse.update.error", name=name, error=str(exc))
flash = f"update failed: {exc}"
return RedirectResponse(f"/admin/pulse?flash={flash}", status_code=303)
@app.post("/admin/pulse/{name}/run")
def pulse_run_now(request: Request, name: str) -> RedirectResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
try:
outcome, result = pulse.run_now(name)
flash = f"{name}{outcome}: {result[:120]}"
except ValueError as exc:
flash = f"run failed: {exc}"
return RedirectResponse(f"/admin/pulse?flash={flash}", status_code=303)
@app.post("/admin/pulse/respond-config")
def pulse_respond_config(
request: Request,
threshold: str = Form(...),
require_quorum: Optional[str] = Form(None),
local_only: Optional[str] = Form(None),
) -> RedirectResponse:
if not _admin_ok(request):
return RedirectResponse("/admin", status_code=303)
try:
sev = Severity(threshold)
pulse.set_respond_auto_threshold(sev)
pulse.set_respond_require_quorum(require_quorum is not None)
pulse.set_respond_local_only(local_only is not None)
flash = (
f"respond gates updated: threshold={sev.value}, "
f"quorum={'on' if require_quorum is not None else 'off'}, "
f"local-only={'on' if local_only is not None else 'off'}"
)
except ValueError as exc:
flash = f"respond-config failed: {exc}"
return RedirectResponse(f"/admin/pulse?flash={flash}", status_code=303)
@app.on_event("startup")
async def _start_pulse_loop() -> None:
# Fire-and-forget; the loop catches its own exceptions and self-restarts.
asyncio.create_task(pulse.start_background_loop(interval_seconds=TICK_INTERVAL_SECONDS))
_log.info("pulse.routes.registered", tick=TICK_INTERVAL_SECONDS)

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,780 @@
/* psyc — federation explorer (public, transparency view).
*
* Forked from federation_network.js, adapted for the public surface:
* • data source is /federation/explore/data (signed, CORS-enabled)
* • clicking a peer opens a walk-to-peer card with a primary CTA
* that full-page-navigates to that peer's own /federation/explore
* • the transparency log can be re-verified live from the page
* • inbound vouches (who vouches for THIS node) get their own section
* • severity/IOC-type breakdowns are intentionally NOT surfaced —
* those stay admin-only to avoid sector-leaking via the public page
*
* Pure vanilla JS, no deps. Mirrors topology.js's proven force-sim loop.
*/
(function () {
"use strict";
const svg = document.getElementById("federation-network-graph");
const loadingEl = document.getElementById("fn-loading");
const errorEl = document.getElementById("fn-error");
const tooltipEl = document.getElementById("fn-tooltip");
const walkEl = document.getElementById("fe-walk");
const directCountEl = document.getElementById("fe-direct-count");
const transitiveCountEl = document.getElementById("fe-transitive-count");
const kpiPeers = document.getElementById("fe-kpi-peers");
const kpiVouchesOut = document.getElementById("fe-kpi-vouches-out");
const kpiVouchesIn = document.getElementById("fe-kpi-vouches-in");
const kpiSignals = document.getElementById("fe-kpi-signals");
const kpiCorroboration = document.getElementById("fe-kpi-corroboration");
const kpiTranslog = document.getElementById("fe-kpi-translog");
const kpiVerify = document.getElementById("fe-kpi-verify");
const verifyBtn = document.getElementById("fe-verify-btn");
const verifyResult = document.getElementById("fe-verify-result");
const vouchesInList = document.getElementById("fe-vouches-in-list");
const vouchesInCountEl = document.getElementById("fe-vouches-in-count");
const settings = window.PSYC_EXPLORE || {};
if (!svg) return;
// ---------- shared escape -----------------------------------------------
function esc(s) {
return String(s == null ? "" : s).replace(/[&<>"']/g, c =>
({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" }[c]));
}
function shortFp(fp) {
if (!fp) return "—";
if (fp.length >= 16) return fp.slice(0, 8) + "…" + fp.slice(-8);
return fp;
}
function fmtAge(iso) {
if (!iso) return "—";
const ts = new Date(iso);
if (isNaN(ts.getTime())) return "—";
const secs = Math.floor((Date.now() - ts.getTime()) / 1000);
if (secs < 0) return "just now";
if (secs < 60) return secs + "s ago";
if (secs < 3600) return Math.floor(secs / 60) + "m ago";
if (secs < 86400) return Math.floor(secs / 3600) + "h ago";
return Math.floor(secs / 86400) + "d ago";
}
fetch("/federation/explore/data", { credentials: "omit" })
.then(r => {
if (!r.ok) throw new Error("HTTP " + r.status);
return r.json();
})
.then(data => {
if (loadingEl) loadingEl.style.display = "none";
render(data);
})
.catch(err => {
if (loadingEl) loadingEl.style.display = "none";
if (errorEl) {
errorEl.style.display = "block";
errorEl.textContent = "✗ failed to load explore payload: " + err.message;
}
});
// ---------- verify button — fetch /federation/log/verify ---------------
if (verifyBtn) {
verifyBtn.addEventListener("click", () => {
verifyBtn.disabled = true;
verifyResult.textContent = "verifying…";
verifyResult.classList.remove("fe-verify-ok", "fe-verify-bad");
fetch("/federation/log/verify", { credentials: "omit" })
.then(r => r.json().then(b => ({ status: r.status, body: b })))
.then(({ status, body }) => {
if (status === 200 && body.verified != null) {
verifyResult.textContent = "✓ verified " + body.verified + " entries · head " + (body.head_hash || "").slice(0, 12) + "…";
verifyResult.classList.add("fe-verify-ok");
if (kpiVerify) {
kpiVerify.textContent = "✓ ok";
kpiVerify.classList.add("fe-verify-ok");
kpiVerify.classList.remove("fe-verify-bad");
}
} else {
verifyResult.textContent = "✗ " + (body.error || "chain invalid");
verifyResult.classList.add("fe-verify-bad");
if (kpiVerify) {
kpiVerify.textContent = "✗ broken";
kpiVerify.classList.add("fe-verify-bad");
kpiVerify.classList.remove("fe-verify-ok");
}
}
})
.catch(err => {
verifyResult.textContent = "✗ fetch failed: " + err.message;
verifyResult.classList.add("fe-verify-bad");
})
.finally(() => { verifyBtn.disabled = false; });
});
}
function render(data) {
const node = data.node || {};
const selfFp = data.fingerprint || node.fingerprint || "";
const peersData = data.peers || [];
const transitiveData = data.transitive_peers || [];
const vouchesIn = data.vouches_in || [];
const vouchesOut = data.vouches_out || data.vouches || [];
// ---------- KPI strip ------------------------------------------------
if (kpiPeers) kpiPeers.textContent = String(node.peer_count != null ? node.peer_count : peersData.length);
if (kpiVouchesOut) kpiVouchesOut.textContent = String(node.vouches_out_count != null ? node.vouches_out_count : vouchesOut.length);
if (kpiVouchesIn) kpiVouchesIn.textContent = String(node.vouches_in_count != null ? node.vouches_in_count : vouchesIn.length);
if (kpiSignals) kpiSignals.textContent = String(node.signals_count_24h || 0);
if (kpiCorroboration) kpiCorroboration.textContent = String(node.corroboration_count_24h || 0);
if (kpiTranslog) kpiTranslog.textContent = String(node.translog_entry_count || 0);
if (kpiVerify) kpiVerify.textContent = "unverified";
if (directCountEl) directCountEl.textContent = String(peersData.length);
if (transitiveCountEl) transitiveCountEl.textContent = String(transitiveData.length);
// ---------- node + edge model ----------------------------------------
// The explore payload doesn't ship edges directly; we derive them from
// the vouches + per-peer signal counts so the graph reads the same way
// the admin view does.
const peerByFp = Object.create(null);
const nodes = [];
// Self at the center.
const selfNode = {
id: selfFp, fp: selfFp,
domain: settings.selfDomain || node.domain || "",
label: settings.selfDomain || (node.domain || "self"),
status: "self",
is_self: true,
distance: 0,
stats: null,
r: 38,
intensity: 1,
x: 0, y: 0, vx: 0, vy: 0, fixed: false,
};
nodes.push(selfNode);
peerByFp[selfFp] = selfNode;
// Max signal count for log-intensity normalization.
let maxSig = 0;
for (const p of peersData) {
if ((p.signal_count_24h || 0) > maxSig) maxSig = p.signal_count_24h || 0;
}
for (const p of peersData) {
const fp = p.fingerprint;
if (!fp || fp === selfFp) continue;
const sig = p.signal_count_24h || 0;
let intensity = 1;
if (maxSig > 0) {
const num = Math.log2(sig + 1);
const den = Math.log2(maxSig + 1) || 1;
intensity = 0.20 + 0.80 * (num / den);
}
const n = {
id: fp, fp,
domain: p.domain || "",
label: p.domain || shortFp(fp),
status: "trusted",
is_self: false,
distance: 1,
stats: p,
r: 16,
intensity,
x: 0, y: 0, vx: 0, vy: 0, fixed: false,
};
nodes.push(n);
peerByFp[fp] = n;
}
for (const t of transitiveData) {
const fp = t.fingerprint;
if (!fp || peerByFp[fp]) continue;
const n = {
id: fp, fp,
domain: t.domain || "",
label: t.domain || shortFp(fp),
status: "unknown",
is_self: false,
distance: 2,
stats: null,
via: t.via_peer_fingerprint || "",
r: 9,
intensity: 0.7,
x: 0, y: 0, vx: 0, vy: 0, fixed: false,
};
nodes.push(n);
peerByFp[fp] = n;
}
// Edges. Per-peer signal counts → signal edges; outbound vouches →
// vouch edges; vouches_in → bidirectional vouch edges; transitive
// "via" → knows edges.
const edges = [];
for (const p of peersData) {
const fp = p.fingerprint;
if (!fp || fp === selfFp) continue;
if ((p.signal_count_24h || 0) > 0) {
edges.push({
source: fp, target: selfFp, kind: "signal",
weight: p.signal_count_24h, label: p.signal_count_24h + " signals/24h",
bidirectional: false,
});
}
}
// Outbound vouches.
const outbound = new Set();
for (const v of vouchesOut) {
const tgt = v.target_fingerprint;
if (!tgt || !peerByFp[tgt]) continue;
outbound.add(tgt);
edges.push({
source: selfFp, target: tgt, kind: "vouch",
weight: 1, label: "vouched", bidirectional: false,
});
}
// Inbound vouches — collapse onto existing outbound where possible.
for (const v of vouchesIn) {
const src = v.voucher_fingerprint;
if (!src || !peerByFp[src]) continue;
if (outbound.has(src)) {
const existing = edges.find(e => e.kind === "vouch"
&& e.source === selfFp && e.target === src);
if (existing) {
existing.bidirectional = true;
existing.label = "vouched ↔";
continue;
}
}
edges.push({
source: src, target: selfFp, kind: "vouch",
weight: 1, label: "vouches us", bidirectional: false,
});
}
// Transitive "knows" edges.
for (const t of transitiveData) {
const parent = t.via_peer_fingerprint;
if (!parent || !peerByFp[parent] || !peerByFp[t.fingerprint]) continue;
edges.push({
source: parent, target: t.fingerprint, kind: "knows",
weight: 0.5, label: "knows", bidirectional: false,
});
}
// ---------- viewport + seeding ---------------------------------------
function viewport() {
const W = svg.clientWidth || 900;
const H = parseInt(getComputedStyle(svg).height, 10) || 560;
return { W, H };
}
let { W, H } = viewport();
svg.setAttribute("viewBox", `0 0 ${W} ${H}`);
(function seed() {
const cx = W / 2, cy = H / 2;
nodes.forEach((n, i) => {
if (n.is_self) { n.x = cx; n.y = cy; return; }
const ring = n.distance >= 2 ? Math.min(W, H) * 0.40 : Math.min(W, H) * 0.22;
const ang = (i / Math.max(1, nodes.length)) * Math.PI * 2;
n.x = cx + ring * Math.cos(ang) + (Math.random() - 0.5) * 20;
n.y = cy + ring * Math.sin(ang) + (Math.random() - 0.5) * 20;
});
})();
const REPULSION = 1500;
const SPRING_K = 0.035;
const SPRING_REST_BASE = 110;
const DAMP = 0.82;
const CENTER_PULL = 0.005;
function tick() {
for (let i = 0; i < nodes.length; i++) {
const a = nodes[i];
for (let j = i + 1; j < nodes.length; j++) {
const b = nodes[j];
const dx = b.x - a.x, dy = b.y - a.y;
const d2 = dx * dx + dy * dy + 0.1;
const d = Math.sqrt(d2);
const f = REPULSION / d2;
const fx = (dx / d) * f, fy = (dy / d) * f;
if (!a.fixed) { a.vx -= fx; a.vy -= fy; }
if (!b.fixed) { b.vx += fx; b.vy += fy; }
}
}
for (const e of edges) {
const a = peerByFp[e.source], b = peerByFp[e.target];
if (!a || !b) continue;
const rest = e.kind === "knows" ? SPRING_REST_BASE + 60 : SPRING_REST_BASE;
const dx = b.x - a.x, dy = b.y - a.y;
const d = Math.sqrt(dx * dx + dy * dy) + 0.1;
const f = (d - rest) * SPRING_K;
const fx = (dx / d) * f, fy = (dy / d) * f;
if (!a.fixed) { a.vx += fx; a.vy += fy; }
if (!b.fixed) { b.vx -= fx; b.vy -= fy; }
}
for (const n of nodes) {
if (n.fixed) { n.vx = 0; n.vy = 0; continue; }
n.vx += (W / 2 - n.x) * CENTER_PULL;
n.vy += (H / 2 - n.y) * CENTER_PULL;
n.vx *= DAMP; n.vy *= DAMP;
n.x += n.vx; n.y += n.vy;
n.x = Math.max(n.r, Math.min(W - n.r, n.x));
n.y = Math.max(n.r, Math.min(H - n.r, n.y));
}
}
for (let i = 0; i < 280; i++) tick();
// ---------- render SVG groups ----------------------------------------
const ns = "http://www.w3.org/2000/svg";
const edgesG = document.createElementNS(ns, "g");
const nodesG = document.createElementNS(ns, "g");
edgesG.setAttribute("class", "fn-edges");
nodesG.setAttribute("class", "fn-nodes");
svg.appendChild(edgesG);
svg.appendChild(nodesG);
const edgeEls = edges.map(e => {
const grp = document.createElementNS(ns, "g");
grp.setAttribute("class", "fn-edge-grp fn-kind-" + e.kind);
grp.dataset.source = e.source;
grp.dataset.target = e.target;
const ln = document.createElementNS(ns, "line");
ln.setAttribute("class", "fn-edge");
if (e.kind === "signal") {
const w = Math.min(5, 1 + Math.log2(e.weight + 1) * 0.8);
ln.setAttribute("stroke-width", w.toFixed(2));
}
grp.appendChild(ln);
if (e.label) {
const lbl = document.createElementNS(ns, "text");
lbl.setAttribute("class", "fn-edge-label");
lbl.textContent = e.label;
grp.appendChild(lbl);
}
edgesG.appendChild(grp);
return { line: ln, label: grp.querySelector("text"), grp };
});
function _classFor(n) {
if (n.is_self) return "fn-node fn-self";
const dist = n.distance >= 2 ? " fn-distance-2" : " fn-distance-1";
return "fn-node fn-status-" + n.status + dist;
}
const nodeEls = nodes.map(n => {
const g = document.createElementNS(ns, "g");
g.setAttribute("class", _classFor(n));
g.dataset.fp = n.fp;
let shape;
if (n.is_self) {
const sz = n.r;
shape = document.createElementNS(ns, "rect");
shape.setAttribute("x", -sz); shape.setAttribute("y", -sz);
shape.setAttribute("width", sz * 2); shape.setAttribute("height", sz * 2);
shape.setAttribute("rx", 10); shape.setAttribute("ry", 10);
g.appendChild(shape);
} else {
shape = document.createElementNS(ns, "circle");
shape.setAttribute("r", n.r);
shape.setAttribute("fill-opacity", n.intensity.toFixed(2));
g.appendChild(shape);
}
const text = document.createElementNS(ns, "text");
text.setAttribute("class", "fn-label");
text.setAttribute("dy", n.r + 13);
text.textContent = n.label;
g.appendChild(text);
if (!n.is_self) {
const sub = document.createElementNS(ns, "text");
sub.setAttribute("class", "fn-sublabel");
sub.setAttribute("dy", n.r + 24);
sub.textContent = n.fp.slice(0, 8) + "…";
g.appendChild(sub);
if (n.stats) {
const badge = document.createElementNS(ns, "text");
badge.setAttribute("class", "fn-stat-badge");
badge.setAttribute("dy", n.r + 36);
badge.textContent =
"↓ " + (n.stats.signal_count_24h || 0) +
" · ⚡ " + (n.stats.quorum_contribution_24h || 0);
g.appendChild(badge);
}
}
const title = document.createElementNS(ns, "title");
title.textContent = (n.is_self ? "self · " : "") + (n.domain || n.label || n.fp);
g.appendChild(title);
nodesG.appendChild(g);
return g;
});
function paint() {
for (let i = 0; i < edges.length; i++) {
const e = edges[i];
const a = peerByFp[e.source], b = peerByFp[e.target];
if (!a || !b) continue;
const els = edgeEls[i];
els.line.setAttribute("x1", a.x); els.line.setAttribute("y1", a.y);
els.line.setAttribute("x2", b.x); els.line.setAttribute("y2", b.y);
if (els.label) {
const mx = (a.x + b.x) / 2, my = (a.y + b.y) / 2;
els.label.setAttribute("x", mx); els.label.setAttribute("y", my - 3);
}
}
for (let i = 0; i < nodes.length; i++) {
nodeEls[i].setAttribute("transform", `translate(${nodes[i].x},${nodes[i].y})`);
}
}
paint();
// ---------- tooltip --------------------------------------------------
function showTooltip(n, clientX, clientY) {
if (!tooltipEl) return;
const rows = [];
rows.push(`<div class="fn-tooltip-title">${esc(n.domain || n.label || n.fp.slice(0,12))}</div>`);
if (n.is_self) {
rows.push(`<div class="fn-tooltip-row"><span class="k">role</span><span class="v">self</span></div>`);
rows.push(`<div class="fn-tooltip-row"><span class="k">peers</span><span class="v">${node.peer_count || 0}</span></div>`);
rows.push(`<div class="fn-tooltip-row"><span class="k">signals 24h</span><span class="v">${node.signals_count_24h || 0}</span></div>`);
} else if (n.distance >= 2) {
rows.push(`<div class="fn-tooltip-row"><span class="k">distance</span><span class="v">2 hops (transitive)</span></div>`);
if (n.via) {
const parent = peerByFp[n.via];
const via = parent ? (parent.domain || shortFp(parent.fp)) : shortFp(n.via);
rows.push(`<div class="fn-tooltip-row"><span class="k">via</span><span class="v">${esc(via)}</span></div>`);
}
rows.push(`<div class="fn-tooltip-row"><span class="k">tip</span><span class="v">click to walk →</span></div>`);
} else {
const s = n.stats || {};
rows.push(`<div class="fn-tooltip-row"><span class="k">status</span><span class="v">trusted</span></div>`);
rows.push(`<div class="fn-tooltip-row"><span class="k">signals 24h</span><span class="v">${s.signal_count_24h || 0}</span></div>`);
rows.push(`<div class="fn-tooltip-row"><span class="k">quorum hits</span><span class="v">${s.quorum_contribution_24h || 0}</span></div>`);
rows.push(`<div class="fn-tooltip-row"><span class="k">last seen</span><span class="v">${esc(fmtAge(s.last_seen))}</span></div>`);
rows.push(`<div class="fn-tooltip-row"><span class="k">tip</span><span class="v">click to walk →</span></div>`);
}
tooltipEl.innerHTML = rows.join("");
tooltipEl.classList.add("is-visible");
positionTooltip(clientX, clientY);
}
function positionTooltip(clientX, clientY) {
if (!tooltipEl) return;
const parent = svg.parentElement;
if (!parent) return;
const rect = parent.getBoundingClientRect();
let x = clientX - rect.left + 14;
let y = clientY - rect.top + 14;
const tw = tooltipEl.offsetWidth || 240;
const th = tooltipEl.offsetHeight || 100;
if (x + tw > parent.clientWidth - 4) x = clientX - rect.left - tw - 14;
if (y + th > parent.clientHeight - 4) y = clientY - rect.top - th - 14;
tooltipEl.style.left = x + "px";
tooltipEl.style.top = y + "px";
}
function hideTooltip() {
if (tooltipEl) tooltipEl.classList.remove("is-visible");
}
// ---------- drag + click + hover ------------------------------------
let dragging = null, dragOffset = { x: 0, y: 0 };
let pressedNode = null, pressedAt = null, moved = false;
let energyBudget = 40;
function svgPoint(clientX, clientY) {
const pt = svg.createSVGPoint(); pt.x = clientX; pt.y = clientY;
return pt.matrixTransform(svg.getScreenCTM().inverse());
}
nodeEls.forEach((g, i) => {
const n = nodes[i];
g.addEventListener("mousedown", ev => {
ev.preventDefault();
pressedNode = n;
pressedAt = { x: ev.clientX, y: ev.clientY };
moved = false;
dragging = n;
const p = svgPoint(ev.clientX, ev.clientY);
dragOffset.x = p.x - dragging.x; dragOffset.y = p.y - dragging.y;
if (currentLayout === "force") dragging.fixed = true;
g.classList.add("dragging");
});
g.addEventListener("mouseenter", ev => showTooltip(n, ev.clientX, ev.clientY));
g.addEventListener("mousemove", ev => positionTooltip(ev.clientX, ev.clientY));
g.addEventListener("mouseleave", hideTooltip);
});
document.addEventListener("mousemove", ev => {
if (pressedAt) {
const dx = ev.clientX - pressedAt.x, dy = ev.clientY - pressedAt.y;
if (dx * dx + dy * dy > 16) moved = true;
}
if (!dragging) return;
const p = svgPoint(ev.clientX, ev.clientY);
dragging.x = p.x - dragOffset.x; dragging.y = p.y - dragOffset.y;
dragging.vx = 0; dragging.vy = 0;
energyBudget = 80;
});
document.addEventListener("mouseup", () => {
if (dragging) {
const g = nodesG.querySelector(`[data-fp="${CSS.escape(dragging.fp)}"]`);
if (g) g.classList.remove("dragging");
if (currentLayout === "force") dragging.fixed = false;
dragging = null;
}
if (pressedNode && !moved) selectNode(pressedNode);
pressedNode = null; pressedAt = null;
});
// ---------- walk-to-peer card ---------------------------------------
function selectNode(n) {
nodeEls.forEach(el => el.classList.remove("selected"));
const me = nodesG.querySelector(`[data-fp="${CSS.escape(n.fp)}"]`);
if (me) me.classList.add("selected");
renderWalk(n);
}
function jumpToFp(fp) {
const t = peerByFp[fp];
if (!t) return;
selectNode(t);
// Scroll the graph stage into view so the user sees the highlight.
svg.scrollIntoView({ behavior: "smooth", block: "center" });
}
function vouchersFor(fp) {
// Inbound vouches naming `fp`. Right now we only have inbound vouches
// for SELF in the public payload; for any other peer we don't see
// who-vouches-for-them from this page.
if (fp === selfFp) return vouchesIn.map(v => v.voucher_fingerprint);
return [];
}
function renderWalk(n) {
if (!walkEl) return;
const isSelf = n.is_self;
const isTransitive = n.distance >= 2;
const stats = n.stats || {};
const targetDomain = n.domain || (isSelf ? settings.selfDomain : "");
const peerHref = targetDomain
? `https://${targetDomain}/federation/explore`
: "";
const statsHtml = [];
if (isSelf) {
statsHtml.push(`<span><span class="k">peers</span> <span class="v">${node.peer_count || 0}</span></span>`);
statsHtml.push(`<span><span class="k">signals 24h</span> <span class="v">${node.signals_count_24h || 0}</span></span>`);
statsHtml.push(`<span><span class="k">corroborations</span> <span class="v">${node.corroboration_count_24h || 0}</span></span>`);
statsHtml.push(`<span><span class="k">translog</span> <span class="v">${node.translog_entry_count || 0} entries</span></span>`);
} else if (isTransitive) {
const parent = peerByFp[n.via];
const via = parent ? (parent.domain || shortFp(parent.fp)) : shortFp(n.via);
statsHtml.push(`<span><span class="k">distance</span> <span class="v">2 hops</span></span>`);
statsHtml.push(`<span><span class="k">learned via</span> <span class="v">${esc(via)}</span></span>`);
statsHtml.push(`<span><span class="k">stats</span> <span class="v">— (peer-side only)</span></span>`);
} else {
statsHtml.push(`<span><span class="k">status</span> <span class="v">trusted</span></span>`);
statsHtml.push(`<span><span class="k">signals 24h</span> <span class="v">${stats.signal_count_24h || 0}</span></span>`);
statsHtml.push(`<span><span class="k">cases / iocs 24h</span> <span class="v">${stats.cases_24h || 0} / ${stats.iocs_24h || 0}</span></span>`);
statsHtml.push(`<span><span class="k">quorum hits</span> <span class="v">${stats.quorum_contribution_24h || 0}</span></span>`);
statsHtml.push(`<span><span class="k">last seen</span> <span class="v">${esc(fmtAge(stats.last_seen))}</span></span>`);
}
const cta = peerHref
? `<a class="fe-walk-cta" href="${esc(peerHref)}">View this peer's federation <span aria-hidden="true">→</span></a>`
: `<span class="fe-walk-cta fe-walk-cta-disabled" title="no public domain on file for this peer">no public address known</span>`;
walkEl.innerHTML = `
<div class="fe-walk-card">
<div class="fe-walk-card-body">
<h3 class="fe-walk-card-title">${esc(n.domain || n.label || shortFp(n.fp))}</h3>
<div class="fe-walk-card-fp">${esc(n.fp)}</div>
<div class="fe-walk-card-stats">${statsHtml.join("")}</div>
</div>
${cta}
</div>`;
}
// ---------- inbound vouches list ------------------------------------
function renderVouchesIn() {
if (!vouchesInList) return;
if (vouchesInCountEl) vouchesInCountEl.textContent = String(vouchesIn.length);
if (!vouchesIn.length) {
vouchesInList.innerHTML = `<li class="fe-vouches-in-empty">no inbound vouches yet</li>`;
return;
}
const items = vouchesIn.map(v => {
const fp = v.voucher_fingerprint || "";
const peer = peerByFp[fp];
const label = peer ? (peer.domain || shortFp(fp)) : shortFp(fp);
const clickable = peer ? `data-jump="${esc(fp)}"` : `data-jump=""`;
return `<li>
<span class="fp">
<button type="button" class="fn-fp-jump" ${clickable}>${esc(label)}</button>
<code style="margin-left:8px;color:var(--muted);font-size:11px;">${esc(fp)}</code>
</span>
<span class="ts">${esc(v.issued_at || "")}</span>
</li>`;
}).join("");
vouchesInList.innerHTML = items;
vouchesInList.querySelectorAll(".fn-fp-jump").forEach(btn => {
btn.addEventListener("click", () => {
const fp = btn.getAttribute("data-jump") || "";
if (fp) jumpToFp(fp);
});
});
}
renderVouchesIn();
// ---------- copy buttons on the static page -------------------------
document.querySelectorAll(".fn-copy-btn").forEach(btn => {
btn.addEventListener("click", () => {
const v = btn.getAttribute("data-copy") || "";
if (!v) return;
if (navigator.clipboard && navigator.clipboard.writeText) {
navigator.clipboard.writeText(v).catch(() => {});
}
const t = btn.textContent;
btn.textContent = "copied";
setTimeout(() => { btn.textContent = t; }, 1100);
});
});
// ---------- idle animation ------------------------------------------
function loop() {
let moving = false;
for (const n of nodes) {
if (Math.abs(n.vx) > 0.04 || Math.abs(n.vy) > 0.04) { moving = true; break; }
}
if (moving || energyBudget > 0 || dragging) {
tick(); paint();
if (energyBudget > 0) energyBudget--;
}
requestAnimationFrame(loop);
}
loop();
// ---------- edge liveness + flow toggle -----------------------------
edges.forEach((e, i) => {
const ln = edgeEls[i].line;
if (e.kind === "signal") ln.classList.add("alive");
if (e.kind === "knows") ln.classList.add("dim");
});
const flowToggle = document.getElementById("fn-flow");
function applyFlowToggle() {
svg.classList.toggle("flow-off", !(flowToggle && flowToggle.checked));
}
applyFlowToggle();
if (flowToggle) flowToggle.addEventListener("change", applyFlowToggle);
// ---------- layout modes --------------------------------------------
function unfix() { for (const n of nodes) n.fixed = false; }
function clearVel() { for (const n of nodes) { n.vx = 0; n.vy = 0; } }
function applyForce() {
unfix();
for (const n of nodes) {
if (n.is_self) { n.x = W / 2; n.y = H / 2; n.fixed = true; continue; }
n.vx = (Math.random() - 0.5) * 5;
n.vy = (Math.random() - 0.5) * 5;
}
energyBudget = 300;
}
function applyHierarchical() {
const self = nodes.find(n => n.is_self);
const direct = nodes.filter(n => !n.is_self && n.distance === 1);
const transitive = nodes.filter(n => n.distance >= 2);
if (self) { self.x = W / 2; self.y = 70; self.fixed = true; }
direct.forEach((n, i) => {
n.x = W * (i + 1) / (direct.length + 1);
n.y = H * 0.42;
n.fixed = true;
});
const tCount = transitive.length || 1;
transitive.forEach((n, i) => {
n.x = W * (i + 1) / (tCount + 1);
n.y = H * 0.78;
n.fixed = true;
});
clearVel(); paint();
}
function applyRadial() {
const self = nodes.find(n => n.is_self);
const direct = nodes.filter(n => !n.is_self && n.distance === 1);
const transitive = nodes.filter(n => n.distance >= 2);
const R1 = Math.min(W, H) * 0.22;
const R2 = Math.min(W, H) * 0.40;
if (self) { self.x = W / 2; self.y = H / 2; self.fixed = true; }
const dCount = direct.length || 1;
direct.forEach((n, i) => {
const a = (i / dCount) * Math.PI * 2 - Math.PI / 2;
n.x = W / 2 + R1 * Math.cos(a);
n.y = H / 2 + R1 * Math.sin(a);
n.fixed = true;
});
const tCount = transitive.length || 1;
transitive.forEach((n, i) => {
const a = (i / tCount) * Math.PI * 2 - Math.PI / 2;
n.x = W / 2 + R2 * Math.cos(a);
n.y = H / 2 + R2 * Math.sin(a);
n.fixed = true;
});
clearVel(); paint();
}
const LAYOUTS = { force: applyForce, hier: applyHierarchical, radial: applyRadial };
let currentLayout = "force";
const selfNodeRef = nodes.find(n => n.is_self);
if (selfNodeRef) { selfNodeRef.x = W / 2; selfNodeRef.y = H / 2; selfNodeRef.fixed = true; }
document.querySelectorAll(".topo-layout").forEach(btn => {
btn.addEventListener("click", () => {
const mode = btn.dataset.layout;
if (!LAYOUTS[mode] || mode === currentLayout) return;
document.querySelectorAll(".topo-layout").forEach(b => b.classList.toggle("is-active", b === btn));
currentLayout = mode;
LAYOUTS[mode]();
});
});
const resetBtn = document.getElementById("fn-reset");
if (resetBtn) {
resetBtn.addEventListener("click", () => {
if (currentLayout === "force") {
for (const n of nodes) {
if (n.is_self) continue;
n.vx = (Math.random() - 0.5) * 6;
n.vy = (Math.random() - 0.5) * 6;
}
energyBudget = 200;
} else {
LAYOUTS[currentLayout]();
}
});
}
// ---------- wheel zoom + resize -------------------------------------
let zoom = 1, panX = 0, panY = 0;
svg.addEventListener("wheel", ev => {
ev.preventDefault();
const delta = ev.deltaY > 0 ? 1.1 : 0.9;
zoom = Math.max(0.3, Math.min(2.5, zoom * delta));
const vw = W / zoom, vh = H / zoom;
svg.setAttribute("viewBox", `${(W - vw) / 2 + panX} ${(H - vh) / 2 + panY} ${vw} ${vh}`);
}, { passive: false });
window.addEventListener("resize", () => {
const v = viewport();
W = v.W; H = v.H;
svg.setAttribute("viewBox", `0 0 ${W} ${H}`);
energyBudget = 60;
});
// ---------- focus-peer query param ----------------------------------
// ?peer=<domain> auto-selects that peer in the graph so deep links work.
if (settings.focusPeer) {
const target = nodes.find(n => n.domain && n.domain === settings.focusPeer);
if (target) selectNode(target);
}
}
})();

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,17 @@
{
"name": "psyc — defensive CTI cockpit",
"short_name": "psyc",
"description": "Defensive cyber-threat-intelligence routing, sealing, and human-gated response.",
"start_url": "/",
"scope": "/",
"display": "standalone",
"background_color": "#0f1115",
"theme_color": "#1ec8ff",
"orientation": "any",
"icons": [
{ "src": "/static/psyc-logo.png", "sizes": "192x192", "type": "image/png", "purpose": "any" },
{ "src": "/static/psyc-logo.png", "sizes": "512x512", "type": "image/png", "purpose": "any" },
{ "src": "/static/psyc-logo.png", "sizes": "1024x1024", "type": "image/png", "purpose": "any maskable" }
],
"categories": ["security", "productivity"]
}

View File

@@ -0,0 +1,67 @@
// psyc — minimal service worker.
// Strategy:
// • static assets (CSS/JS/PNG) → cache-first, fall back to network
// • HTML pages and API responses → network-first (always fresh data)
// This makes the cockpit installable as a PWA and survives flaky connections,
// without serving stale operational data behind the operator's back.
const CACHE_VERSION = "psyc-v11";
const STATIC_ASSETS = [
"/static/cockpit.css",
"/static/psyc-tokens.css",
"/static/psyc-logo.png",
"/static/nn-sc-icon.png",
"/static/manifest.json",
];
self.addEventListener("install", (event) => {
event.waitUntil(
caches.open(CACHE_VERSION).then((c) => c.addAll(STATIC_ASSETS)).catch(() => {})
);
self.skipWaiting();
});
self.addEventListener("activate", (event) => {
event.waitUntil(
caches.keys().then((keys) =>
Promise.all(keys.filter((k) => k !== CACHE_VERSION).map((k) => caches.delete(k)))
)
);
self.clients.claim();
});
function isStatic(req) {
return /\.(css|js|png|svg|ico|woff2?)$/.test(new URL(req.url).pathname) ||
new URL(req.url).pathname.startsWith("/static/");
}
self.addEventListener("fetch", (event) => {
const req = event.request;
if (req.method !== "GET") return;
if (isStatic(req)) {
// stale-while-revalidate: serve from cache for speed, refresh in the
// background so subsequent loads pick up new CSS/JS without a manual
// version bump. Falls back to network if the cache is cold.
event.respondWith(
caches.open(CACHE_VERSION).then((cache) =>
cache.match(req).then((hit) => {
const network = fetch(req).then((resp) => {
if (resp && resp.ok) cache.put(req, resp.clone()).catch(() => {});
return resp;
}).catch(() => hit);
return hit || network;
})
)
);
} else {
// network-first for HTML + API
event.respondWith(
fetch(req).catch(() => caches.match(req).then((hit) => hit || new Response(
"<!doctype html><meta charset=utf-8><title>psyc offline</title>" +
"<style>body{background:#0f1115;color:#d8dee9;font-family:sans-serif;padding:40px;text-align:center}h1{color:#1ec8ff}</style>" +
"<h1>psyc · offline</h1><p>The cockpit is offline. Reconnect to load fresh data.</p>",
{ headers: { "Content-Type": "text/html" } }
)))
);
}
});

View File

@@ -0,0 +1,543 @@
/* psyc — Docker topology force-directed graph.
*
* Networks are hubs, containers radiate around them, edges = membership.
* Pure vanilla JS, no deps. Layout settles in ~250 ticks then runs an idle
* animation loop that only redraws while velocities matter. Drag a node to
* pin/rearrange; the "re-settle" button kicks the simulation again.
*/
(function () {
"use strict";
const dataEl = document.getElementById("topo-data");
const svg = document.getElementById("topology-graph");
if (!dataEl || !svg) return;
const data = JSON.parse(dataEl.textContent);
// Build nodes + edges from the topology snapshot.
const nodes = [];
const nodeById = Object.create(null);
// Synthetic host node — the physical machine. Networks gateway to it; published
// ports leave the container through this node to the outside world.
const host = {
id: "h:host", type: "host",
label: (data.host && data.host.name) || "docker host",
sub: data.host && data.host.os ? data.host.os : "",
r: 36, x: 0, y: 0, vx: 0, vy: 0, fixed: false,
tooltip: `host: ${(data.host && data.host.name) || "docker host"}\n${(data.host && data.host.os) || ""}`,
};
nodes.push(host); nodeById[host.id] = host;
for (const net of data.networks || []) {
const r = 26 + Math.sqrt((net.containers || []).length) * 4.5;
const n = {
id: "n:" + net.name, type: "net",
label: net.name, driver: (net.driver || "bridge").toLowerCase(),
sub: net.subnet || "",
gateway: net.gateway || "",
internal: !!net.internal,
r, x: 0, y: 0, vx: 0, vy: 0, fixed: false,
tooltip: `switch: ${net.name}\ndriver: ${net.driver} (${net.scope})${net.subnet ? "\nsubnet: " + net.subnet : ""}${net.gateway ? "\ngateway: " + net.gateway : ""}\ncontainers: ${(net.containers || []).length}`,
};
nodes.push(n); nodeById[n.id] = n;
}
for (const c of data.containers || []) {
const n = {
id: "c:" + c.name, type: "cont",
label: c.name, state: c.state || "unknown",
published_ports: c.published_ports || [],
r: 11, x: 0, y: 0, vx: 0, vy: 0, fixed: false,
tooltip: `${c.name}\n${c.image}\n${c.status}${c.published_ports && c.published_ports.length ? "\npublishes: " + c.published_ports.join(", ") : ""}`,
};
nodes.push(n); nodeById[n.id] = n;
}
const edges = [];
// 1) container -> network, labeled with the container's IP on that network
for (const c of data.containers || []) {
for (const cn of c.networks || []) {
const sid = "c:" + c.name;
const tid = "n:" + cn.name;
if (nodeById[sid] && nodeById[tid]) {
edges.push({ source: sid, target: tid, kind: "wire", label: cn.ip || "" });
}
}
}
// 2) non-internal network -> host (uplink with the gateway as label)
for (const net of data.networks || []) {
if (net.internal) continue;
const tid = "n:" + net.name;
if (nodeById[tid]) {
edges.push({ source: host.id, target: tid, kind: "uplink", label: net.gateway || "" });
}
}
// 3) container -> host for published ports (ingress paths from the world)
for (const c of data.containers || []) {
if (!c.published_ports || !c.published_ports.length) continue;
const sid = "c:" + c.name;
if (nodeById[sid]) {
edges.push({ source: sid, target: host.id, kind: "publish", label: c.published_ports.join(" ") });
}
}
// Viewport.
function viewport() {
const W = svg.clientWidth || 900;
const H = parseInt(getComputedStyle(svg).height, 10) || 560;
return { W, H };
}
let { W, H } = viewport();
svg.setAttribute("viewBox", `0 0 ${W} ${H}`);
// Seed positions in a circle around the center so the initial frame isn't a glob.
(function seed() {
const cx = W / 2, cy = H / 2;
nodes.forEach((n, i) => {
const ang = (i / nodes.length) * Math.PI * 2;
const rad = (n.type === "net" ? 60 : 180) + Math.random() * 40;
n.x = cx + rad * Math.cos(ang);
n.y = cy + rad * Math.sin(ang);
});
})();
// Force-sim parameters — tuned for ~50 nodes.
const REPULSION = 1400;
const SPRING_K = 0.045;
const SPRING_REST = 110;
const DAMP = 0.82;
const CENTER_PULL = 0.004;
function tick() {
// Repulsion (O(n^2) — fine here).
for (let i = 0; i < nodes.length; i++) {
const a = nodes[i];
for (let j = i + 1; j < nodes.length; j++) {
const b = nodes[j];
const dx = b.x - a.x, dy = b.y - a.y;
const d2 = dx * dx + dy * dy + 0.1;
const d = Math.sqrt(d2);
const f = REPULSION / d2;
const fx = (dx / d) * f, fy = (dy / d) * f;
if (!a.fixed) { a.vx -= fx; a.vy -= fy; }
if (!b.fixed) { b.vx += fx; b.vy += fy; }
}
}
// Spring attraction along edges.
for (const e of edges) {
const a = nodeById[e.source], b = nodeById[e.target];
const dx = b.x - a.x, dy = b.y - a.y;
const d = Math.sqrt(dx * dx + dy * dy) + 0.1;
const f = (d - SPRING_REST) * SPRING_K;
const fx = (dx / d) * f, fy = (dy / d) * f;
if (!a.fixed) { a.vx += fx; a.vy += fy; }
if (!b.fixed) { b.vx -= fx; b.vy -= fy; }
}
// Gentle center gravity + damping + integrate.
for (const n of nodes) {
if (n.fixed) { n.vx = 0; n.vy = 0; continue; }
n.vx += (W / 2 - n.x) * CENTER_PULL;
n.vy += (H / 2 - n.y) * CENTER_PULL;
n.vx *= DAMP; n.vy *= DAMP;
n.x += n.vx; n.y += n.vy;
n.x = Math.max(n.r, Math.min(W - n.r, n.x));
n.y = Math.max(n.r, Math.min(H - n.r, n.y));
}
}
// Pre-settle so the first frame isn't chaos.
for (let i = 0; i < 300; i++) tick();
// ---------- rendering ----------------------------------------------------
const ns = "http://www.w3.org/2000/svg";
const edgesG = document.createElementNS(ns, "g");
const nodesG = document.createElementNS(ns, "g");
edgesG.setAttribute("class", "topo-edges");
nodesG.setAttribute("class", "topo-nodes");
svg.appendChild(edgesG);
svg.appendChild(nodesG);
const edgeEls = edges.map(e => {
const grp = document.createElementNS(ns, "g");
grp.setAttribute("class", "topo-edge-grp topo-kind-" + e.kind);
const ln = document.createElementNS(ns, "line");
ln.setAttribute("class", "topo-edge");
grp.appendChild(ln);
const lbl = document.createElementNS(ns, "text");
lbl.setAttribute("class", "topo-edge-label");
lbl.textContent = e.label || "";
grp.appendChild(lbl);
edgesG.appendChild(grp);
return { line: ln, label: lbl };
});
function _classFor(n) {
if (n.type === "host") return "topo-node topo-host";
if (n.type === "net") return "topo-node topo-net topo-driver-" + n.driver + (n.internal ? " topo-internal" : "");
return "topo-node topo-cont topo-state-" + n.state;
}
const nodeEls = nodes.map(n => {
const g = document.createElementNS(ns, "g");
g.setAttribute("class", _classFor(n));
g.dataset.id = n.id;
if (n.type === "host") {
// Render the host as a rounded square so it visually reads as the "outside world" anchor.
const sz = n.r;
const rect = document.createElementNS(ns, "rect");
rect.setAttribute("x", -sz); rect.setAttribute("y", -sz);
rect.setAttribute("width", sz * 2); rect.setAttribute("height", sz * 2);
rect.setAttribute("rx", 8); rect.setAttribute("ry", 8);
g.appendChild(rect);
} else if (n.type === "net") {
// Switches: hexagon-ish double-stroke for a "device" feel.
const c = document.createElementNS(ns, "circle");
c.setAttribute("r", n.r); g.appendChild(c);
const c2 = document.createElementNS(ns, "circle");
c2.setAttribute("r", n.r - 5); c2.setAttribute("class", "topo-net-inner"); g.appendChild(c2);
} else {
const c = document.createElementNS(ns, "circle");
c.setAttribute("r", n.r); g.appendChild(c);
}
const text = document.createElementNS(ns, "text");
text.setAttribute("class", "topo-label");
text.setAttribute("dy", (n.type === "host" ? n.r + 14 : n.r + 13));
text.textContent = n.label;
g.appendChild(text);
if (n.sub) {
const sub = document.createElementNS(ns, "text");
sub.setAttribute("class", "topo-sublabel");
sub.setAttribute("dy", (n.type === "host" ? n.r + 26 : n.r + 24));
sub.textContent = n.sub;
g.appendChild(sub);
}
const title = document.createElementNS(ns, "title");
title.textContent = n.tooltip;
g.appendChild(title);
nodesG.appendChild(g);
return g;
});
function paint() {
for (let i = 0; i < edges.length; i++) {
const e = edges[i], a = nodeById[e.source], b = nodeById[e.target];
const els = edgeEls[i];
els.line.setAttribute("x1", a.x); els.line.setAttribute("y1", a.y);
els.line.setAttribute("x2", b.x); els.line.setAttribute("y2", b.y);
if (els.label.textContent) {
const mx = (a.x + b.x) / 2, my = (a.y + b.y) / 2;
els.label.setAttribute("x", mx); els.label.setAttribute("y", my - 3);
}
}
for (let i = 0; i < nodes.length; i++) {
nodeEls[i].setAttribute("transform", `translate(${nodes[i].x},${nodes[i].y})`);
}
}
paint();
// ---------- drag + click ------------------------------------------------
let dragging = null, dragOffset = { x: 0, y: 0 };
let pressedNode = null, pressedAt = null, moved = false;
function svgPoint(clientX, clientY) {
const pt = svg.createSVGPoint(); pt.x = clientX; pt.y = clientY;
return pt.matrixTransform(svg.getScreenCTM().inverse());
}
nodeEls.forEach((g, i) => {
g.addEventListener("mousedown", ev => {
ev.preventDefault();
pressedNode = nodes[i];
pressedAt = { x: ev.clientX, y: ev.clientY };
moved = false;
dragging = nodes[i];
const p = svgPoint(ev.clientX, ev.clientY);
dragOffset.x = p.x - dragging.x; dragOffset.y = p.y - dragging.y;
if (currentLayout === "force") dragging.fixed = true;
g.classList.add("dragging");
});
});
document.addEventListener("mousemove", ev => {
if (pressedAt) {
const dx = ev.clientX - pressedAt.x, dy = ev.clientY - pressedAt.y;
if (dx * dx + dy * dy > 16) moved = true; // > 4px = drag, not click
}
if (!dragging) return;
const p = svgPoint(ev.clientX, ev.clientY);
dragging.x = p.x - dragOffset.x; dragging.y = p.y - dragOffset.y;
dragging.vx = 0; dragging.vy = 0;
energyBudget = 80;
});
document.addEventListener("mouseup", () => {
if (dragging) {
const g = nodesG.querySelector(`[data-id="${CSS.escape(dragging.id)}"]`);
if (g) g.classList.remove("dragging");
if (currentLayout === "force") dragging.fixed = false;
dragging = null;
}
if (pressedNode && !moved) selectNode(pressedNode);
pressedNode = null; pressedAt = null;
});
// Click on empty graph area clears selection.
svg.addEventListener("click", ev => {
if (!ev.target.closest(".topo-node")) clearSelection();
});
// ---------- spec panel (click any node) --------------------------------
const detailEl = document.getElementById("topo-detail");
function esc(s) {
return String(s == null ? "" : s).replace(/[&<>"']/g, c =>
({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" }[c]));
}
function selectNode(n) {
nodeEls.forEach(el => el.classList.remove("selected"));
const me = nodesG.querySelector(`[data-id="${CSS.escape(n.id)}"]`);
if (me) me.classList.add("selected");
renderDetail(n);
}
function clearSelection() {
nodeEls.forEach(el => el.classList.remove("selected"));
if (detailEl) detailEl.innerHTML = '<p class="topo-detail-empty">Click any node in the graph above to inspect it.</p>';
}
function _kvRow(k, v) { return `<dt>${esc(k)}</dt><dd>${v}</dd>`; }
function _stateChip(s) {
const cls = s === "running" ? "state-running" : (s === "exited" || s === "dead") ? "state-exited" : (s === "paused" || s === "restarting") ? "state-paused" : "";
return `<span class="state-badge ${cls}">${esc(s || "?")}</span>`;
}
function renderDetail(n) {
if (!detailEl) return;
const kind = n.id.slice(0, 1);
const name = n.id.slice(2);
let html = "";
if (kind === "h") {
const h = data.host || {};
html = `<div class="td-head"><span class="td-kind td-kind-host">HOST</span>
<h3 class="td-title">${esc(h.name || "docker host")}</h3>
<button type="button" class="td-close" aria-label="close">×</button></div>
<dl class="td-kv">
${_kvRow("OS", esc(h.os || "—"))}
${_kvRow("CPUs", h.ncpu != null ? esc(h.ncpu) : "—")}
${_kvRow("Containers", h.containers != null ? `${esc(h.containers)} (running: ${esc(h.containers_running)})` : "—")}
</dl>`;
} else if (kind === "n") {
const net = (data.networks || []).find(x => x.name === name) || {};
const conts = net.containers || [];
html = `<div class="td-head"><span class="td-kind td-kind-net">SWITCH</span>
<h3 class="td-title">${esc(net.name)}</h3>
<button type="button" class="td-close" aria-label="close">×</button></div>
<dl class="td-kv">
${_kvRow("ID", esc(net.id))}
${_kvRow("Driver", esc(net.driver) + " · " + esc(net.scope))}
${_kvRow("Subnet", esc(net.subnet || "—"))}
${_kvRow("Gateway", esc(net.gateway || "—"))}
${_kvRow("Internal", net.internal ? "yes" : "no")}
</dl>
<h4 class="td-sec">Attached containers (${conts.length})</h4>
${conts.length ? `<table class="td-tbl">
<thead><tr><th>Name</th><th>IPv4</th><th>MAC</th></tr></thead>
<tbody>${conts.map(c => `<tr><td><a href="#" class="td-jump" data-id="c:${esc(c.name)}">${esc(c.name)}</a></td><td>${esc(c.ip || "—")}</td><td>${esc(c.mac || "—")}</td></tr>`).join("")}</tbody>
</table>` : `<p class="empty">No containers attached.</p>`}`;
} else {
const c = (data.containers || []).find(x => x.name === name) || {};
const nets = c.networks || [];
const pub = c.published_ports || [];
html = `<div class="td-head"><span class="td-kind td-kind-cont">CONTAINER</span>
<h3 class="td-title">${esc(c.name)}</h3>
${_stateChip(c.state)}
<button type="button" class="td-close" aria-label="close">×</button></div>
<dl class="td-kv">
${_kvRow("ID", esc(c.id))}
${_kvRow("Image", `<code>${esc(c.image)}</code>`)}
${_kvRow("Status", esc(c.status))}
${_kvRow("Published", pub.length ? pub.map(p => `<span class="port-pill">${esc(p)}</span>`).join(" ") : "—")}
</dl>
<h4 class="td-sec">Networks (${nets.length})</h4>
${nets.length ? `<table class="td-tbl">
<thead><tr><th>Switch</th><th>IPv4</th><th>MAC</th><th>Gateway</th></tr></thead>
<tbody>${nets.map(nn => `<tr><td><a href="#" class="td-jump" data-id="n:${esc(nn.name)}">${esc(nn.name)}</a></td><td>${esc(nn.ip || "—")}</td><td>${esc(nn.mac || "—")}</td><td>${esc(nn.gateway || "—")}</td></tr>`).join("")}</tbody>
</table>` : `<p class="empty">No networks attached.</p>`}
${(c.ports || []).length ? `<h4 class="td-sec">All ports</h4>
<ul class="td-portlist">${c.ports.map(p => `<li>${esc(p)}</li>`).join("")}</ul>` : ""}`;
}
detailEl.innerHTML = html;
detailEl.classList.add("has-selection");
// Wire close + cross-jumps.
const close = detailEl.querySelector(".td-close");
if (close) close.addEventListener("click", clearSelection);
detailEl.querySelectorAll(".td-jump").forEach(a => {
a.addEventListener("click", ev => {
ev.preventDefault();
const id = a.dataset.id;
const target = nodeById[id];
if (target) selectNode(target);
});
});
// Smooth scroll the panel into view.
detailEl.scrollIntoView({ behavior: "smooth", block: "nearest" });
}
// ---------- idle animation ---------------------------------------------
let energyBudget = 40;
function loop() {
let moving = false;
for (const n of nodes) {
if (Math.abs(n.vx) > 0.04 || Math.abs(n.vy) > 0.04) { moving = true; break; }
}
if (moving || energyBudget > 0 || dragging) {
tick(); paint();
if (energyBudget > 0) energyBudget--;
}
requestAnimationFrame(loop);
}
loop();
// ---------- traffic flow on edges --------------------------------------
// An edge is "alive" if its container endpoint is running, or if it's an
// uplink (host↔switch) — those are considered backbone. Dead edges fade.
function markEdgeLiveness() {
edges.forEach((e, i) => {
const a = nodeById[e.source], b = nodeById[e.target];
const contAlive =
(a.type === "cont" && a.state === "running") ||
(b.type === "cont" && b.state === "running");
const alwaysOn = e.kind === "uplink";
const alive = contAlive || alwaysOn;
const ln = edgeEls[i].line;
ln.classList.toggle("alive", alive);
ln.classList.toggle("dead", !alive);
});
}
markEdgeLiveness();
const flowToggle = document.getElementById("topo-flow");
function applyFlowToggle() {
svg.classList.toggle("flow-off", !(flowToggle && flowToggle.checked));
}
applyFlowToggle();
if (flowToggle) flowToggle.addEventListener("change", applyFlowToggle);
// ---------- layout modes (force | hierarchical | radial) ----------------
function unfix() { for (const n of nodes) n.fixed = false; }
function clearVel() { for (const n of nodes) { n.vx = 0; n.vy = 0; } }
function applyForce() {
unfix();
for (const n of nodes) { n.vx = (Math.random() - 0.5) * 5; n.vy = (Math.random() - 0.5) * 5; }
energyBudget = 300;
}
function applyHierarchical() {
const switches = nodes.filter(n => n.type === "net");
const conts = nodes.filter(n => n.type === "cont");
// Group containers by their primary (first) connected switch.
const groups = {};
for (const c of conts) {
const edge = edges.find(e => e.source === c.id && nodeById[e.target] && nodeById[e.target].type === "net");
const swId = edge ? edge.target : "_unattached";
(groups[swId] = groups[swId] || []).push(c);
}
host.x = W / 2; host.y = 60; host.fixed = true;
switches.forEach((sw, i) => {
sw.x = W * (i + 1) / (switches.length + 1);
sw.y = H * 0.36;
sw.fixed = true;
});
for (const [swId, group] of Object.entries(groups)) {
const sw = nodeById[swId];
const cx = sw ? sw.x : 40;
const cy = sw ? sw.y + 90 : H - 50;
const cols = Math.max(1, Math.ceil(Math.sqrt(group.length)));
group.forEach((c, idx) => {
const col = idx % cols, row = Math.floor(idx / cols);
c.x = cx + (col - (cols - 1) / 2) * 38;
c.y = cy + row * 38;
c.fixed = true;
});
}
clearVel(); paint();
}
function applyRadial() {
const switches = nodes.filter(n => n.type === "net");
const conts = nodes.filter(n => n.type === "cont");
const R1 = Math.min(W, H) * 0.22;
const R2 = Math.min(W, H) * 0.42;
host.x = W / 2; host.y = H / 2; host.fixed = true;
switches.forEach((sw, i) => {
const a = (i / switches.length) * Math.PI * 2 - Math.PI / 2;
sw.x = W / 2 + R1 * Math.cos(a);
sw.y = H / 2 + R1 * Math.sin(a);
sw._angle = a; sw.fixed = true;
});
// Bucket containers by their primary switch.
const groups = {};
for (const c of conts) {
const edge = edges.find(e => e.source === c.id && nodeById[e.target] && nodeById[e.target].type === "net");
const swId = edge ? edge.target : "_unattached";
(groups[swId] = groups[swId] || []).push(c);
}
const slice = switches.length ? (Math.PI * 2) / switches.length : Math.PI;
for (const [swId, group] of Object.entries(groups)) {
const sw = nodeById[swId];
const baseAng = sw ? sw._angle : Math.PI;
group.forEach((c, idx) => {
const t = group.length === 1 ? 0 : (idx / (group.length - 1) - 0.5);
const a = baseAng + t * slice * 0.75;
c.x = W / 2 + R2 * Math.cos(a);
c.y = H / 2 + R2 * Math.sin(a);
c.fixed = true;
});
}
clearVel(); paint();
}
const LAYOUTS = { force: applyForce, hier: applyHierarchical, radial: applyRadial };
let currentLayout = "force";
document.querySelectorAll(".topo-layout").forEach(btn => {
btn.addEventListener("click", () => {
const mode = btn.dataset.layout;
if (!LAYOUTS[mode] || mode === currentLayout) return;
document.querySelectorAll(".topo-layout").forEach(b => b.classList.toggle("is-active", b === btn));
currentLayout = mode;
LAYOUTS[mode]();
});
});
// ---------- controls + resize -------------------------------------------
const resetBtn = document.getElementById("topo-reset");
if (resetBtn) {
resetBtn.addEventListener("click", () => {
if (currentLayout === "force") {
for (const n of nodes) { n.vx = (Math.random() - 0.5) * 6; n.vy = (Math.random() - 0.5) * 6; }
energyBudget = 200;
} else {
LAYOUTS[currentLayout]();
}
});
}
// Wheel zoom on the SVG (changes viewBox).
let zoom = 1, panX = 0, panY = 0;
svg.addEventListener("wheel", ev => {
ev.preventDefault();
const delta = ev.deltaY > 0 ? 1.1 : 0.9;
zoom = Math.max(0.3, Math.min(2.5, zoom * delta));
const vw = W / zoom, vh = H / zoom;
svg.setAttribute("viewBox", `${(W - vw) / 2 + panX} ${(H - vh) / 2 + panY} ${vw} ${vh}`);
}, { passive: false });
window.addEventListener("resize", () => {
const v = viewport();
W = v.W; H = v.H;
svg.setAttribute("viewBox", `0 0 ${W} ${H}`);
energyBudget = 60;
});
})();

View File

@@ -0,0 +1,67 @@
{% extends "base.html" %}
{% block title %}Admin — psyc{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Admin Control Center</h1>
<span class="count">{{ members|length }} member{{ '' if members|length == 1 else 's' }} enrolled</span>
</div>
<p class="page-intro">The secured zone — TOTP-gated, hidden from the nav. Manage who can get in here, and (next) watch the live infrastructure.</p>
<div class="verdict verdict-clean">✓ Admin session active — expires after 60 min idle.</div>
</section>
<section class="panel">
<div class="panel-head">
<h2>Access Control</h2>
<span class="count">{{ members|length }} enrolled</span>
</div>
<p class="page-intro">Every member enrolls their own authenticator — no shared secret. To offboard someone, revoke their slot; everyone else keeps working, no re-enrollment.</p>
{% if new_qr %}
<div class="enroll-card">
<div class="gate-qr-frame" style="margin:0;"><img class="gate-qr" src="{{ new_qr }}" alt="enrollment QR"></div>
<div class="enroll-body">
<h3>Enroll “{{ new_label or 'member' }}”</h3>
<p>Have them scan this <strong>once</strong> with Google&nbsp;Authenticator / Authy. It won't be shown again — reload to hide it.</p>
</div>
</div>
{% endif %}
<table class="ledger">
<thead><tr><th>Member</th><th>Enrolled</th><th>Last used</th><th></th></tr></thead>
<tbody>
{% for m in members %}
<tr class="ledger-row">
<td><strong>{{ m.label }}</strong></td>
<td class="lg-ts">{{ (m.created_at or '')[:16] | replace('T', ' ') }}</td>
<td class="lg-ts">{{ ((m.last_used or '')[:16] | replace('T', ' ')) or '—' }}</td>
<td>
<form method="post" action="/admin/members/{{ m.id }}/revoke" class="queue-action"
data-confirm-revoke="member" data-confirm-name="{{ m.label }}">
<button type="submit" class="btn btn-reject">revoke</button>
</form>
</td>
</tr>
{% endfor %}
</tbody>
</table>
<form method="post" action="/admin/members" class="lookup-form" style="margin-top:14px;">
<input type="text" name="label" placeholder="new member name (e.g. alice)" class="lookup-input" maxlength="40">
<button type="submit" class="btn btn-enforce">+ Add member</button>
</form>
</section>
<section class="panel">
<div class="panel-head">
<h2>Infrastructure</h2>
<a href="/admin/docker" class="lg-sub">open topology →</a>
</div>
<div class="admin-grid">
<a href="/admin/docker" class="admin-tile admin-tile-link">
<h2>Docker topology</h2>
<p>Live container roster + network map, read-only via socket-proxy. Click to open.</p>
</a>
</div>
</section>
{% endblock %}

View File

@@ -0,0 +1,72 @@
{% extends "base.html" %}
{% block title %}Discovery — psyc admin{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Peer discovery</h1>
<span class="count">{{ candidates|length }} candidate{{ '' if candidates|length == 1 else 's' }}</span>
</div>
<p class="page-intro">Walk DNS-SD records from a seed domain you know runs psyc, then recurse through its public peer list. Newly-found peers land here with status <code>unknown</code> — vouching is what eventually promotes them. Once seeds exist, enabling the <code>peer-pull</code> pulse pipeline runs this on a cadence.</p>
<p class="back"><a href="/admin/federation">← back to federation</a></p>
{% if flash %}
<div class="verdict verdict-clean">{{ flash }}</div>
{% endif %}
</section>
<section class="panel">
<div class="panel-head">
<h2>Seed domains</h2>
<span class="count">{{ seeds|length }} configured</span>
</div>
<p class="page-intro">One domain per line. Each seed is resolved via <code>_psyc._tcp.&lt;domain&gt;</code> SRV+TXT; its <code>/federation/peers/public</code> is fetched and recursed.</p>
<form method="post" action="/admin/federation/discovery/seeds" style="display:grid; gap:10px; max-width:680px;">
<textarea name="seeds" rows="6" class="lookup-input" style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:12px;" placeholder="peer1.example.com&#10;peer2.example.org">{{ seeds_text }}</textarea>
<div style="display:flex; gap:10px;">
<button type="submit" class="btn btn-enforce">save seeds</button>
</div>
</form>
<form method="post" action="/admin/federation/discovery/walk" style="margin-top:14px;">
<button type="submit" class="btn btn-approve" {% if not seeds %}disabled{% endif %}>walk now</button>
</form>
</section>
<section class="panel">
<div class="panel-head">
<h2>Recent candidates</h2>
<span class="count">{{ candidates|length }} in registry</span>
</div>
<p class="page-intro">Every peer the walker has ever found, newest first. Trusted/blocked statuses are preserved across re-walks — discovery never demotes a peer the operator has classified.</p>
{% if candidates %}
<table class="ledger">
<thead><tr><th>Domain</th><th>Fingerprint</th><th>Status</th><th>Discovered</th><th>Last seen</th><th>Notes</th></tr></thead>
<tbody>
{% for p in candidates %}
<tr class="ledger-row">
<td><strong>{{ p.domain }}</strong></td>
<td class="lg-sub" style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace;">{{ p.fingerprint[:8] }}…{{ p.fingerprint[-8:] }}</td>
<td>
{% if p.status == 'trusted' %}
<span class="sev-badge" style="background:rgba(74,222,128,0.10); color:var(--green); border-color:var(--green);">trusted</span>
{% elif p.status == 'blocked' %}
<span class="sev-badge" style="background:rgba(248,113,113,0.10); color:var(--red); border-color:var(--red);">blocked</span>
{% else %}
<span class="sev-badge">{{ p.status }}</span>
{% endif %}
</td>
<td class="lg-ts">{{ (p.discovered_at or '')[:16] | replace('T', ' ') }}</td>
<td class="lg-ts">{{ ((p.last_seen or '')[:16] | replace('T', ' ')) or '—' }}</td>
<td class="lg-sub">{{ (p.notes or '')[:60] }}</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="page-intro">(no candidates yet — add a seed above and walk)</p>
{% endif %}
</section>
{% endblock %}

View File

@@ -0,0 +1,84 @@
{% extends "base.html" %}
{% block title %}Docker topology — psyc admin{% endblock %}
{% block body_class %}wide{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Docker Topology</h1>
<span class="count">{{ topo.containers|length }} containers · {{ topo.networks|length }} networks</span>
</div>
<p class="page-intro">Live read-only view of this host's Docker daemon, routed through <code>{{ topo.proxy }}</code>. The proxy exposes only GET on containers and networks — psyc cannot start, stop, exec into, or modify anything from here.</p>
<p class="back"><a href="/admin">← back to admin</a></p>
{% if topo.error %}
<div class="gate-error">✗ Socket-proxy unreachable: {{ topo.error }}</div>
{% endif %}
{% if topo.containers %}
<div class="topo-stage">
<div class="topo-toolbar">
<div class="topo-layouts" role="tablist">
<button type="button" class="topo-layout is-active" data-layout="force" title="organic force-directed">Force</button>
<button type="button" class="topo-layout" data-layout="hier" title="host on top, switches in a row, containers grouped below">Hierarchical</button>
<button type="button" class="topo-layout" data-layout="radial" title="host in center, switches on a ring, containers on the outer ring">Radial</button>
</div>
<label class="topo-toggle"><input type="checkbox" id="topo-flow" checked> traffic flow</label>
<span class="topo-legend"><span class="lg-swatch lg-net"></span>switch</span>
<span class="topo-legend"><span class="lg-swatch lg-run"></span>running</span>
<span class="topo-legend"><span class="lg-swatch lg-stop"></span>exited</span>
<button type="button" id="topo-reset" class="btn">re-settle</button>
<span class="topo-hint">drag · scroll to zoom</span>
</div>
<svg id="topology-graph" preserveAspectRatio="xMidYMid meet"></svg>
<script id="topo-data" type="application/json">{{ topo|tojson }}</script>
<script src="/static/topology.js" defer></script>
</div>
{% endif %}
<div id="topo-detail" class="topo-detail">
<p class="topo-detail-empty">Click any node in the graph above to inspect it.</p>
</div>
<h2 style="margin-top:18px;">Networks</h2>
<div class="net-grid">
{% for n in topo.networks %}
<div class="net-card net-driver-{{ n.driver }}">
<div class="net-card-head">
<div>
<div class="net-name">{{ n.name }}</div>
<div class="net-meta">{{ n.driver }} · {{ n.scope }}{% if n.internal %} · internal{% endif %}</div>
</div>
<span class="net-count">{{ n.containers|length }}</span>
</div>
{% if n.containers %}
<div class="net-members">
{% for c in n.containers %}
<span class="net-chip"><span class="net-chip-name">{{ c.name }}</span><span class="net-chip-ip">{{ c.ip or '—' }}</span></span>
{% endfor %}
</div>
{% else %}
<div class="net-empty">(no attached containers)</div>
{% endif %}
</div>
{% endfor %}
</div>
<h2 style="margin-top:24px;">Containers</h2>
<table class="ledger">
<thead><tr><th>Name</th><th>Image</th><th>State</th><th>Networks</th><th>Ports</th></tr></thead>
<tbody>
{% for c in topo.containers %}
<tr class="ledger-row">
<td><strong>{{ c.name }}</strong><div class="lg-sub">{{ c.id }}</div></td>
<td class="lg-dest">{{ c.image }}</td>
<td><span class="state-badge state-{{ c.state }}">{{ c.state }}</span></td>
<td>
{% for net in c.networks %}<span class="net-chip mini"><span class="net-chip-name">{{ net.name }}</span><span class="net-chip-ip">{{ net.ip or '—' }}</span></span>{% endfor %}
</td>
<td class="lg-sub">{% for p in c.ports %}{{ p }}{% if not loop.last %}<br>{% endif %}{% endfor %}</td>
</tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -0,0 +1,132 @@
{% extends "base.html" %}
{% block title %}Federation — psyc admin{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Federation Identity</h1>
<span class="count">{{ peers|length }} peer{{ '' if peers|length == 1 else 's' }}</span>
</div>
<p class="page-intro">This node's Ed25519 identity. The fingerprint goes into a DNS TXT record so other psyc nodes can discover this one. The public key lets them verify any feed we publish — the private key never leaves this box.</p>
<p class="back"><a href="/admin">← back to admin</a> &nbsp;·&nbsp; <a href="/admin/federation/discovery">discovery</a> &nbsp;·&nbsp; <a href="/admin/federation/vouches">vouches</a> &nbsp;·&nbsp; <a href="/admin/federation/quorum">quorum config</a> &nbsp;·&nbsp; <a href="/admin/federation/log">transparency log</a> &nbsp;·&nbsp; <a href="/admin/federation/network">network</a></p>
<div class="card" style="margin-bottom:14px;">
<div class="lg-sub">node fingerprint</div>
<div style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:18px; word-break:break-all; margin:6px 0 12px; color:var(--accent); text-shadow:0 0 12px var(--accent-glow);">{{ fingerprint }}</div>
<details>
<summary class="lg-sub" style="cursor:pointer;">public key (PEM)</summary>
<pre style="background:var(--panel-2); border:1px solid var(--border); border-radius:6px; padding:10px; margin-top:8px; overflow-x:auto; font-size:11.5px;">{{ pubkey_pem }}</pre>
</details>
</div>
</section>
<section class="panel">
<div class="panel-head">
<h2>Publish via DNS</h2>
<span class="count">SRV + TXT records</span>
</div>
<p class="page-intro">Paste these into your zone file. Once they're live, any peer that knows your domain can discover the node and pin the right key without out-of-band coordination.</p>
<form method="get" action="/admin/federation" class="lookup-form" style="margin-bottom:12px;">
<input type="text" name="domain" value="{{ suggested_domain }}" class="lookup-input" placeholder="domain to publish on (e.g. psyc.example.com)">
<button type="submit" class="btn btn-enforce">regenerate</button>
</form>
<pre style="background:var(--panel-2); border:1px solid var(--border); border-radius:6px; padding:12px; overflow-x:auto; font-size:12px; line-height:1.5;">{{ dns.human_instructions }}</pre>
</section>
<section class="panel">
<div class="panel-head">
<h2>Known Peers</h2>
<span class="count">{{ peers|length }} registered</span>
</div>
<p class="page-intro">Trusted peers' feeds are signature-verified on every poll. Blocked peers are recorded but ignored. Unknown peers are kept for review — nothing flows from them until you set them trusted.</p>
{% if peers %}
<table class="ledger">
<thead><tr><th>Domain</th><th>Fingerprint</th><th>Status</th><th>Discovered</th><th>Last seen</th><th></th></tr></thead>
<tbody>
{% for p in peers %}
<tr class="ledger-row">
<td><strong>{{ p.domain }}</strong></td>
<td class="lg-sub" style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace;">{{ p.fingerprint[:8] }}…{{ p.fingerprint[-8:] }}</td>
<td>
{% if p.status == 'trusted' %}
<span class="sev-badge" style="background:rgba(74,222,128,0.10); color:var(--green); border-color:var(--green);">trusted</span>
{% elif p.status == 'blocked' %}
<span class="sev-badge" style="background:rgba(248,113,113,0.10); color:var(--red); border-color:var(--red);">blocked</span>
{% else %}
<span class="sev-badge">unknown</span>
{% endif %}
</td>
<td class="lg-ts">{{ (p.discovered_at or '')[:16] | replace('T', ' ') }}</td>
<td class="lg-ts">{{ ((p.last_seen or '')[:16] | replace('T', ' ')) or '—' }}</td>
<td>
<form method="post" action="/admin/federation/peers/{{ p.domain }}/status" class="queue-action">
<input type="hidden" name="status" value="trusted">
<button type="submit" class="btn btn-enforce" {% if p.status == 'trusted' %}disabled{% endif %}>trust</button>
</form>
<form method="post" action="/admin/federation/peers/{{ p.domain }}/status" class="queue-action">
<input type="hidden" name="status" value="blocked">
<button type="submit" class="btn btn-reject" {% if p.status == 'blocked' %}disabled{% endif %}>block</button>
</form>
<form method="post" action="/admin/federation/peers/{{ p.domain }}/remove" class="queue-action"
data-confirm-revoke="peer" data-confirm-name="{{ p.domain }}">
<button type="submit" class="btn">remove</button>
</form>
</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="page-intro">(no peers yet — add one below)</p>
{% endif %}
</section>
<section class="panel">
<div class="panel-head">
<h2>Add Peer</h2>
</div>
<p class="page-intro">Pin a peer's identity manually: their domain, their fingerprint (from their DNS TXT record), and the public key they publish at <code>/federation/key</code>.</p>
<form method="post" action="/admin/federation/peers/add" style="display:grid; gap:10px; max-width:680px;">
<input type="text" name="domain" placeholder="peer domain (e.g. peer.example.com)" class="lookup-input" required>
<input type="text" name="fingerprint" placeholder="fingerprint (32 hex chars)" class="lookup-input" maxlength="64" required style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace;">
<textarea name="pubkey_pem" placeholder="-----BEGIN PUBLIC KEY-----&#10;…&#10;-----END PUBLIC KEY-----" rows="6" class="lookup-input" required style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:11.5px;"></textarea>
<select name="status" class="lookup-input">
<option value="unknown">unknown — record only, don't trust yet</option>
<option value="trusted">trusted — count toward quorum</option>
<option value="blocked">blocked — ignore</option>
</select>
<button type="submit" class="btn btn-enforce">+ register peer</button>
</form>
</section>
<section class="panel">
<div class="panel-head">
<h2>Recent Signals</h2>
<span class="count">last {{ signals|length }} of buffer</span>
</div>
<p class="page-intro">Verified federation signals from peers — case + IOC reports awaiting quorum. The signal buffer is what later quorum logic will count over.</p>
{% if signals %}
<table class="ledger">
<thead><tr><th>Received</th><th>Peer</th><th>Type</th><th>Id</th><th>Hash</th></tr></thead>
<tbody>
{% for s in signals %}
<tr class="ledger-row">
<td class="lg-ts">{{ (s.received_at or '')[:19] | replace('T', ' ') }}</td>
<td class="lg-sub" style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace;">{{ s.peer_fingerprint[:8] }}…</td>
<td><span class="sev-badge">{{ s.signal_type }}</span></td>
<td style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:11.5px;">{{ s.signal_id[:48] }}</td>
<td class="lg-sub" style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:11.5px;">{{ s.signal_hash[:16] }}…</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="page-intro">(no signals received yet — quorum stage will populate this)</p>
{% endif %}
</section>
{% endblock %}

View File

@@ -0,0 +1,63 @@
{% extends "base.html" %}
{% block title %}Transparency Log — psyc admin{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Transparency Log</h1>
<span class="count">id @ {{ head_id }}</span>
</div>
<p class="page-intro">Every signal we accept from a peer is appended to a signed merkle chain. Each entry references the previous entry's hash, so tampering with any historical row invalidates every entry after. Auditors can re-walk and detect a bad peer historically — even one we trusted at the time.</p>
<p class="back"><a href="/admin/federation">← back to federation</a> &nbsp;·&nbsp; <a href="/admin/federation/vouches">vouches</a> &nbsp;·&nbsp; <a href="/admin/federation/quorum">quorum config</a> &nbsp;·&nbsp; <a href="/federation/log/verify">public verify endpoint</a></p>
<div class="card" style="margin-bottom:14px;">
<div class="lg-sub">chain verification</div>
{% if verify_status.ok %}
<div style="margin:6px 0;"><span class="sev-badge" style="background:rgba(74,222,128,0.10); color:var(--green); border-color:var(--green);">verified</span> &nbsp; {{ verify_status.verified }} entries walked, no breaks</div>
{% else %}
<div style="margin:6px 0;"><span class="sev-badge" style="background:rgba(248,113,113,0.10); color:var(--red); border-color:var(--red);">BROKEN</span> &nbsp; {{ verify_status.reason }}</div>
{% endif %}
{% if head_hash %}
<div class="lg-sub" style="margin-top:8px;">head hash</div>
<div style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:13px; word-break:break-all; color:var(--accent);">{{ head_hash }}</div>
{% endif %}
</div>
</section>
<section class="panel">
<div class="panel-head">
<h2>Recent Entries</h2>
<span class="count">{{ entries|length }} of last 200</span>
</div>
<p class="page-intro">Newest first. Hashes are truncated for display — full values are at <code>/federation/log</code>.</p>
{% if entries %}
<table class="ledger">
<thead><tr><th>id</th><th>When</th><th>Type</th><th>Peer / target</th><th>Signal id</th><th>Hash</th></tr></thead>
<tbody>
{% for e in entries %}
<tr class="ledger-row">
<td>{{ e.id }}</td>
<td class="lg-ts">{{ (e.timestamp or '')[:19] | replace('T', ' ') }}</td>
<td><span class="sev-badge">{{ e.entry_type }}</span></td>
<td class="lg-sub" style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:11.5px;">
{% if e.entry_type == 'signal' %}
{{ (e.entry_data.peer_fingerprint or '')[:8] }}…
{% elif e.entry_type == 'vouch' %}
{{ (e.entry_data.voucher_fingerprint or '')[:8] }}…→{{ (e.entry_data.target_fingerprint or '')[:8] }}…
{% else %}
{% endif %}
</td>
<td style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:11.5px;">{{ ((e.entry_data.signal_id or e.entry_data.target_fingerprint or '') | string)[:32] }}</td>
<td class="lg-sub" style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:11.5px;">{{ e.entry_hash[:16] }}…</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="page-intro">(chain empty — no signals appended yet)</p>
{% endif %}
</section>
{% endblock %}

View File

@@ -0,0 +1,67 @@
{% extends "base.html" %}
{% block title %}Federation network — psyc admin{% endblock %}
{% block body_class %}wide{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Federation Network</h1>
<span class="count">{{ stats.total_peers }} direct · <span id="fn-transitive-count"></span> transitive</span>
</div>
<p class="page-intro">Force-directed map of the federation this node sits inside. Self at the center, directly-registered peers at distance 1, peers-of-peers (fetched from each trusted peer's <code>/federation/network</code>) at distance 2. Edges: <em>vouches</em> (solid), <em>signals</em> (dashed, thickness ∝ 24h volume), <em>knows</em> (dotted grey).</p>
<p class="back"><a href="/admin/federation">← federation hub</a> &nbsp;·&nbsp; <a href="/admin/federation/discovery">discovery</a> &nbsp;·&nbsp; <a href="/admin/federation/vouches">vouches</a> &nbsp;·&nbsp; <a href="/admin/federation/quorum">quorum</a> &nbsp;·&nbsp; <a href="/admin/federation/log">log</a></p>
<div class="fn-stats">
<div class="fn-stat"><div class="fn-stat-label">direct peers</div><div class="fn-stat-value">{{ stats.total_peers }}</div></div>
<div class="fn-stat"><div class="fn-stat-label">vouched / trusted</div><div class="fn-stat-value">{{ stats.vouched_peers }}</div></div>
<div class="fn-stat"><div class="fn-stat-label">vouches issued</div><div class="fn-stat-value">{{ stats.vouches_issued }}</div></div>
<div class="fn-stat"><div class="fn-stat-label">signals (24h)</div><div class="fn-stat-value">{{ stats.signals_buffered_24h }}</div></div>
<div class="fn-stat"><div class="fn-stat-label">distinct hashes</div><div class="fn-stat-value">{{ stats.distinct_signal_hashes_24h }}</div></div>
<div class="fn-stat"><div class="fn-stat-label">quorum-met</div><div class="fn-stat-value">{{ stats.quorum_met_count }}</div></div>
</div>
<div class="fn-search-bar">
<label for="fn-search">filter</label>
<input type="search" id="fn-search" class="fn-search-input" placeholder="domain or fingerprint substring…" autocomplete="off" spellcheck="false">
<span id="fn-search-count" class="fn-search-count"></span>
</div>
<div class="topo-stage">
<div class="topo-toolbar">
<div class="topo-layouts" role="tablist">
<button type="button" class="topo-layout is-active" data-layout="force" title="organic force-directed">Force</button>
<button type="button" class="topo-layout" data-layout="hier" title="self on top, direct peers in a row, transitives below">Hierarchical</button>
<button type="button" class="topo-layout" data-layout="radial" title="self at center, peers on a ring, transitives outside">Radial</button>
</div>
<label class="topo-toggle"><input type="checkbox" id="fn-flow" checked> signal flow</label>
<span class="topo-legend"><span class="lg-swatch fn-lg-self"></span>self</span>
<span class="topo-legend"><span class="lg-swatch fn-lg-trusted"></span>trusted</span>
<span class="topo-legend"><span class="lg-swatch fn-lg-vouched"></span>vouched</span>
<span class="topo-legend"><span class="lg-swatch fn-lg-unknown"></span>unknown</span>
<span class="topo-legend"><span class="lg-swatch fn-lg-blocked"></span>blocked</span>
<button type="button" id="fn-reset" class="btn">re-settle</button>
<span class="topo-hint">drag · scroll to zoom · hover for tooltip</span>
</div>
<svg id="federation-network-graph" preserveAspectRatio="xMidYMid meet"></svg>
<div id="fn-tooltip" class="fn-tooltip"></div>
<div id="fn-loading" class="page-intro" style="text-align:center; margin-top:10px;">loading federation network…</div>
<div id="fn-error" class="gate-error" style="display:none;"></div>
</div>
<div id="fn-detail" class="topo-detail">
<p class="topo-detail-empty">Click any node in the graph above to inspect it.</p>
</div>
<div class="fn-timeline-wrap">
<div class="fn-timeline-head">
<h3>signals · last 24h</h3>
<span class="meta" id="fn-timeline-meta"></span>
</div>
<div id="fn-timeline" class="fn-timeline" aria-label="signals received per hour for the last 24 hours"></div>
<div id="fn-timeline-axis" class="fn-timeline-axis"></div>
</div>
<p class="page-intro" style="margin-top:18px;">Self fingerprint: <code style="color:var(--accent);">{{ fingerprint }}</code></p>
</section>
<script src="/static/federation_network.js" defer></script>
{% endblock %}

View File

@@ -0,0 +1,92 @@
{% extends "base.html" %}
{% block title %}Quorum Config — psyc admin{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Quorum Configuration</h1>
<span class="count">trust={{ cfg.trust_min_vouchers }} k={{ cfg.signal_quorum_k }}</span>
</div>
<p class="page-intro"><strong>trust_min_vouchers</strong> — distinct trusted vouchers required to make a new peer listening-eligible. <strong>signal_quorum_k</strong> — distinct listening-eligible peers required to consider a signal_hash quorum-met. Both gates live in pulse_settings; raising them tightens trust, lowering them relaxes it.</p>
<p class="back"><a href="/admin/federation">← back to federation</a> &nbsp;·&nbsp; <a href="/admin/federation/vouches">vouches</a> &nbsp;·&nbsp; <a href="/admin/federation/log">transparency log</a></p>
<form method="post" action="/admin/federation/quorum/save" style="display:grid; gap:10px; max-width:520px;">
<label class="lg-sub">trust_min_vouchers</label>
<input type="number" name="trust_min_vouchers" value="{{ cfg.trust_min_vouchers }}" min="1" max="50" class="lookup-input" required>
<label class="lg-sub">signal_quorum_k</label>
<input type="number" name="signal_quorum_k" value="{{ cfg.signal_quorum_k }}" min="1" max="50" class="lookup-input" required>
<button type="submit" class="btn btn-enforce">save config</button>
</form>
</section>
<section class="panel">
<div class="panel-head">
<h2>Per-Peer Listening Eligibility</h2>
<span class="count">{{ peer_rows|length }}</span>
</div>
<p class="page-intro">A peer's feed gets ingested only when its fingerprint is eligible (directly trusted or vouched into trust).</p>
{% if peer_rows %}
<table class="ledger">
<thead><tr><th>Domain</th><th>Fingerprint</th><th>Status</th><th>Vouched</th><th>Eligible</th></tr></thead>
<tbody>
{% for row in peer_rows %}
<tr class="ledger-row">
<td><strong>{{ row.peer.domain }}</strong></td>
<td class="lg-sub" style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace;">{{ row.peer.fingerprint[:8] }}…{{ row.peer.fingerprint[-8:] }}</td>
<td>
{% if row.peer.status == 'trusted' %}
<span class="sev-badge" style="background:rgba(74,222,128,0.10); color:var(--green); border-color:var(--green);">trusted</span>
{% elif row.peer.status == 'blocked' %}
<span class="sev-badge" style="background:rgba(248,113,113,0.10); color:var(--red); border-color:var(--red);">blocked</span>
{% else %}
<span class="sev-badge">{{ row.peer.status }}</span>
{% endif %}
</td>
<td>{% if row.vouched %}<span class="sev-badge" style="background:rgba(74,222,128,0.10); color:var(--green); border-color:var(--green);">yes</span>{% else %}<span class="sev-badge">no</span>{% endif %}</td>
<td>{% if row.eligible %}<span class="sev-badge" style="background:rgba(74,222,128,0.10); color:var(--green); border-color:var(--green);">listening</span>{% else %}<span class="sev-badge" style="background:rgba(248,113,113,0.10); color:var(--red); border-color:var(--red);">muted</span>{% endif %}</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="page-intro">(no peers registered yet)</p>
{% endif %}
</section>
<section class="panel">
<div class="panel-head">
<h2>Signal Hashes in Buffer</h2>
<span class="count">{{ hash_summary|length }} hashes</span>
</div>
<p class="page-intro">Distinct eligible-peer counts per signal hash. Quorum is met when count ≥ {{ cfg.signal_quorum_k }}.</p>
{% if hash_summary %}
<table class="ledger">
<thead><tr><th>Latest</th><th>Type</th><th>Signal id</th><th>Hash</th><th>Distinct peers</th><th>Eligible</th><th>Quorum</th></tr></thead>
<tbody>
{% for r in hash_summary %}
<tr class="ledger-row">
<td class="lg-ts">{{ (r.latest or '')[:19] | replace('T', ' ') }}</td>
<td><span class="sev-badge">{{ r.signal_type }}</span></td>
<td style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:11.5px;">{{ r.signal_id[:48] }}</td>
<td class="lg-sub" style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:11.5px;">{{ r.signal_hash[:16] }}…</td>
<td>{{ r.distinct_peers }}</td>
<td>{{ r.distinct_eligible }}</td>
<td>
{% if r.quorum_met %}
<span class="sev-badge" style="background:rgba(74,222,128,0.10); color:var(--green); border-color:var(--green);">met</span>
{% else %}
<span class="sev-badge">below</span>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="page-intro">(no signals in buffer yet)</p>
{% endif %}
</section>
{% endblock %}

View File

@@ -0,0 +1,110 @@
{% extends "base.html" %}
{% block title %}Federation Vouches — psyc admin{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Web of Trust</h1>
<span class="count">{{ our_vouches|length }} issued</span>
</div>
<p class="page-intro">A vouch is an Ed25519-signed assertion that we trust another node's fingerprint. Peers gossip our vouches with their feeds, so trust accumulates: once {{ cfg.trust_min_vouchers }} of our trusted peers vouches for a new fingerprint, it becomes <em>listening-eligible</em> — its signed feeds get ingested.</p>
<p class="back"><a href="/admin/federation">← back to federation</a> &nbsp;·&nbsp; <a href="/admin/federation/quorum">quorum config</a> &nbsp;·&nbsp; <a href="/admin/federation/log">transparency log</a></p>
<div class="card" style="margin-bottom:14px;">
<div class="lg-sub">our fingerprint</div>
<div style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:14px; word-break:break-all; color:var(--accent);">{{ fingerprint }}</div>
</div>
</section>
<section class="panel">
<div class="panel-head">
<h2>Vouches We've Issued</h2>
<span class="count">{{ our_vouches|length }}</span>
</div>
<p class="page-intro">We've signed these — peers that fetch our feed will see them and may extend trust accordingly.</p>
{% if our_vouches %}
<table class="ledger">
<thead><tr><th>Target fingerprint</th><th>Issued</th><th>Expires</th><th></th></tr></thead>
<tbody>
{% for v in our_vouches %}
<tr class="ledger-row">
<td style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:12px;">{{ v.target_fingerprint }}</td>
<td class="lg-ts">{{ v.issued_at.isoformat()[:16] | replace('T', ' ') }}</td>
<td class="lg-ts">{{ v.expires_at.isoformat()[:16] | replace('T', ' ') if v.expires_at else '—' }}</td>
<td>
<form method="post" action="/admin/federation/vouches/revoke" class="queue-action"
onsubmit="return confirm('Revoke vouch for {{ v.target_fingerprint[:8] }}…?');">
<input type="hidden" name="target_fingerprint" value="{{ v.target_fingerprint }}">
<button type="submit" class="btn btn-reject">revoke</button>
</form>
</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="page-intro">(no vouches issued yet)</p>
{% endif %}
</section>
<section class="panel">
<div class="panel-head">
<h2>Issue a Vouch</h2>
</div>
<p class="page-intro">Vouch for a peer's fingerprint. Trusted peers see this and may treat the target as listening-eligible.</p>
<form method="post" action="/admin/federation/vouches/issue" style="display:grid; gap:10px; max-width:680px;">
<input type="text" name="target_fingerprint" placeholder="target fingerprint (32 hex chars)" class="lookup-input" maxlength="64" required style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace;">
<input type="number" name="ttl_days" value="90" min="1" max="3650" class="lookup-input" placeholder="ttl days">
<button type="submit" class="btn btn-enforce">+ issue vouch</button>
</form>
</section>
<section class="panel">
<div class="panel-head">
<h2>Per-Peer Quorum Status</h2>
<span class="count">{{ peer_rows|length }} peers</span>
</div>
<p class="page-intro">Threshold: {{ cfg.trust_min_vouchers }} distinct trusted vouchers required to make a non-trusted peer listening-eligible.</p>
{% if peer_rows %}
<table class="ledger">
<thead><tr><th>Peer</th><th>Status</th><th>Vouches</th><th>Quorum met</th><th>Eligible</th></tr></thead>
<tbody>
{% for row in peer_rows %}
<tr class="ledger-row">
<td><strong>{{ row.peer.domain }}</strong><br><span class="lg-sub" style="font-family:ui-monospace,SFMono-Regular,Menlo,monospace;">{{ row.peer.fingerprint[:8] }}…{{ row.peer.fingerprint[-8:] }}</span></td>
<td>
{% if row.peer.status == 'trusted' %}
<span class="sev-badge" style="background:rgba(74,222,128,0.10); color:var(--green); border-color:var(--green);">trusted</span>
{% elif row.peer.status == 'blocked' %}
<span class="sev-badge" style="background:rgba(248,113,113,0.10); color:var(--red); border-color:var(--red);">blocked</span>
{% else %}
<span class="sev-badge">{{ row.peer.status }}</span>
{% endif %}
</td>
<td>{{ row.vouches|length }}</td>
<td>
{% if row.vouched %}
<span class="sev-badge" style="background:rgba(74,222,128,0.10); color:var(--green); border-color:var(--green);">yes</span>
{% else %}
<span class="sev-badge">no</span>
{% endif %}
</td>
<td>
{% if row.eligible %}
<span class="sev-badge" style="background:rgba(74,222,128,0.10); color:var(--green); border-color:var(--green);">listening</span>
{% else %}
<span class="sev-badge" style="background:rgba(248,113,113,0.10); color:var(--red); border-color:var(--red);">muted</span>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="page-intro">(no peers registered yet)</p>
{% endif %}
</section>
{% endblock %}

View File

@@ -0,0 +1,56 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>psyc · restricted</title>
<meta name="robots" content="noindex,nofollow">
<link rel="icon" type="image/png" href="/static/psyc-logo.png">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;600;700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="/static/cockpit.css">
<link rel="stylesheet" href="/static/psyc-tokens.css">
</head>
<body class="gate-body">
<div class="gate-bg"></div>
<div class="gate-wrap">
<div class="gate-card">
<span class="corner tl"></span><span class="corner tr"></span>
<span class="corner bl"></span><span class="corner br"></span>
<div class="gate-emblem">
<svg viewBox="0 0 48 48" width="46" height="46" aria-hidden="true">
<path d="M24 3 L42 10 V24 C42 35 34 42 24 45 C14 42 6 35 6 24 V10 Z"
fill="none" stroke="currentColor" stroke-width="2.2" stroke-linejoin="round"/>
<circle cx="24" cy="21" r="5" fill="none" stroke="currentColor" stroke-width="2.2"/>
<rect x="22" y="24" width="4" height="8" rx="2" fill="currentColor"/>
</svg>
<span class="emblem-ring"></span>
</div>
<div class="gate-status"><span class="pulse-dot"></span> SECURE CHANNEL · TOTP</div>
<h1 class="gate-title">RESTRICTED ZONE</h1>
<p class="gate-sub">psyc · admin control center</p>
{% if not provisioned %}
<div class="gate-setup">
<div class="gate-qr-frame">
<img class="gate-qr" src="{{ qr }}" alt="TOTP enrollment QR">
<span class="scanline"></span>
</div>
<p class="gate-step">Scan with Google&nbsp;Authenticator or Authy to enroll</p>
</div>
{% endif %}
{% if error %}<div class="gate-error">✗ Invalid or expired code</div>{% endif %}
<form method="post" action="/admin/verify" class="gate-form">
<input type="text" name="code" inputmode="numeric" pattern="[0-9]*" autocomplete="one-time-code"
maxlength="6" placeholder="••••••" class="gate-input" autofocus>
<button type="submit" class="gate-btn">UNLOCK</button>
</form>
<p class="gate-hint">6-digit code · rotates every 30 s</p>
</div>
</div>
</body>
</html>

View File

@@ -0,0 +1,181 @@
{% extends "base.html" %}
{% block title %}Pulse — psyc admin{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Pulse — autonomous heartbeat</h1>
<span class="count">{{ pipelines|length }} pipeline{{ '' if pipelines|length == 1 else 's' }}</span>
</div>
<p class="page-intro">Cron-style scheduler that drives every psyc line on a cadence without human input. Each pipeline has an autonomy mode and a cadence in seconds. The kill switch halts everything instantly — it overrides cadence, mode, and the enabled flag.</p>
<p class="back"><a href="/admin">← back to admin</a></p>
{% if flash %}
<div class="verdict verdict-clean">{{ flash }}</div>
{% endif %}
</section>
<section class="panel">
<div class="panel-head">
<h2>Global kill switch</h2>
<span class="count">{{ 'ARMED' if kill_switch else 'OFF' }}</span>
</div>
{% if kill_switch %}
<div class="verdict" style="background:rgba(248,113,113,0.12); border:1px solid rgba(248,113,113,0.45); color:#fca5a5; padding:14px 18px; border-radius:6px; font-weight:700; letter-spacing:0.04em;">
✗ KILL SWITCH ARMED — every pipeline is paused. tick() returns "skipped" for everything. Run-now is also blocked. Toggle off to resume.
</div>
<form method="post" action="/admin/pulse/kill" style="margin-top:14px;">
<button type="submit" class="btn btn-approve">Disarm kill switch</button>
</form>
{% else %}
<div class="verdict verdict-clean">✓ Pulse is live — the background loop ticks every {{ tick_interval }}s.</div>
<form method="post" action="/admin/pulse/kill" style="margin-top:14px;"
onsubmit="return confirm('Arm the kill switch? Every pipeline halts immediately.');">
<button type="submit" class="btn btn-reject">Arm kill switch</button>
</form>
{% endif %}
</section>
<section class="panel">
<div class="panel-head">
<h2>AUTO-RESPONSE STATE</h2>
<span class="count">{{ respond_auto_fired_24h }} auto-fired in last 24h</span>
</div>
<p class="page-intro">When the <code>respond</code> pipeline runs in <code>auto-execute</code>, every PROPOSED action that passes all three gates fires automatically. Below shows the live config + audit trail.</p>
<div style="display:flex; gap:14px; flex-wrap:wrap; margin:14px 0;">
{# Mode badge — traffic-light coloring. auto-execute is "armed" (red), auto-propose is amber, manual is green/safe. #}
{% if respond_mode == 'auto-execute' %}
<div style="padding:10px 14px; border-radius:6px; background:rgba(248,113,113,0.12); border:1px solid rgba(248,113,113,0.45); color:#fca5a5; font-weight:700; letter-spacing:0.04em;">
MODE: auto-execute (ARMED)
</div>
{% elif respond_mode == 'auto-propose' %}
<div style="padding:10px 14px; border-radius:6px; background:rgba(250,204,21,0.12); border:1px solid rgba(250,204,21,0.45); color:#fde047; font-weight:700; letter-spacing:0.04em;">
MODE: auto-propose (staging only)
</div>
{% else %}
<div style="padding:10px 14px; border-radius:6px; background:rgba(74,222,128,0.10); border:1px solid var(--green); color:var(--green); font-weight:700; letter-spacing:0.04em;">
MODE: manual (no proposals)
</div>
{% endif %}
<div style="padding:10px 14px; border-radius:6px; background:var(--panel-2); border:1px solid var(--border); color:var(--text);">
Threshold: <strong>{{ respond_threshold|upper }}+</strong>
</div>
<div style="padding:10px 14px; border-radius:6px; background:var(--panel-2); border:1px solid var(--border); color:var(--text);">
Quorum: <strong style="color: {{ 'var(--green)' if respond_require_quorum else 'var(--muted)' }};">{{ 'ON' if respond_require_quorum else 'OFF' }}</strong>
</div>
<div style="padding:10px 14px; border-radius:6px; background:var(--panel-2); border:1px solid var(--border); color:var(--text);">
Local-only: <strong style="color: {{ 'var(--green)' if respond_local_only else 'var(--muted)' }};">{{ 'ON' if respond_local_only else 'OFF' }}</strong>
</div>
</div>
<form method="post" action="/admin/pulse/respond-config" style="display:flex; gap:10px; align-items:center; flex-wrap:wrap; margin-top:14px;">
<label style="font-size:12px;">Min severity:
<select name="threshold" style="background:var(--panel-2); color:var(--text); border:1px solid var(--border); border-radius:4px; padding:4px 6px;">
{% for s in severity_choices %}
<option value="{{ s }}" {% if s == respond_threshold %}selected{% endif %}>{{ s }}</option>
{% endfor %}
</select>
</label>
<label style="display:inline-flex; align-items:center; gap:4px; font-size:12px;">
<input type="checkbox" name="require_quorum" value="1" {% if respond_require_quorum %}checked{% endif %}> require quorum
</label>
<label style="display:inline-flex; align-items:center; gap:4px; font-size:12px;">
<input type="checkbox" name="local_only" value="1" {% if respond_local_only %}checked{% endif %}> local-only
</label>
<button type="submit" class="btn">save gates</button>
</form>
{% if respond_audit_recent %}
<table class="ledger" style="margin-top:18px;">
<thead>
<tr>
<th style="width:18%;">timestamp</th>
<th style="width:12%;">decision</th>
<th style="width:18%;">case</th>
<th>detail</th>
</tr>
</thead>
<tbody>
{% for row in respond_audit_recent %}
<tr class="ledger-row">
<td class="lg-ts">{{ row.timestamp }}</td>
<td>
{% if row.action == 'auto-fire' %}<span style="color: var(--green);">✓ {{ row.action }}</span>
{% elif row.action == 'error' %}<span style="color: var(--red);">✗ {{ row.action }}</span>
{% else %}<span style="color: var(--muted);">⊘ {{ row.action }}</span>{% endif %}
</td>
<td class="lg-sub"><code>{{ row.case_id or '—' }}</code>{% if row.action_id %} · #{{ row.action_id }}{% endif %}</td>
<td class="lg-sub">{{ row.detail or '' }}</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="lg-sub" style="margin-top:14px;">No auto-response decisions logged yet.</p>
{% endif %}
</section>
<section class="panel">
<div class="panel-head">
<h2>Pipelines</h2>
<span class="count">{{ pipelines|selectattr('enabled')|list|length }} enabled</span>
</div>
<p class="page-intro">Mode: <code>auto-execute</code> fires the line, <code>auto-propose</code> stages proposals for human approval, <code>manual</code> runs only when you press “Run now”. Cadence is the gap between ticks; the loop wakes up every {{ tick_interval }}s.</p>
<table class="ledger">
<thead>
<tr>
<th style="width:24%;">Pipeline</th>
<th>Mode · cadence · enabled</th>
<th style="width:11%;">Last fired</th>
<th style="width:11%;">Next fire</th>
<th>Last result</th>
<th style="width:1%;"></th>
</tr>
</thead>
<tbody>
{% for p in pipelines %}
<tr class="ledger-row">
<td>
<strong>{{ p.title }}</strong>
<div class="lg-sub"><code>{{ p.name }}</code> · {{ p.description }}</div>
</td>
<td>
<form method="post" action="/admin/pulse/{{ p.name }}/update" class="queue-action" style="display:flex; gap:8px; align-items:center; flex-wrap:wrap;">
<select name="mode" style="background:var(--panel-2); color:var(--text); border:1px solid var(--border); border-radius:4px; padding:4px 6px;">
<option value="auto-execute" {% if p.mode.value == 'auto-execute' %}selected{% endif %}>auto-execute</option>
<option value="auto-propose" {% if p.mode.value == 'auto-propose' %}selected{% endif %}>auto-propose</option>
<option value="manual" {% if p.mode.value == 'manual' %}selected{% endif %}>manual</option>
</select>
<input type="number" name="cadence_seconds" value="{{ p.cadence_seconds }}" min="1" step="1"
style="background:var(--panel-2); color:var(--text); border:1px solid var(--border); border-radius:4px; padding:4px 6px; width:84px;" title="cadence in seconds">
<label style="display:inline-flex; align-items:center; gap:4px; font-size:12px;">
<input type="checkbox" name="enabled" value="1" {% if p.enabled %}checked{% endif %}> enabled
</label>
<button type="submit" class="btn">save</button>
</form>
</td>
<td class="lg-ts">
{% if p.last_fired %}<span title="{{ p.last_fired.isoformat() }}">{{ relative(p.last_fired) }}</span>{% else %}—{% endif %}
</td>
<td class="lg-ts">
{% if p.next_fire %}<span title="{{ p.next_fire.isoformat() }}">{{ relative(p.next_fire) }}</span>{% else %}—{% endif %}
</td>
<td class="lg-sub" style="max-width:0;">
{% if p.last_outcome == 'ok' %}<span style="color: var(--green);"></span>
{% elif p.last_outcome == 'err' %}<span style="color: var(--red);"></span>
{% elif p.last_outcome == 'skipped' %}<span style="color: var(--muted);"></span>
{% endif %}
{{ (p.last_result or '—')[:60] }}{% if (p.last_result or '')|length > 60 %}…{% endif %}
</td>
<td>
<form method="post" action="/admin/pulse/{{ p.name }}/run" class="queue-action">
<button type="submit" class="btn btn-enforce" title="Fire now, regardless of cadence">Run now</button>
</form>
</td>
</tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -2,25 +2,101 @@
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<meta name="theme-color" content="#0f1115">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
<meta name="apple-mobile-web-app-title" content="psyc">
<title>{% block title %}psyc cockpit{% endblock %}</title>
<link rel="icon" type="image/png" href="/static/psyc-logo.png">
<link rel="apple-touch-icon" href="/static/psyc-logo.png">
<link rel="manifest" href="/static/manifest.json">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;600;700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="/static/cockpit.css">
<link rel="stylesheet" href="/static/psyc-tokens.css">
<!-- analytics — Umami, cookieless, defer'd so it never blocks render -->
<script defer src="https://analytics.neuronetz.ai/script.js" data-website-id="34c354e5-780e-4c42-a5ce-49b13ff3f088"></script>
<script>
if ("serviceWorker" in navigator) {
window.addEventListener("load", () => {
navigator.serviceWorker.register("/sw.js", { scope: "/" }).catch(() => {});
});
}
// Mobile nav toggle. Topbar gets .nav-open; aria-expanded mirrors it.
document.addEventListener("DOMContentLoaded", () => {
const btn = document.querySelector(".nav-toggle");
const bar = document.querySelector(".topbar");
if (!btn || !bar) return;
btn.addEventListener("click", () => {
const open = bar.classList.toggle("nav-open");
btn.setAttribute("aria-expanded", open ? "true" : "false");
});
// Close the drawer when a nav link is clicked (so navigating dismisses it).
document.querySelectorAll(".nav a").forEach(a => a.addEventListener("click", () => {
bar.classList.remove("nav-open");
btn.setAttribute("aria-expanded", "false");
}));
});
// data-driven confirms (used by /admin and /admin/federation revoke/remove
// buttons; replaces inline onsubmit which was XSS-vulnerable when the
// confirm prompt interpolated a member label or peer domain).
document.addEventListener("DOMContentLoaded", () => {
document.querySelectorAll("form[data-confirm-revoke]").forEach(form => {
form.addEventListener("submit", ev => {
const kind = form.getAttribute("data-confirm-revoke") || "item";
const name = form.getAttribute("data-confirm-name") || "";
const msg = kind === "peer"
? `Remove ${name}? Their signals will no longer count toward quorum.`
: `Revoke ${name}? Their codes stop working immediately.`;
if (!confirm(msg)) ev.preventDefault();
});
});
});
</script>
</head>
<body>
<body class="{% block body_class %}{% endblock %}">
<header class="topbar">
<a class="brand" href="/cases">
<img class="brand-icon" src="/static/psyc-logo.png" alt="psyc">
<span class="brand-sub">operations cockpit</span>
</a>
<nav class="nav">
<button class="nav-toggle" type="button" aria-expanded="false" aria-controls="primary-nav" aria-label="Menu">
<span class="nav-toggle-bar"></span>
<span class="nav-toggle-bar"></span>
<span class="nav-toggle-bar"></span>
</button>
<nav class="nav" id="primary-nav">
<a href="/cases">Cases</a>
<a href="/lookup">Lookup</a>
<a href="/response">Response</a>
<a href="/queue">Queue</a>
<a href="/ledger">Ledger</a>
<a href="/train">Trainline</a>
{% if request.session.get('admin_who') %}
<a href="/admin" class="nav-admin" title="Admin control center">
<svg viewBox="0 0 24 24" width="14" height="14" aria-hidden="true" style="vertical-align:-2px;">
<path d="M12 2 L20 5 V12 C20 17 16.5 20.5 12 22 C7.5 20.5 4 17 4 12 V5 Z"
fill="none" stroke="currentColor" stroke-width="1.8" stroke-linejoin="round"/>
<circle cx="12" cy="11" r="2.4" fill="none" stroke="currentColor" stroke-width="1.8"/>
<rect x="11" y="12.5" width="2" height="4" rx="1" fill="currentColor"/>
</svg>
Admin
</a>
<a href="/admin/pulse" class="nav-admin" title="Autonomy scheduler">Pulse</a>
<a href="/admin/federation" class="nav-admin" title="Federation peers">Federation</a>
{% endif %}
</nav>
{% if request.session.get('admin_who') %}
<span class="admin-chip" title="Admin session · click to manage">
<a href="/admin" class="admin-chip-body">
<span class="admin-chip-dot"></span>
<span class="admin-chip-label">ADMIN · {{ request.session.get('admin_who') }}</span>
</a>
<a href="/admin/logout" class="admin-chip-lock" title="Lock the admin zone (sign out)" aria-label="lock"></a>
</span>
{% endif %}
<span class="model-status" id="model-status" data-state="checking" title="checking…">
<span class="model-status-dot"></span><span class="model-status-text">model</span>
</span>

View File

@@ -42,7 +42,7 @@
<h2>Source</h2>
<dl>
<dt>Type</dt><dd>{{ case.source_type }}</dd>
<dt>Reference</dt><dd>{% if case.source_ref %}<a href="{{ case.source_ref }}" target="_blank" rel="noopener">{{ case.source_ref }}</a>{% else %}—{% endif %}</dd>
<dt>Reference</dt><dd>{% if case.source_ref %}{% if case.source_ref.startswith(('http://', 'https://')) %}<a href="{{ case.source_ref }}" target="_blank" rel="noopener">{{ case.source_ref }}</a>{% else %}<code>{{ case.source_ref }}</code>{% endif %}{% else %}—{% endif %}</dd>
<dt>Observed</dt><dd class="muted">{{ case.observed_at.strftime('%Y-%m-%d %H:%M UTC') }}</dd>
<dt>Ingested</dt><dd class="muted">{{ case.ingested_at.strftime('%Y-%m-%d %H:%M UTC') }}</dd>
</dl>

View File

@@ -6,12 +6,12 @@
<h1>Case Queue</h1>
<span class="count">{{ total }} case{{ '' if total == 1 else 's' }}</span>
</div>
<p class="page-intro">Every threat case psyc is tracking — ingested from URLhaus, CISA KEV, and Feodo Tracker, then classified by severity and TLP. The live queue of what the platform currently knows about; click any case to follow it through the pipeline.</p>
<p class="page-intro">Every threat case psyc is tracking — ingested from URLhaus, CISA KEV, Feodo Tracker, ThreatFox, MalwareBazaar, and AlienVault OTX, then classified by severity and TLP. The live queue of what the platform currently knows about; click any case to follow it through the pipeline.</p>
<details class="page-help">
<summary>how to use this view</summary>
<div class="help-body">
<p><b>How to use.</b> Scan the severity and TLP badges to triage; click any case ID to open its full record and Worker Mesh journey. Run <code>psyc fetch-all</code> to pull fresh cases from the feeds.</p>
<p><b>What you're seeing.</b> Each row is one normalized Case object — ingested by Scoutline from URLhaus, CISA KEV, or Feodo Tracker, then rated by Classifyline.</p>
<p><b>What you're seeing.</b> Each row is one normalized Case object — ingested by Scoutline from any of six feeds (URLhaus, CISA KEV, Feodo Tracker, ThreatFox, MalwareBazaar, OTX), then rated by Classifyline.</p>
<p><b>Why it matters.</b> A defender needs one place that answers "what is happening right now" before deciding what to act on — this queue is that place.</p>
</div>
</details>

View File

@@ -0,0 +1,148 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<meta name="theme-color" content="#0f1115">
<meta name="apple-mobile-web-app-title" content="psyc · federation explorer">
<title>Federation Explorer — psyc</title>
<link rel="icon" type="image/png" href="/static/psyc-logo.png">
<link rel="apple-touch-icon" href="/static/psyc-logo.png">
<link rel="manifest" href="/static/manifest.json">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;600;700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="/static/cockpit.css">
<link rel="stylesheet" href="/static/psyc-tokens.css">
<script defer src="https://analytics.neuronetz.ai/script.js" data-website-id="34c354e5-780e-4c42-a5ce-49b13ff3f088"></script>
<script>
if ("serviceWorker" in navigator) {
window.addEventListener("load", () => {
navigator.serviceWorker.register("/sw.js", { scope: "/" }).catch(() => {});
});
}
</script>
</head>
<body class="wide fe-page">
<header class="topbar fe-topbar">
<a class="brand" href="/federation/explore">
<img class="brand-icon" src="/static/psyc-logo.png" alt="psyc">
<span class="brand-sub">operations cockpit</span>
</a>
<span class="fe-badge" title="Public, no auth — this view is auditable by anyone">
<span class="fe-badge-dot"></span>PUBLIC · TRANSPARENT
</span>
<div class="family">
<img class="family-icon" src="/static/nn-sc-icon.png" alt="NN-sc — Security/Control" title="NN-sc · Security">
</div>
</header>
<main class="content">
<section class="panel fe-hero">
<div class="fe-hero-head">
<h1 class="fe-title">Federation Explorer</h1>
<p class="fe-sub" id="fe-node-domain">{{ domain or fingerprint }}</p>
</div>
<p class="fe-intro">
This is a <strong>public transparency view</strong> of the psyc federation this node sits inside.
Anyone on the internet can verify the trust network — who has vouched for whom,
which signals are corroborated, and that the transparency log hasn't been rewritten.
<strong>Click any peer in the graph to inspect it, then jump to that peer's own explorer.</strong>
</p>
<p class="fe-intro fe-intro-sub">
Self fingerprint:
<code class="fe-fp">{{ fingerprint }}</code>
<button type="button" class="fn-copy-btn" data-copy="{{ fingerprint }}">copy</button>
</p>
</section>
<section class="panel fe-kpi-panel">
<div class="fn-stats fe-kpis">
<div class="fn-stat"><div class="fn-stat-label">direct peers</div><div class="fn-stat-value" id="fe-kpi-peers"></div></div>
<div class="fn-stat"><div class="fn-stat-label">vouches out</div><div class="fn-stat-value" id="fe-kpi-vouches-out"></div></div>
<div class="fn-stat"><div class="fn-stat-label">vouches in</div><div class="fn-stat-value" id="fe-kpi-vouches-in"></div></div>
<div class="fn-stat"><div class="fn-stat-label">signals (24h)</div><div class="fn-stat-value" id="fe-kpi-signals"></div></div>
<div class="fn-stat"><div class="fn-stat-label">corroborations (24h)</div><div class="fn-stat-value" id="fe-kpi-corroboration"></div></div>
<div class="fn-stat"><div class="fn-stat-label">translog entries</div><div class="fn-stat-value" id="fe-kpi-translog"></div></div>
<div class="fn-stat fe-kpi-verify"><div class="fn-stat-label">log integrity</div><div class="fn-stat-value" id="fe-kpi-verify"></div></div>
</div>
<div class="fe-verify-row">
<button type="button" id="fe-verify-btn" class="btn fe-verify-btn">Verify this log →</button>
<span id="fe-verify-result" class="fe-verify-result"></span>
<a class="fe-verify-link" href="/federation/log" target="_blank" rel="noopener">raw log JSON</a>
<a class="fe-verify-link" href="/federation/explore/data" target="_blank" rel="noopener">signed payload</a>
</div>
</section>
<section class="panel">
<div class="panel-head">
<h2>Trust network</h2>
<span class="count"><span id="fe-direct-count"></span> direct · <span id="fe-transitive-count"></span> transitive</span>
</div>
<p class="page-intro">
Self at the center, directly-trusted peers around it,
peers-of-peers (learned from each trusted peer's signed feed) on the outer ring.
Edges: <em>vouches</em> (solid), <em>signals</em> (dashed, thickness ∝ 24h volume),
<em>knows</em> (dotted grey), <em>corroborate</em> (faint pulse — two peers reporting the same signal).
</p>
<div class="topo-stage fe-stage">
<div class="topo-toolbar">
<div class="topo-layouts" role="tablist">
<button type="button" class="topo-layout is-active" data-layout="force" title="organic force-directed">Force</button>
<button type="button" class="topo-layout" data-layout="hier" title="self on top, direct peers in a row, transitives below">Hierarchical</button>
<button type="button" class="topo-layout" data-layout="radial" title="self at center, peers on a ring, transitives outside">Radial</button>
</div>
<label class="topo-toggle"><input type="checkbox" id="fn-flow" checked> signal flow</label>
<span class="topo-legend"><span class="lg-swatch fn-lg-self"></span>self</span>
<span class="topo-legend"><span class="lg-swatch fn-lg-trusted"></span>trusted</span>
<span class="topo-legend"><span class="lg-swatch fn-lg-unknown"></span>transitive</span>
<button type="button" id="fn-reset" class="btn">re-settle</button>
<span class="topo-hint">drag · scroll to zoom · click a peer to walk to its explorer</span>
</div>
<svg id="federation-network-graph" preserveAspectRatio="xMidYMid meet"></svg>
<div id="fn-tooltip" class="fn-tooltip"></div>
<div id="fn-loading" class="page-intro" style="text-align:center; margin-top:10px;">loading federation network…</div>
<div id="fn-error" class="gate-error" style="display:none;"></div>
</div>
<div id="fe-walk" class="fe-walk">
<p class="fe-walk-empty">Click any peer in the graph above to inspect it and walk to its federation view.</p>
</div>
</section>
<section class="panel fe-vouches-panel">
<div class="panel-head">
<h2>Who vouches for this node</h2>
<span class="count"><span id="fe-vouches-in-count"></span> inbound</span>
</div>
<p class="page-intro">
Each entry is a signed vouch naming this node as target, issued by a peer we currently trust.
Click a fingerprint to highlight that peer in the graph above.
</p>
<ul id="fe-vouches-in-list" class="fe-vouches-in-list">
<li class="fe-vouches-in-empty">no inbound vouches yet</li>
</ul>
</section>
</main>
<footer class="footer fe-footer">
psyc · defensive CTI · transparent by design ·
powered by <a href="https://neuronetz.ai" target="_blank" rel="noopener">neuronetz.ai</a>
</footer>
<script>
// The walk-to-peer param ("?peer=domain") tells the JS to focus that peer
// in the graph as soon as the data lands.
// Use tojson so values are honest JS literals — does not depend on the
// subtle HTML-entity-vs-script-context parser rules. Empty values become
// null/"" cleanly. Reach the values via PSYC_EXPLORE.* in the script.
window.PSYC_EXPLORE = {
selfFingerprint: {{ fingerprint | tojson }},
selfDomain: {{ (domain or "") | tojson }},
focusPeer: {{ (peer or "") | tojson }}
};
</script>
<script src="/static/federation_explore.js" defer></script>
</body>
</html>

View File

@@ -0,0 +1,162 @@
{% extends "base.html" %}
{% block title %}psyc — operations cockpit{% endblock %}
{% block content %}
<section class="hero">
<div class="hero-text">
<h1 class="hero-title">Defensive CTI in motion</h1>
<p class="hero-sub">What psyc has seen and done — at a glance.</p>
<p class="hero-meta"><a class="hero-explore" href="/federation/explore">Federation Explorer →</a> <span class="hero-explore-sub">public · auditable trust network</span></p>
</div>
<a class="hero-cta" href="/cases">All cases →</a>
</section>
<section class="kpis">
<a class="kpi" href="/cases">
<div class="kpi-num">{{ kpis.cases }}</div>
<div class="kpi-label">cases tracked</div>
</a>
<a class="kpi" href="/lookup">
<div class="kpi-num">{{ kpis.iocs }}</div>
<div class="kpi-label">IOCs indexed</div>
</a>
<div class="kpi">
<div class="kpi-num">+{{ kpis.new_24h }}</div>
<div class="kpi-label">new in 24 h</div>
</div>
<div class="kpi kpi-warn">
<div class="kpi-num">{{ kpis.high_total }}</div>
<div class="kpi-label">high / critical</div>
</div>
<a class="kpi kpi-accent" href="/response">
<div class="kpi-num">⚡ {{ kpis.enforcements_24h }}</div>
<div class="kpi-label">enforced 24 h</div>
</a>
<a class="kpi" href="/ledger">
<div class="kpi-num">{{ kpis.ledger_total }}</div>
<div class="kpi-label">ledger entries</div>
</a>
</section>
{% if featured %}
<section class="featured-section">
<header class="featured-section-head sev-{{ featured.classification.severity.value }}">
<div class="featured-section-title">
<span class="featured-bracket"></span>
<span class="featured-section-label">Featured Threat</span>
<span class="featured-section-pulse"></span>
</div>
<div class="featured-section-meta">
<span class="sev-badge">{{ featured.classification.severity.value if featured.classification.severity else '—' }}</span>
<span class="tlp-badge tlp-{{ featured.classification.tlp.value }}">{{ featured.classification.tlp.value }}</span>
<span class="muted">{{ featured.source_metadata.feed or 'unknown' }} · {{ featured.ingested_at.strftime('%Y-%m-%d %H:%M') }}</span>
</div>
</header>
<a class="featured sev-{{ featured.classification.severity.value }}" href="/cases/{{ featured.case_id }}">
<div class="featured-hero">
<div class="featured-grid" aria-hidden="true"></div>
<div class="featured-particles" aria-hidden="true">{{ featured_hero|safe }}</div>
<svg class="featured-radar" viewBox="0 0 100 100" aria-hidden="true">
<defs>
<linearGradient id="radar-sweep" x1="0" y1="50" x2="50" y2="50" gradientUnits="userSpaceOnUse">
<stop offset="0%" stop-color="currentColor" stop-opacity="0"/>
<stop offset="100%" stop-color="currentColor" stop-opacity="0.55"/>
</linearGradient>
</defs>
<circle cx="50" cy="50" r="46" fill="none" stroke="currentColor" stroke-opacity="0.35" stroke-width="0.6"/>
<circle cx="50" cy="50" r="32" fill="none" stroke="currentColor" stroke-opacity="0.25" stroke-width="0.5"/>
<circle cx="50" cy="50" r="18" fill="none" stroke="currentColor" stroke-opacity="0.18" stroke-width="0.4"/>
<line x1="50" y1="4" x2="50" y2="96" stroke="currentColor" stroke-opacity="0.10" stroke-width="0.4"/>
<line x1="4" y1="50" x2="96" y2="50" stroke="currentColor" stroke-opacity="0.10" stroke-width="0.4"/>
<g class="radar-sweep-arm">
<path d="M50,50 L50,4 A46,46 0 0,1 96,50 Z" fill="url(#radar-sweep)"/>
</g>
<circle cx="50" cy="50" r="2.5" fill="currentColor" fill-opacity="0.85"/>
</svg>
<div class="featured-overlay">
<h3 class="featured-title">{{ featured.summary }}</h3>
<div class="featured-stats">
{% if featured.classification.incident_type %}
<span class="stat-chip stat-incident">⌧ {{ featured.classification.incident_type.value }}</span>
{% endif %}
{% set obs = featured.observables %}
{% set total_iocs = (obs.urls|length) + (obs.domains|length) + (obs.ips|length) + (obs.hashes|length) + (obs.cves|length) %}
{% if total_iocs %}
<span class="stat-chip stat-iocs" title="URLs/Domains/IPs/Hashes/CVEs">⛛ {{ total_iocs }} IOCs · {{ obs.urls|length }}U/{{ obs.domains|length }}D/{{ obs.ips|length }}I/{{ obs.hashes|length }}H/{{ obs.cves|length }}C</span>
{% endif %}
{% if featured.victim.country %}
<span class="stat-chip stat-country">📍 {{ featured.victim.country }}</span>
{% endif %}
{% if featured.confidence and featured.confidence.level %}
<span class="stat-chip stat-confidence">⊞ {{ featured.confidence.level }} confidence</span>
{% endif %}
{% if featured.source_metadata.malware %}
<span class="stat-chip stat-family">⌬ {{ featured.source_metadata.malware }}</span>
{% endif %}
</div>
<div class="featured-cta">Open case <span class="featured-arrow"></span></div>
</div>
</div>
</a>
</section>
{% endif %}
<div class="home-grid">
<section class="panel home-news">
<div class="panel-head">
<h2>Recent activity</h2>
<span class="count">{{ total_items }} items</span>
</div>
<p class="page-intro">Live feed of what psyc has detected and what it has done about it.</p>
{% if not buckets %}
<p class="empty">Nothing recent yet — start with <code>psyc fetch-all</code>.</p>
{% endif %}
{% for b in buckets %}
<h3 class="bucket-head">{{ b.label }} <span class="bucket-count">{{ b.items|length }}</span></h3>
<ol class="news-list">
{% for i in b.items %}
{% set has_link = i.case_id %}
<li class="news-item news-kind-{{ i.kind }}{% if i.severity %} sev-{{ i.severity }}{% endif %}{% if has_link %} is-link{% endif %}">
{% if has_link %}<a class="news-cardlink" href="/cases/{{ i.case_id }}" aria-label="Open {{ i.case_id }}"></a>{% endif %}
<div class="news-icon">
{% if i.kind == 'case' and i.case_id and case_glyphs.get(i.case_id) %}
{{ case_glyphs[i.case_id]|safe }}
{% else %}
<span class="news-icon-glyph">{{ i.icon }}</span>
{% endif %}
</div>
<div class="news-body">
<div class="news-head">
<span class="news-headline">{{ i.headline }}</span>
{% if i.severity %}<span class="sev-badge">{{ i.severity }}</span>{% endif %}
</div>
<div class="news-sub">{{ i.body }}</div>
<div class="news-meta">
<time>{{ i.timestamp.strftime('%H:%M' if b.label == 'Today' else '%Y-%m-%d %H:%M') }}</time>
{% if i.case_id %} · <span class="news-case-id">{{ i.case_id }}</span>{% endif %}
</div>
</div>
{% if has_link %}<span class="news-arrow" aria-hidden="true"></span>{% endif %}
</li>
{% endfor %}
</ol>
{% endfor %}
</section>
<aside class="panel home-side">
<div class="panel-head"><h2>Feed health</h2></div>
<p class="page-intro">Where psyc's data is coming from.</p>
<ul class="feed-health">
{% for f in feeds %}
<li class="feed-row">
<span class="feed-name">{{ f.feed }}</span>
<span class="feed-count">{{ f.count }}</span>
<span class="feed-time">{{ f.latest.strftime('%m-%d %H:%M') if f.latest else '—' }}</span>
</li>
{% endfor %}
</ul>
</aside>
</div>
{% endblock %}

View File

@@ -74,7 +74,7 @@
{% set rule_sev = case.classification.severity.value if case.classification.severity else '' %}
{% set agrees = b.model_answer == rule_sev %}
<p class="bot-model {{ 'model-agree' if agrees else 'model-differ' }}">
<span class="model-chip">psyc-v4 · live model</span>
<span class="model-chip">{{ model_label }} · live model</span>
severity: <strong>{{ b.model_answer | upper }}</strong>
{% if agrees %}<span class="model-verdict">✓ agrees with the rule</span>
{% else %}<span class="model-verdict">✗ differs — rule said {{ rule_sev | upper }}</span>{% endif %}

View File

@@ -0,0 +1,66 @@
{% extends "base.html" %}
{% block title %}Lookup — psyc{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Indicator Lookup</h1>
<span class="count">{{ total_iocs }} indicators indexed</span>
</div>
<p class="page-intro">Paste any indicator — IP, domain, URL, file hash, or CVE — and psyc tells you whether it's known-bad across the whole case corpus, which feed flagged it, and at what severity. This is the "is this thing dangerous?" desk check.</p>
<details class="page-help">
<summary>how to use this view</summary>
<div class="help-body">
<p><b>How to use.</b> Type or paste an indicator and hit Look up. A green banner means it's clean (not in the corpus); a red banner means it matched known threat intel — open the case to see the full context.</p>
<p><b>What you're seeing.</b> Matches come from the IOC index built across all {{ total_iocs }} indicators in the corpus. Lookup is case- and format-insensitive (EVIL.COM = evil.com).</p>
<p><b>Why it matters.</b> A defender investigating an alert needs a fast verdict on a raw indicator — and a way to push the whole known-bad set into a firewall or DNS sinkhole (see Blocklist export below).</p>
</div>
</details>
<form method="get" action="/lookup" class="lookup-form">
<input type="text" name="q" value="{{ query }}" placeholder="1.2.3.4 · evil.com · http://… · &lt;sha256&gt; · CVE-2024-3721" class="lookup-input" autofocus>
<button type="submit" class="btn btn-approve">Look up</button>
</form>
{% if searched %}
{% if matches %}
<div class="verdict verdict-bad"><strong>{{ query }}</strong> is KNOWN-BAD — {{ matches|length }} match(es) in the corpus</div>
<table class="ledger">
<thead>
<tr><th>Type</th><th>Case</th><th>Feed</th><th>Severity</th><th>First seen</th></tr>
</thead>
<tbody>
{% for m in matches %}
<tr class="ledger-row sev-{{ m.severity or 'none' }}">
<td>{{ m.ioc_type }}</td>
<td class="lg-case"><a href="/cases/{{ m.case_id }}">{{ m.case_id }}</a></td>
<td class="lg-dest">{{ m.feed or '—' }}</td>
<td>{% if m.severity %}<span class="sev-badge">{{ m.severity }}</span>{% else %}—{% endif %}</td>
<td class="lg-ts">{{ (m.first_seen or '')[:10] }}</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<div class="verdict verdict-clean"><strong>{{ query }}</strong> is not in the corpus — no known-bad match</div>
{% endif %}
{% endif %}
</section>
<section class="panel">
<div class="panel-head"><h2>Blocklist export</h2></div>
<p class="page-intro">Download the deduplicated set of known-bad indicators of one type as plain text — ready to paste into a firewall denylist, DNS sinkhole, or SIEM watchlist.</p>
<table class="ledger">
<thead><tr><th>Type</th><th>Count</th><th>Download (all)</th><th>Download (high+)</th></tr></thead>
<tbody>
{% for t, n in counts.items() %}
<tr class="ledger-row">
<td>{{ t }}</td>
<td>{{ n }}</td>
<td><a href="/export/blocklist?type={{ t }}" target="_blank">{{ t }} blocklist ▾</a></td>
<td><a href="/export/blocklist?type={{ t }}&min_severity=high" target="_blank">{{ t }} (high+) ▾</a></td>
</tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -0,0 +1,73 @@
{% extends "base.html" %}
{% block title %}Approval Queue — psyc{% endblock %}
{% block content %}
<section class="panel">
<div class="panel-head">
<h1>Submission Approval Queue</h1>
<span class="count">{{ counts.pending }} pending · {{ counts.approved }} approved · {{ counts.rejected }} rejected</span>
</div>
<p class="page-intro">Nothing leaves psyc to an authority destination without a human signing off. Routing builds the payload, freezes it here, and waits. You approve — Courier dispatches and the Ledger records. You reject — nothing leaves, and the rejection is recorded too.</p>
<details class="page-help">
<summary>how to use this view</summary>
<div class="help-body">
<p><b>How to use.</b> Each row is a payload waiting on you. Click the case to inspect it before deciding. Approve sends it now; Reject blocks it forever (the case can still be re-submitted later from CLI if appropriate).</p>
<p><b>What you're seeing.</b> Pending submissions for destinations marked <code>requires_approval=True</code> — CERT-Bund by default, or <em>all</em> destinations when <code>PSYC_REQUIRE_APPROVAL=1</code>.</p>
<p><b>Why it matters.</b> The dossier mandates a human gate before evidence reaches real authority systems. This is that gate. The frozen payload guarantees the reviewer approves exactly what gets sent — not a re-derived version that might have drifted.</p>
</div>
</details>
<div class="queue-tabs">
<a class="queue-tab{% if current_status == 'pending' %} is-active{% endif %}" href="/queue?status=pending">pending ({{ counts.pending }})</a>
<a class="queue-tab{% if current_status == 'approved' %} is-active{% endif %}" href="/queue?status=approved">approved ({{ counts.approved }})</a>
<a class="queue-tab{% if current_status == 'rejected' %} is-active{% endif %}" href="/queue?status=rejected">rejected ({{ counts.rejected }})</a>
<a class="queue-tab{% if current_status == 'all' %} is-active{% endif %}" href="/queue?status=all">all</a>
</div>
{% if not rows %}
<p class="empty">Queue is clear. Either nothing is pending, or no destination on the current cases requires approval. Set <code>PSYC_REQUIRE_APPROVAL=1</code> to force every routable submission through this gate.</p>
{% else %}
<table class="ledger">
<thead>
<tr>
<th>#</th>
<th>Created</th>
<th>Case</th>
<th>Destination</th>
<th>Payload</th>
<th>TLP</th>
<th>Hash</th>
<th>Status</th>
<th>Action</th>
</tr>
</thead>
<tbody>
{% for p in rows %}
<tr class="ledger-row{% if p.status.value == 'rejected' %} is-rejected{% elif p.status.value == 'approved' %} is-actioned{% endif %}">
<td>#{{ p.id }}</td>
<td class="lg-ts">{{ p.created_at.strftime('%Y-%m-%d %H:%M:%S') }}</td>
<td class="lg-case"><a href="/cases/{{ p.case_id }}">{{ p.case_id }}</a></td>
<td class="lg-dest">{{ p.destination_name }}</td>
<td>{{ p.payload_kind }}</td>
<td><span class="tlp-badge tlp-{{ p.tlp.value }}">{{ p.tlp.value }}</span></td>
<td class="lg-hash">{{ p.payload_hash[:12] }}…</td>
<td><span class="outcome-badge outcome-{{ 'submitted' if p.status.value == 'approved' else ('rejected' if p.status.value == 'rejected' else 'pending_approval') }}">{{ p.status.value }}</span></td>
<td>
{% if p.status.value == 'pending' %}
<form method="post" action="/queue/approve/{{ p.id }}" class="queue-action">
<button type="submit" class="btn btn-approve">approve</button>
</form>
<form method="post" action="/queue/reject/{{ p.id }}" class="queue-action">
<input type="text" name="reason" placeholder="reason (optional)" class="reject-reason">
<button type="submit" class="btn btn-reject">reject</button>
</form>
{% else %}
<span class="lg-sub">{{ p.reviewer or '—' }}{% if p.reason %} · {{ p.reason }}{% endif %}</span>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
{% endif %}
</section>
{% endblock %}

View File

@@ -0,0 +1,81 @@
{% extends "base.html" %}
{% block title %}Response — psyc{% endblock %}
{% block content %}
{% if fired %}
<div class="disco" id="disco">
<div class="disco-strobe"></div>
<div class="disco-core">
<div class="disco-bolt"></div>
<div class="disco-headline">ENFORCED</div>
<div class="disco-sub">action #{{ fired }} · {{ fired_kind }} pushed to the perimeter</div>
<div class="disco-iocs">
{% for i in range(8) %}<span class="ioc-fly" style="--d: {{ i * 0.08 }}s; --x: {{ (i - 4) * 60 }}px;"></span>{% endfor %}
</div>
</div>
</div>
<script>
setTimeout(function () {
var d = document.getElementById('disco');
if (d) { d.classList.add('disco-out'); setTimeout(function(){ d.remove(); }, 700); }
}, 2600);
</script>
{% endif %}
<section class="panel">
<div class="panel-head">
<h1>Response Actions</h1>
<span class="count">{{ counts.proposed }} proposed · {{ counts.executed }} enforced · {{ counts.rejected }} declined{% if counts.failed %} · {{ counts.failed }} failed{% endif %}</span>
</div>
<p class="page-intro">When a high-severity case lands, psyc proposes what to <em>do</em> about it — alert the SOC, push its IOCs to the perimeter firewall + DNS. Nothing fires on its own: you approve, psyc enforces, the ledger records it. Detection that acts, with a human on the trigger.</p>
<details class="page-help">
<summary>how to use this view</summary>
<div class="help-body">
<p><b>How to use.</b> Each proposed action is one defensive move. Hit <b>⚡ Enforce</b> to fire it (and enjoy the disco), or Decline to drop it. Both decisions are logged to the immutable ledger.</p>
<p><b>What you're seeing.</b> Actions generated by Respondline for HIGH/CRITICAL cases. The frozen payload is exactly what gets pushed to the enforcement sink on approval.</p>
<p><b>Why it matters.</b> Closing the loop — intel → decision → enforcement → audit — is what separates a threat <em>viewer</em> from a threat <em>response</em> platform. The human gate keeps automation safe.</p>
</div>
</details>
<div class="queue-tabs">
<a class="queue-tab{% if current_status == 'proposed' %} is-active{% endif %}" href="/response?status=proposed">proposed ({{ counts.proposed }})</a>
<a class="queue-tab{% if current_status == 'executed' %} is-active{% endif %}" href="/response?status=executed">enforced ({{ counts.executed }})</a>
<a class="queue-tab{% if current_status == 'rejected' %} is-active{% endif %}" href="/response?status=rejected">declined ({{ counts.rejected }})</a>
<a class="queue-tab{% if current_status == 'all' %} is-active{% endif %}" href="/response?status=all">all</a>
</div>
{% if not actions %}
<p class="empty">No actions here. Propose some with <code>psyc respond &lt;case_id&gt;</code> on a HIGH/CRITICAL case, or run <code>psyc demo</code>.</p>
{% else %}
<table class="ledger">
<thead>
<tr><th>#</th><th>Type</th><th>Case</th><th>Sev</th><th>What it does</th><th>Status</th><th>Action</th></tr>
</thead>
<tbody>
{% for a in actions %}
<tr class="ledger-row sev-{{ a.severity or 'none' }}{% if a.status.value == 'rejected' %} is-rejected{% elif a.status.value == 'executed' %} is-actioned{% endif %}">
<td>#{{ a.id }}</td>
<td><span class="act-type act-{{ a.action_type.value }}">{{ a.action_type.value }}</span></td>
<td class="lg-case"><a href="/cases/{{ a.case_id }}">{{ a.case_id }}</a></td>
<td><span class="sev-badge">{{ a.severity or '—' }}</span></td>
<td>{{ a.summary }}</td>
<td><span class="outcome-badge outcome-{{ 'actioned' if a.status.value == 'executed' else ('rejected' if a.status.value == 'rejected' else ('failed' if a.status.value == 'failed' else 'pending_approval')) }}">{{ a.status.value }}</span></td>
<td>
{% if a.status.value == 'proposed' %}
<form method="post" action="/response/approve/{{ a.id }}" class="queue-action">
<button type="submit" class="btn btn-enforce">⚡ Enforce</button>
</form>
<form method="post" action="/response/reject/{{ a.id }}" class="queue-action">
<button type="submit" class="btn btn-reject">decline</button>
</form>
{% else %}
<span class="lg-sub">{{ a.approver or '—' }}</span>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
{% endif %}
</section>
{% endblock %}

View File

@@ -17,11 +17,13 @@ from sqlalchemy import (
Table,
Text,
create_engine,
event,
func,
insert,
select,
)
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy.pool import NullPool
from psyc import DATA_DIR, log
from psyc.models import Case
@@ -64,16 +66,176 @@ ledger = Table(
Index("ledger_case_idx", ledger.c.case_id)
Index("ledger_time_idx", ledger.c.timestamp.desc())
pending = Table(
"pending_submissions", _metadata,
Column("id", Integer, primary_key=True, autoincrement=True),
Column("case_id", String, nullable=False),
Column("destination_name", String, nullable=False),
Column("payload_kind", String, nullable=False),
Column("payload_hash", String, nullable=False),
Column("payload_json", Text, nullable=False),
Column("tlp", String, nullable=False),
Column("created_at", String, nullable=False),
Column("status", String, nullable=False),
Column("reviewer", String, nullable=True),
Column("reviewed_at", String, nullable=True),
Column("reason", String, nullable=True),
)
Index("pending_status_idx", pending.c.status)
Index("pending_case_idx", pending.c.case_id)
response_actions = Table(
"response_actions", _metadata,
Column("id", Integer, primary_key=True, autoincrement=True),
Column("case_id", String, nullable=False),
Column("action_type", String, nullable=False),
Column("target", String, nullable=False),
Column("summary", Text, nullable=False),
Column("payload_json", Text, nullable=False),
Column("severity", String, nullable=True),
Column("status", String, nullable=False),
Column("created_at", String, nullable=False),
Column("approver", String, nullable=True),
Column("executed_at", String, nullable=True),
Column("detail", Text, nullable=True),
)
Index("actions_status_idx", response_actions.c.status)
Index("actions_case_idx", response_actions.c.case_id)
iocs = Table(
"iocs", _metadata,
Column("id", Integer, primary_key=True, autoincrement=True),
Column("value", String, nullable=False), # normalized indicator
Column("ioc_type", String, nullable=False), # url | domain | ip | hash | cve
Column("case_id", String, nullable=False),
Column("feed", String, nullable=True),
Column("severity", String, nullable=True),
Column("first_seen", String, nullable=True),
)
Index("iocs_value_idx", iocs.c.value)
Index("iocs_type_idx", iocs.c.ioc_type)
Index("iocs_case_idx", iocs.c.case_id)
pulse_pipelines = Table(
"pulse_pipelines", _metadata,
Column("name", String, primary_key=True),
Column("title", String, nullable=False),
Column("description", Text, nullable=False, default=""),
Column("mode", String, nullable=False), # manual | auto-propose | auto-execute
Column("cadence_seconds", Integer, nullable=False),
Column("enabled", Boolean, nullable=False, default=True),
Column("last_fired", String, nullable=True), # ISO timestamp or NULL
Column("next_fire", String, nullable=True), # ISO timestamp or NULL
Column("last_result", Text, nullable=False, default=""),
Column("last_outcome", String, nullable=False, default=""), # ok | err | skipped | ""
)
pulse_settings = Table(
"pulse_settings", _metadata,
Column("key", String, primary_key=True),
Column("value", String, nullable=False),
)
pulse_audit = Table(
"pulse_audit", _metadata,
Column("id", Integer, primary_key=True, autoincrement=True),
Column("pipeline", String, nullable=False), # 'respond' | 'fetch' | ...
Column("action", String, nullable=False), # 'auto-fire' | 'skipped' | 'error'
Column("action_id", Integer, nullable=True), # response_actions.id when relevant
Column("case_id", String, nullable=True),
Column("detail", Text, nullable=True),
Column("timestamp", String, nullable=False), # ISO
)
Index("pulse_audit_pipeline_idx", pulse_audit.c.pipeline, pulse_audit.c.timestamp.desc())
Index("pulse_audit_action_id_idx", pulse_audit.c.action_id)
peers = Table(
"peers", _metadata,
Column("domain", String, primary_key=True),
Column("fingerprint", String, nullable=False),
Column("pubkey_pem", Text, nullable=False),
Column("status", String, nullable=False), # unknown | trusted | blocked
Column("discovered_at", String, nullable=False),
Column("last_seen", String, nullable=True),
Column("notes", Text, nullable=True),
)
Index("peers_fp_idx", peers.c.fingerprint)
Index("peers_status_idx", peers.c.status)
federation_signals = Table(
"federation_signals", _metadata,
Column("id", Integer, primary_key=True, autoincrement=True),
Column("peer_fingerprint", String, nullable=False),
Column("signal_type", String, nullable=False), # case | ioc
Column("signal_id", String, nullable=False), # case_id or ioc value
Column("signal_hash", String, nullable=False), # sha256 of canonical record
Column("received_at", String, nullable=False),
Column("raw_json", Text, nullable=False),
)
Index("federation_signals_hash_idx", federation_signals.c.signal_hash)
Index("federation_signals_peer_idx", federation_signals.c.peer_fingerprint)
Index("federation_signals_received_idx", federation_signals.c.received_at.desc())
# Web-of-trust vouches — voucher signs an attestation that target is OK to listen to.
# Quorum is reached when enough distinct trusted vouchers sign for the same target.
vouches = Table(
"vouches", _metadata,
Column("id", Integer, primary_key=True, autoincrement=True),
Column("voucher_fingerprint", String, nullable=False),
Column("target_fingerprint", String, nullable=False),
Column("issued_at", String, nullable=False),
Column("expires_at", String, nullable=True),
Column("signature", Text, nullable=False), # base64 ed25519 sig
)
Index("vouches_unique_idx", vouches.c.voucher_fingerprint, vouches.c.target_fingerprint, unique=True)
Index("vouches_target_idx", vouches.c.target_fingerprint)
# Transparency log — append-only signed hash chain over every signal we receive.
# Each entry references the previous entry's hash; tampering with any row breaks
# verify_chain on every subsequent row.
translog = Table(
"translog", _metadata,
Column("id", Integer, primary_key=True, autoincrement=True),
Column("prev_hash", String, nullable=False),
Column("entry_type", String, nullable=False), # signal | vouch | config
Column("entry_data", Text, nullable=False), # canonical JSON of payload
Column("timestamp", String, nullable=False),
Column("entry_hash", String, nullable=False),
)
Index("translog_hash_idx", translog.c.entry_hash)
Index("translog_time_idx", translog.c.timestamp.desc())
_log = log.get(__name__)
_engine: Optional[Engine] = None
def engine(db_path: Path = DB_PATH) -> Engine:
"""Lazy-init the SQLite engine.
Uses NullPool — SQLite doesn't benefit from connection pooling (it's a
file, opens are cheap) and the default QueuePool starved the classify +
federation + cockpit-request workers under real load. WAL journal mode
+ a 30s busy timeout let readers and a writer share the file safely.
"""
global _engine
if _engine is None:
db_path.parent.mkdir(parents=True, exist_ok=True)
_engine = create_engine(f"sqlite:///{db_path}", future=True)
_engine = create_engine(
f"sqlite:///{db_path}",
future=True,
poolclass=NullPool,
connect_args={"check_same_thread": False, "timeout": 30},
)
@event.listens_for(_engine, "connect")
def _sqlite_pragmas(dbapi_conn, _connection_record): # noqa: D401
cur = dbapi_conn.cursor()
cur.execute("PRAGMA journal_mode=WAL")
cur.execute("PRAGMA synchronous=NORMAL")
cur.execute("PRAGMA busy_timeout=30000")
cur.close()
return _engine
@@ -133,3 +295,327 @@ def case_count(db_path: Path = DB_PATH) -> int:
stmt = select(func.count()).select_from(cases)
with engine(db_path).connect() as conn:
return conn.execute(stmt).scalar_one()
# ---------- IOC index ----------------------------------------------------
def replace_iocs(rows: List[dict], db_path: Path = DB_PATH) -> int:
"""Rebuild the IOC index: clear it, then bulk-insert rows. Returns count."""
with engine(db_path).begin() as conn:
conn.execute(iocs.delete())
if rows:
conn.execute(iocs.insert(), rows)
return len(rows)
def find_iocs(value: str, db_path: Path = DB_PATH) -> List[dict]:
"""Exact-match lookup of one normalized indicator. Returns matching index rows."""
stmt = select(iocs).where(iocs.c.value == value).order_by(iocs.c.first_seen.desc())
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def iocs_by_type(ioc_type: str, db_path: Path = DB_PATH) -> List[dict]:
"""All index rows of one type, newest first — caller filters/dedupes."""
stmt = select(iocs).where(iocs.c.ioc_type == ioc_type).order_by(iocs.c.first_seen.desc())
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def ioc_count(db_path: Path = DB_PATH) -> int:
stmt = select(func.count()).select_from(iocs)
with engine(db_path).connect() as conn:
return conn.execute(stmt).scalar_one()
# ---------- pulse scheduler ----------------------------------------------
def get_pulse_state(db_path: Path = DB_PATH) -> List[dict]:
"""Every registered pipeline, ordered by name."""
stmt = select(pulse_pipelines).order_by(pulse_pipelines.c.name)
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def upsert_pulse_pipeline(row: dict, db_path: Path = DB_PATH) -> None:
"""Insert or update one pipeline by name."""
stmt = sqlite_insert(pulse_pipelines).values(**row)
stmt = stmt.on_conflict_do_update(
index_elements=[pulse_pipelines.c.name],
set_=dict(
title=stmt.excluded.title,
description=stmt.excluded.description,
mode=stmt.excluded.mode,
cadence_seconds=stmt.excluded.cadence_seconds,
enabled=stmt.excluded.enabled,
last_fired=stmt.excluded.last_fired,
next_fire=stmt.excluded.next_fire,
last_result=stmt.excluded.last_result,
last_outcome=stmt.excluded.last_outcome,
),
)
with engine(db_path).begin() as conn:
conn.execute(stmt)
def kill_switch_get(db_path: Path = DB_PATH) -> bool:
stmt = select(pulse_settings.c.value).where(pulse_settings.c.key == "kill_switch")
with engine(db_path).connect() as conn:
row = conn.execute(stmt).fetchone()
if row is None:
return False
return str(row.value) == "1"
def kill_switch_set(armed: bool, db_path: Path = DB_PATH) -> None:
value = "1" if armed else "0"
stmt = sqlite_insert(pulse_settings).values(key="kill_switch", value=value)
stmt = stmt.on_conflict_do_update(
index_elements=[pulse_settings.c.key],
set_=dict(value=stmt.excluded.value),
)
with engine(db_path).begin() as conn:
conn.execute(stmt)
def pulse_setting_get(key: str, db_path: Path = DB_PATH) -> Optional[str]:
"""Fetch one row from pulse_settings by key; returns None if absent."""
stmt = select(pulse_settings.c.value).where(pulse_settings.c.key == key)
with engine(db_path).connect() as conn:
row = conn.execute(stmt).fetchone()
return str(row.value) if row else None
def pulse_setting_set(key: str, value: str, db_path: Path = DB_PATH) -> None:
"""Upsert one (key, value) into pulse_settings."""
stmt = sqlite_insert(pulse_settings).values(key=key, value=value)
stmt = stmt.on_conflict_do_update(
index_elements=[pulse_settings.c.key],
set_=dict(value=stmt.excluded.value),
)
with engine(db_path).begin() as conn:
conn.execute(stmt)
# ---------- pulse audit trail --------------------------------------------
def pulse_audit_record(row: dict, db_path: Path = DB_PATH) -> int:
"""Append one pulse_audit row. Returns its id.
`row` must include 'pipeline', 'action', 'timestamp'. action_id, case_id,
detail are optional. Caller controls timestamp so tests can pin it.
"""
stmt = insert(pulse_audit).values(**row)
with engine(db_path).begin() as conn:
res = conn.execute(stmt)
return int(res.inserted_primary_key[0])
def pulse_audit_recent(pipeline: str, limit: int = 25, db_path: Path = DB_PATH) -> List[dict]:
"""Most-recent audit rows for one pipeline (newest first)."""
stmt = (
select(pulse_audit)
.where(pulse_audit.c.pipeline == pipeline)
.order_by(pulse_audit.c.timestamp.desc())
.limit(limit)
)
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def pulse_audit_count_since(
pipeline: str, action: str, since_iso: str, db_path: Path = DB_PATH
) -> int:
"""Count audit rows for (pipeline, action) at or after `since_iso`."""
stmt = (
select(func.count())
.select_from(pulse_audit)
.where(pulse_audit.c.pipeline == pipeline)
.where(pulse_audit.c.action == action)
.where(pulse_audit.c.timestamp >= since_iso)
)
with engine(db_path).connect() as conn:
return int(conn.execute(stmt).scalar_one())
# ---------- federation: peers + signal buffer ----------------------------
def upsert_peer(row: dict, db_path: Path = DB_PATH) -> None:
"""Insert-or-update a peer by domain. `row` must include `domain`."""
stmt = sqlite_insert(peers).values(**row)
update_cols = {k: stmt.excluded[k] for k in row if k != "domain"}
stmt = stmt.on_conflict_do_update(index_elements=[peers.c.domain], set_=update_cols)
with engine(db_path).begin() as conn:
conn.execute(stmt)
def list_peers(db_path: Path = DB_PATH) -> List[dict]:
stmt = select(peers).order_by(peers.c.discovered_at.desc())
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def get_peer(domain: str, db_path: Path = DB_PATH) -> Optional[dict]:
stmt = select(peers).where(peers.c.domain == domain)
with engine(db_path).connect() as conn:
row = conn.execute(stmt).fetchone()
return dict(row._mapping) if row else None
def set_peer_status(domain: str, status: str, db_path: Path = DB_PATH) -> None:
from sqlalchemy import update as sa_update
stmt = sa_update(peers).where(peers.c.domain == domain).values(status=status)
with engine(db_path).begin() as conn:
conn.execute(stmt)
def remove_peer(domain: str, db_path: Path = DB_PATH) -> None:
stmt = peers.delete().where(peers.c.domain == domain)
with engine(db_path).begin() as conn:
conn.execute(stmt)
def record_signal(row: dict, db_path: Path = DB_PATH) -> int:
"""Append one federation signal. Returns the inserted row id."""
stmt = insert(federation_signals).values(**row)
with engine(db_path).begin() as conn:
res = conn.execute(stmt)
return int(res.inserted_primary_key[0])
def signals_for_hash(signal_hash: str, db_path: Path = DB_PATH) -> List[dict]:
"""All recorded signals matching `signal_hash` — quorum-lookup primitive."""
stmt = select(federation_signals).where(federation_signals.c.signal_hash == signal_hash)
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def recent_signals(limit: int = 200, db_path: Path = DB_PATH) -> List[dict]:
stmt = select(federation_signals).order_by(federation_signals.c.received_at.desc()).limit(limit)
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
# ---------- federation: pulse_settings get/set (shared scratch kv) -------
def setting_get(key: str, db_path: Path = DB_PATH) -> Optional[str]:
"""Read one pulse_settings value by key. Returns None if absent."""
stmt = select(pulse_settings.c.value).where(pulse_settings.c.key == key)
with engine(db_path).connect() as conn:
row = conn.execute(stmt).fetchone()
return None if row is None else str(row.value)
def setting_set(key: str, value: str, db_path: Path = DB_PATH) -> None:
"""Upsert one pulse_settings entry."""
stmt = sqlite_insert(pulse_settings).values(key=key, value=value)
stmt = stmt.on_conflict_do_update(
index_elements=[pulse_settings.c.key],
set_=dict(value=stmt.excluded.value),
)
with engine(db_path).begin() as conn:
conn.execute(stmt)
# ---------- federation: vouches ------------------------------------------
def upsert_vouch(row: dict, db_path: Path = DB_PATH) -> None:
"""Insert-or-update one vouch. Unique on (voucher_fp, target_fp)."""
stmt = sqlite_insert(vouches).values(**row)
update_cols = {k: stmt.excluded[k] for k in row if k not in ("voucher_fingerprint", "target_fingerprint")}
stmt = stmt.on_conflict_do_update(
index_elements=[vouches.c.voucher_fingerprint, vouches.c.target_fingerprint],
set_=update_cols,
)
with engine(db_path).begin() as conn:
conn.execute(stmt)
def list_vouches(db_path: Path = DB_PATH) -> List[dict]:
stmt = select(vouches).order_by(vouches.c.issued_at.desc())
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def vouches_by_target(target_fingerprint: str, db_path: Path = DB_PATH) -> List[dict]:
stmt = select(vouches).where(vouches.c.target_fingerprint == target_fingerprint)
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def vouches_by_voucher(voucher_fingerprint: str, db_path: Path = DB_PATH) -> List[dict]:
stmt = select(vouches).where(vouches.c.voucher_fingerprint == voucher_fingerprint)
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def delete_vouch(voucher_fingerprint: str, target_fingerprint: str, db_path: Path = DB_PATH) -> None:
stmt = vouches.delete().where(
(vouches.c.voucher_fingerprint == voucher_fingerprint)
& (vouches.c.target_fingerprint == target_fingerprint)
)
with engine(db_path).begin() as conn:
conn.execute(stmt)
# ---------- transparency log ---------------------------------------------
def translog_append(row: dict, db_path: Path = DB_PATH) -> int:
"""Append one transparency-log entry. Returns inserted id."""
stmt = insert(translog).values(**row)
with engine(db_path).begin() as conn:
res = conn.execute(stmt)
return int(res.inserted_primary_key[0])
def translog_head(db_path: Path = DB_PATH) -> Optional[dict]:
"""Highest-id (latest) entry, or None if chain empty."""
stmt = select(translog).order_by(translog.c.id.desc()).limit(1)
with engine(db_path).connect() as conn:
row = conn.execute(stmt).fetchone()
return dict(row._mapping) if row else None
def translog_get(entry_id: int, db_path: Path = DB_PATH) -> Optional[dict]:
stmt = select(translog).where(translog.c.id == entry_id)
with engine(db_path).connect() as conn:
row = conn.execute(stmt).fetchone()
return dict(row._mapping) if row else None
def translog_after(entry_id: int, db_path: Path = DB_PATH) -> List[dict]:
"""All entries with id > entry_id, oldest first — for sync."""
stmt = select(translog).where(translog.c.id > entry_id).order_by(translog.c.id.asc())
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def translog_recent(limit: int = 100, db_path: Path = DB_PATH) -> List[dict]:
stmt = select(translog).order_by(translog.c.id.desc()).limit(limit)
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def translog_range(start: int = 0, end: Optional[int] = None, db_path: Path = DB_PATH) -> List[dict]:
"""All entries with start <= id (and id <= end if given), oldest first."""
cond = translog.c.id >= start
if end is not None:
cond = cond & (translog.c.id <= end)
stmt = select(translog).where(cond).order_by(translog.c.id.asc())
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]
def signals_for_case(case_id: str, db_path: Path = DB_PATH) -> List[dict]:
"""All federation signals attached to this case_id (signal_type='case').
Empty list means no peer has ever sent us this case → we generated it
ourselves and it counts as locally-sourced for auto-fire purposes.
"""
stmt = (
select(federation_signals)
.where(federation_signals.c.signal_type == "case")
.where(federation_signals.c.signal_id == case_id)
)
with engine(db_path).connect() as conn:
return [dict(r._mapping) for r in conn.execute(stmt).fetchall()]

View File

@@ -12,6 +12,16 @@ _FEED_INCIDENT = {
"urlhaus": IncidentType.MALWARE,
"feodo": IncidentType.BOTNET,
"cisa-kev": IncidentType.EXPLOIT,
"malware-bazaar": IncidentType.MALWARE,
"otx": IncidentType.MALWARE, # default; OTX pulses span many types
}
# ThreatFox carries its own type signal — map it instead of using a feed default.
_THREATFOX_THREAT_TYPE = {
"botnet_cc": IncidentType.BOTNET,
"payload_delivery": IncidentType.MALWARE,
"payload": IncidentType.MALWARE,
"phishing": IncidentType.PHISHING,
}
@@ -33,7 +43,11 @@ def classify(case: Case) -> Case:
def _classify_incident_type_and_tlp(case: Case) -> None:
if case.classification.incident_type is not None:
return
incident = _FEED_INCIDENT.get(case.source_metadata.get("feed", ""))
feed = case.source_metadata.get("feed", "")
if feed == "threatfox":
incident = _THREATFOX_THREAT_TYPE.get(case.source_metadata.get("threat_type", ""), IncidentType.MALWARE)
else:
incident = _FEED_INCIDENT.get(feed)
if incident is None and case.observables.urls:
incident = IncidentType.MALWARE # fallback for un-tagged feeds
if incident is None:

View File

@@ -1,18 +1,21 @@
"""Courier — payload building + HTTP submission to destination endpoints."""
"""Courier — payload building + HTTP submission, with optional approval queue."""
from __future__ import annotations
import hashlib
import json
import os
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
import httpx
from pydantic import BaseModel, Field
from sqlalchemy import select, update
from psyc import log
from psyc import db, log
from psyc.lines import ledger as ledger_line
from psyc.lines.route import BlockedRoute, Route, endpoint_for
from psyc.models import Case, Outcome, SealedPackage
from psyc.models import ApprovalStatus, Case, Outcome, PendingSubmission, SealedPackage, TLP
from psyc.result import Err, Ok, Result
@@ -122,8 +125,32 @@ def execute_blocked_routes(case: Case, blocked: List[BlockedRoute]) -> None:
)
def _force_approval() -> bool:
return os.environ.get("PSYC_REQUIRE_APPROVAL", "").lower() in ("1", "true", "yes")
def _enqueue_pending(case: Case, route: Route, payload: Dict[str, Any], payload_hash: str) -> int:
now = datetime.now(timezone.utc).isoformat()
stmt = db.pending.insert().values(
case_id=case.case_id,
destination_name=route.destination_name,
payload_kind=route.payload_kind,
payload_hash=payload_hash,
payload_json=json.dumps(payload, sort_keys=True),
tlp=case.classification.tlp.value,
created_at=now,
status=ApprovalStatus.PENDING.value,
)
with db.engine().begin() as conn:
res = conn.execute(stmt)
pid = int(res.inserted_primary_key[0])
_log.info("courier.queued", case_id=case.case_id, destination=route.destination_name, pending_id=pid)
return pid
def execute_routes(case: Case, routes: List[Route], sealed_pkg: Optional[SealedPackage] = None) -> List[SubmitResult]:
results: List[SubmitResult] = []
force = _force_approval()
for r in routes:
endpoint = endpoint_for(r.destination_name)
if endpoint is None:
@@ -140,6 +167,14 @@ def execute_routes(case: Case, routes: List[Route], sealed_pkg: Optional[SealedP
continue
payload = build_payload(case, r.payload_kind, sealed_pkg)
payload_hash = _hash_payload(payload)
if r.requires_approval or force:
pid = _enqueue_pending(case, r, payload, payload_hash)
results.append(SubmitResult(
destination_name=r.destination_name,
outcome=Outcome.PENDING_APPROVAL,
detail=f"pending_id={pid}",
))
continue
result = submit(endpoint, payload)
if isinstance(result, Err):
ledger_line.write(
@@ -166,3 +201,129 @@ def execute_routes(case: Case, routes: List[Route], sealed_pkg: Optional[SealedP
)
results.append(SubmitResult(destination_name=r.destination_name, outcome=outcome, receipt_id=receipt.receipt_id))
return results
def _row_to_pending(row: Any) -> PendingSubmission:
return PendingSubmission(
id=row.id,
case_id=row.case_id,
destination_name=row.destination_name,
payload_kind=row.payload_kind,
payload_hash=row.payload_hash,
payload_json=row.payload_json,
tlp=TLP(row.tlp),
created_at=datetime.fromisoformat(row.created_at),
status=ApprovalStatus(row.status),
reviewer=row.reviewer,
reviewed_at=datetime.fromisoformat(row.reviewed_at) if row.reviewed_at else None,
reason=row.reason,
)
def list_pending(status: Optional[ApprovalStatus] = ApprovalStatus.PENDING, limit: int = 200) -> List[PendingSubmission]:
stmt = select(db.pending)
if status is not None:
stmt = stmt.where(db.pending.c.status == status.value)
stmt = stmt.order_by(db.pending.c.created_at.desc()).limit(limit)
with db.engine().connect() as conn:
rows = conn.execute(stmt).fetchall()
return [_row_to_pending(r) for r in rows]
def get_pending(pid: int) -> Result[PendingSubmission, str]:
stmt = select(db.pending).where(db.pending.c.id == pid)
with db.engine().connect() as conn:
row = conn.execute(stmt).fetchone()
if row is None:
return Err(f"pending submission not found: {pid}")
return Ok(_row_to_pending(row))
def pending_count(status: ApprovalStatus = ApprovalStatus.PENDING) -> int:
from sqlalchemy import func as sa_func
stmt = select(sa_func.count()).select_from(db.pending).where(db.pending.c.status == status.value)
with db.engine().connect() as conn:
return int(conn.execute(stmt).scalar_one())
def dispatch_pending(pid: int, reviewer: str = "operator") -> Result[SubmitResult, str]:
"""Approve and submit a pending entry — POST to destination, write ledger, mark approved."""
pending_r = get_pending(pid)
if isinstance(pending_r, Err):
return Err(pending_r.reason)
p = pending_r.value
if p.status != ApprovalStatus.PENDING:
return Err(f"pending submission {pid} is already {p.status.value}")
endpoint = endpoint_for(p.destination_name)
if endpoint is None:
return Err(f"no endpoint configured for {p.destination_name}")
payload = json.loads(p.payload_json)
result = submit(endpoint, payload)
now = datetime.now(timezone.utc).isoformat()
if isinstance(result, Err):
ledger_line.write(
case_id=p.case_id,
destination=p.destination_name,
payload_hash=p.payload_hash,
submitter_identity=SUBMITTER_IDENTITY,
tlp=p.tlp,
outcome=Outcome.FAILED,
detail=result.reason,
)
with db.engine().begin() as conn:
conn.execute(update(db.pending).where(db.pending.c.id == pid).values(
status=ApprovalStatus.APPROVED.value,
reviewer=reviewer,
reviewed_at=now,
reason=f"submit failed: {result.reason}",
))
return Ok(SubmitResult(destination_name=p.destination_name, outcome=Outcome.FAILED, detail=result.reason))
receipt = result.value
outcome = _STATUS_TO_OUTCOME.get(receipt.status, Outcome.SUBMITTED)
ledger_line.write(
case_id=p.case_id,
destination=p.destination_name,
payload_hash=p.payload_hash,
submitter_identity=SUBMITTER_IDENTITY,
tlp=p.tlp,
outcome=outcome,
response_id=receipt.receipt_id,
detail=f"approved_by={reviewer}",
)
with db.engine().begin() as conn:
conn.execute(update(db.pending).where(db.pending.c.id == pid).values(
status=ApprovalStatus.APPROVED.value,
reviewer=reviewer,
reviewed_at=now,
))
_log.info("courier.approved", pending_id=pid, reviewer=reviewer, outcome=outcome.value)
return Ok(SubmitResult(destination_name=p.destination_name, outcome=outcome, receipt_id=receipt.receipt_id))
def reject_pending(pid: int, reviewer: str = "operator", reason: str = "") -> Result[None, str]:
"""Reject a pending entry — write ledger reject row, mark rejected. Nothing leaves."""
pending_r = get_pending(pid)
if isinstance(pending_r, Err):
return Err(pending_r.reason)
p = pending_r.value
if p.status != ApprovalStatus.PENDING:
return Err(f"pending submission {pid} is already {p.status.value}")
now = datetime.now(timezone.utc).isoformat()
ledger_line.write(
case_id=p.case_id,
destination=p.destination_name,
payload_hash=p.payload_hash,
submitter_identity=SUBMITTER_IDENTITY,
tlp=p.tlp,
outcome=Outcome.REJECTED,
detail=f"rejected_by={reviewer}: {reason}" if reason else f"rejected_by={reviewer}",
)
with db.engine().begin() as conn:
conn.execute(update(db.pending).where(db.pending.c.id == pid).values(
status=ApprovalStatus.REJECTED.value,
reviewer=reviewer,
reviewed_at=now,
reason=reason or None,
))
_log.info("courier.rejected", pending_id=pid, reviewer=reviewer)
return Ok(None)

73
src/psyc/lines/defang.py Normal file
View File

@@ -0,0 +1,73 @@
"""Defanging — IOC obfuscation styles common in real CTI prose.
Real advisories don't write `1.2.3.4` and `http://evil.com` verbatim; they
defang IOCs into bracket/paren/word forms (`1[.]2[.]3[.]4`, `hxxp://evil[.]com`)
so indicators don't auto-link in email/chat clients. Training the IOC extractor
purely on canonical inputs leaves it brittle. This module corrupts canonical
IOCs into common defanged forms for use as training-time data augmentation.
"""
from __future__ import annotations
import random
from typing import List, Optional
# Dot replacement styles seen in the wild, in rough frequency order.
_DOT_FORMS = ("[.]", "(.)", "[dot]", "{.}")
_PROTOCOL_FORMS = {
"http://": "hxxp://",
"https://": "hxxps://",
}
def _rng(r: Optional[random.Random]) -> random.Random:
return r if r is not None else random.Random()
def defang_ip(ip: str, rng: Optional[random.Random] = None) -> str:
"""`1.2.3.4` → `1[.]2[.]3[.]4` (one randomly chosen dot style)."""
return ip.replace(".", _rng(rng).choice(_DOT_FORMS))
def defang_domain(domain: str, rng: Optional[random.Random] = None) -> str:
"""`evil.com` → `evil[.]com`."""
return domain.replace(".", _rng(rng).choice(_DOT_FORMS))
def defang_url(url: str, rng: Optional[random.Random] = None) -> str:
"""`http://evil.com/x` → `hxxp://evil[.]com/x` — protocol + dot defanging."""
r = _rng(rng)
out = url
for proto, replacement in _PROTOCOL_FORMS.items():
if out.startswith(proto):
out = replacement + out[len(proto):]
break
out = out.replace(".", r.choice(_DOT_FORMS))
return out
def defang_text(
text: str,
ips: List[str],
domains: List[str],
urls: List[str],
rng: Optional[random.Random] = None,
) -> str:
"""Defang every occurrence of the given IOCs inside a free-text body.
URLs are replaced before domains (URLs contain domain substrings, so
domain-first would corrupt the URL match). Likewise IPs last. Each
occurrence picks its own dot-style independently — real advisories don't
keep one style consistent across paragraphs.
"""
r = _rng(rng)
out = text
for u in sorted(set(urls), key=len, reverse=True):
out = out.replace(u, defang_url(u, r))
for d in sorted(set(domains), key=len, reverse=True):
out = out.replace(d, defang_domain(d, r))
for i in sorted(set(ips), key=len, reverse=True):
out = out.replace(i, defang_ip(i, r))
return out

357
src/psyc/lines/discovery.py Normal file
View File

@@ -0,0 +1,357 @@
"""Discovery — DNS-SD resolver + BFS peer-walker for internet-wide federation.
Federation identity (psyc.lines.federation) gives every node a stable Ed25519
keypair, a 32-hex fingerprint, and a DNS record format. This module is the
*finder*: given a seed domain you suspect runs psyc, walk its DNS-SD records
to learn the fingerprint, fetch its public peer list, and recurse.
Newly-discovered peers always enter the `peers` table with status="unknown"
they do NOT become trusted by being seen; vouching is a separate concern
(sibling module). Discovery only populates the candidate set.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional, Set, Tuple
import dns.exception
import dns.rdatatype
import dns.resolver
import httpx
from pydantic import BaseModel, Field
from psyc import db, log
from psyc.lines import federation
from psyc.result import Err, Ok, Result
_log = log.get(__name__)
DEFAULT_TIMEOUT = 5.0
DEFAULT_PORT = 443
DEFAULT_MAX_DEPTH = 2
DEFAULT_MAX_PEERS = 200
_VALID_STATUSES = {"unknown", "trusted", "blocked", "vouched"}
# ---------- candidate model --------------------------------------------------
class PeerCandidate(BaseModel):
"""A peer found by the resolver/walker — not yet vetted, just observed."""
domain: str
fingerprint: str
port: int = DEFAULT_PORT
source: str # "dns-sd" | "peer-walk:<source-domain>"
discovered_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
# ---------- DNS-SD resolver --------------------------------------------------
def _parse_txt_value(txt: str) -> Result[Dict[str, str], str]:
"""Parse `v=psyc1 fp=<hex> alg=ed25519 path=...` → dict. Tolerant of order."""
out: Dict[str, str] = {}
for token in txt.strip().split():
if "=" not in token:
return Err(f"malformed TXT token (no '='): {token!r}")
k, v = token.split("=", 1)
out[k.strip()] = v.strip()
if out.get("v") != federation.FEED_VERSION:
return Err(f"unsupported version: {out.get('v')!r}")
fp = out.get("fp", "")
if len(fp) != 32 or any(c not in "0123456789abcdef" for c in fp.lower()):
return Err(f"bad fingerprint: {fp!r}")
if out.get("alg") and out["alg"] != federation.FEED_ALG:
return Err(f"unsupported alg: {out.get('alg')!r}")
return Ok(out)
def _flatten_txt(rdata: Any) -> str:
"""DNS TXT records are a sequence of byte-strings — join them. Tolerant of mocks."""
strings = getattr(rdata, "strings", None)
if strings is None:
return str(rdata).strip('"')
parts: List[str] = []
for s in strings:
if isinstance(s, bytes):
parts.append(s.decode("utf-8", errors="replace"))
else:
parts.append(str(s))
return "".join(parts)
def resolve_psyc(domain: str, timeout: float = DEFAULT_TIMEOUT) -> Result[PeerCandidate, str]:
"""Look up `_psyc._tcp.<domain>` SRV + TXT → return a candidate.
SRV gives target+port; TXT carries `v=psyc1 fp=... alg=ed25519 path=...`.
Any DNS failure, parse failure, or missing record returns Err.
"""
name = f"_psyc._tcp.{domain}"
resolver = dns.resolver.Resolver()
resolver.lifetime = timeout
resolver.timeout = timeout
# SRV: target host + port.
port = DEFAULT_PORT
try:
srv_answers = resolver.resolve(name, "SRV")
except dns.resolver.NXDOMAIN:
return Err(f"no SRV record at {name} (NXDOMAIN)")
except dns.resolver.NoAnswer:
return Err(f"no SRV record at {name} (NoAnswer)")
except dns.exception.Timeout:
return Err(f"SRV lookup timed out for {name}")
except dns.exception.DNSException as exc:
return Err(f"SRV lookup failed for {name}: {exc}")
srv_list = list(srv_answers)
if not srv_list:
return Err(f"no SRV record at {name} (empty)")
try:
port = int(srv_list[0].port)
except Exception:
port = DEFAULT_PORT
# TXT: fingerprint + protocol metadata.
try:
txt_answers = resolver.resolve(name, "TXT")
except dns.resolver.NXDOMAIN:
return Err(f"no TXT record at {name} (NXDOMAIN)")
except dns.resolver.NoAnswer:
return Err(f"no TXT record at {name} (NoAnswer)")
except dns.exception.Timeout:
return Err(f"TXT lookup timed out for {name}")
except dns.exception.DNSException as exc:
return Err(f"TXT lookup failed for {name}: {exc}")
txt_list = list(txt_answers)
if not txt_list:
return Err(f"no TXT record at {name} (empty)")
last_err: Optional[str] = None
for rdata in txt_list:
value = _flatten_txt(rdata)
parsed = _parse_txt_value(value)
if isinstance(parsed, Ok):
return Ok(PeerCandidate(
domain=domain,
fingerprint=parsed.value["fp"].lower(),
port=port,
source="dns-sd",
))
last_err = parsed.reason
return Err(f"no usable psyc TXT record at {name}: {last_err}")
# ---------- HTTP probes ------------------------------------------------------
def _base_url(domain: str, port: int) -> str:
if port == 443:
return f"https://{domain}"
return f"https://{domain}:{port}"
def fetch_peer_info(domain: str, port: int = DEFAULT_PORT, timeout: float = DEFAULT_TIMEOUT,
expected_fingerprint: Optional[str] = None) -> Result[Dict[str, Any], str]:
"""GET /federation/info on a peer. Cross-checks fingerprint if provided.
The cross-check defends against MITM-injected TXT records — DNS said one
fingerprint, the live node's HTTPS-served info MUST agree.
"""
url = _base_url(domain, port) + "/federation/info"
try:
with httpx.Client(timeout=timeout) as client:
r = client.get(url)
r.raise_for_status()
data = r.json()
except httpx.HTTPStatusError as exc:
return Err(f"HTTP {exc.response.status_code} from {url}")
except httpx.RequestError as exc:
return Err(f"network error fetching {url}: {exc}")
except ValueError as exc:
return Err(f"non-JSON response from {url}: {exc}")
except Exception as exc: # noqa: BLE001 — anything weird is a failure
return Err(f"fetch failed for {url}: {exc}")
if not isinstance(data, dict):
return Err(f"unexpected info shape from {url}: {type(data).__name__}")
declared = str(data.get("fingerprint", "")).lower()
if expected_fingerprint and declared != expected_fingerprint.lower():
return Err(
f"fingerprint mismatch for {domain}: DNS said {expected_fingerprint!r} "
f"but /federation/info said {declared!r}"
)
return Ok(data)
def fetch_public_peers(domain: str, port: int = DEFAULT_PORT,
timeout: float = DEFAULT_TIMEOUT) -> Result[List[Dict[str, Any]], str]:
"""GET /federation/peers/public on a peer. Returns the list as-is for the walker to dedupe."""
url = _base_url(domain, port) + "/federation/peers/public"
try:
with httpx.Client(timeout=timeout) as client:
r = client.get(url)
r.raise_for_status()
data = r.json()
except httpx.HTTPStatusError as exc:
return Err(f"HTTP {exc.response.status_code} from {url}")
except httpx.RequestError as exc:
return Err(f"network error fetching {url}: {exc}")
except ValueError as exc:
return Err(f"non-JSON response from {url}: {exc}")
except Exception as exc: # noqa: BLE001
return Err(f"fetch failed for {url}: {exc}")
if not isinstance(data, list):
return Err(f"unexpected peers shape from {url}: {type(data).__name__}")
out: List[Dict[str, Any]] = []
for item in data:
if isinstance(item, dict) and item.get("domain") and item.get("fingerprint"):
out.append(item)
return Ok(out)
# ---------- BFS walker -------------------------------------------------------
def walk(seeds: List[str], max_depth: int = DEFAULT_MAX_DEPTH,
max_peers: int = DEFAULT_MAX_PEERS, timeout: float = DEFAULT_TIMEOUT) -> List[PeerCandidate]:
"""Breadth-first walk from `seeds` → discovered candidates.
For each domain: DNS-SD resolve → fetch /info to verify fingerprint →
fetch /peers/public → enqueue its domains for the next layer. Dedupes on
(domain, fingerprint). Skips our own fingerprint to avoid loops. All
errors are logged but non-fatal — one bad peer doesn't abort the walk.
"""
own_fp = ""
try:
own_fp = federation.node_fingerprint()
except Exception as exc: # noqa: BLE001 — discovery should work even pre-keygen
_log.info("discovery.walk.no_own_fp", error=str(exc))
seen_pairs: Set[Tuple[str, str]] = set()
seen_domains: Set[str] = set()
out: List[PeerCandidate] = []
# Queue of (domain, depth, source). Seeds enter at depth 0.
frontier: List[Tuple[str, int, str]] = [(d.strip(), 0, "dns-sd") for d in seeds if d and d.strip()]
next_layer: List[Tuple[str, int, str]] = []
while frontier:
domain, depth, source = frontier.pop(0)
if domain in seen_domains:
if not frontier:
frontier, next_layer = next_layer, []
continue
seen_domains.add(domain)
if len(out) >= max_peers:
_log.info("discovery.walk.cap.peers", cap=max_peers)
break
resolved = resolve_psyc(domain, timeout=timeout)
if isinstance(resolved, Err):
_log.info("discovery.resolve.skip", domain=domain, reason=resolved.reason)
if not frontier:
frontier, next_layer = next_layer, []
continue
cand = resolved.value
cand.source = source if depth > 0 else "dns-sd"
if cand.fingerprint == own_fp:
_log.info("discovery.skip.self", domain=domain)
if not frontier:
frontier, next_layer = next_layer, []
continue
pair = (cand.domain, cand.fingerprint)
if pair in seen_pairs:
if not frontier:
frontier, next_layer = next_layer, []
continue
seen_pairs.add(pair)
# Verify the live endpoint's fingerprint matches DNS. If we can't reach
# it, still record the DNS-discovered candidate — vouching can vet it
# later, and we don't want one HTTP outage to abort the walk.
info_res = fetch_peer_info(domain, port=cand.port, timeout=timeout,
expected_fingerprint=cand.fingerprint)
if isinstance(info_res, Err):
_log.info("discovery.info.skip", domain=domain, reason=info_res.reason)
out.append(cand)
if not frontier:
frontier, next_layer = next_layer, []
continue
out.append(cand)
# Recurse: fetch this peer's public-peers list, enqueue domains.
if depth + 1 <= max_depth:
peers_res = fetch_public_peers(domain, port=cand.port, timeout=timeout)
if isinstance(peers_res, Err):
_log.info("discovery.peers.skip", domain=domain, reason=peers_res.reason)
else:
for item in peers_res.value:
child_domain = str(item.get("domain", "")).strip()
if not child_domain or child_domain in seen_domains:
continue
child_source = f"peer-walk:{domain}"
next_layer.append((child_domain, depth + 1, child_source))
if not frontier:
frontier, next_layer = next_layer, []
_log.info("discovery.walk.done", seeds=len(seeds), discovered=len(out), max_depth=max_depth)
return out
# ---------- persistence ------------------------------------------------------
def record_candidate(c: PeerCandidate, default_status: str = "unknown") -> None:
"""Upsert a discovered candidate into the peers table.
Preserves any existing trusted/blocked status — discovery NEVER demotes a
peer the operator has already classified. Only updates last_seen.
"""
if default_status not in _VALID_STATUSES:
default_status = "unknown"
existing = db.get_peer(c.domain)
now = c.discovered_at.isoformat()
if existing:
status = existing.get("status") or default_status
if status not in ("trusted", "blocked"):
status = default_status
db.upsert_peer(dict(
domain=c.domain,
fingerprint=existing.get("fingerprint") or c.fingerprint,
pubkey_pem=existing.get("pubkey_pem") or "",
status=status,
discovered_at=existing.get("discovered_at") or now,
last_seen=now,
notes=existing.get("notes"),
))
return
db.upsert_peer(dict(
domain=c.domain,
fingerprint=c.fingerprint,
pubkey_pem="", # populated when we successfully fetch /federation/key during vouching
status=default_status,
discovered_at=now,
last_seen=now,
notes=f"discovered via {c.source}",
))
_log.info("discovery.recorded", domain=c.domain, fp=c.fingerprint, source=c.source)
# ---------- public attestation -----------------------------------------------
def public_peer_attestation() -> List[Dict[str, Any]]:
"""List of peers we'll publicly attest to. Only `trusted` — never leaks unknown/blocked.
This is the surface that other psyc nodes' walkers read from us. We never
expose unknown or blocked peers — those are internal classification state.
"""
out: List[Dict[str, Any]] = []
for row in db.list_peers():
if row.get("status") != "trusted":
continue
out.append({
"domain": row["domain"],
"fingerprint": row["fingerprint"],
"first_seen": row.get("discovered_at"),
})
return out

View File

@@ -0,0 +1,746 @@
"""Federation — node identity, signed feeds, peer registry.
Identity layer for internet-wide federation of psyc nodes. Each node owns
an Ed25519 keypair persisted under DATA_DIR/federation/. The public key
fingerprint (first 16 bytes of SHA256(raw_pubkey) hex-encoded) goes into a
DNS TXT record so peers can discover and authenticate the node, and the
private key signs the outbound feed at /federation/feed.
This module is the *identity* primitives only — discovery walkers,
vouching/quorum, transparency log and auto-pull live in later stages.
"""
from __future__ import annotations
import base64
import hashlib
import json
import os
import re
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, List, Optional, Tuple
# Hostname-with-optional-port pattern for peer domains. Reject anything else at
# registration so a hostile domain string can't reach a render context where
# it could break out of an HTML attr or JS string.
_DOMAIN_RE = re.compile(r"^[A-Za-z0-9._\-]+(:\d{1,5})?$")
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
from pydantic import BaseModel, Field
from psyc import DATA_DIR, db, log
from psyc.lines import translog
from psyc.result import Err, Ok, Result
_log = log.get(__name__)
FED_DIR = DATA_DIR / "federation"
PRIVATE_KEY_PATH = FED_DIR / "node.key"
PUBLIC_KEY_PATH = FED_DIR / "node.pub"
FEED_VERSION = "psyc1"
FEED_ALG = "ed25519"
FEED_PATH = "/federation/feed"
# ---------- keypair persistence -----------------------------------------
def _ensure_dir() -> None:
FED_DIR.mkdir(parents=True, exist_ok=True)
def node_keypair() -> Tuple[ed25519.Ed25519PrivateKey, ed25519.Ed25519PublicKey]:
"""Return the node's Ed25519 keypair, generating + persisting it on first call.
Private key lands at data/federation/node.key (PEM, chmod 0600); public
at data/federation/node.pub (PEM). Idempotent — subsequent calls load
the existing files instead of generating new ones.
"""
_ensure_dir()
if PRIVATE_KEY_PATH.exists() and PUBLIC_KEY_PATH.exists():
priv_pem = PRIVATE_KEY_PATH.read_bytes()
priv = serialization.load_pem_private_key(priv_pem, password=None)
if not isinstance(priv, ed25519.Ed25519PrivateKey):
raise RuntimeError(f"federation key at {PRIVATE_KEY_PATH} is not Ed25519")
return priv, priv.public_key()
priv = ed25519.Ed25519PrivateKey.generate()
priv_pem = priv.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.PKCS8,
encryption_algorithm=serialization.NoEncryption(),
)
pub = priv.public_key()
pub_pem = pub.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
)
PRIVATE_KEY_PATH.write_bytes(priv_pem)
os.chmod(PRIVATE_KEY_PATH, 0o600)
PUBLIC_KEY_PATH.write_bytes(pub_pem)
_log.info("federation.keypair.generated", path=str(PRIVATE_KEY_PATH))
return priv, pub
def public_key_pem() -> str:
"""PEM-encoded public key as text — what peers store + verify against."""
_, pub = node_keypair()
return pub.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
).decode("ascii")
def _raw_pubkey_bytes(pub: ed25519.Ed25519PublicKey) -> bytes:
return pub.public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
def node_fingerprint() -> str:
"""Short stable id for the node — first 16 bytes of SHA256(raw_pubkey), hex.
Lives in DNS TXT records; 32 hex chars is short enough to fit but long
enough to be collision-safe for any plausible peer population.
"""
_, pub = node_keypair()
digest = hashlib.sha256(_raw_pubkey_bytes(pub)).digest()
return digest[:16].hex()
def _fingerprint_for_pubkey_pem(pubkey_pem: str) -> str:
pub = serialization.load_pem_public_key(pubkey_pem.encode("ascii"))
if not isinstance(pub, ed25519.Ed25519PublicKey):
raise ValueError("not an Ed25519 public key")
return hashlib.sha256(_raw_pubkey_bytes(pub)).digest()[:16].hex()
# ---------- DNS record format -------------------------------------------
class DNSRecord(BaseModel):
"""The SRV + TXT pair an admin pastes into their zone file."""
srv_name: str
srv_target: str
srv_port: int
srv_priority: int = 10
srv_weight: int = 10
txt_name: str
txt_value: str
human_instructions: str
def dns_record(domain: str, port: int = 443) -> DNSRecord:
"""Build the DNS-SD-style records that advertise this node at `domain`."""
fp = node_fingerprint()
srv_name = f"_psyc._tcp.{domain}"
srv_target = f"{domain}."
txt_name = srv_name
txt_value = f"v={FEED_VERSION} fp={fp} alg={FEED_ALG} path={FEED_PATH}"
instructions = (
f"; psyc federation records for {domain}\n"
f"; ----------------------------------------------------------\n"
f"; 1) SRV record — locates this psyc node (host + port).\n"
f'{srv_name}. 3600 IN SRV 10 10 {port} {srv_target}\n'
f";\n"
f"; 2) TXT record — declares protocol version, key fingerprint,\n"
f"; signature algorithm, and the feed endpoint path.\n"
f'{txt_name}. 3600 IN TXT "{txt_value}"\n'
f"; ----------------------------------------------------------\n"
f"; Once these are live, federation peers can fetch:\n"
f"; https://{domain}{FEED_PATH} (signed feed JSON)\n"
f"; https://{domain}/federation/key (public key PEM)\n"
f"; https://{domain}/federation/info (capabilities)\n"
)
return DNSRecord(
srv_name=srv_name,
srv_target=srv_target,
srv_port=port,
txt_name=txt_name,
txt_value=txt_value,
human_instructions=instructions,
)
# ---------- signing -----------------------------------------------------
def canonical_json(obj: Dict[str, Any]) -> bytes:
"""Deterministic JSON serialization — what we sign + hash over."""
return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
def sign_payload(payload: bytes) -> bytes:
"""Ed25519 signature over `payload`. Raw 64-byte sig."""
priv, _ = node_keypair()
return priv.sign(payload)
def verify_payload(payload: bytes, signature: bytes, pubkey_pem: str) -> bool:
"""True iff `signature` verifies under `pubkey_pem`. Never raises."""
try:
pub = serialization.load_pem_public_key(pubkey_pem.encode("ascii"))
if not isinstance(pub, ed25519.Ed25519PublicKey):
return False
pub.verify(signature, payload)
return True
except Exception:
return False
# ---------- feed export -------------------------------------------------
def _case_digest(case_record: Dict[str, Any]) -> str:
return hashlib.sha256(canonical_json(case_record)).hexdigest()
def _build_case_records(window_hours: int) -> List[Dict[str, Any]]:
cutoff = datetime.now(timezone.utc) - timedelta(hours=window_hours)
out: List[Dict[str, Any]] = []
for case in db.list_cases(limit=10_000):
if case.ingested_at < cutoff:
continue
record: Dict[str, Any] = {
"case_id": case.case_id,
"summary": case.summary,
"severity": case.classification.severity.value if case.classification.severity else None,
"incident_type": case.classification.incident_type.value if case.classification.incident_type else None,
"observed_at": case.observed_at.isoformat(),
"feed_source": case.source_metadata.get("feed", ""),
"iocs": (
[{"value": v, "type": "url"} for v in case.observables.urls]
+ [{"value": v, "type": "domain"} for v in case.observables.domains]
+ [{"value": v, "type": "ip"} for v in case.observables.ips]
+ [{"value": v, "type": "hash"} for v in case.observables.hashes]
+ [{"value": v, "type": "cve"} for v in case.observables.cves]
),
}
record["digest_sha256"] = _case_digest(
{k: v for k, v in record.items() if k != "digest_sha256"}
)
out.append(record)
return out
def _build_ioc_records(window_hours: int) -> List[Dict[str, Any]]:
cutoff = datetime.now(timezone.utc) - timedelta(hours=window_hours)
out: List[Dict[str, Any]] = []
seen: set = set()
for ioc_type in ("url", "domain", "ip", "hash", "cve"):
for row in db.iocs_by_type(ioc_type):
first_seen = row.get("first_seen")
if first_seen:
try:
if datetime.fromisoformat(first_seen) < cutoff:
continue
except ValueError:
pass
key = (row["value"], row["ioc_type"])
if key in seen:
continue
seen.add(key)
record = {
"value": row["value"],
"type": row["ioc_type"],
"severity": row.get("severity"),
"first_seen": first_seen,
}
record["digest_sha256"] = hashlib.sha256(canonical_json(record)).hexdigest()
out.append(record)
return out
def build_signed_feed(window_hours: int = 24) -> Dict[str, Any]:
"""Build the JSON feed peers will pull from /federation/feed.
Pulls cases ingested in the last `window_hours` plus the corresponding
IOC slice, attaches per-record `digest_sha256` (so peers can later
quorum-match across nodes), and signs the canonical JSON of the whole
payload-minus-signature with our Ed25519 key.
"""
payload: Dict[str, Any] = {
"version": FEED_VERSION,
"fingerprint": node_fingerprint(),
"generated_at": datetime.now(timezone.utc).isoformat(),
"window_hours": window_hours,
"cases": _build_case_records(window_hours),
"iocs": _build_ioc_records(window_hours),
# Vouches we've issued ride along with the feed so peers can learn
# who we trust and accumulate quorum on shared targets.
"vouches": [v.model_dump() for v in our_vouches()],
}
sig = sign_payload(canonical_json(payload))
payload["signature"] = base64.b64encode(sig).decode("ascii")
return payload
# ---------- import + quorum-signal buffer -------------------------------
class ImportSummary(BaseModel):
peer_fingerprint: str
cases_seen: int
iocs_seen: int
signal_ids: List[Tuple[str, str]] = Field(default_factory=list)
def import_signed_feed(feed: Dict[str, Any], expected_pubkey_pem: str) -> Result[ImportSummary, str]:
"""Verify + record a peer's feed into the federation_signals buffer.
Does NOT merge into the local case store — that's the quorum stage's
job. The buffer is the per-hash signal log that quorum logic later
aggregates ("3 trusted peers reported this same IOC → promote").
"""
sig_b64 = feed.get("signature")
if not sig_b64:
return Err("missing signature")
try:
signature = base64.b64decode(sig_b64)
except Exception:
return Err("malformed signature (not base64)")
unsigned = {k: v for k, v in feed.items() if k != "signature"}
if not verify_payload(canonical_json(unsigned), signature, expected_pubkey_pem):
return Err("signature verification failed")
peer_fp = feed.get("fingerprint", "")
if not peer_fp:
return Err("missing fingerprint")
if peer_fp == node_fingerprint():
return Err("loop: own feed")
# Cross-check the declared fingerprint matches the pubkey we verified with.
try:
if _fingerprint_for_pubkey_pem(expected_pubkey_pem) != peer_fp:
return Err("fingerprint does not match provided pubkey")
except Exception as exc:
return Err(f"bad pubkey: {exc}")
# Listening gate: only accept signals from peers we explicitly trust or
# that quorum of trusted peers vouches for. Unknown peers don't land here.
if not peer_is_listening_eligible(peer_fp):
return Err(f"peer not trusted: {peer_fp}")
now = datetime.now(timezone.utc).isoformat()
signal_ids: List[Tuple[str, str]] = []
cases = feed.get("cases") or []
iocs = feed.get("iocs") or []
feed_vouches = feed.get("vouches") or []
for c in cases:
case_id = c.get("case_id") or ""
digest = c.get("digest_sha256") or hashlib.sha256(canonical_json(c)).hexdigest()
db.record_signal(dict(
peer_fingerprint=peer_fp,
signal_type="case",
signal_id=case_id,
signal_hash=digest,
received_at=now,
raw_json=json.dumps(c, sort_keys=True),
))
signal_ids.append(("case", digest))
try:
translog.append("signal", {
"peer_fingerprint": peer_fp,
"signal_type": "case",
"signal_id": case_id,
"signal_hash": digest,
})
except Exception as exc: # transparency log is best-effort, never block ingest
_log.warning("federation.translog.append.fail", error=str(exc))
for i in iocs:
value = i.get("value") or ""
digest = i.get("digest_sha256") or hashlib.sha256(canonical_json(i)).hexdigest()
db.record_signal(dict(
peer_fingerprint=peer_fp,
signal_type="ioc",
signal_id=value,
signal_hash=digest,
received_at=now,
raw_json=json.dumps(i, sort_keys=True),
))
signal_ids.append(("ioc", digest))
try:
translog.append("signal", {
"peer_fingerprint": peer_fp,
"signal_type": "ioc",
"signal_id": value,
"signal_hash": digest,
})
except Exception as exc:
_log.warning("federation.translog.append.fail", error=str(exc))
# Vouch propagation — peer asserts who they trust. We only accept vouches
# whose declared voucher fingerprint matches the peer we just authenticated
# (so a peer can't forge vouches "from" someone else through us).
for v_raw in feed_vouches:
if not isinstance(v_raw, dict):
continue
try:
vouch = Vouch.model_validate(v_raw)
except Exception as exc:
_log.warning("federation.vouch.malformed", error=str(exc))
continue
if vouch.voucher_fingerprint != peer_fp:
_log.warning(
"federation.vouch.voucher_mismatch",
claimed=vouch.voucher_fingerprint, actual=peer_fp,
)
continue
accepted = accept_vouch(vouch, expected_pubkey_pem)
if isinstance(accepted, Err):
_log.warning("federation.vouch.rejected", reason=accepted.reason)
_log.info("federation.import.ok", peer=peer_fp, cases=len(cases), iocs=len(iocs))
return Ok(ImportSummary(
peer_fingerprint=peer_fp,
cases_seen=len(cases),
iocs_seen=len(iocs),
signal_ids=signal_ids,
))
# ---------- peer registry ------------------------------------------------
class Peer(BaseModel):
domain: str
fingerprint: str
pubkey_pem: str
status: str = "unknown" # unknown | trusted | blocked
discovered_at: str
last_seen: Optional[str] = None
notes: Optional[str] = None
def _row_to_peer(row: Dict[str, Any]) -> Peer:
return Peer(
domain=row["domain"],
fingerprint=row["fingerprint"],
pubkey_pem=row["pubkey_pem"],
status=row.get("status") or "unknown",
discovered_at=row.get("discovered_at") or "",
last_seen=row.get("last_seen"),
notes=row.get("notes"),
)
def register_peer(domain: str, fingerprint: str, pubkey_pem: str, status: str = "unknown") -> None:
"""Insert or update a peer in the registry. Idempotent on `domain`.
Rejects malformed domain strings — only hostname chars + optional :port.
Closes a stored-XSS hole where a hostile `domain` would have been rendered
into the admin federation page's confirm() prompt.
"""
domain = (domain or "").strip()
if not _DOMAIN_RE.match(domain):
raise ValueError(f"invalid domain: {domain!r}")
if fingerprint and not re.fullmatch(r"[0-9a-fA-F]{32}", fingerprint):
raise ValueError(f"invalid fingerprint: {fingerprint!r}")
now = datetime.now(timezone.utc).isoformat()
existing = db.get_peer(domain)
discovered_at = existing["discovered_at"] if existing else now
db.upsert_peer(dict(
domain=domain,
fingerprint=fingerprint,
pubkey_pem=pubkey_pem,
status=status,
discovered_at=discovered_at,
last_seen=now,
notes=existing.get("notes") if existing else None,
))
def list_peers() -> List[Peer]:
return [_row_to_peer(r) for r in db.list_peers()]
def get_peer(domain: str) -> Optional[Peer]:
row = db.get_peer(domain)
return _row_to_peer(row) if row else None
def set_peer_status(domain: str, status: str) -> None:
if status not in ("unknown", "trusted", "blocked"):
raise ValueError(f"unknown peer status: {status}")
db.set_peer_status(domain, status)
def remove_peer(domain: str) -> None:
db.remove_peer(domain)
# ---------- vouching + quorum (stage 4) ---------------------------------
#
# The web of trust: a peer's fingerprint becomes "listening-eligible" when
# either we directly trust it (peers.status == "trusted") or at least
# `trust_min_vouchers` of our trusted peers have signed a vouch for it.
#
# Signal-level quorum: a federation_signals row is meaningful only when
# `signal_quorum_k` distinct vouched peers have reported the same signal_hash.
#
# Vouches are short Pydantic records signed with the voucher's Ed25519 key
# over canonical JSON of the body (everything but the signature field).
class Vouch(BaseModel):
voucher_fingerprint: str
target_fingerprint: str
issued_at: datetime
expires_at: Optional[datetime] = None
signature: str = "" # base64 ed25519 sig over vouch_payload_bytes(...)
class QuorumConfig(BaseModel):
trust_min_vouchers: int = 2
signal_quorum_k: int = 2
_QC_TRUST_KEY = "wot_trust_min"
_QC_K_KEY = "wot_quorum_k"
def quorum_config() -> QuorumConfig:
"""Live quorum settings, with sensible defaults if pulse_settings is empty."""
cfg = QuorumConfig()
t = db.setting_get(_QC_TRUST_KEY)
k = db.setting_get(_QC_K_KEY)
if t is not None:
try:
cfg.trust_min_vouchers = max(1, int(t))
except ValueError:
pass
if k is not None:
try:
cfg.signal_quorum_k = max(1, int(k))
except ValueError:
pass
return cfg
def set_quorum_config(cfg: QuorumConfig) -> None:
"""Persist quorum config into pulse_settings."""
db.setting_set(_QC_TRUST_KEY, str(cfg.trust_min_vouchers))
db.setting_set(_QC_K_KEY, str(cfg.signal_quorum_k))
def vouch_payload_bytes(
voucher_fp: str,
target_fp: str,
issued_at: datetime,
expires_at: Optional[datetime],
) -> bytes:
"""Canonical JSON of the unsigned vouch body — what the voucher signs."""
body: Dict[str, Any] = {
"voucher_fingerprint": voucher_fp,
"target_fingerprint": target_fp,
"issued_at": issued_at.isoformat(),
"expires_at": expires_at.isoformat() if expires_at else None,
}
return canonical_json(body)
def _store_vouch(v: Vouch) -> None:
db.upsert_vouch(dict(
voucher_fingerprint=v.voucher_fingerprint,
target_fingerprint=v.target_fingerprint,
issued_at=v.issued_at.isoformat(),
expires_at=v.expires_at.isoformat() if v.expires_at else None,
signature=v.signature,
))
def _row_to_vouch(row: Dict[str, Any]) -> Vouch:
return Vouch(
voucher_fingerprint=row["voucher_fingerprint"],
target_fingerprint=row["target_fingerprint"],
issued_at=datetime.fromisoformat(row["issued_at"]),
expires_at=datetime.fromisoformat(row["expires_at"]) if row.get("expires_at") else None,
signature=row.get("signature") or "",
)
def issue_vouch(target_fingerprint: str, ttl_days: int = 90) -> Vouch:
"""Sign a vouch for `target_fingerprint` with OUR key. Persists + returns it."""
our_fp = node_fingerprint()
issued_at = datetime.now(timezone.utc)
expires_at = issued_at + timedelta(days=ttl_days) if ttl_days > 0 else None
payload = vouch_payload_bytes(our_fp, target_fingerprint, issued_at, expires_at)
sig = sign_payload(payload)
vouch = Vouch(
voucher_fingerprint=our_fp,
target_fingerprint=target_fingerprint,
issued_at=issued_at,
expires_at=expires_at,
signature=base64.b64encode(sig).decode("ascii"),
)
_store_vouch(vouch)
try:
translog.append("vouch", {
"voucher_fingerprint": vouch.voucher_fingerprint,
"target_fingerprint": vouch.target_fingerprint,
"issued_at": vouch.issued_at.isoformat(),
"expires_at": vouch.expires_at.isoformat() if vouch.expires_at else None,
})
except Exception as exc:
_log.warning("federation.translog.append.fail", error=str(exc))
_log.info("federation.vouch.issued", target=target_fingerprint, ttl_days=ttl_days)
return vouch
def accept_vouch(vouch: Vouch, voucher_pubkey_pem: str) -> Result[None, str]:
"""Verify signature + expiry + voucher trust status, then persist.
Failure modes return Err with a short reason so the caller can log them.
A voucher whose status is not "trusted" in our peers table is refused —
we don't accept transitive vouches from unknown peers.
"""
# Expiry first — cheapest check.
now = datetime.now(timezone.utc)
if vouch.expires_at is not None and vouch.expires_at < now:
return Err("vouch expired")
# Voucher must be a directly-trusted peer (no transitive trust at this layer).
voucher_status = None
for row in db.list_peers():
if row.get("fingerprint") == vouch.voucher_fingerprint:
voucher_status = row.get("status")
break
if voucher_status != "trusted":
return Err(f"voucher not trusted: {vouch.voucher_fingerprint}")
# The pubkey must match the declared voucher fingerprint.
try:
if _fingerprint_for_pubkey_pem(voucher_pubkey_pem) != vouch.voucher_fingerprint:
return Err("voucher pubkey does not match fingerprint")
except Exception as exc:
return Err(f"bad voucher pubkey: {exc}")
payload = vouch_payload_bytes(
vouch.voucher_fingerprint,
vouch.target_fingerprint,
vouch.issued_at,
vouch.expires_at,
)
try:
signature = base64.b64decode(vouch.signature)
except Exception:
return Err("vouch signature not base64")
if not verify_payload(payload, signature, voucher_pubkey_pem):
return Err("vouch signature invalid")
_store_vouch(vouch)
try:
translog.append("vouch", {
"voucher_fingerprint": vouch.voucher_fingerprint,
"target_fingerprint": vouch.target_fingerprint,
"issued_at": vouch.issued_at.isoformat(),
"expires_at": vouch.expires_at.isoformat() if vouch.expires_at else None,
"accepted": True,
})
except Exception as exc:
_log.warning("federation.translog.append.fail", error=str(exc))
_log.info("federation.vouch.accepted", voucher=vouch.voucher_fingerprint, target=vouch.target_fingerprint)
return Ok(None)
def revoke_vouch(target_fingerprint: str) -> None:
"""Delete OUR vouch naming `target_fingerprint`. No-op if absent."""
db.delete_vouch(node_fingerprint(), target_fingerprint)
_log.info("federation.vouch.revoked", target=target_fingerprint)
def our_vouches() -> List[Vouch]:
"""Vouches we have issued (filter for voucher_fingerprint == our fp)."""
return [_row_to_vouch(r) for r in db.vouches_by_voucher(node_fingerprint())]
def vouches_for(target_fingerprint: str) -> List[Vouch]:
"""Every vouch stored locally that names `target_fingerprint` as target."""
return [_row_to_vouch(r) for r in db.vouches_by_target(target_fingerprint)]
def is_vouched(target_fingerprint: str, min_vouchers: Optional[int] = None) -> bool:
"""True iff ≥`min_vouchers` distinct non-expired vouches from currently-trusted
peers name `target_fingerprint`.
"""
cfg = quorum_config()
threshold = min_vouchers if min_vouchers is not None else cfg.trust_min_vouchers
if threshold <= 0:
return True
now = datetime.now(timezone.utc)
trusted_fps = {p.fingerprint for p in list_peers() if p.status == "trusted"}
distinct_vouchers: set = set()
for v in vouches_for(target_fingerprint):
if v.expires_at is not None and v.expires_at < now:
continue
if v.voucher_fingerprint not in trusted_fps:
continue
distinct_vouchers.add(v.voucher_fingerprint)
if len(distinct_vouchers) >= threshold:
return True
return False
def peer_is_listening_eligible(fingerprint: str) -> bool:
"""True iff the peer is directly trusted OR vouched into trust.
This is the gate used by `import_signed_feed`. Auto-response will share
this signature — keep it stable.
"""
if not fingerprint:
return False
for p in list_peers():
if p.fingerprint == fingerprint:
if p.status == "trusted":
return True
if p.status == "blocked":
return False
break
return is_vouched(fingerprint)
def is_quorum_met(signal_hash: str, k: Optional[int] = None) -> bool:
"""True iff ≥k distinct vouched peers have reported `signal_hash`.
"Vouched" here means `peer_is_listening_eligible` — the same web-of-trust
set the import gate respects. Self-reports from the local node do not
count (they never end up in federation_signals).
"""
cfg = quorum_config()
threshold = k if k is not None else cfg.signal_quorum_k
if threshold <= 0:
return True
rows = db.signals_for_hash(signal_hash)
distinct: set = set()
for r in rows:
fp = r.get("peer_fingerprint") or ""
if not fp or fp in distinct:
continue
if not peer_is_listening_eligible(fp):
continue
distinct.add(fp)
if len(distinct) >= threshold:
return True
return False
def quorum_evidence(signal_hash: str) -> List[Tuple[str, datetime]]:
"""(peer_fingerprint, received_at) tuples for one signal_hash — for UI display.
Only includes signals from currently listening-eligible peers, deduped
per fingerprint at the earliest receipt.
"""
rows = db.signals_for_hash(signal_hash)
earliest: Dict[str, datetime] = {}
for r in rows:
fp = r.get("peer_fingerprint") or ""
if not fp or not peer_is_listening_eligible(fp):
continue
try:
ts = datetime.fromisoformat(r.get("received_at") or "")
except ValueError:
continue
if fp not in earliest or ts < earliest[fp]:
earliest[fp] = ts
return sorted(earliest.items(), key=lambda kv: kv[1])

102
src/psyc/lines/lookup.py Normal file
View File

@@ -0,0 +1,102 @@
"""Lookupline — IOC index over the case corpus.
Turns the collected cases into a reverse index: indicator -> which cases,
feeds, and severities mention it. This is the shared primitive behind
"paste an indicator, is it known-bad?", asset matching, and blocklist export.
Indicators are normalized so lookups are case- and format-insensitive.
"""
from __future__ import annotations
from typing import Dict, Iterable, List, Optional, Tuple
from psyc import db, log
from psyc.models import Case
_log = log.get(__name__)
# severity ordering for min-severity filters
_SEVERITY_RANK: Dict[str, int] = {"low": 0, "medium": 1, "high": 2, "critical": 3}
IOC_TYPES = ("url", "domain", "ip", "hash", "cve")
def normalize(value: str, ioc_type: str) -> str:
"""Normalize an indicator for storage + lookup. CVEs upper, everything else lower."""
v = value.strip()
if ioc_type == "cve":
return v.upper()
return v.lower()
def iter_case_iocs(case: Case) -> Iterable[Tuple[str, str]]:
"""Yield (normalized_value, ioc_type) for every observable on a case."""
obs = case.observables
for u in obs.urls:
yield normalize(u, "url"), "url"
for d in obs.domains:
yield normalize(d, "domain"), "domain"
for ip in obs.ips:
yield normalize(ip, "ip"), "ip"
for h in obs.hashes:
yield normalize(h, "hash"), "hash"
for c in obs.cves:
yield normalize(c, "cve"), "cve"
def reindex(cases: Iterable[Case]) -> int:
"""Rebuild the whole IOC index from the given cases. Returns rows written."""
rows: List[dict] = []
seen: set = set()
for case in cases:
feed = case.source_metadata.get("feed")
sev = case.classification.severity.value if case.classification.severity else None
first_seen = case.observed_at.isoformat() if case.observed_at else None
for value, ioc_type in iter_case_iocs(case):
if not value:
continue
key = (value, ioc_type, case.case_id)
if key in seen:
continue
seen.add(key)
rows.append(dict(
value=value, ioc_type=ioc_type, case_id=case.case_id,
feed=feed, severity=sev, first_seen=first_seen,
))
written = db.replace_iocs(rows)
_log.info("lookup.reindexed", iocs=written, cases=len(seen))
return written
def lookup(value: str) -> List[dict]:
"""Look up one indicator across all types. Returns matching index rows (may be empty)."""
# Try every type's normalization so callers don't need to know the type.
candidates = {normalize(value, t) for t in IOC_TYPES}
out: List[dict] = []
seen_ids: set = set()
for cand in candidates:
for row in db.find_iocs(cand):
if row["id"] not in seen_ids:
seen_ids.add(row["id"])
out.append(row)
return out
def export_blocklist(ioc_type: str, min_severity: Optional[str] = None) -> List[str]:
"""Distinct indicator values of one type, optionally filtered by min severity."""
if ioc_type not in IOC_TYPES:
raise ValueError(f"unknown ioc_type: {ioc_type}; choices: {', '.join(IOC_TYPES)}")
floor = _SEVERITY_RANK.get(min_severity, -1) if min_severity else -1
values: List[str] = []
seen: set = set()
for row in db.iocs_by_type(ioc_type):
if floor >= 0:
rank = _SEVERITY_RANK.get(row["severity"] or "", -1)
if rank < floor:
continue
v = row["value"]
if v not in seen:
seen.add(v)
values.append(v)
return values

File diff suppressed because it is too large Load Diff

174
src/psyc/lines/news.py Normal file
View File

@@ -0,0 +1,174 @@
"""Newsline — turn ledger + case activity into a human-readable digest.
Surfaces what psyc has *done* and what it has *seen* as a stream of news items
for the start page. Pure read aggregation over the existing case and ledger
stores — no new state.
"""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from typing import Dict, List, Optional
from pydantic import BaseModel
from psyc import db, log
from psyc.lines import ledger as ledger_line
from psyc.models import Case, LedgerEntry, Outcome, Severity
_log = log.get(__name__)
class NewsItem(BaseModel):
timestamp: datetime
kind: str # case | enforced | submitted | rejected | actioned | failed
headline: str
body: str
severity: Optional[str] = None
case_id: Optional[str] = None
icon: str = "" # tiny glyph for the card
class FeedHealth(BaseModel):
feed: str
count: int
latest: Optional[datetime] = None
# ---------- KPI strip ----------------------------------------------------
def kpis() -> Dict[str, int]:
"""Counters shown at the top of the home page."""
cases = db.list_cases(limit=10_000)
today = datetime.now(timezone.utc) - timedelta(hours=24)
new_24h = sum(1 for c in cases if c.ingested_at and c.ingested_at >= today)
high = sum(1 for c in cases
if c.classification.severity in (Severity.HIGH, Severity.CRITICAL))
ledger = ledger_line.list_recent(limit=10_000)
enforcements_24h = sum(1 for e in ledger if e.timestamp >= today and e.outcome is Outcome.ACTIONED)
return {
"cases": len(cases),
"iocs": db.ioc_count(),
"new_24h": new_24h,
"high_total": high,
"enforcements_24h": enforcements_24h,
"ledger_total": ledger_line.count(),
}
# ---------- news items ---------------------------------------------------
_OUTCOME_RENDER = {
Outcome.ACTIONED: ("", "enforced", "psyc enforced the response for {case}{dest}"),
Outcome.SUBMITTED: ("", "submitted", "Submitted to {dest} for {case}"),
Outcome.ACKNOWLEDGED:("", "submitted", "{dest} acknowledged the submission for {case}"),
Outcome.REJECTED: ("", "rejected", "Blocked / declined: {dest} for {case}"),
Outcome.FAILED: ("", "failed", "Delivery to {dest} failed for {case}"),
Outcome.PENDING_APPROVAL: ("", "pending", "Awaiting approval: {dest} for {case}"),
}
def _ledger_to_news(e: LedgerEntry) -> NewsItem:
icon, kind, fmt = _OUTCOME_RENDER.get(e.outcome, ("", "ledger", "Ledger event for {case}"))
headline = fmt.format(case=e.case_id, dest=e.destination)
body = e.detail or f"{e.outcome.value} · TLP:{e.tlp.value}"
return NewsItem(
timestamp=e.timestamp, kind=kind, headline=headline, body=body,
case_id=e.case_id, icon=icon,
)
def _case_to_news(c: Case) -> NewsItem:
sev = c.classification.severity.value if c.classification.severity else None
incident = c.classification.incident_type.value if c.classification.incident_type else "case"
feed = c.source_metadata.get("feed", "feed")
headline = f"New {sev or 'unrated'} {incident} from {feed}"
return NewsItem(
timestamp=c.ingested_at, kind="case",
headline=headline, body=c.summary[:200],
severity=sev, case_id=c.case_id,
icon={"critical": "🚨", "high": "", "medium": "", "low": "·"}.get(sev or "", ""),
)
def recent_items(limit: int = 30, high_only: bool = False) -> List[NewsItem]:
"""Interleave the latest ledger events with the latest case ingests, newest first."""
items: List[NewsItem] = []
for e in ledger_line.list_recent(limit=limit * 2):
items.append(_ledger_to_news(e))
cases = db.list_cases(limit=limit * 2)
for c in cases:
if high_only and c.classification.severity not in (Severity.HIGH, Severity.CRITICAL):
continue
items.append(_case_to_news(c))
items.sort(key=lambda i: i.timestamp, reverse=True)
return items[:limit]
# ---------- feed health (sidebar / footer of home) -----------------------
class Bucket(BaseModel):
label: str
items: List[NewsItem]
def bucket_items(items: List[NewsItem]) -> List[Bucket]:
"""Group items into Today / Yesterday / Earlier this week / Older."""
now = datetime.now(timezone.utc)
today_start = now.replace(hour=0, minute=0, second=0, microsecond=0)
yesterday_start = today_start - timedelta(days=1)
week_start = today_start - timedelta(days=7)
order = ("Today", "Yesterday", "Earlier this week", "Older")
buckets: Dict[str, List[NewsItem]] = {k: [] for k in order}
for i in items:
ts = i.timestamp
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
if ts >= today_start: buckets["Today"].append(i)
elif ts >= yesterday_start: buckets["Yesterday"].append(i)
elif ts >= week_start: buckets["Earlier this week"].append(i)
else: buckets["Older"].append(i)
return [Bucket(label=k, items=buckets[k]) for k in order if buckets[k]]
_SEV_RANK = {Severity.CRITICAL: 3, Severity.HIGH: 2, Severity.MEDIUM: 1, Severity.LOW: 0}
def featured_case() -> Optional[Case]:
"""Pick a case to spotlight: highest-severity from the last 7 days,
breaking ties by recency. Returns None if nothing HIGH+ in the window."""
now = datetime.now(timezone.utc)
week = now - timedelta(days=7)
candidates: List[Case] = []
for c in db.list_cases(limit=2000):
ts = c.ingested_at
if ts and ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
if ts and ts >= week and c.classification.severity in (Severity.HIGH, Severity.CRITICAL):
candidates.append(c)
if not candidates:
return None
candidates.sort(
key=lambda c: (_SEV_RANK.get(c.classification.severity, -1), c.ingested_at),
reverse=True,
)
return candidates[0]
def feed_health() -> List[FeedHealth]:
"""Per-feed counts + most-recent ingest. Useful as a 'sources live?' panel."""
cases = db.list_cases(limit=10_000)
buckets: Dict[str, FeedHealth] = {}
for c in cases:
feed = c.source_metadata.get("feed") or "unknown"
h = buckets.get(feed)
if h is None:
buckets[feed] = FeedHealth(feed=feed, count=1, latest=c.ingested_at)
else:
h.count += 1
if c.ingested_at and (h.latest is None or c.ingested_at > h.latest):
h.latest = c.ingested_at
out = list(buckets.values())
out.sort(key=lambda h: h.count, reverse=True)
return out

View File

@@ -23,9 +23,12 @@ _SHA_RE = re.compile(r"^[a-fA-F0-9]{32,64}$")
# feed -> (Admiralty source reliability A-F, information credibility 1-6)
_FEED_RELIABILITY = {
"cisa-kev": ("A", "1"), # government catalog, confirmed exploited
"urlhaus": ("B", "2"), # established CTI source, confirmed malware
"feodo": ("B", "2"), # established CTI source, confirmed C2
"cisa-kev": ("A", "1"), # government catalog, confirmed exploited
"urlhaus": ("B", "2"), # established CTI source, confirmed malware
"feodo": ("B", "2"), # established CTI source, confirmed C2
"threatfox": ("B", "2"), # abuse.ch CTI source
"malware-bazaar": ("B", "2"), # abuse.ch CTI source, confirmed sample
"otx": ("C", "3"), # community-driven, varying quality
}

682
src/psyc/lines/pulse.py Normal file
View File

@@ -0,0 +1,682 @@
"""Pulseline — cron-style scheduler that drives every psyc pipeline on a cadence.
Each registered pipeline has an autonomy mode (manual / auto-propose /
auto-execute) and a cadence in seconds. tick() iterates every pipeline and
fires whichever ones are due (and enabled, and not manual, and the global kill
switch is off). State persists to SQLite via psyc.db so cadences survive
restarts. A background asyncio loop calls tick() at a fixed interval — the
cockpit lifespan attaches it.
NOTE: federation pipelines (peer-pull, vouch-refresh) are wired as placeholders
that return a no-op string. Real federation lands in a later stage.
"""
from __future__ import annotations
import asyncio
import traceback
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Callable, Dict, List, Optional, Tuple
from pydantic import BaseModel, Field
from psyc import db, log
from psyc.models import Severity
_log = log.get(__name__)
class PulseMode(str, Enum):
MANUAL = "manual"
AUTO_PROPOSE = "auto-propose"
AUTO_EXECUTE = "auto-execute"
# ---------- respond auto-fire gates -----------------------------------------
# Persisted as rows in pulse_settings (key/value pairs). All defaults are
# "safe" — quorum required, HIGH threshold, federation cases permitted only
# when quorum-met.
_KEY_RESPOND_THRESHOLD = "respond_auto_threshold"
_KEY_RESPOND_REQUIRE_QUORUM = "respond_require_quorum"
_KEY_RESPOND_LOCAL_ONLY = "respond_local_only"
_DEFAULT_THRESHOLD = Severity.HIGH
_DEFAULT_REQUIRE_QUORUM = True
_DEFAULT_LOCAL_ONLY = False
def _severity_rank(sev: Optional[Severity]) -> int:
"""Rank order for severity threshold comparison. Unknown / None → -1."""
if sev is None:
return -1
return {
Severity.LOW: 0,
Severity.MEDIUM: 1,
Severity.HIGH: 2,
Severity.CRITICAL: 3,
}.get(sev, -1)
def respond_auto_threshold() -> Severity:
raw = db.pulse_setting_get(_KEY_RESPOND_THRESHOLD)
if raw is None:
return _DEFAULT_THRESHOLD
try:
return Severity(raw)
except ValueError:
return _DEFAULT_THRESHOLD
def set_respond_auto_threshold(sev: Severity) -> None:
if not isinstance(sev, Severity):
raise ValueError(f"not a Severity: {sev!r}")
db.pulse_setting_set(_KEY_RESPOND_THRESHOLD, sev.value)
_log.info("pulse.respond.threshold.changed", severity=sev.value)
def respond_require_quorum() -> bool:
raw = db.pulse_setting_get(_KEY_RESPOND_REQUIRE_QUORUM)
if raw is None:
return _DEFAULT_REQUIRE_QUORUM
return raw == "1"
def set_respond_require_quorum(state: bool) -> None:
db.pulse_setting_set(_KEY_RESPOND_REQUIRE_QUORUM, "1" if state else "0")
_log.info("pulse.respond.quorum.changed", required=bool(state))
def respond_local_only() -> bool:
raw = db.pulse_setting_get(_KEY_RESPOND_LOCAL_ONLY)
if raw is None:
return _DEFAULT_LOCAL_ONLY
return raw == "1"
def set_respond_local_only(state: bool) -> None:
db.pulse_setting_set(_KEY_RESPOND_LOCAL_ONLY, "1" if state else "0")
_log.info("pulse.respond.local-only.changed", local_only=bool(state))
class Pipeline(BaseModel):
name: str
title: str
description: str
mode: PulseMode
cadence_seconds: int
enabled: bool = True
last_fired: Optional[datetime] = None
next_fire: Optional[datetime] = None
last_result: str = ""
last_outcome: str = "" # "ok" | "err" | "skipped" | ""
# ---------- pipeline runners --------------------------------------------------
def _run_fetch() -> str:
"""Fetch every enabled scout source; partial fetch is fine.
Skip-on-fail is critical: keyed feeds (threatfox, malware-bazaar, otx)
raise when their key isn't configured — we don't want one missing key to
block the public ones.
"""
from psyc.lines import scout
plan: Tuple[Tuple[str, Optional[int]], ...] = (
("urlhaus", 50),
("cisa-kev", 100),
("feodo", 50),
("threatfox", 200),
("malware-bazaar", 100),
("otx", 100),
)
total = 0
feeds_ok = 0
feeds_err: List[str] = []
for source, limit in plan:
try:
cases = scout.fetch_and_signal(source, limit=limit)
for c in cases:
db.upsert_case(c)
total += len(cases)
feeds_ok += 1
except Exception as exc: # noqa: BLE001 — partial fetch is the point
feeds_err.append(source)
_log.info("pulse.fetch.skip", source=source, error=str(exc)[:200])
tail = f" (skipped: {', '.join(feeds_err)})" if feeds_err else ""
return f"fetched {total} cases across {feeds_ok} feed(s){tail}"
def _run_classify() -> str:
from psyc.lines import classify
cases = db.list_cases(limit=10_000)
n = 0
for c in cases:
classify.classify(c)
db.upsert_case(c)
n += 1
return f"classified {n} case(s)"
def _run_prove() -> str:
from psyc.lines import proof
cases = db.list_cases(limit=10_000)
n = 0
for c in cases:
proof.prove(c)
db.upsert_case(c)
n += 1
return f"proved {n} case(s)"
def _run_reindex() -> str:
from psyc.lines import lookup
cases = db.list_cases(limit=1_000_000)
written = lookup.reindex(cases)
return f"indexed {written} IOC(s) from {len(cases)} case(s)"
def _propose_for_recent_cases() -> int:
"""Propose response actions for high-severity cases that don't yet have any.
Returns total proposed-action count. Idempotent per case (respond's
propose_for_case skips cases that already have actions).
"""
from psyc.lines import respond
cases = db.list_cases(limit=10_000)
proposed = 0
for c in cases:
ids = respond.propose_for_case(c)
proposed += len(ids)
return proposed
def _current_mode(pipeline_name: str) -> PulseMode:
p = _get_pipeline(pipeline_name)
return p.mode if p is not None else PulseMode.MANUAL
def _is_quorum_met(case_digest_hash: str) -> bool:
"""Wrapper for federation.is_quorum_met that tolerates the sibling agent
not having shipped the function yet.
If federation lacks `is_quorum_met`, we fall back to False — the safe
default ("no quorum signal → don't fire federation cases").
"""
try:
from psyc.lines import federation as _federation
fn = getattr(_federation, "is_quorum_met", None)
if fn is None:
return False
return bool(fn(case_digest_hash))
except Exception as exc: # noqa: BLE001 — defensive: any import / runtime miss → safe-false
_log.warning("pulse.respond.quorum.unavailable", error=str(exc))
return False
def _canonical_json_local(obj: Dict[str, object]) -> bytes:
"""Deterministic JSON serialization — mirrors federation.canonical_json
for the case-digest computation. Local copy so we don't hard-require
federation to be importable.
"""
import json
return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
def _case_digest_hash(case_id: str) -> str:
"""SHA-256 of the canonical JSON of {case_id: ...} — what federation hashes.
Returns "" if the case can't be loaded (e.g. row vanished mid-fire).
"""
import hashlib
from psyc.result import Ok as _Ok
got = db.get_case(case_id)
if not isinstance(got, _Ok):
return ""
case = got.value
# Mirror federation._build_case_records' record shape so digests match.
record = {
"case_id": case.case_id,
"summary": case.summary,
"severity": case.classification.severity.value if case.classification.severity else None,
"incident_type": case.classification.incident_type.value if case.classification.incident_type else None,
"observed_at": case.observed_at.isoformat(),
"feed_source": case.source_metadata.get("feed", ""),
"iocs": (
[{"value": v, "type": "url"} for v in case.observables.urls]
+ [{"value": v, "type": "domain"} for v in case.observables.domains]
+ [{"value": v, "type": "ip"} for v in case.observables.ips]
+ [{"value": v, "type": "hash"} for v in case.observables.hashes]
+ [{"value": v, "type": "cve"} for v in case.observables.cves]
),
}
return hashlib.sha256(_canonical_json_local(record)).hexdigest()
def _case_is_local(case_id: str) -> bool:
"""True if no federation peer has ever pushed us this case_id."""
return len(db.signals_for_case(case_id)) == 0
def _audit(action: str, *, action_id: Optional[int] = None,
case_id: Optional[str] = None, detail: str = "") -> None:
db.pulse_audit_record(dict(
pipeline="respond",
action=action,
action_id=action_id,
case_id=case_id,
detail=detail[:500],
timestamp=_now().isoformat(),
))
def _auto_fire_eligible() -> Tuple[int, int]:
"""Iterate PROPOSED actions and execute the ones that clear every gate.
Returns (fired_count, skipped_count). Records a pulse_audit row for every
decision (fired or skipped-with-reason) so the cockpit can show history.
A single failing action never aborts the batch.
"""
from psyc.lines import respond
from psyc.models import ActionStatus
from psyc.result import Ok as _Ok
threshold = respond_auto_threshold()
threshold_rank = _severity_rank(threshold)
require_quorum = respond_require_quorum()
local_only = respond_local_only()
fired = 0
skipped = 0
actions = respond.list_actions(status=ActionStatus.PROPOSED, limit=100)
for action in actions:
# Re-hydrate severity enum (action.severity is the .value string).
try:
sev_enum = Severity(action.severity) if action.severity else None
except ValueError:
sev_enum = None
if _severity_rank(sev_enum) < threshold_rank:
skipped += 1
_audit(
"skipped",
action_id=action.id,
case_id=action.case_id,
detail=f"below threshold: severity={action.severity!r} < {threshold.value}",
)
continue
is_local = _case_is_local(action.case_id)
if require_quorum and not is_local:
digest = _case_digest_hash(action.case_id)
if not digest or not _is_quorum_met(digest):
if local_only:
# local-only is armed but this case was imported via federation
# → defer (don't fire) until federation grants quorum
skipped += 1
_audit(
"skipped",
action_id=action.id,
case_id=action.case_id,
detail="local-only armed + federation-sourced case",
)
continue
skipped += 1
_audit(
"skipped",
action_id=action.id,
case_id=action.case_id,
detail="no quorum on federation-sourced case",
)
continue
# else: quorum disabled, or case is locally-generated → fire.
try:
result = respond.execute_action(action.id, approver="pulse-auto")
except Exception as exc: # noqa: BLE001 — one bad action shouldn't kill the batch
skipped += 1
_audit(
"error",
action_id=action.id,
case_id=action.case_id,
detail=f"execute raised: {type(exc).__name__}: {exc}",
)
_log.warning("pulse.respond.auto-fire.error",
action_id=action.id, error=str(exc))
continue
if isinstance(result, _Ok):
fired += 1
_log.info("pulse.respond.auto-fire",
action_id=action.id, case_id=action.case_id,
type=action.action_type.value, target=action.target)
_audit(
"auto-fire",
action_id=action.id,
case_id=action.case_id,
detail=f"{action.action_type.value}{action.target}",
)
else:
# Err path — execute_action returned Err (e.g. SOAR sink down)
reason = getattr(result, "reason", "unknown")
skipped += 1
_audit(
"error",
action_id=action.id,
case_id=action.case_id,
detail=f"execute failed: {reason}",
)
_log.warning("pulse.respond.auto-fire.failed",
action_id=action.id, reason=str(reason))
return fired, skipped
def _run_respond() -> str:
"""Propose + (when mode is auto-execute) auto-fire eligible PROPOSED actions.
Two phases:
1. Always propose new actions for high-severity cases (existing behavior).
2. If pipeline mode is auto-execute, iterate PROPOSED actions and execute
those that clear severity/quorum/local-only gates.
"""
propose_count = _propose_for_recent_cases()
mode = _current_mode("respond")
if mode != PulseMode.AUTO_EXECUTE:
return f"proposed {propose_count} actions; mode={mode.value} → no auto-fire"
fired, skipped = _auto_fire_eligible()
return f"proposed {propose_count}; auto-fired {fired}; skipped {skipped} (gate)"
_DISCOVERY_SEEDS_KEY = "discovery_seeds"
def get_discovery_seeds() -> List[str]:
"""Operator-curated seed list for the discovery walker. Newline-separated in DB."""
raw = db.pulse_setting_get(_DISCOVERY_SEEDS_KEY)
if not raw:
return []
return [line.strip() for line in raw.splitlines() if line.strip()]
def set_discovery_seeds(seeds: List[str]) -> None:
"""Replace the seed list. Strips blanks + dedupes preserving order."""
seen: set = set()
cleaned: List[str] = []
for s in seeds:
v = (s or "").strip()
if not v or v in seen:
continue
seen.add(v)
cleaned.append(v)
db.pulse_setting_set(_DISCOVERY_SEEDS_KEY, "\n".join(cleaned))
def _run_peer_pull() -> str:
"""Walk DNS-SD + recurse over peer-public lists from the operator's seeds.
Records every fresh candidate into the `peers` table with status=unknown.
Vouching (sibling stage) is what eventually promotes them.
"""
from psyc.lines import discovery
seeds = get_discovery_seeds()
if not seeds:
return "no seeds configured"
candidates = discovery.walk(seeds)
for c in candidates:
try:
discovery.record_candidate(c)
except Exception as exc: # noqa: BLE001 — one bad write must not abort the batch
_log.warning("pulse.peer_pull.record.error", domain=c.domain, error=str(exc))
return f"discovered {len(candidates)} candidate(s) from {len(seeds)} seed(s)"
def _run_vouch_refresh() -> str:
return "federation not yet active"
# ---------- registry ----------------------------------------------------------
_REGISTRY: Dict[str, Callable[[], str]] = {
"fetch": _run_fetch,
"classify": _run_classify,
"prove": _run_prove,
"reindex": _run_reindex,
"respond": _run_respond,
"peer-pull": _run_peer_pull,
"vouch-refresh": _run_vouch_refresh,
}
# Initial defaults — seeded once on first DB init. Tuples of
# (name, title, description, mode, cadence_seconds, enabled).
_DEFAULTS: Tuple[Tuple[str, str, str, PulseMode, int, bool], ...] = (
("fetch", "Scout · fetch feeds", "Pull every configured threat feed and ingest new cases.",
PulseMode.AUTO_EXECUTE, 900, True),
("classify", "Classify · label cases", "Assign incident type, severity, TLP, and internal class to every case.",
PulseMode.AUTO_EXECUTE, 300, True),
("prove", "Proof · score confidence", "Compute confidence (reliability · credibility · freshness) for every case.",
PulseMode.AUTO_EXECUTE, 300, True),
("reindex", "Lookup · rebuild IOC index","Rebuild the IOC reverse-index over the case corpus.",
PulseMode.AUTO_EXECUTE, 3600, True),
("respond", "Respond · propose actions", "Propose human-gated response actions for newly high-severity cases.",
PulseMode.AUTO_PROPOSE, 600, True),
("peer-pull", "Federation · peer pull", "(placeholder) Pull sealed cases from federated peers.",
PulseMode.MANUAL, 600, False),
("vouch-refresh","Federation · vouch refresh","(placeholder) Refresh per-peer vouching ledgers.",
PulseMode.MANUAL, 3600, False),
)
# ---------- helpers -----------------------------------------------------------
def _now() -> datetime:
return datetime.now(timezone.utc)
def _parse_dt(value: Optional[str]) -> Optional[datetime]:
if not value:
return None
try:
return datetime.fromisoformat(value)
except ValueError:
return None
def _row_to_pipeline(row: dict) -> Pipeline:
return Pipeline(
name=row["name"],
title=row["title"],
description=row["description"],
mode=PulseMode(row["mode"]),
cadence_seconds=int(row["cadence_seconds"]),
enabled=bool(row["enabled"]),
last_fired=_parse_dt(row.get("last_fired")),
next_fire=_parse_dt(row.get("next_fire")),
last_result=row.get("last_result") or "",
last_outcome=row.get("last_outcome") or "",
)
def _pipeline_to_row(p: Pipeline) -> dict:
return dict(
name=p.name,
title=p.title,
description=p.description,
mode=p.mode.value,
cadence_seconds=int(p.cadence_seconds),
enabled=bool(p.enabled),
last_fired=p.last_fired.isoformat() if p.last_fired else None,
next_fire=p.next_fire.isoformat() if p.next_fire else None,
last_result=p.last_result or "",
last_outcome=p.last_outcome or "",
)
def seed_defaults() -> None:
"""Insert any default pipelines that aren't already in the DB. Idempotent."""
existing = {row["name"] for row in db.get_pulse_state()}
for name, title, desc, mode, cadence, enabled in _DEFAULTS:
if name in existing:
continue
p = Pipeline(
name=name, title=title, description=desc,
mode=mode, cadence_seconds=cadence, enabled=enabled,
next_fire=_now(), # first tick after install fires immediately if due
)
db.upsert_pulse_pipeline(_pipeline_to_row(p))
_log.info("pulse.defaults.seeded", count=len(_DEFAULTS))
def state() -> List[Pipeline]:
"""Every pipeline, ordered by name. Seeds defaults on the first call."""
rows = db.get_pulse_state()
if not rows:
seed_defaults()
rows = db.get_pulse_state()
return [_row_to_pipeline(r) for r in rows]
def _get_pipeline(name: str) -> Optional[Pipeline]:
for p in state():
if p.name == name:
return p
return None
def set_mode(name: str, mode: PulseMode) -> None:
p = _get_pipeline(name)
if p is None:
raise ValueError(f"unknown pipeline: {name}")
p.mode = mode
db.upsert_pulse_pipeline(_pipeline_to_row(p))
def set_cadence(name: str, seconds: int) -> None:
if seconds <= 0:
raise ValueError("cadence must be > 0 seconds")
p = _get_pipeline(name)
if p is None:
raise ValueError(f"unknown pipeline: {name}")
p.cadence_seconds = int(seconds)
db.upsert_pulse_pipeline(_pipeline_to_row(p))
def set_enabled(name: str, enabled: bool) -> None:
p = _get_pipeline(name)
if p is None:
raise ValueError(f"unknown pipeline: {name}")
p.enabled = bool(enabled)
db.upsert_pulse_pipeline(_pipeline_to_row(p))
def set_kill_switch(armed: bool) -> None:
db.kill_switch_set(armed)
_log.warning("pulse.killswitch.changed", armed=bool(armed))
def kill_switch_state() -> bool:
return db.kill_switch_get()
# ---------- the heartbeat -----------------------------------------------------
def _fire(p: Pipeline) -> Tuple[str, str]:
"""Run one pipeline. Returns (outcome, result_str). Persists the timestamps.
Outcome is "ok" if the runner returned, "err" if it raised.
"""
runner = _REGISTRY.get(p.name)
if runner is None:
outcome = "err"
result = f"no runner registered for '{p.name}'"
else:
try:
result = runner() or ""
outcome = "ok"
except Exception as exc: # noqa: BLE001 — log + record, don't crash the loop
outcome = "err"
result = f"{type(exc).__name__}: {exc}"
_log.warning("pulse.fire.error", name=p.name, error=result, trace=traceback.format_exc())
now = _now()
p.last_fired = now
p.next_fire = now + timedelta(seconds=max(1, p.cadence_seconds))
p.last_result = result[:500]
p.last_outcome = outcome
db.upsert_pulse_pipeline(_pipeline_to_row(p))
_log.info("pulse.fired", name=p.name, outcome=outcome, result=p.last_result)
return outcome, p.last_result
def _should_fire(p: Pipeline, now: datetime) -> bool:
if not p.enabled:
return False
if p.mode == PulseMode.MANUAL:
return False
if p.next_fire is None:
return True
return now >= p.next_fire
def tick() -> List[Tuple[str, str, str]]:
"""Single scheduler heartbeat. Returns (name, outcome, result_str) per pipeline.
Outcome is "ok" / "err" / "skipped" — every registered pipeline appears in
the return value so callers can see what was skipped and why.
"""
if kill_switch_state():
_log.info("pulse.tick.killed")
return [(p.name, "skipped", "kill switch armed") for p in state()]
now = _now()
out: List[Tuple[str, str, str]] = []
for p in state():
if not _should_fire(p, now):
out.append((p.name, "skipped", "not due"))
continue
outcome, result = _fire(p)
out.append((p.name, outcome, result))
return out
def run_now(name: str) -> Tuple[str, str]:
"""Manually fire one pipeline, bypassing cadence and mode. Honors kill switch.
Returns (outcome, result_str). Raises ValueError on unknown name.
"""
if kill_switch_state():
return ("skipped", "kill switch armed")
p = _get_pipeline(name)
if p is None:
raise ValueError(f"unknown pipeline: {name}")
return _fire(p)
# ---------- background loop ---------------------------------------------------
async def start_background_loop(interval_seconds: int = 30) -> None:
"""Long-running scheduler — calls tick() every interval. Launched from FastAPI lifespan.
Designed to run for the life of the process; cancellation is the normal stop signal.
"""
_log.info("pulse.loop.starting", interval=interval_seconds)
while True:
try:
tick()
except Exception as exc: # noqa: BLE001 — one bad tick must not kill the loop
_log.warning("pulse.loop.tick.error", error=str(exc), trace=traceback.format_exc())
try:
await asyncio.sleep(interval_seconds)
except asyncio.CancelledError:
_log.info("pulse.loop.cancelled")
raise

211
src/psyc/lines/respond.py Normal file
View File

@@ -0,0 +1,211 @@
"""Respondline — human-gated response actions (SOAR-lite).
High-severity cases propose response actions (alert the SOC, push IOCs to
enforcement, open a ticket). Nothing fires automatically: each action sits in
PROPOSED until a human approves it, mirroring the submission approval gate.
On approval the action is dispatched to the configured enforcement sink
(PSYC_SOAR_URL, default = the mock-cert container) and recorded in the ledger.
"""
from __future__ import annotations
import json
import os
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
import httpx
from sqlalchemy import func as sa_func
from sqlalchemy import select, update
from psyc import db, log
from psyc.lines import ledger as ledger_line
from psyc.models import ActionStatus, ActionType, Case, Outcome, ResponseAction, Severity, TLP
from psyc.result import Err, Ok, Result
_log = log.get(__name__)
SOAR_BASE = os.environ.get("PSYC_SOAR_URL", "http://127.0.0.1:8770")
SOAR_ENDPOINT = f"{SOAR_BASE}/soar/enforce"
DEFAULT_TIMEOUT = 10.0
APPROVER_DEFAULT = "operator"
# Only act on cases this severe or worse.
_ACTIONABLE = {Severity.HIGH, Severity.CRITICAL}
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def propose_for_case(case: Case) -> List[int]:
"""Generate response actions for a high-severity case. Returns new action ids.
Idempotent per case: if actions already exist for this case, propose none.
"""
if case.classification.severity not in _ACTIONABLE:
return []
if _action_count_for_case(case.case_id) > 0:
return []
sev = case.classification.severity.value
obs = case.observables
ioc_total = len(obs.ips) + len(obs.domains) + len(obs.urls) + len(obs.hashes)
actions: List[ResponseAction] = []
# 1. Alert the SOC.
actions.append(ResponseAction(
case_id=case.case_id,
action_type=ActionType.ALERT,
target="soc-webhook",
summary=f"Alert SOC: {sev.upper()} {case.classification.incident_type.value if case.classification.incident_type else 'threat'}{case.summary[:80]}",
payload_json=json.dumps({
"kind": "alert", "case_id": case.case_id, "severity": sev,
"summary": case.summary, "ioc_count": ioc_total,
}, ensure_ascii=False),
severity=sev,
created_at=datetime.now(timezone.utc),
))
# 2. Push IOCs to enforcement, if there are any network indicators.
if obs.ips or obs.domains or obs.urls:
actions.append(ResponseAction(
case_id=case.case_id,
action_type=ActionType.BLOCKLIST,
target="perimeter-firewall+dns",
summary=f"Block {len(obs.ips)} IP(s), {len(obs.domains)} domain(s), {len(obs.urls)} URL(s) at the perimeter",
payload_json=json.dumps({
"kind": "blocklist", "case_id": case.case_id, "severity": sev,
"ips": obs.ips, "domains": obs.domains, "urls": obs.urls,
}, ensure_ascii=False),
severity=sev,
created_at=datetime.now(timezone.utc),
))
ids: List[int] = []
with db.engine().begin() as conn:
for a in actions:
res = conn.execute(db.response_actions.insert().values(
case_id=a.case_id, action_type=a.action_type.value, target=a.target,
summary=a.summary, payload_json=a.payload_json, severity=a.severity,
status=ActionStatus.PROPOSED.value, created_at=a.created_at.isoformat(),
))
ids.append(int(res.inserted_primary_key[0]))
_log.info("respond.proposed", case_id=case.case_id, actions=len(ids))
return ids
def _action_count_for_case(case_id: str) -> int:
stmt = select(sa_func.count()).select_from(db.response_actions).where(db.response_actions.c.case_id == case_id)
with db.engine().connect() as conn:
return int(conn.execute(stmt).scalar_one())
def _row_to_action(row: Any) -> ResponseAction:
return ResponseAction(
id=row.id, case_id=row.case_id, action_type=ActionType(row.action_type),
target=row.target, summary=row.summary, payload_json=row.payload_json,
severity=row.severity, status=ActionStatus(row.status),
created_at=datetime.fromisoformat(row.created_at),
approver=row.approver,
executed_at=datetime.fromisoformat(row.executed_at) if row.executed_at else None,
detail=row.detail,
)
def list_actions(status: Optional[ActionStatus] = None, limit: int = 200) -> List[ResponseAction]:
stmt = select(db.response_actions)
if status is not None:
stmt = stmt.where(db.response_actions.c.status == status.value)
stmt = stmt.order_by(db.response_actions.c.created_at.desc()).limit(limit)
with db.engine().connect() as conn:
return [_row_to_action(r) for r in conn.execute(stmt).fetchall()]
def get_action(action_id: int) -> Result[ResponseAction, str]:
stmt = select(db.response_actions).where(db.response_actions.c.id == action_id)
with db.engine().connect() as conn:
row = conn.execute(stmt).fetchone()
if row is None:
return Err(f"action not found: {action_id}")
return Ok(_row_to_action(row))
def action_count(status: ActionStatus) -> int:
stmt = select(sa_func.count()).select_from(db.response_actions).where(db.response_actions.c.status == status.value)
with db.engine().connect() as conn:
return int(conn.execute(stmt).scalar_one())
def execute_action(action_id: int, approver: str = APPROVER_DEFAULT) -> Result[ResponseAction, str]:
"""Approve + fire an action: POST to the enforcement sink, ledger it, mark executed."""
got = get_action(action_id)
if isinstance(got, Err):
return Err(got.reason)
a = got.value
if a.status != ActionStatus.PROPOSED:
return Err(f"action {action_id} is already {a.status.value}")
payload = json.loads(a.payload_json)
payload["action_type"] = a.action_type.value
payload["approved_by"] = approver
try:
with httpx.Client(timeout=DEFAULT_TIMEOUT) as client:
resp = client.post(SOAR_ENDPOINT, json=payload)
resp.raise_for_status()
body = resp.json()
receipt = str(body.get("receipt_id", ""))
ok = True
detail = f"enforced_by={approver}{receipt}"
except Exception as exc: # noqa: BLE001 — network/sink failure is expected-path
ok = False
detail = f"enforcement failed: {exc}"
_log.warning("respond.execute.error", action_id=action_id, error=str(exc))
now = _now()
new_status = ActionStatus.EXECUTED if ok else ActionStatus.FAILED
ledger_line.write(
case_id=a.case_id,
destination=f"SOAR:{a.action_type.value}:{a.target}",
payload_hash="",
submitter_identity="psyc/respond@0.1",
tlp=TLP.AMBER,
outcome=Outcome.ACTIONED if ok else Outcome.FAILED,
detail=detail,
)
with db.engine().begin() as conn:
conn.execute(update(db.response_actions).where(db.response_actions.c.id == action_id).values(
status=new_status.value, approver=approver, executed_at=now, detail=detail,
))
_log.info("respond.executed", action_id=action_id, ok=ok, approver=approver)
if not ok:
return Err(detail)
refreshed = get_action(action_id)
return refreshed if isinstance(refreshed, Ok) else Err("post-execute read failed")
def reject_action(action_id: int, approver: str = APPROVER_DEFAULT, reason: str = "") -> Result[None, str]:
"""Decline a proposed action — nothing fires; ledger records the decision."""
got = get_action(action_id)
if isinstance(got, Err):
return Err(got.reason)
a = got.value
if a.status != ActionStatus.PROPOSED:
return Err(f"action {action_id} is already {a.status.value}")
ledger_line.write(
case_id=a.case_id,
destination=f"SOAR:{a.action_type.value}:{a.target}",
payload_hash="",
submitter_identity="psyc/respond@0.1",
tlp=TLP.AMBER,
outcome=Outcome.REJECTED,
detail=f"declined_by={approver}: {reason}" if reason else f"declined_by={approver}",
)
with db.engine().begin() as conn:
conn.execute(update(db.response_actions).where(db.response_actions.c.id == action_id).values(
status=ActionStatus.REJECTED.value, approver=approver, executed_at=_now(),
detail=reason or None,
))
_log.info("respond.rejected", action_id=action_id, approver=approver)
return Ok(None)

View File

@@ -38,6 +38,7 @@ class Destination(BaseModel):
priority: int
payload_kind: str
countries: List[str] = Field(default_factory=list)
requires_approval: bool = False
class Route(BaseModel):
@@ -45,6 +46,7 @@ class Route(BaseModel):
priority: int
payload_kind: str
max_tlp_allowed: TLP
requires_approval: bool = False
class BlockedRoute(BaseModel):
@@ -61,6 +63,7 @@ DESTINATIONS: List[Destination] = [
priority=1,
payload_kind="sealed_evidence_package",
countries=["DE"],
requires_approval=True,
),
Destination(
name="MISP-Community",
@@ -111,6 +114,7 @@ def plan(case: Case) -> Tuple[List[Route], List[BlockedRoute]]:
priority=d.priority,
payload_kind=d.payload_kind,
max_tlp_allowed=d.max_tlp,
requires_approval=d.requires_approval,
))
routes.sort(key=lambda r: r.priority)
_log.info("route.planned", case_id=case.case_id, allowed=len(routes), blocked=len(blocked))

View File

@@ -10,33 +10,64 @@ from __future__ import annotations
import csv
import io
from datetime import datetime, timezone
from typing import Callable, Dict, Iterable, List, Optional
import os
from datetime import datetime, timedelta, timezone
from typing import Any, Callable, Dict, Iterable, List, Optional
from urllib.parse import urlparse
import httpx
from psyc import log
from psyc.models import Case, Observables
from psyc.models import Case, IncidentType, Observables
USER_AGENT = "psyc/0.1 (defensive CTI; hackathon prototype)"
# CISA's CDN 403s "exotic" UAs from some IPs; a Mozilla-compatible identifier
# is universally accepted and still identifies us honestly. Overridable via env
# if a feed ever wants a specific UA.
USER_AGENT = os.environ.get(
"PSYC_HTTP_USER_AGENT",
"Mozilla/5.0 (compatible; psyc/0.1; +https://psyc.neuronetz.ai)",
)
HTTP_TIMEOUT = 30.0
URLHAUS_RECENT_CSV = "https://urlhaus.abuse.ch/downloads/csv_recent/"
CISA_KEV_JSON = "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"
FEODO_BLOCKLIST_JSON = "https://feodotracker.abuse.ch/downloads/ipblocklist.json"
THREATFOX_API = "https://threatfox-api.abuse.ch/api/v1/"
MALWARE_BAZAAR_API = "https://mb-api.abuse.ch/api/v1/"
OTX_PULSES_API = "https://otx.alienvault.com/api/v1/pulses/subscribed"
_log = log.get(__name__)
def _http_get(url: str) -> httpx.Response:
with httpx.Client(timeout=HTTP_TIMEOUT, headers={"User-Agent": USER_AGENT}, follow_redirects=True) as client:
resp = client.get(url)
def _http(
method: str,
url: str,
headers: Optional[Dict[str, str]] = None,
json_body: Optional[Dict[str, Any]] = None,
form_body: Optional[Dict[str, Any]] = None,
params: Optional[Dict[str, Any]] = None,
timeout: float = HTTP_TIMEOUT,
) -> httpx.Response:
h = {"User-Agent": USER_AGENT}
if headers:
h.update(headers)
with httpx.Client(timeout=timeout, headers=h, follow_redirects=True) as client:
if method.upper() == "POST":
if form_body is not None:
resp = client.post(url, data=form_body, params=params)
else:
resp = client.post(url, json=json_body, params=params)
else:
resp = client.get(url, params=params)
resp.raise_for_status()
return resp
def _http_get(url: str) -> httpx.Response:
return _http("GET", url)
def _parse_dt(value: str, fmt: str) -> datetime:
try:
return datetime.strptime(value, fmt).replace(tzinfo=timezone.utc)
@@ -142,12 +173,221 @@ def _fetch_feodo() -> List[Case]:
return [_feodo_record_to_case(r) for r in data]
# --- ThreatFox — multi-malware IOC feed (abuse.ch) -----------------------
# ThreatFox threat_type values → psyc IncidentType.
THREATFOX_THREAT_TYPE: Dict[str, IncidentType] = {
"botnet_cc": IncidentType.BOTNET,
"payload_delivery": IncidentType.MALWARE,
"payload": IncidentType.MALWARE,
"phishing": IncidentType.PHISHING,
}
def _threatfox_row_to_case(r: Dict[str, Any]) -> Optional[Case]:
# API field is `ioc` (the `_value` alias is older docs); date is `first_seen`.
ioc_value = str(r.get("ioc") or r.get("ioc_value") or "").strip()
ioc_type = str(r.get("ioc_type") or "").lower()
if not ioc_value or not ioc_type:
return None
malware = str(r.get("malware_printable") or r.get("malware") or "unknown")
threat_type = str(r.get("threat_type") or "")
tags_raw = r.get("tags") or []
tags = tags_raw if isinstance(tags_raw, list) else []
obs = Observables()
host = ""
if ioc_type in ("ip:port", "ipv4", "ipv6"):
ip = ioc_value.split(":")[0]
obs.ips = [ip]
elif ioc_type == "domain":
obs.domains = [ioc_value]
host = ioc_value
elif ioc_type == "url":
obs.urls = [ioc_value]
host = urlparse(ioc_value).hostname or ""
if host:
obs.domains = [host]
elif ioc_type in ("sha256_hash", "md5_hash", "sha1_hash"):
obs.hashes = [ioc_value]
else:
return None
threat_label = threat_type.replace("_", " ") or "malware"
summary = f"ThreatFox: {malware} {threat_label}{ioc_value}"
first_seen = str(r.get("first_seen") or r.get("first_seen_utc") or "")
return Case(
case_id=f"PSYC-THREATFOX-{r.get('id', '')}",
summary=summary,
source_type="abuse_feed",
source_ref=str(r.get("reference") or f"https://threatfox.abuse.ch/ioc/{r.get('id', '')}/"),
source_metadata=dict(
feed="threatfox",
malware=malware,
malware_malpedia=str(r.get("malware_malpedia") or ""),
threat_type=threat_type,
threat_type_desc=str(r.get("threat_type_desc") or ""),
ioc_type=ioc_type,
confidence_level=str(r.get("confidence_level", "")),
tags=",".join(t for t in tags if t),
reporter=str(r.get("reporter", "")),
),
observed_at=_parse_dt(first_seen, "%Y-%m-%d %H:%M:%S"),
observables=obs,
)
def _fetch_threatfox() -> List[Case]:
key = os.environ.get("THREATFOX_AUTH_KEY", "").strip()
if not key:
raise RuntimeError("THREATFOX_AUTH_KEY not set — free abuse.ch auth-key from https://auth.abuse.ch/")
data = _http("POST", THREATFOX_API, headers={"Auth-Key": key}, json_body={"query": "get_iocs", "days": 1}).json()
rows = data.get("data") or []
out: List[Case] = []
for r in rows:
c = _threatfox_row_to_case(r)
if c is not None:
out.append(c)
return out
# --- MalwareBazaar — recent malware samples (abuse.ch) -------------------
def _mb_row_to_case(r: Dict[str, Any]) -> Optional[Case]:
sha256 = str(r.get("sha256_hash") or "")
if not sha256:
return None
sha1 = str(r.get("sha1_hash") or "")
md5 = str(r.get("md5_hash") or "")
file_name = str(r.get("file_name") or "unknown")
signature = str(r.get("signature") or "")
file_type = str(r.get("file_type") or "")
tags_raw = r.get("tags") or []
tags = tags_raw if isinstance(tags_raw, list) else []
hashes = [h for h in (sha256, sha1, md5) if h]
label = signature or "unsigned"
summary = f"MalwareBazaar: {label} {file_type} sample — {file_name}"
return Case(
case_id=f"PSYC-MBAZAAR-{sha256[:16]}",
summary=summary,
source_type="abuse_feed",
source_ref=f"https://bazaar.abuse.ch/sample/{sha256}/",
source_metadata=dict(
feed="malware-bazaar",
signature=signature,
file_type=file_type,
file_name=file_name,
tags=",".join(t for t in tags if t),
reporter=str(r.get("reporter", "")),
),
observed_at=_parse_dt(str(r.get("first_seen") or ""), "%Y-%m-%d %H:%M:%S"),
observables=Observables(hashes=hashes),
)
def _fetch_malware_bazaar() -> List[Case]:
key = os.environ.get("THREATFOX_AUTH_KEY", "").strip()
if not key:
raise RuntimeError("THREATFOX_AUTH_KEY not set — abuse.ch auth-key from https://auth.abuse.ch/ also covers MalwareBazaar")
# MalwareBazaar expects form-encoded body (unlike ThreatFox which takes JSON).
data = _http("POST", MALWARE_BAZAAR_API, headers={"Auth-Key": key}, form_body={"query": "get_recent", "selector": "100"}).json()
rows = data.get("data") or []
out: List[Case] = []
for r in rows:
c = _mb_row_to_case(r)
if c is not None:
out.append(c)
return out
# --- AlienVault OTX — curated multi-source pulses ------------------------
_OTX_IOC_LIMIT_PER_PULSE = 50
def _otx_pulse_to_case(p: Dict[str, Any]) -> Optional[Case]:
pulse_id = str(p.get("id") or "")
if not pulse_id:
return None
pulse_name = str(p.get("name") or "OTX pulse")
description = str(p.get("description") or "")
tags_raw = p.get("tags") or []
tags = tags_raw if isinstance(tags_raw, list) else []
tlp_pulse = str(p.get("tlp") or "white").upper()
indicators = p.get("indicators") or []
obs = Observables()
for ind in indicators[:_OTX_IOC_LIMIT_PER_PULSE]:
value = str(ind.get("indicator") or "").strip()
itype = str(ind.get("type") or "").lower()
if not value:
continue
if itype in ("ipv4", "ipv6"):
obs.ips.append(value)
elif itype in ("domain", "hostname"):
obs.domains.append(value)
elif itype == "url":
obs.urls.append(value)
host = urlparse(value).hostname or ""
if host and host not in obs.domains:
obs.domains.append(host)
elif itype in ("filehash-sha256", "filehash-sha1", "filehash-md5"):
obs.hashes.append(value)
elif itype == "cve":
obs.cves.append(value)
if not (obs.urls or obs.domains or obs.ips or obs.hashes or obs.cves):
return None
return Case(
case_id=f"PSYC-OTX-{pulse_id}",
summary=f"OTX: {pulse_name}",
source_type="threat_intel",
source_ref=f"https://otx.alienvault.com/pulse/{pulse_id}",
source_metadata=dict(
feed="otx",
pulse_name=pulse_name,
description=description[:2000],
tags=",".join(t for t in tags if t),
tlp_pulse=tlp_pulse,
),
observed_at=_parse_dt(str(p.get("created") or "").split(".")[0], "%Y-%m-%dT%H:%M:%S"),
observables=obs,
)
def _fetch_otx() -> List[Case]:
key = os.environ.get("OTX_API_KEY", "").strip()
if not key:
raise RuntimeError("OTX_API_KEY not set — free key at https://otx.alienvault.com → settings → API")
# OTX subscribes a new account to many curated feeds, so the unfiltered
# /pulses/subscribed page can 504 on its own backend. modified_since
# narrows to recent pulses; page size 20 caps the response.
since = (datetime.now(timezone.utc) - timedelta(days=7)).strftime("%Y-%m-%dT%H:%M:%S")
data = _http(
"GET", OTX_PULSES_API,
headers={"X-OTX-API-KEY": key},
params={"limit": 20, "modified_since": since},
timeout=120.0,
).json()
pulses = data.get("results") or []
out: List[Case] = []
for p in pulses:
c = _otx_pulse_to_case(p)
if c is not None:
out.append(c)
return out
# --- registry + dispatch -------------------------------------------------
SOURCES: Dict[str, Callable[[], List[Case]]] = {
"urlhaus": _fetch_urlhaus,
"cisa-kev": _fetch_cisa_kev,
"feodo": _fetch_feodo,
"threatfox": _fetch_threatfox,
"malware-bazaar": _fetch_malware_bazaar,
"otx": _fetch_otx,
}

View File

@@ -0,0 +1,228 @@
"""Topology export — sanitized public docker snapshot.
The cockpit's `docker_view.topology()` returns a rich daemon view useful to
the local operator: container env vars, volume mounts, internal IPs, labels,
gateways. None of that may leave the node. This module wraps `docker_view`
with a strict whitelist: only container names, images, states, network names
and high-level driver/health metadata are exposed. Anything not listed in
the Pydantic schemas below is dropped before serialization.
Used by `/federation/topology` so peer admin pages can render every node's
container topology side-by-side with their own.
"""
from __future__ import annotations
import re
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
from pydantic import BaseModel, Field
from psyc import log
from psyc.cockpit import docker_view
from psyc.lines import federation
_log = log.get(__name__)
# Caps keep the response bounded — a runaway node with thousands of
# containers shouldn't blow up the peer's panel.
MAX_CONTAINERS = 200
MAX_NETWORKS = 50
# ---------- data model --------------------------------------------------
class TopologyContainer(BaseModel):
"""One container — sanitized.
Strict whitelist: name, short_id, image (tag-only), state, health,
network names, compose service label, started_at. No env vars, no
volumes, no IPs, no MACs, no port mappings, no full labels dict.
"""
name: str
short_id: str
image: str
state: str
health: str
networks: List[str] = Field(default_factory=list)
service: Optional[str] = None
started_at: Optional[str] = None
class TopologyNetwork(BaseModel):
"""One docker network — sanitized.
Whitelist: name, driver, internal flag, container_count. No subnet,
no gateway, no labels, no attached-container details (those are
surfaced via the container.networks list).
"""
name: str
driver: str
internal: bool
container_count: int
class TopologyExport(BaseModel):
"""Whole-node container snapshot, public-safe."""
node_fingerprint: str
generated_at: str
host_name: str
container_count: int
network_count: int
containers: List[TopologyContainer] = Field(default_factory=list)
networks: List[TopologyNetwork] = Field(default_factory=list)
# ---------- sanitizers --------------------------------------------------
_BASIC_AUTH_RE = re.compile(r"^[^/@]+@")
def _filter_image_name(s: str) -> str:
"""Strip credentials from an image reference and drop digests.
Docker accepts `user:pass@registry/image:tag` for registries with HTTP
basic auth — we strip everything up to and including the `@` so leaked
creds never reach a peer. We also cut content-addressable digests
(`...@sha256:...`) to a clean tag-only form.
Returns the cleaned `repo/image:tag` string. Empty input → "".
"""
if not s:
return ""
raw = str(s).strip()
if not raw:
return ""
# Drop digest suffix, e.g. "nginx:1.25@sha256:abcd…" → "nginx:1.25".
if "@sha256:" in raw:
raw = raw.split("@sha256:", 1)[0]
# Strip basic-auth prefix on the registry component.
# "user:pass@host/repo:tag" → "host/repo:tag" (we never want creds out).
if _BASIC_AUTH_RE.match(raw):
raw = raw.split("@", 1)[1]
# Cap length defensively.
return raw[:160]
def _short_id(raw: Any) -> str:
s = str(raw or "")
return s[:12]
def _parse_health(status: str) -> str:
"""Extract a healthcheck word from the docker "Status" line if present.
docker's container-list "Status" string includes "(healthy)" or
"(unhealthy)" when a healthcheck is configured. We surface just that
one-word state and fall back to "" otherwise — no other free-form
text from the daemon leaks out.
"""
if not status:
return ""
low = status.lower()
if "(healthy)" in low:
return "healthy"
if "(unhealthy)" in low:
return "unhealthy"
if "(starting)" in low or "(health: starting)" in low:
return "starting"
return ""
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
def _empty_export(node_fp: str) -> TopologyExport:
return TopologyExport(
node_fingerprint=node_fp,
generated_at=_now_iso(),
host_name="",
container_count=0,
network_count=0,
containers=[],
networks=[],
)
# ---------- builder ------------------------------------------------------
def build_export() -> TopologyExport:
"""Sanitized snapshot of this node's docker topology.
Calls `docker_view.topology()` and re-projects every field through the
Pydantic whitelist above. If the proxy is unreachable (e.g. dev box
without docker-socket-proxy) we return an empty export rather than
raising — the public endpoint must never 500.
"""
try:
node_fp = federation.node_fingerprint()
except Exception as exc: # noqa: BLE001 — keep endpoint defensive
_log.warning("topology_export.fp.error", error=str(exc))
node_fp = ""
try:
raw = docker_view.topology()
except Exception as exc: # noqa: BLE001 — docker proxy may be down
_log.warning("topology_export.docker.error", error=str(exc))
return _empty_export(node_fp)
# docker_view.topology() returns a dict with `containers`, `networks`,
# `host`, `error` fields. We treat any non-None error as "empty export"
# rather than partially leaking through whatever did succeed.
if raw.get("error"):
return _empty_export(node_fp)
raw_host = raw.get("host") or {}
host_name_raw = str(raw_host.get("name") or "")
# Truncate the docker host id — it can be the actual machine hostname.
# Keep it short, no domain. Defensive even though docker host names are
# generally low-sensitivity.
host_name = host_name_raw[:24]
raw_containers = raw.get("containers") or []
raw_networks = raw.get("networks") or []
containers: List[TopologyContainer] = []
for c in raw_containers[:MAX_CONTAINERS]:
nets_raw = c.get("networks") or []
net_names: List[str] = []
for nd in nets_raw:
nm = nd.get("name") if isinstance(nd, dict) else None
if nm:
net_names.append(str(nm)[:64])
containers.append(TopologyContainer(
name=str(c.get("name") or "?")[:64],
short_id=_short_id(c.get("id")),
image=_filter_image_name(c.get("image") or ""),
state=str(c.get("state") or "")[:24],
health=_parse_health(str(c.get("status") or "")),
networks=net_names[:12],
# docker_view doesn't currently surface the compose service label
# or started_at; leave them None until that lands.
service=None,
started_at=None,
))
networks: List[TopologyNetwork] = []
for n in raw_networks[:MAX_NETWORKS]:
attached = n.get("containers") or []
networks.append(TopologyNetwork(
name=str(n.get("name") or "")[:64],
driver=str(n.get("driver") or "")[:24],
internal=bool(n.get("internal")),
container_count=len(attached),
))
return TopologyExport(
node_fingerprint=node_fp,
generated_at=_now_iso(),
host_name=host_name,
container_count=len(containers),
network_count=len(networks),
containers=containers,
networks=networks,
)

View File

@@ -15,6 +15,7 @@ restricted source types, never empty input/output.
from __future__ import annotations
import json
import random
import re
from datetime import datetime, timezone
from pathlib import Path
@@ -24,10 +25,18 @@ from pydantic import BaseModel, Field
from psyc import DATA_DIR, log
from psyc.lines import classify as classify_line
from psyc.lines import defang as defang_line
from psyc.lines import route as route_line
from psyc.models import Case, TLP
class BuildOptions(BaseModel):
"""Per-build configuration. Currently only ioc_extraction reads any field."""
defang_frac: float = 0.0 # in [0.0, 1.0] — fraction of ioc_extraction inputs to defang
seed: Optional[int] = None # reproducible RNG when set
_log = log.get(__name__)
DATASETS_DIR = DATA_DIR / "datasets"
@@ -60,7 +69,11 @@ class DatasetReport(BaseModel):
# ---------- ExampleBuilder per task ---------------------------------------
def _ex_ioc_extraction(case: Case) -> Optional[Example]:
def _ex_ioc_extraction(
case: Case,
options: Optional["BuildOptions"] = None,
rng: Optional[random.Random] = None,
) -> Optional[Example]:
obs = case.observables
if not (obs.urls or obs.domains or obs.ips or obs.hashes or obs.cves):
return None
@@ -81,6 +94,13 @@ def _ex_ioc_extraction(case: Case) -> Optional[Example]:
body.append("Related CVEs: " + ", ".join(obs.cves) + ".")
if tags:
body.append(f"Tags: {tags}.")
body_text = " ".join(body)
# Defanging augmentation: with probability options.defang_frac, replace IOCs
# in the input with common real-world defanged forms (1[.]2[.]3[.]4,
# hxxp://, etc.). Output stays canonical so the model learns the mapping.
if options is not None and rng is not None and options.defang_frac > 0.0:
if rng.random() < options.defang_frac:
body_text = defang_line.defang_text(body_text, obs.ips, obs.domains, obs.urls, rng)
output_obj = {
"urls": obs.urls,
"domains": obs.domains,
@@ -90,7 +110,7 @@ def _ex_ioc_extraction(case: Case) -> Optional[Example]:
}
return Example(
instruction="Extract all indicators of compromise from the advisory and return JSON with keys: urls, domains, ips, hashes, cves.",
input=" ".join(body),
input=body_text,
output=json.dumps(output_obj, ensure_ascii=False),
task="ioc_extraction",
case_id=case.case_id,
@@ -119,7 +139,11 @@ def severity_features(case: Case) -> Dict[str, object]:
}
def _ex_severity_classification(case: Case) -> Optional[Example]:
def _ex_severity_classification(
case: Case,
options: Optional["BuildOptions"] = None,
rng: Optional[random.Random] = None,
) -> Optional[Example]:
if case.classification.severity is None:
return None
return Example(
@@ -132,7 +156,11 @@ def _ex_severity_classification(case: Case) -> Optional[Example]:
)
def _ex_routing_decision(case: Case) -> Optional[Example]:
def _ex_routing_decision(
case: Case,
options: Optional["BuildOptions"] = None,
rng: Optional[random.Random] = None,
) -> Optional[Example]:
if case.classification.incident_type is None:
return None
routes, blocked = route_line.plan(case)
@@ -158,7 +186,11 @@ def _ex_routing_decision(case: Case) -> Optional[Example]:
)
def _ex_tlp_assignment(case: Case) -> Optional[Example]:
def _ex_tlp_assignment(
case: Case,
options: Optional["BuildOptions"] = None,
rng: Optional[random.Random] = None,
) -> Optional[Example]:
input_obj = {
"source_type": case.source_type,
"incident_type": case.classification.incident_type.value if case.classification.incident_type else None,
@@ -217,10 +249,12 @@ def _next_version(task: str) -> int:
return (max(used) + 1) if used else 1
def build(task: str, cases: Iterable[Case]) -> DatasetReport:
def build(task: str, cases: Iterable[Case], options: Optional[BuildOptions] = None) -> DatasetReport:
if task not in _BUILDERS:
raise ValueError(f"unknown task: {task}; choices: {sorted(_BUILDERS)}")
builder = _BUILDERS[task]
options = options or BuildOptions()
rng = random.Random(options.seed)
version = _next_version(task)
path = DATASETS_DIR / f"{task}-v{version}.jsonl"
written = 0
@@ -230,7 +264,7 @@ def build(task: str, cases: Iterable[Case]) -> DatasetReport:
skipped_empty = 0
with path.open("w", encoding="utf-8") as fh:
for case in cases:
example = builder(case)
example = builder(case, options, rng)
if example is None:
skipped_empty += 1
continue

161
src/psyc/lines/translog.py Normal file
View File

@@ -0,0 +1,161 @@
"""Transparency log — append-only signed merkle chain over federation signals.
Every signal we receive from a peer (case, IOC, or accepted vouch) is appended
as one `LogEntry`. Each entry's `entry_hash = sha256(canonical(prev_hash +
entry_type + entry_data + timestamp))` references the previous head, so any
tampering with a historical row invalidates every subsequent hash. The chain
is public — auditors can re-fetch it and re-run `verify_chain` to detect a
node that quietly mutated history (e.g. to hide a bad signal it accepted).
Hash format: lowercase hex SHA-256 of the canonical JSON of
``{"prev_hash": "...", "entry_type": "...", "entry_data": {...}, "timestamp": "..."}``.
Genesis entries use ``prev_hash = "0" * 64``.
"""
from __future__ import annotations
import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional
from pydantic import BaseModel, Field
from psyc import db, log
from psyc.result import Err, Ok, Result
_log = log.get(__name__)
GENESIS_PREV_HASH = "0" * 64
class LogEntry(BaseModel):
id: int
prev_hash: str
entry_type: str
entry_data: Dict[str, Any] = Field(default_factory=dict)
timestamp: str
entry_hash: str
def _canonical_json(obj: Dict[str, Any]) -> bytes:
return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
def compute_entry_hash(prev_hash: str, entry_type: str, entry_data: Dict[str, Any], timestamp: str) -> str:
"""Hex SHA-256 of canonical(prev_hash + entry_type + entry_data + timestamp)."""
payload: Dict[str, Any] = {
"prev_hash": prev_hash,
"entry_type": entry_type,
"entry_data": entry_data,
"timestamp": timestamp,
}
return hashlib.sha256(_canonical_json(payload)).hexdigest()
def _row_to_entry(row: Dict[str, Any]) -> LogEntry:
raw = row.get("entry_data") or "{}"
try:
data = json.loads(raw)
except Exception:
data = {}
return LogEntry(
id=int(row["id"]),
prev_hash=str(row["prev_hash"]),
entry_type=str(row["entry_type"]),
entry_data=data if isinstance(data, dict) else {},
timestamp=str(row["timestamp"]),
entry_hash=str(row["entry_hash"]),
)
def head(db_path: Path = db.DB_PATH) -> Optional[LogEntry]:
"""Latest log entry, or None if the chain is empty."""
row = db.translog_head(db_path=db_path)
return _row_to_entry(row) if row else None
def append(entry_type: str, entry_data: Dict[str, Any], db_path: Path = db.DB_PATH) -> LogEntry:
"""Atomically append one entry to the chain. Returns the persisted entry."""
prev = db.translog_head(db_path=db_path)
prev_hash = str(prev["entry_hash"]) if prev else GENESIS_PREV_HASH
timestamp = datetime.now(timezone.utc).isoformat()
entry_hash = compute_entry_hash(prev_hash, entry_type, entry_data, timestamp)
new_id = db.translog_append(
dict(
prev_hash=prev_hash,
entry_type=entry_type,
entry_data=json.dumps(entry_data, sort_keys=True),
timestamp=timestamp,
entry_hash=entry_hash,
),
db_path=db_path,
)
_log.info("translog.append", id=new_id, entry_type=entry_type, hash=entry_hash[:12])
return LogEntry(
id=new_id,
prev_hash=prev_hash,
entry_type=entry_type,
entry_data=entry_data,
timestamp=timestamp,
entry_hash=entry_hash,
)
def verify_chain(start: int = 0, end: Optional[int] = None, db_path: Path = db.DB_PATH) -> Result[int, str]:
"""Walk entries [start, end] in id order, recompute each hash, compare.
Returns Ok(n_verified) when every entry's recomputed hash equals the
stored one and each prev_hash matches the previous entry's stored hash.
Returns Err with the offending id + expected/got hashes otherwise.
"""
rows = db.translog_range(start=start, end=end, db_path=db_path)
if not rows:
return Ok(0)
# Establish the prior hash anchor — either genesis (if walking from id=1)
# or the entry just before `start`.
first_id = int(rows[0]["id"])
if first_id <= 1:
prior_hash = GENESIS_PREV_HASH
else:
anchor = db.translog_get(first_id - 1, db_path=db_path)
if anchor is None:
return Err(f"missing anchor entry id={first_id - 1}")
prior_hash = str(anchor["entry_hash"])
verified = 0
for row in rows:
stored_prev = str(row["prev_hash"])
if stored_prev != prior_hash:
return Err(
f"broken at id={row['id']} expected_prev={prior_hash} got_prev={stored_prev}"
)
try:
data = json.loads(row.get("entry_data") or "{}")
except Exception:
return Err(f"broken at id={row['id']} entry_data not JSON")
if not isinstance(data, dict):
return Err(f"broken at id={row['id']} entry_data not an object")
recomputed = compute_entry_hash(
stored_prev, str(row["entry_type"]), data, str(row["timestamp"])
)
stored_hash = str(row["entry_hash"])
if recomputed != stored_hash:
return Err(
f"broken at id={row['id']} expected={recomputed} got={stored_hash}"
)
prior_hash = stored_hash
verified += 1
return Ok(verified)
def recent(limit: int = 100, db_path: Path = db.DB_PATH) -> List[LogEntry]:
"""The latest `limit` entries, newest first."""
return [_row_to_entry(r) for r in db.translog_recent(limit=limit, db_path=db_path)]
def entries_after(entry_id: int, db_path: Path = db.DB_PATH) -> List[LogEntry]:
"""All entries with id > entry_id, oldest first — for peer sync."""
return [_row_to_entry(r) for r in db.translog_after(entry_id, db_path=db_path)]

View File

@@ -75,6 +75,13 @@ def submit_abuseipdb(payload: Dict[str, Any]) -> Dict[str, Any]:
return {"receipt_id": sub.receipt_id, "destination": sub.destination, "status": sub.status, "received_at": sub.received_at.isoformat()}
@app.post("/soar/enforce")
def soar_enforce(payload: Dict[str, Any]) -> Dict[str, Any]:
"""Mock enforcement sink — stands in for a firewall/DNS/SOAR webhook."""
sub = _record(f"SOAR:{payload.get('action_type', 'action')}", "enforced", payload)
return {"receipt_id": sub.receipt_id, "destination": sub.destination, "status": sub.status, "received_at": sub.received_at.isoformat()}
@app.get("/received")
def received() -> Dict[str, Any]:
return {"count": len(_submissions), "submissions": [s.model_dump(mode="json") for s in _submissions]}

View File

@@ -133,6 +133,7 @@ class Outcome(str, Enum):
REJECTED = "rejected"
ACTIONED = "actioned"
FAILED = "failed"
PENDING_APPROVAL = "pending_approval"
class LedgerEntry(BaseModel):
@@ -146,3 +147,52 @@ class LedgerEntry(BaseModel):
response_id: Optional[str] = None
outcome: Outcome
detail: Optional[str] = None
class ApprovalStatus(str, Enum):
PENDING = "pending"
APPROVED = "approved"
REJECTED = "rejected"
class PendingSubmission(BaseModel):
id: Optional[int] = None
case_id: str
destination_name: str
payload_kind: str
payload_hash: str
payload_json: str # frozen payload — what will be sent on approval
tlp: TLP
created_at: datetime
status: ApprovalStatus = ApprovalStatus.PENDING
reviewer: Optional[str] = None
reviewed_at: Optional[datetime] = None
reason: Optional[str] = None
class ActionType(str, Enum):
ALERT = "alert" # notify the SOC (webhook/Slack)
BLOCKLIST = "blocklist" # push IOCs to firewall/DNS enforcement
TICKET = "ticket" # open an incident ticket
class ActionStatus(str, Enum):
PROPOSED = "proposed" # generated, awaiting human approval
EXECUTED = "executed" # approved + fired successfully
REJECTED = "rejected" # human declined; nothing fired
FAILED = "failed" # approved but execution errored
class ResponseAction(BaseModel):
id: Optional[int] = None
case_id: str
action_type: ActionType
target: str # enforcement target label
summary: str # human-readable "what this does"
payload_json: str # frozen payload sent on execution
severity: Optional[str] = None
status: ActionStatus = ActionStatus.PROPOSED
created_at: datetime
approver: Optional[str] = None
executed_at: Optional[datetime] = None
detail: Optional[str] = None

86
tests/test_adminauth.py Normal file
View File

@@ -0,0 +1,86 @@
"""Admin gate — per-member TOTP enrollment + revocation isolation."""
from __future__ import annotations
import json
import pyotp
import pytest
from psyc.cockpit import adminauth
@pytest.fixture
def fresh_state(tmp_path, monkeypatch):
monkeypatch.setattr(adminauth, "_STATE_PATH", tmp_path / "admin_auth.json")
yield tmp_path / "admin_auth.json"
def _code_for_secret(secret: str) -> str:
return pyotp.TOTP(secret).now()
def _secret_of(state_path, member_id: str) -> str:
data = json.loads(state_path.read_text())
return next(m["secret"] for m in data["members"] if m["id"] == member_id)
def test_starts_unbootstrapped(fresh_state):
assert adminauth.is_bootstrapped() is False
assert adminauth.members() == []
def test_bootstrap_promotes_pending_to_owner(fresh_state):
code = adminauth.current_code() # pending secret
assert adminauth.verify(code) == "owner"
assert adminauth.is_bootstrapped() is True
assert [m["label"] for m in adminauth.members()] == ["owner"]
def test_add_member_then_each_secret_authenticates(fresh_state):
adminauth.verify(adminauth.current_code()) # bootstrap owner
aid, _ = adminauth.add_member("alice")
bid, _ = adminauth.add_member("bob")
assert adminauth.verify(_code_for_secret(_secret_of(fresh_state, aid))) == "alice"
assert adminauth.verify(_code_for_secret(_secret_of(fresh_state, bid))) == "bob"
def test_revoke_isolates_one_member(fresh_state):
adminauth.verify(adminauth.current_code())
aid, _ = adminauth.add_member("alice")
bid, _ = adminauth.add_member("bob")
a_secret = _secret_of(fresh_state, aid)
b_secret = _secret_of(fresh_state, bid)
assert adminauth.revoke_member(aid) is True
# Alice's code no longer works…
assert adminauth.verify(_code_for_secret(a_secret)) is None
# …but Bob's still does.
assert adminauth.verify(_code_for_secret(b_secret)) == "bob"
# Alice is gone from the active roster, Bob remains.
labels = [m["label"] for m in adminauth.members()]
assert "alice" not in labels and "bob" in labels
def test_members_never_expose_secrets(fresh_state):
adminauth.verify(adminauth.current_code())
adminauth.add_member("alice")
for m in adminauth.members():
assert "secret" not in m
def test_revoke_unknown_id_is_false(fresh_state):
adminauth.verify(adminauth.current_code())
assert adminauth.revoke_member("deadbeef") is False
def test_migrates_old_single_secret_format(fresh_state):
# Simulate the pre-stage-27 state file.
old_secret = pyotp.random_base32()
fresh_state.write_text(json.dumps({
"totp_secret": old_secret, "session_secret": "s", "provisioned": True,
}))
# Old enrolled secret should authenticate as the migrated 'owner'.
assert adminauth.verify(_code_for_secret(old_secret)) == "owner"
assert [m["label"] for m in adminauth.members()] == ["owner"]

View File

@@ -57,3 +57,26 @@ def test_classify_is_idempotent():
first = case.classification.model_copy(deep=True)
classify(case)
assert case.classification == first
def test_threatfox_botnet_cc_is_botnet():
case = make_case(feed="threatfox", ips=["1.2.3.4"])
case.source_metadata["threat_type"] = "botnet_cc"
assert classify(case).classification.incident_type is IncidentType.BOTNET
def test_threatfox_payload_delivery_is_malware():
case = make_case(feed="threatfox", urls=["http://1.2.3.4/x.bin"])
case.source_metadata["threat_type"] = "payload_delivery"
assert classify(case).classification.incident_type is IncidentType.MALWARE
def test_threatfox_phishing_threat_type_is_phishing():
case = make_case(feed="threatfox", urls=["http://login.bad/example"])
case.source_metadata["threat_type"] = "phishing"
assert classify(case).classification.incident_type is IncidentType.PHISHING
def test_malware_bazaar_is_malware():
case = make_case(feed="malware-bazaar", hashes=["a" * 64])
assert classify(case).classification.incident_type is IncidentType.MALWARE

130
tests/test_courier.py Normal file
View File

@@ -0,0 +1,130 @@
"""Courier approval-queue tests — gating, dispatch, rejection."""
from __future__ import annotations
import pytest
from sqlalchemy import create_engine, select
from psyc import db
from psyc.lines import courier, ledger as ledger_line
from psyc.lines.route import Route
from psyc.models import ApprovalStatus, Outcome, TLP
from psyc.result import Ok
from conftest import make_case
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
test_engine = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(test_engine, checkfirst=True)
monkeypatch.setattr(db, "_engine", test_engine)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
def _malware_url_route(requires_approval: bool) -> Route:
return Route(
destination_name="URLhaus",
priority=3,
payload_kind="malware_url_report",
max_tlp_allowed=TLP.GREEN,
requires_approval=requires_approval,
)
def test_execute_routes_enqueues_when_approval_required(fresh_db, monkeypatch):
case = make_case(feed="urlhaus", urls=["http://1.2.3.4/x"], ips=["1.2.3.4"])
case.classification.tlp = TLP.GREEN
# No HTTP should happen — submit must NOT be called on the approval branch.
def boom(*a, **kw):
raise AssertionError("submit() must not be called when approval is required")
monkeypatch.setattr(courier, "submit", boom)
results = courier.execute_routes(case, [_malware_url_route(requires_approval=True)])
assert len(results) == 1
assert results[0].outcome is Outcome.PENDING_APPROVAL
pending = courier.list_pending()
assert len(pending) == 1
assert pending[0].destination_name == "URLhaus"
assert pending[0].status is ApprovalStatus.PENDING
assert pending[0].payload_hash # frozen hash present
assert pending[0].payload_json # frozen payload present
def test_execute_routes_force_approval_via_env(fresh_db, monkeypatch):
case = make_case(feed="urlhaus", urls=["http://1.2.3.4/x"], ips=["1.2.3.4"])
case.classification.tlp = TLP.GREEN
monkeypatch.setenv("PSYC_REQUIRE_APPROVAL", "1")
monkeypatch.setattr(courier, "submit", lambda *a, **kw: (_ for _ in ()).throw(AssertionError("must not submit")))
results = courier.execute_routes(case, [_malware_url_route(requires_approval=False)])
assert results[0].outcome is Outcome.PENDING_APPROVAL
assert courier.pending_count() == 1
def test_dispatch_pending_submits_and_marks_approved(fresh_db, monkeypatch):
case = make_case(feed="urlhaus", urls=["http://1.2.3.4/x"], ips=["1.2.3.4"])
case.classification.tlp = TLP.GREEN
monkeypatch.setattr(courier, "submit", lambda *a, **kw: (_ for _ in ()).throw(AssertionError("queue phase")))
courier.execute_routes(case, [_malware_url_route(requires_approval=True)])
pid = courier.list_pending()[0].id
# Now approve — submit IS called, returns a receipt.
receipt = courier.Receipt(receipt_id="r-001", destination="urlhaus", status="acknowledged", response_body={})
monkeypatch.setattr(courier, "submit", lambda *a, **kw: Ok(receipt))
result = courier.dispatch_pending(pid, reviewer="alice")
assert isinstance(result, Ok)
assert result.value.outcome is Outcome.ACKNOWLEDGED
assert result.value.receipt_id == "r-001"
# Pending row now marked approved.
refreshed = courier.get_pending(pid).value
assert refreshed.status is ApprovalStatus.APPROVED
assert refreshed.reviewer == "alice"
# Ledger has a corresponding row.
entries = ledger_line.list_by_case(case.case_id, limit=10)
assert any(e.outcome is Outcome.ACKNOWLEDGED and e.destination == "URLhaus" for e in entries)
def test_reject_pending_writes_rejection_to_ledger(fresh_db, monkeypatch):
case = make_case(feed="urlhaus", urls=["http://1.2.3.4/x"], ips=["1.2.3.4"])
case.classification.tlp = TLP.GREEN
monkeypatch.setattr(courier, "submit", lambda *a, **kw: (_ for _ in ()).throw(AssertionError("queue phase")))
courier.execute_routes(case, [_malware_url_route(requires_approval=True)])
pid = courier.list_pending()[0].id
# Reject — submit must NOT be called.
def boom(*a, **kw):
raise AssertionError("rejected submissions must not POST")
monkeypatch.setattr(courier, "submit", boom)
result = courier.reject_pending(pid, reviewer="bob", reason="payload looks wrong")
assert isinstance(result, Ok)
refreshed = courier.get_pending(pid).value
assert refreshed.status is ApprovalStatus.REJECTED
assert refreshed.reviewer == "bob"
assert refreshed.reason == "payload looks wrong"
entries = ledger_line.list_by_case(case.case_id, limit=10)
assert any(e.outcome is Outcome.REJECTED and "rejected_by=bob" in (e.detail or "") for e in entries)
def test_double_approve_is_rejected(fresh_db, monkeypatch):
case = make_case(feed="urlhaus", urls=["http://1.2.3.4/x"], ips=["1.2.3.4"])
case.classification.tlp = TLP.GREEN
monkeypatch.setattr(courier, "submit", lambda *a, **kw: (_ for _ in ()).throw(AssertionError("queue phase")))
courier.execute_routes(case, [_malware_url_route(requires_approval=True)])
pid = courier.list_pending()[0].id
receipt = courier.Receipt(receipt_id="r-002", destination="urlhaus", status="acknowledged", response_body={})
monkeypatch.setattr(courier, "submit", lambda *a, **kw: Ok(receipt))
courier.dispatch_pending(pid, reviewer="alice")
# Second approval of the same id must fail — no double-submission.
again = courier.dispatch_pending(pid, reviewer="alice")
assert not isinstance(again, Ok)

71
tests/test_defang.py Normal file
View File

@@ -0,0 +1,71 @@
"""Defanging — IOC obfuscation styles for training-data augmentation."""
from __future__ import annotations
import json
import random
from psyc.lines.defang import defang_domain, defang_ip, defang_text, defang_url
from psyc.lines.train import BuildOptions, _ex_ioc_extraction
from conftest import make_case
def test_defang_ip_breaks_canonical_form():
out = defang_ip("1.2.3.4", random.Random(0))
assert "1.2.3.4" not in out # canonical IP substring no longer appears
assert "1" in out and "4" in out # digits preserved
assert any(form in out for form in ("[.]", "(.)", "[dot]", "{.}"))
def test_defang_domain_preserves_label_text():
out = defang_domain("evil.example.com", random.Random(1))
assert "evil" in out and "example" in out and "com" in out
assert "evil.example.com" not in out # canonical domain broken
def test_defang_url_defangs_protocol_and_breaks_canonical_form():
out = defang_url("http://evil.example.com/payload.bin", random.Random(2))
assert out.startswith("hxxp://") # protocol defanged
assert "http://" not in out
assert "evil.example.com" not in out # host part defanged
def test_defang_url_handles_https():
assert defang_url("https://evil.com/x", random.Random(0)).startswith("hxxps://")
def test_defang_text_substitutes_every_listed_ioc():
text = "See URL http://1.2.3.4/x and IP 1.2.3.4 and domain evil.com please."
out = defang_text(text, ips=["1.2.3.4"], domains=["evil.com"], urls=["http://1.2.3.4/x"], rng=random.Random(3))
# No canonical IOC string should remain anywhere in the corrupted body.
assert "http://" not in out
assert "1.2.3.4" not in out
assert "evil.com" not in out
# Surrounding prose is preserved.
assert "See URL" in out and "please" in out
def test_ioc_extraction_with_defang_frac_1_corrupts_input_only():
case = make_case(feed="urlhaus", urls=["http://1.2.3.4/x"], domains=["1.2.3.4"], ips=["1.2.3.4"])
options = BuildOptions(defang_frac=1.0, seed=42)
rng = random.Random(options.seed)
ex = _ex_ioc_extraction(case, options, rng)
assert ex is not None
# Input has been defanged.
assert "1.2.3.4" not in ex.input
assert "http://" not in ex.input
# Output stays canonical so the model learns the inverse mapping.
output = json.loads(ex.output)
assert "1.2.3.4" in output["ips"]
assert "http://1.2.3.4/x" in output["urls"]
def test_ioc_extraction_with_defang_frac_0_is_canonical():
case = make_case(feed="urlhaus", urls=["http://1.2.3.4/x"], ips=["1.2.3.4"])
options = BuildOptions(defang_frac=0.0, seed=0)
rng = random.Random(0)
ex = _ex_ioc_extraction(case, options, rng)
assert ex is not None
# No defanging → input keeps the canonical IOCs.
assert "http://1.2.3.4/x" in ex.input
assert "1.2.3.4" in ex.input

376
tests/test_discovery.py Normal file
View File

@@ -0,0 +1,376 @@
"""Discovery — DNS-SD parse + resolver, BFS walker, persistence, public endpoint."""
from __future__ import annotations
from typing import Any, Dict, List, Optional
from unittest.mock import MagicMock, patch
import dns.exception
import dns.resolver
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from sqlalchemy import create_engine
from starlette.middleware.sessions import SessionMiddleware
from psyc import db
from psyc.cockpit import federation_routes
from psyc.lines import discovery, federation, pulse
from psyc.lines.discovery import (
PeerCandidate,
_parse_txt_value,
fetch_peer_info,
fetch_public_peers,
public_peer_attestation,
record_candidate,
resolve_psyc,
walk,
)
from psyc.result import Err, Ok
# ---------- fixtures ---------------------------------------------------------
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
@pytest.fixture
def fed_dir(tmp_path, monkeypatch):
d = tmp_path / "federation"
monkeypatch.setattr(federation, "FED_DIR", d)
monkeypatch.setattr(federation, "PRIVATE_KEY_PATH", d / "node.key")
monkeypatch.setattr(federation, "PUBLIC_KEY_PATH", d / "node.pub")
yield d
def _mk_srv(port: int = 443) -> Any:
rd = MagicMock()
rd.port = port
return rd
def _mk_txt(value: str) -> Any:
rd = MagicMock()
rd.strings = [value.encode("utf-8")]
return rd
# ---------- TXT parser -------------------------------------------------------
def test_parse_txt_valid():
txt = "v=psyc1 fp=" + "a" * 32 + " alg=ed25519 path=/federation/feed"
res = _parse_txt_value(txt)
assert isinstance(res, Ok)
assert res.value["fp"] == "a" * 32
assert res.value["alg"] == "ed25519"
def test_parse_txt_tolerates_token_order():
txt = "path=/federation/feed alg=ed25519 fp=" + "f" * 32 + " v=psyc1"
res = _parse_txt_value(txt)
assert isinstance(res, Ok)
def test_parse_txt_rejects_wrong_version():
txt = "v=psyc2 fp=" + "a" * 32 + " alg=ed25519"
res = _parse_txt_value(txt)
assert isinstance(res, Err)
assert "version" in res.reason
def test_parse_txt_rejects_bad_fingerprint_length():
txt = "v=psyc1 fp=deadbeef alg=ed25519"
res = _parse_txt_value(txt)
assert isinstance(res, Err)
assert "fingerprint" in res.reason
def test_parse_txt_rejects_non_hex_fingerprint():
txt = "v=psyc1 fp=" + "z" * 32 + " alg=ed25519"
res = _parse_txt_value(txt)
assert isinstance(res, Err)
def test_parse_txt_rejects_malformed_token():
txt = "v=psyc1 fp=" + "a" * 32 + " alg ed25519"
res = _parse_txt_value(txt)
assert isinstance(res, Err)
assert "malformed" in res.reason
def test_parse_txt_rejects_wrong_alg():
txt = "v=psyc1 fp=" + "a" * 32 + " alg=rsa"
res = _parse_txt_value(txt)
assert isinstance(res, Err)
# ---------- resolve_psyc -----------------------------------------------------
def test_resolve_psyc_happy_path():
fp = "1" * 32
txt = f"v=psyc1 fp={fp} alg=ed25519 path=/federation/feed"
def fake_resolve(self, name, rdtype):
if rdtype == "SRV":
return [_mk_srv(port=8443)]
if rdtype == "TXT":
return [_mk_txt(txt)]
raise dns.exception.DNSException("unexpected")
with patch.object(dns.resolver.Resolver, "resolve", fake_resolve):
res = resolve_psyc("peer.example.com")
assert isinstance(res, Ok)
cand = res.value
assert cand.domain == "peer.example.com"
assert cand.fingerprint == fp
assert cand.port == 8443
assert cand.source == "dns-sd"
def test_resolve_psyc_nxdomain_returns_err():
def fake_resolve(self, name, rdtype):
raise dns.resolver.NXDOMAIN()
with patch.object(dns.resolver.Resolver, "resolve", fake_resolve):
res = resolve_psyc("nothere.example")
assert isinstance(res, Err)
assert "NXDOMAIN" in res.reason
def test_resolve_psyc_txt_malformed_returns_err():
def fake_resolve(self, name, rdtype):
if rdtype == "SRV":
return [_mk_srv()]
return [_mk_txt("v=psyc1 fp=garbage alg=ed25519")]
with patch.object(dns.resolver.Resolver, "resolve", fake_resolve):
res = resolve_psyc("peer.example")
assert isinstance(res, Err)
assert "TXT" in res.reason or "fingerprint" in res.reason
def test_resolve_psyc_no_answer_returns_err():
def fake_resolve(self, name, rdtype):
raise dns.resolver.NoAnswer()
with patch.object(dns.resolver.Resolver, "resolve", fake_resolve):
res = resolve_psyc("peer.example")
assert isinstance(res, Err)
assert "NoAnswer" in res.reason
# ---------- walk -------------------------------------------------------------
def _stub_resolve(catalog: Dict[str, str]):
"""Build a resolve_psyc stub that returns each domain's catalog fingerprint."""
def _stub(domain: str, timeout: float = 5.0):
if domain not in catalog:
return Err(f"no record for {domain}")
return Ok(PeerCandidate(
domain=domain,
fingerprint=catalog[domain],
port=443,
source="dns-sd",
))
return _stub
def _stub_fetch_info_ok(*args, **kwargs):
return Ok({"fingerprint": kwargs.get("expected_fingerprint", "")})
def _stub_fetch_peers_factory(graph: Dict[str, List[Dict[str, str]]]):
def _stub(domain: str, port: int = 443, timeout: float = 5.0):
return Ok(graph.get(domain, []))
return _stub
def test_walk_dedupes_by_fingerprint(fresh_db, fed_dir, monkeypatch):
# Two seeds, same fingerprint via different domains → only one survives the (domain,fp) dedupe
# but distinct domains both surface; the (domain, fp) pair just shouldn't repeat.
fp = "9" * 32
catalog = {"a.example": fp, "b.example": fp}
monkeypatch.setattr(discovery, "resolve_psyc", _stub_resolve(catalog))
monkeypatch.setattr(discovery, "fetch_peer_info", _stub_fetch_info_ok)
monkeypatch.setattr(discovery, "fetch_public_peers", _stub_fetch_peers_factory({}))
out = walk(["a.example", "b.example", "a.example"], max_depth=1)
# both unique domains made it in; the duplicate seed didn't re-enter
assert len(out) == 2
domains = {c.domain for c in out}
assert domains == {"a.example", "b.example"}
def test_walk_respects_max_depth(fresh_db, fed_dir, monkeypatch):
catalog = {"d0.example": "0" * 32, "d1.example": "1" * 32, "d2.example": "2" * 32}
graph = {
"d0.example": [{"domain": "d1.example", "fingerprint": "1" * 32}],
"d1.example": [{"domain": "d2.example", "fingerprint": "2" * 32}],
}
monkeypatch.setattr(discovery, "resolve_psyc", _stub_resolve(catalog))
monkeypatch.setattr(discovery, "fetch_peer_info", _stub_fetch_info_ok)
monkeypatch.setattr(discovery, "fetch_public_peers", _stub_fetch_peers_factory(graph))
out = walk(["d0.example"], max_depth=1)
domains = {c.domain for c in out}
# depth 0: d0; depth 1: d1; depth 2 (d2) is excluded by max_depth=1
assert "d0.example" in domains and "d1.example" in domains
assert "d2.example" not in domains
def test_walk_respects_max_peers(fresh_db, fed_dir, monkeypatch):
catalog = {f"d{i}.example": f"{i:032x}" for i in range(10)}
graph = {"d0.example": [{"domain": f"d{i}.example", "fingerprint": f"{i:032x}"} for i in range(1, 10)]}
monkeypatch.setattr(discovery, "resolve_psyc", _stub_resolve(catalog))
monkeypatch.setattr(discovery, "fetch_peer_info", _stub_fetch_info_ok)
monkeypatch.setattr(discovery, "fetch_public_peers", _stub_fetch_peers_factory(graph))
out = walk(["d0.example"], max_depth=2, max_peers=3)
assert len(out) <= 3
def test_walk_skips_own_fingerprint(fresh_db, fed_dir, monkeypatch):
own = federation.node_fingerprint()
catalog = {"self.example": own, "peer.example": "f" * 32}
monkeypatch.setattr(discovery, "resolve_psyc", _stub_resolve(catalog))
monkeypatch.setattr(discovery, "fetch_peer_info", _stub_fetch_info_ok)
monkeypatch.setattr(discovery, "fetch_public_peers", _stub_fetch_peers_factory({}))
out = walk(["self.example", "peer.example"], max_depth=1)
domains = {c.domain for c in out}
assert "self.example" not in domains
assert "peer.example" in domains
def test_walk_one_failure_does_not_abort(fresh_db, fed_dir, monkeypatch):
catalog = {"good.example": "a" * 32} # bad.example is absent → Err on resolve
monkeypatch.setattr(discovery, "resolve_psyc", _stub_resolve(catalog))
monkeypatch.setattr(discovery, "fetch_peer_info", _stub_fetch_info_ok)
monkeypatch.setattr(discovery, "fetch_public_peers", _stub_fetch_peers_factory({}))
out = walk(["bad.example", "good.example"], max_depth=1)
assert len(out) == 1
assert out[0].domain == "good.example"
# ---------- record_candidate -------------------------------------------------
def test_record_candidate_inserts_as_unknown(fresh_db):
c = PeerCandidate(domain="new.example", fingerprint="a" * 32, source="dns-sd")
record_candidate(c)
row = db.get_peer("new.example")
assert row is not None
assert row["status"] == "unknown"
assert row["fingerprint"] == "a" * 32
def test_record_candidate_preserves_trusted(fresh_db, fed_dir):
federation.register_peer("vip.example", "b" * 32, "PEM", status="trusted")
# walker re-discovers it
c = PeerCandidate(domain="vip.example", fingerprint="b" * 32, source="peer-walk:other.example")
record_candidate(c)
row = db.get_peer("vip.example")
assert row["status"] == "trusted"
def test_record_candidate_preserves_blocked(fresh_db, fed_dir):
federation.register_peer("bad.example", "c" * 32, "PEM", status="blocked")
c = PeerCandidate(domain="bad.example", fingerprint="c" * 32, source="dns-sd")
record_candidate(c)
row = db.get_peer("bad.example")
assert row["status"] == "blocked"
def test_record_candidate_updates_last_seen(fresh_db):
c = PeerCandidate(domain="repeat.example", fingerprint="d" * 32, source="dns-sd")
record_candidate(c)
first = db.get_peer("repeat.example")
# second pass — last_seen advances, discovered_at stays
c2 = PeerCandidate(domain="repeat.example", fingerprint="d" * 32, source="dns-sd")
record_candidate(c2)
second = db.get_peer("repeat.example")
assert second["discovered_at"] == first["discovered_at"]
# ---------- public attestation -----------------------------------------------
def test_public_peer_attestation_only_trusted(fresh_db, fed_dir):
federation.register_peer("trusted.example", "1" * 32, "PEM", status="trusted")
federation.register_peer("unknown.example", "2" * 32, "PEM", status="unknown")
federation.register_peer("blocked.example", "3" * 32, "PEM", status="blocked")
out = public_peer_attestation()
domains = {p["domain"] for p in out}
assert domains == {"trusted.example"}
def test_public_peer_attestation_payload_shape(fresh_db, fed_dir):
federation.register_peer("t.example", "f" * 32, "PEM", status="trusted")
out = public_peer_attestation()
assert len(out) == 1
entry = out[0]
assert set(entry.keys()) == {"domain", "fingerprint", "first_seen"}
# ---------- public endpoint via TestClient -----------------------------------
def _mk_app() -> FastAPI:
app = FastAPI()
app.add_middleware(SessionMiddleware, secret_key="test-secret")
# Templates aren't exercised by the public endpoints we care about here.
from fastapi.templating import Jinja2Templates
import tempfile, os
tdir = tempfile.mkdtemp()
templates = Jinja2Templates(directory=tdir)
federation_routes.register(app, templates)
return app
def test_public_peers_endpoint_excludes_unknown_blocked(fresh_db, fed_dir):
federation.register_peer("ok.example", "a" * 32, "PEM", status="trusted")
federation.register_peer("rude.example", "b" * 32, "PEM", status="blocked")
federation.register_peer("new.example", "c" * 32, "PEM", status="unknown")
# Flush in-memory cache from any earlier test.
federation_routes._PUBLIC_PEERS_CACHE["payload"] = None
federation_routes._PUBLIC_PEERS_CACHE["ts"] = 0.0
app = _mk_app()
client = TestClient(app)
r = client.get("/federation/peers/public")
assert r.status_code == 200
body = r.json()
domains = {p["domain"] for p in body}
assert "ok.example" in domains
assert "rude.example" not in domains
assert "new.example" not in domains
# ---------- pulse integration ------------------------------------------------
def test_discovery_seeds_roundtrip(fresh_db):
assert pulse.get_discovery_seeds() == []
pulse.set_discovery_seeds(["a.example", "b.example", "a.example", "", " "])
# dedupe + strip blanks
assert pulse.get_discovery_seeds() == ["a.example", "b.example"]
def test_peer_pull_pipeline_no_seeds(fresh_db, fed_dir, monkeypatch):
# peer-pull runner returns a clean message when nothing's configured.
outcome, result = pulse.run_now("peer-pull")
assert outcome == "ok"
assert "no seeds" in result
def test_peer_pull_pipeline_with_seeds(fresh_db, fed_dir, monkeypatch):
pulse.set_discovery_seeds(["good.example"])
catalog = {"good.example": "e" * 32}
monkeypatch.setattr(discovery, "resolve_psyc", _stub_resolve(catalog))
monkeypatch.setattr(discovery, "fetch_peer_info", _stub_fetch_info_ok)
monkeypatch.setattr(discovery, "fetch_public_peers", _stub_fetch_peers_factory({}))
outcome, result = pulse.run_now("peer-pull")
assert outcome == "ok"
assert "discovered 1" in result
# And it was recorded.
row = db.get_peer("good.example")
assert row is not None
assert row["status"] == "unknown"

124
tests/test_docker_view.py Normal file
View File

@@ -0,0 +1,124 @@
"""Docker topology — normalization + error handling against the socket-proxy."""
from __future__ import annotations
import pytest
from psyc.cockpit import docker_view
_CONTAINERS_FIXTURE = [
{
"Id": "abcdef1234567890",
"Names": ["/psyc-cockpit-1"],
"Image": "psyc:latest",
"State": "running",
"Status": "Up 5 minutes (healthy)",
"NetworkSettings": {"Networks": {"backend": {"IPAddress": "172.20.0.5"}}},
"Ports": [{"IP": "0.0.0.0", "PrivatePort": 8767, "PublicPort": 8767, "Type": "tcp"}],
},
{
"Id": "fedcba0987654321",
"Names": ["/some-stopped"],
"Image": "alpine",
"State": "exited",
"Status": "Exited (0) 2 hours ago",
"NetworkSettings": {"Networks": {}},
"Ports": [],
},
]
_NETWORKS_FIXTURE = [
{
"Id": "n1", "Name": "backend", "Driver": "bridge", "Scope": "local", "Internal": False,
"IPAM": {"Config": [{"Subnet": "172.20.0.0/16", "Gateway": "172.20.0.1"}]},
"Containers": {
"abcdef1234567890": {
"Name": "psyc-cockpit-1", "IPv4Address": "172.20.0.5/16",
"MacAddress": "02:42:ac:14:00:05",
},
},
},
{"Id": "n2", "Name": "bridge", "Driver": "bridge", "Scope": "local", "Internal": False, "Containers": {}},
]
def _fake_get_factory(monkeypatch, payloads: dict):
"""Patch docker_view._get to return canned payloads by path.
Unknown paths raise DockerProxyError (mimicking a proxy that blocks the
endpoint), so callers like host_info() exercise their fallback paths.
"""
def fake_get(path: str):
for prefix, body in payloads.items():
if path.startswith(prefix):
return body
raise docker_view.DockerProxyError(f"blocked: {path}")
monkeypatch.setattr(docker_view, "_get", fake_get)
def test_list_containers_normalizes_fields(monkeypatch):
_fake_get_factory(monkeypatch, {"/containers/json": _CONTAINERS_FIXTURE})
out = docker_view.list_containers()
by_name = {c["name"]: c for c in out}
# running comes before exited
assert out[0]["state"] == "running"
assert out[-1]["state"] == "exited"
cockpit = by_name["psyc-cockpit-1"]
assert cockpit["id"] == "abcdef123456"
assert cockpit["image"] == "psyc:latest"
assert cockpit["networks"][0]["name"] == "backend"
assert cockpit["networks"][0]["ip"] == "172.20.0.5"
assert "0.0.0.0:8767->8767/tcp" in cockpit["ports"]
def test_list_networks_attaches_containers_with_ip(monkeypatch):
_fake_get_factory(monkeypatch, {"/networks": _NETWORKS_FIXTURE})
out = docker_view.list_networks()
backend = next(n for n in out if n["name"] == "backend")
assert backend["driver"] == "bridge"
assert backend["subnet"] == "172.20.0.0/16"
assert backend["gateway"] == "172.20.0.1"
assert backend["containers"][0]["name"] == "psyc-cockpit-1"
assert backend["containers"][0]["ip"] == "172.20.0.5"
assert backend["containers"][0]["mac"] == "02:42:ac:14:00:05"
# default networks pushed to bottom
assert out[-1]["name"] == "bridge"
def test_list_containers_extracts_published_ports(monkeypatch):
_fake_get_factory(monkeypatch, {"/containers/json": _CONTAINERS_FIXTURE})
out = docker_view.list_containers()
cockpit = next(c for c in out if c["name"] == "psyc-cockpit-1")
assert cockpit["published_ports"] == ["8767/tcp"]
# stopped container with no ports → empty
stopped = next(c for c in out if c["name"] == "some-stopped")
assert stopped["published_ports"] == []
def test_host_info_falls_back_when_proxy_blocks_info(monkeypatch):
def boom(path):
raise docker_view.DockerProxyError("info forbidden")
monkeypatch.setattr(docker_view, "_get", boom)
info = docker_view.host_info()
assert info["name"] == "docker host"
def test_topology_returns_both_with_no_error(monkeypatch):
_fake_get_factory(monkeypatch, {
"/containers/json": _CONTAINERS_FIXTURE,
"/networks": _NETWORKS_FIXTURE,
})
snap = docker_view.topology()
assert snap["error"] is None
assert len(snap["containers"]) == 2
assert len(snap["networks"]) == 2
def test_topology_surfaces_proxy_failure(monkeypatch):
def boom(path):
raise docker_view.DockerProxyError("connection refused")
monkeypatch.setattr(docker_view, "_get", boom)
snap = docker_view.topology()
assert snap["error"] is not None and "connection refused" in snap["error"]
assert snap["containers"] == [] and snap["networks"] == []

427
tests/test_explore_view.py Normal file
View File

@@ -0,0 +1,427 @@
"""Federation explore view — public transparency payload tests.
Sibling to `test_network_view.py`; focused on the explore-only shape:
shape contract, signature round-trip, no-leak invariants, transitive walk,
inbound vouches filter, and the corroboration counter.
"""
from __future__ import annotations
import base64
import json
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, List, Optional
from unittest.mock import patch
import pytest
from fastapi import FastAPI
from fastapi.templating import Jinja2Templates
from fastapi.testclient import TestClient
from sqlalchemy import create_engine
from starlette.middleware.sessions import SessionMiddleware
from psyc import db
from psyc.cockpit import federation_routes
from psyc.lines import federation, network_view, translog
from psyc.lines.network_view import build_explore_view
# ---------- fixtures ----------------------------------------------------
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
@pytest.fixture
def fed_dir(tmp_path, monkeypatch):
d = tmp_path / "federation"
monkeypatch.setattr(federation, "FED_DIR", d)
monkeypatch.setattr(federation, "PRIVATE_KEY_PATH", d / "node.key")
monkeypatch.setattr(federation, "PUBLIC_KEY_PATH", d / "node.pub")
yield d
@pytest.fixture(autouse=True)
def reset_explore_caches(monkeypatch):
"""Prevent route-level cache bleed between tests."""
monkeypatch.setattr(network_view, "_TRANSITIVE_CACHE", {"ts": 0.0, "view": None})
federation_routes._FEED_CACHE["payload"] = None
federation_routes._PUBLIC_PEERS_CACHE["payload"] = None
federation_routes._PUBLIC_NETWORK_CACHE["payload"] = None
if hasattr(federation_routes, "_EXPLORE_CACHE"):
federation_routes._EXPLORE_CACHE["payload"] = None
yield
def _make_peer_pubkey() -> tuple[str, str]:
"""Return (fingerprint, pubkey_pem) for a synthetic peer keypair."""
import hashlib
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
priv = ed25519.Ed25519PrivateKey.generate()
pub = priv.public_key()
pem = pub.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
).decode("ascii")
raw = pub.public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
fp = hashlib.sha256(raw).digest()[:16].hex()
return fp, pem
def _silence_explore_fetch():
return patch.object(network_view, "_fetch_peer_explore", return_value=None)
def _silence_network_fetch():
return patch.object(network_view, "_fetch_peer_network", return_value=None)
# ---------- schema ------------------------------------------------------
def test_explore_view_top_level_shape(fresh_db, fed_dir):
with _silence_explore_fetch(), _silence_network_fetch():
payload = build_explore_view(node_domain="me.example")
for key in (
"version", "fingerprint", "generated_at",
"node", "peers", "vouches", "vouches_out", "vouches_in",
"transitive_peers", "corroboration_count_24h", "signature",
):
assert key in payload, f"missing {key}"
node = payload["node"]
for key in (
"fingerprint", "domain", "generated_at",
"transparency_log_head_hash", "translog_entry_count",
"peer_count", "vouches_out_count", "vouches_in_count",
"corroboration_count_24h", "signals_count_24h",
):
assert key in node, f"missing node.{key}"
assert node["domain"] == "me.example"
assert node["fingerprint"] == federation.node_fingerprint()
def test_explore_peer_carries_public_safe_stats(fresh_db, fed_dir):
fp, pem = _make_peer_pubkey()
federation.register_peer("peer.example", fp, pem, status="trusted")
now_iso = datetime.now(timezone.utc).isoformat()
for i in range(3):
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id=f"1.2.3.{i}",
signal_hash=f"hash-{i}",
received_at=now_iso,
raw_json=json.dumps({"type": "ip", "value": f"1.2.3.{i}"}),
))
with _silence_explore_fetch(), _silence_network_fetch():
payload = build_explore_view()
assert len(payload["peers"]) == 1
p = payload["peers"][0]
# Public-safe stats present.
for key in (
"signal_count_24h", "signal_count_total",
"cases_24h", "iocs_24h",
"quorum_contribution_24h", "last_seen",
):
assert key in p
assert p["signal_count_24h"] == 3
assert p["iocs_24h"] == 3
assert p["cases_24h"] == 0
# Sensitive fields are not surfaced per-peer.
assert "severity_breakdown" not in p
assert "ioc_type_breakdown" not in p
assert "recent_translog" not in p
# ---------- signature round-trip ---------------------------------------
def test_explore_view_signature_round_trip(fresh_db, fed_dir):
fp, pem = _make_peer_pubkey()
federation.register_peer("trusted.example", fp, pem, status="trusted")
federation.issue_vouch(fp, ttl_days=30)
with _silence_explore_fetch(), _silence_network_fetch():
payload = build_explore_view()
assert "signature" in payload
sig = base64.b64decode(payload["signature"])
unsigned = {k: v for k, v in payload.items() if k != "signature"}
assert federation.verify_payload(
federation.canonical_json(unsigned),
sig,
federation.public_key_pem(),
) is True
# ---------- no-leak invariants -----------------------------------------
def test_explore_view_has_no_ioc_values_or_case_ids_or_raw_json(fresh_db, fed_dir):
"""Public payload must not expose IOC values, case_ids in raw form, or raw_json.
This is the core transparency-vs-leakage contract: anyone can see who's
talking to whom and how much, but never what they're saying.
"""
fp, pem = _make_peer_pubkey()
federation.register_peer("trusted.example", fp, pem, status="trusted")
now_iso = datetime.now(timezone.utc).isoformat()
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id="evil-domain-do-not-leak.com",
signal_hash="ioc-hash-leak",
received_at=now_iso,
raw_json=json.dumps({"type": "domain", "value": "evil-domain-do-not-leak.com"}),
))
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="case",
signal_id="CASE-SECRET-42",
signal_hash="case-hash-leak",
received_at=now_iso,
raw_json=json.dumps({"severity": "critical", "case_id": "CASE-SECRET-42"}),
))
with _silence_explore_fetch(), _silence_network_fetch():
payload = build_explore_view()
flat = json.dumps(payload, default=str)
# IOC values.
assert "evil-domain-do-not-leak.com" not in flat
# Case ids (raw).
assert "CASE-SECRET-42" not in flat
# raw_json shape never serialized.
assert "raw_json" not in flat
# Sector-leaking breakdowns.
assert "severity_breakdown" not in flat
assert "ioc_type_breakdown" not in flat
# ---------- transitive peers --------------------------------------------
def test_explore_transitive_peers_populated_from_peer_responses(fresh_db, fed_dir):
direct_fp, direct_pem = _make_peer_pubkey()
federation.register_peer("direct.example", direct_fp, direct_pem, status="trusted")
far_a, _ = _make_peer_pubkey()
far_b, _ = _make_peer_pubkey()
fake_payload: Dict[str, Any] = {
"fingerprint": direct_fp,
"peers": [
{"fingerprint": far_a, "domain": "far-a.example"},
{"fingerprint": far_b, "domain": "far-b.example"},
],
"vouches": [],
}
with patch.object(network_view, "_fetch_peer_explore", return_value=fake_payload):
payload = build_explore_view()
tps = payload["transitive_peers"]
fps = {t["fingerprint"] for t in tps}
assert far_a in fps
assert far_b in fps
via_fps = {t["via_peer_fingerprint"] for t in tps}
assert via_fps == {direct_fp}
def test_explore_transitive_peers_falls_back_to_network_endpoint(fresh_db, fed_dir):
"""If a peer doesn't have /federation/explore/data (older node), fall back
to /federation/network — the public-view shape is the same {fingerprint, peers}."""
direct_fp, direct_pem = _make_peer_pubkey()
federation.register_peer("direct.example", direct_fp, direct_pem, status="trusted")
far_fp, _ = _make_peer_pubkey()
fallback_payload: Dict[str, Any] = {
"fingerprint": direct_fp,
"peers": [{"fingerprint": far_fp, "domain": "far.example"}],
"vouches": [],
}
with patch.object(network_view, "_fetch_peer_explore", return_value=None), \
patch.object(network_view, "_fetch_peer_network", return_value=fallback_payload):
payload = build_explore_view()
assert any(t["fingerprint"] == far_fp for t in payload["transitive_peers"])
def test_explore_transitive_peers_dedupe_against_direct(fresh_db, fed_dir):
"""If a transitive fp is already a direct peer, don't duplicate it."""
direct_fp, direct_pem = _make_peer_pubkey()
federation.register_peer("direct.example", direct_fp, direct_pem, status="trusted")
fake_payload = {
"fingerprint": direct_fp,
# Direct peer's own fp echoed back — must be deduped.
"peers": [{"fingerprint": direct_fp, "domain": "direct.example"}],
"vouches": [],
}
with patch.object(network_view, "_fetch_peer_explore", return_value=fake_payload):
payload = build_explore_view()
assert payload["transitive_peers"] == []
# ---------- vouches_in --------------------------------------------------
def test_explore_vouches_in_filters_to_target_self_and_trusted_vouchers(fresh_db, fed_dir):
"""vouches_in includes ONLY entries naming us as target whose voucher we trust."""
our_fp = federation.node_fingerprint()
fp_trusted, pem_t = _make_peer_pubkey()
fp_unknown, pem_u = _make_peer_pubkey()
federation.register_peer("trusted.example", fp_trusted, pem_t, status="trusted")
federation.register_peer("unknown.example", fp_unknown, pem_u, status="unknown")
now = datetime.now(timezone.utc).isoformat()
# Trusted peer vouches for us.
db.upsert_vouch(dict(
voucher_fingerprint=fp_trusted,
target_fingerprint=our_fp,
issued_at=now,
expires_at=None,
signature="trusted-sig",
))
# Unknown peer also "vouches" for us — must NOT leak.
db.upsert_vouch(dict(
voucher_fingerprint=fp_unknown,
target_fingerprint=our_fp,
issued_at=now,
expires_at=None,
signature="rogue-sig",
))
# Vouch naming someone else — must NOT appear in vouches_in.
other_fp, _ = _make_peer_pubkey()
db.upsert_vouch(dict(
voucher_fingerprint=fp_trusted,
target_fingerprint=other_fp,
issued_at=now,
expires_at=None,
signature="other-sig",
))
with _silence_explore_fetch(), _silence_network_fetch():
payload = build_explore_view()
vouchers = {v["voucher_fingerprint"] for v in payload["vouches_in"]}
assert vouchers == {fp_trusted}
# And the rogue signature is not anywhere in the payload.
assert "rogue-sig" not in json.dumps(payload, default=str)
# ---------- corroboration counter --------------------------------------
def test_explore_corroboration_count_matches_distinct_shared_hashes(fresh_db, fed_dir):
fp_a, pem_a = _make_peer_pubkey()
fp_b, pem_b = _make_peer_pubkey()
federation.register_peer("a.example", fp_a, pem_a, status="trusted")
federation.register_peer("b.example", fp_b, pem_b, status="trusted")
now_iso = datetime.now(timezone.utc).isoformat()
# Two shared hashes between A and B.
for h in ("shared-1", "shared-2"):
for fp in (fp_a, fp_b):
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id="x",
signal_hash=h,
received_at=now_iso,
raw_json="{}",
))
# One solo hash — must NOT count.
db.record_signal(dict(
peer_fingerprint=fp_a,
signal_type="ioc",
signal_id="solo",
signal_hash="solo-hash",
received_at=now_iso,
raw_json="{}",
))
with _silence_explore_fetch(), _silence_network_fetch():
payload = build_explore_view()
assert payload["corroboration_count_24h"] == 2
assert payload["node"]["corroboration_count_24h"] == 2
# ---------- transparency log headline ----------------------------------
def test_explore_node_translog_headline_reflects_chain(fresh_db, fed_dir):
translog.append("vouch", {"foo": "bar"})
translog.append("signal", {"x": 1})
with _silence_explore_fetch(), _silence_network_fetch():
payload = build_explore_view()
node = payload["node"]
assert node["translog_entry_count"] == 2
assert isinstance(node["transparency_log_head_hash"], str)
assert len(node["transparency_log_head_hash"]) == 64 # hex sha256
# ---------- HTTP routes -------------------------------------------------
def _mk_app() -> FastAPI:
app = FastAPI()
app.add_middleware(SessionMiddleware, secret_key="test-secret")
import tempfile
from pathlib import Path as _Path
# We need real templates for /federation/explore HTML response.
here = _Path(__file__).resolve().parent.parent / "src" / "psyc" / "cockpit" / "templates"
templates = Jinja2Templates(directory=str(here))
federation_routes.register(app, templates)
return app
def test_federation_explore_endpoint_returns_html(fresh_db, fed_dir):
with _silence_explore_fetch(), _silence_network_fetch():
client = TestClient(_mk_app())
r = client.get("/federation/explore")
assert r.status_code == 200
assert "text/html" in r.headers.get("content-type", "")
# Banner + page title are present.
body = r.text
assert "Federation Explorer" in body
def test_federation_explore_data_returns_signed_json(fresh_db, fed_dir):
fp, pem = _make_peer_pubkey()
federation.register_peer("trusted.example", fp, pem, status="trusted")
with _silence_explore_fetch(), _silence_network_fetch():
client = TestClient(_mk_app())
r = client.get("/federation/explore/data")
assert r.status_code == 200
data = r.json()
assert "signature" in data
assert "node" in data
sig = base64.b64decode(data["signature"])
unsigned = {k: v for k, v in data.items() if k != "signature"}
assert federation.verify_payload(
federation.canonical_json(unsigned),
sig,
federation.public_key_pem(),
) is True
def test_federation_explore_data_has_cors_header(fresh_db, fed_dir):
"""Other psyc nodes' explore pages need to fetch this from the browser."""
with _silence_explore_fetch(), _silence_network_fetch():
client = TestClient(_mk_app())
r = client.get("/federation/explore/data")
assert r.status_code == 200
assert r.headers.get("access-control-allow-origin") == "*"
def test_federation_info_has_explore_and_cors(fresh_db, fed_dir):
client = TestClient(_mk_app())
r = client.get("/federation/info")
assert r.status_code == 200
data = r.json()
assert data.get("explore") == "/federation/explore"
assert r.headers.get("access-control-allow-origin") == "*"
def test_existing_public_endpoints_have_cors_header(fresh_db, fed_dir):
"""All public endpoints must be cross-origin fetchable for the explorer."""
client = TestClient(_mk_app())
for path in (
"/federation/key",
"/federation/feed",
"/federation/vouches",
"/federation/log",
"/federation/log/verify",
"/federation/peers/public",
"/federation/network",
):
r = client.get(path)
assert r.status_code in (200, 409), f"{path} status {r.status_code}"
assert r.headers.get("access-control-allow-origin") == "*", f"{path} missing CORS"

262
tests/test_federation.py Normal file
View File

@@ -0,0 +1,262 @@
"""Federation — identity, signed feed, peer registry, signal buffer."""
from __future__ import annotations
import base64
import os
import stat
import pytest
from sqlalchemy import create_engine
from psyc import db
from psyc.lines import federation
from psyc.lines.federation import (
DNSRecord,
build_signed_feed,
canonical_json,
dns_record,
import_signed_feed,
node_fingerprint,
node_keypair,
public_key_pem,
sign_payload,
verify_payload,
)
from psyc.result import Err, Ok
from conftest import make_case
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
@pytest.fixture
def fed_dir(tmp_path, monkeypatch):
"""Redirect federation key paths to a tmp dir so each test gets a fresh key."""
d = tmp_path / "federation"
monkeypatch.setattr(federation, "FED_DIR", d)
monkeypatch.setattr(federation, "PRIVATE_KEY_PATH", d / "node.key")
monkeypatch.setattr(federation, "PUBLIC_KEY_PATH", d / "node.pub")
yield d
def test_keypair_persisted_with_correct_perms(fed_dir):
priv, pub = node_keypair()
assert federation.PRIVATE_KEY_PATH.exists()
assert federation.PUBLIC_KEY_PATH.exists()
mode = stat.S_IMODE(os.stat(federation.PRIVATE_KEY_PATH).st_mode)
assert mode == 0o600
def test_keypair_idempotent_across_calls(fed_dir):
priv1, pub1 = node_keypair()
priv2, pub2 = node_keypair()
# raw bytes match — same key loaded twice, not regenerated
raw1 = pub1.public_bytes(
encoding=federation.serialization.Encoding.Raw,
format=federation.serialization.PublicFormat.Raw,
)
raw2 = pub2.public_bytes(
encoding=federation.serialization.Encoding.Raw,
format=federation.serialization.PublicFormat.Raw,
)
assert raw1 == raw2
def test_fingerprint_is_stable_and_32_hex(fed_dir):
fp1 = node_fingerprint()
fp2 = node_fingerprint()
assert fp1 == fp2
assert len(fp1) == 32
assert all(c in "0123456789abcdef" for c in fp1)
def test_sign_verify_roundtrip(fed_dir):
payload = b"hello federation"
sig = sign_payload(payload)
assert verify_payload(payload, sig, public_key_pem()) is True
def test_verify_with_wrong_key_returns_false(fed_dir, tmp_path):
payload = b"the truth"
sig = sign_payload(payload)
# Build a *different* keypair in a separate directory and use its pubkey.
other = tmp_path / "other-federation"
other.mkdir()
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
other_priv = ed25519.Ed25519PrivateKey.generate()
other_pub_pem = other_priv.public_key().public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
).decode("ascii")
assert verify_payload(payload, sig, other_pub_pem) is False
def test_verify_with_garbage_pubkey_returns_false_no_raise(fed_dir):
sig = sign_payload(b"x")
assert verify_payload(b"x", sig, "not a pem") is False
def test_canonical_json_is_deterministic():
a = canonical_json({"b": 1, "a": 2, "nested": {"y": 1, "x": 2}})
b = canonical_json({"a": 2, "b": 1, "nested": {"x": 2, "y": 1}})
assert a == b
def test_dns_record_txt_value_matches_spec(fed_dir):
rec = dns_record("example.com")
assert isinstance(rec, DNSRecord)
fp = node_fingerprint()
assert rec.srv_name == "_psyc._tcp.example.com"
assert rec.srv_target == "example.com."
assert rec.srv_port == 443
assert rec.txt_value == f"v=psyc1 fp={fp} alg=ed25519 path=/federation/feed"
# human instructions include both record lines
assert "_psyc._tcp.example.com" in rec.human_instructions
assert "SRV" in rec.human_instructions
assert "TXT" in rec.human_instructions
def test_build_then_import_signed_feed_roundtrip(fresh_db, fed_dir):
case = make_case(feed="urlhaus", ips=["1.1.1.1"], urls=["http://1.1.1.1/x"])
db.upsert_case(case)
feed = build_signed_feed(window_hours=24)
assert feed["version"] == "psyc1"
assert feed["fingerprint"] == node_fingerprint()
assert feed["signature"]
# cases entry made it in
assert any(c["case_id"] == case.case_id for c in feed["cases"])
# Import using our own pubkey against a *different* declared fingerprint:
# swap the fingerprint so import_signed_feed doesn't reject as a loop.
pub = public_key_pem()
# Use a fresh keypair to act as "peer" — sign a feed with that key.
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
import hashlib
peer_priv = ed25519.Ed25519PrivateKey.generate()
peer_pub_pem = peer_priv.public_key().public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
).decode("ascii")
peer_raw = peer_priv.public_key().public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
peer_fp = hashlib.sha256(peer_raw).digest()[:16].hex()
feed["fingerprint"] = peer_fp
unsigned = {k: v for k, v in feed.items() if k != "signature"}
new_sig = peer_priv.sign(canonical_json(unsigned))
feed["signature"] = base64.b64encode(new_sig).decode("ascii")
# Stage 4 listening gate: peer must be trusted to land signals.
federation.register_peer("peer.example", peer_fp, peer_pub_pem, status="trusted")
result = import_signed_feed(feed, peer_pub_pem)
assert isinstance(result, Ok), getattr(result, "reason", "")
summary = result.value
assert summary.peer_fingerprint == peer_fp
assert summary.cases_seen >= 1
def test_import_with_wrong_pubkey_returns_err(fresh_db, fed_dir):
db.upsert_case(make_case(feed="urlhaus", ips=["2.2.2.2"]))
feed = build_signed_feed(window_hours=24)
# build a *different* pubkey to claim verification against
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
other_pub_pem = ed25519.Ed25519PrivateKey.generate().public_key().public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
).decode("ascii")
# also need to change fingerprint so the loop-check doesn't trigger first
feed["fingerprint"] = "deadbeef" * 4
result = import_signed_feed(feed, other_pub_pem)
assert isinstance(result, Err)
def test_import_own_feed_returns_loop_err(fresh_db, fed_dir):
db.upsert_case(make_case(feed="urlhaus", ips=["3.3.3.3"]))
feed = build_signed_feed(window_hours=24)
result = import_signed_feed(feed, public_key_pem())
assert isinstance(result, Err)
assert "loop" in result.reason
def test_record_signal_then_lookup_by_hash(fresh_db):
rid = db.record_signal(dict(
peer_fingerprint="abc123",
signal_type="ioc",
signal_id="1.2.3.4",
signal_hash="hash-aaa",
received_at="2026-01-01T00:00:00+00:00",
raw_json="{}",
))
assert rid > 0
rows = db.signals_for_hash("hash-aaa")
assert len(rows) == 1
assert rows[0]["peer_fingerprint"] == "abc123"
# second peer reports the same hash → both surface for quorum check
db.record_signal(dict(
peer_fingerprint="def456",
signal_type="ioc",
signal_id="1.2.3.4",
signal_hash="hash-aaa",
received_at="2026-01-01T00:01:00+00:00",
raw_json="{}",
))
rows = db.signals_for_hash("hash-aaa")
assert {r["peer_fingerprint"] for r in rows} == {"abc123", "def456"}
def test_peer_registry_crud(fresh_db, fed_dir):
federation.register_peer("peer.example", "ff" * 16, "PEM", status="trusted")
peers = federation.list_peers()
assert len(peers) == 1
assert peers[0].domain == "peer.example"
assert peers[0].status == "trusted"
federation.set_peer_status("peer.example", "blocked")
assert federation.get_peer("peer.example").status == "blocked"
federation.remove_peer("peer.example")
assert federation.list_peers() == []
def test_register_peer_rejects_malformed_domain(fresh_db, fed_dir):
"""XSS guard: domain must look like a hostname (+ optional :port)."""
import pytest
bad = [
"evil.com'); alert(1); //",
"evil.com<script>",
"evil.com onclick=alert(1)",
"",
"evil com", # space
"/etc/passwd",
"evil.com/?phish=1",
]
for d in bad:
with pytest.raises(ValueError):
federation.register_peer(d, "ff" * 16, "PEM")
# And good ones still pass:
for d in ["peer.example.com", "peer.example.com:8443", "peer-1.example", "127.0.0.1:8767"]:
federation.register_peer(d, "ff" * 16, "PEM")
federation.remove_peer(d)
def test_register_peer_rejects_malformed_fingerprint(fresh_db, fed_dir):
"""Defense-in-depth: fingerprint must be 32 hex chars."""
import pytest
with pytest.raises(ValueError):
federation.register_peer("peer.example", "not-hex", "PEM")
with pytest.raises(ValueError):
federation.register_peer("peer.example", "ff" * 8, "PEM") # too short

151
tests/test_inference.py Normal file
View File

@@ -0,0 +1,151 @@
"""Tests for the inference client — both psyc-native and openai-compatible modes."""
from __future__ import annotations
import importlib
import json
from typing import Any, Dict
import httpx
import pytest
from psyc.cockpit import inference
from psyc.models import Case, Classification, Confidence, Severity, TLP, Observables, Evidence, Victim
def _reload_with_env(monkeypatch, **env: str) -> Any:
for k, v in env.items():
monkeypatch.setenv(k, v)
return importlib.reload(inference)
def _case() -> Case:
return Case(
case_id="C-T-1",
summary="test",
source_type="test",
source_ref="",
observed_at="2026-01-01T00:00:00+00:00",
ingested_at="2026-01-01T00:00:00+00:00",
classification=Classification(tlp=TLP.GREEN, severity=Severity.HIGH),
confidence=Confidence(level="medium", source_reliability="B", information_credibility="2"),
observables=Observables(),
evidence=Evidence(),
source_metadata={},
victim=Victim(),
)
def test_no_auth_header_when_token_unset(monkeypatch):
mod = _reload_with_env(monkeypatch, PSYC_INFERENCE_TOKEN="")
assert mod._auth_headers() == {}
def test_bearer_header_when_token_set(monkeypatch):
mod = _reload_with_env(monkeypatch, PSYC_INFERENCE_TOKEN="abc123")
assert mod._auth_headers() == {"Authorization": "Bearer abc123"}
def test_psyc_mode_server_adapter(monkeypatch):
mod = _reload_with_env(monkeypatch, PSYC_INFERENCE_MODE="psyc", PSYC_INFERENCE_URL="http://x")
seen: Dict[str, Any] = {}
def handler(request: httpx.Request) -> httpx.Response:
seen["url"] = str(request.url)
seen["method"] = request.method
return httpx.Response(200, json={"adapter": "/data/adapters/psyc-v5/final"})
transport = httpx.MockTransport(handler)
real_client = httpx.Client
monkeypatch.setattr(httpx, "Client", lambda **kw: real_client(transport=transport, **{k: v for k, v in kw.items() if k != "transport"}))
assert mod.server_adapter() == "/data/adapters/psyc-v5/final"
assert seen["url"].endswith("/healthz")
def test_openai_mode_server_adapter(monkeypatch):
mod = _reload_with_env(
monkeypatch,
PSYC_INFERENCE_MODE="openai",
PSYC_INFERENCE_URL="https://api.example",
PSYC_INFERENCE_TOKEN="t0k",
PSYC_INFERENCE_MODEL="psyc-v5",
)
seen: Dict[str, Any] = {}
def handler(request: httpx.Request) -> httpx.Response:
seen["url"] = str(request.url)
seen["auth"] = request.headers.get("authorization")
return httpx.Response(200, json={"data": [{"id": "llama3"}, {"id": "mistral"}]})
transport = httpx.MockTransport(handler)
real_client = httpx.Client
monkeypatch.setattr(httpx, "Client", lambda **kw: real_client(transport=transport, **{k: v for k, v in kw.items() if k != "transport"}))
assert mod.server_adapter() == "llama3"
assert seen["url"].endswith("/v1/models")
assert seen["auth"] == "Bearer t0k"
def test_openai_mode_severity_request_shape(monkeypatch):
mod = _reload_with_env(
monkeypatch,
PSYC_INFERENCE_MODE="openai",
PSYC_INFERENCE_URL="https://api.example",
PSYC_INFERENCE_TOKEN="t0k",
PSYC_INFERENCE_MODEL="psyc-v5",
)
sent: Dict[str, Any] = {}
def handler(request: httpx.Request) -> httpx.Response:
sent["url"] = str(request.url)
sent["auth"] = request.headers.get("authorization")
sent["body"] = json.loads(request.content.decode())
return httpx.Response(200, json={"choices": [{"message": {"content": "HIGH"}}]})
transport = httpx.MockTransport(handler)
real_client = httpx.Client
monkeypatch.setattr(httpx, "Client", lambda **kw: real_client(transport=transport, **{k: v for k, v in kw.items() if k != "transport"}))
result = mod.model_severity(_case())
assert result == "high"
assert sent["url"].endswith("/v1/chat/completions")
assert sent["auth"] == "Bearer t0k"
assert sent["body"]["model"] == "psyc-v5"
assert sent["body"]["messages"][0]["role"] == "system"
assert sent["body"]["messages"][1]["role"] == "user"
assert sent["body"]["max_tokens"] == 16
def test_psyc_mode_severity_unchanged(monkeypatch):
mod = _reload_with_env(monkeypatch, PSYC_INFERENCE_MODE="psyc", PSYC_INFERENCE_URL="http://x", PSYC_INFERENCE_TOKEN="")
sent: Dict[str, Any] = {}
def handler(request: httpx.Request) -> httpx.Response:
sent["url"] = str(request.url)
sent["auth"] = request.headers.get("authorization")
sent["body"] = json.loads(request.content.decode())
return httpx.Response(200, json={"output": "MEDIUM"})
transport = httpx.MockTransport(handler)
real_client = httpx.Client
monkeypatch.setattr(httpx, "Client", lambda **kw: real_client(transport=transport, **{k: v for k, v in kw.items() if k != "transport"}))
assert mod.model_severity(_case()) == "medium"
assert sent["url"].endswith("/infer")
assert sent["auth"] is None
assert "instruction" in sent["body"]
assert "max_new_tokens" in sent["body"]
def test_server_adapter_returns_none_on_http_error(monkeypatch):
mod = _reload_with_env(monkeypatch, PSYC_INFERENCE_MODE="openai", PSYC_INFERENCE_URL="https://api.example")
def handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(401, json={"error": "unauthorized"})
transport = httpx.MockTransport(handler)
real_client = httpx.Client
monkeypatch.setattr(httpx, "Client", lambda **kw: real_client(transport=transport, **{k: v for k, v in kw.items() if k != "transport"}))
assert mod.server_adapter() is None

85
tests/test_lookup.py Normal file
View File

@@ -0,0 +1,85 @@
"""Lookupline — IOC index, normalization, lookup, blocklist export."""
from __future__ import annotations
import pytest
from sqlalchemy import create_engine
from psyc import db
from psyc.lines import lookup
from psyc.models import Severity
from conftest import make_case
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
def test_normalize_lowercases_except_cve():
assert lookup.normalize("EVIL.COM", "domain") == "evil.com"
assert lookup.normalize(" AbCdEf ", "hash") == "abcdef"
assert lookup.normalize("cve-2026-0001", "cve") == "CVE-2026-0001"
def test_iter_case_iocs_covers_all_types():
case = make_case(
feed="urlhaus",
urls=["http://1.2.3.4/x"], domains=["EVIL.com"], ips=["1.2.3.4"],
hashes=["AABBCC"], cves=["cve-2026-1"],
)
pairs = set(lookup.iter_case_iocs(case))
assert ("http://1.2.3.4/x", "url") in pairs
assert ("evil.com", "domain") in pairs # normalized
assert ("1.2.3.4", "ip") in pairs
assert ("aabbcc", "hash") in pairs # normalized
assert ("CVE-2026-1", "cve") in pairs # upper
def test_reindex_then_lookup_finds_case(fresh_db):
case = make_case(feed="threatfox", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
n = lookup.reindex([case])
assert n == 1
hits = lookup.lookup("9.9.9.9")
assert len(hits) == 1
assert hits[0]["case_id"] == case.case_id
assert hits[0]["feed"] == "threatfox"
assert hits[0]["severity"] == "high"
def test_lookup_is_normalization_insensitive(fresh_db):
case = make_case(feed="urlhaus", domains=["Evil.Example.COM"], severity=Severity.MEDIUM)
lookup.reindex([case])
# Query with different casing than stored — still matches.
assert len(lookup.lookup("evil.example.com")) == 1
assert len(lookup.lookup("EVIL.EXAMPLE.COM")) == 1
def test_lookup_miss_returns_empty(fresh_db):
lookup.reindex([make_case(feed="urlhaus", ips=["1.1.1.1"])])
assert lookup.lookup("8.8.8.8") == []
def test_export_blocklist_dedupes_and_filters_by_severity(fresh_db):
high = make_case(feed="feodo", ips=["10.0.0.1"], severity=Severity.HIGH)
med = make_case(feed="urlhaus", ips=["10.0.0.2"], severity=Severity.MEDIUM)
dup = make_case(feed="threatfox", ips=["10.0.0.1"], severity=Severity.CRITICAL) # same IP as high
lookup.reindex([high, med, dup])
all_ips = lookup.export_blocklist("ip")
assert set(all_ips) == {"10.0.0.1", "10.0.0.2"} # deduped across cases
high_only = lookup.export_blocklist("ip", min_severity="high")
assert "10.0.0.1" in high_only # high + critical pass
assert "10.0.0.2" not in high_only # medium filtered out
def test_export_blocklist_rejects_bad_type(fresh_db):
with pytest.raises(ValueError):
lookup.export_blocklist("mutex")

702
tests/test_network_view.py Normal file
View File

@@ -0,0 +1,702 @@
"""Network view — local + transitive + public payload tests."""
from __future__ import annotations
import base64
import json
from datetime import datetime, timedelta, timezone
from typing import Any, Dict
from unittest.mock import patch
import pytest
from sqlalchemy import create_engine
from psyc import db
from psyc.lines import federation, network_view, translog
from psyc.lines.network_view import (
NetworkEdge,
NetworkNode,
NetworkView,
build_admin_view,
build_explore_view,
build_local_view,
build_public_view,
build_transitive_view,
)
# ---------- fixtures ----------------------------------------------------
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
@pytest.fixture
def fed_dir(tmp_path, monkeypatch):
d = tmp_path / "federation"
monkeypatch.setattr(federation, "FED_DIR", d)
monkeypatch.setattr(federation, "PRIVATE_KEY_PATH", d / "node.key")
monkeypatch.setattr(federation, "PUBLIC_KEY_PATH", d / "node.pub")
yield d
@pytest.fixture(autouse=True)
def reset_transitive_cache(monkeypatch):
"""Prevent cache bleed between tests."""
monkeypatch.setattr(network_view, "_TRANSITIVE_CACHE", {"ts": 0.0, "view": None})
yield
def _make_peer_pubkey() -> tuple[str, str]:
"""Return (fingerprint, pubkey_pem) for a synthetic peer keypair."""
import hashlib
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
priv = ed25519.Ed25519PrivateKey.generate()
pub = priv.public_key()
pem = pub.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
).decode("ascii")
raw = pub.public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
fp = hashlib.sha256(raw).digest()[:16].hex()
return fp, pem
# ---------- local view --------------------------------------------------
def test_local_view_empty_registry_yields_only_self(fresh_db, fed_dir):
view = build_local_view()
assert isinstance(view, NetworkView)
assert len(view.nodes) == 1
self_node = view.nodes[0]
assert self_node.is_self is True
assert self_node.distance == 0
assert self_node.status == "self"
assert self_node.fingerprint == federation.node_fingerprint()
assert view.edges == []
assert view.stats["total_peers"] == 0
assert view.stats["vouched_peers"] == 0
assert view.stats["signals_buffered_24h"] == 0
def test_local_view_one_trusted_peer_no_edges(fresh_db, fed_dir):
peer_fp, peer_pem = _make_peer_pubkey()
federation.register_peer("peer.example", peer_fp, peer_pem, status="trusted")
view = build_local_view()
assert len(view.nodes) == 2
peer_node = next(n for n in view.nodes if not n.is_self)
assert peer_node.fingerprint == peer_fp
assert peer_node.status == "trusted"
assert peer_node.distance == 1
assert peer_node.domain == "peer.example"
assert view.edges == []
assert view.stats["total_peers"] == 1
assert view.stats["vouched_peers"] == 1
def test_local_view_outbound_vouch_creates_edge(fresh_db, fed_dir):
peer_fp, peer_pem = _make_peer_pubkey()
federation.register_peer("peer.example", peer_fp, peer_pem, status="trusted")
federation.issue_vouch(peer_fp, ttl_days=30)
view = build_local_view()
vouch_edges = [e for e in view.edges if e.kind == "vouch"]
assert len(vouch_edges) == 1
e = vouch_edges[0]
assert e.source_fingerprint == federation.node_fingerprint()
assert e.target_fingerprint == peer_fp
assert view.stats["vouches_issued"] == 1
def test_local_view_inbound_vouch_creates_edge(fresh_db, fed_dir):
"""Vouches received that name us as target → peer → self edge."""
peer_fp, peer_pem = _make_peer_pubkey()
federation.register_peer("peer.example", peer_fp, peer_pem, status="trusted")
# Insert a vouch where peer vouches FOR us, bypassing accept_vouch (which
# we don't need to exercise here — the question is render shape).
our_fp = federation.node_fingerprint()
now = datetime.now(timezone.utc)
db.upsert_vouch(dict(
voucher_fingerprint=peer_fp,
target_fingerprint=our_fp,
issued_at=now.isoformat(),
expires_at=(now + timedelta(days=30)).isoformat(),
signature="x" * 88,
))
view = build_local_view()
vouch_edges = [e for e in view.edges if e.kind == "vouch"]
assert len(vouch_edges) == 1
e = vouch_edges[0]
assert e.source_fingerprint == peer_fp
assert e.target_fingerprint == our_fp
def test_local_view_bidirectional_vouches_collapse(fresh_db, fed_dir):
peer_fp, peer_pem = _make_peer_pubkey()
federation.register_peer("peer.example", peer_fp, peer_pem, status="trusted")
federation.issue_vouch(peer_fp, ttl_days=30)
# And peer vouches back at us.
our_fp = federation.node_fingerprint()
now = datetime.now(timezone.utc)
db.upsert_vouch(dict(
voucher_fingerprint=peer_fp,
target_fingerprint=our_fp,
issued_at=now.isoformat(),
expires_at=(now + timedelta(days=30)).isoformat(),
signature="x" * 88,
))
view = build_local_view()
vouch_edges = [e for e in view.edges if e.kind == "vouch"]
assert len(vouch_edges) == 1
assert vouch_edges[0].bidirectional is True
def test_local_view_signal_edge_weight_is_24h_count(fresh_db, fed_dir):
peer_fp, peer_pem = _make_peer_pubkey()
federation.register_peer("peer.example", peer_fp, peer_pem, status="trusted")
now_iso = datetime.now(timezone.utc).isoformat()
# Three signals from this peer within the window.
for i in range(3):
db.record_signal(dict(
peer_fingerprint=peer_fp,
signal_type="ioc",
signal_id=f"1.2.3.{i}",
signal_hash=f"hash-{i}",
received_at=now_iso,
raw_json="{}",
))
# One stale signal outside the window — must be ignored.
stale = (datetime.now(timezone.utc) - timedelta(hours=48)).isoformat()
db.record_signal(dict(
peer_fingerprint=peer_fp,
signal_type="ioc",
signal_id="9.9.9.9",
signal_hash="stale",
received_at=stale,
raw_json="{}",
))
view = build_local_view()
sig_edges = [e for e in view.edges if e.kind == "signal"]
assert len(sig_edges) == 1
assert sig_edges[0].weight == 3.0
assert sig_edges[0].source_fingerprint == peer_fp
assert sig_edges[0].target_fingerprint == federation.node_fingerprint()
assert view.stats["signals_buffered_24h"] == 3
assert view.stats["distinct_signal_hashes_24h"] == 3
def test_local_view_blocked_peer_renders_with_blocked_status(fresh_db, fed_dir):
fp, pem = _make_peer_pubkey()
federation.register_peer("blocked.example", fp, pem, status="blocked")
view = build_local_view()
peer = next(n for n in view.nodes if not n.is_self)
assert peer.status == "blocked"
# ---------- public view + signature round-trip --------------------------
def test_public_view_excludes_unknown_and_blocked(fresh_db, fed_dir):
fp_t, pem_t = _make_peer_pubkey()
fp_u, pem_u = _make_peer_pubkey()
fp_b, pem_b = _make_peer_pubkey()
federation.register_peer("trusted.example", fp_t, pem_t, status="trusted")
federation.register_peer("unknown.example", fp_u, pem_u, status="unknown")
federation.register_peer("blocked.example", fp_b, pem_b, status="blocked")
payload = build_public_view()
fps = {p["fingerprint"] for p in payload["peers"]}
assert fp_t in fps
assert fp_u not in fps
assert fp_b not in fps
def test_public_view_signature_round_trip(fresh_db, fed_dir):
fp, pem = _make_peer_pubkey()
federation.register_peer("trusted.example", fp, pem, status="trusted")
federation.issue_vouch(fp, ttl_days=30)
payload = build_public_view()
assert "signature" in payload
assert payload["fingerprint"] == federation.node_fingerprint()
sig = base64.b64decode(payload["signature"])
unsigned = {k: v for k, v in payload.items() if k != "signature"}
assert federation.verify_payload(
federation.canonical_json(unsigned),
sig,
federation.public_key_pem(),
) is True
# Vouch we issued is in the payload.
targets = {v["target_fingerprint"] for v in payload["vouches"]}
assert fp in targets
def test_public_view_omits_signals(fresh_db, fed_dir):
"""Public payload must not leak who's reporting what."""
fp, pem = _make_peer_pubkey()
federation.register_peer("trusted.example", fp, pem, status="trusted")
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id="1.2.3.4",
signal_hash="secret-hash",
received_at=datetime.now(timezone.utc).isoformat(),
raw_json="{}",
))
payload = build_public_view()
# No signal-shaped fields anywhere in the payload.
flat = str(payload)
assert "secret-hash" not in flat
assert "signals" not in payload
# ---------- transitive view ---------------------------------------------
def test_transitive_view_adds_distance_2_nodes(fresh_db, fed_dir):
direct_fp, direct_pem = _make_peer_pubkey()
federation.register_peer("direct.example", direct_fp, direct_pem, status="trusted")
# The peer reports two peers of its own.
far_fp_a, _ = _make_peer_pubkey()
far_fp_b, _ = _make_peer_pubkey()
fake_payload: Dict[str, Any] = {
"fingerprint": direct_fp,
"peers": [
{"fingerprint": far_fp_a, "domain": "far-a.example"},
{"fingerprint": far_fp_b, "domain": "far-b.example"},
],
"vouches": [],
}
with patch.object(network_view, "_fetch_peer_network", return_value=fake_payload):
view = build_transitive_view(force_refresh=True)
distances = sorted(n.distance for n in view.nodes)
assert 0 in distances and 1 in distances and 2 in distances
transitive_fps = {n.fingerprint for n in view.nodes if n.distance == 2}
assert far_fp_a in transitive_fps
assert far_fp_b in transitive_fps
# "knows" edges from direct peer to each transitive.
knows = [e for e in view.edges if e.kind == "knows"]
assert len(knows) == 2
assert all(e.source_fingerprint == direct_fp for e in knows)
assert view.stats["transitive_nodes"] == 2
def test_transitive_view_failed_fetch_does_not_abort(fresh_db, fed_dir):
fp_a, pem_a = _make_peer_pubkey()
fp_b, pem_b = _make_peer_pubkey()
federation.register_peer("peer-a.example", fp_a, pem_a, status="trusted")
federation.register_peer("peer-b.example", fp_b, pem_b, status="trusted")
far_fp, _ = _make_peer_pubkey()
def fake_fetch(domain, timeout=4.0):
if domain == "peer-a.example":
return None # simulate a fetch failure
return {
"fingerprint": fp_b,
"peers": [{"fingerprint": far_fp, "domain": "far.example"}],
"vouches": [],
}
with patch.object(network_view, "_fetch_peer_network", side_effect=fake_fetch):
view = build_transitive_view(force_refresh=True)
# Direct nodes both present, transitive only from B.
assert any(n.fingerprint == fp_a for n in view.nodes)
assert any(n.fingerprint == fp_b for n in view.nodes)
assert any(n.fingerprint == far_fp and n.distance == 2 for n in view.nodes)
assert view.stats["transitive_nodes"] == 1
def test_transitive_view_skips_only_unknown_peers(fresh_db, fed_dir):
"""Unknown peers shouldn't be queried — fetcher only called for trusted."""
fp_unknown, pem_u = _make_peer_pubkey()
fp_trusted, pem_t = _make_peer_pubkey()
federation.register_peer("unknown.example", fp_unknown, pem_u, status="unknown")
federation.register_peer("trusted.example", fp_trusted, pem_t, status="trusted")
calls = []
def fake_fetch(domain, timeout=4.0):
calls.append(domain)
return None
with patch.object(network_view, "_fetch_peer_network", side_effect=fake_fetch):
build_transitive_view(force_refresh=True)
assert "trusted.example" in calls
assert "unknown.example" not in calls
# ---------- admin view: per-peer enrichment + corroboration + timeline ---
def _no_transitive():
"""patch.object helper — silence network fetches in admin-view tests."""
return patch.object(network_view, "_fetch_peer_network", return_value=None)
def test_admin_view_includes_stats_on_peer_nodes(fresh_db, fed_dir):
"""Every non-self node must carry a `stats` dict in the admin view."""
fp, pem = _make_peer_pubkey()
federation.register_peer("peer.example", fp, pem, status="trusted")
with _no_transitive():
view = build_admin_view(include_transitive=False)
self_nodes = [n for n in view["nodes"] if n["is_self"]]
peer_nodes = [n for n in view["nodes"] if not n["is_self"]]
assert len(self_nodes) == 1
assert len(peer_nodes) == 1
# Self has no stats; peers do.
assert self_nodes[0]["stats"] is None
peer_stats = peer_nodes[0]["stats"]
assert isinstance(peer_stats, dict)
for key in (
"signals_24h", "signals_total", "cases_24h", "iocs_24h",
"severity_breakdown", "ioc_type_breakdown",
"vouches_in_count", "vouches_out_count",
"quorum_contribution", "last_seen", "last_seen_relative",
"recent_translog",
):
assert key in peer_stats, f"missing {key}"
# last_seen is None when no signals have landed yet.
assert peer_stats["last_seen"] is None
assert peer_stats["last_seen_relative"] == ""
def test_admin_view_signals_24h_excludes_stale(fresh_db, fed_dir):
"""signals_24h must count only rows inside the 24h window."""
fp, pem = _make_peer_pubkey()
federation.register_peer("peer.example", fp, pem, status="trusted")
now_iso = datetime.now(timezone.utc).isoformat()
stale_iso = (datetime.now(timezone.utc) - timedelta(hours=30)).isoformat()
for i in range(3):
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id=f"v{i}",
signal_hash=f"h{i}",
received_at=now_iso,
raw_json="{}",
))
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id="stale",
signal_hash="stale-hash",
received_at=stale_iso,
raw_json="{}",
))
with _no_transitive():
view = build_admin_view(include_transitive=False)
peer = next(n for n in view["nodes"] if not n["is_self"])
assert peer["stats"]["signals_24h"] == 3
# All-time total still sees the stale row.
assert peer["stats"]["signals_total"] == 4
# last_seen is populated and the relative is a short string.
assert peer["stats"]["last_seen"] is not None
assert peer["stats"]["last_seen_relative"] != ""
def test_admin_view_severity_and_ioc_breakdown_from_raw_json(fresh_db, fed_dir):
"""severity_breakdown is read from raw_json for case rows, ioc_type for ioc rows."""
fp, pem = _make_peer_pubkey()
federation.register_peer("peer.example", fp, pem, status="trusted")
now_iso = datetime.now(timezone.utc).isoformat()
cases = [
{"severity": "critical", "case_id": "c1"},
{"severity": "critical", "case_id": "c2"},
{"severity": "high", "case_id": "c3"},
{"severity": "low", "case_id": "c4"},
]
for c in cases:
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="case",
signal_id=c["case_id"],
signal_hash=f"hash-{c['case_id']}",
received_at=now_iso,
raw_json=json.dumps(c),
))
iocs = [
{"type": "url", "value": "https://a"},
{"type": "url", "value": "https://b"},
{"type": "domain", "value": "x.com"},
{"type": "ip", "value": "1.2.3.4"},
]
for ioc in iocs:
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id=ioc["value"],
signal_hash=f"hash-{ioc['value']}",
received_at=now_iso,
raw_json=json.dumps(ioc),
))
with _no_transitive():
view = build_admin_view(include_transitive=False)
stats = next(n for n in view["nodes"] if not n["is_self"])["stats"]
assert stats["cases_24h"] == 4
assert stats["iocs_24h"] == 4
sev = stats["severity_breakdown"]
assert sev == {"critical": 2, "high": 1, "medium": 0, "low": 1}
ioc_t = stats["ioc_type_breakdown"]
assert ioc_t == {"url": 2, "domain": 1, "ip": 1, "hash": 0, "cve": 0}
def test_admin_view_vouch_in_out_counts(fresh_db, fed_dir):
"""vouches_in_count counts vouches naming this peer; out counts what they've issued."""
fp_a, pem_a = _make_peer_pubkey()
fp_b, pem_b = _make_peer_pubkey()
federation.register_peer("a.example", fp_a, pem_a, status="trusted")
federation.register_peer("b.example", fp_b, pem_b, status="trusted")
now = datetime.now(timezone.utc).isoformat()
# A vouches for B; we vouch for B too — B sees vouches_in=2.
db.upsert_vouch(dict(
voucher_fingerprint=fp_a,
target_fingerprint=fp_b,
issued_at=now, expires_at=None, signature="x",
))
federation.issue_vouch(fp_b, ttl_days=30)
# B vouches for A — A sees vouches_in=1, B sees vouches_out=1.
db.upsert_vouch(dict(
voucher_fingerprint=fp_b,
target_fingerprint=fp_a,
issued_at=now, expires_at=None, signature="y",
))
with _no_transitive():
view = build_admin_view(include_transitive=False)
by_fp = {n["fingerprint"]: n for n in view["nodes"] if not n["is_self"]}
assert by_fp[fp_a]["stats"]["vouches_in_count"] == 1
assert by_fp[fp_a]["stats"]["vouches_out_count"] == 1 # vouches for B
assert by_fp[fp_b]["stats"]["vouches_in_count"] == 2 # A + us
assert by_fp[fp_b]["stats"]["vouches_out_count"] == 1 # vouches for A
def test_admin_view_corroborated_signals(fresh_db, fed_dir):
"""Pairs of peers reporting the same signal_hash → corroborated entry + edge."""
fp_a, pem_a = _make_peer_pubkey()
fp_b, pem_b = _make_peer_pubkey()
federation.register_peer("a.example", fp_a, pem_a, status="trusted")
federation.register_peer("b.example", fp_b, pem_b, status="trusted")
now_iso = datetime.now(timezone.utc).isoformat()
for peer_fp in (fp_a, fp_b):
db.record_signal(dict(
peer_fingerprint=peer_fp,
signal_type="ioc",
signal_id="evil.com",
signal_hash="shared-hash-1",
received_at=now_iso,
raw_json="{}",
))
# A also reports a hash B doesn't — should NOT corroborate.
db.record_signal(dict(
peer_fingerprint=fp_a,
signal_type="ioc",
signal_id="solo.com",
signal_hash="solo-hash",
received_at=now_iso,
raw_json="{}",
))
with _no_transitive():
view = build_admin_view(include_transitive=False)
corr = view["stats"]["corroborated_signals"]
hashes = {c["signal_hash"] for c in corr}
assert "shared-hash-1" in hashes
assert "solo-hash" not in hashes
shared = next(c for c in corr if c["signal_hash"] == "shared-hash-1")
assert set(shared["peer_fingerprints"]) == {fp_a, fp_b}
assert shared["peer_count"] == 2
# One corroborate edge between the pair (orientation-independent).
corr_edges = [e for e in view["edges"] if e["kind"] == "corroborate"]
assert len(corr_edges) == 1
pair = {corr_edges[0]["source_fingerprint"], corr_edges[0]["target_fingerprint"]}
assert pair == {fp_a, fp_b}
assert corr_edges[0]["weight"] == 1.0
def test_admin_view_signal_timeline_24h_returns_24_buckets(fresh_db, fed_dir):
"""signal_timeline_24h is a 24-bucket list with correct totals."""
fp, pem = _make_peer_pubkey()
federation.register_peer("peer.example", fp, pem, status="trusted")
now = datetime.now(timezone.utc)
# Two signals one hour ago, three signals five hours ago.
one_h = (now - timedelta(hours=1, minutes=5)).isoformat()
five_h = (now - timedelta(hours=5, minutes=5)).isoformat()
for i in range(2):
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id=f"a{i}",
signal_hash=f"h-a-{i}",
received_at=one_h,
raw_json="{}",
))
for i in range(3):
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id=f"b{i}",
signal_hash=f"h-b-{i}",
received_at=five_h,
raw_json="{}",
))
# Stale signal — must NOT show up.
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id="stale",
signal_hash="stale-hash",
received_at=(now - timedelta(hours=48)).isoformat(),
raw_json="{}",
))
with _no_transitive():
view = build_admin_view(include_transitive=False)
buckets = view["stats"]["signal_timeline_24h"]
assert isinstance(buckets, list)
assert len(buckets) == 24
totals = [b["total"] for b in buckets]
assert sum(totals) == 5 # stale excluded
# Bucket hour_offsets are 0..23 in oldest-first order.
assert [b["hour_offset"] for b in buckets] == list(range(24))
def test_admin_view_quorum_contribution(fresh_db, fed_dir):
"""quorum_contribution counts this peer's distinct hashes that are quorum-met."""
fp_a, pem_a = _make_peer_pubkey()
fp_b, pem_b = _make_peer_pubkey()
federation.register_peer("a.example", fp_a, pem_a, status="trusted")
federation.register_peer("b.example", fp_b, pem_b, status="trusted")
now_iso = datetime.now(timezone.utc).isoformat()
# Shared hash → both peers report it → quorum-met (default k=2).
for peer_fp in (fp_a, fp_b):
db.record_signal(dict(
peer_fingerprint=peer_fp,
signal_type="ioc",
signal_id="shared",
signal_hash="quorum-hash",
received_at=now_iso,
raw_json="{}",
))
# Solo hash from A → not quorum-met.
db.record_signal(dict(
peer_fingerprint=fp_a,
signal_type="ioc",
signal_id="solo",
signal_hash="solo-hash",
received_at=now_iso,
raw_json="{}",
))
with _no_transitive():
view = build_admin_view(include_transitive=False)
by_fp = {n["fingerprint"]: n for n in view["nodes"] if not n["is_self"]}
assert by_fp[fp_a]["stats"]["quorum_contribution"] == 1
assert by_fp[fp_b]["stats"]["quorum_contribution"] == 1
def test_admin_view_recent_translog_per_peer(fresh_db, fed_dir):
"""recent_translog lists entries where entry_data.peer_fingerprint matches."""
fp_a, pem_a = _make_peer_pubkey()
fp_b, pem_b = _make_peer_pubkey()
federation.register_peer("a.example", fp_a, pem_a, status="trusted")
federation.register_peer("b.example", fp_b, pem_b, status="trusted")
# Append translog rows that name each peer.
translog.append("signal", {"peer_fingerprint": fp_a, "signal_type": "ioc", "signal_id": "x"})
translog.append("signal", {"peer_fingerprint": fp_b, "signal_type": "case", "signal_id": "y"})
translog.append("signal", {"peer_fingerprint": fp_a, "signal_type": "case", "signal_id": "z"})
with _no_transitive():
view = build_admin_view(include_transitive=False)
by_fp = {n["fingerprint"]: n for n in view["nodes"] if not n["is_self"]}
a_log = by_fp[fp_a]["stats"]["recent_translog"]
b_log = by_fp[fp_b]["stats"]["recent_translog"]
assert len(a_log) == 2
assert len(b_log) == 1
# Each row carries the documented shape.
for row in a_log + b_log:
assert set(row.keys()) == {"id", "entry_type", "timestamp", "hash"}
def test_explore_view_omits_ioc_values_case_ids_and_raw_json(fresh_db, fed_dir):
"""The public explore payload must NEVER expose IOC values, case_ids, or raw_json.
This is the load-bearing transparency-vs-leakage contract that lives at
the network-view layer — anyone can audit who's talking to whom and how
much, but never *what* they're saying.
"""
fp, pem = _make_peer_pubkey()
federation.register_peer("trusted.example", fp, pem, status="trusted")
now_iso = datetime.now(timezone.utc).isoformat()
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id="evil-domain-do-not-leak.com",
signal_hash="ioc-hash-leak",
received_at=now_iso,
raw_json=json.dumps({"type": "domain", "value": "evil-domain-do-not-leak.com"}),
))
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="case",
signal_id="CASE-SECRET-42",
signal_hash="case-hash-leak",
received_at=now_iso,
raw_json=json.dumps({"severity": "critical", "case_id": "CASE-SECRET-42"}),
))
with patch.object(network_view, "_fetch_peer_explore", return_value=None), \
patch.object(network_view, "_fetch_peer_network", return_value=None):
payload = build_explore_view()
flat = json.dumps(payload, default=str)
assert "evil-domain-do-not-leak.com" not in flat
assert "CASE-SECRET-42" not in flat
assert "raw_json" not in flat
# Sector-leaking breakdowns must not appear either.
assert "severity_breakdown" not in flat
assert "ioc_type_breakdown" not in flat
# And peer rows carry only public-safe counts.
for p in payload.get("peers", []):
assert "severity_breakdown" not in p
assert "ioc_type_breakdown" not in p
assert "recent_translog" not in p
def test_public_view_still_has_no_stats(fresh_db, fed_dir):
"""Public payload must not surface admin-only enrichments — sensitive.
Even after `build_admin_view` has been invoked (which mutates node.stats
on the cached transitive view), the public view path must stay clean.
"""
fp, pem = _make_peer_pubkey()
federation.register_peer("trusted.example", fp, pem, status="trusted")
# Seed signals + corroborated hash so admin view has rich state.
now_iso = datetime.now(timezone.utc).isoformat()
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id="leak",
signal_hash="leak-hash",
received_at=now_iso,
raw_json=json.dumps({"type": "url", "value": "https://leak"}),
))
# Build admin view first so any caching kicks in.
with _no_transitive():
build_admin_view(include_transitive=False)
# Now build the public view and assert no admin-only fields leak.
payload = build_public_view()
flat = json.dumps(payload, default=str)
assert "signals_24h" not in flat
assert "severity_breakdown" not in flat
assert "corroborated_signals" not in flat
assert "signal_timeline_24h" not in flat
assert "recent_translog" not in flat
assert "leak-hash" not in flat
# Peer entries in the public view never carry a `stats` field.
for p in payload.get("peers", []):
assert "stats" not in p

102
tests/test_news.py Normal file
View File

@@ -0,0 +1,102 @@
"""Newsline — start-page digest aggregation."""
from __future__ import annotations
import pytest
from sqlalchemy import create_engine
from psyc import db
from psyc.lines import ledger as ledger_line
from psyc.lines import news
from psyc.models import Outcome, Severity, TLP
from conftest import make_case
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
def test_kpis_count_basic(fresh_db):
# vary age_days so make_case() doesn't collide its case_id
db.upsert_case(make_case(feed="urlhaus", urls=["http://1.1.1.1/x"], severity=Severity.LOW, age_days=1))
db.upsert_case(make_case(feed="feodo", ips=["2.2.2.2"], severity=Severity.HIGH, age_days=0))
db.upsert_case(make_case(feed="urlhaus", urls=["http://3.3.3.3/x"], severity=Severity.CRITICAL, age_days=2))
k = news.kpis()
assert k["cases"] == 3
assert k["high_total"] == 2 # high + critical
def test_recent_items_interleaves_ledger_and_cases(fresh_db):
c = make_case(feed="feodo", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(c)
ledger_line.write(
case_id=c.case_id, destination="CERT-Bund", payload_hash="",
submitter_identity="x", tlp=TLP.AMBER, outcome=Outcome.ACTIONED,
)
items = news.recent_items(limit=10)
kinds = {i.kind for i in items}
assert "case" in kinds
assert "enforced" in kinds
# newest-first ordering
assert items == sorted(items, key=lambda i: i.timestamp, reverse=True)
def test_feed_health_groups_by_feed(fresh_db):
db.upsert_case(make_case(feed="urlhaus", urls=["http://a/"], age_days=1))
db.upsert_case(make_case(feed="urlhaus", urls=["http://b/"], age_days=2))
db.upsert_case(make_case(feed="otx", ips=["1.1.1.1"], age_days=1))
h = news.feed_health()
by_feed = {f.feed: f for f in h}
assert by_feed["urlhaus"].count == 2
assert by_feed["otx"].count == 1
# highest count first
assert h[0].feed == "urlhaus"
def test_bucket_items_groups_by_recency(fresh_db):
from datetime import datetime, timedelta, timezone
now = datetime.now(timezone.utc)
items = [
news.NewsItem(timestamp=now, kind="case", headline="t1", body="", icon=""),
news.NewsItem(timestamp=now - timedelta(days=1), kind="case", headline="t2", body="", icon=""),
news.NewsItem(timestamp=now - timedelta(days=3), kind="case", headline="t3", body="", icon=""),
news.NewsItem(timestamp=now - timedelta(days=14), kind="case", headline="t4", body="", icon=""),
]
labels = [b.label for b in news.bucket_items(items)]
# all four buckets should appear, in chronological order
assert labels == ["Today", "Yesterday", "Earlier this week", "Older"]
def test_featured_case_picks_highest_severity(fresh_db):
db.upsert_case(make_case(feed="urlhaus", urls=["http://1.1.1.1/x"], severity=Severity.LOW, age_days=0))
db.upsert_case(make_case(feed="urlhaus", urls=["http://2.2.2.2/x"], severity=Severity.HIGH, age_days=1))
db.upsert_case(make_case(feed="urlhaus", urls=["http://3.3.3.3/x"], severity=Severity.CRITICAL, age_days=2))
f = news.featured_case()
assert f is not None
assert f.classification.severity is Severity.CRITICAL
def test_featured_case_none_when_nothing_high(fresh_db):
db.upsert_case(make_case(feed="urlhaus", urls=["http://1.1.1.1/x"], severity=Severity.LOW, age_days=0))
db.upsert_case(make_case(feed="urlhaus", urls=["http://2.2.2.2/x"], severity=Severity.MEDIUM, age_days=1))
assert news.featured_case() is None
def test_outcome_kinds_match_render_map(fresh_db):
# Every Outcome should produce a NewsItem (no KeyError).
c = make_case(feed="urlhaus", ips=["1.2.3.4"])
db.upsert_case(c)
for outcome in Outcome:
ledger_line.write(
case_id=c.case_id, destination="X", payload_hash="",
submitter_identity="x", tlp=TLP.AMBER, outcome=outcome,
)
items = news.recent_items(limit=200)
# at least one item per outcome we wrote
assert len([i for i in items if i.kind != "case"]) >= len(list(Outcome))

218
tests/test_pulse.py Normal file
View File

@@ -0,0 +1,218 @@
"""Pulse scheduler tests — cadence, kill switch, mode gating, persistence."""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
import pytest
from sqlalchemy import create_engine
from psyc import db
from psyc.lines import pulse
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
"""A temp SQLite + a swapped-in registry of fast, deterministic runners.
Real runners would talk to scout / classify / proof / etc — none of which
we want to exercise from a pulse-only test. We replace _REGISTRY in-place
so tick() drives the scheduler logic against trivial counters.
"""
test_db = tmp_path / "pulse.db"
test_engine = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(test_engine, checkfirst=True)
monkeypatch.setattr(db, "_engine", test_engine)
monkeypatch.setattr(db, "DB_PATH", test_db)
counters = {name: 0 for name in pulse._REGISTRY}
def _make(name: str):
def runner() -> str:
counters[name] += 1
return f"{name}: tick {counters[name]}"
return runner
fake_registry = {name: _make(name) for name in pulse._REGISTRY}
monkeypatch.setattr(pulse, "_REGISTRY", fake_registry)
return counters
def _set_last_fired(name: str, when: datetime, cadence: int) -> None:
"""Force a pipeline's last_fired + next_fire to a known instant."""
for p in pulse.state():
if p.name == name:
p.last_fired = when
p.next_fire = when + timedelta(seconds=cadence)
db.upsert_pulse_pipeline(pulse._pipeline_to_row(p))
return
raise AssertionError(f"unknown pipeline {name}")
def test_defaults_seeded_on_first_state(fresh_db):
pipelines = pulse.state()
names = {p.name for p in pipelines}
assert names == {"fetch", "classify", "prove", "reindex", "respond", "peer-pull", "vouch-refresh"}
# Spot-check a couple of defaults from the spec.
by_name = {p.name: p for p in pipelines}
assert by_name["fetch"].mode == pulse.PulseMode.AUTO_EXECUTE
assert by_name["fetch"].cadence_seconds == 900
assert by_name["peer-pull"].enabled is False
assert by_name["respond"].mode == pulse.PulseMode.AUTO_PROPOSE
def test_state_persists_across_reads(fresh_db):
pulse.set_cadence("fetch", 42)
pulse.set_mode("classify", pulse.PulseMode.MANUAL)
pulse.set_enabled("reindex", False)
by_name = {p.name: p for p in pulse.state()}
assert by_name["fetch"].cadence_seconds == 42
assert by_name["classify"].mode == pulse.PulseMode.MANUAL
assert by_name["reindex"].enabled is False
def test_tick_skips_not_due_pipelines(fresh_db):
# Push every pipeline well into the future — nothing should fire.
now = datetime.now(timezone.utc)
for p in pulse.state():
_set_last_fired(p.name, now, p.cadence_seconds)
results = pulse.tick()
outcomes = {name: outcome for name, outcome, _ in results}
assert all(o == "skipped" for o in outcomes.values()), outcomes
assert all(c == 0 for c in fresh_db.values()), fresh_db
def test_tick_fires_due_auto_pipelines(fresh_db):
# Force "fetch" to be due (last fired far in the past) and re-tick.
past = datetime.now(timezone.utc) - timedelta(hours=24)
for p in pulse.state():
_set_last_fired(p.name, past, p.cadence_seconds)
results = pulse.tick()
by_name = {name: (outcome, result) for name, outcome, result in results}
# auto-execute + auto-propose modes should fire; manual should skip.
assert by_name["fetch"][0] == "ok"
assert by_name["classify"][0] == "ok"
assert by_name["prove"][0] == "ok"
assert by_name["reindex"][0] == "ok"
assert by_name["respond"][0] == "ok"
# Manual + disabled pipelines stay skipped.
assert by_name["peer-pull"][0] == "skipped"
assert by_name["vouch-refresh"][0] == "skipped"
def test_tick_respects_manual_mode(fresh_db):
past = datetime.now(timezone.utc) - timedelta(hours=24)
_set_last_fired("fetch", past, 900)
pulse.set_mode("fetch", pulse.PulseMode.MANUAL)
results = pulse.tick()
by_name = {n: o for n, o, _ in results}
assert by_name["fetch"] == "skipped"
assert fresh_db["fetch"] == 0
def test_tick_respects_disabled(fresh_db):
past = datetime.now(timezone.utc) - timedelta(hours=24)
_set_last_fired("fetch", past, 900)
pulse.set_enabled("fetch", False)
results = pulse.tick()
by_name = {n: o for n, o, _ in results}
assert by_name["fetch"] == "skipped"
assert fresh_db["fetch"] == 0
def test_kill_switch_halts_everything(fresh_db):
past = datetime.now(timezone.utc) - timedelta(hours=24)
for p in pulse.state():
_set_last_fired(p.name, past, p.cadence_seconds)
pulse.set_kill_switch(True)
results = pulse.tick()
assert all(o == "skipped" for _, o, _ in results)
assert all(c == 0 for c in fresh_db.values())
# And killswitch state itself round-trips.
assert pulse.kill_switch_state() is True
pulse.set_kill_switch(False)
assert pulse.kill_switch_state() is False
def test_tick_updates_last_fired_and_next_fire(fresh_db):
past = datetime.now(timezone.utc) - timedelta(hours=24)
_set_last_fired("fetch", past, 900)
before = datetime.now(timezone.utc)
pulse.tick()
p = {x.name: x for x in pulse.state()}["fetch"]
assert p.last_fired is not None
assert p.last_fired >= before - timedelta(seconds=2)
assert p.next_fire is not None
assert p.next_fire > p.last_fired
delta = (p.next_fire - p.last_fired).total_seconds()
# 900s cadence, allow a 2s rounding window.
assert 898 <= delta <= 902
assert p.last_outcome == "ok"
assert "fetch: tick" in p.last_result
def test_run_now_bypasses_cadence(fresh_db):
# Pipeline isn't due — run_now must still fire.
future = datetime.now(timezone.utc) + timedelta(hours=1)
for p in pulse.state():
p.last_fired = datetime.now(timezone.utc)
p.next_fire = future
db.upsert_pulse_pipeline(pulse._pipeline_to_row(p))
outcome, result = pulse.run_now("fetch")
assert outcome == "ok"
assert "fetch: tick" in result
assert fresh_db["fetch"] == 1
def test_run_now_respects_kill_switch(fresh_db):
pulse.set_kill_switch(True)
outcome, result = pulse.run_now("fetch")
assert outcome == "skipped"
assert "kill switch" in result
assert fresh_db["fetch"] == 0
def test_run_now_even_when_manual(fresh_db):
"""Manual mode blocks tick() but NOT run_now — that's the whole point of manual."""
pulse.set_mode("peer-pull", pulse.PulseMode.MANUAL)
pulse.set_enabled("peer-pull", True)
outcome, _ = pulse.run_now("peer-pull")
assert outcome == "ok"
assert fresh_db["peer-pull"] == 1
def test_runner_exception_recorded_as_err(fresh_db, monkeypatch):
def boom() -> str:
raise RuntimeError("nope")
# Inject a failing runner just for "fetch".
bad_registry = dict(pulse._REGISTRY)
bad_registry["fetch"] = boom
monkeypatch.setattr(pulse, "_REGISTRY", bad_registry)
past = datetime.now(timezone.utc) - timedelta(hours=24)
_set_last_fired("fetch", past, 900)
results = pulse.tick()
by_name = {n: (o, r) for n, o, r in results}
assert by_name["fetch"][0] == "err"
assert "nope" in by_name["fetch"][1]
# And the failure is persisted, with next_fire still pushed forward so we
# don't busy-retry a broken runner every tick.
p = {x.name: x for x in pulse.state()}["fetch"]
assert p.last_outcome == "err"
assert p.next_fire is not None and p.next_fire > p.last_fired
def test_unknown_pipeline_raises(fresh_db):
with pytest.raises(ValueError):
pulse.set_mode("does-not-exist", pulse.PulseMode.MANUAL)
with pytest.raises(ValueError):
pulse.set_cadence("does-not-exist", 60)
with pytest.raises(ValueError):
pulse.run_now("does-not-exist")
def test_invalid_cadence_raises(fresh_db):
with pytest.raises(ValueError):
pulse.set_cadence("fetch", 0)
with pytest.raises(ValueError):
pulse.set_cadence("fetch", -5)

314
tests/test_pulse_respond.py Normal file
View File

@@ -0,0 +1,314 @@
"""Pulseline auto-response gating — severity threshold, quorum, local-only.
The runner here is the live `_run_respond` from pulse.py. We point it at a
temp DB, monkeypatch federation.is_quorum_met to a controllable function, and
swap respond.execute_action for a counter so we don't reach the SOAR sink.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import List, Tuple
import pytest
from sqlalchemy import create_engine
from psyc import db
from psyc.lines import pulse, respond
from psyc.lines import federation
from psyc.models import (
ActionStatus,
ActionType,
Case,
Classification,
Observables,
ResponseAction,
Severity,
TLP,
)
from psyc.result import Ok
from conftest import make_case
# ----- fixtures --------------------------------------------------------------
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
"""Temp SQLite + the real runner registry. Mode pinned to auto-execute."""
test_db = tmp_path / "respond.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
@pytest.fixture
def fired(monkeypatch):
"""Capture every execute_action(action_id, approver=...) — no SOAR sink call."""
log: List[Tuple[int, str]] = []
def fake_execute(action_id: int, approver: str = "operator"):
log.append((action_id, approver))
# Re-read the action so we can return a realistic Ok value
got = respond.get_action(action_id)
return got if isinstance(got, Ok) else got
monkeypatch.setattr(respond, "execute_action", fake_execute)
return log
@pytest.fixture
def quorum_yes(monkeypatch):
monkeypatch.setattr(federation, "is_quorum_met",
lambda h, k=None: True, raising=False)
@pytest.fixture
def quorum_no(monkeypatch):
monkeypatch.setattr(federation, "is_quorum_met",
lambda h, k=None: False, raising=False)
def _set_respond_mode(mode: pulse.PulseMode) -> None:
pulse.set_mode("respond", mode)
def _propose_one(case: Case) -> int:
db.upsert_case(case)
ids = respond.propose_for_case(case)
assert ids, "test setup expected at least one action proposed"
return ids[0]
# ----- severity rank ---------------------------------------------------------
def test_severity_rank_ordering():
assert pulse._severity_rank(Severity.LOW) == 0
assert pulse._severity_rank(Severity.MEDIUM) == 1
assert pulse._severity_rank(Severity.HIGH) == 2
assert pulse._severity_rank(Severity.CRITICAL) == 3
assert pulse._severity_rank(None) == -1
# ----- runner mode gating ----------------------------------------------------
def test_runner_no_auto_fire_when_mode_is_propose(fresh_db, fired, quorum_yes):
case = make_case(feed="feodo", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
# default seed mode for respond is auto-propose → no auto-fire even with PROPOSED actions
result = pulse._run_respond()
assert "no auto-fire" in result
assert fired == []
def test_runner_no_auto_fire_when_manual(fresh_db, fired, quorum_yes):
case = make_case(feed="feodo", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
_set_respond_mode(pulse.PulseMode.MANUAL)
result = pulse._run_respond()
assert "no auto-fire" in result
assert fired == []
# ----- severity threshold ----------------------------------------------------
def test_below_threshold_is_skipped(fresh_db, fired, quorum_yes):
# Propose an action carrying severity=MEDIUM by hand — propose_for_case
# only generates HIGH/CRITICAL actions, but the gate must still work for
# any below-threshold severity we drop in.
case = make_case(feed="feodo", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
respond.propose_for_case(case)
# Demote every action's severity to MEDIUM so all should be skipped under HIGH threshold.
from sqlalchemy import update as sa_update
with db.engine().begin() as conn:
conn.execute(sa_update(db.response_actions).values(severity=Severity.MEDIUM.value))
pulse.set_respond_auto_threshold(Severity.HIGH)
_set_respond_mode(pulse.PulseMode.AUTO_EXECUTE)
pulse._run_respond()
assert fired == [], "below-threshold action must not fire"
audit = db.pulse_audit_recent("respond", limit=5)
assert any(r["action"] == "skipped" and "below threshold" in (r["detail"] or "") for r in audit)
# ----- quorum gate -----------------------------------------------------------
def test_federation_case_no_quorum_skipped(fresh_db, fired, quorum_no):
case = make_case(feed="urlhaus", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
# Mark this case as federation-sourced by inserting a signal row for it.
db.record_signal(dict(
peer_fingerprint="peer-a",
signal_type="case",
signal_id=case.case_id,
signal_hash="dummyhash",
received_at=datetime.now(timezone.utc).isoformat(),
raw_json="{}",
))
respond.propose_for_case(case)
pulse.set_respond_require_quorum(True)
pulse.set_respond_local_only(False)
pulse.set_respond_auto_threshold(Severity.HIGH)
_set_respond_mode(pulse.PulseMode.AUTO_EXECUTE)
pulse._run_respond()
assert fired == []
audit = db.pulse_audit_recent("respond", limit=5)
assert any(r["action"] == "skipped" and "no quorum" in (r["detail"] or "") for r in audit)
def test_local_case_fires_when_quorum_required(fresh_db, fired, quorum_no):
"""Locally-generated cases bypass quorum — they're our own work."""
case = make_case(feed="urlhaus", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
# No federation_signals row → locally-generated
respond.propose_for_case(case)
pulse.set_respond_require_quorum(True)
pulse.set_respond_local_only(True) # both armed; local cases still fire
pulse.set_respond_auto_threshold(Severity.HIGH)
_set_respond_mode(pulse.PulseMode.AUTO_EXECUTE)
pulse._run_respond()
assert len(fired) >= 1
audit = db.pulse_audit_recent("respond", limit=10)
assert any(r["action"] == "auto-fire" for r in audit)
def test_local_case_fires_local_only_off(fresh_db, fired, quorum_no):
"""Even with local_only OFF, a locally-generated case still fires (no quorum needed)."""
case = make_case(feed="urlhaus", ips=["1.1.1.1"], severity=Severity.CRITICAL)
db.upsert_case(case)
respond.propose_for_case(case)
pulse.set_respond_require_quorum(True)
pulse.set_respond_local_only(False)
pulse.set_respond_auto_threshold(Severity.HIGH)
_set_respond_mode(pulse.PulseMode.AUTO_EXECUTE)
pulse._run_respond()
assert len(fired) >= 1
def test_federation_case_with_quorum_fires(fresh_db, fired, quorum_yes):
case = make_case(feed="urlhaus", ips=["2.2.2.2"], severity=Severity.HIGH)
db.upsert_case(case)
db.record_signal(dict(
peer_fingerprint="peer-b",
signal_type="case",
signal_id=case.case_id,
signal_hash="dummyhash2",
received_at=datetime.now(timezone.utc).isoformat(),
raw_json="{}",
))
respond.propose_for_case(case)
pulse.set_respond_require_quorum(True)
pulse.set_respond_local_only(False)
pulse.set_respond_auto_threshold(Severity.HIGH)
_set_respond_mode(pulse.PulseMode.AUTO_EXECUTE)
pulse._run_respond()
assert len(fired) >= 1
def test_quorum_off_fires_federation_case(fresh_db, fired, quorum_no):
"""With quorum gating disabled entirely, federation cases fire too."""
case = make_case(feed="urlhaus", ips=["3.3.3.3"], severity=Severity.HIGH)
db.upsert_case(case)
db.record_signal(dict(
peer_fingerprint="peer-c",
signal_type="case",
signal_id=case.case_id,
signal_hash="dummyhash3",
received_at=datetime.now(timezone.utc).isoformat(),
raw_json="{}",
))
respond.propose_for_case(case)
pulse.set_respond_require_quorum(False)
pulse.set_respond_auto_threshold(Severity.HIGH)
_set_respond_mode(pulse.PulseMode.AUTO_EXECUTE)
pulse._run_respond()
assert len(fired) >= 1
# ----- kill switch -----------------------------------------------------------
def test_kill_switch_blocks_tick(fresh_db, fired, quorum_yes):
"""The parent tick() skips everything when kill switch is armed."""
case = make_case(feed="feodo", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
respond.propose_for_case(case)
_set_respond_mode(pulse.PulseMode.AUTO_EXECUTE)
pulse.set_kill_switch(True)
results = pulse.tick()
assert all(o == "skipped" for _, o, _ in results)
assert fired == []
# ----- audit -----------------------------------------------------------------
def test_pulse_audit_records_fire_and_skip(fresh_db, fired, quorum_no):
# Local case → should fire and audit auto-fire
local = make_case(feed="urlhaus", ips=["10.0.0.1"], severity=Severity.HIGH, age_days=1)
db.upsert_case(local)
respond.propose_for_case(local)
# Federation-sourced case w/o quorum → should skip and audit skip
fedcase = make_case(feed="urlhaus", ips=["10.0.0.2"], severity=Severity.HIGH, age_days=2)
db.upsert_case(fedcase)
db.record_signal(dict(
peer_fingerprint="peer-x",
signal_type="case",
signal_id=fedcase.case_id,
signal_hash="xhash",
received_at=datetime.now(timezone.utc).isoformat(),
raw_json="{}",
))
respond.propose_for_case(fedcase)
pulse.set_respond_require_quorum(True)
pulse.set_respond_local_only(False)
pulse.set_respond_auto_threshold(Severity.HIGH)
_set_respond_mode(pulse.PulseMode.AUTO_EXECUTE)
pulse._run_respond()
audit = db.pulse_audit_recent("respond", limit=20)
actions = {r["action"] for r in audit}
assert "auto-fire" in actions
assert "skipped" in actions
def test_audit_count_since(fresh_db, fired, quorum_no):
case = make_case(feed="urlhaus", ips=["8.8.8.8"], severity=Severity.HIGH)
db.upsert_case(case)
respond.propose_for_case(case)
pulse.set_respond_require_quorum(True)
pulse.set_respond_auto_threshold(Severity.HIGH)
_set_respond_mode(pulse.PulseMode.AUTO_EXECUTE)
pulse._run_respond()
from datetime import timedelta
since = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()
assert db.pulse_audit_count_since("respond", "auto-fire", since) >= 1
# ----- config round-trip -----------------------------------------------------
def test_config_round_trips(fresh_db):
assert pulse.respond_auto_threshold() == Severity.HIGH
assert pulse.respond_require_quorum() is True
assert pulse.respond_local_only() is False
pulse.set_respond_auto_threshold(Severity.CRITICAL)
pulse.set_respond_require_quorum(False)
pulse.set_respond_local_only(True)
assert pulse.respond_auto_threshold() == Severity.CRITICAL
assert pulse.respond_require_quorum() is False
assert pulse.respond_local_only() is True

144
tests/test_respond.py Normal file
View File

@@ -0,0 +1,144 @@
"""Respondline — proposal gating, human-gated execution, rejection."""
from __future__ import annotations
import pytest
from sqlalchemy import create_engine
from psyc import db
from psyc.lines import ledger as ledger_line
from psyc.lines import respond
from psyc.models import ActionStatus, ActionType, Outcome, Severity
from psyc.result import Ok
from conftest import make_case
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
def test_low_severity_proposes_nothing(fresh_db):
case = make_case(feed="urlhaus", ips=["1.2.3.4"], severity=Severity.LOW)
db.upsert_case(case)
assert respond.propose_for_case(case) == []
def test_high_severity_proposes_alert_and_blocklist(fresh_db):
case = make_case(feed="feodo", ips=["9.9.9.9"], domains=["evil.com"], severity=Severity.HIGH)
db.upsert_case(case)
ids = respond.propose_for_case(case)
assert len(ids) == 2
actions = respond.list_actions(status=ActionStatus.PROPOSED)
types = {a.action_type for a in actions}
assert types == {ActionType.ALERT, ActionType.BLOCKLIST}
def test_proposal_is_idempotent_per_case(fresh_db):
case = make_case(feed="feodo", ips=["9.9.9.9"], severity=Severity.CRITICAL)
db.upsert_case(case)
respond.propose_for_case(case)
assert respond.propose_for_case(case) == [] # second call adds nothing
assert respond.action_count(ActionStatus.PROPOSED) == 2
def test_blocklist_skipped_when_no_network_iocs(fresh_db):
case = make_case(feed="malware-bazaar", hashes=["a" * 64], severity=Severity.HIGH)
db.upsert_case(case)
respond.propose_for_case(case)
actions = respond.list_actions()
# hash-only case → alert yes, blocklist no
assert {a.action_type for a in actions} == {ActionType.ALERT}
def test_execute_fires_and_marks_executed(fresh_db, monkeypatch):
case = make_case(feed="feodo", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
aid = respond.propose_for_case(case)[0]
captured = {}
class _Resp:
def raise_for_status(self): pass
def json(self): return {"receipt_id": "MOCK-AB12"}
class _Client:
def __init__(self, *a, **k): pass
def __enter__(self): return self
def __exit__(self, *a): return False
def post(self, url, json):
captured["url"] = url
captured["payload"] = json
return _Resp()
monkeypatch.setattr(respond.httpx, "Client", _Client)
result = respond.execute_action(aid, approver="alice")
assert isinstance(result, Ok)
assert result.value.status is ActionStatus.EXECUTED
assert result.value.approver == "alice"
assert captured["payload"]["approved_by"] == "alice"
# ledger has an ACTIONED row
entries = ledger_line.list_by_case(case.case_id, limit=10)
assert any(e.outcome is Outcome.ACTIONED for e in entries)
def test_execute_failure_marks_failed(fresh_db, monkeypatch):
case = make_case(feed="feodo", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
aid = respond.propose_for_case(case)[0]
class _Client:
def __init__(self, *a, **k): pass
def __enter__(self): return self
def __exit__(self, *a): return False
def post(self, url, json): raise RuntimeError("sink down")
monkeypatch.setattr(respond.httpx, "Client", _Client)
result = respond.execute_action(aid)
assert not isinstance(result, Ok)
assert respond.get_action(aid).value.status is ActionStatus.FAILED
def test_reject_fires_nothing_and_records(fresh_db, monkeypatch):
case = make_case(feed="feodo", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
aid = respond.propose_for_case(case)[0]
def boom(*a, **k):
raise AssertionError("reject must not POST")
monkeypatch.setattr(respond.httpx, "Client", boom)
result = respond.reject_action(aid, approver="bob", reason="false positive")
assert isinstance(result, Ok)
a = respond.get_action(aid).value
assert a.status is ActionStatus.REJECTED
assert a.approver == "bob"
entries = ledger_line.list_by_case(case.case_id, limit=10)
assert any(e.outcome is Outcome.REJECTED and "false positive" in (e.detail or "") for e in entries)
def test_double_execute_refused(fresh_db, monkeypatch):
case = make_case(feed="feodo", ips=["9.9.9.9"], severity=Severity.HIGH)
db.upsert_case(case)
aid = respond.propose_for_case(case)[0]
class _Resp:
def raise_for_status(self): pass
def json(self): return {"receipt_id": "X"}
class _Client:
def __init__(self, *a, **k): pass
def __enter__(self): return self
def __exit__(self, *a): return False
def post(self, url, json): return _Resp()
monkeypatch.setattr(respond.httpx, "Client", _Client)
respond.execute_action(aid)
again = respond.execute_action(aid)
assert not isinstance(again, Ok) # already executed

View File

@@ -2,7 +2,14 @@
from __future__ import annotations
from psyc.lines.scout import _feodo_record_to_case, _kev_vuln_to_case, _parse_urlhaus_csv
from psyc.lines.scout import (
_feodo_record_to_case,
_kev_vuln_to_case,
_mb_row_to_case,
_otx_pulse_to_case,
_parse_urlhaus_csv,
_threatfox_row_to_case,
)
URLHAUS_CSV = """\
# comment line
@@ -47,3 +54,96 @@ def test_feodo_record_to_case():
assert case.source_metadata["feed"] == "feodo"
assert case.source_metadata["malware"] == "Emotet"
assert case.source_metadata["status"] == "online"
def test_threatfox_row_url_to_case():
row = {
"id": "1234567",
"ioc": "http://1.2.3.4/x.bin",
"ioc_type": "url",
"threat_type": "payload_delivery",
"malware_printable": "Cobalt Strike",
"first_seen": "2026-05-19 10:00:00",
"confidence_level": 100,
"tags": ["c2", "stager"],
"reporter": "anon",
}
case = _threatfox_row_to_case(row)
assert case is not None
assert case.case_id == "PSYC-THREATFOX-1234567"
assert case.observables.urls == ["http://1.2.3.4/x.bin"]
assert case.observables.domains == ["1.2.3.4"]
assert case.source_metadata["feed"] == "threatfox"
assert case.source_metadata["malware"] == "Cobalt Strike"
assert case.source_metadata["threat_type"] == "payload_delivery"
def test_threatfox_row_ip_port_to_case():
row = {
"id": "9999",
"ioc": "5.6.7.8:443",
"ioc_type": "ip:port",
"threat_type": "botnet_cc",
"malware_printable": "Qakbot",
"first_seen": "2026-05-18 10:00:00",
}
case = _threatfox_row_to_case(row)
assert case is not None
assert case.observables.ips == ["5.6.7.8"] # port stripped
def test_threatfox_row_rejects_unknown_type():
assert _threatfox_row_to_case({"id": "1", "ioc": "x", "ioc_type": "ja3_fp"}) is None
def test_malware_bazaar_row_to_case():
row = {
"sha256_hash": "a" * 64,
"sha1_hash": "b" * 40,
"md5_hash": "c" * 32,
"file_name": "invoice.exe",
"signature": "AgentTesla",
"file_type": "exe",
"first_seen": "2026-05-19 10:00:00",
"tags": ["RAT", "stealer"],
}
case = _mb_row_to_case(row)
assert case is not None
assert case.case_id == "PSYC-MBAZAAR-" + "a" * 16
assert case.observables.hashes == ["a" * 64, "b" * 40, "c" * 32]
assert case.source_metadata["feed"] == "malware-bazaar"
assert case.source_metadata["signature"] == "AgentTesla"
def test_otx_pulse_to_case_multi_indicator():
pulse = {
"id": "pulse-abc",
"name": "APT-X campaign Q2 2026",
"description": "Threat actor APT-X distributed Cobalt Strike via spear-phishing emails targeting EU energy firms. The following indicators were recovered:",
"created": "2026-05-15T12:00:00.000000",
"tlp": "white",
"tags": ["apt-x", "energy"],
"indicators": [
{"indicator": "1.2.3.4", "type": "IPv4"},
{"indicator": "evil.example", "type": "domain"},
{"indicator": "http://evil.example/payload.bin", "type": "URL"},
{"indicator": "d" * 64, "type": "FileHash-SHA256"},
{"indicator": "CVE-2026-1111", "type": "CVE"},
{"indicator": "irrelevant", "type": "Mutex"}, # ignored
],
}
case = _otx_pulse_to_case(pulse)
assert case is not None
assert case.case_id == "PSYC-OTX-pulse-abc"
assert case.observables.ips == ["1.2.3.4"]
assert "evil.example" in case.observables.domains
assert case.observables.urls == ["http://evil.example/payload.bin"]
assert case.observables.hashes == ["d" * 64]
assert case.observables.cves == ["CVE-2026-1111"]
assert "APT-X" in case.source_metadata["description"]
assert case.source_metadata["feed"] == "otx"
def test_otx_pulse_skips_when_no_recognized_indicators():
pulse = {"id": "p1", "name": "x", "description": "", "indicators": [{"indicator": "x", "type": "Mutex"}]}
assert _otx_pulse_to_case(pulse) is None

View File

@@ -0,0 +1,313 @@
"""Topology export — whitelist sanitization + endpoint contract.
The big invariant: nothing from docker_view.topology() escapes that isn't
in the Pydantic schema. We assert via model_fields introspection AND via a
JSON-dump scan over a fixture that contains every dangerous field.
"""
from __future__ import annotations
import json
from typing import Any, Dict, List
from unittest.mock import patch
import pytest
from fastapi import FastAPI
from fastapi.templating import Jinja2Templates
from fastapi.testclient import TestClient
from sqlalchemy import create_engine
from starlette.middleware.sessions import SessionMiddleware
from psyc import db
from psyc.cockpit import docker_view, federation_routes
from psyc.lines import federation, topology_export
from psyc.lines.topology_export import (
TopologyContainer,
TopologyExport,
TopologyNetwork,
_filter_image_name,
build_export,
)
# ---------- fixtures ----------------------------------------------------
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
@pytest.fixture
def fed_dir(tmp_path, monkeypatch):
d = tmp_path / "federation"
monkeypatch.setattr(federation, "FED_DIR", d)
monkeypatch.setattr(federation, "PRIVATE_KEY_PATH", d / "node.key")
monkeypatch.setattr(federation, "PUBLIC_KEY_PATH", d / "node.pub")
yield d
@pytest.fixture(autouse=True)
def reset_topology_cache():
if hasattr(federation_routes, "_TOPOLOGY_CACHE"):
federation_routes._TOPOLOGY_CACHE["payload"] = None
federation_routes._TOPOLOGY_CACHE["ts"] = 0.0
yield
# ---------- fixture data: hostile docker_view output --------------------
# This payload has every leaky field docker_view *could* surface, plus
# nested env-style data — used to prove the export is whitelist-only.
_LEAKY_TOPOLOGY: Dict[str, Any] = {
"containers": [
{
"id": "abcdef1234567890ffff",
"name": "psyc-cockpit-1",
"image": "registry.example/psyc:1.2",
"state": "running",
"status": "Up 5 minutes (healthy)",
"networks": [
{"name": "backend", "ip": "172.20.0.5", "gateway": "172.20.0.1", "mac": "02:42:ac:14:00:05"},
{"name": "frontend", "ip": "172.21.0.7", "gateway": "172.21.0.1", "mac": "02:42:ac:15:00:07"},
],
"ports": ["0.0.0.0:8767->8767/tcp"],
"published_ports": ["8767/tcp"],
# These are NOT current docker_view fields but defend in depth —
# if a future docker_view change adds them, sanitizer drops them.
"env": ["SECRET_TOKEN=abc123", "DB_PASSWORD=hunter2"],
"mounts": ["/var/run/docker.sock", "/etc/secrets:/secrets"],
"labels": {"com.docker.compose.project": "psyc", "secret_label": "shh"},
},
{
"id": "fedcba0987654321",
"name": "some-stopped",
"image": "alpine",
"state": "exited",
"status": "Exited (0) 2 hours ago",
"networks": [],
"ports": [],
"published_ports": [],
},
],
"networks": [
{
"id": "n1", "name": "backend", "driver": "bridge", "scope": "local",
"internal": False, "subnet": "172.20.0.0/16", "gateway": "172.20.0.1",
"containers": [
{"id": "abcdef123456", "name": "psyc-cockpit-1", "ip": "172.20.0.5", "mac": "02:42:ac:14:00:05"},
],
},
{
"id": "n2", "name": "internal-only", "driver": "bridge", "scope": "local",
"internal": True, "subnet": "10.99.0.0/16", "gateway": "10.99.0.1",
"containers": [],
},
],
"host": {"name": "docker-host-secret-internal.example.com", "os": "linux", "ncpu": 8},
"error": None,
"proxy": "http://docker-socket-proxy:2375",
}
# Sensitive strings that MUST NOT appear anywhere in the export JSON.
_FORBIDDEN_STRINGS = (
"SECRET_TOKEN", "DB_PASSWORD", "hunter2", "abc123",
"/var/run/docker.sock", "/etc/secrets",
"secret_label", "shh",
"172.20.0.5", "172.21.0.7", # IPs
"02:42:ac", # MAC prefix
"172.20.0.1", # gateway
"172.20.0.0/16", "10.99.0.0/16", # subnets
"0.0.0.0:8767", # port mapping
"internal.example.com", # full host
)
# ---------- model field introspection -----------------------------------
def test_container_model_has_no_dangerous_fields():
fields = set(TopologyContainer.model_fields.keys())
# whitelist — must match the design contract exactly
assert fields == {
"name", "short_id", "image", "state", "health",
"networks", "service", "started_at",
}
# explicit deny-list, double-belt
for forbidden in ("env", "environment", "mounts", "volumes",
"labels", "ip", "ip_address", "ipaddress",
"ports", "published_ports", "mac", "gateway"):
assert forbidden not in fields, f"{forbidden} must not be a field"
def test_network_model_has_no_dangerous_fields():
fields = set(TopologyNetwork.model_fields.keys())
assert fields == {"name", "driver", "internal", "container_count"}
for forbidden in ("subnet", "gateway", "labels", "ipam",
"containers", "scope", "id"):
assert forbidden not in fields, f"{forbidden} must not be a field"
def test_export_model_top_level_fields():
fields = set(TopologyExport.model_fields.keys())
assert fields == {
"node_fingerprint", "generated_at", "host_name",
"container_count", "network_count", "containers", "networks",
}
# ---------- image-name filter -------------------------------------------
def test_filter_image_strips_basic_auth_prefix():
# user:pass@host/repo:tag → host/repo:tag (creds gone)
assert _filter_image_name("user:pass@host/repo:tag") == "host/repo:tag"
def test_filter_image_drops_digest_suffix():
assert _filter_image_name(
"nginx:1.25@sha256:abcdef0123"
) == "nginx:1.25"
def test_filter_image_passes_clean_refs_untouched():
assert _filter_image_name("psyc:latest") == "psyc:latest"
assert _filter_image_name(
"ghcr.io/example/psyc:v0.3.1"
) == "ghcr.io/example/psyc:v0.3.1"
def test_filter_image_handles_empty():
assert _filter_image_name("") == ""
assert _filter_image_name(None) == "" # type: ignore[arg-type]
# ---------- build_export contract ---------------------------------------
def test_build_export_returns_empty_when_docker_view_raises(fresh_db, fed_dir, monkeypatch):
def boom():
raise docker_view.DockerProxyError("connection refused")
monkeypatch.setattr(docker_view, "topology", boom)
out = build_export()
assert isinstance(out, TopologyExport)
assert out.container_count == 0
assert out.containers == []
assert out.networks == []
# fingerprint is still real (federation key was generated)
assert len(out.node_fingerprint) == 32
def test_build_export_returns_empty_when_docker_view_reports_error(fresh_db, fed_dir, monkeypatch):
monkeypatch.setattr(docker_view, "topology", lambda: {
"containers": [], "networks": [], "host": {"name": "x"},
"error": "containers: refused", "proxy": "x",
})
out = build_export()
assert out.container_count == 0
assert out.containers == []
def test_build_export_sanitizes_every_field(fresh_db, fed_dir, monkeypatch):
monkeypatch.setattr(docker_view, "topology", lambda: _LEAKY_TOPOLOGY)
out = build_export()
# Containers came through, but as TopologyContainer (no leaky attrs).
assert out.container_count == 2
by_name = {c.name: c for c in out.containers}
cp = by_name["psyc-cockpit-1"]
assert cp.short_id == "abcdef123456"
assert cp.image == "registry.example/psyc:1.2"
assert cp.state == "running"
assert cp.health == "healthy"
assert cp.networks == ["backend", "frontend"]
assert cp.service is None
# Networks came through, sanitized.
assert out.network_count == 2
by_net = {n.name: n for n in out.networks}
assert by_net["backend"].driver == "bridge"
assert by_net["backend"].internal is False
assert by_net["backend"].container_count == 1
assert by_net["internal-only"].internal is True
def test_export_json_contains_no_dangerous_strings(fresh_db, fed_dir, monkeypatch):
"""Strict no-leak: serialize and grep for everything sensitive."""
monkeypatch.setattr(docker_view, "topology", lambda: _LEAKY_TOPOLOGY)
out = build_export()
blob = json.dumps(out.model_dump(mode="json"))
for forbidden in _FORBIDDEN_STRINGS:
assert forbidden not in blob, f"leak: {forbidden!r} appeared in export JSON"
def test_build_export_caps_at_max_containers(fresh_db, fed_dir, monkeypatch):
fake = {
"containers": [
{"id": f"id{i:04d}", "name": f"c{i}", "image": "x", "state": "running", "status": "Up", "networks": []}
for i in range(topology_export.MAX_CONTAINERS + 50)
],
"networks": [], "host": {"name": "h"}, "error": None, "proxy": "",
}
monkeypatch.setattr(docker_view, "topology", lambda: fake)
out = build_export()
assert out.container_count == topology_export.MAX_CONTAINERS
# ---------- HTTP endpoint -----------------------------------------------
def _mk_app() -> FastAPI:
app = FastAPI()
app.add_middleware(SessionMiddleware, secret_key="test-secret")
from pathlib import Path as _Path
here = _Path(__file__).resolve().parent.parent / "src" / "psyc" / "cockpit" / "templates"
templates = Jinja2Templates(directory=str(here))
federation_routes.register(app, templates)
return app
def test_federation_topology_endpoint_returns_json_with_cors(fresh_db, fed_dir, monkeypatch):
monkeypatch.setattr(docker_view, "topology", lambda: _LEAKY_TOPOLOGY)
client = TestClient(_mk_app())
r = client.get("/federation/topology")
assert r.status_code == 200
assert r.headers.get("access-control-allow-origin") == "*"
data = r.json()
# Schema check.
for key in ("node_fingerprint", "generated_at", "host_name",
"container_count", "network_count", "containers", "networks"):
assert key in data
assert data["container_count"] == 2
assert len(data["containers"]) == 2
# No leaks in the wire response either.
blob = r.text
for forbidden in _FORBIDDEN_STRINGS:
assert forbidden not in blob, f"leak via endpoint: {forbidden!r}"
def test_federation_topology_endpoint_resilient_when_docker_unavailable(fresh_db, fed_dir, monkeypatch):
def boom():
raise docker_view.DockerProxyError("proxy down")
monkeypatch.setattr(docker_view, "topology", boom)
client = TestClient(_mk_app())
r = client.get("/federation/topology")
assert r.status_code == 200
data = r.json()
assert data["container_count"] == 0
assert data["containers"] == []
def test_federation_topology_cache_short_circuits_repeated_calls(fresh_db, fed_dir, monkeypatch):
"""Within TTL, a second hit must not re-call docker_view."""
calls = {"n": 0}
def counted():
calls["n"] += 1
return _LEAKY_TOPOLOGY
monkeypatch.setattr(docker_view, "topology", counted)
client = TestClient(_mk_app())
r1 = client.get("/federation/topology")
r2 = client.get("/federation/topology")
assert r1.status_code == 200 and r2.status_code == 200
assert calls["n"] == 1, "cache should suppress the second docker_view call"

118
tests/test_translog.py Normal file
View File

@@ -0,0 +1,118 @@
"""Transparency log — append, verify, tamper detection, sync slices."""
from __future__ import annotations
import json
import pytest
from sqlalchemy import create_engine, update
from psyc import db
from psyc.lines import translog
from psyc.lines.translog import GENESIS_PREV_HASH
from psyc.result import Err, Ok
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
def test_first_append_uses_genesis_prev_hash(fresh_db):
e = translog.append("signal", {"x": 1})
assert e.prev_hash == GENESIS_PREV_HASH
assert e.id >= 1
assert e.entry_type == "signal"
assert e.entry_data == {"x": 1}
# head matches
h = translog.head()
assert h is not None
assert h.id == e.id
assert h.entry_hash == e.entry_hash
def test_append_chains_prev_hash(fresh_db):
e1 = translog.append("signal", {"a": 1})
e2 = translog.append("signal", {"b": 2})
e3 = translog.append("vouch", {"c": 3})
assert e2.prev_hash == e1.entry_hash
assert e3.prev_hash == e2.entry_hash
head = translog.head()
assert head is not None
assert head.entry_hash == e3.entry_hash
def test_verify_chain_ok_round_trip(fresh_db):
translog.append("signal", {"a": 1})
translog.append("signal", {"b": 2})
translog.append("vouch", {"c": 3})
result = translog.verify_chain()
assert isinstance(result, Ok)
assert result.value == 3
def test_verify_chain_empty_returns_zero(fresh_db):
result = translog.verify_chain()
assert isinstance(result, Ok)
assert result.value == 0
def test_verify_chain_detects_tampered_data(fresh_db):
e1 = translog.append("signal", {"a": 1})
e2 = translog.append("signal", {"b": 2})
# Mutate entry_data of the first row directly in the DB; entry_hash stays
# the same but no longer matches the recomputed hash.
with db.engine().begin() as conn:
conn.execute(
update(db.translog)
.where(db.translog.c.id == e1.id)
.values(entry_data=json.dumps({"a": 999}, sort_keys=True))
)
result = translog.verify_chain()
assert isinstance(result, Err)
assert "broken at id=" in result.reason
def test_verify_chain_detects_tampered_prev_hash(fresh_db):
translog.append("signal", {"a": 1})
e2 = translog.append("signal", {"b": 2})
# Flip e2.prev_hash so it no longer matches e1.entry_hash.
with db.engine().begin() as conn:
conn.execute(
update(db.translog)
.where(db.translog.c.id == e2.id)
.values(prev_hash="f" * 64)
)
result = translog.verify_chain()
assert isinstance(result, Err)
assert "broken at id=" in result.reason
def test_entries_after_returns_correct_slice(fresh_db):
e1 = translog.append("signal", {"a": 1})
e2 = translog.append("signal", {"b": 2})
e3 = translog.append("signal", {"c": 3})
after_zero = translog.entries_after(0)
assert [e.id for e in after_zero] == [e1.id, e2.id, e3.id]
after_e1 = translog.entries_after(e1.id)
assert [e.id for e in after_e1] == [e2.id, e3.id]
after_e3 = translog.entries_after(e3.id)
assert after_e3 == []
def test_recent_newest_first(fresh_db):
e1 = translog.append("signal", {"a": 1})
e2 = translog.append("signal", {"b": 2})
e3 = translog.append("signal", {"c": 3})
recent = translog.recent(limit=10)
assert [e.id for e in recent] == [e3.id, e2.id, e1.id]

336
tests/test_vouching.py Normal file
View File

@@ -0,0 +1,336 @@
"""Vouching + quorum — sign/verify, threshold logic, import gate."""
from __future__ import annotations
import base64
import hashlib
from datetime import datetime, timedelta, timezone
import pytest
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
from sqlalchemy import create_engine
from psyc import db
from psyc.lines import federation
from psyc.lines.federation import (
QuorumConfig,
Vouch,
accept_vouch,
build_signed_feed,
canonical_json,
import_signed_feed,
is_quorum_met,
is_vouched,
issue_vouch,
node_fingerprint,
our_vouches,
peer_is_listening_eligible,
public_key_pem,
quorum_config,
register_peer,
revoke_vouch,
set_quorum_config,
vouch_payload_bytes,
)
from psyc.result import Err, Ok
@pytest.fixture
def fresh_db(tmp_path, monkeypatch):
test_db = tmp_path / "test.db"
eng = create_engine(f"sqlite:///{test_db}", future=True)
db._metadata.create_all(eng, checkfirst=True)
monkeypatch.setattr(db, "_engine", eng)
monkeypatch.setattr(db, "DB_PATH", test_db)
yield test_db
@pytest.fixture
def fed_dir(tmp_path, monkeypatch):
d = tmp_path / "federation"
monkeypatch.setattr(federation, "FED_DIR", d)
monkeypatch.setattr(federation, "PRIVATE_KEY_PATH", d / "node.key")
monkeypatch.setattr(federation, "PUBLIC_KEY_PATH", d / "node.pub")
yield d
def _make_peer():
"""Generate an Ed25519 keypair + matching fingerprint for a fake peer."""
priv = ed25519.Ed25519PrivateKey.generate()
pub = priv.public_key()
pub_pem = pub.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
).decode("ascii")
raw = pub.public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
fp = hashlib.sha256(raw).digest()[:16].hex()
return priv, pub_pem, fp
def _sign_vouch(priv, voucher_fp, target_fp, issued_at, expires_at):
payload = vouch_payload_bytes(voucher_fp, target_fp, issued_at, expires_at)
sig = priv.sign(payload)
return base64.b64encode(sig).decode("ascii")
# ---------- self-issued vouch round-trip --------------------------------
def test_issue_vouch_roundtrip(fresh_db, fed_dir):
target = "ab" * 16
v = issue_vouch(target, ttl_days=30)
assert v.voucher_fingerprint == node_fingerprint()
assert v.target_fingerprint == target
assert v.expires_at is not None
# round-trip from storage
listed = our_vouches()
assert len(listed) == 1
assert listed[0].target_fingerprint == target
assert listed[0].signature == v.signature
# signature verifies under our own pubkey
payload = vouch_payload_bytes(
v.voucher_fingerprint, v.target_fingerprint, v.issued_at, v.expires_at
)
sig = base64.b64decode(v.signature)
assert federation.verify_payload(payload, sig, public_key_pem())
def test_revoke_vouch_removes_only_our_entry(fresh_db, fed_dir):
target = "cd" * 16
issue_vouch(target, ttl_days=30)
assert len(our_vouches()) == 1
revoke_vouch(target)
assert our_vouches() == []
# ---------- accept_vouch validation -------------------------------------
def test_accept_vouch_rejects_expired(fresh_db, fed_dir):
priv, pem, fp = _make_peer()
register_peer("voucher.example", fp, pem, status="trusted")
issued = datetime.now(timezone.utc) - timedelta(days=10)
expired = datetime.now(timezone.utc) - timedelta(days=1)
sig = _sign_vouch(priv, fp, "target", issued, expired)
v = Vouch(voucher_fingerprint=fp, target_fingerprint="target",
issued_at=issued, expires_at=expired, signature=sig)
result = accept_vouch(v, pem)
assert isinstance(result, Err)
assert "expired" in result.reason
def test_accept_vouch_rejects_bad_signature(fresh_db, fed_dir):
priv, pem, fp = _make_peer()
register_peer("voucher.example", fp, pem, status="trusted")
issued = datetime.now(timezone.utc)
expires = issued + timedelta(days=30)
# Sign a different target then claim it's for "real-target".
real_sig = _sign_vouch(priv, fp, "other-target", issued, expires)
v = Vouch(voucher_fingerprint=fp, target_fingerprint="real-target",
issued_at=issued, expires_at=expires, signature=real_sig)
result = accept_vouch(v, pem)
assert isinstance(result, Err)
assert "signature" in result.reason
def test_accept_vouch_rejects_voucher_not_trusted(fresh_db, fed_dir):
priv, pem, fp = _make_peer()
# Voucher exists but is "unknown" not "trusted".
register_peer("voucher.example", fp, pem, status="unknown")
issued = datetime.now(timezone.utc)
expires = issued + timedelta(days=30)
sig = _sign_vouch(priv, fp, "target", issued, expires)
v = Vouch(voucher_fingerprint=fp, target_fingerprint="target",
issued_at=issued, expires_at=expires, signature=sig)
result = accept_vouch(v, pem)
assert isinstance(result, Err)
assert "not trusted" in result.reason
def test_accept_vouch_ok_for_trusted_voucher(fresh_db, fed_dir):
priv, pem, fp = _make_peer()
register_peer("voucher.example", fp, pem, status="trusted")
issued = datetime.now(timezone.utc)
expires = issued + timedelta(days=30)
sig = _sign_vouch(priv, fp, "target", issued, expires)
v = Vouch(voucher_fingerprint=fp, target_fingerprint="target",
issued_at=issued, expires_at=expires, signature=sig)
result = accept_vouch(v, pem)
assert isinstance(result, Ok)
# ---------- is_vouched threshold ----------------------------------------
def test_is_vouched_needs_distinct_vouchers(fresh_db, fed_dir):
"""Two vouches from the same peer must NOT clear a threshold of 2."""
priv, pem, fp = _make_peer()
register_peer("voucher.example", fp, pem, status="trusted")
issued = datetime.now(timezone.utc)
expires = issued + timedelta(days=30)
sig = _sign_vouch(priv, fp, "target", issued, expires)
v1 = Vouch(voucher_fingerprint=fp, target_fingerprint="target",
issued_at=issued, expires_at=expires, signature=sig)
assert isinstance(accept_vouch(v1, pem), Ok)
# Newer vouch from the SAME voucher — upsert replaces, count stays 1.
issued2 = issued + timedelta(seconds=1)
sig2 = _sign_vouch(priv, fp, "target", issued2, expires)
v2 = Vouch(voucher_fingerprint=fp, target_fingerprint="target",
issued_at=issued2, expires_at=expires, signature=sig2)
assert isinstance(accept_vouch(v2, pem), Ok)
assert is_vouched("target", min_vouchers=2) is False
# Threshold of 1 should pass.
assert is_vouched("target", min_vouchers=1) is True
def test_is_vouched_two_distinct_clear_threshold(fresh_db, fed_dir):
priv_a, pem_a, fp_a = _make_peer()
priv_b, pem_b, fp_b = _make_peer()
register_peer("a.example", fp_a, pem_a, status="trusted")
register_peer("b.example", fp_b, pem_b, status="trusted")
issued = datetime.now(timezone.utc)
expires = issued + timedelta(days=30)
va = Vouch(voucher_fingerprint=fp_a, target_fingerprint="target",
issued_at=issued, expires_at=expires,
signature=_sign_vouch(priv_a, fp_a, "target", issued, expires))
vb = Vouch(voucher_fingerprint=fp_b, target_fingerprint="target",
issued_at=issued, expires_at=expires,
signature=_sign_vouch(priv_b, fp_b, "target", issued, expires))
assert isinstance(accept_vouch(va, pem_a), Ok)
assert isinstance(accept_vouch(vb, pem_b), Ok)
assert is_vouched("target", min_vouchers=2) is True
assert is_vouched("target", min_vouchers=3) is False
# ---------- quorum on signal_hash ---------------------------------------
def test_is_quorum_met_counts_distinct_vouched_peers_only(fresh_db, fed_dir):
# Two trusted peers + one untrusted peer report the same signal_hash.
_, pem_a, fp_a = _make_peer()
_, pem_b, fp_b = _make_peer()
_, pem_c, fp_c = _make_peer()
register_peer("a.example", fp_a, pem_a, status="trusted")
register_peer("b.example", fp_b, pem_b, status="trusted")
register_peer("c.example", fp_c, pem_c, status="unknown") # not eligible
for fp in (fp_a, fp_b, fp_c, fp_a): # fp_a duplicated → still 1 distinct
db.record_signal(dict(
peer_fingerprint=fp,
signal_type="ioc",
signal_id="1.2.3.4",
signal_hash="h-aaa",
received_at=datetime.now(timezone.utc).isoformat(),
raw_json="{}",
))
assert is_quorum_met("h-aaa", k=2) is True
assert is_quorum_met("h-aaa", k=3) is False # only 2 eligible distincts
# ---------- quorum config persistence -----------------------------------
def test_quorum_config_defaults_and_persistence(fresh_db, fed_dir):
cfg = quorum_config()
assert cfg.trust_min_vouchers == 2
assert cfg.signal_quorum_k == 2
set_quorum_config(QuorumConfig(trust_min_vouchers=3, signal_quorum_k=4))
cfg2 = quorum_config()
assert cfg2.trust_min_vouchers == 3
assert cfg2.signal_quorum_k == 4
# ---------- import gate enforces listening eligibility ------------------
def _signed_feed_from_peer(peer_priv, peer_fp, vouches=None):
"""Build a feed claiming origin=peer_fp, signed with peer_priv."""
payload = {
"version": federation.FEED_VERSION,
"fingerprint": peer_fp,
"generated_at": datetime.now(timezone.utc).isoformat(),
"window_hours": 24,
"cases": [],
"iocs": [{
"value": "9.9.9.9",
"type": "ip",
"severity": "high",
"first_seen": datetime.now(timezone.utc).isoformat(),
"digest_sha256": "abc123",
}],
"vouches": vouches or [],
}
sig = peer_priv.sign(canonical_json(payload))
payload["signature"] = base64.b64encode(sig).decode("ascii")
return payload
def test_import_feed_rejects_unknown_peer(fresh_db, fed_dir):
peer_priv, peer_pem, peer_fp = _make_peer()
feed = _signed_feed_from_peer(peer_priv, peer_fp)
result = import_signed_feed(feed, peer_pem)
assert isinstance(result, Err)
assert "not trusted" in result.reason
def test_import_feed_accepts_directly_trusted_peer(fresh_db, fed_dir):
peer_priv, peer_pem, peer_fp = _make_peer()
register_peer("peer.example", peer_fp, peer_pem, status="trusted")
feed = _signed_feed_from_peer(peer_priv, peer_fp)
result = import_signed_feed(feed, peer_pem)
assert isinstance(result, Ok), getattr(result, "reason", "")
def test_import_feed_accepts_vouched_peer(fresh_db, fed_dir):
# Two trusted peers vouch for a third — third becomes listening-eligible.
priv_a, pem_a, fp_a = _make_peer()
priv_b, pem_b, fp_b = _make_peer()
priv_c, pem_c, fp_c = _make_peer()
register_peer("a.example", fp_a, pem_a, status="trusted")
register_peer("b.example", fp_b, pem_b, status="trusted")
issued = datetime.now(timezone.utc)
expires = issued + timedelta(days=30)
va = Vouch(voucher_fingerprint=fp_a, target_fingerprint=fp_c,
issued_at=issued, expires_at=expires,
signature=_sign_vouch(priv_a, fp_a, fp_c, issued, expires))
vb = Vouch(voucher_fingerprint=fp_b, target_fingerprint=fp_c,
issued_at=issued, expires_at=expires,
signature=_sign_vouch(priv_b, fp_b, fp_c, issued, expires))
assert isinstance(accept_vouch(va, pem_a), Ok)
assert isinstance(accept_vouch(vb, pem_b), Ok)
assert peer_is_listening_eligible(fp_c) is True
feed = _signed_feed_from_peer(priv_c, fp_c)
result = import_signed_feed(feed, pem_c)
assert isinstance(result, Ok), getattr(result, "reason", "")
def test_import_feed_propagates_vouches_in_payload(fresh_db, fed_dir):
"""A trusted peer's feed carries a vouch the peer issued — we should
accept_vouch it and store it locally."""
peer_priv, peer_pem, peer_fp = _make_peer()
register_peer("peer.example", peer_fp, peer_pem, status="trusted")
target_fp = "ff" * 16
issued = datetime.now(timezone.utc)
expires = issued + timedelta(days=30)
peer_vouch = Vouch(
voucher_fingerprint=peer_fp,
target_fingerprint=target_fp,
issued_at=issued,
expires_at=expires,
signature=_sign_vouch(peer_priv, peer_fp, target_fp, issued, expires),
)
feed = _signed_feed_from_peer(peer_priv, peer_fp, vouches=[peer_vouch.model_dump(mode="json")])
result = import_signed_feed(feed, peer_pem)
assert isinstance(result, Ok), getattr(result, "reason", "")
# The vouch is now in our local store under the peer's fingerprint.
stored = federation.vouches_for(target_fp)
assert any(v.voucher_fingerprint == peer_fp for v in stored)