neuronetz-gateway/docs/ARCHITECTURE.md

neuronetz-gateway — Architecture

Distilled from scope-docs/SPEC.md §4. The SPEC is the source of truth.

The gateway is the hot path of the Neuronetz API: a secure, multi-tenant proxy in front of an Ollama instance. The Ollama backend must never be reachable directly from the public internet — all access flows through this gateway. Administration (dashboards, tenant self-service) lives in a separate service, neuronetz-console, and is out of scope here.

---

Component diagram (SPEC §4.1)

                          Internet
                              │ TLS
                              ▼
                  ┌──────────────────────┐
                  │ Caddy (sidecar)      │  Let's Encrypt for api.neuronetz.ai
                  │ - TLS termination    │  HSTS, security headers
                  │ - HTTP/2, HTTP/3     │
                  └──────────┬───────────┘
                             │ HTTP/1.1 internal
                  ┌──────────▼───────────┐
                  │ neuronetz-gateway    │  FastAPI + uvicorn
                  │  - authn             │
                  │  - rate limit        │
                  │  - budget check      │
                  │  - proxy + stream    │
                  │  - token count       │
                  │  - audit write       │
                  └──┬────────┬──────┬───┘
                     │        │      │
              ┌──────▼──┐  ┌──▼───┐  │
              │Postgres │  │Redis │  │
              │ schema: │  │ keys │  │
              │ gateway │  │bucket│  │
              └─────────┘  └──────┘  │
                                     │ internal network only
                              ┌──────▼──────┐
                              │   Ollama    │
                              │ 127.0.0.1   │
                              └─────────────┘

Same Compose stack also hosts (separate from this SPEC):
  - neuronetz-console (PHP/Nibiru) → reads schema `console`, reads schema `gateway` (SELECT)

Only Caddy publishes ports. Postgres, Redis and (critically) Ollama have no published ports and are reachable only on the internal Docker network.

---

Database schemas (SPEC §4.2)

A single Postgres instance with two schemas:

• gateway — owned by this service; full DDL. Tables: tenants, tenant_limits, api_keys, key_limits, budget_usage, audit_log, prompt_log, revocations (see SPEC §5 for the full DDL).
• console — owned by neuronetz-console (out of scope). The console role gets SELECT on all gateway.* tables and INSERT on gateway.revocations only.

If the console needs to mutate gateway state (e.g. revoke a key), it does so by inserting into the gateway.revocations outbox table, which the gateway tails (see Revocation below).

Limit inheritance: limits and budgets resolve key → tenant. A NULL key-level value inherits the tenant value. For allow_all_models, a non-NULL key value overrides the tenant flag; otherwise the tenant flag applies (SPEC §13.7).

---

Request lifecycle (SPEC §4.3)

1. Caddy terminates TLS and forwards to the gateway on the internal port.
2. Middleware extracts Authorization: Bearer <key>.
3. The 12-char prefix is the Redis cache key. On miss, look up gateway.api_keys by prefix, verify the full key with argon2id, and cache resolved metadata in Redis (TTL 60 s).
4. Rate limit check — sliding window in Redis (Lua-atomic): per-key RPM + per-tenant RPM.
5. Budget check — Redis counter for the current period; Postgres ledger is the source of truth on reset.
6. Concurrency semaphore — Redis INCR with TTL.
7. Model allowlist check — resolve the effective set (see below); the request model must be in it, else a generic 403.
8. Endpoint allowlist check — mutating endpoints are hard-blocked.
9. Body validation — size, schema, num_predict cap.
10. If an OpenAI-compat path, translate the request to the Ollama schema.
11. Open an httpx async stream to Ollama.
12. Stream the response back to the client, accumulating the final prompt_eval_count + eval_count.
13. On stream close: write the gateway.audit_log row; decrement the budget; release the semaphore; if prompt logging is enabled, write gateway.prompt_log.
14. On any failure: sanitized error to the client, audit row with the status code, semaphore released.

Streaming integrity: token counting and the audit write happen after stream close, never on the hot path — time-to-first-byte is not degraded by bookkeeping (SPEC §9).

---

Model discovery (SPEC §4.6)

The set of usable models is never hand-maintained; it is extracted live from Ollama.

• A background task (started in the app lifespan, alongside the revocation listener) polls Ollama GET /api/tags every MODEL_DISCOVERY_REFRESH_S seconds.
• The parsed set (names + sanitized metadata: family, parameter size, quantization, size, modified-at) is cached in Redis under gateway:models:discovered with TTL MODEL_DISCOVERY_CACHE_TTL_S, and held in-process for hot reads on the request path.
• An initial fetch runs at startup; if Ollama is unreachable the discovered set is empty.
• Fail-closed: an empty or expired-and-unrefreshable discovered set means no model resolves and requests are denied. Discovery never opens access on failure.
• Auto-grant: because the effective set intersects with discovered (or is discovered when allow_all_models), a model pulled into Ollama out-of-band becomes usable to allow_all tenants on the next refresh — no per-tenant config change.
• Discovery is read-only against Ollama and uses only the allowlisted /api/tags endpoint; it never triggers a model pull.

Effective-set resolution (SPEC §4.3 step 7)

allow_all := key.allow_all_models ?? tenant.allow_all_models
effective := discovered                                          if allow_all
             (key.allowed_models ?? tenant.allowed_models) ∩ discovered   otherwise

/api/tags and /v1/models return exactly this effective set, so the listing never reveals models outside the tenant's reach. A model that is installed-but-unpermitted and one that is not installed both return the same generic 403 — no existence disclosure (SPEC §13.6).

---

Failure modes — fail-closed (SPEC §4.4)

Subsystem	If down	Behavior
Postgres (read)	Key lookup fails	503 with retry-after; nothing proxied.
Postgres (write)	Audit write fails	Request still succeeds; audit row buffered in-memory ring (max 1000), drained on recovery; if the buffer fills, switch to deny mode.
Redis	Rate limit / budget unavailable	503 — fail closed. Never "allow because we can't check."
Ollama	Upstream unreachable	502 with retry-after; circuit breaker opens after 5 consecutive failures, half-open after 30 s.
Caddy	Not a gateway concern	—

The governing rule (AGENT_PROMPT non-negotiable #1): if a security or budgeting check cannot be performed, deny. Never default to allow.

---

Cache invalidation / key revocation (SPEC §4.5)

The console revokes a key by inserting into gateway.revocations(key_id, ts, reason). A background task in the gateway lifespan:

• LISTENs on the Postgres channel key_revoked (the gateway emits NOTIFY on its own write path; the console's INSERT fires a trigger that emits it).
• On notification, evicts the Redis cache entry for that key's prefix.

This makes revocation effectively immediate (≤ Redis RTT) with no cross-service HTTP.

---

Observability

• Structured logs (structlog), JSON in production. Secrets/keys are never logged.
• Prometheus /metrics (loopback only): gateway_requests_total{tenant,model,status}, gateway_tokens_total{tenant,model,direction}, gateway_request_duration_seconds{tenant,model} (histogram). Labelled by tenant, never by key_id (cardinality — SPEC §13.3); per-key data lives in Postgres.
• Audit log — always-on request metadata. Prompt log — opt-in per key, TTL'd.