The gateway can now aggregate models across SEVERAL Ollama backends and
route each request to the correct one. Opt-in via OLLAMA_BACKENDS in .env
— single-backend deployments are unaffected (effective_backends()
synthesizes a single "default" backend from the legacy OLLAMA_BASE_URL /
OLLAMA_AUTH_TOKEN fields when the list is empty).
Behavior:
- Discovery polls EVERY configured backend in parallel each tick; the
cache stores per-backend model lists plus a model → backends priority
list (config order = priority order).
- /api/tags and /v1/models surface the DEDUPLICATED UNION of all
backends' models.
- A request's model is looked up in the priority list and proxied to the
FIRST backend that hosts it. If that backend errors on the request, the
pipeline transparently fails over to the next backend that has the
same model (the streaming-failover probes the first chunk before
releasing the response, so we never serve partial bytes from a dead
backend).
- No existence disclosure: a model not hosted by any backend yields the
same generic 403 as "model not allowed" (SPEC §13.6 preserved).
Components:
- config.py: new BackendSpec model + ollama_backends list field + an
effective_backends() helper.
- proxy/router.py (new): BackendRouter (clients_for_with_failover),
build_http_clients() builds one httpx client per backend with its own
auth headers, build_backend_headers() exposes the per-backend header
composition for the CLI probe.
- proxy/discovery.py: DiscoveryCache.set_per_backend() + backends_for(),
refresh_all_backends() polls all in parallel, discovery_loop_multi()
replaces the single-backend loop in production; the legacy single-
backend functions are kept for the dependency-override tests.
- proxy/pipeline.py: Pipeline accepts an optional router; the four proxy
methods now retry against each candidate backend in priority order on
transport error.
- lifespan.py: constructs the per-backend client dict, stores the router
on app.state, launches discovery_loop_multi.
- deps.py: get_backend_router provider + BackendRouterDep type alias;
get_pipeline passes the router into Pipeline.
- cli/manage.py: probe-ollama iterates every backend and reports per-
backend status; list-models groups its output by backend and prints
the union count + Redis cache size for sanity.
- .env.example + docker-compose.yml: document and pass through
OLLAMA_BACKENDS with a real example.
Verified: ruff check (clean), mypy --strict src/ + tests/ (clean,
66 source files), pytest (60 passed + 39 skipped — same baseline as
before this change; integration tests are Docker-socket-gated).
Initial project structure for neuronetz-gateway per scope-docs/SPEC.md:
- Python 3.12 / FastAPI / SQLAlchemy 2.0 (async) / Redis / Postgres stack
managed by uv. Multi-stage non-root Dockerfile, prod + dev compose files
(ollama service is NEVER published in either), Caddyfile + systemd unit,
justfile, GitHub Actions CI (ruff, mypy --strict, pytest, bandit, pip-audit).
- Pydantic-Settings config covering every env var from SPEC §7, including the
MODEL_DISCOVERY_* keys for the dynamic-discovery feature (§4.6).
- Alembic 0001_initial creates the full gateway schema (8 tables, 3 enums,
notify_key_revoked() trigger), incl. allow_all_models on tenant_limits and
key_limits for the per-tenant auto-grant toggle.
- Working /healthz, /readyz (fail-closed when deps unreachable), and a
Prometheus /metrics stub. Sanitizing error handlers that attach X-Request-ID
to every response and never leak upstream internals.
- SPEC + AGENT_PROMPT included under scope-docs/ (source of truth).
