Architecture

QuantumLane is a single-box data platform that demonstrates production patterns at the smallest scale that still requires real engineering. The full architecture document with ADRs lives in the repo; what follows is the summary.

Principles

Boring tech that runs forever beats novel tech that runs for a month.
Observability is a first-class feature, not an afterthought.
Document the trade-off, not the tool.
Schema is contract. Forward-only migrations, never ALTER TABLE in psql.
Local dev = production in a smaller box.
Model by access pattern, not by volume. The right store and grain follow from how data is read, not how much there is.
Cost discipline is part of the design.

Stack

Layer	Choice	Why
Orchestration	Dagster	Asset-centric model; lighter than Airflow at single-box scale.
Database	PostgreSQL 16 + PostGIS	One store at this scale; best-in-class geospatial.
Object storage	Amazon S3	Cold-tier Parquet archive with Hive-style partitioned keys.
API	FastAPI + Pydantic v2	Async, auto OpenAPI, modern validation.
Website	Plain HTML + Tailwind CDN	A few pages, monthly updates — no build system needed.
Reverse proxy	Caddy	Automatic TLS, dead-simple config.
Host	Hetzner CPX21	Far cheaper than AWS for the same patterns.

Hot / cold split

Real-time data lands in a hot tier (PostgreSQL/PostGIS) that serves live API reads and near-real-time queries. A daily job archives it to a cold tier — Parquet on S3, partitioned by UTC day with Hive-style keys — for historical analytics. The two paths are deliberately separate: live operational queries and historical aggregation have different access patterns, so they get different stores rather than one general-purpose table doing both jobs.

Both ingestion and archival are written to stay within memory on a small box. Large static GTFS files (e.g. stop_times, ~4M rows) stream row-by-row into COPY from the open archive; the Parquet export reads through a server-side cursor into a held-open writer rather than materializing the result set.

Delay & reliability

Three features share the word "delay" but are modelled separately, each by its access pattern:

Headway regularity — real-time only, no schedule dependency. (The TTC RT delay fields arrive all-NULL, so headway is the viable RT-only signal.)
Live schedule-adherence delay — hot-tier, computed on read against the static schedule.
Historical reliability — daily OLAP aggregation of event-time-computed delays, not a recompute from raw real-time plus schedule. This avoids keeping a history of schedule snapshots and sidesteps a daily-refresh race.

The mechanism underneath is a stop-level overwrite: one row per (trip_id, stop_sequence), upserted as predictions arrive and finalized on arrival. A large volume of superseded predictions collapses to one durable row per stop-event, which then feeds both the live gauge and the historical aggregate from a single source.

MCP server

QuantumLane runs a public Model Context Protocol server, so LLM clients — Claude and ChatGPT — can answer live transit questions in plain English: vehicles on a route (with rider-language → route_id resolution), nearest stops via PostGIS KNN, and route lookup. It deliberately wraps the deployed public API rather than the database — no separate data path, no DB credentials in the MCP layer, one source of truth.

Guardrails are right-sized for a small shared box: a per-IP rate limit and a route-catalog cache whose TTL matches the catalog's actual change cadence (the daily static-GTFS reload) — not auth or quotas, which the access pattern doesn't justify. Today's tools are live and spatial; the analytical tools ("how reliable is the 504 usually") arrive with the OLAP layer below. How to connect →

What's in v0.4

TTC GTFS-RT ingestion: vehicle positions and trip updates every minute, service alerts every 5 minutes.
TTC static GTFS daily full-replace (04:00 Toronto), with the large stop_times file streamed via COPY.
PostgreSQL 16 + PostGIS hot tier; daily-partitioned real-time tables with a 3-day retention window and daily partition maintenance.
Daily Parquet export to S3 cold storage, partitioned by UTC day with Hive-style keys.
FastAPI public read-only API with rate limiting, including a nearest-stops endpoint (PostGIS KNN, distances in metres).
Public MCP server over the API — three tools, remote streamable HTTP, per-IP rate limiting, route-catalog caching — with a connect guide and demo video.
Per-feed freshness telemetry, refreshed every minute and surfaced on this site.
Schema-drift detection via field-population signatures.
Dagster run monitoring for stuck/zombie runs.

Planned

Historical reliability — the daily OLAP aggregation described above, once the stop-level finalize path is built out. This is also what unlocks the analytical MCP tools — the questions a live-feed wrapper structurally can't answer.
Iceberg cold tier — PySpark writing the expiring hot partitions to Iceberg on S3, restoring the archive-then-drop ordering the retention job was designed for.
Other agencies — GO Transit, MiWay, and other GTA feeds, with the schema-heterogeneity story.

Deliberately not here

Streaming framework (Kafka, etc.) — overkill at this message rate.
ML / forecasting — needs more historical data first.
Authentication — public data, public API.
Pretty map UIs — not the point.

Cost

Cost discipline is treated as a design constraint, not an afterthought: a single small Hetzner box, an S3 cold tier sized to what the analytics actually need, and free-tier DNS and CI. The architecture is chosen so the running cost stays low and predictable — the cold-tier read path in particular is designed around egress, which is why historical bulk access is handled differently from the always-free hot-tier live queries.