1.3 KiB
Relay Lab
A local reliability engineering demo with a SQLite-backed queue, two competing workers, and a live dashboard. The consumer is simulated. The database transactions, retries, and lease recovery are real.
Requires Python 3.10 or newer. From this directory:
python3 app.py
Open http://127.0.0.1:4176 and select a scenario. Use --db PATH to choose a
database and --port PORT to choose a local port.
Failure scenarios
- Transient failure retries after two and four seconds before succeeding.
- Worker interruption abandons the first lease. Another claim recovers the job once that lease expires.
- Permanent failure reaches the dead-letter state after three attempts.
- Resubmit reuses an idempotency key and returns the same job ID. Reusing that key for a different scenario returns HTTP 409.
Delivery guarantees
Job creation and its first event commit together. Workers serialize claims with a SQLite write transaction and use unique lease tokens to reject stale results. Retry timing, terminal state, and event history persist across restarts.
A worker can lose its lease after the consumer has processed an event and before the result is saved. A real consumer would need to deduplicate by job ID. This lab binds to loopback and is intended for a trusted local environment.