A local reliability engineering demo with a SQLite-backed queue, two competing workers, and a live dashboard. The consumer is simulated. The database transactions, retries, and lease recovery are real.

Requires Python 3.10 or newer. From this directory:

python3 app.py

Open http://127.0.0.1:4176 and select a scenario. Use --db PATH to choose a database and --port PORT to choose a local port.

Failure scenarios

Transient failure retries after two and four seconds before succeeding.
Worker interruption abandons the first lease. Another claim recovers the job once that lease expires.
Permanent failure reaches the dead-letter state after three attempts.
Resubmit reuses an idempotency key and returns the same job ID. Reusing that key for a different scenario returns HTTP 409.

Delivery guarantees

Job creation and its first event commit together. Workers serialize claims with a SQLite write transaction and use unique lease tokens to reject stale results. Retry timing, terminal state, and event history persist across restarts.

A worker can lose its lease after the consumer has processed an event and before the result is saved. A real consumer would need to deduplicate by job ID. This lab binds to loopback and is intended for a trusted local environment.