# Relay Lab

A local reliability engineering demo with a SQLite-backed queue, two competing
workers, and a live dashboard. The consumer is simulated. The database
transactions, retries, and lease recovery are real.

Requires Python 3.10 or newer. From this directory:

```sh
python3 app.py
```

Open http://127.0.0.1:4176 and select a scenario. Use `--db PATH` to choose a
database and `--port PORT` to choose a local port.

## Failure scenarios

- **Transient failure** retries after two and four seconds before succeeding.
- **Worker interruption** abandons the first lease. Another claim recovers the
  job once that lease expires.
- **Permanent failure** reaches the dead-letter state after three attempts.
- **Resubmit** reuses an idempotency key and returns the same job ID. Reusing
  that key for a different scenario returns HTTP 409.

## Delivery guarantees

Job creation and its first event commit together. Workers serialize claims with
a SQLite write transaction and use unique lease tokens to reject stale results.
Retry timing, terminal state, and event history persist across restarts.

A worker can lose its lease after the consumer has processed an event and
before the result is saved. A real consumer would need to deduplicate by job ID.
This lab binds to loopback and is intended for a trusted local environment.