Durable jobs

Trestle persists asynchronous work before execution and uses atomic SQLite claims, finite leases and bounded retry state.

Lifecycle

pending → running → succeeded
                  ↘ pending (backoff)
                  ↘ dead
pending/running → cancelled

Workers recover expired running leases at startup and during ordinary polling. Attempts use exponential backoff capped to prevent unbounded shifts. An idempotency key prevents duplicate enqueue records for the same producer operation.

Administration

GET  /admin/v1/jobs?status=dead
POST /admin/v1/jobs/{id} {"action":"retry"}
POST /admin/v1/jobs/{id} {"action":"cancel"}

The dashboard exposes payload, attempts, terminal error and retry/cancel controls. Payloads should contain stable identifiers rather than copied secrets.

Provider storage

Durable job claiming is deliberate per provider: SQLite serializes through its single owned connection, while PostgreSQL uses row locking with SKIP LOCKED so concurrent workers each claim a distinct job. Leases, retries, cancellation, dead letters and terminal states are provider-neutral, and a rolled-back transaction never publishes its job.