Designing an Idempotent Job Queue

Every durable queue we've used promises at-least-once delivery. That's a polite way of saying "sometimes twice." If your job isn't idempotent, retries corrupt data.

The idempotency key

Give each job a stable key and record completion atomically with the work:

async function runOnce(jobKey: string, work: () => Promise<void>) {
  const inserted = await db.query(
    "INSERT INTO job_runs (key) VALUES ($1) ON CONFLICT DO NOTHING",
    [jobKey],
  );
  if (inserted.rowCount === 0) return; // already ran
  await work();
}

The database row that proves a job ran is more valuable than the job's result. Write it in the same transaction as the side effect, or you've built a race.