How a Worker Takes a Job

How a Worker Takes a Job A workflow diagram generated by Archify. 01 / apps/mail 02 / Redis queue 03 / Worker task · ECS ×N 04 / Child processes · C per task, pooled 05 / Postgres · blob storage Hand off Queue Claim + run Finish Store + commit · raw MIME, then rows · apps/mail › Hand off Store + commit raw MIME, then rows Add job · jobId = email id · apps/mail › Hand off Add job jobId = email id Waiting list · oldest first · Redis queue › Queue Waiting list oldest first Delayed / stalled · 3 tries, then failed · Redis queue › Claim + run Delayed / stalled 3 tries, then failed Sweeper · repeatable job · Worker task · ECS ×N › Queue Sweeper repeatable job Claim + lock · lock renewed while running · Worker task · ECS ×N › Claim + run Claim + lock lock renewed while running Complete · unlock, take next · Worker task · ECS ×N › Finish Complete unlock, take next Load + read · row, then raw MIME · Child processes · C per task, pooled › Claim + run Load + read row, then raw MIME Process + save · parse, classify, extract · Child processes · C per task, pooled › Claim + run Process + save parse, classify, extract email_message · rawRef · status · Postgres · blob storage › Queue email_message rawRef · status Blob storage · raw/<sha>.eml · Postgres · blob storage › Claim + run Blob storage raw/<sha>.eml Pipeline tables · run · step · artifact · Postgres · blob storage › Claim + run Pipeline tables run · step · artifact email id hand to child raw MIME row by id done save artifacts crash or lock expiry retry after backoff no run yet then write MIME commit rows re-add job wakes one Legend mail or worker step job state in Redis Postgres or blob