Mail Sync to Processing: The Seam

Mail Sync to Processing: The Seam A data-flow diagram generated by Archify. 01 / Microsoft 365 02 / apps/mail 03 / Handoff 04 / apps/worker: listener 05 / apps/worker: one job Microsoft Graph · Inbox delta · $value · 01 / Microsoft 365 Microsoft Graph Inbox delta · $value Mail sync + ingest · sync → ingest job · 02 / apps/mail Mail sync + ingest sync → ingest job Mail tables · email_* rows · 03 / Handoff Mail tables email_* rows Processing queue · Redis · BullMQ · 03 / Handoff Processing queue Redis · BullMQ Blob storage · local disk or S3 · 03 / Handoff Blob storage local disk or S3 Sweeper · repeatable job · 04 / apps/worker: listener Sweeper repeatable job Listener · BullMQ Worker · 04 / apps/worker: listener Listener BullMQ Worker ECS autoscaling · backlog per task · 04 / apps/worker: listener ECS autoscaling backlog per task Pipeline tables · run · step · artifact · 05 / apps/worker: one job Pipeline tables run · step · artifact Job handler · parse, classify, extract · 05 / apps/worker: one job Job handler parse, classify, extract FlowOS API · enquiry entered by hand · 01 / Microsoft 365 FlowOS API enquiry entered by hand History queue · mail-history · BullMQ · 01 / Microsoft 365 History queue mail-history · BullMQ delta + MIME read-only ① commit rows EXTRACTION_PENDING ② add job email id only mail/raw/<sha>.eml content-addressed no run yet ingested re-add job one per email id email id blocking read child per job pooled, reused raw MIME via rawRef adds tasks desired count artifacts run · step store message no mailbox ask for earlier mail up to 50 ids · off by default fetch earlier mail one at a time Legend primary data not built yet data store data flow

How jobs run

  • • Mail commits rows, then adds a job carrying only the id
  • • An enquiry entered by hand comes from the API, not a mailbox
  • • Each worker runs one listener waiting on Redis
  • • Earlier mail: up to 50 fetched, then one history job
  • • Sweeper re-adds missed jobs

Decided at the seam

  • • D1 Job handler reads direction from the MIME
  • • D2 Job handler lists attachments from the MIME
  • • D3 Blob storage config shared with the worker
  • • D4 Mail tables: mail stops at EXTRACTION_PENDING; the worker sets the rest
  • • D5 Pipeline tables keep ids stable; link duplicates

Scaling, planned

  • • Capacity = tasks × concurrency (child processes)
  • • Autoscaling adds tasks when backlog per task passes target
  • • Keep at least 1 task so the sweeper can report backlog
  • • SQS reports backlog natively; AGENTS.md still names it