Mail to Processing: Data Flow

Mail to Processing: Data Flow A data-flow diagram generated by Archify. 01 / Microsoft 02 / Mail side 03 / Handoff 04 / Worker 05 / Python Microsoft 365 · shared + own mailboxes · 01 / Microsoft · Graph Microsoft 365 shared + own mailboxes Graph apps/mail · one copy, Graph only · 02 / Mail side · no cleaning apps/mail one copy, Graph only no cleaning Files · raw, HTML, new text · 03 / Handoff · S3 Files raw, HTML, new text S3 Records · one shared database · 03 / Handoff · Postgres Records one shared database Postgres Message · process:email · 03 / Handoff · SQS Message process:email SQS apps/worker · 1 to N, owns DB · 04 / Worker · orchestrates apps/worker 1 to N, owns DB orchestrates Extractor · reads files, cleans · 05 / Python · no DB Extractor reads files, cleans no DB Classifier · finds + checks fields · 05 / Python · no DB Classifier finds + checks fields no DB new mail Inbox delta the files step 2 the records step 3 email id step 4 the job email id read records at-least-once safe temporary link presigned GET file links HTTP cleaned text HTTP Legend primary data async batch data store data flow

Mail side builds

  • • Inbox delta per mailbox, one cursor, one process
  • • Files first, records in one transaction, message last
  • • Cleans nothing: the structure must survive for processing

The handoff is three things

  • • Files in S3, records in one shared Postgres (@flowos/db), one queue message: the email id
  • • New today: the new-text body and the job row
  • • A lost message is re-sent by a 5-minute sweep from the job row

Processing builds

  • • Reads the records, sends Python links and text, saves every answer
  • • Python never touches the database or Microsoft
  • • Every step is safe to run twice