AWS dev deployment

How the dev environment is provisioned on AWS, and how a code change reaches it.

FlowOS runs in one real AWS environment today, called dev. It exists so the team can review work outside a laptop — it is not the production topology, and it is not built to be one.

Architecture

• Compute: one t4g.medium (Graviton, arm64) EC2 instance runs api, web, mail, worker, units and redis as Docker containers, fronted by Caddy for automatic Let’s Encrypt TLS. Caddy serves web on the app domain and api on its own API domain; web reaches api at http://api:4000 over the Compose network. mail’s bull-board listens only on the box’s loopback at :4001, reached through an SSM port-forwarding session. mail syncs the enrolled mailboxes from Microsoft Graph and hands each stored email to worker over BullMQ on redis; the worker is capped at 1 GB so a large document conversion can only restart the worker, never api or web. units converts units for the worker at http://units:8005, capped at 512 MB and reachable only inside the Compose network. The box keeps a 2 GB swapfile, which every deploy creates if it is missing.
• Mail storage: an S3 bucket holds raw mail, bodies and attachments — written by mail and worker, read by api, all through the instance role. It is retained if the stack is deleted, and admits the web origin for presigned attachment previews.
• Logs: every container logs to CloudWatch Logs, one group per service under /flowos/dev/, kept for 7 days. docker logs on the box still works.
• Load: the flowos-dev CloudWatch dashboard shows the box, its containers and its queues — see Watching the load.
• First boot: mail sync starts paused, and the worker’s model calls and its requirement-gathering calls start off. A Developer turns each on from /debug/mail; a later deploy never pauses a sync again.
• Database: Postgres runs as a separate Amazon Lightsail managed database, not a container on the box — so its data survives replacing the instance. It is publicly reachable, protected by TLS in transit and a strong generated password rather than a network-level restriction (Lightsail databases have no IP allow-list at the infrastructure-as-code level). DATABASE_URL needs ?sslmode=require&uselibpqcompat=true, not just sslmode=require — @prisma/adapter-pg (the driver packages/db/src/client.ts uses, so this applies to apps/api itself, not just one-off scripts) now treats a bare sslmode=require as verify-full, which fails against Lightsail’s certificate with self-signed certificate in certificate chain. uselibpqcompat=true restores the older, TLS-in-transit-only behavior this environment’s security model already assumes. prisma migrate deploy (the Rust engine, not adapter-pg) isn’t affected either way — confirmed by actually running both against the live database, not just reading the driver’s changelog.
• Images: five ECR repositories hold the api, web, mail, worker and units images, and scan each one on push. GitHub Actions, on the org’s arm64 CodeBuild runner fleet, builds and pushes all five on every push to dev. api, mail and worker share one infra/node-service.Dockerfile, whose runtime image holds production dependencies in one layer and compiled code in the next, so a code-only change pulls tens of megabytes rather than hundreds. web runs as a Next.js standalone server, and units builds from services/units/server.Dockerfile. Migrations run from the api image. Everything builds natively for arm64 (no --platform pin), matching the runner and the box.
• Secrets: as in local development, flowos-secrets fetches every secret from Heizen Studio’s dev-aws environment at container start. The box’s own .env, placed once by hand, holds only the Studio bootstrap keys, Caddy’s three domains and the mail bucket’s name; the Compose file sets the Redis and storage wiring. mail needs the Microsoft Graph app credentials and a bull-board login in dev-aws, and the worker needs its model keys before model calls are turned on. infra/AGENTS.md lists every key and where it lives.
• No static AWS credentials anywhere. The EC2 instance authenticates as itself via an IAM instance role; GitHub Actions authenticates via GitHub’s OIDC token exchanged for a short-lived role scoped to this repository’s dev-aws environment.
• This site itself is served the same way, as a second Caddy site block on the same box — a static blume build (apps/docs) at docs.flowtech-os.heizen.tech, no server, no container beyond Caddy, served straight from /srv/docs at the domain root. apps/web’s own /docs redirects here (DOCS_SITE_URL, without the /docs prefix, which the standalone site does not have), so the Guide entry in the product opens this site.

Provisioning

All of the above — the network, the five ECR repositories, the mail bucket, the log groups, the dashboard, the EC2 instance, the Lightsail database, the CI deploy role and CI’s Turbo cache — is defined as AWS CDK (TypeScript) in infra/cdk/. It is provisioned once, by a human operator with real AWS access, not by CI:

cd infra/cdk
npx cdk bootstrap aws://<account-id>/ap-south-1   # once per account/region
AWS_PROFILE=<profile> pnpm cdk-deploy \
  -c githubImmutableSubject=<owner>@<owner-id>/<repo>@<repo-id>

The CI deploy role’s trust condition matches GitHub’s OIDC sub claim using its new immutable subject format — repo:OWNER@OWNER-ID/REPO@REPO-ID:environment:NAME — because this repository, created 2026-08-17 (after GitHub’s 2026-07-15 cutover), gets that format automatically, not the classic repo:OWNER/REPO:environment:NAME. Trusting the classic form failed every deploy with Not authorized to perform sts:AssumeRoleWithWebIdentity. Matching repository_id and environment as separate claims instead of sub was tried next and failed differently: AWS’s IAM API rejects any GitHub OIDC trust policy whose condition doesn’t include a scoped sub or job_workflow_ref — “which is not scoped to all” — so at least one of those two claims is mandatory, not optional, regardless of what else the condition checks. Both failures were confirmed against real cdk deploy runs and GitHub’s own OIDC reference, not assumed from either alone. Find <owner>/<owner-id>/<repo>/<repo-id> with gh api repos/<org>/<repo> --jq '{owner: .owner.login, owner_id: .owner.id, repo: .name, repo_id: .id}'.

CI’s Turbo cache is the flowos-ci-turbo-cache-<account-id> bucket in the FlowosDevCiRole stack, with two roles on the same immutable subject. Pull requests (the pull_request context) assume flowos-ci-turbo-cache-read, which can only read the bucket. Pushes to testing, dev and main assume flowos-ci-turbo-cache, which can read and write it. So only code already on a trunk ever writes an entry that a later run replays. Entries expire after 14 days.

Deploying a change

Once the infrastructure exists, .github/workflows/deploy.yml runs on every push to dev:

1. Assume the deploy role via GitHub OIDC — no stored AWS credential.
2. Build and push the api, web, mail, worker and units images to ECR.
3. Build this docs site (blume build) and package its static output.
4. Upload the deploy script, the Compose file, Caddy config and the metrics publisher to S3, and the packaged docs site both under docs/<commit-sha>.tar.gz, kept for 365 days, and as the root docs-dist.tar.gz that older deploy scripts read.
5. Run infra/deploy-dev-box.sh on the instance over SSM. It makes sure the 2 GB swapfile and the pinned Compose version are in place, stops unless the box’s .env names the mail bucket, pulls the new files, installs the timer that publishes the box’s load, pulls that release’s docs site and its images, runs migrations, starts Redis and pauses mail sync once per Redis volume, then runs docker compose up -d --wait and restarts Caddy. The restart is required, not cosmetic — Caddy only reads its config at startup, and docker compose up -d does not recreate a container over a bind-mounted file’s contents changing.

A deploy fails — and the Actions run goes red — when api, web, mail or units is not healthy within five minutes, or when any service is restarting or has exited 30 seconds later. The run prints docker compose ps -a and the last 50 log lines of each failed service. Nothing is rolled back: the new containers stay up, and api and web do not depend on mail or worker.

cdk deploy goes before the first push that adds a service: the workflow can only push to repositories CDK has already created, and a container whose log group does not exist yet starts with its logs dropped.

A push to dev is the only way this environment’s application code changes. The infrastructure itself only changes when someone re-runs cdk deploy.

Rolling back

Each ECR repository keeps its 50 newest tagged images, latest and buildcache among them, so about the last 48 releases stay available under their commit SHA. To run one of those again, dispatch the workflow against dev with that release’s full commit SHA:

gh workflow run deploy.yml --ref dev -f image_tag=<full-commit-sha>

GitHub shows a Run workflow button only when the default branch (main) carries the workflow, which it does not, so the CLI is the way in.

The run checks every image that commit’s Compose file names exists, skips the build, uses that commit’s Compose file, Caddyfile, deploy script and docs site, and skips migrations — the database stays on the newer schema. That is safe only when every migration since that release only added to the schema. Only releases deployed after the deploy script was introduced can be rolled back to this way. A release deployed before bundles were kept per release runs its own older deploy script, which reads the root docs-dist.tar.gz — always the newest docs — so its rollback leaves the docs site as it is. A later release whose bundle has expired after 365 days does the same.

Watching the load

The flowos-dev dashboard in CloudWatch (ap-south-1) is where to look before deciding the box is too small, or when mail or requirements seem slow:

EC2 reports CPU and credits itself. Everything else is published once a minute by infra/publish-box-metrics.sh, which a systemd timer on the box runs: it reads memory and disk from the host, asks Docker for each container, and counts each queue’s jobs in Redis. A service or queue added later appears on the dashboard without a change to it. Failed jobs are also listed one by one on mail’s bull-board.

Nothing alerts yet: the dashboard has to be opened to be read.

Four commands from the repository root reach all of it. They use AWS_PROFILE, or flowtech-pilot when it is unset:

pnpm box:dashboard              # the dashboard's address
pnpm box:logs worker            # one service's logs, live: api web mail worker units redis caddy
pnpm box:logs api --since 30m   # any `aws logs tail` option passes through
pnpm box:deploy                 # watch the newest deploy run
pnpm box:queues                 # tunnel to bull-board at localhost:14001/queues

pnpm box:queues needs the AWS Session Manager plugin (brew install --cask session-manager-plugin on macOS), and stays open until it is interrupted. Its local port is 14001 so that it does not collide with a local mail, which listens on 4001.

What this is not

This is a dev environment for review, not a production one:

• No staging or production tier exists yet.
• No ECS, Fargate, Aurora, or load balancer — the full production topology (multi-AZ, Aurora Serverless v2, ECS services) is a separate, larger design that this environment does not build toward incrementally so much as precede. When it’s built, this EC2 box does not become it.
• No autoscaling. mail runs as exactly one container (one Graph delta cursor per mailbox) and worker as one; ECS Fargate is where the worker scales on queue depth when that is needed.
• DNS is mapped by hand, not by this infrastructure — a wildcard *.flowtech-os.heizen.tech A record covers app. and docs., and the domain’s DNS is not hosted in the AWS account this environment runs in.