Deploying to AWS, For Real: The Ten Bugs a Live Account Found That No Review Could

Every prior pass over this infrastructure — the linter, the security read, the walkthrough of what a real request would do — had one thing in common: none of it had ever touched a real AWS account. That changed the evening of 2026-08-09. Running DEPLOYMENT_GUIDE.md's Phase 0 through Phase 4 against real infrastructure for the first time found ten more bugs, and every one of them was invisible to everything that came before, by construction — a validator can only catch what it knows to check for, and none of these were on that list.

Bugs that only exist once AWS actually answers

Several of the ten are the kind that only a real API response can produce. us-east-1 rejects an explicit LocationConstraint on S3 bucket creation — every other region requires exactly that argument, and the bootstrap script had it unconditionally. A timestamp() call inside default_tags broke plan/apply consistency on nearly every resource, because tags_all evaluates differently at plan time than at apply time — removed outright rather than worked around. ElastiCache rejected its own maintenance and snapshot windows for overlapping every Sunday, which AWS simply refuses to create. And the account itself defaulted to the post-2025 "Free Plan," which blocks RDS from provisioning anything past a trivial backup retention period — confirmed via AWS's own docs that upgrading to the Paid Plan doesn't forfeit the signup credit, so that's what happened.

A separate cluster were ordering bugs — the kind where the Terraform is internally consistent but wrong about what exists yet. The ALB module resolved its VPC through a tag-filter lookup instead of a real dependency, which could never resolve on a first apply since no VPC exists to find. A route table association counted off its own sibling resource instead of the actual availability-zone variable, which broke terraform import's refresh step outright. And the ECR repos and the Secrets Manager secret collided with the manual CLI creation steps from an earlier phase — those needed terraform import, not create, now documented inline so the next account bootstrap doesn't rediscover it the hard way.

The one only a real browser session could find

The most consequential bug wasn't in the Terraform at all. The frontend's API_BASE_URL doubled into /api/api/... once built for the ALB's same-origin routing, because every call site in api.js already appends /api/... — behind the ALB, the base needs to resolve to an explicitly empty string, not /api. The code used || to supply a fallback, and || can't represent "explicitly empty" — an empty string is falsy, so it kept getting overridden. Nothing about that shows up as a build error, a lint failure, or a failing unit test. It shows up as "Create Account does nothing" when a real person clicks a real button, which is exactly how it got caught: a chrome-devtools MCP browser session reproducing the failure live against the deployed URL, then confirming the fix by watching the same click actually work. Two smaller runtime bugs sat next to it — Kestrel defaults to port 8080 in .NET 8's base image, which the local Docker Compose port mapping papers over but Fargate's awsvpc networking doesn't, so ASPNETCORE_URLS had to be set explicitly; and the ECS health check's grace period defaulted to zero, so the service got killed mid-startup on the very first failed health check, because Program.cs runs EF migrations and a full five-tenant demo reseed — hundreds of sequential queries — before the app ever binds the port at all.

A cost estimate that had quietly missed its own biggest line item

Standing up real infrastructure also surfaced a planning gap that had nothing to do with a bug: AWS_DEPLOYMENT_READINESS.md's roughly $150/month pilot estimate had never actually accounted for NAT Gateway cost, which turned out to be a real, fixed ~$93/month while the environment is simply running, deploy or no deploy. A staging-only single_nat_gateway option went in to cut that — one NAT Gateway instead of two, saving roughly $32–35/month — alongside a decision that mattered more than the option itself: since this environment only needs to be up for testing and demoing, not serving a paying customer yet, the agreed pattern is destroy-and-recreate between sessions rather than leaving it running around the clock. Demo data reseeds automatically and idempotently on first boot, so nothing real is lost on teardown, and ECR images plus the Secrets Manager secret are deliberately excluded from the destroy since they cost next to nothing and rebuilding container images is the actual slow part of any redeploy.

Verified live via that same chrome-devtools MCP session: registration, login persistence, and the core add-a-beer/progress flow all confirmed working through the full real chain — ALB to API target group to RDS, and ALB to the web target group's nginx-served bundle.

Getting the pipeline itself to actually work

A working manual deploy isn't the same claim as a working CI/CD pipeline, and the next two days were about closing that gap specifically. The GitHub Actions workflow needed the AWS_ROLE_ARN secret wired into the deploy step directly, and the ECR access policy needed splitting so GetAuthorizationToken — which AWS scopes at the account level, not per-repository — could actually be granted without over-broadening everything else. deploy-staging.sh and destroy-staging.sh convenience scripts went in the same day, so the destroy/recreate pattern didn't require remembering the right sequence of Terraform commands from memory each time.

The pipeline still wasn't reliable yet. A verification run on 2026-08-11 passed its test job cleanly and then failed in deployment — aws ecs wait services-stable timed out after the AWS CLI's hardcoded ten minutes, and that session's findings never made it into a written record, which meant the failure was genuinely lost until it resurfaced. Tracing the root cause the next day found a real gap, not a fluke: beer-api's health-check grace period had been calibrated against whatever the seed happened to take on the day it was set, with essentially no margin against a fully empty, freshly-migrated database — exactly the state every redeploy starts from under a destroy/recreate pattern. beer-web's service had no grace period configured at all. Neither service had a deployment circuit breaker, so a genuinely broken rollout just cycled new tasks for the full ten minutes with no clear signal back to CI. Three changes closed it: beer-api's grace period went from 120 to 300 seconds, both services gained a circuit breaker so a real failure surfaces fast instead of hanging out the clock, and the workflow's built-in wait step was replaced with a custom polling loop that reads each service's actual rollout state and surfaces the real failure reason instead of a generic timeout message. The fix got its live test almost immediately, on a different failure than the one that prompted it — a Postgres password mismatch after a credential rotation — and the circuit breaker did exactly its job: failed fast with a clear reason instead of silently consuming another ten minutes.

The PM angle

The pattern connecting all ten deployment bugs and the pipeline failure that followed is the same one the pre-deployment review already found in miniature: static checks can only fail on the shape of a thing, never on how it behaves once something real is running behind it. terraform validate had passed clean for days. None of that told anyone that AWS itself would reject the tag block, that a browser click would silently do nothing, or that a cold database takes longer to seed than a grace period assumed. The discipline worth keeping isn't "review harder" — it's treating "passes validation" and "actually deployed, watched, and clicked through by a real session" as two separate milestones, because the second one is where all ten of these were actually hiding.

The other habit worth naming: writing down the CI failure from 2026-08-11 immediately, even though the session that hit it had already ended, is what let 2026-08-12 diagnose a root cause instead of just retrying and hoping. An undocumented failure isn't a fixed failure, and this one nearly became a repeat incident instead of a closed one purely because the write-up happened a day late rather than not at all.

Where the project stands

Ten deployment bugs fixed in one commit, the pipeline stabilization following in a half-dozen more across the next three days, and a working, repeatable destroy/recreate cycle in place by 2026-08-12 — verified end-to-end via real browser sessions each time, not just green CI. Sprint 18–21 of the Monetization Roadmap also got written down into tracking the same week, now that deployment itself was no longer the open question blocking everything else.

The very next session needed staging up anyway, to validate a Redis connection over real TLS — which is where the circuit-breaker fix got its second live test, and where Sprint 18 actually starts, covered next.