Deploying to AWS: The Bugs a Careful Review Found Before an Account Even Existed

About 1,550 lines of Terraform across seven modules, plus the deployment guide and the GitHub Actions workflow meant to drive it, had been sitting on disk for over a week. None of it had ever been committed, and none of it had ever touched a real AWS account. Before any of that could mean anything, it needed to survive more than one kind of scrutiny, because a syntax linter, a security-minded read, and an "imagine this actually running" walkthrough each catch a completely different class of mistake.

Committing the warts on purpose

The first commit staged everything exactly as it stood, placeholder passwords and a fake account ID included, rather than quietly cleaning it up on the way in. If something in the original draft was wrong, the history should show that it was wrong from the start and got fixed deliberately, not that it was clean from day one. That commit also needed force-adding two config files a bug in the ignore rules would otherwise have silently dropped from git entirely — an un-ignore pattern that only excluded a directory entry, not the individual files a broader wildcard pattern upstream had already matched.

What a linter catches, and what it can't

Running the infrastructure through its own validator surfaced real, hard failures: an invalid attribute on a variable block, a malformed provisioner block using the wrong syntax entirely, and a database cache resource configured with a setting that resource type doesn't actually support — a production configuration that would never have applied, full stop. Those are the good kind of bug: a tool designed to catch exactly that class of mistake did.

A linter has no opinion on whether a configuration is safe, only whether it's valid, and the same pass fixed three things no linter would ever flag. Staging and production had been sharing one hardcoded location to store their infrastructure's own state, with no per-environment separation — deploying production after staging would have operated on staging's already-provisioned resources under production's naming, which is the kind of mistake you only get to make once. That got split so the state location is required explicitly at setup time, per environment, instead of defaulting to one shared value. A newly added HTTPS option for a future custom domain got left commented out rather than half-wired to nothing real, since no domain exists yet. And the database password got pulled out of the checked-in config files entirely, now required as an environment variable with no default.

What a linter can't see at all

A second, security-focused pass over the same commits found a different category of problem, the kind that's perfectly valid Terraform and still a real hole. The trust policy controlling which GitHub Actions runs were allowed to assume the deployment role accepted a run from any branch or pull request, not just the ones the deployment workflow actually triggers on — scoped down to exactly the two triggers that matter, a push to the main branch or a version tag. The new caching layer had no encryption turned on, at rest or in transit — both enabled. And the application tier's security group had a blanket rule allowing every port from the entire private network on top of the correctly scoped load-balancer-only rules, which quietly undoes the whole point of separating tiers with security groups in the first place — removed, since the properly scoped rules already covered every real path traffic takes.

The bugs that only show up once you imagine a real request hitting it

A later review, done before an AWS account existed at all, walked through what would actually happen once a real request landed on a real server, and found two that mattered most. The signing key used to issue login tokens falls back to a hardcoded local-development value if the real one isn't wired up correctly — invisible in every environment tested so far, because every environment tested so far was local development. And the routine that seeds demo accounts on startup had been running unconditionally on every boot, including a production boot, which after the new platform-admin role shipped meant re-granting a cross-business admin role to a guessable account every single time the app restarted, in production, forever. Both got gated by environment: the seed data now runs only in development and staging, never production; the signing key now has to come from a real environment variable or the app won't start.

A cluster of smaller ones would have quietly broken the very first deploy rather than exposing anything. No health-check endpoint existed anywhere, despite the load balancer and the container runtime both being configured to poll one — every health check would have failed forever, and the service would never have been considered healthy enough to receive traffic. The tool that health check depends on to make its own request wasn't even present in the base container image. The login rate limiter keyed off the caller's network address, which is identical for every single request once real traffic is arriving through a shared load balancer instead of directly — collapsing a per-visitor brute-force limit into one shared bucket for the entire internet. A module ordering bug meant the very first apply would have failed outright, looking up load-balancer resources by name before the module that creates them had a chance to run — fixed by having that module hand over the actual identifiers instead of relying on a name lookup that depended on apply order. An environment variable name didn't match what the application code actually reads, meaning the real signing secret would have been silently ignored in favor of the hardcoded development value even after that value was supposedly wired up.

The one I'd flag as most consequential: the frontend's container image ran the development server unconditionally, the same image the deployment pipeline builds and pushes. That's not just slower — a development server ships unminified source, source maps, and a known advisory affecting requests from any origin, live on the public internet behind a load balancer. It hadn't caused visible problems yet only because of an unrelated port mismatch that happened to mask it. Fixing that mismatch without fixing this first would have exposed it immediately. It's now a proper two-stage build: one target for local development exactly as before, and a separate default target that builds the real static bundle and serves it through a minimal web server, which is what any untargeted build now produces by default.

The PM angle

Every one of these got caught before a dollar of AWS spend and before a single real request touched a real server, because the review happened before the account existed rather than after the first incident. That ordering is the whole value: catching a production secret defaulting to a hardcoded value is worth vastly more before it's deployed than after.

The other habit worth naming: no single kind of review would have caught all of this. A linter found the syntax-valid-but-broken configurations. A security-focused read found the ones that were both valid and would apply cleanly, and still wrong. A walkthrough of an actual request found the ones that only show up once you stop reading the code as a specification and start reading it as a sequence of things that will really happen, in order, to a real user. Treating "committed," "passes validation," "reviewed for security," and "actually deployed" as four separate milestones, not one, is what let each pass do the job the others couldn't.

Where the project stands

The backend suite stands at 537 tests passing, up from 535, covering the two new environment-gated behaviors directly. terraform fmt, terraform init, and terraform validate all run clean with no AWS credentials touched — plan and apply haven't been attempted, on purpose, because no AWS account exists yet. A short checklist for standing one up, root account hardening and a dedicated non-root user included, is now written down as the literal next step. Nothing here has been deployed. All of it is what had to be true before deploying could be attempted at all.