Five Sprints In: A Post-Mortem on the Bugs the Tests Didn't Catch

Five sprints in — foundation and password auth, the mug club core loop, the customer-facing phone experience, social sign-in, and now the admin experience — I stopped adding scope for a session and wrote a post-mortem instead. Not a status update on what shipped; EPICS_AND_SPRINTS.md already tracks that automatically. A look back at what actually went wrong, how it got caught, and whether there's a pattern worth carrying forward instead of relearning next sprint.

There is one, and it isn't the one I expected going in.

The suite held. That's not the interesting part.

Test suites grew from zero to 236 backend and 149 frontend across five sprints, every one written before its implementation, per a Definition of Done that's been enforced by CI since day one. That discipline held for five straight sprints without drifting, which is worth naming, because "we do TDD" is the kind of policy that quietly erodes under deadline pressure on most projects I've seen. It didn't here, and I think the reason is structural rather than willpower: the CI gate runs on every push, so skipping a test isn't a private shortcut, it's a red X someone (future me) has to explain.

But going through five sprints' worth of session logs looking for what actually broke, the pattern that jumped out wasn't "a test was missing." It was that every real bug fell into one of two categories the test suite structurally can't see: things that only break against real infrastructure, and things where the logic was correct but built on an assumption nobody had actually stated.

Category one: bugs that only exist outside the test host

A plain HttpClient failed to parse an enum back out of JSON during the Customer Phone Experience sprint, because the enum's JSON converter had been registered on the server's own JsonOptions — which the in-process test host inherits automatically, and any other consumer of the API doesn't. The fix was a one-line change (put the converter on the enum type itself instead), but the bug was invisible to the test suite by construction: the test host is the thing that had the workaround baked in.

Same category, different sprint: docker compose up --build web silently kept serving a stale bundle for an entire session, because the compose file had no volume mount for the frontend service, and a project doc claimed otherwise. No unit test touches Docker Compose. The only way to find that bug was to run the actual stack and notice the page wasn't updating.

Neither of these is a testing gap in the sense of "write more tests." They're a reminder that the test suite validates the code; it doesn't validate the environment the code actually runs in. That only gets checked by running the real thing.

Category two: bugs where nothing was technically wrong

The more interesting failures weren't bugs in the traditional sense at all — they were places where the code did exactly what it was told, and what it was told turned out to be an unstated guess. The admin dashboard needed an "active members" count. Nothing in the ticket defined "active." An earlier planning document had already defined it — as members who'd actually confirmed a beer recently, not as accounts that simply hadn't been deactivated — and those are genuinely different numbers. The cheap version was buildable in an afternoon from data already on hand; the correct version needed a new query. Guessing wrong here doesn't throw an exception. It ships a dashboard with a plausible-looking, wrong number on it, and nobody finds out until the number stops making sense to whoever's reading it.

The admin dashboard was also supposed to become the actual landing page for the admin role. It didn't, by default — the home screen had no idea an admin was signed in, and would have kept showing an admin their own beer-drinking progress, which makes no sense for someone who isn't a customer. That's not a logic bug either. It's a feature description ("becomes the landing page") that was true in the ticket and false in the code, until someone checked.

The sharpest version of this showed up mid-sprint, in a planning document rather than in the code at all: a draft design sketch for a data-comparison, written to think through an approach, that a later pass mistook for real logic — because it was formatted exactly like the real code around it. Nothing had shipped yet, so nothing broke in production. But it's the same failure mode one step earlier: an assumption (this pseudocode is just a placeholder, obviously) that nobody had actually written down, sitting one step away from becoming real.

What actually catches this stuff

Not more unit tests — I want to be direct about that, because "write more tests" is the reflexive answer and it's not the right one here. What caught every item above was one of two habits: running the real stack and clicking through the feature as an actual user, or stopping before writing code to ask "what does this word/gap/decision actually mean" instead of picking the reading that's easiest to implement. Both are cheap. Both are also easy to skip when the plan already looks reasonable and the deadline (even a self-imposed one) is close.

The genuinely useful discipline that came out of this sprint was making the second habit explicit rather than incidental: before writing code for anything with real logic, take one more pass and ask what could be wrong, not just what the plan says to build. Twice this sprint, doing that on purpose — not because a bug had already appeared, but as a standing step — caught a problem before a single line of implementation existed. That's a much better trade than catching it in review, and a far better trade than catching it live.

The PM angle

The uncomfortable finding here is that the metric I'd normally point to — test count, suite passing, green CI — was never actually where the risk lived. It's a necessary signal, not a sufficient one. The stuff that actually cost time was always one level up: does the environment behave the way the tests assume it does, and does everyone (including me, a session later) agree on what the feature is actually supposed to mean. Neither shows up in a coverage report. Both show up the moment a real person uses the real thing, or asks a question nobody had thought to ask yet.

Where this leaves things

Nothing here is a reason to change the process wholesale — TDD held, and it's staying. The change is smaller and more specific: treat "click through it live" and "define the ambiguous term before building it" as required steps on anything nontrivial, not optional polish if there's time left. Five sprints of history says that's where the real bugs were hiding the whole time.