Sprint 18 Closes: A Distributed Cache, a Concurrency Bug It Exposed, and a Test That Couldn't Quite Prove Its Own Fix

Every cache the app had — beer lookups, OAuth exchange codes, the Data Protection key ring — lived in one process's memory, which is fine for one Fargate task and silently wrong the moment a second one exists. Sprint 18 moved all of it onto Redis, eight steps deep, run in dependency order: infrastructure before C#, TLS validated against a live endpoint before any client code got written against it.

Proving the infrastructure works before writing a line of C#

The plan's own suggestion for validating TLS — "from an ECS exec session or a bastion host" — doesn't exist in this infrastructure, and building either just to run one check would have been disproportionate. A throwaway aws ecs run-task did the job instead, reusing the live service's own task definition and network config. The first attempt still failed, for a specific reason: ECS's RunTask container overrides only accept command, not entryPoint, and the image's ENTRYPOINT ["dotnet","BeerApi.dll"] has no CMD to override — supplying just a command would have run dotnet BeerApi.dll sh -c "...", not the validation script at all. Registering a separate throwaway task definition with the entrypoint baked in directly, then deregistering it afterward, worked cleanly: redis-cli --tls against the real ElastiCache endpoint returned PONG, and a plaintext control attempt timed out, which is exactly the confirmation needed that ssl=true in the connection string is load-bearing and not decorative.

Bringing staging up just to run that one check surfaced its own friction, unrelated to Redis: the right AWS credentials profile, a stale Terraform lock from two days earlier, and a secret that had been deleted-but-recoverable earlier the same session leaving Terraform's state and AWS's actual state out of sync until an explicit terraform import reconciled them. Then the API service failed its first deploy outright — a Postgres authentication error, because the RDS instance had been fully recreated with a new master password but the restored secret still held the old connection string. Fixed by updating just that one field while merging in the existing JWT secret without ever exposing it in the process. The circuit breaker from getting the deploy pipeline stable a day earlier did exactly its job here: the failure surfaced fast, with a clear reason, instead of the deployment hanging for ten minutes before anyone found out why.

An interface shaped by its hardest caller, not its easiest one

ICacheStore went in test-first — the test file written against a RedisCacheStore that didn't exist yet, confirmed to fail to compile, then implemented to make it pass. Four methods, but the interesting one is TryConsumeAsync: an atomic get-and-delete, backed by Redis's own GETDEL when Redis is configured, or a semaphore-guarded get-then-remove when it isn't. That method exists because two of the six call sites this cache eventually needed to serve — the OAuth link ticket and exchange code — require a genuine single-use guarantee that plain IDistributedCache can't express in one round trip. Designing the interface around its hardest future consumer, rather than the simplest one, is what made steps 4 and 5 straightforward instead of requiring a second interface later.

The config wiring deliberately kept the existing AddMemoryCache() registration alongside the new branch, rather than replacing it outright the way the original step sketch suggested — the services that still depended on IMemoryCache directly hadn't been migrated yet, and removing it early would have broken the app for the two steps in between. A real bug still slipped past the new tests and got caught by the step's own acceptance criterion instead: confirming docker compose up still worked surfaced a YAML syntax error from the prior step — Redis__InstanceName: mugclub-local: has a trailing colon, which YAML parses as the start of a new mapping key, breaking the entire file. Nobody had actually run docker compose up since that config landed. Application-level tests can't see a Docker Compose file at all; only running the real stack could have found it.

Migrating six cache sites, but not their failure policies, together

The two remaining production call sites — a beer-catalog service and an Open Brewery DB lookup service — got their own small, private fail-open helpers, deliberately not shared between them even though the pattern looks identical, because the OAuth sites needed a different policy against the same underlying cache. AuthController's two cache sites replaced an old TryGetValue followed by a separate Remove call with the new atomic consume, and that wasn't just a cleanup — it closed a genuine concurrency bug. Two simultaneous requests presenting the same exchange code could both read it before either one removed it, meaning the same one-time code could be consumed twice. A new test proves the fix under real concurrency by firing two requests at once and asserting exactly one comes back 200 and the other 401. The failure policy split cleanly along a line worth naming: reads fail closed (a cache miss is a 401, never a 500), writes fail loud (a 503, or a redirect carrying an explicit error — never a silent false success that looks like it worked).

The bug that was already live, and the test that couldn't quite prove its own fix

Sharing the Data Protection key ring through Redis is the one part of this sprint that was fixing something broken today, not just preparing for a future that hasn't happened yet. Without a shared key ring, every Fargate task generates its own, and with no load-balancer session stickiness, an OAuth sign-in issued by one task can get validated by a different one — a validation failure with no useful error message attached. The same gap invalidates every outstanding password-reset token on every container restart, silently.

Verifying the fix produced a genuinely interesting false negative. Trying to prove it red-then-green by reverting the change first, the "red" version of the test still passed — because this development machine already had a Data Protection key directory sitting in its own home folder, left over from some earlier, unrelated run. ASP.NET Core's unconfigured default falls back to a filesystem path scoped by machine and user account, which two in-process test hosts running on the same laptop share by accident, even though two separate ECS tasks in production never would. The honest conclusion wasn't to force a red result some other way — it's that the original bug simply doesn't reproduce locally, for a specific, explainable reason, and the test still proves the fix's actual mechanism works correctly against real Redis. Writing that down as a real limitation is a better outcome than quietly presenting a red-to-green run that wasn't actually testing what it claimed to.

Making a green CI run mean what it claims

The last piece of real engineering gave CI an actual Redis container rather than continuing to skip every Redis-dependent test. A single environment variable gates them — unset locally, they no-op; set, in CI or locally after starting the Redis service by hand, they run for real. Two of them are worth naming: one proves TTL expiry against Redis's actual SETEX semantics, not a fake clock, and one proves genuine cross-instance cache sharing — verified to actually discriminate by temporarily pointing two service instances at separate in-memory caches, confirming the test fails in that configuration, and then reverting. A test that can't fail when the thing it's testing is actually broken isn't proving anything, and checking that directly, rather than assuming a clean-looking assertion does its job, is what makes that claim credible.

The PM angle

The concurrency bug in the OAuth exchange flow is the finding I'd point to first. Nobody set out to look for a race condition — it fell out of doing the Redis migration correctly, because building TryConsumeAsync properly required actually thinking about what "single-use" means under concurrent access, and the old code had never been asked that question. That's a real argument for treating infrastructure migrations as an opportunity to re-examine the logic sitting on top of them, not just a mechanical swap of one backing store for another.

The Data Protection test is the smaller but more instructive one. The instinct when a "red" test unexpectedly passes is to force it red some other way until the story is clean. The better move was asking why, finding a genuine and specific reason, and writing that reason down instead of the tidier version. A false "it reproduced and I fixed it" claim would have been strictly worse than an honest "it doesn't reproduce locally, and here's exactly why."

Where the project stands

Eight steps, roughly ten commits, 571 tests passing at close — up from 563 at the sprint's start — with zero regressions surfacing at any step along the way. Every cache in the app now runs through one interface, backed by Redis when configured and by in-memory storage when it isn't, with the two OAuth sites specifically hardened against the exact concurrency bug this migration exposed. A closing documentation pass caught this log file's own ordering convention had quietly broken — two of this sprint's own entries got inserted at the top instead of appended in place, a small inconsistency flagged rather than silently fixed, per the same project convention that governs every other known doc drift on this project.

The next work after this ran ten days later and started an entirely new epic rather than continuing this one — a decision about what the app should look like, not just what it should be capable of.