A Reliability-Hardening Sprint: Isolation, Guardrails, and a Query Plan That Went From 45 Million to 206

The easiest reliability work to justify is the kind that follows an outage. It's much harder to carve out time for the kind that doesn't, fixes made because a risk is visible and worth closing, not because it already caused damage. Most of what's in this piece shipped in a single week without a single one of these changes being a response to a live incident. One of them was a response to an external report, handled the way you'd want it handled. Together they're a good picture of what proactive reliability work actually looks like, as opposed to the reactive kind that gets written up more often because it comes with a dramatic story attached.

Deciding which environments actually need their own failure domain

Multiple blockchain environments were originally sharing a single default Redis instance for caching. That's a reasonable starting point when the system is small, and a real liability once one environment's traffic pattern starts affecting another's: a cache invalidation problem, a memory pressure spike, or a slow operation in one environment can degrade or contaminate the cache for every other environment sharing that same instance, even though those environments have nothing to do with each other functionally. The fix wasn't to give every environment its own instance unconditionally, it was to identify which ones actually warranted the isolation and pull just those out: mainnet and one specific microchain, the one carrying enough independent traffic and importance to justify it, each got their own dedicated Redis instance, enforced strictly enough in code that the service refuses to start if that instance's address isn't configured. Everything else still shares a common default instance. Nothing about this fix was triggered by an incident. It closed a risk that was visible in the architecture before it had the chance to become one, and it did so by isolating exactly the environments that needed it rather than paying the operational cost of isolating all of them.

A timeout is a blast-radius decision, not a performance tweak

Every GraphQL resolver in the backend service, every query and every mutation, got wrapped with a wall-clock timeout, applied schema-wide rather than added resolver by resolver. It's tempting to think of a timeout as a performance optimization, something you add to make slow things feel faster. That's not really what this fix does. Its actual job is bounding how much damage a single slow or hanging query can do to everything around it: without a timeout, one resolver stuck waiting on a slow database call or a hung upstream request can hold a connection, a worker thread, or a piece of shared capacity indefinitely, degrading the whole service for requests that have nothing to do with the slow one. A timeout doesn't make the underlying slow query fast. It makes sure that whatever caused it to be slow stays contained to that one request instead of spreading.

The number worth quoting on its own

A separate fix, found during a broader review of query performance, is the single most quotable result in this entire piece. A function fetching network-wide transaction statistics was filtering its query by a timestamp column instead of the table's actual partition key. That distinction sounds academic until you see what it costs: filtering on the wrong column meant Postgres couldn't use partition pruning at all, forcing a full scan across every partition, including one that alone held over 400 gigabytes of data. The query planner's own estimated cost for that query, captured at the time, was just under 45 million. Correcting the filter to use the actual partition key was the first fix, and it dropped that same estimated cost to 206, from a query the planner expected to be genuinely expensive to one it treated as nearly free from a single change of which column a filter clause referenced. The system has since moved a step further than that filter fix. The current version of this function doesn't scan the partitioned transaction table at all, even with a correct filter: it reads instead from a separate, precomputed stats snapshot table, sidestepping the scan entirely. The comment left in that code still cites the original ~45 million cost as the reason the snapshot approach exists, a permanent, visible record of the problem the current design was built to avoid.

A related fix in the same family addressed a query fetching transaction lists that was missing partition-key predicates specifically inside its join clauses, which defeated the database's runtime partition pruning and forced scans across well over a hundred partitions where only a small number should have been touched. Both fixes are variations on the same underlying lesson: a partitioned table only delivers its performance benefit if every query touching it, including the parts buried inside joins and CTEs, actually filters on the partition key. Miss it in one place, and the partitioning strategy silently stops helping for that specific query while still looking correct.

An external report handled the right way

Not every fix here was found internally. An unauthenticated search endpoint was found, by an external report rather than an internal audit, to accept oversized input in a way that could trigger server errors. The response was layered rather than a single patch: a hard length cap on the search input rejects oversized requests outright before they reach anything expensive, the cache key for a search result is now a hash of the input rather than the input itself, so even a maliciously oversized string can't be used to construct an arbitrarily large cache key and exhaust cache memory that way, and a dedicated GraphQL security library was added on top, enforcing limits on query depth, aliases, and token count, with batched HTTP requests disabled outright so those same limits can't be sidestepped by sending an array of operations instead of one. The one control deliberately left off was a cost-limiting layer, everything else in that same security library was turned on. The detail worth noting isn't the specific fix, it's that an externally-reported issue got a defense-in-depth response instead of the narrowest possible patch that would have made the specific report stop reproducing.

The small bug that's a reminder about write-path consistency

One more fix, smaller in scope but worth including for what it represents: an address being inserted into a detail table without the padding format that the corresponding foreign-key target table always expected, causing insert failures against that constraint. It's a small, almost mundane bug, but it's a recognizable category: two different code paths writing related data that quietly disagree about a formatting convention neither one documented explicitly. It's the same underlying failure shape as the type-mismatch issues described elsewhere in this series, just at a smaller scale and caught before it caused a bigger one.

What this demonstrates

None of these seven changes, cache isolation, resolver timeouts, two separate partition-pruning fixes, a security hardening response, and a small write-path consistency fix, are individually dramatic. What makes them worth writing up together is that none of them were forced by an outage severe enough to demand attention. They were found by looking, in the middle of a normal week, for risks that hadn't yet become incidents, and closing them anyway. That's a much less visible kind of engineering work than fixing something already on fire, and it's usually the reason there's less to write up after the fact, not because less happened, but because nothing broke.