This is a walkthrough of that background layer: the Write-Ahead Log, checkpoints, the wait events that make both of them visible, and the vacuum/analyze cycle that cleans up after every write and keeps the query planner honest.

WAL: the black box that makes crash recovery possible

Postgres never writes a change directly to a table's data file first. It writes a record of the change to the Write-Ahead Log (WAL) first, and only later, asynchronously, does the actual data file get updated. This is the same idea as an aircraft's flight data recorder: it doesn't prevent anything from going wrong, but if the plane goes down, the black box is what lets you reconstruct exactly what happened, in order, right up to the last recorded moment.

If Postgres crashes between "WAL record written" and "data file updated," the data file is allowed to be behind. On restart, Postgres replays the WAL from the last confirmed checkpoint forward and reconstructs every change that was durably logged but not yet applied to the heap. That's the entire crash-recovery guarantee, and it's also the entire replication guarantee: a replica is, at its core, a process continuously replaying WAL records shipped from the primary.

Why this matters in production: synchronous_commit controls whether a transaction waits for its WAL record to actually hit disk before returning success to the client. Turn it off (or set it to a weaker level like local) and writes get faster, because you're no longer waiting on an fsync. You're also now willing to lose the last few transactions if the server crashes before that WAL hits disk. That's a real trade-off teams make deliberately for high-throughput, loss-tolerant write paths, and a real trade-off teams make by accident when they copy a config from a benchmark blog post.

Checkpoints: the point WAL gets allowed to forget the past

WAL can't grow forever, and replaying an unbounded WAL after a crash would make recovery take unbounded time. A checkpoint is Postgres periodically flushing every dirty page sitting in shared buffers out to the actual data files, so that everything the WAL described up to that point is now durably reflected on disk. Once a checkpoint completes, the WAL segments before it are no longer needed for crash recovery and can be recycled or removed (subject to what replication and archiving still need them for).

Checkpoints run on a schedule controlled by two settings: checkpoint_timeout (time-based, five minutes by default) and max_wal_size (a soft cap on how much WAL can accumulate before a checkpoint is forced early). Whichever threshold is hit first triggers the checkpoint.

Why this matters in production: a checkpoint means writing out every dirty page at once, which is a burst of I/O. If that burst happens all at once, you get the classic "checkpoint spike," periodic latency jumps that correlate with wall-clock time, not query volume. checkpoint_completion_target exists specifically to spread that I/O out over the checkpoint interval instead of dumping it all at once. If your latency graphs show a sawtooth pattern with a period matching your checkpoint_timeout, that's usually the first setting worth checking, not the query itself.

Wait events: the window into both of the above

pg_stat_activity has two columns, wait_event_type and wait_event, that tell you what a backend is actually blocked on right now, if it's blocked on anything. This is the diagnostic surface that turns "the query is slow" into "the query is waiting on something specific":

SELECT pid, wait_event_type, wait_event, state, query
FROM pg_stat_activity
WHERE wait_event IS NOT NULL;

The wait_event_type column groups events into categories: Lock (waiting on a row, table, or advisory lock held by another transaction), LWLock (waiting on an internal lightweight lock protecting shared memory structures), IO (waiting on an actual disk read or write), BufferPin, IPC, Timeout, and a few others. A backend showing IO wait events tied to WAL activity, or an LWLock wait tied to the WAL insert path, is a backend that's been caught behind the exact mechanism described above: it's trying to get its change durably logged and something (usually disk throughput, sometimes a checkpoint mid-flush) is making it wait its turn.

Why this matters in production: the specific wait event names attached to WAL and checkpoint activity have changed across major Postgres versions as the I/O statistics system was reworked, so don't memorize a table of exact strings from a blog post (including this one) and assume it matches your version. Instead, learn the query above, run it against your own instance during a slow period, and read whatever wait_event_type/wait_event pair comes back against the current docs for your version. That habit outlives any specific release.

Vacuum: cleaning up after MVCC, not just reclaiming disk

Postgres never overwrites a row in place on UPDATE or DELETE. MVCC (multi-version concurrency control) means an UPDATE writes a new row version and marks the old one dead, and a DELETE just marks a row dead, because other transactions that started before yours might still need to see the old version. Those dead row versions don't disappear on their own. VACUUM is the process that finds them, confirms no transaction still needs them, and marks that space reusable by future inserts on the same table.

Plain VACUUM does not shrink the file on disk (only VACUUM FULL does, by rewriting the whole table, which takes an exclusive lock). It reclaims space within the existing file for future rows to reuse. autovacuum runs this automatically, table by table, once the number of dead tuples crosses a threshold (autovacuum_vacuum_threshold plus a percentage of the table's row count, autovacuum_vacuum_scale_factor, which defaults to 20%). Vacuum also has a second job unrelated to space at all: it's what prevents transaction ID wraparound, a much rarer but far more serious failure mode where an un-vacuumed table's transaction ID counter runs out of room entirely.

Why this matters in production: vacuum generates its own WAL records, since it's modifying pages, so a large autovacuum run on a big table shows up as real write volume, not a free background chore. It also competes for the same I/O bandwidth a checkpoint is using, which is why an aggressive autovacuum kicking off right as a checkpoint is flushing dirty pages is a common, specific cause of a latency spike that doesn't correlate with any query change. Database Efficiency 101 goes deeper on the bloat side of this; the piece worth adding here is that vacuum isn't an isolated background job, it's sharing the same WAL and I/O path as everything above.

ANALYZE: the step that keeps the planner honest

ANALYZE is a separate operation from vacuum, even though autovacuum triggers both and people often say "autovacuum" to mean either. ANALYZE samples a table's rows and updates the planner's statistics (row counts, most common values, distribution histograms) stored in pg_statistic. The query planner uses those statistics, not a live count, to estimate how many rows a WHERE clause will match and decide whether a sequential scan, index scan, or bitmap heap scan will be cheapest.

Stale statistics don't cause a query to return wrong results. They cause the planner to misjudge how many rows it's dealing with, and a bad row-count estimate is one of the most common root causes behind "this query used to be fast and now it isn't," with no schema change and no data corruption involved, just a planner making a good decision on outdated information. autovacuum_analyze_threshold and autovacuum_analyze_scale_factor (10% by default) control when this runs automatically, similarly to vacuum's thresholds, and an insert-heavy, delete-free table (an append-only events table, for instance) still needs ANALYZE on a schedule even though it may rarely need vacuuming for dead tuples.

Why this matters in production: after a large bulk load, either via a migration or an initial data import, running ANALYZE explicitly before serving real traffic is worth the few seconds it costs. Waiting for autovacuum to notice on its own schedule means the planner may make its first few hours of decisions on statistics from before the table had any real data in it.

Putting the five back together

None of these are separate systems bolted onto Postgres. They're one pipeline: WAL makes every write durable and replicable, checkpoints periodically settle that WAL's guarantees into the actual data files and bound how long crash recovery takes, wait events are the live diagnostic surface showing you when a backend is stuck somewhere in that path, vacuum reclaims the space MVCC leaves behind and prevents wraparound, and ANALYZE keeps the planner's model of your data close enough to reality to keep picking good plans. A slow query is sometimes a bad query. Just as often, in a system that's been running fine for months, it's one of these five processes falling behind, and pg_stat_activity is the first place that tells you which one.

That's what Prisma, Drizzle, TypeORM, and Sequelize do to your relationship with Postgres. They don't lie. They just don't tell you everything, and the gaps are exactly where production incidents live.

This post is not "ORMs are bad, use raw SQL." Most apps should use one. It's a map of the specific places where the ORM's model of your database and Postgres's actual behavior diverge, so you know where to look when something is slow, wrong, or gone.

Why teams reach for an ORM in the first place

Before the shortcomings, the honest case for using one:

• Types generated from your schema. Rename a column, your build breaks at compile time instead of at 2am in production.
• Migrations as code. Schema changes are version-controlled, reviewable in a pull request, and reproducible across environments.
• Boilerplate CRUD disappears. Most application code touching the database is SELECT, INSERT, UPDATE, DELETE on one or two tables. An ORM turns that into a few lines instead of a hand-written query and a row-mapping function, every time.

None of that is wrong. The problem is what it costs you when you stop thinking about the SQL underneath.

The N+1 query, hiding in plain sight

This is the failure mode that catches the most teams, because the ORM makes it look identical to the correct version.

const users = await prisma.user.findMany();
for (const user of users) {
  const posts = await prisma.post.findMany({ where: { authorId: user.id } });
}

That's 1 query to fetch users, plus N more queries, one per user, to fetch their posts. With 10 users you might not notice. With 10,000 users, those extra round trips can turn an otherwise simple endpoint into a timeout.

The dangerous part isn't that this pattern exists. Every engineer knows N+1 queries are bad. The dangerous part is that the ORM's relational syntax looks the same whether it's doing this or a single join:

const users = await prisma.user.findMany({
  include: { posts: true },
});

Relation loading isn't necessarily equivalent to the join you would write by hand. Depending on the ORM and its configuration, that one line can compile down to a single query with a join, or to a batch of separate queries stitched together in application code. Prisma, for example, defaults to the second approach for include on one-to-many relations: it batches the related rows with a SQL IN clause rather than emitting a native JOIN, unless you're on a relation mode or raw query that says otherwise. You can't reliably tell which one you got just by reading the application code. Turn on query logging and look.

Why this matters in production: N+1 doesn't necessarily mean N new connections. Queries can reuse the pool. The problem is the number of database round trips. Under concurrent traffic, many requests running their own N+1 loops can keep pooled connections busy for longer, increasing pool contention and request latency. This is where an ORM's convenience directly creates the incident.

The SQL your ORM actually generates is not the SQL you'd write

Turn on query logging for a week on a real Prisma or TypeORM project and read what comes out. It is rarely what you'd write by hand.

With nested includes, some ORMs, Prisma among them, can issue several separate queries and stitch the results together in application code instead of one query with joins. That's not a bug, it's a design trade-off, and it varies by ORM, version, and configuration. But it means EXPLAIN ANALYZE on the query you think you're running and the query Postgres actually saw can tell two different stories, and only one of them is visible in your codebase.

Why this matters in production: if you've read through the query planning module of the PostgreSQL In-Depth course, you know Postgres picks between a sequential scan, an index scan, and a bitmap heap scan based on cost estimates for the specific query it receives. An ORM that reshapes your intent into three smaller queries instead of one joined query gives the planner three separate decisions to make, rather than one plan for the operation as a whole. You lose the planner's ability to optimize across the whole operation.

The fix isn't abandoning the ORM. It's treating query logging as mandatory, not optional, and periodically running EXPLAIN ANALYZE on queries that matter to your application's hot paths or have noticeable latency.

Migrations: the auto-diff can silently drop data

Most ORMs generate migrations by diffing your schema file against the last known state and producing SQL. This works well for additive changes. It gets dangerous on renames.

Rename a column in your schema file, and a schema diff doesn't always have enough information to know that you intended a rename rather than a drop-and-add. Some ORMs can detect renames, while others may generate a drop-and-add migration depending on the schema change and migration workflow. The generated migration can be:

ALTER TABLE users DROP COLUMN full_name;
ALTER TABLE users ADD COLUMN display_name text;

instead of the safe version:

ALTER TABLE users RENAME COLUMN full_name TO display_name;

The first version runs without error. It also silently discards every value in that column on deploy.

Why this matters in production: a migration that runs clean and drops a column's worth of data doesn't announce itself. Nobody gets an error. You find out when a support ticket comes in asking why a field is suddenly empty. This is one of the highest-leverage things to check before merging any ORM-generated migration: read the actual SQL it produces, not just the schema diff you wrote.

Connection pooling: every ORM has different defaults, and it matters more than ever in serverless

If you've read Connection Pooling in the Serverless Era, you already know that each connection Postgres accepts consumes server resources, whether or not it's doing anything, and enough concurrent connections becomes a significant resource cost. ORMs each ship their own default pool size, their own idle-timeout behavior, and their own assumptions about how long your process lives.

Those defaults were mostly written assuming a long-lived server process. In a serverless or edge function, where every invocation can be a fresh process, an ORM's default pool of, say, 10 connections means every cold start tries to open 10 connections to Postgres. Multiply that by concurrent invocations during a traffic spike and you can exhaust your database's max connection limit in seconds, well before you exhaust CPU or memory on the compute side.

Why this matters in production: this is a common cause of "it works locally, it falls over under load" for teams deploying ORM-based apps to serverless platforms. The fix usually involves an external pooler (PgBouncer, RDS Proxy, or a managed equivalent) sitting between your functions and Postgres, plus explicitly tuning the ORM's own pool size down, not trusting the default. One trap worth knowing before you reach for a pooler: PgBouncer's transaction-mode pooling, the mode most serverless setups need, doesn't preserve a session across queries, and ORMs that rely on prepared statements (Prisma and TypeORM both do, by default) can fail against it in ways that only show up under load. Check your pooler's mode against your ORM's prepared-statement behavior before you assume adding one fixes everything.

Write-heavy systems: where the ORM's convenience becomes the bottleneck

Everything above assumes a fairly typical application: reads dominate, writes happen when a user submits a form or an admin edits a record. That's most CRUD apps, and it's exactly where an ORM's ergonomics pay for themselves. It is not a blockchain indexer, an event pipeline, or anything else ingesting a continuous, high-volume stream of writes. Those systems are where an ORM's row-at-a-time model of the world stops being a convenience and starts being the bottleneck.

A blockchain indexer is a clean example because the write pattern is unforgiving: every new block can carry hundreds or thousands of events (transfers, contract calls, state changes), and all of them need to land in Postgres before the indexer can call itself caught up. Three places an ORM adds real cost in that path:

• Row-by-row INSERT instead of COPY. Postgres's COPY protocol is built specifically for loading large volumes of rows in one streamed operation, and it is meaningfully faster than even a well-batched multi-row INSERT. Most ORMs don't expose COPY at all. Their bulk-insert APIs (createMany, bulkCreate, and similar) are a real improvement over inserting one row per statement, but under the hood they're still building INSERT statements with many value tuples, not streaming through COPY. For an indexer processing thousands of events per block, that gap compounds every single block.
• Object hydration on the way in and out. An ORM that maps every row to and from a model instance is doing real work per row: validation, type coercion, applying defaults. That cost is invisible at CRUD volumes and very visible at ingestion volumes, where it adds CPU and allocation overhead to every event in the stream, not just the ones a human is waiting on.
• More, smaller transactions than the workload needs. An ORM's ergonomic defaults tend toward one transaction per logical operation. At indexer-scale throughput, that can mean far more transaction and WAL overhead than a design that deliberately batches writes into fewer, larger transactions per block.

Why this matters in production: all three of these show up as the same symptom eventually, insert throughput that can't keep up with the source it's indexing, and the indexer falls further behind with every block. If you've read Taming PostgreSQL Replication Lag in Real-Time Blockchain Indexers, this is the same category of problem from a different angle: it's not always replication configuration that causes an indexer to lag, sometimes it's the write path itself, one ORM abstraction away from the bulk-loading tools Postgres actually gives you for this job.

None of this means an ORM is disqualified from a write-heavy system. It means the write path deserves the same scrutiny as any other hot path in this article: measure it, and if it's the bottleneck, that's exactly the place to drop to COPY, hand-batched multi-row INSERTs, or a dedicated bulk-loading library, and keep the ORM everywhere else it's not in the way.

When the abstraction stops being useful

Every ORM's query builder is designed around the common case: filter, sort, join, paginate. The moment your requirement moves past that, you're negotiating with the abstraction instead of using it. In practice, that means:

• Window functions (running totals, ranking within groups) — most ORM query builders don't model these at all.
• Recursive CTEs (org charts, threaded comments, category trees) — almost always raw SQL, even in projects that otherwise avoid it entirely.
• Bulk operations — an ORM's .update() in a loop is N statements; a single UPDATE ... WHERE id = ANY($1) is one. The difference compounds fast on large batches.
• ON CONFLICT (upsert) and RETURNING — supported by some ORMs, but often with a narrower API than the SQL clause itself allows, especially bulk upserts (ON CONFLICT DO UPDATE across many rows at once) and the newer OLD/NEW aliases in RETURNING.
• SKIP LOCKED for queue-style workloads — the standard pattern for letting multiple workers pull from the same table without blocking on each other's locked rows. Most ORM query builders have no vocabulary for it at all, so it's raw SQL from the start.
• Partial and expression indexes — PostgreSQL can use indexes defined with a WHERE clause or an expression, but the generated SQL still needs to satisfy the conditions that make those indexes usable. An ORM won't necessarily make that obvious from the application code.
• JSONB queries with GIN indexes — the operators that make these fast (@>, ?, #>>) are Postgres-specific and rarely have first-class ORM support.
• Row-level and advisory locking — SELECT ... FOR UPDATE, pg_advisory_lock, and friends are concurrency primitives most ORMs expose thinly or not at all.
• Reading EXPLAIN ANALYZE output — no ORM will do this for you. It's the one skill that stays entirely yours no matter which tool you pick.

None of this means avoid the ORM for these cases. It means know, ahead of time, that you'll drop to raw SQL here, and treat that as normal rather than a sign something went wrong.

Is your ORM a black box? It depends which one

So what should you actually use?

Not all ORMs hide the same amount, and the trade-off usually runs opposite to how much boilerplate they remove:

There's no winner in that table on purpose. A team shipping CRUD-heavy internal tools benefits enormously from Prisma's or Sequelize's ergonomics and rarely hits the edges above. A team running a write-heavy, high-traffic public API needs to know exactly which SQL is running on every hot path, and might be better served by Drizzle or hand-written queries where it matters most. Plenty of production codebases mix approaches deliberately: an ORM for the CRUD majority, raw SQL for the handful of queries that are actually hot.

The actual takeaway

The ORM isn't the problem. Forgetting that an ORM is generating database operations is the problem. It doesn't remove the need to understand what Postgres is doing, it just moves the moment you're forced to learn it, from day one of the project to the incident where your database is drowning in unnecessary queries because the ORM issued forty small UPDATE statements instead of one bulk operation.

Use the ORM for what it's good at: the 80% of your code that's routine CRUD, type-safe and fast to write. But keep query logging on, read the SQL your ORM generates for anything on a hot path, and read the actual migration SQL before it hits production, not just the schema diff. The ORM is a very good translator. It is still worth knowing enough of the language yourself to catch it when it gets a sentence wrong.

The easiest reliability work to justify is the kind that follows an outage. It's much harder to carve out time for the kind that doesn't, fixes made because a risk is visible and worth closing, not because it already caused damage. Most of what's in this piece shipped in a single week without a single one of these changes being a response to a live incident. One of them was a response to an external report, handled the way you'd want it handled. Together they're a good picture of what proactive reliability work actually looks like, as opposed to the reactive kind that gets written up more often because it comes with a dramatic story attached.

Deciding which environments actually need their own failure domain

Multiple blockchain environments were originally sharing a single default Redis instance for caching. That's a reasonable starting point when the system is small, and a real liability once one environment's traffic pattern starts affecting another's: a cache invalidation problem, a memory pressure spike, or a slow operation in one environment can degrade or contaminate the cache for every other environment sharing that same instance, even though those environments have nothing to do with each other functionally. The fix wasn't to give every environment its own instance unconditionally, it was to identify which ones actually warranted the isolation and pull just those out: mainnet and one specific microchain, the one carrying enough independent traffic and importance to justify it, each got their own dedicated Redis instance, enforced strictly enough in code that the service refuses to start if that instance's address isn't configured. Everything else still shares a common default instance. Nothing about this fix was triggered by an incident. It closed a risk that was visible in the architecture before it had the chance to become one, and it did so by isolating exactly the environments that needed it rather than paying the operational cost of isolating all of them.

A timeout is a blast-radius decision, not a performance tweak

Every GraphQL resolver in the backend service, every query and every mutation, got wrapped with a wall-clock timeout, applied schema-wide rather than added resolver by resolver. It's tempting to think of a timeout as a performance optimization, something you add to make slow things feel faster. That's not really what this fix does. Its actual job is bounding how much damage a single slow or hanging query can do to everything around it: without a timeout, one resolver stuck waiting on a slow database call or a hung upstream request can hold a connection, a worker thread, or a piece of shared capacity indefinitely, degrading the whole service for requests that have nothing to do with the slow one. A timeout doesn't make the underlying slow query fast. It makes sure that whatever caused it to be slow stays contained to that one request instead of spreading.

The number worth quoting on its own

A separate fix, found during a broader review of query performance, is the single most quotable result in this entire piece. A function fetching network-wide transaction statistics was filtering its query by a timestamp column instead of the table's actual partition key. That distinction sounds academic until you see what it costs: filtering on the wrong column meant Postgres couldn't use partition pruning at all, forcing a full scan across every partition, including one that alone held over 400 gigabytes of data. The query planner's own estimated cost for that query, captured at the time, was just under 45 million. Correcting the filter to use the actual partition key was the first fix, and it dropped that same estimated cost to 206, from a query the planner expected to be genuinely expensive to one it treated as nearly free from a single change of which column a filter clause referenced. The system has since moved a step further than that filter fix. The current version of this function doesn't scan the partitioned transaction table at all, even with a correct filter: it reads instead from a separate, precomputed stats snapshot table, sidestepping the scan entirely. The comment left in that code still cites the original ~45 million cost as the reason the snapshot approach exists, a permanent, visible record of the problem the current design was built to avoid.

A related fix in the same family addressed a query fetching transaction lists that was missing partition-key predicates specifically inside its join clauses, which defeated the database's runtime partition pruning and forced scans across well over a hundred partitions where only a small number should have been touched. Both fixes are variations on the same underlying lesson: a partitioned table only delivers its performance benefit if every query touching it, including the parts buried inside joins and CTEs, actually filters on the partition key. Miss it in one place, and the partitioning strategy silently stops helping for that specific query while still looking correct.

An external report handled the right way

Not every fix here was found internally. An unauthenticated search endpoint was found, by an external report rather than an internal audit, to accept oversized input in a way that could trigger server errors. The response was layered rather than a single patch: a hard length cap on the search input rejects oversized requests outright before they reach anything expensive, the cache key for a search result is now a hash of the input rather than the input itself, so even a maliciously oversized string can't be used to construct an arbitrarily large cache key and exhaust cache memory that way, and a dedicated GraphQL security library was added on top, enforcing limits on query depth, aliases, and token count, with batched HTTP requests disabled outright so those same limits can't be sidestepped by sending an array of operations instead of one. The one control deliberately left off was a cost-limiting layer, everything else in that same security library was turned on. The detail worth noting isn't the specific fix, it's that an externally-reported issue got a defense-in-depth response instead of the narrowest possible patch that would have made the specific report stop reproducing.

The small bug that's a reminder about write-path consistency

One more fix, smaller in scope but worth including for what it represents: an address being inserted into a detail table without the padding format that the corresponding foreign-key target table always expected, causing insert failures against that constraint. It's a small, almost mundane bug, but it's a recognizable category: two different code paths writing related data that quietly disagree about a formatting convention neither one documented explicitly. It's the same underlying failure shape as the type-mismatch issues described elsewhere in this series, just at a smaller scale and caught before it caused a bigger one.

What this demonstrates

None of these seven changes, cache isolation, resolver timeouts, two separate partition-pruning fixes, a security hardening response, and a small write-path consistency fix, are individually dramatic. What makes them worth writing up together is that none of them were forced by an outage severe enough to demand attention. They were found by looking, in the middle of a normal week, for risks that hadn't yet become incidents, and closing them anyway. That's a much less visible kind of engineering work than fixing something already on fire, and it's usually the reason there's less to write up after the fact, not because less happened, but because nothing broke.

A block height stored as text instead of a number sounds like a small, easily-forgiven decision, and for a long time it is. The chain's own RPC layer returns numeric values like block height as strings in the first place, because some of those numbers exceed what can be safely represented as a standard number in JavaScript, so storing them as text on the way in feels like the path of least resistance. The cost of that decision doesn't show up until someone runs a range query and Postgres compares those "numbers" alphabetically instead of numerically, silently returning wrong results. Fixing that meant migrating the column's actual type from text to a proper integer at its source, the indexer that originally writes it, and then getting every downstream reader of that same column to agree with the new type. The migration itself turned out to be the least risky part of the whole change.

Why the column type was the easy part

Changing a column's underlying type in Postgres is a well-understood operation. The genuine risk in a change like this isn't the ALTER TABLE itself, it's every piece of application code downstream that was written against the old type's behavior without ever stating that assumption explicitly. A function that concatenates a block height into a string for logging, a comparison that relies on lexicographic ordering without realizing it, a serialization layer that formats a string differently than it formats a number, none of these show up as compile errors. They show up as runtime bugs, in production, only when that specific code path executes against the newly-typed column for the first time.

That's exactly what happened here. The migration itself landed cleanly across the backend service and the socket server, both of which read from the same underlying column. Two follow-up hotfixes were needed shortly after, both in functions that hadn't been touched by the migration directly but that broke because they'd been implicitly relying on the column's old string type when computing values like the latest block height and network-wide statistics. Neither hotfix was a large fix. Both were necessary, and both were the direct, predictable cost of a type change rippling through code that had never explicitly declared its assumption about that type in the first place.

What coordinating across services actually requires

The reason this qualifies as a coordination problem, not just a database migration, is that three services, the indexer writing the column, and the backend and the socket server reading it, are all deployed independently, on their own release cycles, each with its own copy of code that depends on this column's type. A schema change made once, in one place, has to be correctly consumed by every service reading that data, and there's no compiler shared across service boundaries to catch a mismatch between what the database now returns and what a given service's code still expects. The two hotfixes here are exactly what "coordination worked, but wasn't perfect on the first attempt" looks like in practice: the underlying migration itself never had to be rolled back or redone, but the code on the consuming side needed a second pass once the new type actually reached it in production.

What this demonstrates

The lesson here isn't "test your migrations more," though that's true of any migration. It's that a type change at the schema level is never really contained to the schema. It's a contract change with every piece of code, across every service, that reads that column, and the services most likely to break are the ones that were never explicitly written against a documented type contract in the first place, because they were written back when the column's actual type was assumed rather than verified. Catching both breakages quickly, shipping targeted hotfixes rather than a broader rollback, and confirming the underlying migration itself held throughout is the realistic version of what "coordinating a schema change across services" looks like when it goes well: not zero surprises, but small, contained ones, caught and fixed fast.

Some engineering work happens in response to an incident. This work happened in response to a feature that hadn't fully shipped yet. Supra's chain infrastructure introduced a new RPC version explicitly designed to support multiple virtual machines processing transactions within the same block, Move and EVM to start, with more planned. At the time this readiness work began, the indexing stack was still built entirely around the older RPC version and its Move-only assumptions. The task wasn't fixing something broken. It was mapping, precisely, everything that would break once the new capability went fully live, before it did.

A wrapper key that touches everything downstream

The documented change between RPC versions looks small in isolation: several top-level fields on a transaction, the sender, the payload, the output, the authenticator, gain a wrapper key identifying which virtual machine produced that data. The interesting part is that the codebase's own type definitions have already started anticipating this shape ahead of the actual rollout: the response types for these fields already model a VM-keyed wrapper alongside the currently-used one, with a placeholder branch for a second VM sitting unused in the type layer today, well before that second VM's data ever arrives over the wire. That's a reasonable way to de-risk a breaking change, let the types absorb the new shape early, in a place where being wrong costs a compile error, rather than doing it all at once under time pressure once the new format is actually live.

Type-level readiness isn't the same as working end-to-end, though, and that gap is the real scope of what's left. Every place in both the indexer and the backend that reads a transaction's sender, payload, output, or authenticator still needs to actually branch on which VM produced it and unwrap accordingly, across multiple files in both repositories, not just have a type that permits doing so. Event parsing, which currently reads from one fixed location in the response, needs equivalent VM-aware branching. None of these are individually difficult changes. All of them need to happen together, correctly, and be exercised against real second-VM data before the system can claim genuine multi-VM support rather than a type layer that's merely ready for it.

Why this is a cross-cutting change, not a parser update

The reason this readiness work spans far more than the RPC-consuming code is what multi-VM support actually means at the data layer. A single block will be able to contain transactions from more than one virtual machine simultaneously, and those transaction types don't share a schema. An EVM transaction's meaningful fields, sender, recipient, calldata, value, look nothing like a Move transaction's sender, function, and payload. That difference has to be handled at every layer: the indexer needs a distinct parser per virtual machine, plus a column identifying which VM produced each row, since a single flat schema can't represent both shapes cleanly. The database question is still genuinely open rather than settled: either add a dedicated table per virtual machine, one for EVM alongside the existing Move-shaped ones, and eventually a Solana or Sui equivalent, or keep one shared transaction table and add a JSONB column to hold whatever VM-specific fields don't fit the common shape. Neither is free. Separate tables keep every VM's data cleanly typed and indexed but mean every cross-VM query, "show me the last hundred transactions regardless of which VM ran them," has to union across tables that don't share a schema. A shared table with a JSONB escape hatch keeps queries simple at the cost of reintroducing exactly the kind of untyped, hard-to-index column this same series describes paying down elsewhere. That tension hasn't been resolved yet, and naming it as unresolved is more honest than picking a side before the second VM actually arrives and reveals which cost matters more in practice. Even something as basic as the wallet triggers described elsewhere in this series need updating regardless of which path gets picked, because EVM addresses use a different length and format than Move addresses, and any logic keyed on address shape needs to handle both correctly. The backend needs VM-aware logic in every resolver that returns transaction details, and the frontend needs to render two structurally different transaction types side by side in the same block view, with address formatting that doesn't assume every wallet address looks the same.

Doing the scoping before the deadline, not during it

At the time of this work, a multi-VM development network already existed and was publicly reachable, with a frontend deployed against it, but the actual multi-VM parsing, schema, and UI work hadn't been built yet, meaning that live deployment likely only reflected Move transactions in practice despite technically running against multi-VM-capable infrastructure. The rollout plan follows a pattern the team had already used successfully once before, when the previous RPC version transition happened: enable the new endpoint behind a feature flag on the testnet environment first, validate the full indexing pipeline end to end against real data in that shape, and only then flip the same flag on production once the chain itself makes the new version available there.

What this demonstrates

The value in this piece of work isn't a shipped feature, because at the time of writing, the full multi-VM parsing and schema support hadn't been built yet either. The value is in having already mapped, field by field and file by file, exactly what breaks and where, ahead of the deadline that would otherwise force that mapping to happen under pressure. Readiness work like this rarely gets noticed when it goes well, because going well looks like nothing happened: the feature arrives, the flag flips, and the system handles it. The alternative, discovering the breaking changes live, in production, once transactions from a second virtual machine actually start arriving, is the outcome this scoping work exists specifically to prevent.

Most engineering work gets justified by its own metrics: faster, cheaper, more reliable. Some of the most consequential work gets justified by someone else's roadmap depending on it. That was the actual shape of the concurrency and trigger-removal work described elsewhere in this series. It wasn't scoped because SupraScan's own dashboards demanded it. It was scoped because a separate team, building automated on-chain task execution, had explicitly tied their own launch decision to it.

A capability that was ready, gated by a system that wasn't yet

The automation team had already done real validation work of their own: their Automation V2 system had been privately tested to handle 100,000 concurrent tasks on a private test network. That's not a small number, and it's not a hypothetical claim either, it was measured. And yet, that team explicitly agreed not to raise the live, on-chain capacity limits for real automated tasks on a real network until the indexer's ability to handle that scale was confirmed.

That's an unusual kind of commitment for one team to make around another team's infrastructure, and it only makes sense once you understand what those capacity limits actually control. The chain enforces a default cap on how many tasks a single account can register, alongside a separate cap on the total gas budget those tasks can consume, plus a lower, distinct ceiling for tasks registered through a governance process rather than a normal account. None of these caps are hard-coded ceilings in the underlying protocol logic; every one of them is adjustable through a standard configuration update, with no code change required. The only thing standing between the current limit and a much higher one is a decision to raise it, which means the real gate on scaling automation usage was never a protocol constraint. It was confidence that the systems consuming the resulting transaction volume, this indexer chief among them, wouldn't buckle the way it already had once, during the four-wallet contention incident that first exposed the trigger-locking problem this same work fixed.

Why the dependency was structurally sound, not just cautious

It would be easy to read this as one team simply being conservative. The more precise read is that the dependency was well-scoped because of how automation execution actually behaves on-chain: each registered task executes at most once per block. That means the worst-case number of executions in any single block scales linearly with whatever the capacity cap is set to, not multiplicatively or unpredictably. A team raising that cap can reason precisely about the additional load it introduces, which is exactly the kind of predictable, boundable increase that makes "we'll raise this once the downstream system is proven ready" a decision you can actually commit to, rather than a vague promise to be careful later.

That precision is also what made this indexer's readiness work legible as a real gate rather than a soft suggestion. The automation team didn't need to trust a general assurance that "the indexer should be fine." They could point to a specific configuration value, know exactly how it maps to worst-case block-level load, and wait for confirmation that the exact failure mode which had already happened once, contention on a small number of hot wallet rows under concurrent load, had been genuinely closed.

What this demonstrates

The trigger-removal and concurrency work described in this series would have been worth doing on its own merits, the four-wallet incident alone justified it. What makes it a more interesting case study than a standard reliability fix is that a different team's product roadmap was explicitly, formally waiting on it. That's a different kind of leverage than fixing your own system's stability. It's being the reason someone else's already-validated capability sat unused on a real network until the work described here shipped, verified, across every environment. Infrastructure work rarely gets to point at a concrete, named dependency like that. When it does, it's worth naming directly instead of folding it quietly into a general reliability narrative.

Capacity planning conversations have a predictable failure mode: someone names a big target number, and the answer that gets built is optimized for sounding capable of hitting it, not for being honest about whether it actually can. This is the opposite of that. A benchmark measured SupraScan's sustained throughput at the time, three concrete scaling levers were evaluated with honest multipliers instead of aspirational ones, and the combined answer, done as carefully as possible, still fell 25 times short of a 500,000 transaction per second target. The useful output of this exercise wasn't a bigger number. It was knowing exactly why that number wasn't reachable with this architecture, and what would actually be required if it mattered enough to pursue.

The baseline, and where the bottleneck actually was

At the time of this planning exercise, a benchmark run measured roughly 110 to 118 transactions per second sustained, across ten running instances. The instinct when a number looks low is to assume the database is the constraint. It wasn't. In the same benchmark window, Postgres itself sat at 50 percent CPU idle, meaning it had considerable spare capacity at that load. The actual bottleneck was the RPC layer: 34 socket-hangup errors inside a 37-second window, with latency climbing past 600 milliseconds. The system wasn't write-constrained. It was constrained by how fast data could be safely pulled out of the chain node in the first place.

Three levers, evaluated without rounding up

Three concrete changes were assessed, each given a multiplier grounded in what it actually does rather than what would sound good in a roadmap slide.

Switching from a pull-based RPC pattern, roughly two RPC calls per block today, to a push-based WebSocket feed removes the single biggest blocker at the current scale: a thundering-herd effect on the RPC endpoint that gets worse as more instances are added, and which currently caps how many instances can even run productively. But this lever is explicitly a ceiling-remover, not a throughput multiplier on its own. It adds zero additional database write capacity by itself; it only clears the obstacle currently in front of scaling out further.

Sharding write traffic is the only one of the three levers that actually multiplies database capacity, because the current architecture writes through a single Postgres primary. Splitting into 8 to 16 shards was judged a plausible, though explicitly not linear, 8 to 16x increase in write ceiling. The honest caveats matter here: cross-shard queries, anything resembling a leaderboard spanning many wallets, get meaningfully harder once data is split this way, and a real, observed pattern of hot-shard skew, one specific address firing continuous automated self-transactions, actively eats into the theoretical gain by concentrating load unevenly across whichever shard that address lands on.

Replacing trigger or sync-based wallet counting with write-ahead-log-derived counters targets the hot-row contention problem directly, the same contention responsible for the multi-day lag incident described elsewhere in this series. Its honest impact is bounded: an estimated 20 to 50 percent reduction in per-transaction database work, explicitly not a throughput multiplier in its own right.

The math, done honestly, and the verdict it produces

Combining all three levers, removing the RPC bottleneck, applying an 8 to 16x sharding gain, and applying a 1.3 to 1.5x reduction from WAL-based counting, produces a combined estimate in the range of 2,000 to 20,000 transactions per second. Against the measured 118 TPS baseline, that's a genuinely significant 15 to 150x improvement, nothing to dismiss. (Production has since grown to 500–1,000 TPS. Even from there, 500K is 500 to 1,000 times away.)

Measured against a 500,000 transaction per second target, even the optimistic end of that range falls roughly 25x short. That's the actual verdict: not reachable this way. Getting to 500,000 for real wouldn't mean tuning the current design harder. It would require a different architecture tier entirely: dozens to hundreds of shards rather than the 8 to 16 evaluated here, a move away from row-by-row ORM inserts with multiple dependent child-table writes per transaction toward bulk or columnar ingestion, since that per-transaction write pattern doesn't scale to that volume regardless of how many shards it's split across, and very likely a genuinely distributed database engine built for that scale, not a set of independently-sharded single-node Postgres instances glued together.

Why saying no was the right call

The easy version of this exercise ends with a roadmap slide showing a path to 500,000 TPS built out of optimistic rounding at each step. The honest version treats a target that large as an architecture-tier decision, a different system built differently from the ground up, not a tuning destination you inch toward by stacking incremental wins. The WAL-based counting work is still worth doing on its own terms, the hot-row contention it targets is real and worth fixing regardless. But it doesn't get oversold internally as a credible step toward a 500,000-tier system, because the math says plainly that it isn't one. Knowing the difference between a real, bounded improvement and a genuine step toward a stated target is most of what capacity planning actually is.

The old design for getting live block and transaction data onto SupraScan's frontend worked, and that's part of what makes it a useful case study. Nothing was broken. The indexer published every block and transaction to Redis pub/sub channels, a socket server subscribed to those channels and rebroadcast the data to connected browsers, and a second, entirely separate Redis stream fanned raw transaction data out to an external team's automated systems. It was a working relay with an extra hop, and the decision to remove that hop, across five separate repositories, is a good example of what disciplined deletion actually looks like.

Old: an extra hop through Redis

  Chain Node --> Indexer --> Redis (pub/sub) --> Socket Server --> Browser
                                  |
                                  +--> Stream Redis --> External team

New: the socket server talks to the chain directly

  Chain Node <==WS==> Socket Server --> Browser
                            |
                            +--> Redis (last 10 blocks / 20 txs, reconnect snapshot only)

The shape of the fix is visible in that comparison: the old path had the indexer sitting between the chain and the live feed even though the indexer's own job is writing to Postgres, not serving live updates. The new path removes it from that role entirely, and Redis goes from being the transport to being a small cache for one specific case, a client reconnecting mid-session.

The new shape: a client, not a relay

The replacement inverts the socket server's role. Instead of subscribing to an internal Redis channel that the indexer populates, the socket server itself becomes a WebSocket client, connecting directly to each blockchain node's own live feed for every environment it serves. The moment a message arrives from the node, it's broadcast straight to every connected browser, with no buffering interval and no per-tick cap holding messages back to smooth them out. Redis still plays a role, but a much smaller one: the last handful of blocks and transactions get cached there purely so a client that reconnects mid-session has something to show immediately, a snapshot, not the primary transport for live data anymore.

Proving the old path was actually dead before deleting it

The discipline here is in what happened before any code was removed, not in the replacement architecture itself. Before deleting the old pub/sub channels, a live subscriber count check against both the QA and production Redis instances confirmed those channels had zero active subscribers. That's a deliberate step, not an assumption: the team didn't reason their way to "nobody should be using this anymore," they checked, live, against production, and only then treated it as confirmed dead traffic. On top of that, the socket server's environment-enable configuration was flipped off for the two environments being migrated, on both QA and production, before the code that implemented the old path was even merged, specifically so there was no window where a live user could hit a code path that had already been half-removed.

The rollout itself crossed five separate repositories, and the merge order mattered: a shared library and the frontend went first, then the socket server, then the backend, then the indexer last. Each merge required a clean build and a grep-verified check that every deleted symbol genuinely had no remaining references anywhere in that repo, before moving on to the next one in the sequence.

Scope that grew on purpose, and a landmine avoided

One piece of functionality got folded into this same change deliberately, not accidentally: a service that computed live transactions-per-second by scanning a sliding window of recent blocks on every single block processed. It turned out to be more computationally expensive than either of the two systems that had already superseded it elsewhere in the stack, a lighter-weight live poll directly against the chain node's own metrics endpoint for the homepage display, and a completely separate metrics-accumulator pipeline feeding the analytics page. Removing a redundant, expensive computation while already touching this code was the right call, and it was made with the author's knowledge rather than discovered as scope creep after the fact.

One specific risk was flagged and handled explicitly during planning: the socket server's environment-enable configuration and its corresponding WebSocket routes had to be removed together, in the same change. Removing one without the other would either leave a live route pointing at handler code that no longer existed, or leave a stale configuration blocking routes that had already been deleted, both of which are the kind of half-finished cleanup that causes an outage days later when someone finally notices.

A second, opposite kind of care went into what wasn't removed. The indexer has its own internal publish and subscribe loop, structurally identical to the exact pattern being deleted everywhere else in this change. But this one is dual-purpose: the indexer's own main processing loop reads its current chain height from that same channel. Removing it under the assumption that it matched the pattern being cleaned up everywhere else would have broken block ingestion itself, not just an external, already-dead feed. It was explicitly flagged and kept, a reminder that the same-looking code in two places doesn't always mean the same thing.

The number worth keeping

A separate but related change in the same era replaced a cold-connect page load's dependency on a live RPC round-trip with a Redis-backed cache instead. The measured result: homepage cold-start hydration latency dropped from 60 seconds to 230 milliseconds. That's not a rounding improvement, it's a roughly 260x reduction in the worst-case time a new visitor waited before seeing live data, and it shipped as part of the same broader effort to make the real-time path leaner and less dependent on hops that weren't earning their cost.

What was accepted, not fixed

The new architecture isn't presented as flawless, and two specific limitations were knowingly accepted rather than solved. There's no per-client replay or acknowledgment: a client that disconnects and reconnects gets whatever the current cached snapshot happens to be, not a replay of exactly what it missed while offline. And the upstream connection from the socket server to each blockchain node has a roughly one-second reconnect gap, during which a block can genuinely be missed by that specific connection. Both are documented, known tradeoffs of favoring a simpler, more direct architecture over a more complete but heavier one, not oversights discovered later.

What this demonstrates

Deleting a working system safely is a different skill than building a new one. Every step here, checking live subscriber counts before removing channels, flipping configuration off before merging code, sequencing five repositories in a specific dependency order, and correctly distinguishing a truly dead code path from a structurally identical but load-bearing one, is about reducing the risk of the deletion itself, not about the destination architecture being clever. The destination here is genuinely simpler than what it replaced. Getting there without an outage was the actual work.

SupraScan runs against five separate blockchain environments today: production mainnet, production testnet, a MultiVM development network, and two independent microchains. All five run the exact same indexer codebase and are served by the exact same backend service. The interesting engineering question isn't "how do you support five environments," it's "how do you add a sixth without touching the code that already works for the first five," and the answer to that turned out to hold up under a genuinely different kind of test than the one it was designed for.

The indexer scales out, the backend scales in

The indexer side of this is deliberately simple: a single codebase, with a configuration variable selecting which blockchain it's pointed at, deployed as a separate running instance per environment. Each deployed instance polls its own RPC endpoint and writes to its own dedicated database. There's no cross-environment coordination at the indexer layer at all, by design; five environments means five independent processes running the same code against five different inputs.

The backend inverts that shape entirely. Instead of five separate backend deployments, one Apollo Federation service serves every environment at once, specifically so a single deployment can serve all of them simultaneously rather than multiplying the number of running backend processes by the number of environments. Every incoming GraphQL request carries a field identifying which environment it's asking about, and a connection manager routes the query to the correct database pool based on that field. The routing isn't a simple two-way split, though, and it's worth being precise about which environments actually share infrastructure today rather than assuming the obvious grouping: mainnet gets its own dedicated pool, one specific microchain has since been given its own dedicated pool as well, and testnet, the MultiVM devnet, and the other microchain share one pool. That's a narrower, more deliberately tiered split than "mainnet alone, everyone else together," and which environments get pulled out of the shared pool has evidently changed at least once since this system was first documented, this microchain's dedicated pool is a more recent addition than the routing scheme's original design.

That drift is worth naming directly: the system's own written documentation still describes the older, simpler grouping, mainnet on one side and testnet, multivm, and every microchain sharing the other, while the actual routing code has since moved one specific environment out of the shared pool without the documentation catching up to reflect it. It's a small, ordinary kind of staleness, the code evolved and the README didn't, but it's a genuine example of why checking live behavior matters even when a design doc already exists and looks authoritative.

There's still a real tension worth sitting with in the environments that do share a pool. A slow query or a resource spike on one environment's traffic runs through the exact same backend process serving every other environment sharing that pool. That's the opposite tradeoff from the one made on the Redis side of this same system, where the two networks judged to need isolation, mainnet and that same promoted microchain, got their own dedicated Redis instances specifically to stop one environment's cache behavior from contaminating another's, while the rest still share a default instance. The database-pool split and the Redis-instance split aren't drawn along quite the same lines, but they're answering the same underlying question, which environments earn dedicated infrastructure and which don't, and the fact that both keep getting revisited as specific environments grow suggests the current split, in either layer, isn't meant to be permanent.

Adding a genuinely new environment to this system is a documented, five-step procedure: register the new environment in a reference table, add its connection configuration to the connection manager in both the indexer and backend codebases, add its RPC endpoint configuration, add a Redis key namespace for it, and deploy a new indexer instance pointed at it. None of those five steps require touching the core indexing or query logic. That's the actual measure of whether an abstraction like this is real: can you add a sixth environment without anyone needing to reason about the first five.

The test this abstraction didn't expect: a different kind of infrastructure entirely

Most of the time, "add a new environment" means the same kind of managed cloud database on a new connection string. A more interesting test came from a different direction: two of the microchain-tier databases were migrated onto plain, self-hosted PostgreSQL running directly on virtual machines, not a managed database service, and not a Kubernetes-orchestrated cluster like some of the other environments use. That's a genuinely different infrastructure category from the managed cloud database clusters the rest of the system runs on, and it's exactly the kind of change that exposes an abstraction that only looked clean on paper.

It held. A direct schema comparison against the equivalent managed database cluster, run live, confirmed 290 tables on the self-hosted box, including the full set of already-created daily partitions, matching the production structure exactly, and a byte-for-byte diff of table and index names between the self-hosted box and its managed counterpart came back identical across all 819 index names. The same application code connected to either backing database correctly, with the only difference being a connection string. Both self-hosted databases also came up with zero database triggers, consistent with the trigger-removal work described elsewhere in this series: any new environment stood up after that work inherits application-level counting logic from day one, with nothing left to migrate. Every environment checked directly, production and QA, mainnet and testnet alike, showed the same result for the two trigger-removal passes confirmed shipped; one further trigger removal from that same series was still awaiting independent confirmation of its production rollout as of this writing, so 'no triggers anywhere' holds for what's been re-verified live, not yet as a fully closed book.

What this demonstrates

A multi-environment architecture is easy to design well when every environment looks the same underneath. This one got tested by an environment that didn't look the same underneath at all, a different hosting model entirely, and the abstraction held without a single code change. That's a more honest test of the design than adding another environment that looks just like the last one, and it's the kind of validation that only shows up when infrastructure decisions made for unrelated reasons, in this case a self-hosting migration on cost or operational grounds, happen to stress-test an architectural boundary nobody built specifically to be tested that way.

JSONB earns its place in a blockchain indexer's schema honestly. Move-based chains produce resources and event payloads with genuinely variable shape, and modeling every one of those shapes as its own narrow table would mean either a schema with hundreds of tables or an unmaintainable entity-attribute-value design. JSONB is the pragmatic answer to that problem. The debt shows up later, once you've been writing JSONB columns for three years and nobody's gone back to ask which of them still earn their keep.

Twelve columns, and the ones nobody was reading

A systematic pass through the schema classified twelve JSONB fields across the indexer's tables into removal phases, ranked by how safe and valuable removal would be. The first tier, already shipped, targeted the clearest cases: a column storing execution statistics on the block table that turned out to be fully derivable from two typed columns already sitting right next to it. A metadata column on the fungible-asset table that nothing in the backend ever actually read, replaced by fetching the same data from the chain's RPC layer on demand instead of storing it at all. A column on an automation gas-assessment table that had been storing an entire event payload when only one derived number from it was ever used anywhere, that number now extracted in memory instead. A signature column on the transaction detail table, dropped the same way, because nothing downstream depended on it being persisted. The rollout for each of these went through an intermediate step before the column disappeared entirely, writes were stopped first, reducing the column to an empty placeholder while everything downstream adjusted, and only then was the column itself dropped. Checking the live schema today confirms all four have completed that full arc: none of them, execution statistics, the fungible-asset metadata field, the gas-assessment payload, or the transaction signature column, exist anywhere in the current tables anymore, not even as an unused placeholder.

None of these were large individually. The observed savings on a lower-volume environment were around 10 megabytes a day, which sounds unremarkable until you scale it by production's substantially higher write volume and multiply by however many years the column would otherwise have kept accumulating. The backend service that reads from these same underlying tables had its own, independent version of this cleanup running in parallel, removing the same category of unused JSONB columns from its side of the codebase, because each service has its own read and write paths into the same tables and each needed its own audit.

The three columns nobody gets to unilaterally drop

Five more fields were still being classified into later removal phases at the time of this writing, with a further idea under evaluation: for the columns that genuinely need to stay, converting the storage format itself to something more compact than raw JSONB, to shrink the cost of keeping them without removing them outright.

Three specific columns sit in a different category entirely, and they're also the largest storage contributors in the entire schema: a payload field on the transaction detail table, an authenticator field on the core transaction table, and a content field on the event table. These three are read directly by teams outside SupraScan's own codebase, which means dropping or reshaping them isn't a decision this team gets to make alone, regardless of how much storage they cost. The options being weighed for these three are all about reducing their footprint without breaking that external contract: compressing the data at write time into a binary column instead of raw JSONB text, moving them into a separate, colder table linked by a foreign key so the frequently-queried hot tables stay lean, or working directly with the teams that depend on them to see whether an API could replace the need for a direct database read at all. None of those are simple swaps, and none of them are happening unilaterally.

The rule that exists now because this happened

The lasting output of this work isn't any single column removal. It's a standing rule, written down rather than left as tribal knowledge, that gets applied before any new JSONB column is added to this schema: can this be typed columns instead, can this be fetched from the chain's RPC layer on demand rather than stored at all, and is this something that's only ever written and never actually read, in which case it shouldn't be stored in the first place. Three years of JSONB columns accumulated because each individual one seemed like a reasonable, low-friction choice at the time it was added. The rule exists specifically so the next three years don't repeat the same accumulation for the same reasons.

What this demonstrates

The interesting tension in this story isn't "JSONB is bad." It's that a tool chosen correctly for a real problem, modeling genuinely variable-shaped blockchain data, still accumulates debt if nobody revisits whether each individual use of it is still earning its cost. Some of these columns had already stopped being read years before anyone checked. Others are permanently protected not because they're well designed, but because an external dependency makes them expensive to change regardless of their internal cost. Paying this down wasn't one migration. It was a classification exercise, a standing rule to prevent recurrence, and an honest acknowledgment that some of the debt isn't fully payable without a conversation that doesn't belong to this team alone.

Four automation wallets, each targeted by roughly 500 automation tasks landing in the same block, was enough to stall the indexer for three to four days. That's the incident that forced a redesign of how SupraScan counts wallet transaction totals, and the redesign that followed it is a good example of how a fix motivated by one incident ends up exposing a second problem nobody was originally looking for.

What the trigger-based design got wrong under load

Wallet transaction counts were originally maintained by database triggers: insert a transaction row referencing a wallet, and a trigger on that table incremented the wallet's stored count automatically. It's a clean pattern in isolation, and it works fine until the same wallet row is targeted by many concurrent writers at once. When automation tasks fire in bulk against a small number of wallets in the same block, every one of those inserts tries to lock and update the same handful of wallet rows through the trigger, and the database has to serialize all of that contention. Four wallets absorbing roughly 2,000 automation-triggered transactions in a single block was enough to create a multi-day backlog, because the trigger contention on those specific rows became the bottleneck for the entire indexing pipeline, not just for those four wallets.

The fix for the insert side moved wallet-count maintenance out of the database trigger and into application-level logic, giving the code direct control over how and when those counts get updated, instead of leaving it to a mechanism that serializes on row locks by default.

The mirror work on delete, and the review that caught what testing didn't

Fixing the insert side solved the immediate incident, but the same tables also had delete-side triggers doing the equivalent decrement, and those carried the identical contention risk. The mirror work replaced three separate delete triggers, on the sender, receiver, and fee-payer tables, with application-level decrement logic, following the same pattern already proven on the insert side.

What's worth calling out here isn't the mechanism, it's the process around shipping it. Three real correctness bugs were found and fixed during code review, before merge, and two of those three were caught by a reviewer, not by the person who wrote the change. That's a detail easy to leave out of a case study because it isn't flattering in the usual sense, but it's exactly the kind of evidence that matters: a change this close to core data integrity got a second set of eyes that found problems the first pass missed, and the process worked the way it's supposed to.

The rollout itself was deliberately cautious given what was at stake. Processing was fully paused, a drift query ran against the still-paused system comparing the stored wallet totals against a value freshly recomputed from the source transaction tables, the trigger-removal migration ran, the same drift query ran again to confirm nothing had changed, and only then did processing resume. QA testnet went first. The full rollout, across every QA environment and both production networks, shipped in early August 2026, with the triggers confirmed dropped and the drift check clean on every environment.

A separate, unrelated bug turned up during this same work: a configuration flag governing in-loop transaction-info removal was found live in production, in a way that violated its own documented pairing rule with the partition-management flag from the storage-scaling work. It was flagged, not fixed immediately, a reasonable call given it wasn't the thing actively causing an incident.

The bigger question this raised, and the assumption that turned out to be wrong

While this work was underway, a separate initiative was considering something much larger: building a full write-ahead-log-based counting system to replace the interim application-level fix entirely, with its own dedicated architecture and integration specs. The stated motivation wasn't the load already observed, it was a theoretical worst-case burst ceiling, on the order of 500,000 transactions per second under a maximum block rate and block size, versus a sustained load at the time closer to six transactions per second. The concern was hot-wallet row contention under a burst nowhere near what had actually happened yet.

Midway through that discussion, a direct check against the live production deployment configuration turned up something that changed the shape of the entire argument: the production indexer was running as a single instance, not the two instances that existing architecture documentation had claimed. That distinction matters enormously for a concurrency argument. A single instance can coordinate its own internal work without needing distributed locking, because there's no second process to race against. The primary justification for building a distributed, Redis-coordinated write-ahead-log system, protecting against cross-instance concurrent writers, simply didn't apply to the system as it was actually deployed. What remained of the WAL system's value after that correction was observability and per-module statistics, useful, but a different and much smaller problem than the one it was originally scoped to solve.

Two more trigger removals, and one that both prior efforts missed

The same interim pattern, application-level counting instead of a database trigger, was later mirrored for automation execution counts on a related table, shipped to QA first with production following shortly after. A third trigger, tied to automation registration counts on the same underlying entity, was found only later, and only because of a full, systematic audit across every live database rather than trusting that two prior rounds of "complete" migration had actually been complete. Two separate migration efforts, each believed finished when it shipped, had both missed a related trigger sitting on the same table family. The audit that finally caught it didn't rely on anyone remembering to check; it re-verified live state directly, which is the only thing that actually caught what two rounds of careful, reviewed work had both missed. Two of those three migration passes, the insert-side and delete-side removals on the sender, receiver, and fee-payer tables, are confirmed shipped and verified clean across every environment. The third, the registration-count trigger this same audit surfaced, was implemented in application code but its merge and production rollout weren't independently reconfirmed as of this writing, worth stating plainly rather than rounding up to 'done' before the same kind of live check that caught it in the first place gets run against it again.

Why this mattered beyond the indexer's own stability

This work wasn't prioritized purely for the indexer's own sake. A separate team building automation tooling had privately validated their own system could handle 100,000 concurrent tasks on a private test network, but had explicitly agreed not to raise the live, on-chain task-capacity limits on the real network until the indexer's concurrency handling was confirmed ready for that scale. The chain's own configuration allows those capacity limits to be raised with no hard ceiling in the underlying code, only a governance action, which meant the actual blocker to a real capacity increase wasn't a protocol limit, it was confidence that the indexer wouldn't repeat the four-wallet incident at ten or a hundred times the concurrent load. This trigger-removal and concurrency-hardening work was the literal, agreed-upon gate that team was waiting on, not preparation for a hypothetical future.

What this demonstrates

The most useful thing about this sequence isn't the trigger-to-application-logic pattern itself, which is a known technique. It's what surrounded it: an incident that motivated an immediate fix, a review process that caught real bugs a test suite hadn't, a live-configuration check that overturned an architectural assumption a much larger initiative had been built on, and an audit discipline that treated a prior migration's own claim of completeness as something to verify, not trust. Concurrency bugs at this scale rarely announce themselves cleanly. Finding them consistently is less about any one clever fix and more about refusing to assume the last fix, or the existing documentation, already got it right.

How many distinct wallets touched the network in the last hour. It sounds like a question with an easy answer until you try to answer it at scale, continuously, across five blockchain environments, without storing every wallet address that was ever active in every window you care about. SupraScan answers it with HyperLogLog sketches: a fixed, tiny amount of memory per time bucket gives an exact distinct-wallet count for any window you ask about, five minutes, an hour, a day, without the storage cost of ever keeping the underlying address list around. That mechanism working reliably in production, continuously, is the achievement. The part worth writing about in detail is what it actually took to get there: in one specific week, the system's reported numbers were wrong five separate times, for five genuinely different reasons, and finding each one is as much a part of "hardening a production system" as the sketch design itself.

Why sum isn't the same as union

The core mechanism is a sketch union: instead of storing every distinct wallet address seen in a time bucket, the system stores a small, fixed-size sketch that can answer "roughly how many distinct items have I seen" using a fraction of the memory a full set would need. The system counts unique senders and receivers this way, and it keeps a sketch for every 5-minute bucket, then needs longer windows, an hour, a day, built from those. The reason those longer windows can't just add up their constituent buckets' counts is structural, not a rounding preference: a wallet that shows up in more than one 5-minute bucket within the hour is a single wallet, and adding the per-bucket counts together counts it once for every bucket it happened to appear in, while simply taking the largest of the per-bucket counts undercounts everything down to whichever single bucket happened to be biggest. A union of the sketches themselves is the only operation that gives the actual right answer for the combined window, which is exactly why the system keeps merging 5-minute sketches upward into longer-lived hourly and daily ones rather than trying to reconstruct a rollup from already-computed counts.

That distinction is not academic. If bucket A has 55 unique senders and bucket B has 2 unique receivers, and some of those addresses appear in both, summing the two counts overcounts by exactly the overlap. Unioning the sketches gives the true combined count, 59 in that specific case, because a union correctly recognizes overlap the way a sum structurally cannot. The first fix in this saga replaced a cardinality sum with a true sketch union, paired with a corresponding change on the backend that consumes the corrected sketches. It's the fix that makes the system exact instead of approximately-too-high, and it's also the fix that makes everything downstream sensitive to a category of bug that a naive counter would never have exposed in the first place.

Five bugs, five different ways to catch them

Once the union fix was in place, four more real correctness bugs surfaced and were fixed over the following days, each caught by a completely different signal.

The second was a double-counting bug on reseal: an operation meant to update a cumulative counter was using simple addition instead of a "take the greater of the two values" comparison, so re-sealing a bucket that had already been sealed once doubled its stored total. This one was caught by an internal invariant checking itself: a 5-minute bucket showed more unique senders than the full hour that was supposed to contain it, which is structurally impossible if the counting is correct, and that impossibility is what surfaced the bug.

The third bundled two separate issues in one commit. The sender-count metric was reading from the wrong source column, quietly undercounting. Separately, a get-then-increment pattern across concurrent block-processing pods created a race window that could inflate peak metrics like maximum TPS or maximum gas price, because two pods could both read the same "current maximum" value before either had written back their update. The fix for the race was to replace the separate read and write with a single atomic compare-and-set, implemented as a Lua script, closing the window entirely rather than trying to make the race smaller. The shape of that fix, illustrative of the pattern rather than a verbatim excerpt, looks like this:

-- KEYS[1] = the peak-metric key for one environment (e.g. a maximum-throughput counter)
-- ARGV[1] = the newly observed value from this pod
local current = tonumber(redis.call('GET', KEYS[1]) or '0')
local candidate = tonumber(ARGV[1])
if candidate > current then
  redis.call('SET', KEYS[1], candidate)
  return candidate
end
return current

The point of running this as a single script isn't the comparison itself, it's that Redis executes a Lua script atomically. Two pods can call this at the same instant and still only one write happens for whichever value is actually higher, because there's no gap between the read and the write for a second caller to land in. A plain GET followed by a separate SET from application code always has that gap, no matter how small it looks in a benchmark.

The fourth was the least intuitive of the five. A bulk-insert call configured to ignore duplicate rows was, by the specific behavior of the ORM version in use, returning every input row regardless of which ones were actually new inserts versus duplicates silently skipped by the underlying conflict-do-nothing clause. That meant a "new wallets" counter had been reporting batch size, not actual new-wallet count, for an unknown period of time, and it stayed undetected until someone traced the number against ground truth: the same insert statement's own returning clause, which only reports rows that were genuinely inserted. Once that comparison was made, the gap was obvious. Before it, the metric looked plausible on every dashboard that displayed it.

The fifth, and most recent, was a data-quality bug rather than a counting-logic bug: event parsers were hardcoding a transaction's origin type to "user" regardless of what the transaction's actual origin was, which inflated sender counts specifically on digital-asset and NFT-related rows that weren't actually user-originated. It shipped alongside a cleanup that removed redundant, duplicate HLL calls sitting in the same code path.

The pipeline underneath, and the incidents it produced on its own

The counting mechanism sits inside a small pipeline of its own: a service increments Redis-backed counters per environment as events happen, and a leader-elected "sealing" process periodically flushes completed time buckets into a durable table. Two separate incidents came out of that pipeline, independent of the five counting bugs above.

The first: the leadership-election check for who becomes the sealing leader only ran once, at process startup. A quick pod restart left a stale leadership heartbeat behind, and no pod ever re-ran the election to notice the old leader was gone. Sealing stalled for roughly 29 hours on one environment before a later restart happened to trigger a fresh election and catch everything up. No data was actually lost, because the underlying Redis counters kept accumulating with their TTLs refreshed the whole time, but the reported metrics were stale for over a day before anyone would have seen it in a dashboard. The fix was straightforward once diagnosed: retry the leadership election on every seal cycle, not only once at startup.

The second was a load problem, not a data problem. During any catch-up or reprocessing run, every block being reprocessed carries a historical, not live, timestamp. That historical timestamp routed metric increments down a direct-to-database write path instead of the normal Redis-then-leader-seals path, and with many pods reprocessing concurrently, that meant hundreds of simultaneous inserts competing for the same handful of bucket rows, with most of them stuck lock-waiting. The practical effect was block ingestion throughput dropping to zero every time this write path triggered during a large catch-up. The workaround in place is to disable that direct-write path entirely during any catch-up or reprocessing run. A proper fix, moving those historical writes through the same leader-only accumulation path as live traffic, hadn't shipped as of this writing.

What this actually demonstrates

The algorithm here is not the hard part, and pretending otherwise would undersell what actually happened. HyperLogLog union is a known, well-documented technique. What's harder, and less often written about, is that a sketch-based counting system sits at the intersection of several failure modes that don't show up in a tutorial: reseal semantics, concurrent-writer races, ORM behavior that silently changes what a "successful insert" means, and a leader-election scheme that has to keep re-checking itself rather than assuming its first decision holds forever. Five real bugs in one week isn't a sign the system was poorly built. It's what building something that reports an exact number, under continuous concurrent load, from a fundamentally approximate data structure, actually looks like when you take correctness seriously enough to keep checking it.

In 2021, SupraScan's transaction tables lived in a single partition each. That was the right call at the time: low volume, simple queries, nothing to gain from splitting data you could scan end to end in milliseconds anyway. Five years later, that same table shape would have been an operational disaster. The interesting part of this story isn't that partitioning strategy changed. It's that it changed five separate times, each change a direct response to a problem the previous grain created, and the last of those changes caused a real production outage before it was fully safe.

Stage one: one partition, then six months, then a monster

The grain tightened in stages as ingestion grew. From 2021 through mid-2024, everything sat in one partition per table. From mid-2024 through the start of 2025, tables were split into six-month partitions, still coarse, but enough to keep individual partitions from growing unbounded.

Then came the full-year partition covering all of 2025, created when the six-month grain was tightened again. That single partition, covering an entire year of transaction events, became the largest structural liability in the entire database. By the time a dedicated storage audit looked at it in April 2026, the prod SupraScan database sat at 10 TB on a single Postgres instance, and that one year-long event partition alone accounted for 4.9 TB of that, roughly half the entire database, driven by three JSONB columns storing content, event data, and identifiers. A separate transaction-detail table was carrying 71 to 86 percent index overhead, meaning its indexes were two to three times larger than the data they indexed. Ingestion at that point was running around 500,000 transactions and 11 million events per day, all still landing in yearly buckets.

The audit, and the decision not to rebuild everything at once

Faced with a 10 TB database and a monster partition eating half of it, the natural instinct is to reach for a bigger architectural change. Three tiering options were seriously evaluated, and it's worth being honest about how each one would actually have played out, not just what it promised on paper. Moving cold data into ClickHouse for the older window would have delivered the best compression and the fastest historical queries of the three, at the real cost of operating a second database engine indefinitely: a new sync pipeline, a second system to monitor, and a second place a query result could quietly drift from the source of truth. Archiving cold data to object storage as Parquet was the cheapest option by a wide margin, but it pushed historical query latency out to 2 to 10 seconds, a real, felt tradeoff for anyone or anything querying older data, not a rounding error. Staying entirely inside Postgres with TimescaleDB's columnar compression was the lowest-effort option precisely because it didn't introduce a second engine at all, at the cost of a compression ceiling lower than a purpose-built columnar store would give.

None of those decisions were made in isolation from a more basic fact: regardless of which tiering approach eventually gets picked, a set of quick wins didn't depend on the choice at all. Extracting frequently-queried JSONB fields into typed columns, auditing and dropping unused indexes on that same transaction-detail table, splitting the year-long monster partition into monthly chunks, and adding Redis caching for hot-path queries were all identified as work worth doing regardless of the bigger architectural direction. That's the pattern worth naming here: the team didn't wait for a strategic decision to start paying down the parts of the debt that were unambiguous.

Tightening the grain again, and the outage that came with it

The partition grain kept tightening after the initial audit: monthly partitions from early 2026, then daily partitions starting in April 2026, auto-created seven days ahead of when the data would land, managed by a dedicated partition-management service running on UTC midnight. The reasoning for going all the way to daily wasn't purely about the production database. QA and testnet environments only need to retain seven days of data, and dropping a daily partition is instant, where deleting the equivalent rows row-by-row was generating real database load on environments that didn't need the data kept at all. Production mainnet runs in create-only mode, keeping every partition forever; QA and the testnet tiers run with cleanup enabled, dropping partitions older than the retention window automatically.

This is also where the evolution produced a real incident. On April 27 and 28, 2026, the automated partition-creation cron silently failed to create the upcoming daily partitions for exactly three tables: the sender, receiver, and fee-payer tables, the only partitioned parents in the schema carrying AFTER DELETE row-level triggers. The reason was subtle: creating a partition on a table with active triggers loses a lock race against the live indexer process writing to that same table, and the create-only rollout hadn't built in a pause for that window. The failures were caught, but only logged at warning level per table, so the automated job looked healthy from the outside while it was quietly failing on exactly the tables that mattered most. The outage hit at 03:57 UTC on April 29, when the indexer tried to insert into a partition that had never been created. It was fixed manually with a defensive CREATE TABLE IF NOT EXISTS ... PARTITION OF statement, and two concrete follow-ups came out of it: pausing the indexer during partition creation, and extending the creation lookahead from two days to seven, so a single missed run has more runway before it becomes a live outage.

Where the numbers landed

A live measurement taken shortly after this work, on May 4, 2026, put the production mainnet database at 4,667 GB, roughly 4.6 TB, down from the 10 TB the April audit had measured on the same instance. It's worth being honest about what does and doesn't explain that drop: the individually-quantified levers, 229 GB from the index audit and roughly 10 MB a day from the first JSONB removals, don't add up to 5.4 TB on their own. The rest is best attributed to the broader restructuring work described here, not to a single measured fix, and that gap is worth naming rather than implying away. The old monster partition's damage was still visible in the numbers even after the paydown: that same year-long transaction-detail partition alone held 452.9 million rows across 1,641 GB, 86 percent of which was index, not data. The new daily partitions for 2026 were running at a much more modest 4.5 to 5 GB per day.

The same audit pass surfaced a second, entirely separate lever: a systematic scan for indexes that had never been scanned, ever, in production. It found 229 GB across 296 zero-scan indexes on prod mainnet alone, with individual offenders as large as a 57 GB primary key index on an automation-fees table with zero recorded scans. That number, sitting right next to the partition work, is the honest reminder that storage debt in a system like this rarely has one cause. Five years of grain changes, a JSONB-heavy schema, and indexes created for a query pattern that never materialized all compound in the same database at the same time, and the fix for each one looks nothing like the fix for the others.

The grain history is still visible directly in the schema today, not just in a report from the time it happened: the earliest partition on the core transaction tables still covers that original 2021-through-2024 span, followed by the six-month partition marking the next grain change, the full 2025 partition still carrying the size of the monster year it covers, three separate monthly partitions for the first quarter of 2026, and daily partitions from spring 2026 onward, currently running several days ahead of the present date exactly as a seven-day creation lookahead would produce. Fifteen separate tables carry this same partitioning strategy today. The evolution this piece describes isn't a closed chapter being reconstructed from memory, it's sitting in the live schema as a literal, dated record of every decision described above.

What the progression actually shows

None of the five grain changes were wrong at the time they were made. Yearly partitions were a reasonable tightening of six-month partitions, until ingestion volume and JSONB column size turned that year into a 4.9 TB single object. What makes this a case study rather than a postmortem is that each stage's cost only became visible under the load the previous stage couldn't have anticipated, and the team's response each time was to tighten the grain further rather than declare the whole model broken and start over. The daily-partition outage is part of that same story, not a separate failure. It's what happens when a mature, working pattern gets pushed one more notch tighter and meets a database-level detail, trigger locking during DDL, that the earlier, coarser grains had never been fast-moving enough to expose.

Then one day the process gets killed between those two lines. Deploy rolled a pod mid-request. The broker connection timed out for four seconds during a network blip. A GC pause pushed the request past a load balancer timeout and the client retried against a different instance while the first one was still finishing up. The order is sitting in the database, fully committed, real, billable. Nobody downstream ever hears about it. No exception was thrown. No alert fired. The system looks healthy. A customer just placed an order that will never ship.

Why You Can't Just Wrap It in a Transaction

The instinct is to reach for a transaction: begin, insert the order, publish the event, commit. That doesn't work, and it's worth being precise about why, because the reason is architectural, not a missing try/catch.

A database transaction gives you atomicity over one resource: the database. Postgres can guarantee the order row either exists or doesn't, with nothing in between visible to other readers. A message broker (Kafka, RabbitMQ, SQS) is a separate system with its own commit protocol, its own durability guarantees, and no shared transaction coordinator with your database. There's no operation that atomically says "commit this Postgres row and this Kafka message together, or neither." You are making two independent network calls to two independent systems, and between them there is always a window where one has succeeded and the other hasn't.

Move the event publish before the database commit and you get the opposite failure: the event goes out, downstream services start acting on an order that doesn't exist yet, and then the database write fails (constraint violation, connection drop, whatever) and now payments has charged a card for an order that was never actually created. Order doesn't matter. Two independent writes to two independent systems, with no atomicity across the boundary, will eventually diverge no matter which one goes first. This is the dual-write problem, and it's not a bug in your code, it's a gap in what a database transaction can promise you.

Why this matters more than it looks like it should: this isn't a rare edge case you can accept as background noise. Every deploy is a process restart. Every autoscaling event is new pods coming up while old ones drain mid-request. Every network blip between your service and your broker is a window for this to happen. At low traffic it might occur once a month and get written off as "weird, must've been a fluke." At real scale, with thousands of writes a minute, this gap fires constantly, and it fires exactly during the conditions you're least equipped to notice it: deploys and incidents, when your attention is already somewhere else.

Two-Phase Commit: The Theoretically Correct Answer Nobody Uses

The textbook fix for "atomically commit across two systems" is Two-Phase Commit (2PC). A coordinator asks every participant (the database, the broker) to prepare, meaning "lock this resource and confirm you can commit, but don't commit yet." Once every participant says yes, the coordinator tells everyone to commit for real. If any participant says no, the coordinator tells everyone to roll back.

It's a real protocol with a real correctness proof, and it's almost never what production systems actually use for this problem, for reasons that show up the moment you operate it instead of just reading about it:

• It's blocking. Once a participant says "yes, I can commit," it has to hold that lock until the coordinator's final decision arrives. If the coordinator crashes after collecting votes but before sending the commit decision, every participant is stuck holding locks indefinitely, waiting for a coordinator that might not come back.
• Most message brokers don't implement the participant side of the protocol at all. Kafka has no native 2PC participant role. You'd be building and maintaining that coordination layer yourself, on top of a system that was never designed to expose it.
• It doesn't survive network partitions gracefully. The entire protocol assumes the coordinator can eventually reach every participant. A partition during the decision phase leaves things in exactly the indeterminate, locked state the protocol was supposed to prevent.

2PC is the right answer to a narrower question: coordinating multiple databases that all speak a compatible protocol, in a controlled environment where you own the failure modes. It's the wrong tool for "coordinate my database with my message broker," which is the shape of the dual-write problem almost everyone actually hits.

The Outbox Pattern: Make the Second Write Boring

The pattern that actually gets used starts from a reframe: stop trying to make two different systems commit atomically, and instead make the second write something the same database transaction can own.

Instead of publishing to the broker directly, you write the event as a row in an outbox table, in the exact same transaction as the order insert:

BEGIN;
INSERT INTO orders (id, customer_id, total_cents, status)
  VALUES ('ord_123', 'cust_456', 4999, 'created');
INSERT INTO outbox (id, aggregate_id, event_type, payload, created_at)
  VALUES (gen_random_uuid(), 'ord_123', 'order.created', '{"order_id":"ord_123","total_cents":4999}', now());
COMMIT;

Now atomicity is trivial, because both writes are ordinary rows in the database you already have transactional guarantees for. Either both rows exist or neither does. There is no window where the order exists but its event doesn't, because "the event exists" now just means "a row in the same table transaction says so."

A separate relay process, running continuously, polls the outbox table for unpublished rows (or reads Postgres's write-ahead log directly via logical replication, which is how tools like Debezium do it without polling), publishes each one to the real message broker, and marks it published. If the relay crashes, it resumes from the last unpublished row on restart. If the broker is briefly unavailable, the events just sit in the table until it recovers. The order write itself never blocks on the broker being up at all.

The trade-off is honest, not hidden: downstream consumers now see events after a short delay (however often the relay polls, typically sub-second to a few seconds), and they can receive the same event more than once if the relay publishes successfully but crashes before marking the row as sent. That second part is why every consumer of these events has to be idempotent, not because it's good hygiene in the abstract, but because at-least-once delivery is the actual guarantee the outbox gives you, and "exactly once" isn't achievable without it.

Sagas: The Same Problem, Stretched Across Multiple Services

The outbox pattern solves atomicity for one write plus one event. A saga is what you reach for when the operation itself spans multiple services, each with its own local database, and there's no way to wrap the whole thing in a transaction even in principle: reserve inventory, charge the payment, create the shipment. Three services, three databases, one logical operation.

A saga runs this as a sequence of local transactions, each one committing on its own, with a compensating action defined for each step in case a later step fails:

1. Inventory service reserves the item. Commits locally.
2. Payment service charges the card. Commits locally.
3. Shipping service creates the shipment. If this fails...
4. ...run the compensations in reverse: refund the payment, release the inventory reservation.

Nothing here is atomic in the ACID sense. There's a real window, between steps 1 and 3, where inventory is reserved and no shipment exists yet. A saga doesn't hide that window, it makes it explicit and time-bounded, and gives you a defined path back to a consistent state if the last step doesn't complete. That's a fundamentally different consistency model than a database transaction (eventual, with defined compensations, instead of immediate and atomic), and pretending otherwise is where sagas go wrong in practice: teams build the happy path, skip writing the compensating actions because "that won't really happen," and then an incident forces someone to write the refund logic live, under pressure, for a case they never tested.

Each step publishing its "I'm done" or "I failed" signal is itself a dual-write problem at a smaller scale, which is why saga implementations lean on the outbox pattern internally for each step, rather than being a separate mechanism from it. The two patterns compose: outbox solves atomic local write-plus-event, sagas solve the multi-step orchestration built on top of that primitive.

What This Actually Demonstrates

None of this is about picking the "advanced" pattern to look sophisticated. It's about recognizing that "write to the database, then tell everyone else" is not one operation, it's two, and the honest response to that is either to make the second write ride inside the first one's transaction (Outbox) or to make the multi-step version of that gap explicit and recoverable (Sagas) — not to assume the gap won't matter until an incident proves otherwise.

That's the actual engineering problem behind SupraScan, the block explorer and indexing layer for the Supra network. Its job sounds simple: consume every block, parse every transaction, write structured data to PostgreSQL, with no missed blocks, no duplicate processing, and no gaps in the indexed history. The interesting part isn't any single piece of that sentence. It's that all of it has to happen continuously, at production throughput, forever, with the failure modes only showing up once you're running at real scale.

What "no pause button" actually costs you

Three constraints shape everything else about this system.

Throughput. Production now runs at 500–1,000 TPS. An earlier benchmark run measured 110-118 TPS sustained across 10 running instances, enough on its own to make naive designs fall over, and at that load Postgres itself sat at 50% CPU idle. The database wasn't the bottleneck. The RPC layer was, throwing socket-hangup errors and climbing past 600ms latency under the same load. That single number reframes the whole architecture problem: you're not primarily fighting the database, you're fighting how fast you can safely pull data out of the chain node and fan it out to workers.

Ordering, without the safety net most indexers get. Blocks have to land in strict sequence, block N fully processed before block N+1 is trusted. Most chain indexers also have to handle reorgs, the chain deciding retroactively that a block it already gave you doesn't count anymore, and rolling back whatever you'd already written. Supra has instant finality, so that entire class of complexity doesn't exist here. It's a real simplification, and it's worth naming, because it's easy to assume every blockchain indexer needs rollback logic. This one doesn't.

No central authority. Every worker instance is equal. Any worker can crash at any moment, mid-block, mid-write, without warning. Nothing about the design can assume a single instance stays alive to coordinate the rest. That constraint is what makes the rest of this interesting.

The shape of the system

Redis is the coordination layer: lock state, progress tracking, health signals. Postgres is the store of record. The blockchain's RPC endpoint is the source of truth for block data itself.

On startup, each pod goes through a fixed sequence: connect to the database, start the partition-management service (creates the next several days of table partitions automatically, ahead of when the data lands), run any pending one-time jobs, and start polling for the current chain height. Several of these services, and a few others, only run on one pod at a time. Leader election over a Redis heartbeat decides who does TPS calculation, fee calculation, and periodic stats. Every pod, leader or not, does the actual work: pulling blocks, indexing transactions, indexing wallets. The leader role exists to stop duplicate work on shared aggregates, not to gate the core pipeline behind a single instance.

Once running, the main loop is intentionally simple: check whether processing is paused, claim the next batch of block numbers, process them, log throughput, and loop again with almost no delay between iterations. The interesting decisions all live one level down, in how "process them" adapts to how far behind the system currently is.

A system that changes its own strategy under load

The indexer runs in one of two modes, and it switches automatically based on measured lag. When it's within a small number of blocks of the chain tip, it processes one block at a time: every block in the current range gets handled concurrently, one RPC call per block, full write pipeline per block, no batching overhead. Once it falls further behind than that, it switches into a batch mode, pulling a larger group of blocks at once and firing all their RPC calls concurrently, trading a bit of per-block latency for a lot more throughput. If a batch RPC call fails, it doesn't just retry the batch. After a couple of failed attempts it falls back to processing that range one block at a time, trading speed for reliability once concurrency itself looks like the problem.

This is a small design decision that pays for itself constantly: a system that's usually keeping up gets the latency profile of single-block processing, and a system that's falling behind (a slow node, an RPC hiccup, a burst of transaction volume) automatically shifts into a mode built for catching up, without anyone paging on-call to flip a setting.

One transaction, a dozen tables

Here's where the fan-out really shows up. A single indexed transaction isn't one row. Inside the write path, one transaction batch triggers somewhere between 15 and 20 sequential database operations: the core transaction record, an "advanced information" record (payload detail), deletion of any stale associated data from a prior attempt, event records, sender/receiver/fee-payer records, then separate tables for coin transfers, fungible asset transfers, NFT transfers, and automation registration, cancellation, execution, and gas-assessment records where relevant. It's a full relational decomposition of one on-chain event, not a JSON blob dropped into a single column.

That decomposition is what makes the system queryable at all, but it's also where a genuinely subtle constraint shows up. Senders, receivers, and fee-payer tables carry database triggers that decrement a wallet's transaction count on delete, and the code that manages this is explicit about why it can't be parallelized: bulk-deleting across multiple transactions concurrently causes concurrent trigger executions on the same wallet row, which either deadlocks or silently miscounts. A comment in the transaction repository says it plainly: deletions happen one at a time specifically to avoid that. It's not an oversight or unfinished optimization. It's a constraint the team hit, understood, and left documented in the code, because the "obvious" faster version is the one that corrupts data under load.

Failures get quarantined, not swallowed

Any block that fails processing three times doesn't get retried forever inline. It drops into a dead-letter topic, out of the main pipeline's way, and a separate reprocessing service picks it back up in chunks. That service is deliberately run on a single instance in production, not because it wouldn't be nice to parallelize, but because its own claim function has no row-locking mechanism to prevent two pods from grabbing the same message. Rather than build that locking immediately, the honest tradeoff was made to run it single-instance and accept the smaller throughput ceiling on the recovery path, which handles a small fraction of total volume, in exchange for correctness without added complexity.

This is the same instinct that shows up in the leader-election pattern for aggregate services: constrain concurrency exactly where correctness demands it, and leave everything else free to scale horizontally.

A concurrency bug the design didn't originally account for, and a different one still open

The main loop used to claim its next batch of blocks by writing a control key to Redis, then reading it back on the next iteration to know where to pick up. That write-then-read pattern looks harmless in isolation. Under real multi-pod load during a catch-up run, it wasn't. A measured benchmark across 10 pods under lag showed a 4.01x duplication ratio: pods were collectively doing four times the useful work in raw completions, because the shared coordination key was being read after a multi-second processing window had already passed, not claimed atomically at the start of it.

It's worth being precise about what this bug did and didn't do. It didn't corrupt data. Every write in the pipeline is idempotent by design, so overlapping work from two pods processing the same range produced the same end state, not conflicting ones. What it cost was infrastructure efficiency: real compute and real RPC calls spent on collisions instead of throughput, at exactly the moments, catch-up under lag, when throughput matters most. It's also specifically a multi-pod problem: a live check of production mainnet's actual deployment configuration confirmed it currently runs as a single instance, which means single-instance coordination didn't strictly need this fix to be correct in that specific environment. The fix shipped anyway, because any environment or future scale-out that does run multiple instances would otherwise inherit the race immediately: the write-then-read claim was replaced with a single atomic INCRBY-based claim, so two pods reading and writing at once now get distinct, non-overlapping ranges by construction instead of racing through a window between two separate calls.

A different, smaller-scope version of the same class of bug is still open, not fixed. The leader-election check that decides which single pod runs the aggregate services, TPS calculation, fee calculation, periodic stats, uses a check-then-set pattern rather than a single atomic operation, and that carries its own theoretical race during the moment two pods could both see no current leader and both try to claim the role. A proposed fix exists, replacing it with a single atomic conditional set, but it hasn't shipped. It's a smaller blast radius than the block-range race was, duplicate leadership would mean redundant aggregate computation, not duplicate block processing, but it's the same category of gap: understood, scoped, and honestly still sitting there unfixed.

It's also worth naming an alternative that was tried at the transport layer and abandoned, not just tuned. Block delivery in this system doesn't run through a message-streaming platform; an earlier version of the architecture did use one, and it was removed in favor of a simpler, database-backed work queue paired with the same Redis coordination layer described above. That's a real, lived example of the opposite failure mode from the one usually worried about in architecture discussions, not under-engineering, but a genuine willingness to walk back a heavier piece of infrastructure once a simpler one covered the same need with less to operate.

What this actually enables

The dead-letter and reprocess design isn't just a safety net for today's failure modes. It's the same mechanism that absorbed a real production incident when the chain started emitting a new transaction type the indexer's release branch hadn't been updated to handle safely, crashing on unguarded property access exactly on that reprocessing path. Because the recovery path already existed, the fix was a scoped hotfix, not an emergency rebuild.

Having closed the multi-pod race mattered for a specific reason beyond the efficiency it recovered at the time. Supra's MultiVM work means blocks will start carrying transactions from multiple virtual machines simultaneously, which increases both volume and per-transaction complexity. A coordination layer that quietly wasted 2-4x its capacity to collisions would have been a much worse foundation for that next phase than one that doesn't. Fixing it ahead of that was directly in service of scaling for what's coming, not abstract cleanup, and the still-open leader-election race is worth closing for the same forward-looking reason, before a busier, multi-VM future makes any coordination gap more expensive to leave sitting.

What this demonstrates

None of the individual pieces here are exotic. Leader election, adaptive batching, dead-letter queues, and idempotent writes are all known patterns. What's harder to fake is the discipline underneath them: a system that adjusts its own throughput strategy based on measured lag instead of a fixed setting, a data model that decomposes correctly under a documented, hard-won concurrency constraint, and a team that can point to a real, unfixed inefficiency in its own coordination layer and explain exactly why it hasn't caused a correctness problem yet. Production systems that run continuously, with no maintenance window and no single point of coordination, get built by making these tradeoffs explicitly and writing down the ones you haven't closed yet, not by pretending you've already closed all of them.

Every team hits the same moment. Search gets slow, someone says "we need Elasticsearch," and two weeks later there's a new cluster, a sync pipeline, and a Slack channel called #search-is-down.

Here's the case for not doing that. PostgreSQL's built-in full-text search handles the kind of workload many teams actually have, and it does it without adding a second database to the architecture.

The real cost isn't the cluster, it's the copy

Adding Elasticsearch doesn't add a search feature. It adds a second brain that has to keep agreeing with your first one.

Your data lives in Postgres. Now it also has to live in Elasticsearch, duplicated, reshaped into documents, kept current through a queue or a change-data-capture pipeline or a reindex job someone wrote in a hurry two years ago. Every insert, update, and delete in Postgres needs a matching write on the other side. That sync layer is where the real cost lives, not in cluster fees:

• A customer updates their email. Postgres has the new one. Elasticsearch still returns the old one for six hours because the CDC consumer fell behind.
• A row gets deleted in Postgres. The reindex job crashed last Tuesday, so it still shows up in search, and support gets a ticket about a ghost customer.
• Someone runs a backfill migration, forgets the matching backfill in Elasticsearch, and search results quietly diverge from the database for a month before anyone notices.

None of these are Elasticsearch bugs. They're the tax you pay for keeping two representations of the same data that don't share a transaction boundary. Postgres full-text search skips this entirely, because there's nothing to sync. The searchable representation lives in the same database and is maintained transactionally with the data.

What Postgres actually gives you: tsvector and tsquery

Full-text search in Postgres is built on two pieces: tsvector, which turns text into a normalized, searchable format, and tsquery, which turns a search string into something you can match against it.

SELECT to_tsvector('english', 'PostgreSQL indexes make searches fast');
-- 'fast':5 'index':2 'make':3 'postgresql':1 'search':4

to_tsvector normalizes the text, removes stopwords according to the text-search configuration (here "make" survives, it isn't one), and applies stemming where the configured dictionary supports it. That's why a search for search can match text containing searching, the same behavior that makes Elasticsearch feel necessary in the first place.

SELECT to_tsvector('english', 'PostgreSQL indexes make searches fast')
       @@ to_tsquery('english', 'search & index');
-- true

The @@ operator matches a tsvector against a tsquery. That's full-text matching as a native Postgres operator, not a separate search service. It doesn't score relevance on its own, that's what ts_rank is for below.

For a real search box, to_tsquery is less useful than it looks, because it throws a syntax error on ordinary user input like an unbalanced quote or a trailing &. websearch_to_tsquery is the better default: it accepts web-search-style syntax (quoted phrases, -exclude, or) and never rejects plain text.

SELECT title, ts_rank(search_vector, query) AS rank
FROM articles, websearch_to_tsquery('english', $1) query
WHERE search_vector @@ query
ORDER BY rank DESC
LIMIT 10;

Reach for to_tsquery when you control the query syntax yourself. Reach for websearch_to_tsquery when the query comes from a search box.

The part that makes it production-ready: GIN indexes

If you compute the tsvector during every query, Postgres has to process the candidate rows to build those vectors on the fly, fine for a thousand rows, a real cost at ten million. For data that's searched regularly, storing the vector and indexing it with a GIN index (Generalized Inverted Index) avoids that repeated work.

ALTER TABLE articles ADD COLUMN search_vector tsvector
  GENERATED ALWAYS AS (to_tsvector('english', title || ' ' || body)) STORED;

CREATE INDEX articles_search_idx ON articles USING GIN (search_vector);

That GENERATED ALWAYS AS ... STORED column is the detail that matters in production. Postgres computes it whenever the underlying row is inserted or updated, and the GIN index is maintained as part of the same database transaction, the same way any index is maintained when its underlying column changes. There's no separate consumer to fall behind and no second system that can temporarily disagree with the row.

A GIN index works like the index at the back of a textbook. Instead of scanning every page for the word "vacuum," you jump straight to the pages listed under V. Postgres does the same: instead of scanning every row's tsvector, it jumps straight to the rows containing your search terms.

Worth saying plainly: that index isn't free. It costs write overhead on every insert and update, and it costs disk, same as any index does. The difference isn't zero cost versus some cost, it's an index cost you were always going to pay somewhere versus that same index cost plus a whole second system to keep in sync. One cost, not two.

ts_rank scores matches by relevance, giving you the ranking step you'd otherwise reach for a search engine to provide. The query runs inside Postgres, alongside the rest of your relational data, without a network hop to a separate search cluster.

Filtering is where a bolted-on search engine shows its seams

Most real search features aren't just "find text," they're "find text, then filter by category, status, or whatever's live right now." If Elasticsearch is populated asynchronously from Postgres, that filter is only as current as the index. A row can go inactive in Postgres while the search index still considers it active, for however long the sync pipeline lags. In Postgres, the filter and the full-text predicate run against the same transactional dataset in the same WHERE clause. There's no separate search index that can lag behind the row, because the filter and the full-text predicate operate on the same Postgres data.

That single detail, an asynchronously maintained index lagging the source of truth, is usually the thing that quietly breaks in production long after the Elasticsearch launch party is over.

Where Elasticsearch still earns its keep

This isn't "Elasticsearch is bad." It's "most teams adopted it for a problem they didn't have."

Elasticsearch is worth reaching for when you need fuzzy matching and aggressive autocomplete beyond what pg_trgm comfortably covers, sophisticated relevance tuning, large-scale aggregations and faceting, distributed search across genuinely large datasets, log and observability workloads, or a search workload you deliberately want isolated from your transactional database. Any one of those is a real reason. None of them are "search felt slow once."

And Postgres full-text search isn't a drop-in replacement for all of that. If your product depends on typo tolerance, heavy autocomplete, deep linguistic analysis, or search-specific aggregations at real scale, the tradeoff changes quickly, and that's exactly when Elasticsearch's cost starts paying for itself.

Postgres does cover more of that ground than people expect, though, particularly the typo tolerance part. pg_trgm handles a different class of problem from full-text search: finding strings that are similar even when the spelling isn't exact, which is what powers a lot of "did you mean" and fuzzy-match behavior.

CREATE EXTENSION IF NOT EXISTS pg_trgm;

CREATE INDEX articles_title_trgm_idx
  ON articles USING GIN (title gin_trgm_ops);

SELECT title
FROM articles
WHERE title % 'elastcsearch'
ORDER BY similarity(title, 'elastcsearch') DESC
LIMIT 10;

Full-text search matches words and lexemes. pg_trgm matches approximate string similarity. Elasticsearch earns its place when you need substantially more search infrastructure than either of those provides.

The actual decision

This was never really "Postgres vs. Elasticsearch." It's "don't add a second system until you've established that Postgres isn't enough."

Start with Postgres. Add full-text search. Add a GIN index. Add pg_trgm if you need fuzzy matching. Measure. Introduce Elasticsearch when the workload actually demands it, not when search first feels slow.

For many teams, that measurement never gets there. The search bar was never the hard part. Keeping two systems telling the same story was.

A service that used to deploy in ninety seconds now takes six minutes. Nobody changed the code that day. The base image hasn't changed, docker push is uploading to the same registry it always has, and the CI runner is the same size it always has been. What changed is that the image itself quietly grew, build after build, revision after revision, until one day the push step timed out and someone finally looked. Nothing broke on purpose. The image just never stopped getting heavier, and nobody was watching the number.

An Image Is a Stack of Layers, Not a Snapshot

A Docker image isn't one file, it's a stack of read-only layers, one per instruction in the Dockerfile that changes the filesystem: a RUN, a COPY, an ADD. Each layer is a diff against the one below it, and at runtime the container's filesystem is presented as the combination of all those layers through a union filesystem, with a thin writable layer on top for the container itself. The layers stay separate on disk; nothing flattens them into one file. Crucially, a layer never shrinks once written. If a RUN apt-get install layer downloads 400MB of packages and a later RUN apt-get clean layer deletes the package cache, the image doesn't get 400MB smaller. The delete happens in a new layer on top; the old layer with the 400MB still sits underneath it, and that layer remains part of the image that nodes need to have available. The deleted file disappears from the filesystem you actually see when the container runs, but the bytes are still physically present in the image and still contribute to its size. If a registry or node doesn't already have that layer, those bytes have to be transferred during a push or pull.

Think of it like a suitcase you never fully unpack between trips. You don't take everything out and repack from scratch, you just add what you need for this trip on top of what's already in there. Take something out and it's not gone, it's still in the suitcase, just buried under a note that says "ignore this." The suitcase only gets heavier. That's a Docker image with unpruned layers: every stale cache, every intermediate build tool, every "temporary" file that got rm'd in a later step is still physically present, just marked as superseded.

Why this shows up in production: a base image with a full OS toolchain (Ubuntu with build-essential, a full Node.js image instead of a smaller variant) starts you 500MB to 1GB heavier before your application code exists at all. Every dependency installed and not cleaned up in the same layer, every COPY . . that grabs node_modules, .git, and test fixtures because there's no .dockerignore, adds weight that never comes back off. None of it fails a build. It just makes every image bigger than the last one, silently, until push and pull times are the bottleneck. A rough breakdown of how an image gets to 3GB:

Base image                900 MB
Build dependencies        700 MB   (compilers, headers, dev packages)
node_modules               500 MB
Source and test files      200 MB
Deleted-but-retained data  700 MB  (old caches, files removed in a later layer)
------------------------------------
Total                     ~3.0 GB

No single layer looks unreasonable on its own. It's the accumulation across all of them, plus the ones that should have been discarded but structurally can't be, that adds up.

Why Bad Layer Ordering Makes Every Build Slower

Layer caching is supposed to make builds fast: Docker walks the Dockerfile instruction by instruction and reuses the cached result for each one, until it hits the first instruction whose cache is invalid, either because the instruction itself changed or because a file it depends on changed. From that instruction onward, every remaining layer has to be rebuilt, even if most of them don't actually depend on what changed. That's the entire point of the layer model, and it only works if the Dockerfile is ordered so that the things which change least often come first, and the things that change on every commit come last.

The single most common mistake is COPY . . before the dependency-install step:

# Bad: any file change invalidates the install layer
COPY . .
RUN npm install

Every commit touches some file in the repo, so that COPY layer's cache is invalidated on every build, and npm install right after it has to rerun from a cold cache every time, even when package.json didn't change. Reordering it fixes the problem directly:

# Better: only a change to package*.json invalidates the install layer
COPY package*.json ./
RUN npm ci
COPY . .

Now changing src/foo.js invalidates only the final COPY, and the dependency-install layer above it stays cached. A build that might otherwise take fifteen seconds because nothing dependency-related moved instead can take three minutes on the bad version, on every revision, forever, because the Dockerfile put the wrong instruction first.

Why this shows up in production: this is the mechanism behind "deploys used to be fast and now they're not," with no code change to point at. It's not the application getting slower, it's the build losing its cache on every run because of layer ordering, on top of an image that's also grown heavier over time, so both the build step and the push/pull step degrade independently and get blamed on each other.

The Registry Doesn't Forget Either

The same "nothing shrinks unless someone tells it to" problem exists one level up, in the artifact registry itself. Every build that pushes a new tag, latest, a commit SHA, a version number, adds a new manifest and a new set of layer blobs to the registry. Layers that are byte-identical get deduplicated by content hash, which is why the registry doesn't grow linearly with every push, but every layer that's actually different (a new dependency version, a new base image patch, a rebuilt application layer) is new data that has to be stored, and old images don't get deleted just because a newer one exists.

A registry doesn't necessarily know which images are no longer useful. Unless you give it retention rules or clean them up yourself, old manifests and their unique layer blobs just accumulate: every feature-branch build, every hotfix, every image built by a CI run that never got cleaned up, all still sitting there, still counted against storage. A registry with a two-year-old image nobody has pulled in eighteen months isn't just wasted storage, it's an image nobody is auditing for CVEs anymore, sitting right next to the one that's actually in production, indistinguishable to anyone browsing tags without a naming convention.

Why this shows up in production: registry storage bills climbing with no obvious cause, and CI jobs or registry maintenance operations getting slower or more expensive as a repository accumulates large numbers of manifests and unique layer blobs. A single docker pull for a specific tag isn't slowed down by unrelated stale tags sitting elsewhere in the repository, it only fetches the layers that tag's manifest actually references, but repository-wide operations, listing tags, running garbage collection, scanning for vulnerabilities across everything stored, all get heavier as the pile grows. The fix isn't a bigger registry plan, it's a retention policy: expire untagged manifests and feature-branch tags after N days, keep only the last M builds per branch, and let content-addressable deduplication do the rest.

Measure Before You Guess

Before reordering a Dockerfile or swapping a base image, find out what's actually taking up space. Two commands answer that directly:

docker image history myapp:latest

Lists every layer in the image with its size, in order, so you can see exactly which instruction added the most weight.

docker image inspect myapp:latest

Gives the full layer manifest and metadata, useful for scripting size checks in CI. For a closer look at what's actually sitting inside the largest layers, a tool like dive walks the filesystem contents layer by layer, which is usually faster than guessing from the Dockerfile alone. Run docker image history first: it takes thirty seconds and tells you whether the problem is the base image, an unpruned dependency cache, or the build context, before you change anything.

What Actually Fixes It

• Order the Dockerfile by change frequency. Dependency manifests (package.json, requirements.txt) get copied and installed first, source code gets copied last. That one reordering is usually the single biggest build-time win available.
• Use multi-stage builds. Build in one stage with the full toolchain, compilers, dev dependencies, and copy only the compiled output into a clean final stage. The build tools never make it into the image that ships. A simplified Node example, for a service that produces a self-contained dist directory:

FROM node:22 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:22-slim
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/package*.json ./
RUN npm ci --omit=dev
CMD ["node", "dist/index.js"]

The final image only contains what COPY --from=build explicitly pulls forward: the compiled output and production dependencies. The compiler, dev dependencies, and source files from the build stage never exist in the shipped image at all.

• Pick a smaller runtime image where it makes sense. -slim, distroless, or Alpine variants are often the difference between a 900MB image and a 90MB one before any application code is added, but they're not a free win in every case: Alpine's musl libc can surface compatibility issues with some native dependencies, so weigh the size/security gain against your actual runtime's tolerance for it rather than defaulting to it blindly.
• Write a real .dockerignore. node_modules, .git, test fixtures, and local env files should never be in the build context in the first place. This keeps the image smaller directly, and it also keeps irrelevant files from ever reaching a COPY layer and invalidating its cache for no reason.
• Set a registry retention policy. Expire untagged and stale feature-branch images automatically instead of relying on someone remembering to clean up. Most registries (ECR, Artifact Registry, GitHub Container Registry, Harbor) support this natively.

None of these are exotic. They're three separate mechanisms with the same operational lesson: unused data doesn't disappear unless you design for it or clean it up on purpose. A service that deploys in six minutes instead of ninety seconds usually isn't a mystery. It's a suitcase that's been repacked on top of itself for a year, and nobody's taken anything out.

A write-heavy ingestion table starts slow on Postgres. Not immediately, but a few weeks in: inserts that used to take a millisecond now take ten, autovacuum is perpetually behind, and EXPLAIN ANALYZE shows time going into index maintenance nobody remembers configuring. The instinct is to blame the schema, or the hardware, or "Postgres doesn't scale." None of those are quite right. The real answer is a decision Postgres made in the 1990s, long before this table existed: it keeps its indexes as B-Trees, and every one of them has to stay sorted, on every write.

Every Write Has to Land Somewhere

Strip a database down to its storage engine and the job is always the same: take a write, put it on disk in a shape that makes future reads fast, and don't lose it. There are two dominant answers to how to do that, and most databases you've heard of lean on one.

B-Trees keep some part of the data sorted on disk, in place, at all times. MySQL's InnoDB and SQLite go all the way: the table itself is a B-Tree, keyed by its primary key or rowid, so the table and its main index are the same structure (a "clustered index"). Postgres is more layered: the table (the heap) is an unordered file with no sort order of its own, rows just go wherever the free space map says there's room, but every index on that table, including the primary key, is a separate B-Tree that has to stay sorted and has to be updated on every write that touches an indexed column.

LSM-Trees, log-structured merge trees (Cassandra, RocksDB, LevelDB, and CockroachDB's Pebble storage engine), take the opposite approach everywhere: a write never touches its final sorted position immediately. It gets appended to whatever's currently open, and getting everything back into sorted order is a job for later, done in the background, in bulk.

That fork, sorted-in-place versus sorted-later, explains almost every practical difference in how these two families of databases behave under load.

B-Trees: Pay at Write Time, Save at Read Time

Think of each B-Tree index as a filing cabinet that's always perfectly alphabetized. Every entry goes directly into its correct folder, in its correct position, the moment it arrives. Finding anything later is fast and predictable, you walk straight to the folder. But filing it correctly in the first place means locating the right spot, possibly shifting other entries out of the way, and writing to a specific place on disk rather than just the next free spot.

Here's what a Postgres insert actually does. The row itself is appended to the heap, close to a sequential write, wherever the free space map finds room. But every index on that table, the primary key, any unique constraint, any column you've indexed for lookups, is a B-Tree, and each one needs a new entry written to its correct leaf page, wherever that page happens to sit on disk. Two indexes on a table means one heap append plus two separate random writes, every single insert.

Updates are the more expensive case, and this is where MVCC changes the arithmetic. Postgres never overwrites a row in place: an UPDATE marks the old row version dead and writes an entirely new version elsewhere in the heap. That means an UPDATE costs everything an INSERT costs, a new heap entry, plus a new entry in every index, plus it leaves a dead tuple behind that autovacuum eventually has to clean up.

Why this shows up in production: early in a table's life, its hot pages and index pages fit comfortably in shared_buffers, so those "random" writes are really just writes to RAM, flushed to disk lazily and cheaply. Once the table and its indexes outgrow memory, every insert or update has a real chance of touching an index page that isn't cached, turning a cheap write into a disk seek, on every indexed column, on every write. Add autovacuum needing to revisit the dead tuples MVCC leaves scattered across heap pages, and you get the exact symptom this post opened with: a table that used to be fast getting steadily slower with no code change, just growth, and just more indexes to maintain per write.

LSM-Trees: Write First, Sort Later

An LSM-Tree handles the same problem by refusing to do that work up front. Picture an inbox tray instead of a filing cabinet: every new entry just gets dropped on top of the pile. Filing is instant, because there's no filing, you're not finding anything, you're just adding to a stack. Periodically, someone takes the whole pile, sorts it properly, and merges it into the already-sorted sections below it.

Mechanically: writes go into an in-memory structure (a memtable) and are also logged to a commit log for durability, both sequential operations. When the memtable fills up, it's flushed to disk as an immutable sorted file (an SSTable). Over time you accumulate many of these sorted files, and a background process called compaction merges them together, discarding values that have been overwritten or deleted along the way.

Deletes are handled the same indirect way updates are avoided: a delete doesn't remove anything immediately, it writes a tombstone, a marker saying "this key is deleted as of this timestamp." The actual old data only disappears once compaction physically rewrites the SSTables that contained it.

Why this shows up in production: writes are cheap and consistently fast, because they're always sequential appends, regardless of how big the dataset gets or where the "right place" for the data would eventually be. That's why Cassandra-style databases are the default reach for write-heavy workloads: time-series ingestion, event logs, metrics pipelines. The cost is deferred, not eliminated. A read might have to check the memtable and several SSTables before it can be sure it has the latest version of a key, that's read amplification, and it's the mirror image of the index-maintenance cost a B-Tree pays on every write. Compaction itself is a real, ongoing background cost too: it consumes disk I/O and CPU that has to be budgeted for, and if compaction ever falls behind sustained write load, both read latency and disk usage climb, in a way that looks a lot like Postgres autovacuum falling behind, just triggered by the opposite kind of pressure.

"Just Switch to Cassandra" Isn't the Fix It Sounds Like

It's tempting to read the above and conclude LSM-Trees are strictly better for anything write-heavy. They're not, they're differently shaped, with their own failure modes:

• Read amplification is real. A point lookup that's a single B-Tree traversal in Postgres might mean checking a memtable plus several SSTables in an LSM-Tree, unless bloom filters and careful compaction keep that bounded.
• Compaction is not free. It's extra write I/O (rewriting data that was already written once), and a compaction strategy mismatched to the workload (size-tiered vs. leveled) can make total write cost in an LSM-Tree worse than a B-Tree's, not better.
• Tombstones linger. Until compaction actually removes the old versions and their tombstones, deleted data still occupies space and still gets scanned past on reads, and in Cassandra specifically, an excess of tombstones scanned in a single read can trip query-side warnings or failures.
• Transactional guarantees differ. Postgres gives you multi-row ACID transactions and foreign keys as first-class features because the B-Tree indexes, MVCC, and WAL were co-designed for exactly that. Most LSM-based systems trade some of that away for horizontal write scale, which is a real cost if your workload actually needs cross-row consistency, not just fast ingestion.

Migrating a workload from Postgres to Cassandra to solve a write-scaling problem, without checking whether the workload also needs the guarantees Postgres was giving up in the trade, is how you end up rebuilding transactional consistency by hand in application code, which is a worse position than the one you started in.

When Each Model Actually Wins

The honest framework isn't "which database is better," it's "which cost am I willing to pay":

• Choose a B-Tree-indexed engine (stay on Postgres, tune it) when reads dominate, or when transactional consistency and relational integrity across tables matter more than raw write throughput. Most application backends, most OLTP workloads, most systems where "this row is correct right now" matters, belong here.
• Choose an LSM-Tree engine when the workload is write-dominated and append-heavy by nature, event streams, time-series metrics, logs, and reads can tolerate either eventual consistency or a slightly higher read cost in exchange for writes that don't degrade as the dataset grows.
• Or, more often than either extreme: the write-heavy table causing the pain doesn't need to be in Postgres at all. Partitioning it, moving it to a purpose-built time-series or log store, or reducing the number of indexes it maintains, frequently solves the actual problem without touching the primary transactional database's engine at all.

Postgres isn't losing to Cassandra when a table gets slow under heavy writes. It's honoring the trade it made on day one: keep every index sorted and correct on every write, so reads stay cheap and consistent forever. Whether that's still the right trade for a specific table, and how many indexes it really needs, is a workload question, not a verdict on the database.

Spend all day at a fish market and you'll leave smelling like fish, even if you never touched one.

Spend your day in a codebase full of shortcuts, and you'll start writing shortcuts too, even if nobody told you to and even if you swore you never would.

That's how environments work. They rub off on you, whether you notice it or not. And in engineering, this isn't a metaphor it shows up in your git log.

The lie of "it's all about discipline"

Most engineers think their habits come from training, courses, or sheer effort. If your code is sloppy, the assumption is you didn't try hard enough. If it's clean, you must have "good discipline."

That's only half true. A huge part of how you code comes from what surrounds you every day, on repeat, long before you're conscious of it:

• The PRs you review, and the ones that get rubber-stamped instead of actually reviewed.
• The Slack threads where someone asks "does this really need a test?" and the answer is a shrug.
• The postmortems you sit through, the ones that hunt for a person to blame versus the ones that hunt for a broken process.
• The teammate whose code you copy-paste from at 2am because it's the closest example you can find, bugs and all.

None of this announces itself as a lesson. Nobody sends a Slack message saying "today I am teaching you to skip error handling." It just seeps in through repetition, the same way an accent seeps in from the people you grow up around.

Where this actually shows up in production

This isn't abstract. It shows up as specific, recognizable symptoms:

Silent try/catch blocks everywhere. Not because anyone taught "swallow your errors," but because the first three examples of error handling you saw in that codebase did exactly that, and you pattern-matched off them without questioning it.

A team where nobody writes tests for "obvious" changes. Six months later, someone ships an "obvious" one-liner that takes down checkout for twenty minutes. The postmortem calls it human error. It was actually a cultural default nobody chose on purpose, it was absorbed.

Incident channels full of finger-pointing instead of timelines. New hires learn within a week that admitting a mistake early gets you blamed, so they learn to sit on problems until they're unavoidable. That delay is the expensive part, not the original mistake.

Commit messages that are just "fix," "update," "wip." You'll find the one senior engineer on the team who started that pattern years ago, still committing "fix" today, and everyone quietly inherited it.

Compare that to a team where the senior engineers write commit messages that explain why, not just what; where someone flags an edge case out loud in standup instead of letting it slide; where an incident review ends with "here's the missing guardrail" instead of "here's who missed it." Sit in that environment for six months and you'll start doing the same things, not because you decided to, but because that's what normal looks like now.

Why "just work harder" doesn't fix this

If your habits were purely a function of effort, then two equally hardworking engineers in different environments should converge on similar code quality. They don't. Drop the same disciplined engineer into a team that treats testing as optional, and within a year their testing habits will have eroded, not because they got lazy, but because the environment stopped rewarding the behavior and stopped modelling it.

This is also why "hire good people and leave them alone" doesn't scale culture. Good habits decay under a bad environment faster than bad habits improve under a good one is slow. Osmosis runs both directions, and it runs constantly, not just during onboarding.

Changing what you're surrounded by is the actual lever

This is why the fastest way to level up often isn't another course or another book, it's deliberately changing what you're marinating in every day:

• Review code from engineers better than you, even when it's not required. You're not looking for bugs; you're absorbing their defaults.
• Sit in on incident calls outside your own team. Watching how a genuinely good incident is run, calm, timestamped, blameless, recalibrates what "normal" looks like faster than any postmortem template.
• Follow people who write and talk publicly about the specific problems you want to get better at. Your feed is an environment too. It rubs off exactly like your team does.
• Choose the side projects and open-source codebases you spend your evenings in as carefully as the one you're paid to work in. If every codebase you touch after 6pm is a mess, don't be surprised when "mess" starts to feel acceptable at 10am.

Your first team teaches you more about "how to engineer" than any style guide, wiki page, or onboarding doc ever will, whether that team is excellent or quietly broken. And it keeps teaching you, every day you stay in it, long after onboarding ends.

The real question

The scent of your environment always follows you into your next commit.

So the question worth asking isn't "am I working hard enough?"

It's "what am I letting rub off on me, and is it actually who I want to become as an engineer?"

That's fine when your data is sitting in memory. It becomes a real problem on modern NVMe storage. An NVMe drive can have 32 or more I/O requests in flight at once, happily serving them in parallel. Postgres, until version 18, never took advantage of that. It asked for one page, waited for it to come back, then asked for the next. The hardware was capable of a highway; Postgres was driving it one car at a time.

PostgreSQL 18 changes that with a real asynchronous I/O subsystem. Here's the mental model worth keeping: synchronous I/O is a single waiter who takes one table's order, walks it to the kitchen, stands there until it's ready, and only then goes to take the next order. Async I/O is that same waiter dropping ten orders at the kitchen window at once and picking up whichever plates are ready as they come up. Same kitchen, same waiter, dramatically less standing around.

What actually changed

Postgres now has a pluggable I/O backend, controlled by a new io_method setting:

io_method = io_uring   # Linux 5.1+, lowest overhead
io_method = worker      # cross-platform fallback (macOS, Windows, older Linux)
io_method = sync        # pre-PG18 behavior

io_uring is the interesting one. It's a Linux kernel interface that gives Postgres two ring buffers shared with the kernel: one to submit I/O requests, one to collect completions. Submitting a batch of reads costs a single syscall instead of one syscall per page. On worker, a pool of background threads makes the blocking calls on your process's behalf so your backend never has to sit and wait itself.

The parameter that actually controls the payoff for read-heavy workloads is effective_io_concurrency. Before PG18 this only affected bitmap heap scan prefetching. Now it controls how many pages ahead a sequential scan, a VACUUM, or a checkpoint will request before it actually needs them:

effective_io_concurrency = 16    # SSD: 16-64
effective_io_concurrency = 200   # NVMe: 64-256
effective_io_concurrency = 2      # HDD: seeks dominate, async barely helps

Why this matters in production

The scenarios where this pays off are the ones that were always I/O-bound: a nightly ETL job doing a full sequential scan of a fact table, VACUUM chewing through a heavily-updated table whose pages aren't in shared_buffers, a checkpoint flushing a large batch of dirty pages, a replication standby trying to flush WAL fast enough to keep lag down.

On paper, a sequential scan of a 10GB table doing 1.3 million page reads at roughly 50 microseconds each costs about 65 seconds fully synchronous. With requests batched 32-deep, that drops toward single-digit seconds in theory. Real workloads don't hit the theoretical ceiling (there's coordination overhead, and the OS page cache intervenes), but a 2-3x throughput improvement on genuinely I/O-bound scans is a realistic, repeatable result, not a marketing number.

The part that matters just as much: if your working set already fits in shared_buffers or the OS page cache, async I/O buys you almost nothing. A cache hit has no I/O latency to overlap in the first place. This isn't a "just turn it on and everything gets faster" feature. It's a fix for exactly one bottleneck: waiting on disk.

Verifying it on your own workload

PostgreSQL 18 also ships pg_stat_io, the first built-in view that breaks down I/O by process type and operation. This is how you check whether async I/O is actually doing anything for you, instead of taking it on faith:

SELECT pg_stat_reset_shared('io');

-- run your actual workload here

SELECT
  backend_type,
  reads,
  read_time,
  ROUND(reads::numeric / NULLIF(read_time, 0) * 1000, 0) AS reads_per_second
FROM pg_stat_io
WHERE reads > 1000
ORDER BY reads DESC;

Run that once with effective_io_concurrency = 1 and once with it set to something like 64, on the same cold query, and compare read_time. If the number barely moves, your data was already cached and async I/O was never going to help. If it drops meaningfully, that's your workload confirming it was genuinely I/O-bound and the new prefetching is doing real work.

What it doesn't change

It's worth being precise about the boundaries. Async I/O doesn't touch MVCC, doesn't change the durability guarantees around WAL flushing, doesn't change lock acquisition, and doesn't reduce the per-connection process overhead that still makes connection pooling mandatory at scale. It's a storage-layer optimization, full stop. Everything above the buffer manager works exactly as it did before.

PostgreSQL isn't getting a new execution model here. It's getting permission to stop waiting in line.

Let's ground this in one real example and stick with it the whole way through: you're building a "find similar support tickets" feature. A customer submits a new ticket, and you want to show them five old tickets that are about the same problem, even if they used totally different words to describe it.

To do that, you can't search for matching keywords, because "my payment failed" and "checkout won't go through" mean the same thing but share zero words. So instead, every ticket gets converted into a list of a few hundred (or a few thousand) numbers, called an embedding, using an AI model. Two tickets about similar problems end up with number-lists that are mathematically close to each other. Two tickets about unrelated problems end up far apart.

That list of numbers is what Postgres calls a vector. "Vector search" just means: given one ticket's list of numbers, find the other tickets whose lists of numbers are closest to it. pgvector is the Postgres extension that lets you store these number-lists in a normal table column and search through them with plain SQL.

The Part Every Tutorial Skips

Every "add AI to your app" tutorial has the same steps: install pgvector, add a vector column, run one search query, done. And for a demo with a thousand tickets, that's genuinely all it takes. It just works.

Here's the part tutorials skip: once your support-ticket table grows to a few hundred thousand rows, that same search query, which used to take 8 milliseconds, now takes 900 milliseconds. Someone on your team says "just add an index," like that fixes it the way adding an index fixes a normal slow query. It doesn't, not in the same simple way. Picking the right kind of index means giving up a small amount of accuracy in exchange for speed, and how much accuracy you give up is the whole subject of this post.

Why a Normal Index Doesn't Work Here

A normal Postgres index (a B-tree) is built for questions like "give me every ticket where status = 'open'." That's an exact match: a row either satisfies it or it doesn't.

A vector search question is different in kind: "of these 2 million tickets, which five have number-lists closest to this new ticket's number-list?" There's no exact match to look up. Every single row has to have its distance calculated and compared.

Postgres can do this the honest way: calculate the distance from your new ticket to every other ticket in the table, one by one, then sort and keep the closest five. This is called an exact search, and it's exactly what your thousand-row demo was quietly doing. At a thousand rows, checking every single one is instant, so you never noticed.

Why this matters in production: the time this takes grows in a straight line with the number of rows. Double your tickets, double the search time. Forever. There's no clever Postgres trick, no ANALYZE, no EXPLAIN setting that fixes this, because the problem isn't that Postgres is being inefficient. It's that checking every row really is the only way to get a guaranteed-correct answer. The only way to get faster is to stop insisting on a guaranteed-correct answer, and accept "almost certainly correct" instead.

Two Ways to Cheat (on Purpose): IVFFlat and HNSW

pgvector gives you two index types that both make this trade deliberately: they return an answer that's usually the true closest five tickets, but occasionally misses one, in exchange for being dramatically faster. This is called approximate nearest neighbor search, or ANN for short. Think of it like asking a knowledgeable local for the nearest coffee shop instead of checking every coffee shop in the city yourself: almost always right, occasionally not the actual single closest one, but you get an answer in two seconds instead of an hour.

The two options, IVFFlat and HNSW, cheat in different ways, and picking between them is the actual skill here.

IVFFlat: Sort Tickets Into Labeled Bins First

Picture a post office that pre-sorts mail into bins by zip code before a carrier ever looks at it. IVFFlat does the same thing to your tickets, before any search happens:

1. When you build the index, Postgres groups all your existing tickets into a fixed number of bins (you choose how many, say 100), based on which tickets' number-lists are similar to each other. Each bin gets a "center point" that represents everything in it, the same way a zip code represents a neighborhood.
2. When a new ticket comes in and you search, Postgres doesn't check all 100 bins. It only checks a handful of the bins whose center point is closest to your new ticket (you choose how many to check, say 10).
3. Inside just those 10 bins, it does the slow, exact, check-every-row search, which is now fast because there are far fewer rows to check.

-- build the index: sort existing tickets into 100 bins
CREATE INDEX ON support_tickets
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

-- when searching: only check the 10 closest bins
SET ivfflat.probes = 10;
SELECT id FROM support_tickets
ORDER BY embedding <=> '[0.1, 0.2, ...]'
LIMIT 5;

The catch nobody mentions in the quickstart: those bins are decided once, on the day you build the index, based on the tickets that existed that day. Six months later, your product has a whole new feature, and a wave of brand-new kinds of tickets start coming in that don't look like anything in your original bins. Postgres still has to put them somewhere, so it shoves each new ticket into whichever old bin is the "least bad fit," even if that bin isn't really a good match. Slowly, quietly, your search results get worse, and there's no error message telling you this is happening. The fix is to periodically rebuild the index (REINDEX) so the bins get redrawn using current data, which on a big table is a real maintenance job, not a background setting you flip once.

HNSW: Build a Map With Highways and Side Streets

HNSW works completely differently: instead of bins, it builds a connected map between all your tickets, like a road network, with some very long "highways" connecting far-apart regions and lots of short local roads connecting nearby tickets.

Think about how you'd actually travel from a small town to another small town on the far side of the country. You wouldn't drive the whole way on back roads. You'd take a short local road to a highway, drive the highway most of the way, then take local roads again at the other end. HNSW searches the exact same way: start on the "highway" layer where a few big jumps get you into the right general area fast, then drop down onto smaller, denser layers to fine-tune and land on the actual closest tickets.

-- build the index: construct the layered map, m = how many roads each point connects to
CREATE INDEX ON support_tickets
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- when searching: how wide an area to explore before settling on an answer
SET hnsw.ef_search = 40;

m is roughly "how many roads does each ticket connect to." More roads means better odds of finding the true closest tickets, but the map itself takes up more memory. ef_search is roughly "how much of the map do you explore before giving up and answering." Explore more, and you get a more accurate answer, but it takes longer.

Why this matters in production: because HNSW never sorts tickets into fixed bins, it doesn't go stale the way IVFFlat does. New kinds of tickets just get woven into the existing map naturally. But that map has real costs. Every single ticket stores a list of which other tickets it's "connected to" by road, and that list has to live somewhere, so memory use grows with how many roads each point has (m) and how many tickets you have, not just with how big your rows are. A table that used to fit comfortably in memory as plain rows can suddenly need a lot more RAM once this road-map is layered on top. Also, building this map isn't a quick one-time calculation like a normal index: Postgres has to insert each ticket into the map one at a time, figuring out its connections as it goes. On a table with tens of millions of tickets, that build can take hours, and depending on your Postgres version it can hold a lock that blocks other people from writing to that table the whole time. If tickets are constantly being added, that's a real scheduling problem you need to plan around, not something to discover mid-build.

Putting Both Side by Side

Neither one is simply "the better index." If your ticket table is built once and doesn't change character much over time, like a fixed product catalog, IVFFlat's cheap build and simple bins are the sensible default. If your table keeps growing and keeps having new, different kinds of data written to it all the time, like a live support-ticket stream, HNSW's steadier accuracy is usually worth its slower, more expensive build.

The One Question That Actually Matters

Before shipping either index, the important question isn't "which one is faster," because both are fast enough for almost any real app. The real question is: how often is it okay for this search to miss the actual best match, and have you ever checked?

That "miss rate" is called recall: the percentage of the true best matches your approximate index actually returns. It doesn't show up anywhere. It won't throw an error, and EXPLAIN ANALYZE won't flag it. The only way anyone finds out recall has gotten bad is a vague complaint months later, something like "the suggested tickets don't feel as relevant as they used to," long after the table has grown large enough for the approximation to start visibly slipping.

The fix is simple to say and easy to skip: while your table is still small enough to run a true exact search for comparison, run one, and compare its results against what your approximate index returns. That tells you your real recall number. Do this early, because once the table is big enough that an exact search is too slow to run anymore, you've lost your only way of checking whether the fast index is still giving you good answers.

500 requests a second are hitting a dependency that is completely dead. Every single one gets a clean timeout after 2 seconds, exactly as configured. And you are still down.

That sentence is the whole problem with treating resilience as a single setting. A timeout did its job, nothing hung forever, and the aggregate is still an outage, because you're holding 1,000 concurrent doomed requests at once, and every one of them is load on a dependency that's trying to restart.

There isn't one pattern that fixes this. There are three, stacked, each one picking up exactly where the last one's guarantee runs out. Get the order wrong, or skip one, and you don't get partial protection, you get a different failure mode wearing the same symptoms.

This is that stack: timeouts and retries, circuit breakers and bulkheads, backpressure and load shedding. What each one actually bounds, why the one before it isn't enough, and the specific way each gets misconfigured in a way that looks fine until the day it doesn't.

Layer One: A Timeout Doesn't Wait Patiently, It Fails Together

Most client libraries default to no overall timeout. fetch in Node has no total deadline unless you add one. Plenty of database drivers wait indefinitely. That default isn't neutral, it's a decision to couple your availability to your slowest dependency, and Little's Law explains exactly how much:

concurrency = arrival rate × service time

A dependency's p99 goes from 50ms to 30 seconds. Your arrival rate hasn't changed. Your concurrency just rose by a factor of 600, and every one of those in-flight requests is holding a socket, a pool connection, a request slot, memory. At 200 requests/second and a 30-second service time, you need 6,000 concurrent slots. You don't have 6,000.

downstream slows → your concurrency climbs → pool exhausted → queue builds
                 → YOUR p99 rises for every endpoint, including ones that
                   never call that dependency → your own callers time out
                 → and they retry, adding load → you are now the outage

Nothing failed. A dependency got slow, and the absence of a bound propagated it outward. A timeout is how you convert someone else's latency problem into your error rate, which sounds like a downgrade and isn't, because an error is bounded and recoverable, while unbounded latency spreads.

The queue at a single service desk, where one customer's transaction is taking forty minutes. With no policy, the twenty people behind them wait forty minutes, the next twenty leave, and the shop's throughput collapses over one difficult case. With a policy, "five minutes per customer, then take a ticket and come back", one customer is inconvenienced and the queue keeps moving.

Only one of your four timeouts actually bounds anything

A single number labelled "timeout" is usually not the number you think it is:

• Connect bounds the TCP handshake. Catches a host that's down or unroutable.
• Time to first byte bounds server thinking time. Catches a slow query or a saturated server.
• Idle / socket bounds the gap between bytes. Catches a stream that stalls mid-response.
• Total / overall bounds the whole operation, including retries. This is the only one that actually protects your resource usage.

A 2-second connect timeout and a 5-second read timeout don't add up to a guarantee, a response trickling one byte every 4 seconds never trips either one. Set the overall deadline, always, and treat the other three as diagnostics that let you fail earlier with a clearer reason why.

Choose the number from the p99.9 of the successful response distribution, not the average. Too short and you abandon requests that would have succeeded, converting them into retries, a load amplifier disguised as a safety measure. Too long and you hold resources through a failure you could have detected sooner.

And here's the arithmetic that catches nearly everyone:

Your caller's timeout:      5s
Your per-attempt timeout:   3s
Your retry policy:          3 attempts
Your worst-case total:      3 + backoff + 3 + backoff + 3 ≈ 10s

At t=5s your caller already gave up. Attempts 2 and 3 are work
for nobody, executed against a dependency that's already struggling.

A retry policy has to fit inside the overall budget: per-attempt timeout is remaining_budget / max_attempts, not a number picked independently.

Not every failure is safe to retry, and a status code isn't proof

Connection refused, DNS failure, 503, 502, safe to retry, nothing happened yet. A 429, safe, and honour Retry-After. A 500 or a timeout, safe only if the operation is idempotent, because both are the canonical ambiguous failure: the request may have already succeeded server-side and you just never heard back. A 400, 422, 404, never retry; identical bytes fail identically.

HTTP method semantics are a hint, not a guarantee. GET and PUT are specified idempotent, and plenty of real handlers aren't, a GET that increments a view counter, a PUT that appends. Decide from what the operation does and whether it carries an idempotency key, not from the verb.

Backoff needs jitter, and jitter isn't a refinement

Fixed-interval retries fail for a specific reason: a thousand clients that failed at the same moment retry at the same moment.

Fixed 1s:      ████    ████    ████     ← full fleet, in phase, forever

Exponential:   ████      ████        ████    ← spaced out, still in phase

Exp + jitter:  ▁▃▂▁▄▂▃▁▂▄▁▃▂▄▁▂▃▁▄▂▃  ← smeared into a manageable trickle

Exponential backoff fixes the frequency of retries. It does nothing for synchronisation. Without jitter, a recovering dependency gets knocked over by the first aligned burst, which restarts the whole cycle.

// Full jitter: sleep is uniform over [0, capped exponential]
const backoff = (attempt: number) => {
  const exp = Math.min(30_000, 200 * 2 ** attempt);   // cap matters, uncapped, attempt 12 waits 9 hours
  return Math.random() * exp;
};

Cap the exponential, bound the attempt count, and make the first retry near-immediate for a plain connection refusal, that failure costs the dependency nothing to answer again.

Retry amplification: the incident that outlives its own cause

Here's the part that turns a thirty-second blip into a two-hour outage.

Retries multiply through layers. A request path where every layer retries three times:

client ──3×──► gateway ──3×──► API ──3×──► service ──3×──► database
                                                  81 attempts at the bottom
                                                  for ONE user request

Each layer is individually reasonable. Together they're an 81× amplifier, and it engages exactly when the deepest component is failing, because that's what triggered the retries in the first place. A database at a 50% error rate, hit with three times its normal load, doesn't recover. It produces more errors. Which produce more retries. Which produce more errors.

This is a metastable failure state: one that sustains itself after its trigger is gone. A 30-second network blip causes a wave of retries. The retries push load above capacity. Being above capacity produces timeouts, which produce more retries. The network has been fine for an hour, and the system does not recover, because the load keeping it down is the load its own failure generated. Removing the original cause changes nothing. The only way out is reducing load: shedding traffic, opening circuit breakers, or manually pulling the service out of rotation until queues drain.

Two rules follow from this. Retry at one layer, not at every layer, pick the one closest to the business logic, the one that holds the idempotency key, and make every other layer pass failures straight through. If a service mesh retries for you, the application must not also retry, or you've silently rebuilt the multiplier.

And bound it structurally with a retry budget, this is the single most valuable thing in this entire layer, and the mechanism most systems lack:

// Allow retries only up to ~10% of successful request volume.
// Errors can spike without retry load spiking, that's the whole point.
if (retryTokens.tryConsume()) { /* attempt */ }   // token bucket, refilled by SUCCESSES

With a per-request retry count, a 100% error rate produces 3× or 81× load. With a 10% retry budget, that same 100% error rate produces 1.1× load, the system fails fast and cheap, and leaves the dependency enough headroom to actually recover.

Why this matters in production: the trigger for these incidents is almost always mundane, a failover, a deploy, a brief partition. The duration is set entirely by whether the system can shed the load its own retries created. A retry budget and a circuit breaker cost about an afternoon each to build, and they're the difference between a five-minute blip and a two-hour incident.

Cancellation has to actually cancel

A client-side timeout does not stop the server. This is the detail that surprises people the most, and it matters most at the database:

A Node query timeout abandons the response. The Postgres backend keeps executing the query, holding the connection, the snapshot, and its locks.

So a client-side timeout on a slow query gives you the worst of both worlds, the client has moved on and may retry, doubling the work, while the server runs the original query to completion anyway. Enforcement has to happen server-side:

SET statement_timeout = '2s';                       -- the actual bound
SET lock_timeout = '1s';                            -- don't queue behind a lock
SET idle_in_transaction_session_timeout = '10s';    -- kill abandoned transactions

Layer Two: A Breaker Stops Making the Calls At All

Layer one bounds each call. It does not stop you from making a thousand doomed calls a second.

Go back to that dependency that's genuinely down, 2-second timeout, 500 requests/second arriving:

500 req/s × 2s each = 1,000 concurrent in-flight requests
                      ...all of which will fail
                      ...each holding a socket, a slot, and memory
                      ...and all of it is load on a dependency trying to restart

The timeout did its job. Each request failed in bounded time. The aggregate is still an outage, you're spending your entire concurrency budget discovering, five hundred times a second, something you already knew.

A circuit breaker is the observation that after enough failures, you can just stop asking. It converts a 2-second failing call into a sub-millisecond local rejection, and that does three things at once: it frees your resources instantly, it removes load from the dependency, giving it room to actually recover, which is the exit from the metastable state above, and it fails fast enough that a fallback becomes viable. A 2-second wait before serving a cached value is a bad experience. A 1-millisecond rejection followed by a cache read is a fine one.

That second point is easy to undervalue and is often the decisive one. A dependency at 100% error rate does not recover while it's still receiving full traffic plus retries. A breaker is how the traffic actually stops, without a human doing it by hand.

The electrical breaker the pattern is named after doesn't protect the appliance that shorted, that appliance is already broken. It protects the rest of the house, by isolating the one circuit so the wiring doesn't overheat and every other room keeps its lights on. And it needs a delay before reclosing, because closing it immediately onto an unfixed short achieves nothing but another trip.

The four ways a breaker gets misconfigured

Trigger on a failure rate over a rolling window, not consecutive failures. A consecutive-failure counter fails in both directions, on a low-traffic endpoint, five consecutive failures might span twenty minutes and open a breaker for a problem long since resolved, and on a dependency where half of calls succeed, a consecutive counter never reaches five at all. Use a rate with a minimum-volume guard, so a single failure after a quiet period doesn't read as a "100% failure rate over a sample of one":

const stats = window.last(10_000);                 // last 10 seconds
if (stats.total >= 20 && stats.failureRate > 0.5) open();

A 4xx is not a failure. This is the misconfiguration that causes the most damage in practice. A 400, 404, or 422 means the dependency is healthy and rejected your request, bad input, a missing resource, failed validation. Count those as breaker failures and a single client bug, a URL scanner, or one malformed integration takes down a perfectly healthy dependency for every other caller.

Threshold on slow calls too, not just errors. A dependency at 100% success and 8 seconds per call is doing just as much damage as one that's down, arguably more, because nothing about it looks like an error. A slow-call-rate threshold ("if 60% of calls exceed 1 second, open") is the check most implementations skip entirely.

Half-open must admit a trickle, not a flood. A cooldown that expires and lets the full 500 requests/second back through at once re-hammers a recovering dependency and reopens the breaker instantly. Admit one to three concurrent probes, require a few successes before closing, and ideally ramp: half-open at 5% of traffic, then 25%, then closed.

Bulkheads: stop one dependency from starving everything else

A ship's hull is divided into watertight compartments so one breach floods a single compartment, not the vessel. The system equivalent: partition the resource requests compete for, so one dependency's slowness can't consume all of it.

Without: 200 concurrent slots, shared.
  The recommendations service slows to 8s. Requests to it accumulate.
  Within seconds all 200 slots hold recommendation calls.
  Checkout, which never calls recommendations, gets no slot. Total outage.

With:  recommendations capped at 20 concurrent.
  Slot 21 onwards is rejected immediately → the fallback runs.
  Checkout's 180 slots are untouched. Recommendations are degraded. Nothing else is.

In Node there's no thread pool to partition, which leads some people to conclude bulkheads don't apply. They do, the shared resource is event-loop time, the socket pool, and memory held by pending promises. The mechanism is a semaphore that rejects when full, not one that queues without bound:

import pLimit from "p-limit";
const limits = {
  recommendations: pLimit(20),     // optional feature: small allowance
  payments:        pLimit(60),     // critical path: generous
};

async function callRecommendations(userId: string) {
  if (limits.recommendations.pendingCount > 50) throw new BulkheadFull();
  return limits.recommendations(() => fetchRecs(userId));
}

A semaphore that queues indefinitely isn't a bulkhead, it's a delay, and the memory held by that queue is exactly the resource you were trying to protect. Size it from Little's Law: a dependency with a 50ms p99 sustaining 200 requests/second needs 10 concurrent slots, not 200. And the single highest-value bulkhead most teams skip is separate connection pools per workload, web traffic and a reporting job should never draw from the same Postgres pool.

Degradation is design work, and it hides the failure that follows

Breakers and bulkheads decide when to stop calling something. Degradation decides what the user gets instead, and it's the part that can't be configured away, it needs a decision per dependency, often a product decision rather than an engineering one.

The rule that catches the most bugs: the fallback must not depend on what failed.

// BROKEN: the fallback path hits the same overloaded database.
try   { return await cache.get(key); }
catch { return await db.expensiveQuery(); }   // ← 100% of traffic now goes here

Redis going down doesn't just remove the cache, it redirects the entire cache hit rate onto the database. A cache at a 95% hit ratio failing means the database sees 20× its normal read load, and the fallback has just converted a cache outage into a database outage. The correct version serves the degraded path behind a bulkhead and a request-coalescing lock, and sheds the excess rather than manufacturing a second failure.

And here's the trap specific to this layer, worth sitting with: degradation removes failure from your error rate. Once it's working, a failing dependency produces no errors. Requests succeed. Latency might even improve, because a fast fallback replaced a slow call. Your dashboard is flat. Your alerts are quiet.

So a system can serve the generic homepage instead of the personalised one for three weeks, because a breaker opened after a deploy changed a hostname and never closed, and nobody notices, because nothing is red. The first report comes from a product manager asking why engagement dropped. The fix is instrumenting the thing degradation hides: breaker state as a metric with an alert on "open for longer than N minutes," and a degraded-serve rate tracked as its own SLI, right next to your error rate, because it is the error rate you decided not to show users.

Layer Three: When Nothing Is Broken and You're Still Down

Layers one and two both answer the same question: a dependency I call is broken, what do I do? This layer answers a different one entirely.

Everything downstream is healthy. Every query is fast. And 9,000 requests a second are arriving at a service that can serve 6,000.

No breaker opens, because nothing is failing. No bulkhead helps, because no single dependency is being monopolized, the resource is being consumed by legitimate, healthy work. Queues just grow, latency climbs uniformly across everything, and eventually it all times out at once.

The counterintuitive fact underneath this: capacity isn't a ceiling you bump into. It's a knee, after which things get worse, not just full. Throughput is requests completed per second. Goodput is requests completed per second that anybody still wanted, inside the caller's deadline. Past the knee, throughput can look almost flat while goodput falls off a cliff, because the work still being completed is work whose caller gave up on seconds ago. A service can sit at 100% CPU, complete 6,000 requests a second, and deliver a genuinely useful 400, and its throughput graph will look perfectly fine the entire time.

A kitchen that takes every order the floor brings. At 30 covers it's fine. At 90, tickets pile up, and by the time each dish is plated the table has left. The kitchen is working flat out, food is going out at the maximum rate the stoves allow, and nobody is eating it. The fix isn't a bigger ticket rail, it's the maître d' at the door saying "we're full, forty-five minute wait." That refusal is the only intervention that gets the guests who are seated actually fed.

An unbounded queue is not a buffer

"Add a queue so we can absorb the spike" is correct advice for a transient burst and catastrophic advice for sustained overload. The arithmetic is Little's Law again:

queue depth 50,000, throughput 500/s → wait time = 100 seconds

Every one of those 50,000 items gets processed eventually. Every single one gets processed after its caller has already timed out. The system does the full amount of work and delivers none of the value, and the memory holding that queue is itself now a failure mode.

Worse, the queue changes the shape of the failure into a less useful one. Without it, request 6,001 gets an immediate 503, the client knows right away, can retry with backoff, can fall back, can tell the user something. With an unbounded queue, request 6,001 gets accepted and times out 30 seconds later, after the client has waited and held its own resources for nothing, learned nothing sooner, and now retries, adding a second item to the queue for work you'd already started.

So: every queue is bounded, and the bound comes from the latency target you actually care about.

max_depth = target_latency × service_rate

Serving at 500/s with a 2-second latency target means a queue of 1,000 items, not one more. Beyond that, reject. This applies to your HTTP accept queue, your connection-pool wait queue, your job queue, every in-process channel. An unbounded queue anywhere in the request path is exactly where the latency will accumulate.

Backpressure where you can, shedding where you can't

Backpressure means the consumer tells the producer to slow down, and the producer can. It's strictly better wherever it's available, because no work gets discarded, the rate just matches. TCP flow control is the original version of this idea; HTTP/2 flow-control windows, reactive streams, bounded channels where the producer blocks, and consumer pause/resume on a Kafka worker are all re-implementations of it. In Node, writable.write() returning false is backpressure, ignoring that return value and writing anyway is the standard way to leak memory in a pipeline.

Backpressure works inside a closed system, your own pipeline, code you control. It fails completely at an open boundary: you can't tell a million browsers, a partner's integration, or a mobile app fleet to send fewer requests. There's no window to shrink. At that boundary, the only option left is load shedding, refuse work immediately and cheaply, so the work you do accept actually completes. Shedding isn't a failure of design. It's the design. The choice was never "shed or serve everyone", it's "shed deliberately, or let the overload choose for you by timing everything out instead."

Shed on the signal that tells the truth earliest

Not every signal is equally honest, and the ranking matters:

Queue wait time is the best signal, it is the latency you're about to violate, and it leads everything else. Concurrency in flight versus your limit is next, direct and cheap. Queue depth is good but needs a service rate to interpret. Event-loop delay in Node measures the actual contended resource. CPU utilisation is mediocre, saturation isn't linear in CPU, and I/O-bound work can show low CPU while queueing badly. Latency p99 lags, by the time it moves, the queue is already deep. And error rate is the worst signal of all: it's the outcome you were trying to prevent in the first place, arriving last.

For Node specifically, event-loop delay is an excellent, cheap, local signal:

import { monitorEventLoopDelay } from "node:perf_hooks";
const h = monitorEventLoopDelay({ resolution: 10 });
h.enable();
// Shed when the loop is persistently behind: the process cannot keep up, full stop.
const overloaded = () => h.mean / 1e6 > 70;   // ms

And a shed response is only cheap if it happens early, reject before authentication, before deserialising a large body, before touching the database. A 503 that costs as much to produce as a 200 protects nothing at all.

The Order Matters, Not Just the Presence

Put together, the three layers cover three genuinely different failures, and none of them substitutes for another:

A timeout bounds one call to a dependency that's slow. It does nothing about the volume of calls you keep making to one that's dead, that's what a circuit breaker stops. A breaker does nothing about legitimate, healthy traffic simply exceeding your own capacity, that's backpressure and shedding. And underneath all three sits the retry logic that, misconfigured, turns any one of these into a self-sustaining outage regardless of how well the other two are built.

This is also, not coincidentally, the fuller version of a claim made in passing in an earlier piece on connection pooling in serverless environments: that platform retry policies, re-invoking every failed serverless request during a connection exhaustion event, are "one of the few failure modes where client retries are strictly harmful." That's retry amplification, in exactly the shape described above, a thousand concurrent invocations failing at once, each one retried by the platform, adding load to a connection table whose entire problem was already too many clients. A retry budget at the right layer, or a breaker that stops the calls outright, is the actual fix; raising max_connections again is not.

None of these three layers is optional once you're running anything with a real dependency graph. But they answer different questions, they fail in different ways when misconfigured, and the order in which you reach for them, bound the call, stop the calls, then shed what you can't backpressure, is the order that actually holds under load.

Sourced from the System Design In-Depth course, Timeouts, Retries, and Backoff, Circuit Breakers, Bulkheads, and Graceful Degradation, and Backpressure and Load Shedding.

Your database CPU is at 20%. Your slowest query is 12ms. Your slow-query log is empty. And your p99 is three seconds. This is what a connection pool failure looks like — and it never looks like a database problem.

Every engineer learns the same sentence about connection pooling: "reuse connections instead of opening a new one per request." It's true, it's useful, and it's where most people stop.

Then you deploy to serverless. Or you turn on autoscaling. Or someone adds a metrics exporter. And you discover that connection pooling isn't a performance optimisation you bolt on — it's a shared, global, hard-capped budget that half your infrastructure is spending without telling anyone.

This article is about the failure modes. Not "what is a connection pool," but the five specific ways pooling breaks in modern deployments, why each one disguises itself as something else, and what the fix actually costs you.

First: A Postgres Connection Is Not a Socket

Engineers price a database connection like an HTTP connection — a file descriptor, some buffers, a few kilobytes. Cheap. Open a thousand of them.

In PostgreSQL, a connection is a forked operating system process. Per connection, you get:

An OS process with its own page tables and scheduler entry, plus a few megabytes of private memory that never becomes shared. This is why connecting to Postgres costs orders of magnitude more than connecting to Redis, and why "one connection per request" is a design error rather than a style preference.

The right to allocate work_mem — and not once per connection, but per sort or hash node in the running query. One query with three hash joins holds three multiples of it. This is how databases get OOM-killed while every dashboard looks calm.

A snapshot, if the connection is inside a transaction, which joins the vacuum horizon. That's how a single forgotten idle in transaction connection blocks dead-tuple cleanup across the entire database.

So max_connections = 100 isn't an arbitrary cap the Postgres developers picked to annoy you. It's a memory and scheduler budget expressed as a count. Raising it to 2,000 doesn't buy you 2,000 connections' worth of throughput — it buys you 2,000 processes contending for the same cores and the same shared_buffers, trading a polite refusal of the 101st client for death by memory pressure.

The analogy worth keeping: your database is a restaurant with a fixed number of tables and a fixed number of cooks. max_connections is the tables. On a busy night the tempting move is to cram in more tables — but the cooks didn't multiply, so every dish arrives late and the kitchen falls behind on all of it at once. The restaurant that keeps ten tables and queues everyone else at the door serves more diners per hour.

That queue at the door is your connection pool's wait queue. It is the cheapest place in your entire system for a request to wait, and the last place anyone thinks to measure it.

Failure Mode 1: The Only Number That Matters Is N × Pool Size

A pool is configured per process. The limit is global. Nearly every exhaustion incident lives in that gap.

Here's a realistic accounting for a modest production system running against max_connections = 100:

Two things make this vicious.

It's a deploy-time cliff, not a load-time one. A rolling deploy briefly runs old and new instances side by side, doubling that top row. Which is why these incidents correlate with deploys rather than with traffic, and why the postmortem keeps looking at the wrong graph.

The bottom rows are invisible. Nobody counts the metrics exporter. Nobody counts the migration container. Those are exactly what tip you over.

The rule, stated properly: allocate max_connections as a budget across all consumers, then derive per-instance pool size by division.

pool_size_per_instance = floor(max_connections / max_instances) - safety_buffer

And the cost of that rule, stated honestly: pool size now depends on replica count. It has to be computed against your autoscaler's ceiling, not your current instance count — which means you deliberately run a pool smaller than any single instance could use at peak.

Why this matters in production: your autoscaler's max-replica setting is a database configuration setting. If max_replicas × pool_size exceeds max_connections, you haven't got a risk. You've configured an outage and scheduled it for the next traffic spike.

Failure Mode 2: Serverless Turns Concurrency Into Connections

A function runtime has no shared pool because it has no shared process. Each concurrent invocation is an isolated environment whose pool has a maximum useful size of one — it serves exactly one request.

Warm containers reuse a connection across invocations, which helps, and which is precisely why this problem is intermittent and maddening to reproduce. But the scaling unit is concurrency, and concurrency is the one thing you don't control.

1,000 concurrent invocations × 1 connection each = 1,000 connections
max_connections = 100, minus 3 reserved, minus everything in the table above

Roughly 90 invocations get a connection. The rest get FATAL: sorry, too many clients already.

There's a specific version of this that catches teams who did read the tutorial:

// Looks correct. Is correct — in a long-running Node server.
import { Pool } from 'pg';

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 10,
});

export default async function handler(req, res) {
  const result = await pool.query('SELECT * FROM transactions LIMIT 10');
  res.json(result.rows);
}

In a long-running process, const pool is created once and reused across every request. Correct. In a serverless function, the module is re-imported on each cold start — so max: 10 isn't a ceiling of 10 connections, it's a ceiling of 10 per concurrent environment. A hundred concurrent invocations makes it 1,000.

The global singleton pattern helps at the margins, because it survives warm starts within a container:

// lib/db.ts
import { Pool } from 'pg';

const globalForPg = global as typeof global & { pgPool?: Pool };

if (!globalForPg.pgPool) {
  globalForPg.pgPool = new Pool({
    connectionString: process.env.DATABASE_URL,
    max: 2,                      // small per instance — the pooler owns the total
    idleTimeoutMillis: 10_000,
    connectionTimeoutMillis: 5_000,
  });
}

export const pool = globalForPg.pgPool;

But be clear about what this is: damage control, not a fix. It caps each environment at 2 connections instead of 10. It does not stop the platform from giving you 500 environments. On Vercel Functions and equivalents, the singleton pattern alone is insufficient — every team shipping a direct Postgres connection plus a singleton is running on borrowed time, and hasn't hit the limit only because they haven't hit enough concurrent traffic yet.

The blast radius is the real story

What makes this an architecture problem rather than a tuning problem is that the failure does not land on the traffic that caused it.

The connection table is global. When it's full, it's full for everybody:

• The internal admin panel stops loading — and its on-call isn't yours.
• The payouts worker's queue backs up silently.
• The metrics exporter fails, so your dashboards go blank during the incident.
• An engineer opens psql to investigate and gets refused.

Then the platform's retry policy re-invokes every failed request, adding load to a resource whose entire problem is too many clients. This is one of the few failure modes where client retries are strictly harmful.

Why this matters in production: connection exhaustion is a tenancy problem as much as a capacity one. If one spiky autoscaled workload can consume the budget of your payments worker, you've coupled two services through a resource neither of them monitors. The cheap mitigation is a per-role cap — ALTER ROLE app_web CONNECTION LIMIT 40 — which converts a shared outage into a contained one, at the cost of one workload hitting its ceiling while slots sit idle elsewhere.

Failure Mode 3: Raising the Pool Makes It Worse

This is the counterintuitive one, and it's the standard incident response.

Requests are timing out waiting for connections. An engineer raises the per-instance pool from 20 to 100 across 10 instances, restarts, and throughput drops further while p99 gets worse.

Little's Law explains why. L = λW — the number of connections you need busy at once is arrival rate times hold time. Take 400 req/s where each request runs one 5ms query, then run it again after a bad plan pushes that query to 200ms:

Healthy:  L = 400/s × 0.005s = 2 connections busy on average
          ρ = (400 × 0.005) / 20 = 0.1   → waits negligible

Degraded: L = 400/s × 0.200s = 80 connections wanted, against a pool of 20
          ρ = (400 × 0.200) / 20 = 4.0   → demand exceeds capacity, queue grows unbounded

Two connections is the honest answer at healthy load. Intuition wants pool size to track concurrent requests; Little's Law says it tracks concurrent requests × the fraction of their life spent inside the database, which is usually small. That gap is why correctly-sized pools look absurdly low to people who haven't done the arithmetic.

So why doesn't a bigger pool help in the degraded row? Because service time S isn't a constant. It depends on how many queries are executing concurrently against fixed cores and a fixed disk. Past hardware saturation, extra in-flight queries add no throughput — they divide the same throughput into slower pieces. Then second-order costs push throughput actively down: context switching between hundreds of runnable backends, contention on buffer-mapping and lock-manager partitions, work_mem allocations evicting your working set from cache, and more concurrent snapshots holding back the vacuum horizon.

Hence the principle worth memorising: queueing outside the database is nearly free; queueing inside it is expensive. A request in a pool's FIFO costs a promise and a timer. A request inside the database costs a process, several megabytes, a snapshot, and a share of every lock partition it touches.

The community starting point, popularised by HikariCP, is connections ≈ (2 × core_count) + effective_spindles. Treat it as a hypothesis to load-test, not a law — effective_spindles is a rotational-disk-era proxy for storage concurrency, and on NVMe at 500K+ random IOPS it means something quite different from its name. Note also that it lands in the low tens for the whole database, not per instance.

Why this matters in production: during pool exhaustion, the right move is almost never "raise the pool." It's to find what raised hold time — a slow query, a lock wait, an external call inside a transaction — because L = λW makes pool demand linear in W, and W is the term you can usually cut by an order of magnitude. Raising C to match a broken W just relocates the collapse into the database, where it costs more and hides better.

Failure Mode 4: Transaction Pooling Silently Eats Session State

The structural fix for serverless is a transaction-mode pooler — PgBouncer, RDS Proxy, Supavisor, Prisma Accelerate. A lightweight process holds a few real backends and multiplexes many clients onto them, lending a backend only for the duration of a transaction. Ten thousand clients, twenty backends.

What it costs is session state, because the backend you get is not the one you had last time. And the way you find out is a production error that says nothing about pooling.

The failure signature is worth committing to memory, because nothing in it mentions pooling: the ORM works perfectly locally against a direct connection, and throws prepared statement "s0" does not exist in production. Or relation "work_items" does not exist halfway through a request. Or — worst of all — no error, just timezone drift producing quietly wrong date arithmetic, because a session SET didn't survive.

For Prisma specifically, the two-connection-string setup is non-negotiable, because the migration engine uses session-level advisory locks and will hang indefinitely through a transaction-mode pooler:

DATABASE_URL="postgresql://user:pass@pooler-host:6432/mydb?pgbouncer=true"
DIRECT_URL="postgresql://user:pass@postgres-host:5432/mydb"

datasource db {
  provider  = "postgresql"
  url       = env("DATABASE_URL")   // pooler, for app queries
  directUrl = env("DIRECT_URL")     // direct, for migrations
}

The general pattern, whatever your stack: two endpoints against one database. A transaction-mode port for application traffic, and a session-mode or direct connection for migrations, LISTEN-based workers, and human debugging. The cost is a second connection string to configure, get wrong once, and document.

One more thing worth saying plainly: session mode is the compatibility escape hatch, not a fix for exhaustion. It holds a backend for the client's entire connection, giving roughly 1:1 reuse — all of a proxy's operational cost with none of the multiplexing benefit.

Failure Mode 5: Holding a Connection Across a Call You Don't Control

This one passes code review every single time.

await client.query('BEGIN');
await client.query('UPDATE orders SET status = $1 WHERE id = $2', ['charging', id]);

// Three seconds of somebody else's p99 — with a backend, a snapshot,
// and a row lock all held open, executing nothing.
const charge = await stripe.charges.create({ amount, currency: 'usd' });

await client.query('UPDATE orders SET charge_id = $1 WHERE id = $2', [charge.id, id]);
await client.query('COMMIT');

Run L = λW on it. At 50 req/s with a 3-second vendor call, you need 150 connections held to do essentially no database work. Your pool is 20.

Your pool size is now a function of a vendor's p99. When their latency doubles, every unrelated endpoint on that instance goes down with it.

And statement_timeout will not save you — this is the genuinely useful part. No statement is running. The backend is idle in transaction, holding a snapshot that blocks vacuum database-wide and row locks that stall other writers, while executing nothing at all. The setting that actually fires is idle_in_transaction_session_timeout, which every application role should have, and which still only acts after the damage, aborting mid-payment.

The fix is structural, not configurational: commit an intent carrying an idempotency key, make the external call holding nothing, then commit the outcome in a second short transaction.

The cost, named honestly: one atomic operation became two, so a crash between them leaves an order stuck in charging. That needs a reconciliation job that finds stale intents and asks the provider what happened — which is exactly why the idempotency key is written in the first transaction rather than generated at call time. The trade is real: a recoverable inconsistency in exchange for not tying your pool to someone else's uptime.

The Metric Nobody Graphs

Every failure mode above shares a diagnostic signature, and it's why these incidents burn hours.

When the pool saturates, latency accumulates before any query runs — so every tool you'd reach for measures the wrong interval:

• pg_stat_statements reports execution time. Your queries look fine at 8ms.
• Slow-query logs are silent. Nothing ran slowly.
• Your APM's database span starts once the driver already holds a connection.
• Database CPU is low, which reads as "the database is healthy" and sends the investigation into application code.

Meanwhile p99 is 3 seconds, of which 2.99 were spent inside pool.connect() — an interval on no component's dashboard, because the app thinks it's database time and the database has never heard of the request.

So instrument the acquisition yourself:

async function acquire(pool: Pool) {
  const start = performance.now();
  const client = await pool.connect();
  // The number that holds your p99 during saturation. Histogram it. Alert on p99.
  metrics.histogram('db.pool.wait_ms', performance.now() - start);
  return client;
}

Also worth having permanently: waiting count (pool.waitingCount) for queue depth, in-use vs idle (totalCount, idleCount) which gives you utilisation, and acquisition timeouts per minute. Set connectionTimeoutMillis — unset means unbounded queueing, which is a queue with no admission control.

On the database side, the first query of any connection incident:

SELECT state, count(*) FROM pg_stat_activity GROUP BY state ORDER BY count DESC;

A large idle in transaction count is Failure Mode 5, live in production.

And if you're running PgBouncer, the single most important view:

SHOW POOLS;
-- cl_waiting: clients waiting for a server connection → THIS SHOULD BE 0

Alert thresholds worth setting today: warning at 70% of max_connections, critical at 85% (you're roughly 90 seconds from user-visible errors), page immediately on more than 10 connections idle in transaction, and alert on sustained cl_waiting > 0.

The Fixes, and What Each One Actually Costs

There is no free option. Pick the bill you'd rather pay.

Mapping that to real deployments:

On that Edge Runtime row, the simplest advice is the best advice: don't query your database from Edge Runtime unless you're on an HTTP driver. Move database access to the Node.js runtime and reserve the edge for work that only touches KV or cache.

The Short Version

A Postgres connection is a forked process with megabytes of private memory, the right to allocate work_mem per plan node, and a snapshot that holds back vacuum. max_connections is a memory budget wearing a counter's clothing.

The only number the database sees is N instances × pool size, plus workers, cron, migrations, exporters, and reserved slots. Rolling deploys briefly double the app tier, which is why these incidents track deploys rather than traffic.

Small pools are faster under load. L = λW puts required concurrency at arrival rate × hold time — usually single digits. When the pool saturates, cut W; don't raise C.

Transaction-mode pooling costs session state, and announces it through errors that mention prepared statements rather than pooling.

Never hold a connection across a call you don't control, because statement_timeout cannot interrupt a backend that isn't executing anything.

And instrument pool-wait time. It's the interval that holds your p99 during every one of these failures, and it appears on nobody's dashboard by default.

Connection pooling isn't a performance optimisation. It's a shared budget with a hard ceiling, spent by more consumers than anyone has counted — and in serverless, the spending is done by a concurrency number you don't control.

Every major PostgreSQL release gets the same LinkedIn headline treatment: "this changes everything." Most of the time it doesn't, it's a dozen genuine improvements wrapped in the language of a paradigm shift. PostgreSQL 19 is a real release with real substance, but the honest way to evaluate it isn't the headline feature, it's asking what specifically didn't work in PG18 that now does, and what the actual cost of adopting it is.

The Shape of the Two Releases

PostgreSQL 18 was primarily an I/O and observability release: the Asynchronous I/O subsystem rewired how the engine reads from disk, pg_stat_io got a near-total overhaul, and B-Tree skip scans fixed a decade-old multicolumn-index limitation. PostgreSQL 19 is primarily a DDL/DML ergonomics and operational-maintenance release: the headline feature (SQL/PGQ) is a new query language surface, but the features that will actually change your on-call life are REPACK CONCURRENTLY, native partition reshaping, and parallel autovacuum.

That distinction matters because it tells you where to look for value. If PG18 already fixed your I/O-bound scan performance, PG19 isn't going to double it again, it's going to fix the maintenance operations that PG18 left untouched.

1. SQL/PGQ: Property Graph Queries

What it is: PostgreSQL 19 adds SQL/PGQ, the SQL:2023 standard for querying graph-shaped data. You define a property graph as a read-only view over existing relational tables (a node table, an edge table), then query it with MATCH and Neo4j-style arrow syntax: MATCH (a:Employee)-[:MANAGES]->(b:Employee). Internally, PostgreSQL rewrites the arrow syntax into ordinary joins against your existing tables before the planner ever runs, so it inherits your existing indexes and your existing row-level security automatically.

Comparison (PG18 vs. PG19): In PG18, and every version before it, modeling a graph relationship (a dependency tree, an org chart, an authorization graph, "who reports to whom") meant writing a recursive CTE, WITH RECURSIVE, walking a self-referencing table one level at a time. Recursive CTEs work, but they're procedural: you write the recursion, you write the termination condition, and the planner frequently struggles to estimate the cost of a deep or unbounded traversal, which is exactly the kind of query that "chokes the planner" that gets complained about on social media. There was no declarative way to say "find all paths matching this pattern" — you had to hand-build the loop yourself in SQL.

Pros:

• No secondary graph database, no sync pipeline. If your graph queries are shallow-to-medium depth over data that's already relational, you get graph-shaped querying without standing up Neo4j and building an ETL job to keep it current.
• Declarative pattern matching gives the planner a clearer picture of intent than a hand-written recursive CTE, which can lead to better plans for the same logical query.
• Inherits your existing security model for free, since a property graph is "just a view."

Cons:

• It's still running joins under the hood. You do not get index-free adjacency, the property that makes a native graph database like Neo4j fast for deep, multi-hop traversal (5+ hops over millions of edges). If your actual workload is that kind of traversal, SQL/PGQ won't rescue you from needing a dedicated graph store.
• New syntax surface means new things to learn and new things the planner can misjudge; it's not yet battle-tested the way recursive CTEs are, having existed for two decades.
• The realistic audience for this feature is "developers who currently reach for a recursive CTE for shallow hierarchical queries," not "teams running production graph analytics at Neo4j scale." Know which one you are before switching.

2. REPACK: Unifying and Unlocking VACUUM FULL / CLUSTER

What it is: PostgreSQL 19 introduces a single REPACK command that replaces both VACUUM FULL (rewrite the table to reclaim bloat) and CLUSTER (rewrite the table in index order). The critical addition is a CONCURRENTLY option — REPACK ... CONCURRENTLY — that rebuilds the table without taking the ACCESS EXCLUSIVE lock that made both of the old commands unusable on a live, high-traffic table.

Comparison (PG18 vs. PG19): In PG18, if a table had accumulated enough bloat (dead row versions from updates/deletes that VACUUM alone couldn't reclaim) that you needed to physically shrink it, your only in-core option was VACUUM FULL, which locks the table exclusively for the entire rewrite. On a table serving live traffic, that's not a maintenance task, it's a scheduled outage. The workaround was the third-party pg_repack extension, which achieves a similar result without full locking by building a shadow copy and swapping it in, but that means installing and trusting an external extension for something this fundamental.

Pros:

• This is the single biggest operational upgrade in the release for anyone who has had to fight table bloat on a production system: the capability that pg_repack (the extension) existed specifically to provide is now in core, with CONCURRENTLY.
• One command name instead of two (VACUUM FULL and CLUSTER remain for backward compatibility, but REPACK is now the unified entry point).
• Removes a dependency on a third-party extension for a core maintenance operation.

Cons:

• CONCURRENTLY avoiding the exclusive lock doesn't mean it's free, it still consumes I/O and CPU rewriting the table, and the new max_repack_replication_slots variable exists because concurrent repack has its own resource considerations to tune.
• If your team already has pg_repack (the extension) working reliably, migrating to core REPACK CONCURRENTLY is a "nice to have simplify," not an urgent fix, don't rip out something that already works without testing the native replacement first.

3. Native Partition Merge/Split

What it is: ALTER TABLE ... MERGE PARTITIONS and ALTER TABLE ... SPLIT PARTITIONS, letting you reshape an existing partitioned table's partition boundaries natively.

Comparison (PG18 vs. PG19): PG18 invested heavily in partition query performance, more efficient planning over many partitions, better partitionwise joins, reduced memory use for partition pruning, but did nothing for partition maintenance. If a monthly partition grew too large and you wanted to split it into two, or several small partitions had accumulated and you wanted to merge them, there was no native command. You did it by hand: create new partition(s), migrate the relevant rows with INSERT ... SELECT, detach and drop the old partition, all while carefully managing locks and application downtime windows.

Pros:

• Turns a multi-step, error-prone, DBA-scripted operation into a single DDL statement.
• Makes it realistic to actually right-size partitions over time as data volume and access patterns change, rather than living with whatever partition scheme you picked at design time.
• Complements PG18's partition query-planning work: PG18 made partitioned tables fast to query, PG19 makes them practical to maintain.

Cons:

• Any partition reshape on a large table still means physically moving rows, this is not a free metadata-only operation, plan for I/O and lock impact accordingly.
• Doesn't retroactively fix a bad initial partitioning key choice, it makes boundary changes easier, not a full re-partitioning strategy change.

4. GROUP BY ALL

What it is: New SELECT syntax, GROUP BY ALL, which automatically groups by every column in the SELECT list that isn't an aggregate or window function, no manual enumeration required.

Comparison (PG18 vs. PG19): PG18 addressed a related but distinct problem at the planner level: it learned to ignore GROUP BY columns that were functionally dependent on other grouped columns (for example, if you group by a table's unique primary key, other same-table columns don't logically need to be listed, and PG18's planner recognized this and dropped them from the actual grouping operation for efficiency). That's an internal cost optimization, it didn't change what you had to type. In PG18 you still hand-wrote every column in the GROUP BY clause, no matter how wide the SELECT list.

Pros:

• Removes the tedious, error-prone task of keeping a long GROUP BY list in sync with the SELECT list, adding a column to one and forgetting the other is a classic source of "column must appear in GROUP BY" errors.
• Particularly valuable for wide analytical queries with 10+ grouping dimensions.

Cons:

• Implicit grouping means it's slightly easier to accidentally group by more (or fewer) columns than you intended if the SELECT list changes later, explicit lists are more self-documenting for complex queries. Use with intention, not as a default habit for every query.

5. FOR PORTION OF: Temporal UPDATE/DELETE

What it is: New clause for UPDATE and DELETE, FOR PORTION OF <period> FROM <start> TO <end>, that lets you modify or delete just a sub-range of a temporal (range-based) row, automatically splitting the surrounding range as needed.

Comparison (PG18 vs. PG19): PG18 introduced the constraint half of temporal tables: WITHOUT OVERLAPS for PRIMARY KEY/UNIQUE and PERIOD for foreign keys, letting you enforce that time ranges in a table never overlap. But it gave you no corresponding operation half. If you had a row valid from Jan–Dec and needed to change just the March–June portion, you had to manually delete the original row and insert two (or three) new rows representing the split ranges yourself, exactly the kind of fiddly, off-by-one-prone logic a database feature should be doing for you.

Pros:

• Closes the gap PG18 left half-finished: PG18 gave you guardrails (non-overlapping constraints), PG19 gives you the verbs (safely editing a slice of history without hand-rolling the split).
• Meaningful for any system tracking effective-dated data, pricing history, contract terms, HR assignment periods, insurance coverage windows.

Cons:

• Temporal tables remain a niche feature relative to PostgreSQL's overall audience, most applications don't model data this way, and adopting WITHOUT OVERLAPS + FOR PORTION OF is a genuine schema-design commitment, not a drop-in swap.

6. INSERT ... ON CONFLICT DO SELECT ... RETURNING

What it is: Extends ON CONFLICT beyond DO NOTHING/DO UPDATE with a new DO SELECT ... RETURNING option, returning (and optionally locking, via FOR UPDATE/FOR SHARE) the row that caused the conflict, without modifying it.

Comparison (PG18 vs. PG19): PG18 made real progress on returning row state, adding OLD/NEW alias support to RETURNING across INSERT/UPDATE/DELETE/MERGE, so you could see before-and-after values in a single statement. But ON CONFLICT itself was unchanged: still only DO NOTHING (silently skip, tell you nothing about the existing row) or DO UPDATE (you must actually modify something to get a RETURNING result). The common "get-or-create" pattern, insert a row if it doesn't exist, otherwise just give me the existing one, had no clean native expression; teams worked around it with a no-op DO UPDATE SET col = col just to trigger a RETURNING clause.

Pros:

• Directly solves get-or-create without a no-op write, which matters for both clarity and for avoiding unnecessary row versions from a fake update (fewer wasted row versions means less bloat, which loops back to why REPACK CONCURRENTLY matters less often).
• Optional row locking (FOR UPDATE/FOR SHARE) on the conflicting row means you can safely read-then-act on it within the same statement, useful for concurrent upsert-adjacent logic.

Cons:

• Another ON CONFLICT branch to learn and to get right in mixed application code that already juggles DO NOTHING and DO UPDATE logic; worth auditing existing upsert helper functions to see where this actually simplifies things versus where it's unnecessary.

7. TOAST Default Compression: pglz → lz4

What it is: The default compression algorithm for TOASTed (large, out-of-line) values changes from pglz to lz4.

Comparison (PG18 vs. PG19): lz4 TOAST compression already existed as an option in PG18 (and earlier), you could set default_toast_compression = lz4 or specify it per-column. But almost nobody did, because defaults are what most schemas actually run with. PG18 shipped with pglz as the out-of-the-box default; PG19 flips that default.

Pros:

• A genuinely free win for anyone storing large JSONB, text, or bytea values who never manually tuned this setting, lz4 compresses and decompresses meaningfully faster than pglz at a comparable ratio.
• Zero migration effort for new databases; the new default just applies.

Cons:

• Existing tables don't retroactively recompress, this only affects newly TOASTed values going forward (or values rewritten via REPACK/VACUUM FULL), so an existing large database won't see the benefit until data is naturally rewritten or explicitly reprocessed.

8. Parallel Autovacuum Workers

What it is: A single autovacuum job on one table can now use multiple parallel workers, controlled globally by autovacuum_max_parallel_workers and per-table by the autovacuum_parallel_workers storage parameter. PG19 also adds a scoring system (autovacuum_vacuum_score_weight, autovacuum_freeze_score_weight, and related variables) to decide which tables get processed first, replacing a simpler threshold-only heuristic.

Comparison (PG18 vs. PG19): PG18 improved autovacuum's behavior significantly: "eager freezing" let normal vacuums freeze all-visible pages proactively (reducing the cost of a later full freeze), autovacuum_worker_slots let you raise the effective worker cap at runtime without a restart, and autovacuum_vacuum_max_threshold let you set a fixed dead-tuple trigger point instead of relying purely on percentages. What PG18 didn't change: each individual vacuum job on a given table still ran as one single worker process. On a genuinely huge, high-churn table, that one worker was the throughput ceiling, no matter how many total autovacuum worker slots your cluster had available.

Pros:

• Directly attacks the specific case that bites teams running very large, high-write tables: a vacuum job that used to take hours on one worker can now split the work across several, on the same table, at the same time.
• The new scoring system for processing order is a real improvement over pure threshold checks, better prioritizing which of many candidate tables actually needs attention most urgently.
• Complements, rather than duplicates, PG18's improvements, PG18 made each vacuum pass smarter and more proactive; PG19 makes the biggest passes faster by parallelizing them.

Cons:

• More parallel workers means more concurrent I/O and CPU contention during vacuum; on a system already tight on resources, this needs the same careful tuning any parallelism feature does, it's not automatically a net win without headroom to spend.
• Per-table autovacuum_parallel_workers is one more tuning knob DBAs now need to understand and set deliberately for their biggest tables, rather than relying entirely on cluster-wide defaults.

9. Logical Replication: Native Sequence Synchronization

What it is: Sequences can now be included in logical replication. CREATE/ALTER PUBLICATION ... ALL SEQUENCES publishes all sequences, and ALTER SUBSCRIPTION ... REFRESH SEQUENCES syncs sequence values (not just existence) on the subscriber to match the publisher. pg_get_sequence_data() lets you inspect sync state directly.

Comparison (PG18 vs. PG19): PG18 made solid logical replication improvements, generated column values could finally be replicated, and the default streaming mode for new subscriptions switched from off to parallel for better apply performance. But sequences were entirely untouched by logical replication in PG18 and every prior version: a subscriber's sequences had no relationship to the publisher's. If you promoted a logically-replicated subscriber to primary during a failover, its sequences would still be wherever they last were on that subscriber, not caught up to the publisher's actual nextval() position, unless you manually reset them yourself before cutting traffic over.

Pros:

• Removes a genuinely dangerous, well-known logical-replication gap: without this, a poorly-timed failover onto a logically-replicated subscriber can hand out primary-key values that collide with rows the old primary already committed, a duplicate-ID bug hiding specifically in your disaster-recovery path, the worst possible place for a bug to hide since it only shows up when you're already in an incident.
• pg_get_sequence_data() gives visibility into sync state that simply didn't exist before, useful for confirming a subscriber is actually safe to promote.

Cons:

• This is a correctness fix for a previously-silent gap, not a performance feature, teams need to actively adopt ALL SEQUENCES and REFRESH SEQUENCES in their subscription setup; it isn't retroactively applied to existing subscriptions without action.

Which of These Actually Change How You Operate Postgres

Ranking by realistic production impact, not headline appeal:

1. REPACK CONCURRENTLY — removes a hard operational constraint (mandatory downtime for bloat reclaim) that has existed since the beginning of Postgres.
2. Parallel autovacuum — directly extends throughput on the exact tables where autovacuum has historically fallen behind.
3. Logical replication sequence sync — closes a real correctness gap sitting specifically in failover paths.
4. Partition merge/split — makes partition schemes maintainable as data grows, rather than fixed at design time.
5. SQL/PGQ — genuinely useful for a specific class of query (shallow hierarchical/relationship modeling), not a universal graph-database replacement, know which camp you're in before treating it as a headline reason to upgrade.
6. Everything else (GROUP BY ALL, FOR PORTION OF, ON CONFLICT DO SELECT, TOAST lz4 default) — real, welcome ergonomics and default-quality improvements, but incremental rather than architectural.

Where This Fits

If you're running the PostgreSQL In-Depth course, the maintenance-and-bloat modules built around pg_repack and manual VACUUM FULL tradeoffs are the direct prerequisite for understanding why REPACK CONCURRENTLY matters as much as it does, and the partitioning phase is the natural place to slot in the new merge/split DDL. For the deep architectural picture PG19 builds on top of (process model, MVCC, the planner, WAL), see "The PostgreSQL Elephant in the Room."

PostgreSQL 19 isn't a rewrite of what Postgres is, it's the same server process, managing files intelligently, that "What the PostgreSQL Server Is Actually Doing" describes, just with more of the maintenance and ergonomic rough edges sanded down. That's not a smaller story than "Postgres killed the recursive CTE", it's a more honest one.

#PostgreSQL #Database #Backend #SoftwareEngineering #DatabaseInternals #SQL #DevOps

A customer clicks "Pay Now" once. The request reaches your payment service, the charge succeeds, but the response times out on the way back. The client, seeing no answer, retries. Somewhere in your logs there are now two successful charges for one click. Nobody wrote a bug. The network just did what networks do.

Retries Are Not Optional, They're the Default

In a single process, a function call either returns or the whole program crashes with it. In a distributed system, that guarantee disappears. A request can fail before it reaches the server, after the server processes it but before the response comes back, or anywhere in between, and from the caller's side, all three failures look identical: silence, or a timeout.

Given that, a caller has exactly two honest choices: give up, or retry. Almost every serious system chooses to retry, because giving up on a payment, an order, or a blockchain transaction just because a router hiccupped is worse than the alternative. The problem is that "the alternative" retrying turns a network problem into a correctness problem, unless the operation you're retrying can tolerate being run more than once.

That property has a name: idempotency. An operation is idempotent if running it once and running it five times leave the system in the same state. SET x = 5 is idempotent — running it twice still leaves x at 5. x = x + 5 is not, run it twice and you've added 10. Retrying a network call is safe by default only when the operation behind it is naturally idempotent. Charging a card, placing an order, sending an email — none of these are. That's the gap idempotency keys exist to close.

How an Idempotency Key Actually Works

The mechanism is simpler than the name suggests. The client generates a unique key, typically a UUID, once, at the moment the user takes the action, before any request is sent. That key is attached to the request, usually as a header (Idempotency-Key: 8f14e...). Every retry of that same logical action, the same click, the same order, reuses the exact same key.

On the server, before doing any real work, the first thing that happens is a lookup: has this key been seen before?

• Not seen before → this is genuinely a new request. Process it (charge the card, place the order), store the key alongside the result, and return.
• Seen before, and finished → this is a retry of something already done. Skip the actual work entirely and return the stored result from the first attempt. The client gets the same success response it would have gotten the first time, and no charge happens twice.
• Seen before, and still in progress → a retry arrived while the original request is still being processed (a slow response, not a failed one). The correct move is to make the retry wait or reject it, never to run the operation again concurrently.

That storage of "key → result" is the entire trick. It turns "run this action" into "run this action, but only the first time you're asked, and remember what happened so every later ask gets the same answer." The database row or cache entry holding that key is doing the same job as the WAL in PostgreSQL, giving you a durable record of intent so a retry, a crash, or a redelivery doesn't have to guess what already happened.

Why This Matters in Production, Not Just in Theory

This isn't a hypothetical edge case; it's the default behavior of almost every reliability mechanism you already rely on:

• Payment gateways retry. Stripe, for instance, builds idempotency keys into its API specifically because a client-side timeout on a successful charge is common enough to have a name in their docs.
• Message queues retry. SQS, Kafka consumers, and most queue systems offer at-least-once delivery, not exactly-once, because exactly-once delivery across a network is provably expensive to guarantee. "At least once, so dedupe on your end" is the honest tradeoff, and idempotency keys are "your end." This is exactly why background job systems like BullMQ build retries with exponential backoff and dead letter queues into the framework itself, rather than leaving it to each job handler to reinvent.
• Mobile clients retry. A user on a bad connection taps "Submit" once, but the app, seeing no response after a few seconds, quietly retries the request behind the scenes. Without an idempotency key, that one tap becomes two orders.
• Load balancers and proxies retry. Some infrastructure automatically retries a request that failed to connect, before your application code ever sees it happen once, let alone twice.
• Webhooks retry, by design. Stripe, GitHub, and most webhook senders will redeliver an event if your endpoint doesn't return a fast 2xx response, on the assumption that a timeout means you never got it. Idempotent webhook processing — checking the event ID before acting on it — is the only thing standing between that redelivery and a duplicate side effect on your end.

Without an idempotency key sitting between "the network is unreliable" and "the operation isn't naturally repeatable," every one of these standard, unremarkable mechanisms becomes a live double-charge, double-order, or double-send bug waiting for the right timing to trigger it.

Example: Why a Blockchain Indexer Cannot Survive Without This

Nowhere is this more visible, or more unforgiving, than in a blockchain indexer, and it's worth walking through concretely.

An indexer's whole job is to watch a chain, pull each new block, extract the events or transfers inside it, and write them into your own database so your application can query them fast. That sounds like a simple one-pass pipeline: get block, process block, move on. In practice it never runs exactly once per block, for reasons entirely outside your control:

• Reorgs. The chain itself can rewind and replay a range of blocks when a fork resolves. Your indexer will see block 18,402,001 more than once, as two different, competing versions of "reality."
• Crash recovery. If your indexer process dies mid-batch, on restart it doesn't know precisely which of the last few blocks were fully written versus half-written, so the safe move is to reprocess a small overlapping window, not trust a fragile "last processed block" pointer to be exactly right.
• RPC retries. The call to fetch a block from a node can itself time out and get retried, occasionally landing you the same block payload twice from two separate requests racing each other.

Now picture what happens without idempotency built in: a transfer event says "500 USDC moved from wallet A to wallet B." If that event gets processed twice, because of a reorg replay or a crash-recovery reprocess, and your indexing logic does balance += 500 each time it sees the event, wallet B's balance is now wrong by 500 USDC in your database, permanently, until someone notices the number doesn't match the chain and re-runs a full backfill. On a system indexing tens of millions of events, that's not a rare accident, it's a near-certainty over enough uptime.

The fix is the exact same pattern as the payment example, just with a different key. Instead of a client-generated UUID, the natural idempotency key is something already unique to the data itself: (transaction_hash, log_index) for an EVM chain, for instance, uniquely identifies one specific event, no matter how many times it's delivered to you. The write becomes an upsert keyed on that pair, not a blind balance += amount. Seen this (tx_hash, log_index) before? Skip it, or overwrite with the identical result, never add on top of it again. Reorgs, crash-recovery reprocessing, and duplicate RPC responses all become harmless, because the operation of "record this event" is now idempotent, exactly the same property a payment API gets from a client-supplied key.

Where to Put the Key

Idempotency keys work at whatever layer needs the guarantee, and the source of the key changes depending on who's best positioned to know "this is the same logical request":

• Client-generated (payments, order placement, form submissions): the client mints a UUID once, before the first attempt, and resends the identical key on every retry of that same user action.
• Data-derived (blockchain events, webhook deliveries, message queue consumers): the key comes from something already unique in the payload itself, a transaction hash, a webhook delivery ID, a message ID, so you never have to coordinate key generation between sender and receiver at all.

Either way, the underlying requirement is the same: the key must be stored durably, checked before any side effect runs, and scoped to an appropriate time window (payment idempotency keys are typically only guaranteed for 24 hours; a blockchain indexer's (tx_hash, log_index) uniqueness is effectively permanent).

Where This Fits in the Full Course

Idempotent webhook processing and BullMQ retry/backoff patterns are both covered hands-on in the Node.js In-Depth course, in the Practitioner phase — module P-7 ("Connecting External Services and Caching") and module P-12 ("Background Jobs and Task Queues with BullMQ"), respectively.

The Question Worth Asking Yourself

Next time you write a piece of code that has a side effect, charges something, sends something, increments something, and sits behind a network call, ask one question before you ship it: if this exact request arrived twice, on purpose or by accident, would the result still be correct?

If the honest answer is no, that's not a rare failure mode you're accepting, it's a bug you've already written. The network will find it for you eventually. It's better to find it first.

#DistributedSystems #SoftwareEngineering #Backend #SystemDesign #Blockchain #APIDesign

Almost every system that makes this speed-up possible Redis, your browser, your database, even a function that "remembers" its last few results is solving the exact same problem underneath: it has limited room, so when that room runs out, it has to decide what to throw away first. The answer nearly all of them land on is the same one: throw away whatever hasn't been touched in the longest time. That policy has a name Least Recently Used, or LRU and the way it's actually built is one of the more satisfying "two weak pieces make one strong piece" stories in software.

The problem neither obvious answer solves

Say you want to build this yourself: a fixed-size cache that instantly tells you if something is stored, and instantly evicts the "stalest" entry the moment you're full.

Your first instinct might be a plain lookup table a dictionary, a hash map, whatever your language calls it. That gives you instant answers to "is this here?" But a lookup table has no concept of time. It doesn't know that key A was touched ten seconds ago and key B was touched ten minutes ago. To find the stalest entry, you'd have to check everything slow, and it gets slower as the cache grows.

So your second instinct might be a simple ordered list keep every entry in a line, most recently used at the front. That solves the ordering problem. But now finding a specific entry means walking the line from the front until you happen to find it. And worse, once you find it, moving it to the front means pulling it out of the middle of the line which, in a plain list, means shifting everything around it.

Neither piece alone works. One gives you instant lookup with no memory of time. The other gives you a sense of time with no instant lookup.

The fix: don't choose, combine

The actual solution used almost everywhere is to stop treating "look something up" and "track its age" as the same job, and instead let one structure handle each, wired together.

Picture a line of people, ordered by who was served most recently front of the line is "just here," back of the line is "hasn't been seen in ages." That's your time-ordering structure, and because it's a doubly linked line (each person knows who's directly in front of and behind them), pulling anyone out of the middle and moving them to the front is instant you're just relinking a few neighbors, not shuffling the whole line.

Now add a cashier holding a notebook that maps every person's name directly to their exact spot in that line. You don't scan the line to find someone you check the notebook, and it points you straight at them. That notebook is your lookup table, except its entries don't hold the person's information directly they hold a pointer to where that person is standing.

Put those two together and something interesting happens: a lookup stops being "search the line" and becomes "check the notebook, then relink two spots in the line." Both steps are instant, regardless of how many people are in line. The lookup table gives you the address; the line gives you a cheap way to reorder once you're there. Neither piece is doing the other's job they're doing their own job, wired to the same underlying entries.

Eviction is the same trick pointed at the back of the line instead of the front: whoever is standing at the very back is, by definition, the stalest entry so when the cache is full, that's who gets dropped, and the notebook forgets their name too.

Why real systems don't build this exactly

Here's the part worth sitting with: at genuinely large scale, most real caching systems don't implement pure LRU, because keeping a perfectly ordered "line" updated on literally every single read becomes its own overhead once you're handling millions of operations a second.

Redis's LRU eviction, for instance, doesn't maintain one global perfectly-ordered line at all it samples a handful of random keys and evicts whichever of those looks stalest, repeated as needed. It's an approximation, and a good one, because it gets 90% of the benefit of true LRU for a fraction of the bookkeeping cost.

Your browser's HTTP cache and CDN edge caches lean on the same core idea evict whatever's gone longest untouched once storage fills up to decide what to drop when they hit their limits, which is exactly why re-visiting a page you loaded five minutes ago feels instant, but a page from three weeks ago fetches fresh.

Databases play the same game one layer down. A database's in-memory buffer pool the pages of the table currently held in RAM instead of read from disk often uses something called clock-sweep instead of textbook LRU: cheaper to maintain, same underlying goal of keeping "hot" pages in memory and letting cold ones get evicted first.

And plenty of ordinary application code hits this without ever calling it LRU by name "cache the last 500 computed results, and once that limit is hit, drop whatever hasn't been asked for in a while" is the identical problem, just at a size where the textbook version runs perfectly well with no approximation needed.

The actual takeaway

The mechanism is almost deceptively small once you see it: pair a lookup table with an ordered line, and let the table's entries point directly into the line instead of duplicating what's in it. That's the whole trick. What's genuinely interesting is how far that one idea travels from a toy interview problem, to the reason your database feels fast after warming up, to the reason revisiting a webpage is instant and revisiting an old one isn't. You've probably relied on a system built this way today without ever knowing it had a name.

A team runs a routine migration: ALTER TABLE transactions ADD COLUMN region TEXT. Nothing dramatic on paper. Twelve minutes later the connection pool is full, every request from every user is failing, and on-call is paged. The migration itself was never the problem. A lock queue was.

A Lock Isn't a Wall, It's a Queue Ticket

The word "lock" makes people picture a barrier: something is locked, everything else waits outside. That's not quite how PostgreSQL locking works, and the difference is exactly what causes outages like this one.

A better picture is a queue ticket. Every statement that touches a table takes a ticket. Some tickets are compatible with each other, holders can be served side by side, no problem. Other tickets are not compatible, and a ticket holder has to wait for the incompatible one ahead of it to finish before it can be served.

Two plain SELECT statements are always compatible. They take the weakest kind of ticket there is, and a hundred of them can run at once without ever noticing each other. The trouble starts when someone joins the line holding the strongest possible ticket.

The Strongest Ticket Blocks Everyone, Even the Compatible Ones

PostgreSQL has a handful of table-level lock strengths, but only one really matters for this story. It's called ACCESS EXCLUSIVE, and it's acquired by ALTER TABLE, DROP TABLE, and TRUNCATE. It conflicts with every other lock in the system, including a plain read.

Here's the part that catches people off guard: PostgreSQL mostly serves lock requests in the order they arrive. So if an ALTER TABLE shows up and has to wait for one long-running SELECT to finish, it doesn't just wait quietly off to the side. It gets in line. And every SELECT that shows up after it also has to get in line, behind the ALTER TABLE, even though those new SELECT statements would have been perfectly happy running alongside the original one.

One slow reader plus one waiting migration is enough to freeze every future read on that table, until the slow reader finally lets go.

How Twelve Minutes Becomes an Outage

This is close to how it actually plays out in production. An analytics query kicks off against a busy table and, because nobody set a timeout on it, keeps running for twelve minutes. A few minutes in, someone ships a routine schema migration on that same table. The migration asks for its ACCESS EXCLUSIVE ticket and starts waiting on the analytics query.

From that moment, every new request the application sends to that table queues up behind the migration. Within half a minute, hundreds of connections are stuck waiting on a lock, not on any actual work. The connection pool, which was never designed to hold hundreds of idle-but-blocked connections, fills up completely. New requests can't even get a connection to wait with. The site goes down, and the root cause line in the postmortem is almost funny in how small it is: one long read, one migration with no timeout.

The Fix Is Two Settings, Not a Rewrite

The defense here isn't clever code, it's two timeouts that should be set before any schema change ever touches a live table.

The first tells the migration itself to give up quickly if it can't get its lock in a few seconds, rather than parking itself at the front of a queue that keeps growing. The second is a ceiling on how long any query is allowed to run at all, so a stray analytics query can never hold a lock for twelve minutes in the first place. Neither setting is exotic. Both are usually just missing.

Set correctly, the same migration either succeeds in under a second because nothing was in its way, or it fails immediately and cleanly so you can retry it a minute later. What you never want is the third option: waiting, silently, gathering a crowd behind it.

Row Locks Are a Separate, Smaller Story

Everything above is about locking the table's structure. There's a second, unrelated locking system for locking individual rows of data, and it solves a different problem: two transactions trying to change the same row at the same time.

The classic example is a balance check before a debit. Read the balance, confirm there's enough money, then subtract the amount. Without a row lock, two concurrent withdrawals can both read the same starting balance and both proceed, and the account ends up wrong. Locking that one row for the duration of the transaction closes the gap.

There's also a neat variant built for queues: instead of locking a row and making every other worker wait for it, a worker can ask to skip any row that's already locked and grab the next free one instead. That's the difference between ten background workers piling up behind each other and ten workers each picking up a different job at the same instant.

Deadlocks: When Two Correct Transactions Still Collide

Occasionally two transactions each hold a lock the other one needs. Transaction A has locked account 1 and wants account 2. Transaction B has locked account 2 and wants account 1. Neither can proceed, and neither will ever let go voluntarily. PostgreSQL notices this after about a second, picks one of the two transactions, and kills it so the other can continue.

The fix isn't a database setting, it's a coding discipline: always acquire locks in the same order, everywhere in the codebase. If every transfer function locks the lower account ID first regardless of direction, the two transactions above stop being able to collide at all. It's the same trick as everyone in a crowded hallway agreeing to keep to the right.

Locks With No Data Attached

PostgreSQL also offers a kind of lock that has nothing to do with any table or row: an application-defined lock, identified by whatever number you choose. It's mainly used for one thing, making sure a background job runs on only one server at a time, even when there are several application instances that could all try to start it. Grab the lock, run the job, and if the process crashes mid-job, the lock releases itself automatically instead of leaving things stuck. It's a lighter, more reliable substitute for the Redis-based mutex a lot of teams reach for by default.

The Takeaway

A lock isn't something you fight, it's a ticket you're standing in line with, and most locking outages don't start with a bad lock. They start with someone who forgot to set a timeout and ended up holding the line for everyone behind them.

Where This Fits in the Full Course

This is Module 13 of the PostgreSQL In-Depth course, from the Architect phase covering the internals senior engineers eventually have to learn the hard way. If this scenario feels familiar, the earlier modules on MVCC and transaction internals build the concurrency model this one depends on.

#PostgreSQL #Database #Backend #SQL #ProductionIncidents #SoftwareEngineering

When you run CREATE DATABASE, PostgreSQL creates a directory. When you run INSERT, it writes a row to a file and logs the write for crash safety. When you run SELECT, it reads that file back. That's the whole trick, once you see it, the "magic" disappears.

What the PostgreSQL Server Is Actually Doing

Most developers use PostgreSQL for years without ever forming a picture of what it's doing when a query runs. SQL goes in, rows come out, and everything in between feels like a sealed box. It isn't a sealed box. It's a program, running on a computer, reading and writing files, same as any other program you've written. This article builds that basic picture, in plain language, with no prior database knowledge assumed.

A Database Is a Directory

Run CREATE DATABASE learning_postgres and PostgreSQL doesn't do anything exotic. It creates a directory on disk to hold that database's data. That's it. Every database you create gets its own folder, sitting under PostgreSQL's data directory, the same way any application on your computer might create a folder to store its files.

This is worth sitting with, because it quietly answers a question that trips up a lot of people: "where does my data actually live?" It lives in a directory on the disk of whatever machine is running the PostgreSQL server. Not in the cloud in some abstract sense, not floating in "the database." On disk, in files, in a folder PostgreSQL controls.

It also explains something you may have noticed without thinking about it: CREATE DATABASE returns almost instantly, milliseconds, not minutes. That's because creating a database is just creating an empty directory. There's no data in it yet. The real work, and the real cost, only begins once rows start filling those files.

A Table Is a File, a Row Is Bytes in That File

Inside that directory, each table you create is backed by its own file (large tables get split across several, but the idea holds). When you run:

INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com');

PostgreSQL takes that row and writes it, as bytes, into the file backing the users table. Nothing more mysterious than that.

When you run:

SELECT * FROM users WHERE email = 'alice@example.com';

PostgreSQL opens that file (or consults an index that points into it, more on indexes in a later module), scans through the rows stored there, checks each one against your WHERE condition, and returns the ones that match.

Think of the file as a very disciplined spreadsheet that only PostgreSQL is allowed to touch directly. You never open it yourself, you always go through SQL, but structurally, that's what it is: rows, stored as bytes, in a file, on disk.

Under the hood, PostgreSQL is really answering a short chain of questions before it hands you an answer: Is the data I need already sitting in memory, or do I have to go to disk for it? Is there a shortcut, an index, that gets me there without reading the whole file? And once I've found a row, is it actually the current, correct version, or an old one nobody needs anymore? You don't write any of that logic. But knowing it happens is what lets you start reasoning about why one query is fast and an almost-identical one is slow.

PostgreSQL Is a Process, Not a Black Box

When you connect to PostgreSQL, whether from psql, a Node.js app, or a GUI tool, you're talking to a running process on a server. That process listens for your connection, hands you off to a dedicated worker just for your session, and from that point on, every query you send is handled by an ordinary program doing ordinary things: reading bytes from files, writing bytes to files, holding some of that data in memory so it doesn't have to touch the disk every single time.

That last part, keeping frequently-used data in memory, is why the second time you query something is usually much faster than the first. The process remembers what it recently read from disk and serves it from memory instead. This is the same idea as any caching you've done in application code, just built into the database itself.

This is also why a query can get slower for no reason you can find in your SQL. If PostgreSQL just restarted, deployed, crashed, rebooted, that memory is empty again. Nothing is cached yet. The first round of queries has to go back to disk to rebuild that cache from scratch, before things speed back up. If you've ever seen a database feel sluggish right after a restart and assumed you'd broken something, this is usually the real reason: the cache went cold, not your code.

The Log That Saves You From Crashes

Here's the one detail that separates "just writing to a file" from what a production database actually needs to guarantee: what happens if the power goes out, or the process crashes, in the middle of a write?

PostgreSQL's answer: before it changes the actual data file, it first writes down what it's about to do, to a separate log. Only after that note is safely on disk does it go ahead and make the real change.

If the server crashes mid-write, it doesn't matter, on restart, PostgreSQL reads that log and replays anything that didn't finish. It's the same instinct as keeping a to-do list before starting a task: if you get interrupted, you don't have to remember what you were doing, you just check the list. This log is one of the reasons people trust PostgreSQL with data they can't afford to lose.

Why This Mental Model Matters

Once "PostgreSQL is a process that reads and writes files, with a safety log protecting every write" is in your head, a lot of things that used to feel like separate pieces of magic start looking like variations on the same idea:

• Indexes are just extra files that let PostgreSQL find the right rows without scanning the whole table.
• Caching (shared buffers) is just PostgreSQL keeping recently-used file contents in memory.
• Replication is just another server reading that same safety log and replaying it, to keep a second copy of the data in sync.
• VACUUM is just PostgreSQL cleaning up old row versions it no longer needs, so the files don't grow forever.

None of these are separate systems bolted onto a mysterious core. They're all extensions of the same basic loop: read files, write files, log before you write, keep useful things in memory.

PostgreSQL isn't magic. It's a server process that manages files intelligently, and every advanced feature it has is that idea, applied more cleverly.

Where This Fits in the Full Course

This is the very first mental model from the Foundation phase of the PostgreSQL In-Depth course, the phase built for zero prior database knowledge. If this clicked, the next modules build on it directly: the client-server connection model in detail, the relational mental model (tables, rows, and how they connect), and your first complete schema.

For readers who want the advanced version of this same territory, process architecture, MVCC, the planner, replication internals, see "The PostgreSQL Elephant in the Room." That article assumes you already have the basic picture this one just gave you.

The Question Worth Asking Yourself

Next time a PostgreSQL query does something you don't expect, slow, fast, returning stale-looking data, try asking the simplest possible question first: what files is it reading or writing right now, and what would that look like from the outside?

You'll be surprised how often that question, on its own, points you toward the answer, long before you need to reach for anything more advanced.

#PostgreSQL #Database #Backend #SoftwareEngineering #DatabaseFundamentals #LearnToCode #SQL

Every promotion in tech has an invisible downside.

• When someone becomes an Intern, thousands of applicants don't.
• When someone becomes a Junior Developer, another candidate receives a rejection email.
• When someone becomes a Senior Developer, someone else waits another review cycle.
• When someone becomes a Tech Lead or Engineering Manager, dozens of equally ambitious engineers aren't selected.

That's the reality of a competitive industry.

But here's where many people get it wrong.

Success isn't about destroying people. It's about becoming the best choice.

You don't need to sabotage a colleague. You don't need office politics. You don't need to hope others fail.

You need to build skills that make the decision obvious.

• Learn continuously.
• Take ownership.
• Communicate clearly.
• Deliver consistently.
• Solve bigger problems than yesterday.

Competition is inevitable. Cruelty is optional.

The goal isn't to terminate careers. The goal is to become so valuable that opportunities naturally come your way.

In the end, the market doesn't reward the loudest engineer.

It rewards the one who consistently creates the most value.

Outperform the competition. Respect the competitors.

#SoftwareEngineering #CareerGrowth #Leadership #TechCareers #Engineering #ContinuousLearning #TechLeadership

The money left Alice's account. Bob never received it. The server crashed in between. If you don't understand transactions, this is how your application loses data, silently, permanently, with no error log.

Transactions and ACID in Practice: What Every Backend Developer Must Know

Here is the scenario that every backend developer eventually hits in production. A bank transfer: debit $100 from Alice, credit $100 to Bob. Two SQL statements. Simple.

The server crashes between them.

Alice has lost $100. Bob never received it. The money has vanished into the gap between two database writes, and your application has no idea it happened.

This is exactly the problem transactions were invented to solve. And yet, after years of reviewing code and debugging production incidents, I can tell you: most developers use transactions far less than they should, misunderstand what isolation levels actually do, and write patterns that hold locks for seconds longer than necessary.

This article covers the full mental model, BEGIN, COMMIT, ROLLBACK, all four isolation levels, SELECT FOR UPDATE, SKIP LOCKED for job queues, and the most common anti-patterns that silently destroy performance at scale. It's based directly on the Transactions & ACID in Practice module from the PostgreSQL In-Depth course.

The Problem Transactions Solve

Consider the transfer code without transactions:

UPDATE accounts SET balance = balance - 100 WHERE id = 1; UPDATE accounts SET balance = balance + 100 WHERE id = 2;

If anything interrupts execution between those two lines, a server crash, an application exception, a network timeout, Alice's account has been debited but Bob's has not been credited. The database is now in an inconsistent state, and it has no knowledge that anything went wrong.

Transactions fix this by grouping operations into an atomic unit: either all of them succeed, or none of them do.

BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = 1; UPDATE accounts SET balance = balance + 100 WHERE id = 2; COMMIT;

If the server crashes after the first UPDATE and before COMMIT, PostgreSQL automatically rolls back the entire transaction on restart. Alice keeps her $100. The database never reaches a half-updated state.

What ACID Actually Means

ACID is not a marketing acronym. Each letter describes a specific guarantee the database makes, and understanding each one changes how you write application code.

Atomicity means all operations in a transaction succeed or none do. There is no partial success. If your transaction debits Alice and the next statement fails, the debit is undone.

Consistency means a transaction takes the database from one valid state to another. Constraints, NOT NULL, FOREIGN KEY, CHECK, are enforced at commit time. If you have a CHECK (balance >= 0) constraint and a debit would push a balance negative, the entire transaction fails at COMMIT and rolls back automatically. The database enforces your business rules, not just your application code.

Isolation means concurrent transactions run as if they are the only transaction in the system. Another session's uncommitted changes are invisible to you. If Session A is updating a row and hasn't committed yet, Session B reads the pre-update value, not the in-progress value.

Durability means once committed, data survives crashes. PostgreSQL's Write-Ahead Log (WAL) ensures this, every committed transaction is written to durable storage before the COMMIT acknowledgement is returned to the client.

The Three Transaction Commands

BEGIN starts a transaction. Without it, every statement is its own transaction, it auto-commits immediately. This is fine for single statements, dangerous for multi-step operations.

COMMIT makes all changes permanent and releases locks.

ROLLBACK undoes all changes since BEGIN and releases locks. If your session disconnects before COMMIT, PostgreSQL automatically rolls back.

In application code (Node.js example):

const client = await pool.connect();
try {
  await client.query('BEGIN');
  await client.query('UPDATE accounts SET balance = balance - $1 WHERE id = $2', [100, 1]);
  await client.query('UPDATE accounts SET balance = balance + $1 WHERE id = $2', [100, 2]);
  await client.query('COMMIT');
} catch (err) {
  await client.query('ROLLBACK');
  throw err;
} finally {
  client.release();
}

The finally block is critical, always release the connection back to the pool, whether the transaction succeeded or failed.

One thing many developers don't realise: once an error occurs inside a transaction in PostgreSQL, the transaction is aborted. Every subsequent statement in that transaction will fail with "current transaction is aborted, commands ignored until end of transaction block", until you issue a ROLLBACK. This has caught many teams off guard in production.

The Four Isolation Levels

PostgreSQL supports four isolation levels. Understanding the difference is not academic, choosing the wrong one causes data integrity bugs that are extremely hard to debug.

READ COMMITTED (the default), each query in your transaction sees a fresh snapshot of committed data at the time that query runs. This means two identical SELECT queries in the same transaction can return different results if another transaction commits between them. This is called a non-repeatable read, and it is expected behaviour at this level. For most application work, READ COMMITTED is correct.

REPEATABLE READ, the entire transaction sees the same snapshot of data as of when the transaction started. If another session inserts 100 rows and commits while your transaction is open, your subsequent queries still see the original row count. Use this when you need consistency across multiple queries in a single transaction, for example, generating a financial report where all queries must reflect the same point in time.

SERIALIZABLE, the strongest level. Transactions execute as if they were serialised one after another. PostgreSQL detects situations where concurrent transactions could produce results different from any serial execution and fails one of them with a serialization error. Use this sparingly, it adds overhead and requires retry logic in your application.

READ UNCOMMITTED, in theory this allows dirty reads (reading uncommitted data from other transactions). In practice, PostgreSQL treats it identically to READ COMMITTED. Postgres never allows dirty reads, regardless of isolation level.

The rule for most applications: use READ COMMITTED for everything. Move to REPEATABLE READ only when you genuinely need a consistent multi-query snapshot. Use SERIALIZABLE only for complex financial operations where you've reasoned through why lower isolation levels produce incorrect results.

SELECT FOR UPDATE: Pessimistic Locking

Here is a classic race condition. Two workers both check an account balance, both decide it's sufficient, and both proceed to debit, leaving the balance negative despite a check constraint.

The naïve code:

SELECT balance FROM accounts WHERE id = 1; -- application checks: if balance >= 100, proceed UPDATE accounts SET balance = balance - 100 WHERE id = 1;

Between the SELECT and the UPDATE, another transaction can change the balance. Your check was valid when you made it and invalid when the UPDATE runs.

The fix is SELECT FOR UPDATE, which locks the row at read time:

BEGIN; SELECT balance FROM accounts WHERE id = 1 FOR UPDATE; -- Row is now locked. No other transaction can modify it until we commit. -- Application checks balance, decides to proceed UPDATE accounts SET balance = balance - 100 WHERE id = 1; COMMIT;

Any other transaction trying to SELECT FOR UPDATE or UPDATE the same row will wait until you commit or rollback. The row is protected for the entire check-then-modify sequence.

FOR UPDATE SKIP LOCKED: The Job Queue Pattern

FOR UPDATE SKIP LOCKED is one of the most useful concurrency primitives PostgreSQL offers, and it is almost unknown outside of experienced backend teams.

The scenario: a table of background jobs, multiple workers running concurrently, each trying to claim and process the next available job. Using plain FOR UPDATE causes workers to queue up waiting for the same row lock, defeating the purpose of concurrency. Using no locking at all causes multiple workers to claim the same job.

SKIP LOCKED solves this exactly:

BEGIN; SELECT id, payload FROM jobs WHERE status = 'pending' ORDER BY created_at LIMIT 1 FOR UPDATE SKIP LOCKED;

UPDATE jobs SET status = 'processing', worker_id = $worker WHERE id = $id; COMMIT;

Each worker instantly skips rows that are already locked by another worker and claims the next available one. No blocking, no duplicate processing. This is the correct pattern for any queue-based workload in PostgreSQL.

The Most Expensive Mistake: Long Transactions

This is the pattern I see most often in production codebases, and it causes damage that compounds over time.

// ❌ Anti-pattern: keeping a transaction open during a network call
await client.query('BEGIN');
const orders = await client.query("SELECT * FROM orders WHERE status = 'pending'");

// Make a payment API call, takes 2-5 seconds
const result = await paymentGateway.charge(orders.rows[0]);

await client.query('UPDATE orders SET status = $1 WHERE id = $2', ['confirmed', orders.rows[0].id]);
await client.query('COMMIT');

What's happening during those 2-5 seconds:

• Locks are held on every row the transaction has touched.
• Other transactions trying to modify those rows are blocked.
• PostgreSQL cannot vacuum dead tuples from any table involved in the transaction, across the entire database, not just the rows you locked.
• Connection pool slots are occupied for the full duration.

At low traffic, this is invisible. At 100 concurrent requests with 3-second payment API calls, you have 100 transactions holding locks simultaneously. Autovacuum falls behind. Tables bloat. Queries slow down. The cause is nearly impossible to identify without knowing what to look for.

The correct pattern:

// ✅ Query outside the transaction
const orders = await pool.query("SELECT * FROM orders WHERE status = 'pending'");

// Do the slow external work, no locks held
const result = await paymentGateway.charge(orders.rows[0]);

// Open a short transaction only for the write
const client = await pool.connect();
try {
  await client.query('BEGIN');
  await client.query('UPDATE orders SET status = $1 WHERE id = $2', ['confirmed', orders.rows[0].id]);
  await client.query('COMMIT');
} catch (err) {
  await client.query('ROLLBACK');
  throw err;
} finally {
  client.release();
}

Transaction open for milliseconds, not seconds. Lock held for milliseconds. Autovacuum unaffected.

SAVEPOINT: Partial Rollback

A SAVEPOINT lets you roll back part of a transaction without abandoning the whole thing. This is useful when you have a series of operations where one failing step should be retried or skipped, not the entire transaction.

BEGIN;

INSERT INTO orders (customer_id, total) VALUES (1, 150.00);

SAVEPOINT before_items;

INSERT INTO order_items (order_id, product_id, quantity) VALUES (1, 999, 1); -- Fails: product 999 doesn't exist

ROLLBACK TO SAVEPOINT before_items; -- The order INSERT is preserved. The failed order_item INSERT is undone.

INSERT INTO order_items (order_id, product_id, quantity) VALUES (1, 2, 1); -- Correct product this time

COMMIT; -- Order + 1 item committed. The failed attempt left no trace.

Where This Fits in the Full Course

The patterns above, along with isolation level trade-offs, deadlock prevention, and the WAL durability mechanics behind durability, are one module of the PostgreSQL In-Depth course at academy.jatinjainsaraf.com.

The course covers three phases: Foundation (absolute basics for beginners, zero assumed knowledge), Practitioner (the patterns working engineers use every week, indexes, schema design, JSONB, full-text search, performance tuning), and Architect (MVCC internals, WAL, autovacuum mechanics, replication, partitioning, and connection pooling failure modes).

Every module is built from years of running PostgreSQL on a live blockchain indexer whose database grew to 10 TB. The examples in this article are from real production scenarios, not documentation.

The Questions Worth Asking About Your Own Codebase

Do any of your API endpoints hold a database transaction open while making an external HTTP call? Even one that takes 200ms?

Do your background job workers use FOR UPDATE SKIP LOCKED, or are they competing for the same row lock?

What happens in your application when a transaction fails mid-way? Does your error handling issue ROLLBACK and release the connection, or does it leave an aborted transaction sitting in the pool?

Do you know which isolation level your most critical queries are running at, and whether that level is actually appropriate for what those queries are doing?

These are not theoretical questions. Each one has a production failure mode attached to it. The good news is that once you have the mental model, the fixes are straightforward.

#PostgreSQL #Database #Backend #SQL #SoftwareEngineering #DatabaseEngineering #Transactions #ACID #NodeJS #BackendDevelopment

You've earned trust. Your instincts are sharp. People come to you before committing to a design. Before merging a PR. Before choosing a database. Before making a call.

And slowly, without noticing it, you stop being an engineer and start being a gate.

The trap looks like success.

You're being consulted because your judgment is valued. You're in every important meeting because your perspective matters. You're the person who "gets it."

But here's what's actually happening:

PRs sit waiting for your review. Decisions stall until you're available. Engineers stop thinking through options, because they'll just ask you anyway. Your calendar fills with meetings that exist to get your approval.

The team isn't scaling. It's depending.

This is what judgment-as-bottleneck looks like:

1. Velocity is tied to your availability, not the team's capacity
2. Junior engineers learn your answers, not your reasoning
3. Decisions that could have been made in an hour wait three days
4. You feel indispensable, but the system is actually fragile

The irony? The more you care about quality, the more likely you are to accidentally create this pattern.

The fix isn't doing less. It's building judgment transfer.

Share the why behind decisions, not just the decision. Write down the heuristics you use, the instincts you've built over years deserve to be externalised. Define decision boundaries: "You own anything under this scope. Come to me only when it crosses these lines." Let people make calls you'd have made differently, then debrief rather than override.

The best engineering leaders I've seen don't protect quality by owning every decision. They protect it by raising the quality of how the team decides.

Your judgment shouldn't be the ceiling. It should be the foundation.

When you make yourself the bottleneck, even unintentionally, you're not protecting quality. You're just delaying it.

The real work isn't being right every time. It's building a team that can be right when you're not in the room.

💬 Have you ever caught yourself becoming the bottleneck, or working under one?

#EngineeringLeadership #TechLead #SoftwareEngineering #LeadershipMindset #TeamScaling

Let's dive into some practical tips for keeping your PostgreSQL database lightning fast!

1. Understand Your Queries with EXPLAIN ANALYZE

Before you can optimize a slow query, you need to know why it's slow. PostgreSQL provides an incredibly powerful tool for this: EXPLAIN. By prepending EXPLAIN ANALYZE to your query, Postgres won't just run the query—it will give you a detailed execution plan showing exactly how it fetched the data.

EXPLAIN ANALYZE 
SELECT * FROM users WHERE last_login > '2023-01-01';

Look for "Seq Scan" (Sequential Scan) in the output. If you see it on a large table, it means Postgres is scanning every single row to find your data. This is your cue that an index might be needed!

2. Index Smartly (But Don't Over-Index)

Indexes are the most common way to speed up read queries. The B-Tree index is the default and works perfectly for equality and range queries.

CREATE INDEX idx_users_last_login ON users(last_login);

However, a common beginner mistake is indexing every column. Remember that every time you INSERT, UPDATE, or DELETE a row, Postgres has to update the indexes too. Too many indexes will slow down your write operations. Only index columns that are frequently used in WHERE, JOIN, or ORDER BY clauses.

3. Leverage the Power of JSONB

One of PostgreSQL's most beloved features is its native support for JSON. But did you know there are two types: json and jsonb?

Always default to jsonb. While json stores an exact copy of the input text, jsonb stores the data in a decomposed binary format. This makes it slightly slower to insert, but significantly faster to process. Even better, you can index jsonb fields!

-- Creating a GIN index on a jsonb column
CREATE INDEX idx_user_metadata ON users USING GIN (metadata);

4. Vacuum Regularly

PostgreSQL uses Multiversion Concurrency Control (MVCC) to handle simultaneous transactions. When you update or delete a row, Postgres doesn't immediately remove the old version—it marks it as a "dead tuple". Over time, these dead tuples cause database bloat.

The VACUUM process cleans up these dead tuples. While Postgres has an autovacuum daemon that handles this automatically, heavily updated tables might need customized autovacuum settings. Keep an eye on your dead tuple counts; if they grow too large, performance will degrade.

5. Use Connection Pooling

Every new connection to PostgreSQL spins up a new OS process, which consumes around 10MB of memory. If your web application opens a new connection for every request, you'll quickly exhaust your server's RAM and CPU.

Instead, use a connection pooler like PgBouncer or Pgpool-II. These tools maintain a pool of active database connections and share them among your application's requests, drastically reducing the overhead on the database server.

Conclusion

PostgreSQL is a massive, feature-rich database engine. While it works beautifully out of the box, understanding how it executes queries, manages memory, and handles concurrency will elevate you from a simple user to a database master.

Try running EXPLAIN ANALYZE on your slowest queries today—you might be surprised by what you find!

If you use a traditional Set data structure to store every unique ID, tracking millions of users will rapidly consume Gigabytes of RAM. This is where Redis HyperLogLog comes to the rescue.

What is HyperLogLog?

HyperLogLog (HLL) is a probabilistic data structure used to estimate the cardinality of a set (the number of unique elements). Instead of storing the actual elements, it hashes them and observes the patterns of the binary representation of the hash.

The magic of Redis' implementation of HyperLogLog is that it can estimate the cardinality of millions of unique items while always using a maximum of 12 KB of memory and maintaining a standard error of just 0.81%.

How it works (The Math made simple)

Imagine you are flipping a coin. If you flip it and get "Heads" 5 times in a row, you'd intuitively know you've probably been flipping that coin for a while.

HLL works on a similar principle using hashing:

1. Every item added to the HLL is hashed into a large binary string (e.g., 101100101...).
2. The algorithm looks for the longest sequence of leading zeros in these binary strings.
3. If the longest sequence of leading zeros across all hashes is N, the estimated number of unique elements is roughly 2^N.

To reduce variance and make the estimate incredibly accurate, Redis divides the data into 16,384 internal registers and averages the results using harmonic means. This brings the memory footprint to exactly 16384 × 6 bits = 12 KB.

Practical Usage in Redis

Using HLL in Redis is delightfully simple. It provides three main commands:

1. PFADD

Adds elements to the HyperLogLog.

PFADD website_visitors:2026-06-06 "user_102" "user_883" "user_911"
# Returns 1 if the internal register was altered

2. PFCOUNT

Returns the approximated cardinality.

PFCOUNT website_visitors:2026-06-06
# Returns: 3

3. PFMERGE

Merges multiple HyperLogLogs into a single one. This is perfect for rolling up daily metrics into weekly or monthly metrics!

PFMERGE visitors:this_week visitors:monday visitors:tuesday visitors:wednesday

When to use HyperLogLog

HyperLogLog is not a silver bullet. You should use it when:

• You need to count massive amounts of unique items (IP addresses, user IDs, search queries).
• You care about memory efficiency more than absolute 100% precision.
• You do not need to retrieve the actual items back from the data structure.

If you need to list the users, or if absolute precision is required for billing purposes, stick to a standard Redis Set or an SQL database. But for analytical dashboards and scale, HyperLogLog is an absolute superpower.

Microservices don’t collapse because they’re “too complex.”

They collapse because teams slowly recreate a monolit, just distributed.

On diagrams, everything looks clean: Neat boxes. Clear boundaries. Perfect arrows.

In production? That’s where the truth shows up.

Services calling each other in long synchronous chains. Retries multiplying failures instead of absorbing them. Shared databases quietly coupling releases. Gateways absorbing business logic. Timeouts missing. Ownership unclear.

Nothing fails dramatically at first.

It just gets slower. Harder to change. Riskier to deploy.

And then one day, a small issue becomes a cascading outage.

Microservices demand discipline more than architecture.

They require:

– Clear domain boundaries – Explicit ownership – Versioning strategy – Observability by default – Thoughtful retry + timeout policies – An actual consistency model – Automation everywhere

Without that, you don’t have microservices.

You have a distributed monolith with extra latency.

Microservices don’t need more services. They need better engineering judgment.

Architecture doesn’t fail on paper. It fails in habits.

#Microservices #SystemDesign #SoftwareArchitecture #DistributedSystems #BackendEngineering #EngineeringLeadership #DevOps

Professionalism Isn’t Rigidity ==> It’s Responsibility

In every organisation, work flows through multiple people, teams, and systems. And whenever that happens, one principle becomes essential:

Clear communication isn’t bureaucracy. It’s accountability.

Documenting a decision in the right channel, informing the relevant teams, or keeping updates transparent, it’s not about “being rigid” or “too formal.” It’s about ensuring that: • Nothing gets lost, • No one gets blindsided, and • Everyone has the same context.

In tech especially, a small correction in the wrong place can trigger a big impact elsewhere. A quiet change might fix a momentary problem, but a documented change prevents future ones.

Professionalism doesn’t mean saying “no.” It means saying “let’s make sure this is visible so the whole team stays aligned.”

Because when work affects users, systems, or data… clarity is not optional, it’s responsibility.

💬 What’s one communication practice you feel every team should follow?

#WorkCulture #Professionalism #TeamCommunication #EngineeringMindset

🧠 Ownership Mindset: Thinking Beyond Assigned Tasks

One thing I’ve learned in engineering (and honestly, in life) is this:

Anyone can complete tasks. Very few take ownership.

Ownership isn’t about working more. It’s about thinking deeper.

It means asking questions like:

“What is the real goal behind this task?” “Is there a better or faster way to achieve it?” “What could break later if we don’t address it now?” “Is this decision aligned with the bigger picture?”

When you operate with an ownership mindset, your work shifts from executing tickets to building outcomes. You stop thinking, “This part isn’t my responsibility,” and start thinking, “How can I make this better for the team, the users, and the system?”

Signs you’re moving from task-based to ownership-based thinking:

You anticipate problems before they show up. You fill gaps even if they’re “not assigned to you.” You communicate proactively instead of reactively. You care about long-term impact, not just short-term delivery. You look at systems holistically, not just your module.

In a world full of skilled developers, the ones who stand out are not just good coders, they’re good owners. They treat the product like something they built, not something they were merely told to work on.

If you want to grow faster, gain trust, and build influence… Start thinking beyond your tasks. Start thinking like an owner.

#OwnershipMindset #Leadership #SoftwareEngineering #CareerGrowth #ProblemSolving #TechMindset #EngineeringCulture #ProfessionalGrowth #Responsibility #MindsetMatters #DevelopersJourney #Teamwork #WorkEthic #ProductThinking

🚨 The React Server Components Vulnerability That Shocked the Web Ecosystem (Dec 2025)

In the last few weeks, the React community faced one of its most serious security incidents, a set of vulnerabilities in React Server Components (RSC) that exposed thousands of production systems to Remote Code Execution (RCE), source-code leakage, and DoS attacks.

Not just another CVE. A wake-up call for every team using modern full-stack React frameworks.

🔥 What Actually Happened? In December 2025, researchers uncovered critical issues in the react-server-dom-* packages used by React 19 and frameworks like Next.js:

1️⃣ CVE-2025-55182: Remote Code Execution (Critical) Attackers could trigger server-side code execution through crafted RSC requests, without authentication. This enabled full server takeover, malware deployment, data extraction, and infrastructure compromise. It was actively exploited by advanced threat groups.

2️⃣ CVE-2025-55184: Denial of Service Malformed payloads could crash RSC handlers and bring applications down instantly.

3️⃣ CVE-2025-55183: Source Code Exposure Some setups mistakenly returned server function source code to clients, exposing business logic and internal secrets.

🌍 Global Impact The impact was widespread: 🔹 600,000+ domains were found vulnerable, from startups to enterprise SaaS. 🔹 Exploitation was confirmed against cloud apps, fintech dashboards, admin portals, and misconfigured Next.js deployments. 🔹 Next.js App Router apps were hit hardest, since Server Actions expanded the execution surface. 🔹 Cloud providers issued emergency advisories, rolled out WAF rules, and scanned hosted projects. 🔹 Many engineering teams had to rotate secrets, rebuild servers, and audit logs, because an RCE incident leaves long-term risk even after patching.

🛡️ How to Mitigate Right Now

If you use React Server Components (React 19, Next.js App Router, or any react-server-dom-* package):

✔️ Update to patched RSC versions — 19.0.1 / 19.1.2 / 19.2.1+ ✔️ Update your framework (Next.js, Remix, etc.) to their security releases ✔️ Restrict public access to server actions where possible ✔️ Add WAF rules to block malicious RSC payloads ✔️ Audit logs since Nov 1 for: • Unknown restarts • Suspicious outbound calls • Unexpected files on disk

💬 Final Thought

React Server Components are powerful, but this incident highlighted a deeper truth: When JavaScript becomes your backend, your frontend developers become part of your security surface. This vulnerability shook the ecosystem, but it also pushed the community toward safer patterns and stronger security awareness. If your application uses React or Next.js in production, patching isn’t optional; it’s urgent.

#ReactJS #Nextjs #Security #WebSecurity #JavaScript #DevSecOps #EngineeringLeadership #FullStackDevelopment

A Redis command that quietly solves one of the hardest problems in high-concurrency systems.

I was diving into the Redis documentation recently and found myself appreciating a highly powerful command: HINCRBYFLOAT.

If you're building real-time analytics, IoT data pipelines, or ad-tech platforms, this command is an absolute lifesaver. It allows you to target a specific field inside a Redis Hash and increment (or decrement, with negative numbers) its decimal value in a single step.

Why It's a Game-Changer for High-Concurrency Systems

1️⃣ 100% Atomic Operations

It completely eliminates the risky "Read-Modify-Write" cycle in your application logic. Redis handles the math in-memory, ensuring zero race conditions even under millions of simultaneous requests.

2️⃣ Automatic Upserts

No need to check if the data exists first. If the hash or field doesn't exist, Redis initializes it at 0 and applies the increment on the fly.

3️⃣ Massive Performance Gain

It reduces network round-trips and complex database locking mechanisms down to a single, lightning-fast line of code.

The Engineering Catch ⚠️

Because it relies on standard IEEE 754 double-precision floating-point math, it can introduce those classic, tiny decimal rounding errors over time (think 0.30000000000000004). It also strips trailing zeros, returning 5.00 as "5".

The Takeaway

For high-frequency aggregations, dashboards, and sensor metrics? HINCRBYFLOAT is perfect.

But if you're dealing with precise ledger data or financial transactions where every micro-penny counts, the industry gold standard is still to multiply your values by 100, store them as integers (cents), and stick to the classic HINCRBY.

It's a great reminder that every powerful tool comes with architectural trade-offs!

Tags: #SoftwareEngineering #Redis #BackendDevelopment #SystemDesign #Database #Caching

The Impact of Words in Professional & Personal Spaces

Words have more weight than we realise, especially when used as “jokes.”

A joke should make the person involved smile, not shrink. But if someone feels stressed, low, or embarrassed because of it…

Then it’s not humour. It’s harm wrapped in laughter.

And context matters more than we think:

Something said lightly between two people can feel very different when repeated in a group. The intention may stay the same, but the impact can change completely.

In professional and personal spaces, we often underestimate how far our words travel and how deeply they can affect someone.

Choosing our words with awareness isn’t just communication; It’s leadership, respect, and humanity.

🌱 The Responsibility Behind Honest Feedback

Feedback is often spoken about as a tool for improvement. But rarely do we talk about the responsibility that comes with giving it.

A performance review isn’t just a formality. It carries weight, real career, emotional, and developmental weight. A single written sentence has the power to uplift someone, redirect them, or hold up an uncomfortable mirror they’ve avoided for too long.

That’s why honest feedback must be delivered with two things: integrity and intention.

Integrity: because hiding the truth helps no one. Intention: because the purpose is growth, not punishment.

Sugarcoating delays development. Harshness without clarity destroys confidence. But constructive, fact-based feedback, even when difficult, is an investment in someone’s future ability.

The real challenge isn’t writing the review. It’s carrying the awareness that those words influence: • someone’s career trajectory • someone’s self-belief • someone’s next year of effort • and sometimes, someone’s livelihood

This is why feedback needs to be grounded, specific, accountable, and compassionate, not emotional, exaggerated, or careless.

When influence is handled responsibly, it guides. When it isn’t, it wounds.

In every organization, feedback isn’t just an HR requirement. It’s a leadership moment, one that quietly shapes people long after the meeting ends.

#LeadershipWisdom #WorkplaceEthics #ResponsibleLeadership #ProfessionalGrowth

What nobody tells you about hitting 10 TB of indexed blockchain data, and why “just optimize it later” is the most expensive sentence in…

Your Blockchain Indexer Is Fine — Until It Isn’t

What nobody tells you about hitting 10 TB of indexed blockchain data, and why “just optimize it later” is the most expensive sentence in this industry.

The first year of building a blockchain indexer is deceptively smooth.

Blocks arrive. Transactions get parsed. Wallets accumulate state. Your PostgreSQL instance hums along. Queries return in milliseconds. Storage is cheap. The team ships features.

Then year two happens. Then year three.

Somewhere between 1 TB and 5 TB, your infrastructure bill starts a quiet conversation with your CTO. By 10 TB, which a high-throughput chain reaches within a few years of production indexing, that conversation gets loud and you are no longer talking about features. You are talking about survival: how do you change anything in a live system of this size without taking it down, corrupting data, or spending six months on a migration that should have taken three weeks?

This is that article. Not the “here’s how to build an indexer” tutorial. The one about what happens after — when the data is already there, it’s already the wrong shape, and the chain doesn’t stop producing blocks while you figure it out.

What Makes Move Different at the Data Layer

Before talking about scale, it’s worth understanding why Move-based blockchains (Aptos, Sui, Supra, and others) create a fundamentally different indexing problem than EVM chains.

EVM state is shallow. An Ethereum indexer primarily deals with: blocks, transactions, logs, and account balances. The schema is wide but not deep. Relationships are relatively flat.

Move state is deep and typed. Move organises on-chain state as typed resources attached to addresses. A single wallet doesn’t just have a balance — it can own dozens of distinct resource types simultaneously: coin stores, object collections, automation registrations, governance positions, staking records. Each resource type has its own structure. Each needs to be indexed, tracked, and queryable.

Move has a first-class object model. In chains like Sui and Supra, everything is an addressable object with an owner, a version, a type path, and a parent-child relationship to other objects. Your indexer isn’t just recording what happened — it’s maintaining a live, queryable snapshot of an entire object graph that mutates with every block.

Events are strongly typed. Move’s event system emits structured, typed payloads — not raw byte arrays like EVM logs. This is powerful for developers querying the indexer, but it means your event storage needs to handle variable schemas across dozens of event types, each with different fields, all arriving in the same stream.

The consequence of all this: a production Move indexer has far more tables, far more relationships, and far more cross-references than an EVM equivalent. A realistic production schema has 35–50 tables. Most of them are interconnected. Most of them grow with every block.

That scale of interconnection is exactly what makes optimisation so painful later.

A Note Before We Go Further

The patterns described in this article were not born from carelessness — they were the sensible choices given what was known at the time. The purpose of laying them out here is not to assign blame, but to share the experience of running into them at scale, so engineers building similar systems today can recognize them early and make the tradeoffs consciously rather than discover the cost years later.

The Compounding Problem: Why Early Decisions Are Permanent

Here is the core tension of blockchain indexer design: you make your schema decisions when you have zero data, but you live with those decisions when you have 10 TB of it.

Every structural inefficiency you introduce on day one doesn’t just cost you N bytes per row. It costs you N bytes multiplied by every row ever written, across every table that inherited the pattern, indexed twice over because PostgreSQL also stores your mistakes in its B-tree indexes.

Let’s make this concrete.

The Repeated String Problem

Imagine you store a short identifier — say, the name of the network environment (“mainnet”, “testnet”) — as a VARCHAR column in every table. Seems harmless. It's just a word.

Now imagine that string is present in 40 tables. In every row of every one of those tables. At 10 billion total rows distributed across those tables — entirely realistic after three years on a fast chain — that one string, repeated, costs you tens to hundreds of gigabytes depending on its length.

But here’s where it gets worse: you can’t just ALTER TABLE your way out of this at 10 TB. An ALTER TABLE that changes a column type on a 2-billion-row table doesn't complete in minutes. It locks the table. It rewrites the heap. It takes hours or days. And you have 40 tables to fix. And the chain is still producing blocks.

The “right” solution — replacing the repeated string with a 2-byte integer foreign key — would have cost 30 minutes on day one. At year three, it costs you a multi-month migration project with significant operational risk.

The Fat Primary Key Problem

A transaction hash is typically 66 characters. A timestamp is 8 bytes. If your transaction table’s primary key is a composite of these plus an environment name, you’ve created a ~90-byte primary key.

That primary key gets copied — physically, on disk — into every foreign key column of every child table. A transaction-heavy Move indexer has 10 or more child tables per transaction: events, state changes, coin transfers, fungible asset movements, sender records, fee payer records, and payload details. Each one stores that full 90-byte reference on every row.

At 10 billion transactions with 8 child rows each on average:

90 bytes × 10 child tables × 8 rows × 10B transactions
= ~72 TB of foreign key storage alone

And PostgreSQL builds a B-tree index on every foreign key. That doubles it.

A 4-byte or 8-byte surrogate integer primary key — decided on day one — reduces this by a factor of 10 to 20. But retrofitting surrogate keys onto live tables with hundreds of billions of rows, active foreign key constraints, and applications reading those columns in real time is one of the hardest classes of database migrations that exists.

The Wrong Type Problem

Move’s RPC layer returns numeric values as strings because they can exceed JavaScript’s safe integer range. blockHeight: "18000000". sequenceNumber: "4294967298". version: "999999999".

Indexer developers, working fast, copy this directly to the database: blockheight VARCHAR(256). version VARCHAR(10). sequencenumber VARCHAR(255).

It works. For a while.

Then someone tries to write WHERE blockheight BETWEEN 15000000 AND 16000000. Postgres does lexicographic comparison on VARCHAR. The query returns wrong results. A bug is filed. A cast is added. The cast prevents index use. The query slows down. An index is added on CAST(blockheight AS BIGINT). Now you have a functional index instead of a simple column index, which is harder to maintain and slower to build.

At 2 billion blocks, changing VARCHAR(256) to BIGINT requires a full table rewrite. At 10 TB, that's a weekend project, not a 10-minute fix.

JSONB: The Tool That Becomes the Problem

JSONB is genuinely useful for a Move indexer. Move resources have variable schemas. Event payloads differ by type. Storing semi-structured data in a relational database without JSONB means either a hundred narrow tables or an unmaintainable EAV design.

But JSONB has a cost that surprises people who haven’t run it at scale.

Storage overhead is fixed regardless of content. A JSONB column storing {} still has ~20–30 bytes of binary representation overhead. When you have millions of rows with small or empty JSON blobs, that overhead dominates.

Updates are expensive. PostgreSQL stores JSONB as an immutable value. When you update a JSONB column — even to change a single key — the database writes a completely new version of the entire document and marks the old one dead. For tables that update JSONB columns frequently (like wallet state, which changes on every relevant transaction), this creates significant table bloat over time. The dead tuples accumulate. VACUUM can’t always keep up. The table grows even if the actual data size doesn’t.

GIN indexes on JSONB are large and slow to build. A GIN index on a full JSONB column is many times larger than a B-tree index on a scalar column. On a table with billions of rows, building or rebuilding a GIN index is a multi-hour operation that consumes significant I/O.

The failure mode at scale is not that JSONB stops working. It’s that it gradually degrades — updates get slower, VACUUM runs longer, storage grows faster than the data warrants — and diagnosing the root cause in a 10 TB database with dozens of tables requires significant forensic work.

The Migration Problem Is the Real Problem

Everything described above has a known correct solution. Surrogate keys. Proper numeric types. Selective JSONB use. Hash storage as binary instead of hex strings. These are not novel ideas.

The reason they matter — the reason this article exists — is not the storage cost in isolation. It’s what that storage cost means when you try to fix it three years into production.

You cannot take a blockchain indexer offline to migrate it. The chain does not pause. Blocks arrive every second. If your indexer falls behind, you create a gap in your data that is either expensive to backfill or permanently lost, depending on your node’s pruning settings.

Schema migrations at this scale require orchestration, not just SQL. You cannot ALTER TABLE indexer_transaction ADD COLUMN transaction_id BIGSERIAL on a 3-billion-row table and move on with your day. You need to: add the column with no constraint, backfill in batches during low-traffic windows, add the constraint as NOT VALID, validate it in a separate step, update all application code to write to both old and new columns during the transition, cut over reads, then drop the old column. Each step has rollback implications. Each step must be tested against a full-size copy of production data.

Foreign key changes require cascade planning. If you want to replace a composite FK with a surrogate integer FK, you need to: add the surrogate column to the parent, backfill it, add it to all child tables, backfill it there, drop the old FK constraints, add new ones, drop the old columns. For 10 child tables, each with billions of rows, this is months of careful work — not because any individual step is hard, but because the sequence must be correct and live traffic must be handled at every stage.

Indexes cannot be rebuilt without significant I/O impact. Replacing a large text-based index with a smaller integer-based index on a table that’s being actively written requires CREATE INDEX CONCURRENTLY — which takes the load without locking, but takes proportionally longer and puts sustained read pressure on the disk. On a busy 10 TB database, a single large index rebuild can take 12–24 hours.

This is the real complexity of running a production Move indexer at scale. Not any single technical problem, but the compounding interaction of many small early decisions that each seemed reasonable at the time — and the enormous operational cost of unwinding them under load, without downtime, while the chain keeps running.

What Good Design Buys You

The flip side of all this is worth stating clearly.

A team that makes the right schema decisions at the start — surrogate integer keys, correct numeric types, binary hash storage, selective JSONB use, properly normalised repeated strings — doesn’t just save storage. They buy themselves optionality.

At 10 TB with a well-designed schema, adding a new index takes hours, not days. Migrating a table to a new structure is a weekend project, not a quarter-long initiative. Backfilling new columns is a batch job, not a rewrite. Query optimisation is about access patterns, not fighting the type system.

The teams that build Move indexers correctly from the start can spend year three shipping new features and supporting developers building on their data. The teams that don’t spend year three fighting their own database.

That’s the real cost of getting it wrong. Not the terabytes. The lost time.

Closing Thought

If you are designing a Move blockchain indexer today — or evaluating one — the questions worth asking are not just “does it index all transaction types?” or “what’s the query latency?” They are:

• What does the schema look like at 100× current data volume?
• How long does it take to add a column to your largest table?
• Can you change a primary key structure without taking the system down?
• How do you handle schema migrations under continuous write load?

The answers reveal a lot about how much technical debt is already priced into the system — and how much of your future engineering capacity it will consume.

Blockchain data is permanent and ever-growing. Your schema decisions are not quite permanent — but at 10 TB, they’re close enough that the difference barely matters.

The Move ecosystem is still early, and the tooling for large-scale indexer operations is still maturing. If you are building in this space and want to compare notes, the comments are open.

By Jatin Jain Saraf on May 5, 2026.

In a high-performance database environment, how we manage storage is just as important as how we write our queries. To keep our systems…

Database Efficiency 101: Understanding Bloat, Vacuum, and the Power of pg_repack

In a high-performance database environment, how we manage storage is just as important as how we write our queries. To keep our systems fast and cost-effective, we need to address a natural phenomenon in PostgreSQL known as Bloat.

1. What is Database Bloat?

PostgreSQL uses a system called MVCC (Multi-Version Concurrency Control). When a row is updated, Postgres doesn’t overwrite the old data; it creates a new version of the row and marks the old one as “dead.” Similarly, deleted rows are merely marked as dead tuples.

These dead tuples are “ghosts” — they occupy physical space on the disk but are invisible to your application. Bloat is the accumulation of these dead tuples within your tables and indexes.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

2. The Maintenance Arsenal

The Autovacuum (Your Internal Autopilot)

The Autovacuum daemon is a background process that automatically cleans up dead tuples so the space can be reused. To keep it efficient, we focus on three main settings:

• Scale Factor (**autovacuum_vacuum_scale_factor**): This determines the threshold for when a vacuum starts. For example, a 20% scale factor means 20% of a table must be "dead" before a cleanup triggers.
• Cost Delay (**autovacuum_vacuum_cost_delay**): This tells the worker to "pause" occasionally to ensure it doesn't consume all your Disk I/O, protecting application performance.
• Workers: This is the number of parallel processes allowed to clean the database at once.

VACUUM FULL (The Heavy Reset)

VACUUM FULL physically rewrites a table to a new file, reclaiming 100% of the bloat and returning that space to the Operating System.

• The Problem: It requires an Access Exclusive Lock, meaning no one can read or write to the table while it’s running. In a 24/7 environment, this is rarely an option.

pg_repack (The Professional Solution)

pg_repack offers the best of both worlds. It reclaims 100% of the bloat and returns space to the OS, but it does so online.

• How it works: It creates a new “clean” version of the table in the background, logs new changes, and then swaps them in a fraction of a second at the very end.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

3. Why is Bloat Removal Necessary?

• Faster Querying: Bloat forces the database to read through “empty” space during scans. Removing bloat reduces Disk I/O and speeds up response times.
• Cost Optimization: We pay for the storage we provision. Our audit identified over 620 GB of recoverable “ghost” space. Reclaiming this reduces our infrastructure spend.
• System Stability: Excessive bloat makes indexes larger and slower to update, eventually leading to performance “jitter” during peak traffic.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

4. Our Execution Strategy

To minimize risk, we process the database using a Sequential Repack Strategy. Instead of a massive operation, we process one table at a time. This ensures that the maximum additional storage required is only the size of the largest single table being processed, rather than a doubling of our entire database footprint.

By clearing out the 90%+ bloat found in our smaller tables first, we create a “buffer” of free space to safely handle our largest multi-terabyte partitions.

By Jatin Jain Saraf on March 13, 2026.

Most developers exclusively use an ORM — Prisma, Sequelize, Hibernate — which means they may never actually learn how to "loop" in SQL. For 90% of scenarios, that's fine. But when you hit a table with 10M+ rows and your ORM-generated join is spilling to disk, you need to understand what's actually happening underneath.

LATERAL joins are the SQL construct that turns a set-based operation into something that behaves like a forEach loop. Once you understand it, a whole class of slow queries becomes fast.

What Is a LATERAL Join?

In a standard SQL join, the subquery in your FROM clause is evaluated once, independently of the outer query. It cannot reference columns from tables listed before it.

A LATERAL subquery breaks that rule. It's re-evaluated for each row of the preceding table, and it can reference columns from that row. That's the entire mental model:

lateral = per-row subquery execution

SELECT *
FROM outer_table o,
LATERAL (
  SELECT *
  FROM inner_table i
  WHERE i.foreign_key = o.id   -- references the current outer row
  LIMIT 3
) sub;

This is semantically equivalent to a forEach in application code:

for (const outerRow of outerTable) {
  const results = innerTable
    .filter(r => r.foreign_key === outerRow.id)
    .slice(0, 3);
}

The difference: it happens entirely in the database, in a single query, with the planner able to use indexes on every iteration.

Use Case 1: Top-N Per Group

The most common LATERAL use case — fetch the N most recent rows for each parent record.

Problem: You have a wallets table and a transactions table. You want the 3 most recent transactions for each wallet.

The naive approach with a window function:

-- Works, but scans and ranks all transactions first
SELECT *
FROM (
  SELECT
    w.id AS wallet_id,
    t.*,
    ROW_NUMBER() OVER (
      PARTITION BY t.wallet_id
      ORDER BY t.created_at DESC
    ) AS rn
  FROM wallets w
  JOIN transactions t ON t.wallet_id = w.id
) ranked
WHERE rn <= 3;

This has to read and rank every transaction before filtering. On 20M rows, that's a full table scan.

With LATERAL:

SELECT w.id AS wallet_id, recent.*
FROM wallets w
CROSS JOIN LATERAL (
  SELECT *
  FROM transactions t
  WHERE t.wallet_id = w.id
  ORDER BY t.created_at DESC
  LIMIT 3
) recent;

With a composite index on (wallet_id, created_at DESC), the planner uses a nested loop: for each wallet, it does an index seek to grab exactly 3 rows. No full scan. No ranking pass.

CREATE INDEX idx_txn_wallet_created
  ON transactions (wallet_id, created_at DESC);

On 500K wallets with 20M transactions: the window function version takes 40+ seconds; the LATERAL version takes under 500ms.

Use Case 2: Forcing Nested Loops — 29 Minutes to 0.3ms

This is where LATERAL becomes a surgical tool for query optimization.

The planner makes join strategy decisions based on table statistics. When it gets it wrong — choosing a hash join or merge join on a large table — you can use LATERAL to force the strategy you know is correct.

A real production example from SupraScan: We had a query joining blocks to events to find the latest event of each type for a given contract. The planner chose a merge join, sorting hundreds of thousands of rows.

-- Before: planner chose merge join — 29 minutes
SELECT DISTINCT ON (e.event_type)
  e.*
FROM blocks b
JOIN events e ON e.block_height = b.height
WHERE b.chain_id = $1
  AND e.contract_address = $2
ORDER BY e.event_type, e.created_at DESC;

After — LATERAL forces a targeted index seek per event type:

-- After: nested loop via LATERAL — 0.3ms
SELECT latest.*
FROM (
  SELECT UNNEST(ARRAY['Transfer', 'Mint', 'Burn', 'Approval']) AS event_type
) types
CROSS JOIN LATERAL (
  SELECT e.*
  FROM events e
  WHERE e.contract_address = $2
    AND e.event_type = types.event_type
  ORDER BY e.created_at DESC
  LIMIT 1
) latest;

-- The index that makes each iteration a single seek
CREATE INDEX idx_events_contract_type_created
  ON events (contract_address, event_type, created_at DESC);

With this index, each LATERAL iteration is one index seek. Four event types = four index seeks, each returning exactly one row. The planner cannot choose anything other than nested loops — which is exactly what you want when your outer set is small and your index is selective.

Result: 29 minutes → 0.3 milliseconds.

The key insight is that LATERAL keeps your working set small at each step. Instead of joining two large tables and filtering afterward, you filter first — at the index level — before any join. Memory stays low, disk spills don't happen, and the index works on every iteration.

Use Case 3: Unnesting JSONB and Arrays

LATERAL joins are essential when working with set-returning functions like UNNEST and jsonb_array_elements, because those functions need per-row context from the outer table.

Example — JSONB array expansion:

-- users table with a preferences JSONB column containing an array
-- Expand each preference entry and filter enabled ones

SELECT u.id, u.email, pref.value
FROM users u
CROSS JOIN LATERAL jsonb_array_elements(u.preferences) AS pref(value)
WHERE pref.value->>'enabled' = 'true';

Without LATERAL, you cannot reference u.preferences inside jsonb_array_elements. The implicit comma syntax (, instead of CROSS JOIN LATERAL) works too — PostgreSQL treats it as LATERAL when a set-returning function appears — but explicit LATERAL makes the intent clear.

Example — plain array unnesting with a join:

-- posts table has a tags integer array; join each tag to a tag_definitions table
SELECT p.id, p.title, t.name AS tag_name
FROM posts p
CROSS JOIN LATERAL UNNEST(p.tag_ids) AS tag_id
JOIN tag_definitions t ON t.id = tag_id;

This pattern replaces multiple round-trips or suboptimal ANY(array) queries with a single clean pass.

Use Case 4: Keyset Pagination

Standard OFFSET pagination breaks on large tables — scanning 500K rows to skip the first 499,990 is a full table read disguised as a feature.

LATERAL enables efficient cursor-based pagination:

-- Get the next 20 transactions per wallet after a given cursor
SELECT w.id, page.*
FROM wallets w
CROSS JOIN LATERAL (
  SELECT id, amount, created_at
  FROM transactions t
  WHERE t.wallet_id = w.id
    AND (t.created_at, t.id) < ($last_seen_at::timestamptz, $last_seen_id::bigint)
  ORDER BY t.created_at DESC, t.id DESC
  LIMIT 20
) page
WHERE w.user_id = $user_id;

Page 500 loads in the same time as page 1. The (created_at, id) tuple comparison is the cursor — the index makes each seek O(log n) regardless of which page you're on.

When NOT to Use LATERAL

LATERAL is the right tool when:

• Your outer set is small and the inner query is highly selective (with a matching index)
• You need top-N per group
• You're working with set-returning functions (UNNEST, jsonb_array_elements)
• You need to override the planner's join strategy choice

Don't reach for LATERAL when:

• You're doing a simple 1:1 join — regular JOIN optimizes better
• Your outer set is large and the inner query isn't selective — you'll get N expensive seeks instead of one efficient scan
• The planner's chosen strategy is actually correct for your data distribution

Always verify with EXPLAIN (ANALYZE, BUFFERS):

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT w.id, recent.*
FROM wallets w
CROSS JOIN LATERAL (
  SELECT * FROM transactions t
  WHERE t.wallet_id = w.id
  ORDER BY t.created_at DESC
  LIMIT 3
) recent;

Look for Nested Loop + Index Scan in the plan output. If you see Hash Join or Seq Scan, the index is missing or statistics are stale — run ANALYZE transactions and re-check.

Summary

A LATERAL join lets a subquery reference the current row of the outer query. Use it when you want the planner to perform N targeted index seeks instead of one large scan — and when you have the index to make each seek fast.

The moment your ORM-generated query starts spilling to disk on a large table, check whether LATERAL can replace it. Drop to raw SQL, write the LATERAL subquery, create the composite index, and run EXPLAIN ANALYZE. The plan will change from Seq Scan → Hash Join → Sort to Nested Loop → Index Scan — and your query time will drop with it.

All examples tested on PostgreSQL 15+. The 29-minute → 0.3ms optimization is from a real production query on SupraScan's blockchain indexer, running on a PostgreSQL database that grew to 10 TB of indexed blockchain data.

PostgreSQL has long been the gold standard for relational databases, robust, reliable, and continuously evolving. With each major release…

PostgreSQL 18: A Paradigm Shift for Modern Data Workloads

PostgreSQL has long been the gold standard for relational databases, robust, reliable, and continuously evolving. With each major release, the community eagerly anticipates new features that enhance performance, developer experience, and operational efficiency. PostgreSQL 18, released in late 2025, isn’t just another incremental update; it’s a paradigm shift that fundamentally re-architects how the database handles I/O, optimises queries, and manages complex deployments.

For developers, DBAs, and architects, understanding the core changes in PG18 isn’t optional — it’s crucial for future-proofing your applications and infrastructure. Let’s dive deep into what makes PG18 a monumental leap from its predecessors, outlining the key features, their impact, and a balanced look at the pros and cons.

The Evolution: A Quick Look Back from PG17 to PG18

Before we dissect PG18, it’s helpful to understand the trajectory. Previous versions, including PG17, brought significant improvements like enhanced JSONB capabilities, better parallelism for certain queries, and continued refinements to the query planner. These were vital, but PG18 tackles some long-standing architectural bottlenecks.

The jump to PG18 is about addressing foundational performance limitations (I/O, index usage) and critical operational hurdles (schema replication, upgrade pain points).

The Big Four: What Defines PostgreSQL 18

While many smaller improvements arrived, four features stand out as the pillars of PostgreSQL 18:

1. Asynchronous I/O (AIO) Subsystem: The End of Waiting

What it is: In a radical departure, PostgreSQL 18 introduces a fully integrated Asynchronous I/O (AIO) subsystem. Previously, PostgreSQL primarily used synchronous I/O; when a process needed data from disk, it would issue a request to the operating system and then wait for the data to be read. This meant CPU cycles were wasted in idle waiting, especially on high-latency storage or during heavy read operations.

PG18 changes this by allowing the database to issue multiple I/O requests concurrently without blocking the initiating process. It leverages modern OS features like Linux’s io_uring for highly efficient I/O batching and completion.

Comparison (PG17 vs. PG18):

• PG17 & Earlier: CPU-bound by disk latency; processes often stalled waiting for I/O.
• PG18: I/O-bound processes can now issue requests and continue processing other tasks, dramatically improving CPU utilisation and throughput.

Pros:

• Massive Performance Boost: Up to 3x faster for read-heavy workloads (scans, joins, VACUUM).
• Increased Concurrency: More transactions per second possible with the same hardware.
• Smoother Operations: Background tasks like VACUUM can run with less impact on active queries.

Cons:

• OS Dependence: Optimal performance leverages specific OS features (io_uring on Linux). While a fallback io_method=worker is available, the full benefit is OS-specific.
• Configuration Nuances: Requires understanding new io_method parameters and potential OS-level tuning.

2. B-Tree Skip Scans: Smart Indexing for Complex Queries

What it is: A common pain point with multicolumn B-Tree indexes (e.g., (column_A, column_B, column_C)) was their strict reliance on the "leftmost prefix rule." If your query filtered only on column_B or column_C without column_A, The index was often ignored, leading to slow sequential scans.

PG18 introduces B-Tree Skip Scans. The query planner can now intelligently “skip” over distinct values of leading index columns to efficiently locate data based on subsequent columns.

Comparison (PG17 vs. PG18):

• PG17 & Earlier: Often required to create multiple, redundant indexes (e.g., (A,B,C), (B,C), (C)) to optimise different query patterns, leading to index bloat and slower writes.
• PG18: A single composite index can now satisfy a wider range of query conditions, significantly reducing index maintenance overhead.

Pros:

• Reduced Index Bloat: Fewer indexes needed to cover diverse query patterns.
• Improved Write Performance: Less overhead during inserts/updates/deletes due to fewer indexes to maintain.
• Faster Ad-Hoc Queries: Analytics and exploratory queries benefit greatly.
• Simplified Schema Design: Easier to manage indexes without sacrificing query speed.

Cons:

• Not a Silver Bullet: The performance gain depends on data distribution and selectivity; not every query will see a huge boost.
• Planner Learning Curve: DBAs need to understand how the planner now utilizes these indexes to avoid over-indexing.

3. Logical Replication for DDL: Seamless Schema Evolution

What it is: Logical replication has been a cornerstone for building highly available, geographically distributed, or multi-tenant PostgreSQL systems. However, a major limitation was its inability to automatically replicate DDL (Data Definition Language) changes like CREATE TABLE, ALTER TABLE, or DROP INDEX. DBAs had to manually synchronize schema changes across all replicas, a process prone to errors and downtime.

PG18 resolves this by natively integrating DDL replication into the logical replication mechanism. Schema changes are now automatically propagated to subscribers.

Comparison (PG17 vs. PG18):

• PG17 & Earlier: Manual DDL sync, often involving custom scripts, change management, and potential replication breaks.
• PG18: Automatic, atomic DDL propagation, making multi-node schema changes significantly safer and simpler.

Pros:

• True Zero-Downtime Migrations: Critical for large-scale, 24/7 applications.
• Simplified Operations: Eliminates complex manual procedures for schema updates.
• Improved Consistency: Reduces the risk of schema drift between publisher and subscribers.
• Enhanced DR Capabilities: Easier to maintain consistent schemas across disaster recovery setups.

Cons:

• Careful Planning Still Needed: While automated, complex DDL changes can still impact application logic on the subscriber side.
• Potential for Conflicts: Requires understanding how certain DDL operations might interact in specific replication topologies.

4. Preserved Optimizer Statistics on Major Upgrades: Hit the Ground Running

What it is: Previously, when performing a major version upgrade using pg_upgrade, the query planner's statistical information was reset. This meant after an upgrade, your database would perform poorly until you ran a full ANALYZE (which could take hours for large databases), as the planner had no information to create efficient execution plans.

PG18 ensures that query planner statistics are preserved during pg_upgrade.

Comparison (PG17 vs. PG18):

• PG17 & Earlier: Post-upgrade “cold start” period with suboptimal query performance until ANALYZE completed.
• PG18: Database performs optimally immediately after the upgrade, minimizing service degradation.

Pros:

• Faster, Smoother Upgrades: Reduces post-upgrade tuning and performance issues.
• Reduced Risk: Eliminates a major source of performance anxiety after major version bumps.
• Improved DBA Experience: Less manual intervention required after an upgrade.

Cons:

• No Functional Impact: This is a quality-of-life improvement for upgrades, not a new runtime feature.

Other Notable Mentions in PostgreSQL 18

• Native UUIDv7 Support: Generates timestamp-ordered UUIDs, drastically improving B-Tree index performance for UUID primary keys by reducing fragmentation.
• Enhanced RETURNING Clause (OLD and NEW): Allows developers to retrieve both the original (OLD) and modified (NEW) row states in UPDATE and DELETE statements, simplifying auditing and application logic.
• Virtual Generated Columns by Default: Generated columns are now virtual (computed on read) by default, saving significant disk space and improving write performance (you can still specify STORED if needed).
• OAuth 2.0 Authentication: Modern enterprise-grade authentication support, simplifying integration with SSO providers.
• MD5 Password Deprecation: Continues the push towards more secure SCRAM-SHA-256 for password authentication, with clear warnings for MD5 usage.

Conclusion: Embracing the Future with PostgreSQL 18

PostgreSQL 18 is a testament to the community’s commitment to pushing the boundaries of relational database technology. The architectural shifts in AIO and the intelligent index utilisation of Skip Scans are foundational changes that will impact performance across the board. Combined with critical operational improvements like Logical DDL Replication and preserved optimizer statistics, PG18 makes managing and scaling PostgreSQL easier and more robust than ever before.

For any organization or developer serious about performance, reliability, and future-proofing their data layer, migrating to PostgreSQL 18 isn’t just an option — it’s a strategic imperative. The benefits far outweigh the considerations, marking a significant milestone in PostgreSQL’s storied history.

By Jatin Jain Saraf on February 21, 2026.

Migrating infrastructure from one cloud provider to another is already a complex process. But migrating a large, production PostgreSQL…

Cloud‑to‑Cloud Database Migration: A Practical Guide for Large PostgreSQL Systems

Migrating infrastructure from one cloud provider to another is already a complex process. But migrating a large, production PostgreSQL database, with schemas, constraints, triggers, partitions, indexes, extensions, and years of operational history, is an entirely different level of engineering.

This article breaks down the real‑world challenges, decisions, risks, and best practices involved in such migrations.

Why Database Migration Is Uniquely Hard

Applications are portable.

Containers can be rebuilt.

CI pipelines can be recreated.

But databases are:

• Stateful
• Performance‑sensitive
• Integrity‑critical
• Operationally fragile

A single wrong command can:

• Drop constraints
• Disable triggers
• Break partitions
• Corrupt encodings
• Destroy indexes
• Or silently degrade performance

Cloud database migration is not data transfer.

It is data engineering + systems engineering + risk management.

Common Use Cases for Cloud‑to‑Cloud Migration

Teams usually migrate because of:

• Cost optimization (AWS → GCP / GCP → DO)
• Vendor lock‑in concerns
• Regulatory or data‑residency requirements
• Performance issues
• Infrastructure consolidation
• Reliability problems

Each use case affects the migration strategy.

Example:

• Cost‑driven migration → minimize downtime
• Compliance migration → prioritize auditability
• Performance migration → re‑design indexes & partitions

What Actually Needs to Be Migrated (Beyond Tables)

Most failures happen because teams think only about data rows. In reality you must migrate:

1. Schemas

• Namespaces
• Ownership
• Permissions

2. Tables

• Data types
• Defaults
• NOT NULL constraints

3. Indexes

• B‑Tree / GIN / GiST / BRIN
• Partial indexes
• Expression indexes

4. Constraints

• Primary keys
• Foreign keys
• Unique constraints
• Check constraints

5. Partitions

• Range / list / hash partitions
• Partition constraints
• Parent‑child table relationships

6. Triggers

• Audit triggers
• Business logic triggers
• Replication triggers

7. Functions & Procedures

• PL/pgSQL
• Extensions

8. Extensions

• PostGIS
• pgcrypto
• uuid‑ossp
• citext
• TimescaleDB

9. Migration history

• Flyway / Liquibase tables
• Schema version tracking

Missing any one of these can break your system in subtle ways.

Strategic Decisions Before You Touch Any Command

1. Downtime vs Live Migration

Options:

• Full downtime migration
• Read‑only window
• Dual‑write period
• Logical replication

Each increases complexity exponentially.

2. Database Engine Parity

Verify:

• PostgreSQL major version
• Extensions availability
• Default configs
• Collation & locale

Even small version mismatches can break indexes or functions.

3. Data Volume

4. Security Model

• Users
• Roles
• Grants
• Secrets

Cloud providers handle IAM very differently.

pg_dump & pg_restore: Where Most Migrations Fail

Choosing the Correct Dump Format

Recommended Commands

Schema only

pg_dump -Fc --schema-only -f schema.dump dbname

Data only

pg_dump -Fc --data-only -f data.dump dbname

Full dump

pg_dump -Fd -j 8 -f dumpdir dbname

Restore Order (Critical)

1. Roles & users
2. Schemas
3. Extensions
4. Tables
5. Data
6. Indexes
7. Constraints
8. Triggers
9. Validation

Wrong order = broken foreign keys or missing triggers.

The Dangerous Commands (Real Production Killers)

pg_dumpall > backup.sql # huge, slow, unsafe

pg_restore --clean # may DROP critical objects

DROP SCHEMA public CASCADE;

SET session_replication_role = replica; # disables triggers

One wrong flag → irreversible damage.

Configuration Differences That Break Systems

Cloud providers ship different defaults:

• work_mem
• shared_buffers
• wal_level
• max_connections
• autovacuum settings
• synchronous_commit

These affect:

• Query latency
• Index usage
• Deadlocks
• Replication stability

Always compare:

SHOW ALL;

Performance Regression After Migration

Common causes:

• Missing indexes
• Statistics not refreshed
• Different query planner
• Changed collation
• IO throughput differences

Fix with:

ANALYZE;
REINDEX DATABASE dbname;

Validation Checklist (Non‑Negotiable)

Structure

SELECT count(*) FROM information_schema.tables;

Triggers

SELECT * FROM information_schema.triggers;

Indexes

SELECT * FROM pg_indexes;

Extensions

SELECT * FROM pg_extension;

Row counts

Compare source vs target.

Application tests

Run full regression tests.

Real Risks Nobody Talks About

• Silent data truncation
• Encoding mismatch
• Timezone corruption
• Floating‑point drift
• Broken audit trails
• Lost permissions

Most of these are detected weeks later in production.

When Your Database Is in Terabytes and Billions of Rows

At small scale, mistakes are painful.

At terabyte scale, mistakes are catastrophic.

When your database contains:

• Multiple TB of data
• Billions of rows
• Hundreds of tables
• Thousands of indexes
• Complex partitions
• Heavy write traffic

…migration becomes a distributed systems problem, not just a database task.

Additional Challenges at TB Scale

1. Network Throughput Becomes the Bottleneck

• Cross-cloud bandwidth limits
• Packet loss
• Throttling
• NAT saturation

Even at 1 Gbps:

1 TB ≈ 2.5 hours (ideal conditions)

5 TB ≈ 12+ hours

10 TB ≈ 24+ hours

Real-world speeds are usually 40–60% of theoretical.

2. Storage IOPS Decide Success or Failure

Cloud disks differ massively:

• GCP PD-SSD
• AWS gp3 / io2
• DigitalOcean volumes

Low IOPS =

• Slow restores
• Checkpoint storms
• Autovacuum backlog
• Index build failures

Always benchmark disk write speed before restore.

3. pg_dump Stops Being Enough

For very large databases:

• pg_dump becomes slow
• Single-threaded metadata phase blocks
• Restore time explodes

Better approaches:

• pg_dump in directory format with high parallelism
• Logical replication
• Chunked table migration
• Physical replication (pg_basebackup)

Migration Strategies for Massive Databases

Strategy 1: Full Downtime (Rarely Acceptable)

• Stop writes
• Dump everything
• Restore
• Start application

Works only for internal systems or very tolerant products.

Strategy 2: Schema First + Batch Data Migration

1. Dump schema
2. Create structure on target
3. Migrate tables in chunks
4. Build indexes later

Reduces risk but complex to orchestrate.

Strategy 3: Logical Replication (Most Common)

• Setup publisher on source
• Setup subscriber on target
• Sync initial snapshot
• Catch up changes
• Switch traffic

Pros:

• Near zero downtime
• Safer cutover

Cons:

• Complex
• Requires identical schema
• Triggers behave differently

Strategy 4: Physical Replication

• Use pg_basebackup
• Copy WAL files
• Promote replica

Fastest but risky across clouds and versions.

The “One Wrong Command” Problem (At Scale)

At TB scale, mistakes are not easily reversible.

Examples:

pg_restore --clean

→ Drops objects while application is live.

pg_restore -j 32

→ Can overload disks and crash the server.

pg_dump --disable-triggers

→ Breaks audit systems.

ALTER SYSTEM SET synchronous_commit = off;

→ Silent data loss during crash.

SET maintenance_work_mem = '64MB';

→ Index creation takes days instead of hours.

Configuration That Can Break Large Restores

Critical settings during restore:

maintenance_work_mem
work_mem
shared_buffers
wal_buffers
checkpoint_timeout
max_wal_size
synchronous_commit
fsync
autovacuum

Bad tuning leads to:

• WAL explosion
• Disk full
• Checkpoint storms
• Table corruption
• Restore freezing at 99%

pg_dump Format Decision Matrix (TB Scale)

Recommended:

pg_dump -Fd -j 16 --no-owner --no-acl -f dumpdir dbname

Restore:

pg_restore -Fd -j 16 -d target_db dumpdir

Data Integrity Verification at Massive Scale

Row counts are not enough.

Also verify:

• Checksums
• Hash totals
• Min/max timestamps
• Foreign key violations
• Application-level aggregates

Example:

SELECT SUM(amount) FROM transactions;

Compare source vs target.

Operational Reality

Large migrations require:

• Dry runs
• Rollback plan
• Monitoring dashboards
• Disk usage alerts
• WAL growth alerts
• CPU saturation alerts

And one truth:

You never migrate once.

You rehearse many times.

Architecture Diagrams (Reference Patterns)

Below are practical reference architectures commonly used for large-scale PostgreSQL migrations.

1. Simple Offline Migration (Full Downtime)

┌──────────────┐ pg_dump / pg_restore ┌──────────────┐
│ Source DB │ ─────────────────────────────▶ │ Target DB │
│ (Cloud A) │ │ (Cloud B) │
└──────────────┘ └──────────────┘
▲ ▲
│ │
└────────────── Application Down ──────────────────┘

Use when:

• Internal tools
• Small datasets
• Downtime acceptable

2. Schema-First + Batch Data Migration

Step 1: Schema
┌──────────────┐ pg_dump --schema-only ┌──────────────┐
│ Source DB │ ───────────────────────▶ │ Target DB │
└──────────────┘ └──────────────┘

      Step 2: Data (in chunks)  

┌──────────────┐ Table batches / jobs ┌──────────────┐
│ Source DB │ ───────────────────────▶ │ Target DB │
└──────────────┘ └──────────────┘

      Step 3: Indexes + Triggers

Use when:

• Medium–large DB
• Partial downtime allowed
• Need more control

3. Logical Replication (Near Zero Downtime)

Initial Snapshot + WAL Streaming
┌──────────────┐ ───────────────────────────────────────▶ ┌──────────────┐
│ Source DB │ │ Target DB │
│ (Publisher) │ │ (Subscriber) │
└──────────────┘ └──────────────┘
▲ ▲
│ │
│ Replication Slot │
└─────────────────────────────────────────────────────────┘

             Traffic Cutover  

┌──────────────┐ ┌──────────────┐
│ Application │ ───────────── switch ───────────▶│ Target DB │
└──────────────┘ └──────────────┘

Use when:

• Production systems
• Billions of rows
• Downtime must be minimal

4. Physical Replication (pg_basebackup)

┌──────────────┐ Base Backup + WAL Files ┌──────────────┐
│ Source DB │ ───────────────────────────────▶ │ Replica DB │
│ (Primary) │ │ (Cloud B) │
└──────────────┘ └──────────────┘
│
▼
Promote to Primary

Use when:

• Same PostgreSQL version
• Compatible storage
• Very large datasets
• Advanced DBA team

5. Enterprise-Grade Migration with Staging Layer

┌─────────────────┐
│ Monitoring & │
│ Validation │
└────────┬────────┘
│
▼
┌──────────────┐ Logical Replication ┌──────────────┐
│ Source DB │ ─────────────────────▶ │ Staging DB │
└──────────────┘ └──────────────┘
│
│ Validation + Index build
▼
┌──────────────┐
│ Target DB │
└──────────────┘
▲
│
Application Cutover

Use when:

• Mission-critical systems
• Regulatory requirements
• Very large datasets
• Strict validation needs

Key Notes for All Architectures

• Always separate schema migration from data migration
• Never build heavy indexes before loading data
• Always validate before traffic cutover
• Monitor WAL growth and disk usage continuously
• Keep rollback architecture ready

Final Thoughts

Cloud database migration at TB scale is not DevOps work.

It is:

• Distributed systems engineering
• Data architecture
• Capacity planning
• Risk management
• Incident prevention

Treat it like a spacecraft launch.

Because at this scale…

one wrong command doesn’t cause a bug.

It causes an outage.

A data incident.

Or a company-level crisis.

By Jatin Jain Saraf on January 11, 2026.

How a social theory quietly explains modern engineering, system failures, and high-impact collaboration.

We usually hear Six Degrees of Separation as a social idea, the belief that any two people on Earth are connected by at most six relationships. But when you look at the systems we work with every day, distributed platforms, microservices, blockchain networks, DevOps pipelines, you realise something interesting: Six Degrees of Separation isn’t just a human phenomenon. It’s a technical one.

Modern engineering systems are deeply interconnected, sometimes visibly, mostly silently. And understanding these hidden connections can change the way we build, debug, and collaborate.

1. Systems don’t exist in isolation; they exist in graphs

Every backend we build is basically a living graph:

• Services call other services
• APIs rely on upstream data
• CI/CD pipelines depend on dozens of tools
• One misconfigured environment variable affects another team’s release
• A single network hop impacts user-facing latency

Your component may feel self-contained, but in reality, it’s never more than a few “degrees” away from critical paths, just like people.

2. Failures spread through weak links, not major roads

Most outages don’t start with big failures. They start with:

• One stale cache
• One retry loop is misbehaving
• One schema mismatch
• One outdated node module
• One engineer assumes “nobody else depends on this”

And like social connections, these tiny issues cascade across services that are only a couple of hops away. Outages propagate the same way news travels in a social network, quietly at first, then everywhere.

3. Communication is infrastructure

In engineering teams, communication behaves like a network protocol. When one link breaks, missing documentation, a misaligned expectation, or unclear ownership, the impact doesn’t stay local. It radiates. You might be 2–3 messaging hops away from someone who needs clarity from you. You may never speak to them directly, but your work will reach them. This is why I often say:

“Your real customers aren’t always the people you talk to

they’re the people 3–4 degrees away consuming your output silently.”

4. Good architecture reduces the degrees of separation

Strong systems intentionally reduce the number of jumps between components:

• Clear API contracts
• Reliable interfaces
• Proper versioning
• Predictable event flows
• Observability baked in
• Cross-team alignment on SLAs and failure scenarios

When components understand each other clearly, the system resembles a well-connected social network, fast, robust, and transparent. When they don’t, it becomes a dysfunctional web full of bottlenecks.

5. The human graph and the tech graph overlap

This part is usually ignored. Behind every API call, there is a team. Behind every service, there is ownership. Behind every dependency there is a relationship. Technology and people fail the same way:

• unclear communication
• ambiguous ownership
• assumptions
• outdated information
• isolated decision-making

The technical graph can only be as strong as the human graph behind it. And that is exactly what Six Degrees of Separation teaches us.

The takeaway

Six Degrees of Separation isn’t just a social curiosity. It’s a design principle. It reminds us that:

• Every decision travels further than you expect
• Every change affects someone you will never meet
• Every service is only a few hops away from a user experience
• Every engineer is part of a much bigger system

We build better when we recognise the invisible connections, in code, in architecture, and in collaboration. Because in today’s world, nothing is truly isolated. Not people. Not systems. Not engineering decisions.

By Jatin Jain Saraf on December 11, 2025.

As a developer building on a high-performance blockchain like Supra, you’ll soon encounter a critical challenge: indexing and understanding…

The Complete Developer’s Guide to Indexing Assets on Supra & Aptos (Coins, FAs, NFTs)

As a developer building on a high-performance blockchain like Supra, you’ll soon encounter a critical challenge: indexing and understanding digital assets. The chain’s history is split into two distinct eras: the legacy standards (like 0x1::coin and 0x3::token) and the modern, object-based standards (like 0x1::fungible_asset and 0x4::digital_asset).

These standards are not interchangeable. They have fundamentally different data models, storage locations, and require completely different indexing strategies. One is simple; the other is a minefield of potential data corruption that will keep you up at night.

For the past 2.5 years, our team at Supra has been building the SupraScan explorer. This is the guide we wish we had. We’ll cover everything from the core data models to the exact RPC calls and, most importantly, the hard-won “corner cases” you will hit.

Part 1: The Move Data Model

It’s built on the Move smart contract language, which organizes all its on-chain data around three foundational principles. The purpose of this model is Safety, Scarcity, and Flexibility.

Modules (The Logic) A smart contract that contains all the logic (functions and procedures) for a program. Modules have no inherent storage; they define the rules for interacting with Resources and Objects. Modules can be updated and iterated upon. Module functions can only access the internal fields of Resources and Objects they define, preventing outside tampering.

• Use Case: The Bank Vault’s Blueprint. Defines the rules for depositing, withdrawing, and securing funds, but doesn’t hold the money itself.

Resources (The Scarcity Primitive) A special data structure that holds state/data. Resources have Move Semantics, meaning they cannot be copied, duplicated, or implicitly dropped/destroyed (like a Rust ownership model). This ensures digital assets behave like physical assets and is fundamental to preventing double-spending.

• Use Case: The Token Balance/Deed. Guarantees that an asset’s data (like a balance of 100 APT) can only exist in one place. It is stored inside an Account’s address or, more commonly now, inside an Object.

Objects (The Container) A user-defined, composable container that has its own unique address on the blockchain. An Object can hold one or more Resources and even other Objects. This is the foundation of the modern asset standards.

• Use Case: The House/Car. A single, unique entity (the Object) with a unique address that contains multiple resources (e.g., the Deed, the Vehicle ID, the Insurance Policy).

Part 2: Fungible Tokens (The “Money”) — FA vs. Coin

This covers standard, interchangeable tokens like USDC or the native SRA/APT coin.

Fungible Assets (FA): The New 0x1::fungible_asset Standard

• Philosophy: Object-Based.
• How it Works: The FA standard uses Objects for metadata and storage. The asset’s metadata (name, symbol) is an Object, and a user’s balance is stored in a FungibleStore resource on an object owned by the user.
• Use Case: Creating a stablecoin ($USDC). The FA model ensures all metadata is stored on-chain in an Object, simplifying integration for wallets and exchanges. It’s flexible and allows for advanced features like custom transfer logic.

Coin Legacy: The Old 0x1::coin Standard

• Philosophy: Resource-Based.
• How it Works: To hold a “Coin,” a user must have a CoinStore resource inside their account. This resource contains a simple value field, which is their balance.
• The Indexer’s Challenge: You cannot ask an account, “Show me all coins you own.” You must know the CoinType in advance to query the specific resource.
• Explorer Solution: An explorer must first build a master list of all CoinInfo resources on the chain, then query a user’s account for every single coin in your master list to build their portfolio.
• Events: Track 0x1::coin::DepositEvent and 0x1::coin::WithdrawEvent.

Part 3: Non-Fungible Tokens (The “Collectibles”)

This is the most complex part of indexing, as the two standards are radically different.

Part 4: The 0x3 Legacy Token Standard (The “Hard Mode”)

PHILOSOPHY: THE “BANK VAULT” MODEL The asset’s metadata (TokenData) is stored in the creator’s account. The asset’s instance (Token) is stored in the owner’s account. The data is completely separate and spread across the chain.

KEY STRUCTS (THE METADATA)

• CollectionData: Contains metadata for an entire collection (name, description, uri, maximum).
• TokenData: Contains metadata for a specific token (name, uri, royalty, properties).
• Royalty: Defined per-token in this standard.
• Properties (PropertyMap): On-chain key-value traits
• Mutability: Booleans for which fields can be edited (e.g., uri: false). An explorer MUST show this.

KEY IDENTIFIERS (THE “HOW”)

• TokenDataId: The unique key for the token’s metadata. It’s a 3-part struct: creator (address), collection (String), and name (String). This is your primary key.
• TokenId: The unique key for the instance a user owns. It’s a struct containing the TokenDataId and a property_version (usually “0”).

ON-CHAIN STORAGE (THE “WHERE”)

• Collections Resource (On Creator’s Account): This resource (0x3::token::Collections) is the key for backtracking. It contains two Table HANDLES (pointers) to the CollectionData and TokenData tables.
• TokenStore Resource (On Owner’s Account): This resource (0x3::token::TokenStore) exists in every owner’s account. It contains a tokens handle, which points to a table of Token structs that this specific wallet owns.

THE PROBLEM: FINDING THE 0x3 OWNER You cannot ask the blockchain, “Who owns this token?” There is no central owner field. To find the owner, you would have to scan every TokenStore on every account, which is impossible.

THE SOLUTION: A TWO-SYSTEM INDEXER

You MUST build two systems to handle this.

System 1: The “Fast” Event Tracker To listen to all events, from genesis, and build its own database of ownership.

• Key Events: CreateCollectionEvent, CreateTokenDataEvent, DepositEvent (Ownership Gained), WithdrawEvent (Ownership Lost).
• The Flaw: If your indexer’s RPC call fails, misses a single block, and that block lands in the dead-letter queue, your database will be permanently stale.

System 2: The “Slow” Reconciliation Validator This is the only way to solve the stale data problem. It’s a background worker that constantly health-checks your database.

1. Job: It runs in a loop, querying your own DB for 0x3 tokens.
2. Verify Owner: It takes the ownerId you have on file and runs the 2-step RPC check to ask the blockchain: “Does Owner A still own this token?” (Get TokenStore handle -> Query Table Item).
3. The Result: If the RPC returns 404 Not Found, you have detected an error. Your data is stale. You cannot find the new owner, but you can fix your database by setting the ownerId to UNKNOWN.

Part 5: The 0x4 Digital Asset (DA) Standard (The “Easy Mode”)

PHILOSOPHY: THE “ASSET IS THE ACCOUNT” MODEL This standard is a dream for indexers. Every NFT and every Collection is a standalone Object with its own unique address.

KEY STRUCTS (RESOURCES ON THE OBJECT)

• 0x1::object::ObjectCore: The most important resource.
• owner (address): THIS IS IT. The single, authoritative field for the current owner.
• 0x4::collection::Collection: Holds name and URI.
• 0x4::royalty::Royalty: Defines the royalty per-collection.
• 0x4::token::Token: Holds the link to the collection and the URI.
• Warning: The ‘name’ field here is often empty (“”).
• 0x4::token::TokenIdentifiers: Holds the real display name (e.g., “Supra Spike #1”).

THE SOLUTION: FINDING THE 0x4 OWNER It’s one simple RPC call: GET /accounts/{NFT_OBJECT_ADDRESS}/resource/0x1::object::ObjectCore ...then just read the owner field.

KEY 0x4 EVENTS

• 0x4::collection::Mint (Token Created)
• 0x1::object::TransferEvent (Ownership Changed - The ONLY event needed).

Part 6: The Universal Challenge — The Metadata URI

Both 0x3 and 0x4 use a uri field to point to metadata. An indexer MUST fetch this uri to get the real image, traits, and other details.

THE PROBLEM: The uri is just a string. It can be a JSON link, a direct image link, an on-chain Data URI, or even a website. Fetching a 2GB video file will crash your indexer.

THE ROBUST FETCHING LOGIC (FOR YOUR BACKEND)

Your indexer cannot just GET every uri. This is the only safe way to process them:

1. Check Prefix:

• If data: -> It's an on-chain SVG. No network call needed.
• If http/ipfs -> Proceed to Step 2.

1. Resolve URI: Convert ipfs:// to a private gateway link (e.g., Pinata). Public gateways will block you.
2. Make HEAD Request: This is the key optimization. Make a fast HEAD request to get the Content-Type header only.
3. Check Content-Type:

• If application/json -> Make a full GET request, download, and parse the metadata.
• If image/png, video/mp4 -> STOP. Do not download. You know the URI is the media itself.
• If HEAD fails -> The link is broken.

Conclusion

Indexing assets on Supra/Aptos is a tale of two eras. The modern 0x4 and FA standards are robust and easy to query. The legacy 0x1 and 0x3 standards are complex and require a dedicated two-system indexer to ensure data accuracy. By handling these standards correctly, and by building a robust URI processor, you can create a fast, reliable explorer for the entire ecosystem.

By Jatin Jain Saraf on November 20, 2025.

PostgreSQL’s reputation for resilience, consistency, and performance isn’t accidental. It’s the result of deliberate, brilliant…

The PostgreSQL Elephant in the Room: A Deep Dive Into the Architecture That Powers Giants

PostgreSQL’s reputation for resilience, consistency, and performance isn’t accidental. It’s the result of deliberate, brilliant architectural decisions made over decades of open-source development. While many of us use it daily, few venture beneath the surface to understand how it all works.

This is that journey. We will dissect the core components of PostgreSQL, from the moment you connect to the database to the complex dance of scaling it to handle billions of transactions.

Chapter 1: The Blueprint — Core Architecture & Memory

PostgreSQL operates on a classic client-server model, but its stability comes from its multi-process architecture. Unlike databases that use a single process with many threads, PostgreSQL isolates connections for maximum stability.

• The Postmaster (The Listener): This is the master daemon, the first process to start. Its primary job is to listen for incoming client connections. When a request arrives, it authenticates the user and then forks — creates an exact copy of itself — which becomes a dedicated backend process for that client.
• The Backend Process (The Workhorse): This dedicated postgres process handles all queries for a single client connection. This isolation is a cornerstone of PostgreSQL's legendary stability. If a complex query causes this single backend process to crash, it has zero effect on other connections or the master process. The Postmaster simply cleans it up and continues listening for new connections.

These processes interact with two main areas of memory:

• Shared Memory: This large memory block is allocated when the server starts and is accessible by all PostgreSQL processes. Its key components include:
• Shared Buffers (**shared_buffers**): This is the database's primary disk cache and the most critical performance-tuning parameter. When you request data, PostgreSQL fetches the 8KB data pages from disk and stores them here. Subsequent requests for the same data can be served directly from this fast memory cache, avoiding slow disk I/O.
• WAL Buffers (**wal_buffers**): Before changes are written to the permanent Write-Ahead Log on disk, they are staged in this small buffer. This allows the system to collect WAL data for multiple transactions and write it out in a single, efficient operation.
• Commit Log (**CLOG**): This small but vital area tracks the status of every transaction: in-progress, committed, or aborted. When a transaction needs to check the visibility of a row, it consults the CLOG to see if the transaction that created it was successfully committed.
• Local Memory: Each backend process has its own private memory for operations specific to its queries. Key parameters include:
• Work Memory (**work_mem**): Used for operations that need space outside of shared buffers, such as sorting data for ORDER BY clauses or building hash tables for joins.
• Maintenance Work Memory (**maintenance_work_mem**): A larger chunk of memory reserved for maintenance tasks like creating indexes (CREATE INDEX) or cleaning up tables (VACUUM).

Chapter 2: The Time Machine — Unraveling MVCC

The magic behind PostgreSQL’s incredible concurrency is Multi-Version Concurrency Control (MVCC). It’s how PostgreSQL allows a long-running analytics query to run at the same time as hundreds of users are writing new data, all without locking each other.

To understand it, you must first understand that PostgreSQL doesn’t really have “rows.” It has tuples, which are versions of a row.

• The Life of a Tuple:

1. **INSERT**: A new tuple is created. A hidden system column, xmin, is stamped with the ID of the inserting transaction.
2. **DELETE**: The tuple is not physically deleted. Instead, its hidden xmax column is stamped with the ID of the deleting transaction. The tuple is now a "dead row."
3. **UPDATE**: An UPDATE is an atomic combination of a DELETE and an INSERT. The current tuple is marked as dead by setting its xmax, and a completely new tuple with the updated data is created, with its xmin set to the current transaction ID.

• Transaction Snapshots and Visibility: When a query begins, the transaction takes a “snapshot” of the database. This snapshot knows which transaction IDs were committed at that exact moment. When scanning a table, the query follows these rules for each tuple it encounters:
• Is the tuple’s xmin from a transaction that was committed before my snapshot?
• Is the tuple’s xmax either null or from a transaction that was not yet committed when my snapshot was taken?

If the answer to both is yes, the tuple is visible to the query. This system elegantly allows different transactions to see different, consistent versions of the database state at the same time. The obvious consequence of this design is the accumulation of dead tuples, leading to a phenomenon called bloat.

Chapter 3: The Janitor — VACUUM and Transaction ID Wraparound

Bloat from dead tuples makes tables and indexes larger than necessary, slowing down queries. The process responsible for cleaning this up is VACUUM.

VACUUM performs three critical tasks:

1. Reclaiming Space: It scans tables, identifies dead tuples, and adds the space they occupy to the table’s Free Space Map (FSM). This space is now available for future INSERTs and UPDATEs. A standard VACUUM does not return this space to the operating system; for that, you need a VACUUM FULL, which exclusively locks the table and rewrites it, a costly operation.
2. Updating Statistics: VACUUM is often run with ANALYZE. This command scans the table's data distribution and updates internal statistics that the query planner uses to make intelligent decisions.
3. Preventing Transaction ID Wraparound: This is arguably VACUUM's most critical job. PostgreSQL uses a 32-bit transaction ID (XID). If this number were to grow indefinitely, it would eventually wrap around. This would be catastrophic, as transactions from the distant past would suddenly appear to be in the future, making all data invisible. VACUUM prevents this by "freezing" the XIDs of very old rows, marking them as permanently ancient and visible to all transactions.

Because running this manually is impractical, the autovacuum daemon handles this automatically in the background, making it a cornerstone of a healthy database.

Chapter 4: The Scribe — Write-Ahead Logging (WAL) and Durability

The “D” in ACID (Durability) means that once a transaction is committed, it is permanent, even if the server crashes immediately after. PostgreSQL guarantees this with its Write-Ahead Log (WAL).

The principle is simple: before any change is written to the actual data files on disk, a record of that change is first written and flushed to the WAL file.

• Performance: Writing sequentially to a log file is much faster than the random I/O required to update different data pages scattered across a disk. This allows PostgreSQL to batch the slower, random writes to the main data files.
• Durability: In case of a crash, PostgreSQL initiates a recovery process on startup. It reads the WAL from the last checkpoint (a known good point on disk) and methodically replays all the logged changes, bringing the database back to a perfectly consistent state.

Chapter 5: The Conductor — From SQL to Results

When you send a query, it goes on a four-stage journey before results are returned.

1. Parser: Checks your SQL for correct syntax and converts the raw text into a structured parse tree.
2. Analyzer/Rewriter: Checks that the tables and columns in the parse tree exist. It also expands views into their underlying queries, producing a query tree.
3. Planner/Optimizer: This is the brain of the operation. It receives the query tree and generates many possible execution plans. Using the statistics gathered by ANALYZE, it estimates the "cost" of each plan (e.g., is it cheaper to use an index or just scan the whole table?). It then selects the plan with the lowest estimated cost. You can inspect this plan using the EXPLAIN command.
4. Executor: This component takes the optimal plan and runs it, fetching the tuples, performing sorts and joins, and finally returning the results to you.

Chapter 6: The Architect — Scaling PostgreSQL

Eventually, a single server isn’t enough. PostgreSQL offers a robust path to scaling.

• Vertical Scaling: The simplest approach — giving the server more powerful CPU, more RAM, and faster disks. This has physical limits and becomes prohibitively expensive.
• Horizontal Read Scaling (Replication): This is achieved via Streaming Replication. A primary server streams its WAL records over the network to one or more standby replicas. A special WAL Sender process on the primary handles this. The replicas consume this WAL stream and apply the changes to their own data files, keeping them nearly in sync. This allows you to offload read-heavy queries to the replicas, freeing up the primary for writes.
• Horizontal Data Scaling (Partitioning): A native feature where a large table is logically divided into smaller physical pieces. For a blockchain indexer’s transactions table, you could PARTITION BY RANGE on the transaction date. When you query for recent transactions, the planner uses partition pruning to ignore all old partitions, leading to massive performance gains.
• Horizontal Write Scaling (Sharding): For workloads that overwhelm even a single primary writer, sharding is the answer. While not automatic, extensions like Citus Data transform PostgreSQL into a distributed cluster. Data is distributed across multiple nodes, and writes can be directed to the node that holds the relevant data shard, breaking the single-writer bottleneck.

Conclusion

PostgreSQL’s architecture is a masterclass in database design. The interplay between its multi-process model for stability, MVCC for non-blocking concurrency, WAL for durability, and a sophisticated planner for performance creates a system that is far more than the sum of its parts. By understanding these internal mechanics, you move from simply using a database to truly engineering a robust, scalable, and reliable data platform.

By Jatin Jain Saraf on July 27, 2025.

Every engineer responsible for a large database fears the “big one” — a problem so severe it brings operations to a crawl. We recently…

From Lag to Leading: A Deep Dive into Fixing a 6-Hour PostgreSQL Replication Lag

Every engineer responsible for a large database fears the “big one” — a problem so severe it brings operations to a crawl. We recently faced this crisis: a high-throughput indexer processing millions of daily transactions on a powerful PostgreSQL server had its read replica fall behind by over six hours. Queries were timing out, and our system was on the brink.

This is the story of how we diagnosed the cascading failures and brought the system back from the edge, not just by treating the lag itself, but by fixing the underlying issues that caused it.

Part 1: The First Clue — Checkpoint Hell

Our investigation started on the primary server. It was struggling to keep up, with disk I/O and CPU usage constantly high. The replication lag was a symptom, but the disease was on the primary.

• The Diagnostic Step: We enabled a crucial logging parameter: log_checkpoints = on. This gave us visibility into one of PostgreSQL's most I/O-intensive operations.
• The Smoking Gun: The logs were terrifying. We saw messages like **checkpoints are occurring too frequently (11 seconds apart)** and a constant stream of **checkpoint starting: wal**. The database was in "checkpoint hell," desperately trying to flush data to disk because it was running out of transaction log space.
• The Fix: The root cause was a **max_wal_size** setting that was too small for our write volume. On a powerful server with 128 GB of RAM, the default 1 GB was a drop in the ocean. We increased it to 16 GB.
• The Result: The effect was immediate. The frantic, size-based checkpoints stopped. The logs shifted to a calm, predictable **checkpoint starting: time**, occurring every 5 minutes. The primary server was stable, but our work wasn't done.

Part 2: The Second Problem — A Bloated Replica

Despite a healthy primary, the replica was still lagged and slow. The bottleneck had shifted. It wasn’t about the primary sending data anymore; it was about the replica’s ability to receive and apply it.

• The Diagnostic Step: We queried the pg_stat_all_tables view to inspect the health of our tables.
• The Horrifying Discovery: We found severe table bloat. One critical table had over 44 million dead tuples. The replica was a digital graveyard. Every query and every replicated write had to sift through this mountain of junk data, killing performance.
• The Quick Check: We also confirmed that **hot_standby_feedback** was enabled on the replica. This is a critical setting that prevents long-running read queries from causing replication conflicts.

Part 3: The Cleanup — Unleashing Autovacuum

The path forward was clear: we had to get rid of the bloat. Instead of a one-time manual VACUUM, we opted to create a sustainable, automated solution by making the autovacuum process hyper-aggressive.

• Global Tuning: First, we increased the number of workers and made them faster by setting **autovacuum_max_workers = 6** and **autovacuum_vacuum_cost_delay = 2ms**.
• The Critical Per-Table Fix: The most important change was telling autovacuum when to start. For our massive tables, the default trigger (20% of rows changed) would never happen. We ran the following command on our most bloated tables:

ALTER TABLE your_bloated_table SET (autovacuum_vacuum_scale_factor = 0.001);

• This forced autovacuum to run after only 0.1% of rows changed, ensuring constant, proactive cleanup.

Part 4: Interpreting the Healing Phase 🩺

After applying the autovacuum settings, we saw a curious pattern: the checkpoint starting: wal messages returned. For a moment, it felt like a step backward, but it was actually a sign of success.

The newly aggressive autovacuum was working so hard cleaning up the bloat that its own operations were generating a massive amount of WAL. This activity was so intense that it temporarily outpaced our 5-minute checkpoint timer. The database wasn't in trouble; it was healing itself. After a few hours, as the worst of the bloat was cleared, the WAL generation subsided, and the checkpoints returned to their stable, time-based schedule.

Conclusion: Our Key Takeaways

This experience reinforced several core database administration principles:

1. Replication lag is a symptom, not the disease. The root causes were I/O bottlenecks and table bloat.
2. Tune your primary first. A stable primary is the foundation of healthy replication. Ensure max_wal_size is appropriate for your workload and hardware.
3. Bloat is a silent killer. For write-intensive systems, aggressive autovacuum isn't just a good idea; it's a requirement. Don't trust the default settings on very large tables.
4. Monitor everything. You can’t fix what you can’t see. Using tools like log_checkpoints and system views like pg_stat_all_tables is essential for proper diagnosis.
5. Be patient. After applying a fix, give the system time to react and stabilize. The “healing phase” can look noisy, but it’s often a sign that your solution is working.

#PostgreSQL #DatabaseTuning #ReplicationLag #PerformanceTuning #DevOps #SRE #Database #HighAvailability #SoftwareEngineering #Tech #CaseStudy #Backend #CloudSQL

By Jatin Jain Saraf on July 23, 2025.

In distributed systems that process massive volumes of real-time data, such as blockchain indexers, database replication is essential for…

Taming PostgreSQL Replication Lag in Real-Time Blockchain Indexers

In distributed systems that process massive volumes of real-time data, such as blockchain indexers, database replication is essential for scalability, fault tolerance, and read optimization. However, when replication isn’t tuned for high-ingestion workloads, it can lead to delays known as replication lag.

This challenge is especially relevant in systems built on PostgreSQL, where replication is robust but also sensitive to write intensity and resource limits.

What is Replication Lag?

Replication lag is the delay between a write occurring on the primary database and the time that the change appears on one or more replica databases. In systems that route read-heavy traffic to replicas (a.k.a. read replicas or followers), replication lag can result in stale reads, inconsistencies between the data being written and what users or services are reading.

Why Indexers Suffer from Replication Lag

Indexers like those powering blockchain explorers, real-time search engines, or analytics dashboards are constantly ingesting and storing high volumes of data. Common patterns include:

• Thousands to millions of insertions per minute
• Frequent bulk writes and upserts
• Continuous schema changes or reprocessing cycles

These stress the primary database, but the replication overhead falls on the replicas, especially in PostgreSQL, which uses Write-Ahead Logging (WAL) to maintain consistency.

Real-World Impact of Replication Lag

• Stale Reads: Applications querying replicas may not see the most recent data.
• Consistency Issues: Time-sensitive operations (e.g., monitoring alerts, transaction lookups) may behave unpredictably.
• Debugging Complexity: Developers and users may encounter discrepancies between data seen on the UI and what’s actually stored.

How PostgreSQL Replication Works (Under the Hood)

PostgreSQL uses physical streaming replication as its most common replication method. Here’s how it works:

1. WAL Generation:
    All changes to the database are first written to WAL (Write-Ahead Log) segments.
2. WAL Shipping to Replicas:
    The primary sends these WAL files to replicas over a TCP connection.
3. WAL Replay:
    Replicas replay the WAL records to reflect the exact state of the primary.
4. Feedback Loop:
    Replicas send acknowledgements (in async setups) indicating the LSN (Log Sequence Number) up to which they’ve replayed.

Why Replication Lag Happens in PostgreSQL

1. WAL Replay Bottlenecks:
    If the replica can’t replay WAL fast enough (due to disk I/O or CPU limitations), it lags.
2. Network Latency:
    Even if WAL files are generated quickly, slow transfer over the network delays replication.
3. Disk IOPS Limitation:
    Replay of WAL on the replica involves I/O-intensive operations. If the replica’s disk isn’t fast enough, WAL gets backlogged.
4. Hot Standby Conflicts:
    In replicas configured for read queries (hot_standby = onLong-running queries can conflict with WAL replay—PostgreSQL may delay applying WAL to avoid cancelling queries.
5. Large Transactions:
    WAL size increases dramatically for bulk operations. Replicas have to process the entire transaction before replaying, delaying visibility.
6. Checkpointing and Vacuuming:
    Postgres background processes like checkpointer and autovacuum can stall WAL replay if the system is already under pressure.

Why Not Use Synchronous Replication?

While synchronous replication guarantees strong consistency, it’s not suitable for indexers due to:

• Performance Impact: The primary must wait for at least one replica to confirm each write, increasing write latency significantly.
• Reduced Throughput: High-volume insert operations slow down to match the speed of the slowest participating replica.
• Availability Risks: If a replica goes offline or slows down, the primary may block new writes, a dangerous bottleneck in real-time systems.

Why We Use Eventual Consistency Instead

For blockchain indexers, eventual consistency via asynchronous replication is more practical:

• High throughput for inserts
• Independent, scalable reads from replicas
• Non-blocking writes, even if replicas lag briefly

For critical paths like user-triggered transaction lookups, we directly query the primary to ensure data freshness. How to Mitigate Replication Lag

Strategies to Reduce Replication Lag

1. Optimize Writes
    Break down bulk inserts into smaller chunks or use batched writes strategically. Avoid unnecessary updates or deletes.
2. Use Synchronous Replication for Critical Data
    For highly sensitive operations, synchronous replication ensures consistency at the cost of some performance.
3. Scaling Replicas Vertically or Horizontally
    Add more compute resources (CPU, memory, IOPS) to replicas or distribute reads across more nodes.
4. Monitoring & Alerting
    Set up real-time monitoring (e.g., PostgreSQL’s pg_stat_replication or MySQL’s Seconds_Behind_Master) to catch replication lag early.
5. Dedicated Replication Channels
    If supported, isolate replication traffic from normal network traffic to prevent interference.
6. Prioritise Write Load Management
    If possible, buffer high-frequency writes using queues or streams (Kafka, RabbitMQ) and process them with controlled throughput.

Tips to Investigate & Reduce Replication Lag in PostgreSQL

Monitor **pg_stat_replication**
 Use this built-in view to see:

• write_lagWAL write delay
• flush_lagWAL flush delay
• replay_lagWAL replay delay

Example query:
SELECT pid, client_addr, state, write_lag, flush_lag, replay_lag
FROM pg_stat_replication;

• Check WAL Size Growth
   Monitor WAL generation rate via tools like pg_stat_wal or pg_stat_bgwriter.
• Tune **max_wal_size**, **checkpoint_timeout**, and **wal_compression**
   These affect how WAL is generated and retained.
• Use **archive_mode = on** + WAL Archiving
   Useful for delayed replicas or if streaming replication is interrupted.
• Tune Disk I/O & IOPS
   Make sure replicas are on high-throughput storage (e.g., NVMe, SSDs).
• Enable Logging of Replication Delays
   Helps identify problematic patterns: log_replication_commands = on
• Avoid Long-Running Queries on Replicas
   They can block WAL replay. Monitor using:

SELECT pid, age(now(), query_start), query FROM pg_stat_activity
WHERE state = 'active' AND backend_type = 'client backend';

Observability = Diagnosability

Replication lag should never be a guessing game. PostgreSQL gives you precise views into where the bottleneck is:

• WAL Write Lag (on primary)
• WAL Shipping Lag (network)
• WAL Replay Lag (on replica)

With this observability, you can implement targeted and effective solutions.

Conclusion

In high-ingestion, real-time systems like blockchain indexers, replication lag is an inevitable but manageable challenge, especially with PostgreSQL’s WAL-based architecture.

By embracing eventual consistency, designing for latency-tolerant reads, and monitoring replication health with PostgreSQL’s native tools, you can build systems that are:

• Fast
• Reliable
• Scalable

With the right strategy and observability, replication lag becomes just another solvable layer in your system architecture, not a blocker.

TL;DR

• Replication lag in PostgreSQL affects read consistency in high-ingestion systems like blockchain indexers.
• Causes include WAL replay lag, network issues, and hot standby conflicts.
• Use async replication for throughput and query primaries for critical paths.
• Monitor withpg_stat_replication, tune WAL settings, and optimise write patterns.

By Jatin Jain Saraf on July 10, 2025.

In Move-based blockchains such as Aptos or Sui, smart contracts are written and deployed using the Move language. While developers interact…

Understanding Source Code vs Bytecode in Move-Based Blockchains

In Move-based blockchains such as Aptos or Sui, smart contracts are written and deployed using the Move language. While developers interact with source code, what runs on-chain is the compiled bytecode. Understanding the distinction between these two is essential for transparency, security, and trust in decentralised applications.

What is Source Code?

Source code refers to the original, human-readable logic written by developers using the Move programming language. It typically includes modules, functions, type definitions, and comments, and it is structured to be easy to read and understand.

For example:

module MyToken {  public fun mint(recipient: address, amount: u64) {  // mint logic  }  }

This is the blueprint developers write, review, and maintain to define how the smart contract behaves.

What is Bytecode?

Bytecode is the compiled, binary representation of the Move source code. It is the program version deployed to and executed by the Move Virtual Machine (MoveVM) on the blockchain.

Bytecode is generated by the Move compiler, such asmovec, and appears in hexadecimal or binary format. It is not human-readable and is optimised for execution by the blockchain runtime. Once deployed, the bytecode is stored on-chain and cannot be changed.

Why Are Source Code and Bytecode Different?

There are several reasons why they may differ:

1. The compilation process transforms the source code by stripping out comments, formatting, and converting high-level logic into low-level operations.
2. Compilers often apply optimisations for performance and compatibility with the virtual machine.
3. In some cases, developers may upload source code that doesn’t exactly match the deployed bytecode — either unintentionally or deliberately.

Implications for Trust and Transparency

These differences can create serious implications:

• If users trust the displayed source code without verifying the bytecode, they may be misled.
• Auditors could spend time analyzing source code that isn’t actually executed.
• Malicious developers could exploit this mismatch to hide unwanted logic.

Why Do Blockchain Explorers Display a Warning?

Many explorers include a warning like:

“The source code is plain text uploaded by the deployer, which can be different from the actual bytecode.”

This line is meant to caution users that what they see on the explorer may not represent the actual logic being executed on-chain. It encourages users to verify the contract to ensure transparency and trust.

Source Code Verification

To solve this issue, some explorers offer a “Verify Source Code” feature. This process involves:

1. Uploading the source code and build settings.
2. Recompiling it using the same compiler version and flags.
3. Comparing the output bytecode with what’s stored on-chain.
4. Marking the contract as verified if there’s a perfect match.

This builds confidence that the code users are reading is exactly what the blockchain is running.

Conclusion

In summary, source code is the readable logic written by developers, while bytecode is the machine-readable version that actually runs on the blockchain. Because only bytecode is deployed and executed, and since source code can be different or even misleading, explorers issue a warning and offer verification features. Verifying contracts helps the ecosystem remain secure, auditable, and trustworthy.

By Jatin Jain Saraf on May 7, 2025.

In today’s rapidly evolving software world, especially where microservices and distributed systems dominate, API gateways have become more…

🔍 Demystifying API Gateway Patterns: Design Choices, Real-World Use Cases & When to Use What

In today’s rapidly evolving software world, especially where microservices and distributed systems dominate, API gateways have become more than reverse proxies — they’re the cornerstone of secure, scalable, and maintainable architectures.

But with multiple gateway patterns, how do you choose the one that fits your architecture best?

In this blog, we’ll explore:

• What an API gateway is (beyond the buzzwords)
• The role it plays in modern systems
• 5 architectural patterns for deploying API gateways
• Real-world use cases for each

🚪 What Is an API Gateway — And Why Should You Care?

Think of an API gateway as the front desk of your application’s backend. It’s the one point of contact that every external client (mobile apps, web apps, third-party integrations) interacts with before reaching your internal services.

But it’s more than a router:

• It enforces security policies
• Aggregates or transforms responses
• Monitors and logs incoming/outgoing traffic
• Handles rate limiting, retries, and timeouts
• Translates protocols (e.g., REST to gRPC)

In short, an API gateway simplifies client interactions while centralizing control of your backend traffic.

🔄 The Rise of Modern API Gateways

Older gateway solutions (like NGINX or Apache-based proxies) still serve basic needs, but today’s cloud-native environments demand more.

That’s where modern solutions like Solo.io’s Gloo Gateway or Kong, built on powerful proxies like Envoy, come into play.

Why they matter:

• Designed for Kubernetes and microservices
• Support for advanced routing, service discovery
• First-class support for WebAssembly, GraphQL, gRPC
• Rich ecosystem of plugins and integrations

Whether you’re running on bare metal, Kubernetes, or a hybrid cloud — modern API gateways are built to scale, perform, and adapt.

🧩 5 Common API Gateway Deployment Patterns

Let’s dive into the different ways you can structure your gateway layer. Each has its pros, cons, and ideal use cases.

1. Centralized Gateway Pattern

Overview: A single entry point for all external API calls. The gateway sits at the edge and handles everything — authentication, routing, rate-limiting, etc.

🧠 Ideal for:

• Simpler systems
• Monolithic-to-microservices transitions
• MVPs or startups

✅ Pros:

• Easier to manage and monitor
• Centralized policy enforcement
• Low operational complexity

⚠️ Cons:

• Can become a bottleneck
• Single point of failure (unless highly available)

2. Tiered (Multi-Level) Gateway Pattern

Overview: Requests first hit an external gateway (client-facing) and then flow through an internal gateway that talks to microservices.

🧠 Ideal for:

• Enterprises with strict separation between external/internal traffic
• Regulated industries (finance, healthcare)

✅ Pros:

• Extra security layer
• Internal traffic is abstracted from public clients
• Enables fine-grained routing at the internal level

⚠️ Cons:

• Slightly higher latency
• More moving parts = more complexity

3. Service-Level (Microgateway) Pattern

Overview: Each service or group of services has its own mini gateway.

🧠 Ideal for:

• Polyglot microservice environments
• Autonomous development teams

✅ Pros:

• Teams own their own API contracts and security rules
• Flexible versioning and rollout strategies

⚠️ Cons:

• Operationally intensive
• Harder to enforce organization-wide standards

4. Pod-Level Gateway Pattern (Kubernetes-centric)

Overview: A dedicated gateway instance (or sidecar proxy) per pod.

🧠 Ideal for:

• Per-tenant traffic isolation
• Use cases involving heavy multitenancy

✅ Pros:

• Traffic isolation at pod level
• Dynamic routing at a fine-grained scale

⚠️ Cons:

• Overhead scales with pods
• It may not be necessary for most use cases

5. Service Mesh + Gateway (Sidecar Pattern)

Overview: Combine an ingress gateway (entry point) with a service mesh (e.g., Istio, Linkerd) that manages service-to-service communication via sidecar proxies.

🧠 Ideal for:

• Zero-trust environments
• Enterprises with complex routing, observability, and policy requirements

✅ Pros:

• Deep observability (metrics, tracing, logging)
• Fine-grained traffic management
• Built-in mTLS, retries, and circuit breaking

⚠️ Cons:

• High complexity and learning curve
• Requires a solid DevOps foundation

🧠 How to Choose the Right Pattern for Your Architecture

1. Quick MVP or startup ===>Centralized Gateway
2. Security layering needed===>Tiered Gateway
3. Independent teams, CI/CD===>Microgateway
4. Pod-level isolation needed===>Pod-level Gateway
5. Need deep traffic insights===>Service Mesh + Sidecar

Still unsure? A good rule of thumb: start simple, scale when needed.

⚙️ Best Practices When Designing API Gateway Architectures

Here are a few lessons learned from real-world deployments:

1. Avoid business logic in gateways
    Gateways are not mini-apps. Keep logic minimal and offload complex tasks to services.
2. Secure all entry points
    Use OAuth2, JWT, mTLS, and rate limiting. Don’t leave unprotected APIs in your mesh.
3. Design for observability
    Collect metrics, logs, and traces right from your gateway layer. It’s your first line of insight.
4. Embrace API versioning
    Avoid breaking changes. Version your APIs and document them clearly.
5. Automate with CI/CD
    Treat your API gateway configs (routes, plugins) as code. Use GitOps or IaC for rollout.

✨ Final Thoughts

The API Gateway isn’t just a reverse proxy anymore — it’s a strategic control point in your architecture.

Whether you’re serving millions of users or just building your first microservice, picking the right pattern can dramatically impact performance, scalability, and developer productivity.

If you’re looking to future-proof your architecture, explore modern tools like Gloo Gateway, Kong, or Envoy that support cloud-native standards out of the box.

👉 Was this helpful? Hit the 💚 or share this with your team.
 Do you have questions or want real-world examples? Let’s chat in the comments!

By Jatin Jain Saraf on April 9, 2025.

Performance optimisation becomes a critical challenge when dealing with millions of records in a database. As datasets grow, queries that…

Handling Large Datasets & High-Traffic Queries: Optimizing Pagination, Sorting & Filtering

Performance optimisation becomes a critical challenge when dealing with millions of records in a database. As datasets grow, queries that once ran smoothly start slowing down, and when multiple users access the system simultaneously, things get even worse.

The Challenge: Why Do Queries Slow Down?

Even with proper indexing, large-scale applications often face the following issues:

• Pagination Performance Drops: Offset-based pagination (OFFSET n LIMIT m) forces the database to scan and discard rows before returning results, making deeper pages increasingly slower.
• Sorting and Filtering Lag: Queries that require sorting by timestamps, prices, or statuses can still become slow, even when indexed.
• Database Overload Due to High-Traffic Queries: If users frequently access high-activity data, like tracking transactions for a busy wallet, caching doesn’t always help because the data is constantly changing.

To ensure fast and scalable queries, we implemented the following optimizations.

Optimizing Pagination for Large Datasets

Problem:

Offset-based pagination (OFFSET n LIMIT m) requires the database to scan and discard n rows before fetching results. The deeper the page, the worse the performance. This is especially problematic for applications handling real-time blockchain transactions, financial records, or high-traffic APIs.

Solution: Cursor-Based Pagination

Instead of using OFFSET, we switched to cursor-based pagination, which fetches records directly from the last seen record. By using a unique, indexed column (e.g., transaction_id or timestamp), we eliminate unnecessary row scans, making pagination significantly faster.

Why it works: ✅ Reduces deep pagination lag ✅ Prevents scanning & discarding large data chunks ✅ Keeps API responses consistently fast

Enhancing Sorting & Filtering Performance

Problem:

Sorting and filtering transactions based on fields like timestamp, status, or amount can become slow, even when indexes are applied. Traditional indexes work well for some queries but struggle with efficiently filtering large datasets.

Solution: Partial Indexes & Materialized Views

• Partial Indexes: Instead of indexing an entire table, we created partial indexes for frequently used filters, such as transactions with status = 'SUCCESS'.
• Materialized Views: For common queries (e.g., “latest transactions”), precomputing results using materialized views significantly improved query performance.

Why it works: ✅ Reduces database load for frequent filtering ✅ Improves query execution times by fetching precomputed data ✅ Keeps results fresh with periodic updates

Handling High-Traffic Queries on Frequently Changing Data

Problem:

When users check frequently updated data (like an active wallet address), caching alone isn’t a viable solution. Since the data changes constantly, relying on Redis or Memcached can lead to outdated information.

Solution: Hybrid Caching + Event-Driven Updates

To handle this challenge efficiently, we combined multiple strategies:

1. Hybrid Caching Strategy: Instead of long-term caching, we used short-lived caches (5–10 seconds) along with background updates to reduce direct database hits.
2. Read Replicas: Queries were offloaded to read replicas, ensuring that high-volume reads didn’t overwhelm the primary database.
3. Event-Driven Updates (CDC — Change Data Capture): Instead of polling the database repeatedly, we implemented CDC using Kafka/PostgreSQL logical replication to push real-time updates whenever new transactions were recorded.
4. Optimized Query Execution: Instead of fetching all transactions for an active wallet, we optimized queries to fetch only new records since the last request, reducing dataset size and improving response time.

Why it works: ✅ Reduces database hits while keeping data fresh ✅ Handles high-traffic wallets with real-time updates ✅ Ensures scalability without compromising performance

Final Outcome & Key Takeaways

With these optimizations, we achieved:

✅ Faster API responses (up to 80% improvement in query execution time) ✅ Smooth pagination, sorting & filtering, even with millions of records ✅ Scalable architecture that handles real-time data efficiently ✅ Balanced database load, ensuring high availability under heavy user traffic

What’s Your Experience?

Have you faced similar challenges in handling large datasets with real-time queries? What worked for you? Let’s discuss your insights in the comments below! 🚀

#DatabaseScaling #Pagination #PerformanceOptimization #HighTrafficApps #BackendEngineering

By Jatin Jain Saraf on February 22, 2025.

Introduction

The Ultimate Git Command Guide: Streamline Your Development Workflow

Introduction

Git is a vital tool for developers, offering effective version control, teamwork, and project administration. Whether you’re solo or in a team, understanding Git commands is vital for smoothing your development process. This overview details the main Git commands you require, giving you a strong base for handling your code, monitoring alterations, and collaborating with fellow developers. Learning these commands can boost efficiency and maintain project momentum. Here are the essential Git commands every software engineer should master.

Configuration

1. git config — Configure Git settings, such as user name and email.
Example: git config --global user.name "Your Name"

2. git init — Initialize a new Git repository.
Example: git init

3. git clone_Purpose: Clone an existing repository._

_Example: git clone_ [_https://github.com/user/repo.git_](https://github.com/user/repo.git)

4. git status_Purpose: Show the working directory and staging area status._

_Example: git status_

5. git add_Purpose: Add file contents to the index (staging area)._

_Example: git add . (add all files)_

6. git commit_Purpose: Record changes to the repository._

_Example: git commit -m "Commit message"_

7. git push_Purpose: Update remote refs along with associated objects._

_Example: git push origin main_

8. git pull_Purpose: Fetch from and integrate with another repository or local branch._

_Example: git pull origin main_

9. git branch_Purpose: List, create, or delete branches._

_Example: git branch new-branch (create new branch)_

10. git checkout_Purpose: Switch branches or restore working tree files._

_Example: git checkout new-branch (switch to branch)_

11. git switch_Purpose: Switch branches._

_Example: git switch new-branch_

12. git merge_Purpose: Join two or more development histories together._

_Example: git merge new-branch (merge new-branch into current branch)_

13. git rebase_Purpose: Reapply commits on top of another base tip._

_Example: git rebase main_

14. git log_Purpose: Show commit logs._

_Example: git log --oneline_

15. git diff_Purpose: Show changes between commits, commit and working tree, etc._

_Example: git diff (show unstaged changes)_

16. git show_Purpose: Show various types of objects._

_Example: git show HEAD (show changes in the last commit)_

17. git stash_Purpose: Stash the changes in a dirty working directory away._

_Example: git stash_

18. git stash pop_Purpose: Apply the changes recorded in the stash to the working directory._

_Example: git stash pop_

19. git clean_Purpose: Remove untracked files from the working directory._

_Example: git clean -fd_

20. git remote_Purpose: Manage set of tracked repositories._

_Example: git remote add origin_ [_https://github.com/user/repo.git_](https://github.com/user/repo.git)

21. git fetch_Purpose: Download objects and refs from another repository._

_Example: git fetch origin_

22. git remote -v_Purpose: Show the URLs that a remote name corresponds to._

_Example: git remote -v_

23. git tag_Purpose: Create, list, delete, or verify a tag object._

_Example: git tag -a v1.0 -m "Version 1.0"_

24. git push origin --tags_Purpose: Push all tags to the remote repository._

_Example: git push origin --tags_

25. git reset_Purpose: Reset current HEAD to the specified state._

_Example: git reset --hard HEAD~1 (reset to previous commit)_

26. git revert_Purpose: Create a new commit that undoes the changes from a previous commit._

_Example: git revert HEAD_

27. git checkout --_Purpose: Discard changes in the working directory._

_Example: git checkout -- file.txt (discard changes in file.txt)_

28. git cherry-pick_Purpose: Apply the changes introduced by some existing commits._

_Example: git cherry-pick <commit-hash>_

29. git branch -d_Purpose: Delete a branch._

_Example: git branch -d branch-name_

30. git branch -D_Purpose: Force delete a branch._

_Example: git branch -D branch-name_

31. git merge --no-ff_Purpose: Create a merge commit even when the merge resolves as a fast-forward._

_Example: git merge --no-ff new-branch_

32. git rebase -i_Purpose: Start an interactive rebase._

_Example: git rebase -i HEAD~3_

33. git diff --staged_Purpose: Show changes between the index and the last commit._

_Example: git diff --staged_

34. git blame

Purpose: Show what revision and author last modified each line of a file.

_Example: git blame file.txt_

35. git log --graph_Purpose: Show a graph of the commit history._

_Example: git log --graph --oneline_

36. git reflog_Purpose: Show a log of all references._

_Example: git reflog_

37. git stash list_Purpose: List all stashes._

_Example: git stash list_

38. git stash apply_Purpose: Apply a stash to the working directory._

_Example: git stash apply stash@{1}_

39. git stash drop_Purpose: Remove a single stash entry from the list of stashes._

_Example: git stash drop stash@{1}_

40. git remote show_Purpose: Show information about the remote repository._

_Example: git remote show origin_

41. git remote rm_Purpose: Remove a remote._

_Example: git remote rm origin_

42. git pull --rebase_Purpose: Fetch and rebase the current branch on top of the upstream branch._

_Example: git pull --rebase origin main_

43. git fetch --all

Purpose: Fetch all remotes.

_Example: git fetch --all_

44. git bisect_Purpose: Use binary search to find the commit that introduced a bug._

_Example: git bisect start_

45. git submodule_Purpose: Initialize, update, or inspect submodules._

_Example: git submodule update --init_

46. git archive_Purpose: Create an archive of files from a named tree._

_Example: git archive --format=tar HEAD > archive.tar_

47. git shortlog_Purpose: Summarize git log output._

_Example: git shortlog -s -n_

48. git describe

Purpose: Give an object a human-readable name based on an available ref.

Example: git describe --tags

49. git rev-parse_Purpose: Parse revision (or other objects) and retrieve its hash._

_Example: git rev-parse HEAD_

50. git tag -d_Purpose: Delete a tag from the local repository._

_Example: git tag -d v1.0_

51. git checkout -b_Purpose: Create and switch to a new branch._

_Example: git checkout -b new-branch_

52. git push origin --delete_Purpose: Delete a remote branch._

_Example: git push origin --delete branch-name_

53. git cherry_Purpose: Find commits not merged upstream._

_Example: git cherry -v_

54. git rm_Purpose: Remove files from the working tree and from the index._

_Example: git rm file.txt_

55. git mv_Purpose: Move or rename a file, directory, or symlink._

_Example: git mv oldname.txt newname.txt_

56. git reset HEAD_Purpose: Unstage changes._

_Example: git reset HEAD file.txt_

57. git log -p_Purpose: Show changes over time for a specific file._

_Example: git log -p file.txt_

58. git diff --cached_Purpose: Show changes between the index and the last commit (same as --staged)._

_Example: git diff --cached_

59. git apply_Purpose: Apply a patch to files and/or to the index._

_Example: git apply patch.diff_

60. git format-patch_Purpose: Prepare patches for e-mail submission._

_Example: git format-patch -1 HEAD_

61. git am_Purpose: Apply a series of patches from a mailbox._

_Example: git am < patch.mbox_

62. git cherry-pick --continue_Purpose: Resume cherry-picking after resolving conflicts._

_Example: git cherry-pick --continue_

63. git fsck_Purpose: Verify the connectivity and validity of objects in the database._

_Example: git fsck_

64. git gc_Purpose: Cleanup unnecessary files and optimize the local repository._

_Example: git gc_

65. git prune_Purpose: Remove unreachable objects from the object database._

_Example: git prune_

66. git notes_Purpose: Add or inspect object notes._

_Example: git notes add -m "Note message"_

67. git whatchanged_Purpose: Show what changed, similar to git log._

_Example: git whatchanged_

68. git show-branch_Purpose: Show branches and their commits._

_Example: git show-branch_

69. git verify-tag_Purpose: Check the GPG signature of tags._

_Example: git verify-tag v1.0_

70. git show-ref_Purpose: List references in a local repository._

_Example: git show-ref_

Twitter Account: Twitter

By Jatin Jain Saraf on July 9, 2024.

Strings are a fundamental part of programming in JavaScript, representing textual data. In this guide, we will explore the intricacies of…

Strings in JavaScript: An In-Depth Guide

Strings are a fundamental part of programming in JavaScript, representing textual data. In this guide, we will explore the intricacies of strings in JavaScript, including their creation, manipulation, and the top 50 methods you can use to handle them effectively.

Introduction to Strings

In JavaScript, a string is a sequence of characters used to represent text. They are immutable, meaning once created, their values cannot be changed.

String Creation

You can create strings in JavaScript using single quotes (`’`), double quotes (`”`), or backticks (`` ` ``) for template literals.

let singleQuote = ‘Hello, world!’;
let doubleQuote = “Hello, world!”;
let templateLiteral = `Hello, world!`;

String Properties

1. `length`
   The `length` property returns the number of characters in a string.

let text = “Hello”;
console.log(text.length); // Output: 5

2. `charAt()`
Returns the character at a specified index.

let text = “Hello”;
console.log(text.charAt(0)); // Output: H

3. `charCodeAt()`
Returns the Unicode of the character at a specified index.

let text = “Hello”;
console.log(text.charCodeAt(0)); // Output: 72

4. `concat()`
Joins two or more strings.

let text1 = “Hello”;
let text2 = “World”;
console.log(text1.concat(“ “, text2)); // Output: Hello World

5. `includes()`
Checks if a string contains a specified value.

let text = “Hello World”;
console.log(text.includes(“World”)); // Output: true

6. `endsWith()`
Checks if a string ends with a specified value.

let text = “Hello World”;
console.log(text.endsWith(“World”)); // Output: true

7. `indexOf()`
   Returns the index of the first occurrence of a specified value.

let text = “Hello World”;
console.log(text.indexOf(“World”)); // Output: 6

8. `lastIndexOf()`
Returns the index of the last occurrence of a specified value.

let text = “Hello World World”;
console.log(text.lastIndexOf(“World”)); // Output: 12

9. `match()`
Searches a string for a match against a regular expression.

let text = “Hello World”;
console.log(text.match(/World/)); // Output: [“World”]

10. `matchAll()`
Returns an iterator of all results matching a string against a regular expression.

let text = “test1test2”;
const matches = text.matchAll(/t(e)(st(\d?))/g);
for (const match of matches) {
console.log(match);
}
// Output:
// [“test1”, “e”, “st1”, “1”]
// [“test2”, “e”, “st2”, “2”]

11. `padEnd()`
Pads the current string with another string until the resulting string reaches the given length.

let text = “Hello”;
console.log(text.padEnd(10, “*”)); // Output: Hello*****

12. `padStart()`
Pads the current string with another string until the resulting string reaches the given length.

let text = “Hello”;
console.log(text.padStart(10, “*”)); // Output: *****Hello

13. `repeat()`
Returns a new string with a specified number of copies of an existing string.

let text = “Hello”;
console.log(text.repeat(3)); // Output: HelloHelloHello

14. `replace()`
Searches for a match between a regular expression and a string, and replaces the matched substring with a new substring.

let text = “Hello World”;
console.log(text.replace(“World”, “Everyone”)); // Output: Hello Everyone

15. `replaceAll()`
Returns a new string with all matches of a pattern replaced by a replacement.

let text = “Hello World World”;
console.log(text.replaceAll(“World”, “Everyone”)); // Output: Hello Everyone Everyone

16. `search()`
    Executes a search for a match between a regular expression and a specified string.

let text = “Hello World”;
console.log(text.search(“World”)); // Output: 6

17. `slice()`
Extracts a section of a string and returns it as a new string.

let text = “Hello World”;
console.log(text.slice(0, 5)); // Output: Hello

18. `split()`
Splits a string into an array of substrings.

let text = “Hello World”;
console.log(text.split(“ “)); // Output: [“Hello”, “World”]

19. `startsWith()`
Checks if a string starts with a specified value.

let text = “Hello World”;
console.log(text.startsWith(“Hello”)); // Output: true

20. `substring()`
Returns a subset of a string between one index and another.

let text = “Hello World”;
console.log(text.substring(0, 5)); // Output: Hello

21. `toLowerCase()`
Converts a string to lowercase letters.

let text = “Hello World”;
console.log(text.toLowerCase()); // Output: hello world

22. `toUpperCase()`
    Converts a string to uppercase letters.

let text = “Hello World”;
console.log(text.toUpperCase()); // Output: HELLO WORLD

23. `trim()`
    Removes whitespace from both ends of a string.

let text = “ Hello World “;
console.log(text.trim()); // Output: Hello World

24. `trimEnd()`
Removes whitespace from the end of a string.

let text = “ Hello World “;
console.log(text.trimEnd()); // Output: “ Hello World”

25. `trimStart()`

Removes whitespace from the start of a string.

let text = “ Hello World “;
console.log(text.trimStart()); // Output: “Hello World “

26. `valueOf()`
Returns the primitive value of a string object.

let text = new String(“Hello World”);
console.log(text.valueOf()); // Output: “Hello World”

27. `codePointAt()`
Returns a non-negative integer that is the Unicode code point value.

let text = “Hello”;
console.log(text.codePointAt(0)); // Output: 72

28. `fromCharCode()`
Creates a string from the specified sequence of UTF-16 code units.

console.log(String.fromCharCode(72, 101, 108, 108, 111)); // Output: “Hello”

29. `fromCodePoint()`
Creates a string from the specified sequence of code points.

console.log(String.fromCodePoint(128512)); // Output: “😀”

30. `raw()`
Returns a raw string from a template string.

console.log(String.raw`Hello\nWorld`); // Output: “Hello\\nWorld”

31. `localeCompare()`
Compares two strings in the current locale.

let text1 = “a”;
let text2 = “b”;
console.log(text1.localeCompare(text2)); // Output: -1

32. `normalize()`
    Returns the Unicode Normalization Form of the string.

let text = “\u004F\u030C”;
console.log(text.normalize()); // Output: “Ǒ”

33. `toString()`
    Returns a string representing the specified object.

let text = new String(“Hello”);
console.log(text.toString()); // Output: “Hello”

34. `toLocaleLowerCase()`
Converts a string to lowercase letters, according to the host’s current locale.

let text = “Hello World”;
console.log(text.toLocaleLowerCase()); // Output: hello world

35. `toLocaleUpperCase()`
Converts a string to uppercase letters, according to the host’s current locale.

let text = “Hello World”;
console.log(text.toLocaleUpperCase()); // Output: HELLO WORLD

36. **`anchor()`
**Creates an HTML anchor.

let text = “Hello”;
console.log(text.anchor(“myanchor”)); // Output: Hello

37. `big()`
Creates a string to be displayed in a big font.

let text = “Hello”;
console.log(text.big()); // Output: Hello

38. `blink()`
Creates a string to be displayed as blinking text.

let text = “Hello”;
console.log(text.blink()); // Output: Hello

39. `bold()`
Creates a string to be displayed as bold.

let text = “Hello”;
console.log(text.bold()); // Output: Hello

40. `fixed()`
Creates a string to be displayed in a fixed-pitch font.

let text = “Hello”;
console.log(text.fixed()); // Output: Hello

41. `fontcolor()`
Creates a string to be displayed in a specified color.

let text = “Hello”;
console.log(text.fontcolor(“red”)); // Output: Hello

42. `fontsize()`
Creates a string to be displayed in a specified size.

let text = “Hello”;
console.log(text.fontsize(7)); // Output: <font size=”7">Hello

43. `italics()`
Creates a string to be displayed as italic.

let text = “Hello”;
console.log(text.italics()); // Output: Hello

44. `link()`
Creates a string to be displayed as a hyperlink.

let text = “Hello”;
console.log(text.link(“https://www.example.com")); // Output: <a href=”https://www.example.com"\>Hello

45. `small()`
Creates a string to be displayed in a small font.

let text = “Hello”;
console.log(text.small()); // Output: Hello

46. `strike()`
Creates a string to be displayed with a strikethrough.

let text = “Hello”;
console.log(text.strike()); // Output: Hello

47. `sub()`
Creates a string to be displayed as subscript.

let text = “Hello”;
console.log(text.sub()); // Output: _Hello

48. `sup()`
Creates a string to be displayed as superscript.

let text = “Hello”;
console.log(text.sup()); // Output: ^Hello

49. `trimLeft()`
    Removes whitespace from the start of a string.

let text = “ Hello World “;
console.log(text.trimLeft()); // Output: “Hello World “

50. `trimRight()`
Removes whitespace from the end of a string.

let text = “ Hello World “;
console.log(text.trimRight()); // Output: “ Hello World”

Conclusion

Strings are an essential part of JavaScript, and knowing how to manipulate them effectively is crucial for any developer. This guide covered the basics of strings and provided a comprehensive list of methods to work with them. Whether you’re searching, replacing, or formatting text, these methods will help you handle strings efficiently in your JavaScript projects.

By Jatin Jain Saraf on June 21, 2024.

When working with TypeScript in Node.js, you often need to define the shape of objects. This is where TypeScript’s type and interface come…

Choosing Between Type and Interface in TypeScript: A Detailed Guide

When working with TypeScript in Node.js, you often need to define the shape of objects. This is where TypeScript’s type and interface come into play. Both are used to describe the structure of an object, but choosing between them depends on several factors. Here’s a detailed guide to help you make an informed decision.

1. Usage Intentions

• Interfaces: Best for defining the structure of objects and classes. They serve as contracts that ensure a class or object adheres to a particular shape.
• Types: Ideal for creating aliases for primitive types, union types, tuple types, and more complex type expressions.

2. Extensibility

• Interfaces: Can be extended using the extends keyword. This feature supports hierarchical and flexible designs, making it easy to create new interfaces that build upon existing ones.
• Types: Can be extended using intersection types (&), which combine multiple types into one.

interface User {
id: number;
username: string;
email: string;
}

interface Admin extends User {
adminLevel: number;
}

type User = {
id: number;
username: string;
email: string;
};

type Admin = User & {
adminLevel: number;
};

3. Declaration Merging

• Interfaces: Support declaration merging, which means you can define the same interface multiple times, and TypeScript will merge them into a single definition.
• Types: Do not support declaration merging. Attempting to redefine a type alias will result in an error.

interface User {
id: number;
username: string;
}

interface User {
email: string;
}

// Merged User interface: { id: number; username: string; email: string; }

type User = {
id: number;
username: string;
};

type User = {
email: string;
}; // Error: Duplicate identifier 'User'

4. Complex Types

• Types: More powerful for defining complex types such as union types, intersection types, or tuples.

type User = {
id: number;
username: string;
email: string;
};

type ApiResponse = User | { error: string };

Practical Examples

• Interface Example:

interface User {
id: number;
username: string;
email: string;
}

function getUserById(id: number): User {
return { id, username: "john_doe", email: "john@example.com" };
}

Type Example:

type User = {
id: number;
username: string;
email: string;
};

function getUserById(id: number): User {
return { id, username: "john_doe", email: "john@example.com" };
}

When to Use Each

Use **interface** when:

• Defining the shape of an object or class.
• Expecting the type to be extended or implemented by other types.
• Benefiting from declaration merging.

Use **type** when:

• Defining complex types (e.g., union, intersection, tuple).
• Creating type aliases for primitives, union, and intersection types.
• Utilising type inference extensively.

Conclusion

Both type and interface are essential tools in TypeScript, each with its unique strengths. Interfaces are excellent for defining object shapes and supporting extensibility and declaration merging. Types excel at creating complex types and leveraging TypeScript’s powerful type inference. Understanding these distinctions will help you write more robust, maintainable, and scalable TypeScript code.

Choose wisely based on your specific use case, and don’t hesitate to use both where appropriate to take full advantage of TypeScript’s type system. Happy coding!

By Jatin Jain Saraf on June 19, 2024.

Introduction:

Unveiling the Power of Descriptive Naming Conventions in JavaScript

Introduction:

In the world of programming, clarity and readability are paramount. Among the many tools at a developer’s disposal, one often overlooked but potent technique is the art of naming variables descriptively. In JavaScript, where agility and efficiency are prized, the choice of variable names can significantly impact code comprehension and maintainability. Let’s delve into the profound impact of descriptive naming conventions and explore the diverse perspectives developers encounter when crafting variable names in JavaScript.

**Why Descriptive Naming Matters:
**Clear and descriptive variable names serve as a form of self-documentation within the codebase. They act as signposts, guiding fellow developers through the logic and functionality encapsulated within each variable. Imagine stumbling upon a cryptically named variable like x or temp. Deciphering its purpose requires extra mental effort and potentially interrupts the flow of understanding. Conversely, encountering a well-named variable such as userProfile or totalSales instantly communicates its role, reducing cognitive overhead and enhancing code comprehension.

Furthermore, descriptive naming fosters code maintainability by reducing the need for extraneous comments or documentation. Instead of relying solely on inline explanations, meaningful variable names convey intent and context, making the code more self-explanatory. This not only streamlines the development process but also facilitates smoother collaboration among team members, as everyone can quickly grasp the purpose and function of each variable.

**The Journey of Naming Variables:
**Naming variables in JavaScript is more than a mechanical task; it’s a journey that encapsulates various perspectives and considerations. Developers often tread the fine line between brevity and clarity, striving to strike a balance that ensures concise yet descriptive names. This balance becomes particularly crucial in the fast-paced environment of JavaScript development, where efficiency is key.

Consider the different perspectives developers adopt when naming variables. Some may prioritize brevity, opting for succinct names like num for "number" or str for "string." While these names save keystrokes, they sacrifice clarity and may lead to confusion, especially in larger codebases. Others lean towards verbosity, choosing names that explicitly spell out the variable's purpose, such as customerFirstName instead of fname. While descriptive, excessively long names can clutter the code and hinder readability.

Navigating these perspectives requires an understanding of the underlying trade-offs. Abbreviated names offer brevity but risk ambiguity, while verbose names provide clarity but may introduce noise. Finding the sweet spot often involves considering factors such as the variable’s scope, lifespan, and the broader context of the codebase.

**Best Practices for Descriptive Naming:
**To harness the full power of descriptive naming in JavaScript, developers should adhere to a set of best practices:

1. Prioritize Clarity: Choose names that convey the purpose and usage of the variable. Aim for names that are self-explanatory and require minimal additional context to understand.
2. Use Meaningful Context: Incorporate relevant context into variable names, such as function or scope. Prefixes like is for booleans or get for getter functions can provide valuable hints about a variable's behaviour.
3. Be Consistent: Establish and adhere to consistent naming conventions across projects and teams. Consistency fosters predictability and reduces the cognitive load when navigating different codebases.
4. Avoid Ambiguity: Steer clear of ambiguous or misleading names that can lead to confusion. Choose descriptive terms that accurately reflect the variable’s purpose and avoid generic placeholders like data or value.

Case Studies and Examples: Let’s examine a few real-world examples to illustrate the impact of descriptive naming on code readability:

Example 1: Poorly Named Variable

// Poorly named variable
let a = 10;
// Requiring additional context
if (a > 5) {
// What does "a" represent?
console.log("Greater than 5");
}

Example 2: Well-Named Variable

// Well-named variable
let numberOfStudents = 10;
// Clear and self-explanatory
if (numberOfStudents > 5) {
console.log("Class size is adequate");
}

In the first example, the variable a provides little insight into its purpose, requiring additional mental effort to decipher its meaning within the code. Conversely, the second example uses a descriptive name numberOfStudents, instantly clarifying its role and eliminating ambiguity.

Navigating Common Pitfalls:
While embracing descriptive naming conventions, developers should be wary of common pitfalls:

1. Overly Verbose Names: Avoid excessively long variable names that add unnecessary verbosity to the code without providing significant clarity.
2. Excessive Abbreviations: While brevity is essential, excessive abbreviations can obscure the meaning of variable names. Strike a balance between brevity and clarity.
3. Lack of Consistency: Inconsistent naming conventions within a codebase can lead to confusion. Establish clear guidelines and ensure consistency across all variables.

Conclusion
Descriptive naming conventions are a cornerstone of effective JavaScript development, enhancing code readability, comprehension, and maintainability. By prioritizing clarity and context in variable names, developers can streamline collaboration, reduce cognitive overhead, and foster more robust codebases. Embrace the power of descriptive naming, and unlock the full potential of your JavaScript projects.

**Call to Action
**Share your experiences and insights into naming variables in JavaScript. How do you approach the challenge of crafting descriptive names? Join the conversation and contribute to the collective knowledge of the developer community.

By Jatin Jain Saraf on May 29, 2024.

As a Full-stack developer with 4 years of experience, I’ve learned firsthand the importance of strategic skill development at every stage…

Navigating Your Career: Deciding Which Skills to Develop at Each Stage

As a Full-stack developer with 4 years of experience, I’ve learned firsthand the importance of strategic skill development at every stage of my career journey. Deciding which skills to cultivate can significantly impact your professional growth and trajectory. Here’s how I approach this critical decision-making process.

Setting a Career Roadmap

Before diving into skill development, it’s crucial to map out your career trajectory. Identify your long-term goals — whether it’s becoming a technical architect, leading a development team, or starting your venture. Understanding where you want to be in 5 or 10 years helps in charting a path for skill acquisition.

Assessing Current Skills

Evaluate your existing skill set. As a Full Stack developer, you likely possess proficiency in web development technologies, frameworks, databases, and possibly some soft skills like problem-solving and communication. Recognize your strengths and areas for improvement.

Understanding Career Stages

Career progression can be broadly categorized into stages — entry-level, mid-level, and senior/expert level. Each stage demands a specific skill set:

Entry-Level (0–2 years):

Focus on foundational technical skills (e.g., programming languages, version control, web frameworks).
Develop collaboration and basic project management skills.

Mid-Level (2–5 years):

Deepen technical expertise in specific domains (e.g., front-end, back-end, databases).
Enhance leadership and mentoring capabilities by taking on more responsibilities.

Senior/Expert Level (5+ years):

Master advanced technologies and architectures (e.g., cloud computing, microservices).
Cultivate strategic thinking, decision-making, and team leadership skills.

Identifying Industry Trends

Stay abreast of industry trends and emerging technologies. Research which skills are in demand and align with your career goals. For instance, with the rise of AI and machine learning, understanding these concepts can be advantageous for future opportunities.

Considering Personal Interests

Factor in your passions and interests when choosing skills to develop. Enjoy working with data? Consider diving deeper into data analytics or data science. Prefer creating seamless user experiences? Sharpen your front-end development skills.

Seeking Feedback and Mentorship

Engage with mentors or colleagues who have navigated similar career paths. Seek feedback on your skill development plan and identify blind spots or areas that might need attention.

Continual Learning and Adaptation

Lastly, remember that the tech landscape evolves rapidly. Continual learning is paramount. Allocate time for self-study, online courses, workshops, and conferences to keep your skills sharp and relevant.

Conclusion

Deciding which skills to develop at each career stage requires a blend of foresight, self-awareness, and adaptability. By strategically investing in the right skills, you pave the way for sustained professional growth, increased job satisfaction, and the fulfilment of your career aspirations. Remember, your skills are the building blocks of your career — choose wisely and embark confidently on your path to excellence.

Chart your course. Develop with purpose. Excel in every stage. 🚀

#TechCareer #SkillDevelopment #FullStackDeveloper #ProfessionalGrowth #CareerDevelopment #TechSkills #ContinuousLearning #WebDevelopment #SoftwareEngineering

By Jatin Jain Saraf on April 26, 2024.

The most recent version of the popular React Framework for building and developing web applications, Next.js 14, was released on October…

Next Js 14

The most recent version of the popular React Framework for building and developing web applications, Next.js 14, was released on October 26th, 2023. It promises to simplify and improve development speed, making it an excellent choice for any developer who wants to begin building fast and scalable web applications.

Before moving forward with the next 14 one should know about react

What do nextjs do

Next.js 14 includes several updates and improvements that will make the development of web applications a lot faster, let’s look at some of the important updates and improvements:

Turbopack

Turbopack, an exciting feature introduced in Next.js 14, represents a significant leap forward in enhancing local development speed. Building on a continuous improvement journey that began with Next.js 13, it has been fine-tuned for optimizing local development in both Pages and the App Router.

The Rust-based compiler is on the brink of stability, with a renewed focus on ensuring compatibility with all Next.js features.

Turbopack has powered 5,000 integration tests for the Next.js devs, which span seven years of bug fixes and reproductions. While on vercel.com, the results have been remarkable:

• Local server startup speeds have surged by up to 53.3%.
• Fast Refresh now delivers code updates up to 94.7% faster.

These gains aren’t just theoretical; they reflect tangible improvements, especially in larger applications with extensive module graphs. With 90% of the Next.js dev tests now successfully passing, you can anticipate consistently faster and more reliable performance when using next dev — turbo.

As we approach the milestone of 100% test passage, Turbopack will transition to a stable state in an upcoming minor release. Rest assured, we remain committed to supporting the use of Webpack for custom configurations and ecosystem plugins.

Server Actions

In Next.js 14, Server Actions represent a substantial evolution in granting developers greater authority over server-side rendering. This empowering enhancement equips developers with the capability to fetch essential data from the server before page generation, ensuring that vital information is readily available when the page loads. This not only accelerates the initial load times but also minimizes superfluous client-side requests.

These Server Actions are seamlessly woven into the fabric of the entire App Router model, offering a versatile toolkit that enables developers to:

• Dynamically revalidate cached data using functions like revalidatePath() or revalidateTag().
• Effortlessly steer users to different routes with the redirect() method.
• Manage cookies efficiently through the cookies() function, both for setting and retrieval.
• Harness optimistic UI updates, enhancing user experience, with the useOptimistic() utility.
• Handle and elegantly display server-side errors through the useFormState() function.
• Keep users informed with loading states on the client side, thanks to the useFormStatus() function.

This multifaceted suite of capabilities empowers developers to not only fine-tune server-side rendering but also to create more responsive and robust web applications with Next.js 14.

Enhanced Metadata Options

Importantly, the incorporation of meta tags into the initial page content ensures a seamless user experience. This practice prevents issues like page flickering due to theme color changes or layout shifts resulting from viewport adjustments.

Moreover, Next.js 14 distinguishes between blocking and non-blocking metadata, enabling a smoother experience. Only a select few metadata options are marked as blocking, ensuring that non-blocking metadata won’t hinder partially prerendered pages from serving the static shell.

Notably, the following metadata options are now deprecated and will be phased out in future major releases:

• viewport: Previously responsible for setting initial zoom and viewport properties.
• colorScheme: Used for specifying support modes (light/dark) for the viewport.
• themeColor: Defined the colour for the chrome surrounding the viewport.

Next.js 14 introduces the replacement options of viewport and generateViewport while retaining all other metadata options.

Partial Prerendering

The newest star in the release of Next.js 14 is Partial Prerendering. In a world where there is a constant battle between SSR and SSG, Next.js 14 gives you the best of both worlds. It provides you with a fast initial static response whilst streaming dynamic content based on your React Suspense boundaries, all this whilst eliminating the need to learn any new APIs. Thus giving you the speed of static sites and the dynamism of server-rendered applications.

Conclusion

Next.js 14 represents a significant step forward for the framework. Focusing on improving existing features rather than adding new ones, this version offers developers a sleeker, more efficient, and powerful experience. With Turbopack, Server Actions, Partial Prerendering, and integration with Strapi and artificial intelligence, Next.js 14 stands out as an ideal solution for developing modern and high-performance web applications. Developers now have an even more robust and versatile tool for creating innovative and effective web experiences.

To begin development with Next.js 14, you can make use of the official documentation or reference the Next.js 14 learn available on the website.

By Jatin Jain Saraf on February 26, 2024.

File and Directory Commands: ls: List files and directories. Syntax: ls [options] [directory] cd: Change directory. Syntax: cd [directory]…

Linux Terminal Basics Commands:

File and Directory Commands:
ls: List files and directories. Syntax: ls [options] [directory]
cd: Change directory. Syntax: cd [directory]
pwd: Print working directory. Syntax: pwd
mkdir: Create a new directory. Syntax: mkdir [directory]
cp: Copy files or directories. Syntax: cp [options] source destination
mv: Move or rename files/directories. Syntax: mv [options] source destination
rm: Remove/delete files or directories. Syntax: rm [options] file

File Manipulation:
cat: Concatenate and display the content of files. Syntax: cat [file]
nano Text editors to create or edit files. Syntax (nano): nano [file]_
Vim_: Text editors to create or edit files.Syntax (vim): vim [file]
touch: Create an empty file or update timestamp. Syntax: touch [file]

System Information:
uname -a: Display system information. Syntax: uname -a
df -h: Show disk space usage. Syntax: df -h
free -h: Display RAM usage. Syntax: free -h
t_op or htop_: Display running processes. Syntax: top or htop

Debian/Ubuntu:
sudo apt update: Update package lists. Syntax: sudo apt update
sudo apt upgrade: Upgrade installed packages. Syntax: sudo apt upgrade
sudo apt install : Install a new package. Syntax: sudo apt install [package]

Red Hat/Fedora:
sudo yum update: Update packages. Syntax: sudo yum update
sudo yum install: Install a new package. Syntax: sudo yum install [package]

Node.js
node -v: Check Node.js version. Syntax: node -v
npm -v: Check npm version. Syntax: npm -v
npm init: Initialize a new Node.js project. Syntax: npm init
node app.js: Run a Node.js script. Syntax: node [script]

NPM:
npm install : Install a Node.js package.Syntax: npm install [package]
npm install -g : Install a global package.Syntax: npm install -g [package]
npm start: Start the application (as defined in package.json).Syntax: npm start
npm test: Run tests.Syntax: npm test
npm run Run a custom script.Syntax: npm run [script]

ES6

ES Conventions

ES6

• The let keyword
• The const keyword
• Arrow Functions
• The … Operator
• For/of
• Map Objects
• Set Objects
• Classes
• Promises
• Symbol
• Default Parameters
• Function Rest Parameter
• String.includes()
• String.startsWith()
• String.endsWith()
• Array.from()
• Array keys()
• Array find()
• Array findIndex()
• New Math Methods
• New Number Properties
• New Number Methods
• New Global Methods
• Object entries
• JavaScript Modules

ES7

• JavaScript Exponentiation (**)
• JavaScript Exponentiation assignment (**=)
• JavaScript Array includes()

ES8

• JavaScript String padding
• JavaScript Object entries()
• JavaScript Object values()
• JavaScript async and await
• JavaScript Object.getOwnPropertyDescriptors

ES9

• Asynchronous Iteration
• Promise Finally
• Object Rest Properties
• New RegExp Features
• JavaScript Shared Memory

ES10

• String.trimStart()
• String.trimEnd()
• Object.fromEntries
• Optional catch binding
• Array.flat()
• Array.flatMap()
• Revised Array.Sort()
• Revised JSON.stringify()
• Separator symbols allowed in string litterals
• Revised Function.toString()

ES11

• BigInt
• String matchAll()
• The Nullish Coalescing Operator (??)
• The Optional Chaining Operator (?.)
• Logical AND Assignment Operator (&&=)
• Logical OR Assignment (||=)
• Nullish Coalescing Assignment (??=)
• Promise allSettled():
  Promise.allSettled([prom1,prom2,prom3]).then {}
• Dynamic Import

ES12

• Promise any()
  const first = await Promise.any([prom1,prom2,prom3]);
• String replaceAll()
• Numeric Separators (_)

By Jatin Jain Saraf on December 19, 2023.

When working on JavaScript, React, or Node.js projects, following best practices is essential for code readability, maintainability, and…

40 Best Practices for Writing Clean JavaScript, React, and Node.js Code

When working on JavaScript, React, or Node.js projects, following best practices is essential for code readability, maintainability, and overall code quality. In this article, we’ll explore 40 best practices to help you write clean and efficient code with these technologies.

1. Import Order Should be Correct: Organize imports logically, typically grouping third-party libraries, local modules, and built-in modules. This enhances code readability and maintainability. Source
2. Constant Values and Strings Should be in a Separate File: Centralize constants, strings, and static values in a separate file for better management and code reusability. Source
3. 2 Space Indentation: Maintain consistent indentation (typically two spaces) to improve code consistency and readability. Source
4. Capitalized Validation Messages: Format validation messages with proper capitalization for a professional look and feel. Source
5. Use Hooks in Every Case: Utilize React hooks for managing state, effects, and other React-related functionalities. They provide a more concise and functional approach. Source
6. Convert Class Components to Functional Components: Refactor class-based components to functional components using hooks, as they are the modern and recommended approach. Source
7. Convert Normal Function to ES6 Arrow Function: Refactor traditional functions to ES6 arrow functions for a more concise syntax and lexical scoping benefits. Source
8. Avoid Using Lodash `__get` for Static Data Access: When accessing static data, avoid using special libraries like Lodash for simplicity and readability. Direct access is preferred. Source
9. Use Nullish Coalescing and Optional Chaining: Utilize nullish coalescing (`??`) and optional chaining (`?.`) operators for safer and more concise code, especially when dealing with potentially null or undefined values. Source
10. Destructure Repeatedly Used Properties: Destructure object properties when they’re used multiple times within a function or component to make the code cleaner and more readable. Source
11. Remove Unused/Redundant Props: Eliminate unused or redundant props from components to streamline code and reduce clutter. Source
12. Check Console for Warnings After Refactoring: Ensure that the console doesn’t show any warning messages after refactoring code to maintain code quality and stability.
13. Review Data from API Calls: Examine API responses in the network tab to identify and remove unnecessary data not being used in the application for improved performance.
14. Camel Case Naming Convention: Follow the camel case naming convention for variables, functions, and other identifiers to maintain code consistency and readability. Source
15. Avoid Function Names Starting With Underscore: Refrain from using underscores as the starting character in function names, as it may imply privacy in some contexts not applicable in JavaScript. Use meaningful names instead. Source
16. Remove Unnecessary Div/Fragments: Eliminate unnecessary `
    ` or `` elements to simplify the component structure and improve code readability. Source
17. Use `console.error` and `console.info`: Utilize `console.error` for displaying error messages and `console.info` for informative messages in the console, aiding in debugging and monitoring. Source
18. Proper Try-Catch Blocks: Implement appropriate try-catch blocks for error handling, especially when dealing with potentially failing operations. Source
19. Avoid One-Line Functions for State Changes: Refrain from using one-liners for complex state changes or side effects to maintain code readability and clarity.
20. Extract Reusable Components: Identify patterns in your code and extract common logic into reusable components or custom hooks to reduce duplication and follow the DRY (Don’t Repeat Yourself) principle. Source
21. Avoid Magic Numbers: Replace numeric literals with named constants or variables that describe their purpose for better code comprehension. Source
22. Limit Function/Component Length: Break down large functions or components into smaller, focused units to improve readability, maintainability, and testability. Source
23. Use Descriptive Variable Names: Choose descriptive variable and function names that convey their purpose and usage, reducing the need for excessive comments. Source
24. Consistent Naming Conventions: Maintain consistent naming conventions across the codebase for better understanding and collaboration. Source
25. Optimize Render Performance: Use memoization techniques like `React.memo`, `useMemo`, and `useCallback` to optimize rendering performance of components by avoiding unnecessary re-renders. Source
26. Separation of Concerns: Ensure that each component or function has a clear and singular responsibility, adhering to the principle of separation of concerns, which improves code modularity and maintainability. Source
27. Use CSS Modules or Styled Components: Implement CSS Modules or a CSS-in-JS solution like Styled Components to encapsulate styling within components, preventing global style conflicts and enhancing maintainability. Source
28. Responsive Design: Ensure your components and UI are responsive across different screen sizes and devices for an enhanced user experience. Source
29. Document Code: Include comments, documentation, and explanations for complex logic, algorithms, or non-obvious decisions to aid in understanding and maintaining the codebase. Source
30. Use PropTypes or TypeScript: Enforce type checking and validation using PropTypes (for JavaScript) or TypeScript (for TypeScript projects) to catch type-related errors early and improve code reliability. Source
31. Avoid Deep Nesting: Refrain from excessive component nesting, which can make code harder to read and maintain. Find a balance between reusability and nesting depth. Source
32. Consolidate Imports: Keep import statements organized by consolidating multiple imports from the same module into a single statement, enhancing code readability. Source
33. Use CSS Flexbox or Grid: Utilize modern CSS layout techniques like Flexbox and CSS Grid for building responsive and flexible layouts, improving UI design. Source
34. Avoid Inline Styles: Separate styling from components by using external stylesheets or styling solutions like CSS Modules for better code maintainability and separation of concerns. Source
35. Unit Testing: Write unit tests using tools like Jest and testing libraries to ensure the correctness of your codebase, prevent regressions, and support continuous integration. Source
36. Code Reviews: Regularly conduct code reviews with team members to ensure adherence to coding standards, best practices, and to catch potential issues early in the development process. Source
37. Keep Dependencies Updated: Regularly update third-party dependencies to benefit from bug fixes, performance improvements, and new features while ensuring security and stability. Source
38. Performance Profiling: Use tools like React DevTools to profile and optimize the performance of your application, ensuring a smooth user experience. Source
39. Error Boundary Components: Implement error boundaries to gracefully handle errors and prevent crashes from propagating through the entire application, improving overall stability. Source
40. Version Control: Use version control systems like Git effectively to track changes, collaborate with team members, and maintain a history of your codebase, ensuring code integrity. Source

By following these best practices, you can significantly improve the quality of your JavaScript, React, and Node.js code. It will make your codebase more maintainable, scalable, and easier to collaborate on with other developers. Happy coding!

By Jatin Jain Saraf on September 1, 2023.

There are many excellent naming conventions regarding git branches and commits. But what if you want something very lean and simple?

Branching Guidelines and Conventions

There are many excellent naming conventions regarding git branches and commits. But what if you want something very lean and simple?

Here is a proposition.

Categories of branches

Every code space should have a main and a development branch.
Apart from them, you can create any number of branches in your code base.

• main branch: Changes merged into the main branch will be reflected in a production environment.
• develop branch: Changes merged into the develop branch will be reflected in a test environment. All new feature branches of current sprint will be created from the develop branch. A git branch should start with a category. Pick one of these: feature, bugfix, hotfix, test, nojira.
• feature is for adding, refactoring, or removing a feature that is mentioned on any ticket or story created on Jira/ Youtrack. The feature branch should contain code related to only a single feature or ticket, which will be eventually merged into the develop branch. It should be created like feature/.
• bugfix is for fixing a bug mentioned on Jira/ Youtrack. These are the fixes to deploy on the test and stage environment. It should be created like bugfix/.
• hotfix is for changing code with a temporary solution and/or without following the usual process (usually because of an emergency). It is also used to deploy hotfix on production. It should be created like hotfix/ or hotfix/.
• test is for experimenting outside of an issue/ticket or having a spike ticket on Jira/ Youtrack. It should be created like test/.
• release is for the stage environment Changes merged into the release branch will be reflected in the stage environment. It should be created like release/
• Nojira is for the work that is not listed on Jira/Youtrack or work that has no reference ticket. It should be created like nojira/

Git Process

During Sprint

1. Create a feature branch with ticket no. from the develop branch.
2. Work on that feature branch and at the end of the day, push your changes into that branch with the proper commit message below format: 
   feature/ | <user id / username> | proper commit description
   feature | SU-12 | Updated Error occurring in getting user API
3. Once your work is done, rebase your feature branch with the develop branch and create a Pull request against the develop branch, The PR title should be as below:
   feature/ | SU-12|  
   feature/ | SU-12| Generate random order number in admin
4. If the reviewer has posted improvements/fixes in PR, then push commits with the same message format with the proper description
5. Once PR is ready to merge the reviewer should squash and merge the branch into develop and delete the feature branch
6. If QA reports bugs in a test environment, create a new branch from the develop branch with format (bugfix/) and generate PR against the develop branch. Reviewer should squash and merge the branch into develop and delete the fix branch

Code freeze (stage release/regression phase)

1. Create a new release branch from the main branch in this format (release_/<_tag> release/1.0.0, release/1.0.1) If all changes from the develop branch are required for the next release, rebase the develop branch on the newly created release branch by running the below command:
    Git checkout release/1.0.0
   Git rebase develop
   Git push origin release/1.0.0 (this will trigger stage deployment)
2. If only a few features(partial changes) are required for release, then cherry-pick commits from the develop branch into the release branch and push new commits.
3. Any changes pushed to the release branch will trigger stage deployment.
4. If any issues/bugs are found during code freeze, the fix needs to deploy on the stage environment first and then on a test environment.
5. For new bugs, create a new release branch from the last release branch, for example, release/1.0.1 from release/1.0.0, then create a new fix branch out of the new release branch with the below format: bugfix/
6. Generate fix branch PR against new release branch(release/1.0.1). The reviewer should squash and merge the branch into the release branch and delete the fix branch
7. Once QA gives confirmation that the bug is fixed on the stage environment, the Developer should then rebase the release branch on the develop branch, so that the test environment will also have a new fix

Production release:

1. If all features/changes from the stage(release branch) environment are required to be released, then rebase the last release branch(e.g. release/1.0.6) on the main branch.
    Git checkout main
   Git rebase release/1.0.6
   Git push origin main
2. Once the changes are pushed, manually trigger production deployment from GitHub actions by selecting the main branch

Hotfix

1. Create a new release branch(release/1.2.0) from the master and create a hotfix branch out of the new release branch with the below format: hotfix/
2. Work on hotfix and generate PR of hotfix branch against new release branch.
3. Reviewer should squash and merge the branch into the release branch and delete the hotfix branch
4. Once QA gives the confirmation of the hotfix on the stage environment, rebase the release branch on the main branch and trigger manual deployment from GitHub actions in order to deploy the hotfix on production.
5. Developer should then rebase the release branch on the develop branch so that the hotfix will be deployed on the test environment as well.

By Jatin Jain Saraf on August 24, 2023.

Introduction to Sequelize

Sequelize using Postgres dialect

Introduction to Sequelize

Sequelize is a popular Object-Relational Mapping (ORM) library for Node.js, which allows you to interact with relational databases like PostgreSQL using JavaScript/Node.js. It provides an easy way to manage your database operations and simplifies the interaction with the database, abstracting SQL queries into JavaScript code. Some key features of Sequelize are listed below:-

Database Support: Sequelize supports various relational databases, including PostgreSQL, MySQL, SQLite, and MSSQL. This flexibility allows you to switch between different database systems easily.

Models and Associations: Sequelize enables you to define models that represent database tables and their relationships (associations) with other tables. These models provide a structured way to interact with the database and handle complex data relationships.

Data Validation: Sequelize offers built-in validation support for your models, allowing you to define validation rules for each attribute to ensure data integrity and consistency.

Migrations: With Sequelize, you can use migrations to manage changes to the database schema over time. Migrations help keep your database schema in sync with your application’s codebase and facilitate database version control.

Querying: Sequelize provides a straightforward API for querying the database, including support for complex queries like filtering, ordering, aggregating, and pagination.

Hooks: Sequelize allows you to define hooks that get executed before or after certain events, such as before saving a record or after fetching data. This feature is helpful for performing additional actions or validation during these events.

Transactions: Sequelize supports transactions, which are crucial for maintaining data consistency and integrity when dealing with multiple related operations that must either all succeed or all fail.

Raw SQL Queries: While Sequelize provides a high-level API for interacting with the database, it also allows you to execute raw SQL queries directly when more complex operations are required.

Eager Loading: Sequelize supports eager loading, enabling you to fetch related data along with the main query, which can help avoid the N+1 problem and improve query performance.

Pluggable Dialects: The library is built to support multiple SQL dialects, making it adaptable to various relational database systems.

Introduction to PostgreSQL

Postgres, also known as PostgreSQL, is a powerful open-source relational database management system (RDBMS). It was initially developed at the University of California, Berkeley in the 1980s and has since grown into a widely used and respected database system. It is a versatile and reliable database system that is widely used by developers and organizations worldwide for various applications ranging from simple web applications to complex enterprise solutions. It has a vibrant community and is continually being improved with each release. Some key features of Sequelize are listed below:-

Relational Database Management System: PostgreSQL is based on the relational model, allowing you to define tables with rows and columns to store data.

Open Source: It is released under the PostgreSQL License, which is a permissive open-source license.

Cross-Platform: PostgreSQL runs on various platforms, including Windows, macOS, Linux, and other UNIX-like operating systems.

Extensible: PostgreSQL supports user-defined data types, functions, and procedures, which allows you to create custom extensions to meet specific needs.

Highly Scalable: It can handle large amounts of data and concurrent users, making it suitable for both small projects and large-scale enterprise applications.

Advanced Data Types: PostgreSQL supports a wide range of data types, including numeric types, string types, date/time types, JSON, arrays, and more.

ACID Compliant: PostgreSQL follows the principles of ACID (Atomicity, Consistency, Isolation, Durability) to ensure data integrity and reliability.

Full Text Search: PostgreSQL provides advanced full-text search capabilities, making it suitable for applications that require efficient text search functionality.

Foreign Data Wrappers (FDW): PostgreSQL supports FDWs, which allow you to access external data sources like other databases or APIs seamlessly.

Highly Customizable: PostgreSQL allows fine-grained configuration settings, enabling you to optimize its behavior for specific workloads.

Triggers and Stored Procedures: PostgreSQL supports triggers and stored procedures, enabling you to define complex business logic directly within the database.

Replication: PostgreSQL offers various replication options to ensure high availability and fault tolerance.

Postgres Installation

To install PostgreSQL on Ubuntu 20.04 (or any other Ubuntu version), follow these steps:

Install PostgreSQL and its required components:

sudo apt install postgresql postgresql-contrib

Start the PostgreSQL service:

sudo systemctl start postgresql

You can now connect to PostgreSQL using the `psql` command-line tool. Switch to the `postgres` user and then run `psql`:

bash
sudo -u postgres psql

Create a New User Role: Replace _newusername_ with the desired username for the new role.

CREATE USER newusername;

Set Password (Optional): You can set a password for the new user role. Replace _newusername_ and _newpassword_ with the actual username and password

ALTER USER <newusername> WITH PASSWORD <newpassword>;

Grant Permissions (Optional): You can grant specific permissions to the new user role. For example, to grant the user role the ability to create databases and roles

ALTER USER newusername CREATEDB;
ALTER USER newusername CREATEROLE;

Sequelize Installation

To install follow these setps

npm install sequelize sequelize-cli pg

_sequelize_: The Sequelize library itself.

_sequelize-cli_: The Sequelize Command Line Interface (CLI) for managing database migrations, models, etc.

_pg_: The PostgreSQL database driver.

After installing Sequelize, you’ll need to set up the Sequelize configuration and models for your project. Run the following command to initialize Sequelize in your project:

npx sequelize-cli init

This will create the necessary directories and files for Sequelize in your project.

Configure Database: Open the _config/config.json_ file that was generated by Sequelize initialization and configure your database connection settings. You'll need to provide the database name, username, password, host, and dialect (e.g., "postgres" for PostgreSQL).

Create Models: Models represent your database tables and are defined using Sequelize. Create a new model using the Sequelize CLI

npx sequelize-cli model:generate --name User --attributes firstName:string,lastName:string,email:string

Migrations and Database Setup: Sequelize uses migrations to manage changes to the database schema. To create the initial migration and apply it to the database, run

npx sequelize-cli db:migrate

This will create the necessary tables in your database based on your model definitions.

Establish a connection to Postgres using sequelize

Configure Sequelize Connection:

const { Sequelize, DataTypes } = require('sequelize');

// Configure database connection
const sequelize = new Sequelize('database', 'username', 'password', {
host: 'localhost',
dialect: 'postgres',
});

// Define User model
const User = sequelize.define('User', {
username: {
type: DataTypes.STRING,
allowNull: false,
},
email: {
type: DataTypes.STRING,
allowNull: false,
},
});

// Define Post model
const Post = sequelize.define('Post', {
title: {
type: DataTypes.STRING,
allowNull: false,
},
content: {
type: DataTypes.TEXT,
allowNull: false,
},
});

// Set up associations
User.hasMany(Post);
Post.belongsTo(User);

// Sync models with the database
sequelize.sync({ force: true }) // WARNING: This will drop existing tables and recreate them
.then(() => {
console.log('Models synchronized with database.');
})
.catch((error) => {
console.error('Error synchronizing models:', error);
});

Model Usages

In Sequelize, models serve as JavaScript representations of database tables. They define the structure of the data that will be stored in the database and provide an interface for querying, creating, updating, and deleting records. Let’s explore how to use models in Sequelize:

Defining a Model: To define a model, you use the _sequelize.define()_ method. Here's an example of defining a _User_ model:

const { Sequelize, DataTypes } = require('sequelize');

const sequelize = new Sequelize('database', 'username', 'password', {
host: 'localhost',
dialect: 'postgres',
});

const User = sequelize.define('User', {
username: {
type: DataTypes.STRING,
allowNull: false,
},
email: {
type: DataTypes.STRING,
allowNull: false,
},
});

Creating Records: You can create records using the _create()_ method on the model:

User.create({
username: 'john_doe',
email: 'john@example.com',
})
.then((user) => {
console.log('New user created:', user.toJSON());
})
.catch((error) => {
console.error('Error creating user:', error);
});

Querying Records: You can query records using methods like _findOne()_, _findAll()_, and custom query methods. For example:

User.findOne({
where: { username: 'john_doe' }
})
.then((user) => {
if (user) {
console.log('Found user:', user.toJSON());
} else {
console.log('User not found.');
}
})
.catch((error) => {
console.error('Error querying user:', error);
});

Updating Records: You can update records using the _update()_ method:

User.update({ username: 'new_username' }, {
where: { id: 1 }
})
.then((result) => {
console.log('Updated user:', result);
})
.catch((error) => {
console.error('Error updating user:', error);
});

Deleting Records: You can delete records using the _destroy()_ method:

User.destroy({
where: { id: 1 }
})
.then(() => {
console.log('User deleted.');
})
.catch((error) => {
console.error('Error deleting user:', error);
});

DataTypes

In the above snippet, we have used DataTypes & Sequelize, Sequelize, and DataTypes are key components of the Sequelize library, which is an Object-Relational Mapping (ORM) tool for Node.js. Sequelize simplifies database interactions by allowing you to work with databases using JavaScript objects and methods, rather than writing raw SQL queries. Here’s a brief introduction to both Sequelize and DataTypes:

1.) Sequelize is a popular ORM library that provides an abstraction layer for working with relational databases, making it easier to manage and manipulate database records through JavaScript code. It supports various relational database systems, including PostgreSQL, MySQL, SQLite, and MSSQL.Key features of Sequelize include: 
Model Definition: Sequelize allows you to define models, which are JavaScript classes that represent tables in your database. Models define the structure of the table, including columns, data types, and associations with other tables.Migrations: Sequelize provides a way to manage database schema changes over time using migrations. Migrations are scripts that define changes to the database schema, such as adding or modifying tables and columns.Querying: Sequelize offers a query interface for creating, reading, updating, and deleting records from the database. It supports both simple and complex queries.Associations: Sequelize allows you to define relationships (associations) between different models, such as one-to-one, one-to-many, and many-to-many relationships.Hooks: Sequelize provides hooks (similar to event listeners) that allow you to perform actions before or after certain database operations, such as creating or updating records.Validation: You can define validation rules for model attributes to ensure that data meets specific requirements before being stored in the database.

2.) DataTypes is an object provided by Sequelize that represents the different data types you can use when defining model attributes. It maps JavaScript data types to the corresponding data types in your database.

Common DataTypes include:

_STRING_: Represents variable-length character strings.

_INTEGER_: Represents whole numbers.

_BOOLEAN_: Represents true or false values.

_DATE_: Represents date and time values.

_FLOAT_, _DOUBLE_: Represent floating-point numbers.

_TEXT_: Represents large blocks of text.

_ENUM_: Represents a fixed set of values.

When defining a model, you use DataTypes to specify the type of each attribute in the model. Here’s a basic example of using Sequelize and DataTypes to define a _User_ model with a _username_ and an _email_ attribute:

const { Sequelize, DataTypes } = require('sequelize');

const sequelize = new Sequelize('database', 'username', 'password', {
host: 'localhost',
dialect: 'postgres',
});

const User = sequelize.define('User', {
username: {
type: DataTypes.STRING,
allowNull: false,
},
email: {
type: DataTypes.STRING,
allowNull: false,
},
});

In this example, we’re defining a _User_ model with two attributes: _username_ of type _STRING_ and _email_ of type _STRING_, both of which are required (_allowNull: false_).

Sequelize’s DataTypes and model definition make it easy to work with databases using JavaScript, abstracting away the complexities of raw SQL queries.

Using Associations

In the context of the above example we have used the models in the following snippet

(async () => {
try {
// Create a user with associated posts
const user = await User.create({
username: 'john_doe',
email: 'john@example.com',
Posts: [
{
title: 'First Post',
content: 'This is the content of the first post.',
},
{
title: 'Second Post',
content: 'This is the content of the second post.',
},
],
}, {
include: Post,
});

// Find a user and retrieve their posts  
const foundUser = await User.findOne({  
  where: { username: 'john\_doe' },  
  include: Post,  
});  

console.log('User with associated posts:', foundUser.toJSON());  

// Find a post and retrieve its user  
const foundPost = await Post.findOne({  
  where: { title: 'First Post' },  
  include: User,  
});  

console.log('Post with associated user:', foundPost.toJSON());  

} catch (error) {
console.error('Error:', error);
} finally {
await sequelize.close();
}
})();

Raw Queries

Raw queries refer to the use of direct SQL statements in your code to interact with a database, bypassing the abstractions provided by Object-Relational Mapping (ORM) libraries like Sequelize. While using an ORM like Sequelize is often preferred for its ease of use, raw queries can be useful in certain situations where complex queries or optimizations are required.Here’s an overview of raw queries, including when and how to use them:

When to Use Raw Queries: _You might consider using raw queries in the following scenarios:
_Performance Optimization: In cases where a specific query optimization is needed, raw queries can be used to fine-tune database performance.Advanced Queries: For complex queries that are difficult to express using an ORM’s query builder methods.Stored Procedures: When you need to call database-stored procedures directly.Database-Specific Features: For utilizing database-specific features that are not supported by the ORM.

Using Raw Queries with Sequelize

Sequelize allows you to execute raw SQL queries using the _sequelize.query()_ method. Here's how you can use it:

const { Sequelize } = require('sequelize');

const sequelize = new Sequelize('database', 'username', 'password', {
host: 'localhost',
dialect: 'postgres',
});

async function runRawQuery() {
try {
const results = await sequelize.query('SELECT * FROM users WHERE age > :age', {
replacements: { age: 25 }, // Bind parameters
type: Sequelize.QueryTypes.SELECT,
});

console.log('Raw Query Results:', results);  

} catch (error) {
console.error('Error executing raw query:', error);
} finally {
await sequelize.close();
}
}

runRawQuery();

In the example above, we’re using _sequelize.query()_ to execute a raw SQL query that retrieves users older than a specified age. The _replacements_ option is used to bind parameters, enhancing security by preventing SQL injection.

Preventing SQL Injection:_When using raw queries, it’s important to properly sanitize and validate user inputs to prevent SQL injection attacks. Sequelize’s parameter binding, as demonstrated in the example above, helps mitigate this risk.
_Cautions and Considerations: _Raw queries can make your code less portable across different database systems. They can bypass some of the built-in validations and features of the ORM. Use raw queries judiciously and consider whether an ORM’s query builder can fulfill your requirements.
_Escaping Identifiers: If you need to use raw identifiers (table/column names) in your SQL queries, you can use the _sequelize.literal()_ function to safely escape them:

sequelize.query(`SELECT * FROM ${sequelize.literal('Users')} WHERE age > :age`, {
replacements: { age: 25 },
type: Sequelize.QueryTypes.SELECT,
});

Indexing

Indexing is a database optimization technique that improves the speed of data retrieval operations (such as searching, sorting, and filtering) by creating a data structure that allows for more efficient access to the data. In relational databases like PostgreSQL, indexing plays a crucial role in improving query performance.

In the context of Sequelize and PostgreSQL, you can create indexes on specific columns of your tables to speed up queries involving those columns. Here’s how you can work with indexing in Sequelize:

Creating Indexes: In Sequelize, you can define indexes when defining your model using the _indexes_ option. Indexes can be created on single or multiple columns. Here's an example of creating an index on the _email_ column of the _User_ model:

const { Sequelize, DataTypes } = require('sequelize');

const sequelize = new Sequelize('database', 'username', 'password', {
host: 'localhost',
dialect: 'postgres',
});

const User = sequelize.define('User', {
username: {
type: DataTypes.STRING,
allowNull: false,
},
email: {
type: DataTypes.STRING,
allowNull: false,
},
}, {
indexes: [
{
unique: true,
fields: ['email'], // Index on the 'email' column
}
]
});

In this example, we’re creating a unique index on the _email_ column. Unique indexes ensure that no two rows have the same value in the indexed column.

Using Existing Indexes: If you have existing indexes in your database, Sequelize can use them for query optimization. Sequelize will automatically use the indexes when executing queries that involve indexed columns.

Custom Index Names: By default, Sequelize generates index names based on the table name and indexed columns. You can specify custom index names using the _name_ property in the _indexes_ array:

indexes: [
{
unique: true,
fields: ['email'],
name: 'custom_email_index_name'
}
]

Composite Indexes: You can create composite indexes by specifying multiple columns in the _fields_ array:

indexes: [
{
fields: ['column1', 'column2']
}
]

Removing Indexes: To remove an index, you can use the _sequelize.drop_ method with the _options_ parameter:

User.drop({
indexes: [
'custom_email_index_name', // Name of the index to drop
]
});

_Keep in mind that dropping indexes will result in a change to your database schema, so use caution.
_Indexes can significantly improve query performance, especially on large datasets. However, overindexing (creating too many unnecessary indexes) can lead to performance issues during data updates and inserts. Therefore, it’s essential to carefully consider which columns should be indexed based on the types of queries your application performs most frequently.Always monitor query performance and database usage when using indexes, and be prepared to adjust your indexing strategy as needed.

Widely used Sequelize Function

_sequelize.define_: Defines a new model with attributes and options.
sequelize.sync(): Synchronize model definitions with the database.
_belongsTo_: Creates an association between two models, where one model belongs to another.
_hasMany_: Defines a one-to-many relationship between models.
_belongsToMany_: Creates a many-to-many association between models.
_Model.create_: Creates a new instance of a model and inserts it into the database.
_Model.findAll_: Retrieves all instances of a model from the database.
_Model.findByPk_: Finds a single instance by its primary key.
_Model.update_: Updates instances that match the given conditions.
_Model.destroy_: Deletes instances that match the given conditions.
_Model.findOne_: Finds the first instance that matches the given conditions.
_Model.findAll_: Retrieves instances based on specific conditions.
_Model.count_: Counts the number of instances that match the given conditions.
Model.bulkCreate(): Bulk insert multiple records.
_Model.findAll({ order, limit, offset })_: Allows sorting and pagination of query results.
_sequelize.fn_: Provides access to various SQL functions (e.g., _SUM_, _COUNT_, _AVG_, etc.).
_sequelize.col_: Represents a column in the database.
_sequelize.query_: Executes raw SQL queries and returns the results.

There are numerous functions available for individuals to use according to their requirements. If you wish to learn more about how to use sequelize with Postgres, you can refer to the official documentation of Postgres and sequelize for detailed information.

Sequelize: https://sequelize.org/

Postgres: https://www.postgresql.org/

By Jatin Jain Saraf on August 7, 2023.

When multiple CPU cores are available, Node.js does not utilize them all by default. Fortunately, Node.js has a native cluster module that…

Clustering In NodeJs

When multiple CPU cores are available, Node.js does not utilize them all by default. Fortunately, Node.js has a native cluster module that allows for the creation of child processes (workers) that can run simultaneously while sharing the same server port. Each child process has its own event loop, memory, and V8 instance. These child processes communicate with the main parent Node.js process through interprocess communication. Clusters of Node.js processes can be used to distribute workloads among application threads, or if process isolation is not needed, the module can be used to run multiple application threads within a single Node.js instance. The cluster module makes it easy to create child processes that can share server ports.

The need for clustering in Node.js

An instance of Node.js runs on a single thread (you can read more about threads in Node.js here). The official Node.js “About” page states: “Node.js being designed without threads doesn’t mean you can’t take advantage of multiple cores in your environment.” That’s where it points to the cluster module.

The cluster module doc adds: “To take advantage of multi-core systems, the user will sometimes want to launch a cluster of Node.js processes to handle the load.” So, to take advantage of the multiple processors on the system running Node.js, we should use the cluster module.

Exploiting the available cores to distribute the load between them gives our Node.js app a performance boost. As most modern systems have multiple cores, we should be using the cluster module in Node.js to get the most performance juice out of these newer machines.

How does the Node.js cluster module work?

In a nutshell, the Node.js cluster module acts as a load balancer. It distributes a load to the child processes running simultaneously on a shared port. Node.js is not great with blocking code, so if there is only one processor and it’s blocked by a heavy and CPU-intensive operation, other requests are just waiting in the queue for this operation to complete.

With multiple processes, if one process is busy with a relatively CPU-intensive operation, other processes can take up the other requests coming in and utilize the other CPUs/cores available. This is the power of the cluster module — workers share the load and the app does not stop.

The master process can distribute the load to the child process in two ways. The first (and default) is a round-robin fashion. The second way is the master process listens to a socket and sends the work to interested workers. The workers then process the incoming requests.

However, the second method is not super clear and easy to comprehend like the basic round-robin approach.

Prerequisites

To follow this guide about Node.js clustering, you should have the following:

• Node.js running on your machine, the latest LTS is advisable. It is Node.js 18 at the time of writing.
• Working knowledge of Node.js and Express
• Basic knowledge of how processes and threads work
• Working knowledge of Git and GitHub

Implementation & Configuration

import cluster from 'node:cluster';
import http from 'node:http';
import { availableParallelism } from 'node:os';
import process from 'node:process';

const numCPUs = availableParallelism();

if (cluster.isPrimary) {
console.log(`Primary ${process.pid} is running`);

// Fork workers.
for (let i = 0; i < numCPUs; i++) {
cluster.fork();
}

cluster.on('exit', (worker, code, signal) => {
console.log(`worker ${worker.process.pid} died`);
});
} else {
// Workers can share any TCP connection
// In this case it is an HTTP server
http.createServer((req, res) => {
res.writeHead(200);
res.end('hello world\n');
}).listen(8000);

console.log(`Worker ${process.pid} started`);
}

Running Node.js will now share port 8000 between the workers:

$ node server.js
Primary 3596 is running
Worker 4324 started
Worker 4520 started
Worker 6056 started
Worker 5644 started

Before implementing clustering in your Node.js project, it's crucial to comprehend its advantages and constraints to determine if it's the most suitable option for your application's requirements.

Benefits of Clustering:

1. By using clustering, you can make the most of the multiple CPU cores on your machine. This can greatly enhance the speed and efficiency of your application, particularly for tasks that rely heavily on the CPU.
2. By clustering, your application can effectively handle a greater number of simultaneous requests through the distribution of workload among several worker processes. This results in improved scalability and performance.
3. In terms of fault tolerance and availability, the ability of the remaining worker processes to handle incoming requests even if one crashes is a significant advantage.

Limitations of Clustering:

1. Adding clustering to your application code can increase its complexity. You must be careful in managing inter-process communication, handling shared resources, and avoiding potential race conditions.
2. The usage of memory increases with each worker process, which could be problematic if the server has limited memory resources.
3. Managing states can become complicated with clustering, as each worker process has its own memory space. If your application heavily depends on in-memory states, you will have to develop methods to synchronize and share states among worker processes.
4. Clustering may not always be necessary for applications that are I/O-bound, such as network requests or database queries. This is because the bottleneck often lies in the I/O operations rather than CPU processing, making clustering less beneficial in these cases.

When to Use Clustering:

If your application is CPU-bound and receives high requests, clustering can be a valuable solution. It is particularly useful for tasks that involve complex computations, image processing, video encoding, and specific types of data analysis. By distributing the workload, clustering can enhance overall efficiency and help overcome any limitations to performance caused by CPU utilization.

By Jatin Jain Saraf on August 7, 2023.