Unlocking the Black Box: How Postgres Logical Replication Consumes Disk Space and How to Reclaim Control

By Tech & Database Reporting Desk
Published: September 2024

For database administrators managing enterprise-grade PostgreSQL deployments, the sudden illumination of a disk-full alert is a moment of immediate, cold dread. When those monitors flag the cryptic pg_replslot directory—watching it swell with gigabytes of anonymous artifacts seemingly at random—system engineers are forced into high-stakes forensics. What is writing to this directory? Why is it ballooning? And crucially, how can database teams stop it before production workloads crash?

The answers to these questions do not lie within obscure operating system logs or broken hardware. Instead, they are hidden in plain sight within the foundational syntax of PostgreSQL logical replication: an innocuous, easily overlooked configuration line embedded inside the CREATE SUBSCRIPTION command.

Main Facts: The Hidden Costs of Logical Decoding

Logical replication has been a core feature of PostgreSQL since version 10, praised for its flexibility in moving data between disparate database schemas and external pipelines. The syntax governing subscriptions is deceptively simple. Executing CREATE SUBSCRIPTION requires little more than a unique name, a target connection string, a list of publications, and a single, innocuous trailing line:

[ WITH ( subscription_parameter [= value ] [, ... ] ) ]

To the casual user, this line is an afterthought. Subscriptions created without any explicit WITH clause operate perfectly out of the box. Most database instances in the wild run with default settings, and administrators rarely think about—or are even aware of—the dozens of advanced options hidden behind that single line of syntax.

However, beneath this simplicity lies a complex machinery of transaction decoding, memory buffers, and disk spill management. When production systems host multiple downstream logical replicas, every active subscription spawns an independent decoding process on the publisher. If transactions exceed internal memory limits, PostgreSQL redirects the overflow directly to disk inside the pg_replslot directory, multiplying storage consumption across every single active subscriber.

Chronology: The Evolution of a Production Mystery

To understand how a database cluster moves from healthy operations to a disk-exhaustion crisis, one must trace the lifecycle of a logical transaction from the primary node to the downstream replica.

1. The Anatomy of the pg_replslot Directory

A preliminary investigation of a troubled PostgreSQL data directory usually points toward /var/lib/postgresql/data/pg_replslot/. Inside this directory, the database maintains one subdirectory for every active replication slot.

A standard directory check using standard shell commands reveals the culprit:

$> ls -l /var/lib/postgresql/data/pg_replslot/sub_a/

total 116608
-rw------- 1 postgres postgres      200 Sep  3 14:01 state
-rw------- 1 postgres postgres 14179546 Sep  3 14:01 xid-741-lsn-0-1000000.spill
-rw------- 1 postgres postgres 20805450 Sep  3 14:01 xid-741-lsn-0-2000000.spill
-rw------- 1 postgres postgres 20805450 Sep  3 14:01 xid-741-lsn-0-3000000.spill

Alongside a small state file tracking the replication position, administrators find an array of .spill files—totaling hundreds of megabytes or even gigabytes. These files carry transaction IDs and Log Sequence Number (LSN) boundaries. Yet, querying standard administrative views like pg_replication_slots often shows all slots operating as active and reserved, with no immediate indicators of lag.

2. Waiting for the Commit Record

The root cause of these spill files rests in PostgreSQL’s architectural requirement for transactional integrity. Logical decoding executes entirely on the publisher. Each active subscription slot assigns a dedicated walsender process to read the Write-Ahead Log (WAL) and load an output plugin.

Because subscribers must receive changes in strict commit order, and because WAL records from concurrent transactions naturally interleave, the walsender cannot simply stream changes as it reads them. A transaction might roll back long after another commits. Therefore, the walsender loads in-flight changes into an internal memory structure called the reorderbuffer.

The memory budget for this buffer is dictated by the configuration parameter logical_decoding_work_mem, which defaults to a modest 64MB. When a transaction—such as a massive bulk insert or an uncommitted batch job—exceeds this memory threshold, PostgreSQL dumps the excess data straight to disk as a spill file inside the slot’s directory.

3. The Multiplicative Effect: Death by a Thousand Decoders

Compounding this behavior is the architecture of PostgreSQL subscriptions. Each subscription requires its own dedicated replication slot, its own walsender, and its own reorderbuffer. Nothing is shared.

If a production database maintains three distinct logical subscribers, a single 300,000-row transaction is decoded, buffered, and—if it exceeds memory limits—written to disk three separate times.

SELECT count(*) AS slots,
       pg_size_pretty(sum(spill_bytes)) AS total_spilled,
       pg_size_pretty(max(spill_bytes)) AS per_slot
  FROM pg_stat_replication_slots;

 slots | total_spilled | per_slot 
-------+---------------+----------
     3 | 335 MB        | 112 MB

Once the transaction finally commits, PostgreSQL purges the spill files instantly. An administrator checking the server post-incident will find an empty directory, leaving behind only elusive cumulative counters and a frightened operations team.

Supporting Data: Empirical Evidence and Metrics

To verify how PostgreSQL handles these memory thresholds under pressure, database engineers can simulate high-load environments by intentionally constraining resources. Lowering logical_decoding_work_mem on a test publisher and running a large transactional insert immediately forces the database to trigger disk spilling.

Monitoring the pg_stat_replication_slots dynamic view exposes the exact mechanics of this process:

SELECT slot_name, spill_txns, spill_count,
       pg_size_pretty(spill_bytes) AS spill_bytes,
       stream_txns, stream_bytes
  FROM pg_stat_replication_slots
 ORDER BY slot_name;

When subscriptions operate under default legacy settings (streaming = off), the metrics show high spill_count and spill_bytes, while stream_bytes remains locked at zero. The publisher shoulder-checks the entire burden of holding uncommitted transaction states.

The Solution: Streaming Large Transactions

To alleviate publisher disk pressure, PostgreSQL provides the streaming subscription parameter. When configured to on or parallel (available in PostgreSQL 16 and higher), the publisher no longer waits for the entire transaction to complete before taking action. Instead, it streams transaction contents as they are decoded, forwarding the storage burden directly to the subscriber.

-- Upgrading a subscription to leverage parallel streaming
ALTER SUBSCRIPTION sub_b SET (streaming = parallel);

When parallel streaming is engaged, temporary files migrate from the publisher’s pg_replslot directory to the subscriber’s temporary file storage (pgsql_tmp), effectively isolating the primary database from catastrophic disk exhaustion.

Even with streaming enabled, however, edge cases remain. Workloads containing heavy TOAST data (such as large out-of-line text payloads) can still force PostgreSQL to generate spill files locally on the publisher, proving that monitoring must remain vigilant even after configuration adjustments.

Official Responses and Documentation Gaps

The database community and core PostgreSQL contributors have long debated the balance between default out-of-the-box convenience and enterprise-grade resilience. Historically, PostgreSQL favored safety and simplicity, setting streaming = off by default up through version 17.

However, acknowledging the severe operational friction caused by unmanaged disk spilling, PostgreSQL 18 fundamentally shifted the paradigm by changing the default streaming setting from off to parallel.

Despite this major core improvement, database reliability engineers have pointed out persistent gaps in official documentation:

• The Monitoring Blind Spot: The official logical replication monitoring chapters historically emphasized WAL sender lag while completely omitting references to pg_stat_replication_slots, pg_replslot, or the dangers of unmanaged logical_decoding_work_mem allocations.
• Configuration Discovery: Critical resource settings like logical_decoding_work_mem are filed under general resource consumption chapters rather than logical replication guides, leaving administrators to discover these parameters through painful trial and error.

Implications for Enterprise Architecture

For organizations running PostgreSQL at scale, these architectural realities carry profound implications for capacity planning, infrastructure monitoring, and version upgrade cycles.

1. Immediate Audit Requirements: Database administrators managing PostgreSQL versions 10 through 17 must proactively audit their subscription catalogs. Running a simple diagnostic query reveals immediate exposure:

   SELECT subname, subenabled,
         CASE substream WHEN 'f' THEN 'off'
                        WHEN 't' THEN 'on'
                        WHEN 'p' THEN 'parallel' END AS streaming
    FROM pg_subscription ORDER BY subname;

2. Upgrading as a Mitigation Strategy: Organizations lagging on legacy versions (such as PostgreSQL 13 and older, which lack streaming replication options entirely) face compounding security and operational risks. Upgrading to PostgreSQL 18 or newer provides native relief by automating parallel streaming out of the box.
3. Redefining Monitoring Metrics: Infrastructure teams must expand their observability pipelines. Monitoring raw disk usage on primary database nodes is no longer sufficient; operations centers must actively track replication slot statistics, spill byte counters, and per-slot memory allocations to catch runaway decoding processes before they threaten cluster stability.

Logical replication remains one of PostgreSQL’s most powerful tools for distributed data architectures. Yet, as with any complex machinery, treating it as a black box invites operational failure. By shining a light into the hidden corners of pg_replslot and mastering the esoteric options of the WITH clause, database professionals can finally tame the replication stream and ensure long-term production resilience.