How to Speed Up Postgres Recovery 16x on IOPS-Limited Storage

As part of Multigres development, we have been testing recovery of Postgres server as part of dealing with graceful switchovers as well as failovers of servers in a variety of different scenarios and with different setups, in particular, with volumes that are I/O limited.

We noted that recovery, which should have taken a couple of seconds, was crawling. Checking closer, we discovered that read syscalls were blocked and hence WAL replay had stopped in its tracks, preventing recovery from proceeding.

A word about network mounted block storage

Cloud block storage, such as AWS EBS, are billed based on IOPS, not on the amount of data transferred. When you configure the volume, you pick a volume type such as gp3 and can also pick the IOPS. Each IOPS represents one read or write operation but is agnostic to the number of bytes read, up to a certain limit.

On AWS EBS, for SSD-backed volumes, that unit is 256 KiB: a read or write of up to 256 KiB counts as one IOPS, and anything larger just gets split into multiple IOPS (a 512 KiB request costs 2 IOPS, and so on).

This means that reading many small pages from an EBS volume is more expensive (counted in IOPS) than reading one large page, assuming they are contiguous. As a result, the maximum bandwidth you can get is based on the device interface performance, but also the IOPS that the volume supports.

For example, on a baseline gp3 volume (3,000 IOPS, 125 MiB/s interface bandwidth):

Read size	Max achievable bandwidth	What limits you
8 KiB	~23 MiB/s	You hit the 3,000 IOPS ceiling
256 KiB	125 MiB/s	You hit the device interface bandwidth

With these figures, that’s roughly a 5x difference in achievable throughput, on the exact same volume, if you could combine several 8 KiB reads into 256 KiB reads.

Postgres io_combine_limit option

Postgres already has the right tool for it: io_combine_limit, added alongside the read-stream infrastructure in Postgres 17. When Postgres reads several logically-adjacent blocks it needs anyway, it can combine them into one larger I/O request instead of issuing one syscall per block. Sequential scans, bitmap heap scans, and vacuum all already benefit from this via the read-stream infrastructure. Unfortunately, the recovery code does not use this option.

Where is recovery code used and how does it work

After a crash, the current on-disk state can be inconsistent (partially applied transactions, partial writes, etc.). Postgres brings the system back into a consistent state by reading that state into memory from disk and replaying the WAL to update it to be consistent.

The recovery procedure for recovering is roughly:

1. StartupProcessMain() is the entry point of the process. It sets up recovery-specific signal handlers/timeouts.
2. StartupXLOG() reads the control file and backup label, determines the checkpoint to start from, and initializes the WAL recovery structures.
3. PerformWalRecovery() locates the first record logically following the checkpoint, then enters the main redo loop, which repeats until end-of-WAL or a recovery target is reached:
   1. Read the next WAL record via XLogReadRecord() which decodes one record from the buffer of the reader. This calls XLogPageRead() whenever it needs more raw bytes than are already buffered.
   2. Apply the WAL record via ApplyWalRecord(), which dispatches to the record’s resource manager redo routine (e.g. heap_redo, btree_redo). That routine reads in the data page to modify via XLogReadBufferExtended(), applies the change, and marks the page dirty.
   3. Loop back to step 1 for the next record.
4. Once there is no more WAL available, or a recovery target is reached, FinishWalRecovery() performs end-of-recovery checks/cleanup.

The details of this are out of scope for this post, but you can find a good explanation in the Reliability and the Write-Ahead Log chapter in the documentation, or in the excellent writeup PostgreSQL Recovery Internals by Imran Zaheer.

Focusing on what is done in this post, recovery needs to access disk in three circumstances:

1. When replaying the WAL and applying the records to buffered pages. This is a sequential read.
2. When reading in data pages into the buffer. This is largely a random-access read.
3. When checkpointing the recovered system. This is largely a random-access write.

Point 1 is what is interesting here, where the focus is on WAL replay. Note that WAL replay plays a role also in streaming replication and in applying recovered archived WAL, not just when recovering from a crash, but we focus on recovery from a crash, which was the most I/O intensive operation in this case.

Recovery is handled by a process called startup in the pg_stat_io view. Its entry point is StartupProcessMain, which eventually calls PerformWalRecovery to run the actual redo loop. (The network-receiving and -sending sides of streaming replication are separate processes, walreceiver and walsender, with their own entries in pg_stat_io.)

The WAL-reading side of the loop pulls bytes off disk through the function XLogPageRead, which is a function that reads exactly one WAL page of 8 KiB per call.

Since this is the single entry point for reading the WAL during recovery, this is a perfect place to add a simple cache. (Other WAL consumers, such as logical decoding, pg_waldump, and pg_rewind, have their own separate read callbacks.)

The patch: a small read-ahead cache honoring io_combine_limit

Given recovery reads WAL almost entirely sequentially, the fix doesn’t need to be clever. The patch adds a small cache:

1. If recovery asks for a page not cached, read up to io_combine_limit worth of contiguous WAL in one syscall.
2. Copy the individual 8 KiB pages from the cache and return to the caller.

The main recovery procedure, which calls XLogPageRead, does not have to change at all. Nothing outside the general code path needs to know pages now arrive pre-fetched in bigger batches. There is no need to make any pre-fetch or similar (something for the future, perhaps). Very simple patch that does not touch anything unnecessary. This approach runs the risk of reading some pages multiple times, but since WAL replay is mostly monotonically increasing, there should be little overlap between the read chunks of data.

What are the improvements?

Using the default setting of io_combine_limit we get these figures before and after the patch.

	reads	bytes read	avg bytes/read
WAL, before	31,283	256,270,336	8,192
WAL, after	1,955	256,131,072	131,013

Approximately the same total bytes read but approximately 16 times fewer read operations. As a result, it is no longer possible to saturate the interface and it will be possible to recover at full performance without stalling I/O.

A word about prefetching

The recovery_prefetch machinery in Postgres looks ahead in the WAL and issues prefetch hints for the data pages a WAL record is about to touch (the redo/apply step of the recovery loop above), so they’re warm in the OS page cache by the time redo needs to apply a change to them. This addresses data-page reads, not the WAL stream itself.

This means that this patch and recovery_prefetch aren’t competing solutions to the same problem; they’re fixes for two different reads that happen to both live inside the same recovery loop.

(For what it’s worth: data-page reads have their own version of this same 8-KiB-at-a-time gap but is a fair bit more involved to fix properly, since those reads aren’t purely sequential the way WAL is, and I’d rather see this simpler, lower-risk WAL-side fix land and get exercised first.)

Concluding remark

I’ve posted the patch and these numbers to pgsql-hackers. It is a small, self-contained change, and given io_combine_limit already exists precisely for this kind of thing, hooking up the WAL reads in recovery felt like a natural, low-risk improvement. Given that this will allow us to not saturate EBS volumes, the patch should offer immediate value to any AWS deployments.