Building a Table Access Method, Part 3: shared memory in Postgres

In Part 2, the TAM callbacks needed for CREATE TABLE ... USING memtable were wired up, but there is not yet any storage, so there is no point in making INSERT, DELETE, UPDATE, and SELECT work. pg_memtable stores table data in shared memory, so before writing a single line of pg_memtable-specific code, it’s worth covering how Postgres actually lets backends share memory.

This post will cover what different kinds of shared memory Postgres has implementations for and what advantages and disadvantages there are for each. It will then focus on the DSM system, which is what will be used when implementing pg_memtable and go through how to create shared memory segments, ensure that they stay in memory, and how to release them back to the system.

Three ways to share memory in Postgres

Postgres gives you three different ways to put data in shared memory, and they solve different problems.

Traditional shared memory is the simplest one: a single, fixed-size segment allocated once when the postmaster starts, requested via RequestAddinShmemSpace during _PG_init. Every backend inherits it automatically at fork time, so there is no attach step and no per-backend setup cost. The tradeoff is that the size has to be known before the first backend ever starts. It cannot grow, it cannot be freed, and it sits reserved for the server’s entire lifetime whether anything is using it or not. This is very useful if you want to have a small piece of shared memory to communicate between backends, but not very useful as a generic shared memory allocation mechanism.

Dynamic Shared Memory (DSM) segments are the opposite: created and destroyed at runtime with dsm_create/dsm_destroy, sized however large a particular use case needs, and mapped into a backend’s address space only when that backend explicitly calls dsm_attach. Nothing needs to be reserved up front, and any number of independent segments can exist at once, each with its own lifetime.

Dynamic Shared memory Areas (DSA) sit on top of DSM. A dsa_area is a general-purpose allocator, dsa_allocate/dsa_free, that works inside one or more DSM segments and grows by attaching more of them as needed. It is the shared-memory equivalent of palloc/pfree: convenient for lots of small, independently-sized, independently-freed objects. Postgres itself uses it internally for parallel hash join, dshash tables, and other structures that need genuinely dynamic allocation. That convenience has a cost: every access goes through the allocator’s own bookkeeping and an extra layer of indirection, since a dsa_pointer has to be resolved to a real address on every dereference.

The implementation of pg_memtable tried the DSA route first. In an earlier version tuples were stored in a DSA-backed red-black tree, and it was 4-5x too slow on sequential scans, entirely down to that indirection. Every tree traversal step meant resolving another dsa_pointer. The fix was to drop DSA entirely and go straight to a raw DSM segment with direct byte offsets into it, with no allocator indirection at all.

Creating and sharing a DSM segment

There are four functions of interest here when creating and sharing DSM segments:

extern dsm_segment *dsm_create(Size size, int flags);
extern dsm_segment *dsm_attach(dsm_handle h);
extern void dsm_detach(dsm_segment *seg);
extern dsm_handle dsm_segment_handle(dsm_segment *seg);

Creating a DSM segment is straightforward: call dsm_create with the size you want, and it hands back a dsm_segment*. The segment is a pointer to the segment information and can be used to, for example, get the base address of the segment. It is a backend-local pointer and cannot be passed to other backends, since the address is only meaningful inside this one’s address space. Instead, you can pass a segment handle to other backends and they can use dsm_attach(handle) to get a segment pointer for that backend.

A dsm_handle is just a uint32, so it is usually easy to pass to other backends. The usual method to deal with this is to reserve a small, fixed-size chunk of traditional shared memory at startup, and stash the handle there. Every backend already has that reserved memory mapped in at fork time, so finding the handle is straightforward.

For pg_memtable this is not necessary, and pg_memtable uses another trick to share the handle between the backends. This is the subject of the next post.

The last call is dsm_detach, which detaches the backend from the segment and allows Postgres to release it if no other backend is using it.

Pinning and unpinning segments

A freshly created DSM segment doesn’t survive on its own by default. Like most Postgres resources, it’s tied to the resource owner that created it and Postgres tears it down automatically when that resource owner goes away.

To not do this and instead keep the segment around for an indefinite time, it is possible to pin it, either its lifetime with dsm_pin_segment or this backend’s mapping of it with dsm_pin_mapping, and the two solve different problems:

• dsm_pin_segment detaches the lifetime of the segment from the backend. Without it, the segment disappears the moment the backend that created it exits.
• dsm_pin_mapping is about the mapping of the segment in the current backend. Without it, the mapping created by dsm_attach is torn down at the end of the transaction by resource-owner cleanup. The underlying segment is still alive, but the next statement in the backend would have to call dsm_attach again.

Only dsm_pin_segment has a way back: dsm_unpin_segment releases that pin and lets Postgres clean up the segment once no backend is using it. A pinned mapping has no equivalent unpin call; it stays pinned in that backend for as long as the backend lives.

Note that a pinned segment has to be explicitly unpinned by something, eventually, or it leaks for the life of the server. This is worth keeping in mind when creating and destroying storage for pg_memtable in the next posts.

_PG_init and the shared memory startup hook

_PG_init was briefly covered in a previous post: it’s the function Postgres calls automatically when a shared library is loaded. For a library listed in shared_preload_libraries this happens very early, before the postmaster has even finished setting up shared memory with CreateSharedMemoryAndSemaphores. Anything that touches real shared state cannot be executed there yet, but Postgres provides a second hook, shmem_startup_hook, that runs after the shared memory subsystem has been set up, for exactly this kind of initialization.

pg_memtable needs to do some shared memory initialization of its own, via a memtable_shmem_startup_hook function installed from _PG_init below.

static shmem_startup_hook_type prev_shmem_startup = NULL;

void _PG_init(void) {
  ...
  prev_shmem_startup = shmem_startup_hook;
  shmem_startup_hook = memtable_shmem_startup_hook;
  ...
}

prev_shmem_startup saves whatever hook, if any, another extension already installed, so memtable_shmem_startup_hook can chain to it instead of silently replacing it.

Shared memory locks in Postgres

In order to prevent shared memory from being modified by more than one backend at the same time, a lock of some sort is needed. Fortunately, Postgres has support for LWLock (meaning “LightWeight Lock”).

It is, however, useful to be able to monitor these locks for debugging and performance purposes, and it is useful to group these locks by the kind of lock it is so that it shows up with meaningful information in pg_stat_activity.

To deal with this, a shared LWLock tranche identifier is created using LWLockNewTrancheId and registered with a name using LWLockRegisterTranche, from the shmem_startup_hook just introduced:

static int MemTableArenaTranche = -1;

static void memtable_shmem_startup_hook(void) {
  if (prev_shmem_startup)
    prev_shmem_startup();

  MemTableArenaTranche = LWLockNewTrancheId();
  LWLockRegisterTranche(MemTableArenaTranche, MEMTABLE_EXTNAME "_arena");
}

The following posts connect every arena lock to this tranche, so it shows up there with the correct name. MemTableArenaTranche itself is nothing more than an int, saved once at startup and read every time an arena header is initialized.

Next

The next post covers how to set up and tear down a shared segment when a pg_memtable table is created and dropped.