A backend keeps its mapping of the arena in its cache. As a result, if another backend drops or truncates the table, the memory of the old arena is not returned until that backend disconnects, so the cache has to be invalidated when the relation changes. This post uses the relcache invalidation messages of Postgres …
Continue reading Building a Table Access Method, Part 8: Cache invalidation
Building a Table Access Method, Part 7: Storing and reading rows
Every memtable table now has an arena with a proper lifecycle, but nothing is stored in it. This post stores rows in the arena and reads them back with a full table scan. Along the way it shows what tuple format to use for efficiency, how any backend finds the arena of a table, how …
Continue reading Building a Table Access Method, Part 7: Storing and reading rows
Building a Table Access Method, Part 6: TRUNCATE
In the previous two posts you could see how to manage storage for create and drop and also to work with transactions. In this post you’ll learn how to handle TRUNCATE, which replaces the arena of a table with a new one. Doing that safely means choosing which arena to release based on how the …
Continue reading Building a Table Access Method, Part 6: TRUNCATE
Building a Table Access Method, Part 5: commit, abort, and savepoints
The arena of a memtable relation does not yet survive a rollback. A rolled-back CREATE TABLE leaks its arena, and a rolled-back DROP TABLE loses an arena the table still needs. In this post you will see how to tie the lifetime of the arena to the outcome of the transaction to avoid both problems.
Building a Table Access Method, Part 4: creating and dropping the arena
Every table in pg_memtable gets its own arena, a Dynamic Shared Memory segment. The tricky part is how other backends find it. This post stores the segment’s handle in pg_class.relfilenode, so any backend can attach with one call, and hooks CREATE TABLE and DROP TABLE to create and release the arena.
Building a Table Access Method, Part 3: shared memory in Postgres
In Part 2, the TAM callbacks needed for CREATE TABLE … USING memtable were wired up, but there is not yet any storage, so there is no point in making INSERT, DELETE, UPDATE, and SELECT work. pg_memtable stores table data in shared memory, so before writing a single line of pg_memtable-specific code, it’s worth covering …
Continue reading Building a Table Access Method, Part 3: shared memory in Postgres
Building a Table Access Method, Part 2: first steps
In Part 1 the difference between table access methods and foreign data wrappers was covered. This post starts implementing one. This post starts by looking at what you actually want to get out of an in-memory table, drawing on research into why memory-resident engines are fast, then starts writing a table access method (TAM) named …
Continue reading Building a Table Access Method, Part 2: first steps
Building a Table Access Method, Part 1
The standard storage engine of Postgres is called the “heap” because of the storage method. The data is stored in “heap files” in no particular order, just added wherever there’s room and extending the file at the end only when there are no space inside the pages of the file. This storage engine is simple …
Continue reading Building a Table Access Method, Part 1
How to Speed Up Postgres Recovery 16x on IOPS-Limited Storage
As part of Multigres development, we have been testing recovery of Postgres server as part of dealing with graceful switchovers as well as failovers of servers in a variety of different scenarios and with different setups, in particular, with volumes that are I/O limited. We noted that recovery, which should have taken a couple of …
Continue reading How to Speed Up Postgres Recovery 16x on IOPS-Limited Storage
Don’t Put Secrets in Your PostgreSQL Environment Variables
Using the official Postgres Docker image to start a new Postgres server is straightforward. The Docker entrypoint script reads POSTGRES_PASSWORD, sets the superuser password via initdb, and then keeps running. The container starts. Your application connects. Everything looks fine. This is very convenient, but is it a good practice from a security perspective? As you …
Continue reading Don’t Put Secrets in Your PostgreSQL Environment Variables
