A slow walk through the inside of a database — pages, trees, logs, locks — rebuilt from sketches and small animations instead of walls of code.
The opening post of the series. We draw the map — the flat data model, the stream (a hash of the fields you choose, not a property of the log line), and the two LSM trees per day — then watch it all land on disk, file by file.
The opening post of the series. We draw the map — the flat data model, the stream (a hash of the fields you choose, not a property of the log line), and the two LSM trees per day — then watch it all land on disk, file by file.
Post 1 left us holding a buffer of typed rows in arrival order. The naive move is to flush it to a file as-is — and it fails three ways. Those three failures are exactly the three moves that build a part, sort by (streamID, timestamp), cut the rows into blocks, and freeze the result into an immutable directory you can skip without opening.
Writing one immutable part per flush would bury the disk in tiny files. We follow the in-memory buffer and the background merges that turn a storm of small parts into a few large ones — the log-structured merge tree at the heart of datadb.
Open a block byte by byte. Fields are stored per column, not per row — each column typed, range-indexed with min/max, and compressed with dictionary and const encodings. This is where most of the "15x less disk" actually comes from.
A full-text query that reads every block is a non-starter at terabyte scale. We read the tokenizer and the per-block bloom filters that let a query skip the blocks it cannot possibly match — search as a process of elimination, not a scan.
The centerpiece. The real history where 20-field logs caused roughly 1000x read amplification, and the format redesign that fixed it — FormatVersion 3, columns_header_index, and the per-column indirection that makes wide events cheap to read.
The second LSM tree, indexdb. How a query like {app="api"} is resolved to a set of stream ids through postings and set algebra — the mergeset that lets the engine find streams without ever touching the log data.
The finale ties every layer into one read pipeline. Per-day partitions, how retention drops whole days instead of deleting rows, and the full path a query takes — from time filter to stream index to the exact blocks it reads.