VictoriaLogs storage internals: the data model and the map of the engine

A log store eats a firehose: millions of lines an hour, from thousands of sources. Storing them is the easy part — a file and some disk will do. The hard part is getting any of it back fast: “show me the errors from this one pod in the last fifteen minutes,” answered in milliseconds, over terabytes, without reading the other 99.99% of the data. Over this series we’ll read one engine that pulls that off — VictoriaLogs — from the bottom up, until “fast” stops looking like magic and starts looking like an obvious consequence of how the bytes are laid out.

Why this engine is worth reading

The payoff is big enough to state up front. These are the project’s own published figures — “up to N×”, best case — not an independent benchmark, so treat them as the ceiling and measure on your own logs. Still, the direction isn’t subtle:

The interesting part, for us, is that every one of those numbers is a direct consequence of a design decision we’re going to read — columnar storage, per-stream grouping, bloom filters, opt-in streams. The brag list is really a table of contents in disguise.

How we’ll read it

Every chapter after this one follows the same move — we build the naive version of one mechanism, hit the wall it runs into, then read how VictoriaLogs actually solves it.

Two rules keep us honest, because a systems post that’s confidently wrong is worse than no post at all:

• Every non-obvious claim is anchored to the real source, checked against the current code.
• The tests are the oracle: when a mental model disagrees with the engine’s own test, the model is wrong.

Where are we headed:

1. The data model — fields, streams, tenants, and the map (this chapter).
2. Append-only ingestion — how a line gets in, and why “just append it” dies at the first query.
3. Sorting & immutable parts — turning a pile of lines into a frozen, searchable part.
4. The LSM merge engine — the in-memory buffer and background merges that avoid a file per flush.
5. Columnar blocks — per-column, typed, min/max, dictionary and const columns.
6. Full-text without scanning — tokenization and bloom filters that skip blocks a query can’t match.
7. The wide-events refactor — the real history where 20-field logs caused ~1000× read amplification, and the fix.
8. The stream inverted index — indexdb: resolving {app="api"} through postings set-algebra.
9. Partitions, retention & the query flow — every layer tied into one read pipeline.

One thing this series is not :

A tour of the query language. LogsQL — its pipes and stats functions — is a big topic that deserves its own series; here we only follow how a query executes against storage, not its grammar.

It all rests on one idea: the stream. Get the stream wrong and nothing else about the system quite makes sense — so that’s where we’ll start.

Let’s start with the atom. A log entry is just a flat bag of fields — key = value pairs, both strings:

{ "app": "api", "pod": "api-7", "level": "error",
  "trace_id": "t-aaa", "_msg": "db timeout" }

“Flat” and “strings” are enforced, not assumed — VictoriaLogs reshapes every incoming log into that form at ingest time. Nested objects are flattened, joining keys with a dot:

{ "host": { "os": { "version": "1.2.3" } } }   →   { "host.os.version": "1.2.3" }

and everything becomes a string — arrays, numbers and booleans are all stringified, so full-text search works the same way over every field:

{ "offset": 12345, "is_error": false }   →   { "offset": "12345", "is_error": "false" }

One more rule surprises people: an empty value is the same as a missing field. {"_msg":"x","region":""} and {"_msg":"x"} are identical to VictoriaLogs — the empty region is dropped before anything else happens. (Keep that one in mind; it quietly changes how a stream is computed, a few paragraphs down.)

So the payload is nothing but a flat bag of string→string fields. All the structure that makes queries fast lives in how four special field names are treated — everything else is an ordinary “other” field:

Field	What it is	The subtlety
_msg	the log message	internally its name is the empty string "", not _msg — and it gets its own files on disk, apart from every other column
_time	the timestamp	it doesn't just power the time filter — it decides which day-partition the line is stored in. Time is physical layout here, not just a searchable value
_stream	the stream’s labels, {k="v",…}	derived, not ingested — you don’t send it
_stream_id	a unique id for the stream	also derived — a hash, computed as the line comes in

You send _msg, maybe _time, and your ordinary fields. The last two — _stream and _stream_id — the engine derives for you. And how it does that turns out to be the whole game.

A stream is a hash of the fields you choose

A stream is meant to be “all the logs from one running instance.” But the engine can’t guess which fields identify an instance, so you tell it — you nominate a few stream fields at ingestion time (say, app and pod). Those fields, and only those, become the line’s identity.

Toggle them below and watch what comes out:

↺ reset to app, pod

Three things that diagram is quietly proving, and each is a real property of the code:

• Only the stream fields count. level, trace_id, even _msg change nothing about the stream. Two lines with totally different messages are the same stream as long as app and pod match. That’s the whole point: putting one instance’s lines together is what makes them compress well and what lets an {app="api"} query skip everyone else’s data.
• Order and empties don’t matter. The chosen fields are sorted into a canonical form before hashing, and empty ones were already dropped. So {pod, app} and {app, pod, region=""} land on the exact same stream.
• The id carries the tenant. The real _stream_id is tenant + hash128(tags) — a pair of account/project integers prepended to a 128-bit hash. That’s why the id in the diagram starts with a run of zeros: a single-tenant store is just tenant (0, 0).

If you nominate no stream fields, every line falls into the single empty stream {}. It still works — you’ve just quietly thrown away the compression and the query locality that streams exist to give you in the first place.

The map: where that line ends up

Now zoom out. That stream doesn’t float free — it lives in a nested structure worth having in your head before any of the later chapters. And there’s less of it than you’d expect: there is one binary. “Single-node”, “insert”, and “select” aren’t separate programs — they’re the same executable behaving differently based on flags. We’ll stay on the single-node path, where the whole engine is one Go package.

Inside it, the containment goes three levels deep. A Storage owns a list of partitions, one per day (a line’s _time is what sorts it into its day). And here’s the fact to carry through the rest of the series: each partition keeps two completely independent LSM trees — one for the log data, one for the stream index.

The datadb is where the actual log fields live, stored by column (that’s chapters of its own: parts, blocks, and columns). The indexdb is a different tree entirely — it answers “which streams carry the tag app="api"?” so a query can find streams without scanning them. Two trees, two background mergers, two jobs. It’s worth keeping them separate in your head from day one — a lot of confusion later comes from quietly blurring the two.

Do it yourself: from a log line to bytes on disk

Everything above you can watch happen on your own machine in about five minutes — no Docker, no config, nothing to sign up for. It’s worth following along; there’s something clarifying about seeing the data model stop being a diagram and turn into real files you can ls.

1 · Get VictoriaLogs

Make a scratch directory, then find out which build you need — the archive must match your OS and CPU architecture:

mkdir vl-quick-start && cd vl-quick-start
uname -sm        # your OS + arch, e.g. "Darwin arm64" or "Linux x86_64"

Map that to the archive name on the releases page — victoria-logs-<os>-<arch>-<version>.tar.gz:

uname -sm	Archive
Darwin arm64	victoria-logs-darwin-arm64-…  (Apple Silicon: M1/M2/M3/M4)
Darwin x86_64	victoria-logs-darwin-amd64-…  (Intel Mac)
Linux x86_64	victoria-logs-linux-amd64-…
Linux aarch64	victoria-logs-linux-arm64-…

For example, on an Apple-Silicon Mac:

curl -L -O https://github.com/VictoriaMetrics/VictoriaLogs/releases/download/v1.52.0/victoria-logs-darwin-arm64-v1.52.0.tar.gz
tar xzf victoria-logs-darwin-arm64-v1.52.0.tar.gz    # unpacks a single `victoria-logs-prod`

2 · Start it

Just run it — no arguments needed:

./victoria-logs-prod

It listens on port 9428 and creates a victoria-logs-data/ directory next to the binary — that’s where everything we’re about to inspect will land. Leave it running and open a second terminal for the next steps.

3 · Ingest five log lines

Put these five lines in sample.jsonl. They’re designed to prove the stream rules: the first three share app=api, pod=api-7 but differ in message — and line 3 even lists the fields in a different order and carries an empty region. The last two change pod, then app:

{"_time":"2026-10-31T10:00:01Z","app":"api","pod":"api-7","level":"info","trace_id":"t-aaa","_msg":"user logged in"}
{"_time":"2026-10-31T10:00:02Z","app":"api","pod":"api-7","level":"error","trace_id":"t-bbb","_msg":"db timeout after 3s"}
{"_time":"2026-10-31T10:00:03Z","pod":"api-7","app":"api","region":"","_msg":"field order and empty region do not change the stream"}
{"_time":"2026-10-31T10:00:04Z","app":"api","pod":"api-9","_msg":"different pod = different stream"}
{"_time":"2026-10-31T10:00:05Z","app":"worker","pod":"w-2","_msg":"different app = different stream"}

Send them to the jsonline endpoint, naming app and pod as the stream fields:

curl -X POST -H 'Content-Type: application/stream+json' --data-binary @sample.jsonl \
  'http://localhost:9428/insert/jsonline?_stream_fields=app,pod'

The Content-Type header matters — without it the body is read as a web form and nothing is stored. A 200 with an empty body means success.

4 · Query them back

Ask the server for each line’s derived _stream and _stream_id (the query is LogsQL; _time:1d just means “the last day”):

curl http://localhost:9428/select/logsql/query \
  -d 'query=_time:1d | sort by (_time) | keep _stream, _stream_id, _msg'

{app="api",pod="api-7"}    0000000000000000f9857ac5ce08a117958ea2f865a13e65   user logged in
{app="api",pod="api-7"}    0000000000000000f9857ac5ce08a117958ea2f865a13e65   db timeout after 3s
{app="api",pod="api-7"}    0000000000000000f9857ac5ce08a117958ea2f865a13e65   field order and empty region do not change the stream
{app="api",pod="api-9"}    0000000000000000311d0b67ca7c3f9d942fbbf2440e9470   different pod = different stream
{app="worker",pod="w-2"}   00000000000000005bf978436a7c2bfaac1faf4208696354   different app = different stream

There it is: the first three lines collapse to one _stream_id despite different messages, reordered fields, and that empty region. Change a stream field and the id changes. And the id’s shape is exactly tenant + hash — the leading 0000000000000000 is tenant (0,0); the rest is the 128-bit hash of the sorted tags. Count the distinct streams and you get 3, not 5:

curl http://localhost:9428/select/logsql/query -d 'query=_time:1d | count_uniq(_stream_id)'
# {"count_uniq(_stream_id)":"3"}

(Prefer clicking? The same data is browsable in the built-in Web UI at /select/vmui.)

5 · Watch the files appear

In-memory data is flushed to disk every few seconds (-inmemoryDataFlushInterval, default 5s). Wait a moment, then list the data directory — and the containment map from earlier draws itself in real directories:

victoria-logs-data/
└── partitions/
    └── 20261031/                    ← one partition — today (YYYYMMDD)
        ├── datadb/                  ← LSM tree #1: the log DATA
        │   ├── parts.json           ← the list of live parts
        │   └── 18E396AD68651529/    ← a "part": an immutable directory
        └── indexdb/                 ← LSM tree #2: the stream INDEX
            ├── parts.json
            └── 18E396AD6862BF81/    ← a part, but a different shape (see below)

The per-day partition, and inside it the two independent LSM trees — not a diagram now, but folders you can cd into.

6 · Anatomy of a part

List that datadb part directory and you get a dozen small files (sizes from this exact run):

   200  metadata.json              53  metaindex.bin
   133  index.bin                   5  timestamps.bin
    44  column_names.bin           27  values.bin0
   259  columns_header.bin          0  bloom.bin0
    37  columns_header_index.bin    9  message_values.bin
     7  column_idxs.bin             0  message_bloom.bin

They look cryptic, but they fall into four jobs:

Start with metadata.json — the part’s summary:

{ "FormatVersion": 3, "RowsCount": 5, "BlocksCount": 3,
  "CompressedSizeBytes": 574, "UncompressedSizeBytes": 729, "BloomValuesShardsCount": 1 }

Three blocks for five rows. That’s the data model showing through: rows are sorted by (stream, time) and a block is cut at every stream change, so our three streams become three blocks — one block never mixes two streams.

Two details worth noticing now, both of which pay off later:

• message_values.bin and message_bloom.bin exist as separate files. _msg really is stored apart from every other field — the empty-name column, as promised.
• "FormatVersion": 3, and files like columns_header_index.bin and column_idxs.bin exist. Those are the wide-events refactor — a real redesign we’ll devote a whole chapter to.

Finally, glance at the other tree. The indexdb part has a completely different file set — items.bin, lens.bin — because it’s a generic sorted key→value store (a “mergeset”), not columnar at all. Its metadata.json counts items, not rows:

{ "ItemsCount": 11, "BlocksCount": 1,
  "FirstItem": "000000000000000000311d0b67ca7c3f9d942fbbf2440e9470", … }

Look closely at that FirstItem: buried in the hex is 311d0b67… — the same stream-id hash we saw for pod=api-9 up in step 4. The index literally stores the streams, so a {app="api"} query can find them without touching datadb. Two trees, two jobs, two formats — exactly the map we drew. Cracking open those .bin files is what the rest of the series is for.

What you now know

1. A log entry is a flat bag of string→string fields; nested JSON is flattened, every value is stringified, and empty equals missing.
2. Four fields are special: _msg (stored under the empty name), _time (which picks the day-partition), and the derived _stream / _stream_id.
3. A stream is tenant + hash128 of the chosen stream fields only, in sorted order — so messages, field order, and empty values never change it.
4. Choosing stream fields is your biggest lever: good ones group an instance together; a high-cardinality one shatters storage into a stream per line.
5. The shape on disk: one binary → a Storage of per-day partitions → two independent LSM trees each (datadb for data, indexdb for the stream index).

So one line becomes a field bag, a chosen handful of those fields hashes it into a stream, and that stream lands in a per-day partition split across two trees. Next we follow that line the rest of the way in — from an HTTP request to sorted, immutable bytes inside datadb — and watch the first naive design hit its first wall.