#1 VictoriaLogs storage internals: How the logs get in
Every log storage engine is a wager about where to pay. Writes are relentless and dull — gigabytes an hour, each line worth almost nothing on its own. Reads are rare but desperate: it’s 3 a.m., something is on fire, and someone needs this error, from this service, in the last five minutes, now. You can spend the cost of that bet at write time or at read time, but you cannot escape it. Where you choose to pay decides everything else about the system you end up with.
The honest way to feel that choice is to make the laziest one possible and watch it break. So that’s where we’ll start: we’ll build the simplest log store that still genuinely works, use it, and let it fail on its own terms. Everything VictoriaLogs does afterward will read not as cleverness for its own sake, but as a considered answer to a failure we felt with our own hands.
The laziest store that works
Call it naive-log. One rule governs it: it must not be a strawman. If it were, every move VictoriaLogs makes later would feel staged — a magician palming the card he’s about to “discover.” naive-log has to be something you’d actually reach for at small scale, something whose appeal is real. Only then is its breaking point real too.
Here it is, in full — small enough to hold in your head:
// Append: O(1) amortized. Just bytes on the end of a file.
func Append(f *os.File, fields map[string]string) error {
b, _ := json.Marshal(fields)
_, err := f.Write(append(b, '\n'))
return err
}
// Query: read the WHOLE file, parse EVERY line, filter in memory.
func Scan(path string, min, max time.Time, match map[string]string) ([]map[string]string, error) {
f, _ := os.Open(path)
defer f.Close()
var out []map[string]string
sc := bufio.NewScanner(f)
for sc.Scan() { // (1) no skipping — reads every byte
var row map[string]string
json.Unmarshal(sc.Bytes(), &row) // (2) parses every field to check one
t, _ := time.Parse(time.RFC3339, row["_time"])
if t.Before(min) || t.After(max) {
continue
}
ok := true
for k, v := range match {
if row[k] != v { // (3) compares after full parse — no value index
ok = false
break
}
}
if ok {
out = append(out, row)
}
}
return out, sc.Err()
}
A line comes in; we append it to the end of a file on disk. That is the entire write side. There is no index, no sort, no bookkeeping — which means that when a question arrives, there is also no shortcut. To answer anything at all, we have to read the whole file, because nothing in it offers a sublinear path to what we’re looking for.
Why the lazy choice is so tempting
Look at the write path and you’ll understand why anyone would build this on purpose: an incoming log becomes JSON, and those bytes plus a newline go on the end of the file. That’s all.
That is the whole seduction. Writing is trivial, fast, and free of coordination — three steps and no thinking. But nothing is actually free; the cost didn’t vanish, it moved. Every bit of work we skipped at write time is now waiting at read time, with interest. That deferred bill is, quite literally, the rest of this series.
Where the bill comes due
Now send the kind of question people actually ask: give me svc=api and level=error between 09:00 and 09:05. Watch what the store has to do to answer it.
Two lines match. To find them, we read all of them. And this is the cruel part: the cost has nothing to do with the size of the answer and everything to do with the size of the history. A one-row result in a billion-row file still reads a billion rows. The store doesn’t get slow under load — it gets slower forever, a little more with every line ever written.
The tax hiding inside a single line
It’s tempting to think the problem is just “too many lines.” But the waste goes deeper than that: even reading one field out of one line is expensive, because of how the line is stored.
So the tax compounds in two directions at once. Across the file you pay for every line, even the ones that don’t match. Within each line you pay for every field, even the ones the query never mentions. A filter on level drags the entire row — timestamps, trace IDs, request paths, message bodies — through the parser just to glance at one word.
Why it’s doomed
Six walls
The slowness isn’t a bug we could patch. It falls out of six structural choices baked into “just append JSON to a file” — and each one is a wall the rest of this series spends its time climbing. It’s worth naming them precisely, because every later move VictoriaLogs makes is aimed squarely at one of these.
1 · No ordering. Lines sit in the file in whatever order they happened to arrive, which is nobody’s idea of sorted. A matching row could be anywhere, so there’s no binary search to reach for, no range to seek to — only the march from the top.
2 · No grouping. The file knows nothing about which lines belong together. “All the logs from the api service” isn’t a place you can jump to; it’s a verdict you reach one line at a time, after testing every line in the file.
3 · Row-oriented. Each line is one glued-together blob. To read a single field you must parse the whole row — every field before it to find it, every field after it to know where the line ends.
4 · No value index. Even if you’ve reached the right field, there’s no index over its values. A search for an error code or a trace ID reads every value in the file; the data can’t tell you where its own contents live.
5 · Unbounded growth. Append-only cuts both ways. You can’t punch a hole in the middle of a file, so deleting old data means rewriting the whole thing. Retention by age — the one thing every logging system needs — is impossible without copying everything you’re keeping.
6 · No compression that survives reads. Logs are wildly repetitive and should compress beautifully. But plain JSONL only compresses generically, and gzipping the whole file destroys random access: to read line N you have to decompress from byte zero. Compression and seekability are at war, and naive-log loses both.
What logs are actually like
Before we can judge any fix, we need to know the material we’re working with. naive-log fails not just because it’s lazy, but because it ignores everything particular about logs — and logs are a peculiar kind of data. A good log engine isn’t built in spite of these quirks; it’s built to exploit every one of them. Hold these six next to the six walls above: each wall is one of these properties left unused, and every design choice still ahead is one of them finally turned to advantage.
1 · Write-once. A log line is never updated in place. It is written once and from then on only read. A storage engine that never has to modify a row in the middle of a file is free to do things a general-purpose database cannot — most of all, to treat its files as immutable.
2 · Torrential, but cheap. Logs arrive by the gigabyte or terabyte a day, yet any single row is worth almost nothing. That ratio rules out any design that spends real money per row — a precise index on every field, say — because you’d pay a fortune to catalogue data you’ll likely never look at.
3 · Nearly ordered, never exactly. When a line shows up at the ingest endpoint, its _time is usually close to now, and usually later than the line before it. But usually is not always — clocks skew, buffers flush late, a batch arrives out of order. The whole gap between “usually” and “always” is where real design decisions get made; an engine that assumed perfect order would corrupt itself on the first late arrival.
4 · Drowning in repetition. Look at a thousand consecutive log lines and you’ll see the same level, the same service, the same host, the same Kubernetes labels, over and over. The information is in what changes; the rest is near-constant. An engine that stores those repeats naively is paying again and again for a value it already knows.
5 · Narrow, predictable questions. For all their volume, logs get asked a surprisingly narrow set of questions: a time range, a few field filters, and some free-text search. You almost never join logs or aggregate across all of history. That narrowness is a gift — you can optimize hard for the shape of query that actually shows up.
6 · Short-lived. Unlike the records in a business database, logs age out fast — days or weeks, not years. The engine doesn’t need to keep data forever; it needs to drop it cheaply and constantly, which pulls retention and deletion to the center of the design instead of the margins.
The four fixes you’re already shouting
If you’ve read this far, you’re probably arguing with the screen — just do X, obviously. Good. There are four obvious fixes, and a careful reader thinks of all of them. The interesting thing is that none of them is simply wrong; each is half-right, and the exact way each one fails is the motivation for a specific thing VictoriaLogs does later. So let’s take the objections seriously, one at a time.
“Just index every field.” This is roughly what Elasticsearch does, and it works — at a price. Indexing everything means logs are too high-volume and too cheap-per-row to afford it: you build an expensive, precise catalogue of data that mostly never gets queried. Property 2 (torrential but cheap) says no. The bill shows up as hardware.
Index everything and you pay for everything. VictoriaLogs indexes the few fields that identify a stream and leaves the rest as cheap, compressed columns — which is a large part of why it runs the same log workload on a fraction of the RAM and disk. The naive “index it all” instinct isn’t free; it’s just a bill paid in machines.
“Just keep the file sorted by time.” The moment you insist on sorted order, you destroy the O(1) append that made naive-log attractive in the first place — every insert now has to find its place. And even then, one sort order can only answer questions about that order; sorting by time still can’t jump to “this stream’s fields.” VictoriaLogs’ answer is the LSM move, and it’s the backbone of Post 2: never re-sort one giant file. Instead, write many small, immutable, already-sorted parts and merge them in the background, so you get sorted data without ever paying to sort the whole history at once.
“Just use one file per day.” This one is genuinely half-right — it’s partitioning, and VictoriaLogs really does split data by time for exactly this reason. It bounds the time range a query must consider, and it makes deleting old data as cheap as dropping a file (there’s wall 5 handled). But it’s coarse: within a day you’re back to a full scan. Necessary, not sufficient.
“Just gzip the file.” Compression is the right instinct — logs are absurdly repetitive — but whole-file gzip is the wrong shape for it. It trades away random access: to read line N you must decompress from byte 0, so you’ve swapped one scan for another. The fix isn’t don’t compress, it’s compress in blocks, so you decompress only the block you actually need to read. That tension — compression versus seekability — is resolved on disk in Post 4.
The front door
Those four fixes fail in four different ways, but they share one blind spot: each tries to get clever after the data is already lying on disk as opaque lines. That’s the instinct to resist. You cannot optimize your way out of a bad decision about what you’re storing — an index over meaningless blobs is a meticulous catalogue of the wrong thing.
So VictoriaLogs’ first real move isn’t an index, or a sort, or a compressor. It’s streamID from them, and lays the row into a column-friendly shape — all before storage is touched.
That’s the wager restated in one sentence each. naive-log decides it is storing “a line of text.” VictoriaLogs decides it is storing “a typed row that already knows which stream it belongs to.” Everything later in the series — sorting, columns, bloom filters, the inverted index — is only possible because of that one decision at the door. So let’s stand at the door and watch a single line come through, slowed right down.
The first thing the door has to handle is chaos at the threshold. Logs arrive in roughly ten different wire formats — Loki, Elasticsearch, OpenTelemetry, syslog, journald, Datadog, Splunk, jsonline, and VictoriaLogs’ own native protocol — and no sane storage engine wants to know about all of them. So the entire ingest layer exists to do one job: melt every one of those formats down into a single neutral shape — (timestamp, []Field) — and hand it across one narrow interface. Everything protocol-specific lives and dies on the ingest side of that boundary; the storage engine beneath never grows a branch for “is this Loki?”
The funnel — many doors, one convergence
The router ( app/vlinsert/main.go 42 ) sends each URL path to its own protocol handler. But every handler, however different its input, does the same job and ends at the same call:
The neutral shape is deliberately dumb — rows.go 18 :
type Field struct {
Name string
Value string
}
Both fields are strings. At ingest, everything is stringly-typed — even req.ms=12 arrives as the string “12”.
jsonline, traced end to end
We’ll pick one handler and follow it all the way down, because the others are variations on its theme. jsonline is the plainest of them — a newline-delimited stream of JSON objects — which makes it the clearest place to see the shared skeleton every protocol wears.
The heart is readLine ( jsonline.go 98 ):
p := logstorage.GetJSONParser()
defer logstorage.PutJSONParser(p)
if err := p.ParseLogMessage(line, preserveKeys, ""); err != nil { // ① parse + flatten
return true, fmt.Errorf("%w; line contents: %q", err, line)
}
ts, err := insertutil.ExtractTimestampFromFields(timeFields, p.Fields) // ② pull out the timestamp
if err != nil {
return true, fmt.Errorf("%w; line contents: %q", err, line)
}
logstorage.RenameField(p.Fields, msgFields, "_msg") // ③ normalize the message field
lmp.AddRow(ts, p.Fields, -1)
Note the loop’s error policy in processStreamInternal: a bad line is logged and skipped; the request only fails if every line failed ( jsonline.go 88 ). One malformed log doesn’t poison a batch.
The three transformations — where the data model is born
Here is the conceptual heart of ingestion, and it’s worth slowing down for. The wire knows nothing of _time or _msg; those are VictoriaLogs’ ideas, not the sender’s. The ingest layer invents them, per request, out of whatever arbitrary JSON showed up. Three small moves turn a stranger’s object into VictoriaLogs’ own data model:
Flattening matters: {"req":{"path":"/x"}} becomes a field literally named req.path. That’s why VL’s data model is flat even though JSON is nested — the flattening happens right here at the door.
CommonParams — how the door knows what you meant
Those three transformations, and the streamID step just ahead, all depend on something the handler cannot guess: which incoming field is the timestamp, which is the message, which fields are the identity. Is the time in ts, in timestamp, in @timestamp? A generic handler has no way to know. So every request has to say — it declares the mapping — and GetCommonParams ( common_params.go 44 ) gathers that declaration into a single struct the rest of the pipeline reads from.
Each of these settings can be given either as a query arg or as a VL-* header — getArray checks both — so someone poking at it with curl and a production log shipper sending headers drive the exact same code path.
The settings that matter for this post:
| Setting | arg / header | Feeds |
|---|---|---|
TimeFields | _time_field / VL-Time-Field | transformation ② — extract _time |
MsgFields | _msg_field / VL-Msg-Field | transformation ③ — rename _msg |
StreamFields | _stream_fields / VL-Stream-Fields | the streamID (next section) |
IgnoreFields, ExtraFields, Debug, … | various | drop fields · stamp constant fields · dry-run |
The seam — one interface, injected once
The ingest layer never imports the storage code. The moment a batch is ready, it pushes it through a two-method interface ( common_params.go 173 ):
type LogRowsStorage interface {
MustAddRows(lr *logstorage.LogRows) // store this batch
CanWriteData() error // can you accept writes right now?
}
Think of it as a wall socket. Ingest holds a plug that fits the socket and nothing more — it has no idea what’s behind the wall. Exactly one line, at startup, decides what’s plugged in ( app/victoria-logs/main.go 50 ):
insertutil.SetLogRowsStorage(&vlstorage.Storage{}) // single-node: the local-disk storage
That one injection is the whole of the single-node-versus-cluster decision. Every file we’ve read, and every file left in this post, only ever presses that socket — none of them branch on “am I a cluster?”
Inside the buffer
Our line has been parsed, normalized, and handed to AddRow. It is now inside VictoriaLogs — but not yet stored, not yet given its stream identity, not yet laid out. The rest of ingest happens here, in the space between “a request handed us fields” and “a batch crossed into storage.” Three things happen to our line in that space: it waits in a buffer, it earns a streamID, and it gets laid down in a deliberately strange shape. We’ll take them in that order.
Buffering — one processor, one cart, three triggers
AddRow does not store a row. A single request can carry thousands of lines, and writing each one to storage individually would recreate the naive store’s tiny-write problem. So for each request the handler opens one processor owning one reusable buffer — call it the cart — and every line is dropped into that same cart until something decides to wheel it away.
The cart isn’t freshly allocated; it’s borrowed from a pool (GetLogRows), already labelled with this request’s CommonParams so it knows how to shape each line without being re-told. At terabytes a day, make-and-discard would bury the garbage collector; reuse is the Post-0 “cheap per row” economics showing up in code.
AddRow, per line ( common_params.go 253 )
lmp.mu.Lock(); defer lmp.mu.Unlock() // claim the cart
lmp.unflushedRows++; lmp.unflushedBytes += EstimatedJSONRowLen(fields) // tally (for metrics)
if len(fields) > *MaxFieldsPerLine { drop + warn; return } // guard: default 1000
lmp.lr.MustAdd(cp.TenantID, timestamp, fields, streamFieldsLen) // into the cart → streamID + layout
if lmp.cp.Debug { render; ResetKeepSettings; return } // dry-run: show, don't keep
if lmp.lr.NeedFlush() { lmp.flushLocked() } // size trigger
The lock exists because a background timer goroutine can also reach for the same cart (next). The two counters are for metrics (vl_rows_ingested_total and friends) — the “is the cart full?” decision is a separate measurement the cart keeps on itself (NeedFlush: more than 1.75 MB of bytes, or more than ~20,971 rows).
The three triggers
flushLocked ( common_params.go 319 ) is the drain: MustAddRows(lr) pushes the whole batch through the socket, then ResetKeepSettings() empties the cart but keeps its label — so it’s the same cart start to finish, emptied and refilled, never swapped. MustClose does a final flush and returns the cart to the pool, so no line is ever stranded in a half-full cart.
The visibility gap — two buffers in series
Here’s the landmine, and it’s one the verification pass corrected. A flush does not make a row searchable. Crossing the socket only lands the batch in the storage layer’s own buffer, which runs its own ~1-second timer before it builds the in-memory part that queries actually read.
So the honest phrasing for the post: a freshly ingested row isn’t searchable for ~1–2s — the ingest buffer hands it to storage immediately on request close, but the storage layer buffers it ~1s more before building the in-memory part queries read. That timer is a storage-layer (Post 3) mechanism, not the ingest cart. Tests skip the wait with DebugFlush.
streamID — four doors, one formula
This is the step that gives the post its name — the one thing naive-log never does. naive-log appends an opaque line; VictoriaLogs appends a row that already knows which stream it belongs to. Everything downstream — the per-stream grouping, the cheap sort, the skippable blocks — is bought here, at the cost of one hash computed on the way in.
Post 0 taught the recipe: a streamID is tenant + hash128 of the chosen stream fields, sorted into a canonical form so order never matters. What Post 0 left open is the ingest-time question — when a raw row arrives, how does VictoriaLogs even know which fields are the stream fields? The protocols disagree about where that information lives, so MustAdd ( log_rows.go 359 ) offers four doors into the same recipe.
Inside MustAdd, which door is taken is a plain if / else-if / else:
- Door A — positional (
streamFieldsLen >= 0): the firstNfields are the stream fields. OpenTelemetry and Loki, where the protocol already groups identity fields up front. - Door B — configured names: walk all fields, keep the ones whose name is in the request’s
_stream_fieldslist. jsonline, Elasticsearch, Datadog, Splunk — this is the_stream_fields=serviceroute. - Door C — embedded
_stream: the row already carries a{service="api"}label (re-importing VL’s own exported logs); parse it back. No label either → the single empty stream{}. - The native door (
MustAddInsertRow): in a cluster the id was computed once at the edge and shipped precomputed, so this door skips the recipe entirely.
Then all of them converge:
bb.B = st.MarshalCanonical(bb.B) // sort tags → canonical bytes (order-independent)
sid.tenantID = tenantID
sid.id = hash128(bb.B) // the 128-bit hash
The gate and the layout
The streamID done, the row reaches mustAddInternal ( log_rows.go 428 ) — the last stop in this post. It does two jobs: check the row is allowed in, then lay it into the cart in a deliberately odd shape.
The gate — three limits
Fail any one, and the whole row is dropped with a throttled warning:
| Check | Limit | Why |
|---|---|---|
| fields per row | 2000 | bounds the columns in a block |
| field-name length | 128 bytes | names become column metadata — keep it small |
| estimated row size | 2 MiB | one row must fit inside one block |
A normal log — a handful of short-named fields, a few KB — sails straight through. The gate only exists to refuse the pathological row that would wreck the on-disk format later.
Structure of arrays
You’d expect the cart to keep a list of rows, each row a self-contained bundle of its time, id, and fields. It does the opposite: it keeps several parallel columns, and reconstructs row i by reading index i from each.
The payoffs are both downstream, but the shape is locked in now: separated key arrays make Post 2’s sort by (streamID, timestamp) cheap (shuffle small keys and slice headers, not field bytes), and starting from columns makes Post 4’s transpose to on-disk columns natural.
Store once — the repetition trick
The field text itself gets one more optimization, and it exploits the Post-0 fact that logs are massively repetitive. When a field is identical to the field in the same column of the row directly above, VictoriaLogs doesn’t write the text again — it reuses it, like a ditto mark in a hand-written ledger (addFieldsInternal, log_rows.go 489 ).
Before that, each field runs a quick filter — empty values are dropped (empty = missing), _time is dropped (it lives in the timestamps column), _stream/_stream_id are rejected, and _msg is canonicalized to the empty name. And one guarantee: every stored row ends up with a _msg, falling back to -defaultMsgValue if the line had none.
Be precise about the limits of what we’ve built: this is not yet columnar (each row’s fields are still grouped in its window), and not yet sorted. The sort by (streamID, timestamp), the transpose into real columns, and the freeze into an immutable part all happen past the socket — and that’s where Post 2 opens.
Where this leaves us
The whole path, and the line we won’t cross
HTTP POST /insert/jsonline (bytes)
│ parse+flatten · extract _time · rename _msg
▼ (timestamp, []Field) ──▶ lmp.AddRow
│ accumulate in one pooled cart; flush on size/time/close
▼ MustAdd
│ pick a door → MarshalCanonical → hash128 → streamID
▼ mustAddInternal
│ gate (2000/128B/2MiB) · drop empties · store-once · SoA
▼ [ cart: arrival order · not sorted · not columnar ]
═══ the socket: MustAddRows ═══ ← Post 1 ends here
▼ storage buffers ~1s → in-memory part (searchable)
Prove it to yourself
None of this has to be taken on faith — the whole chapter is visible from a terminal. Spin up the server from Post 0, send one line naming service as the stream field, then ask for it back:
curl -X POST 'http://localhost:9428/insert/jsonline?_time_field=ts&_msg_field=message&_stream_fields=service' \
-H 'Content-Type: application/json' \
--data-binary '{"ts":"2026-11-01T09:00:00Z","message":"boom","service":"api","level":"error","req":{"path":"/x","ms":12}}'
# wait ~1-2s (the visibility gap), then read it back
curl http://localhost:9428/select/logsql/query -d 'query=service:api | sort by (_time)'
What comes back proves every transformation at once:
{ "_msg":"boom", "_stream":"{service=\"api\"}",
"_stream_id":"0000000000000000 06ea53c6272f075b2cc7f8692a30c236",
"_time":"2026-11-01T09:00:00Z",
"level":"error", "req.ms":"12", "req.path":"/x", "service":"api" }
req was flattened to req.path / req.ms; req.ms is the string "12"; ts was extracted to _time and is gone as a field; message was renamed to _msg; the _stream_id is tenant(0,0) + hash128; and service=api is in _stream and still a stored field — identity and stored.
And a probe for the streamID rules — ingest a few variants and count the distinct ids:
| Ingested | _stream_id | Proves |
|---|---|---|
app, pod, msg:a | …114142fbb… | baseline |
pod, app, msg:b (reversed) | …114142fbb… (same) | order-independent (canonical sort) |
app, pod, msg:c, level:debug | …114142fbb… (same) | non-stream field ignored by the hash |
app, pod:p9, msg:d | …e1a0fd79… (different) | a new stream-field value → a new stream |
What you now know
- Before you can be fast you must decide what you store. The naive store says “a line of text”; VictoriaLogs says “a typed row that already knows its stream” — and it decides this at the front door.
- ~10 protocols funnel into one neutral shape,
(timestamp, []Field), via three transformations — flatten, extract_time, rename_msg— all driven by the per-request CommonParams. - Rows pile into one pooled buffer per request, flushed by size, a ~1s timer, or close — and pushed through a two-method socket that is the entire single-node-vs-cluster fork.
- The streamID is one recipe reached by four doors; a stream field is identity and stored, and only stream fields are hashed.
- In the buffer, a row is laid out as parallel columns, not a bundle, with repeated field values stored once — a shape chosen to make the next post’s sort and columns cheap.
The row is now in memory: stamped with its stream, in arrival order, not sorted and not yet a file. Next we cross the socket and watch that pile of rows become a sorted, columnar, immutable part — the first real on-disk artifact of the engine.
One line, the whole journey — follow the line ▸ strip down the page and watch the same log change shape at every stage, while the mechanism around it (counters, flush triggers, the hash recipe, the gate) explains why: