Write-Ahead
← the log
Log storage

#1 VictoriaLogs storage internals: How the logs get in

Every log storage engine is a wager about where to pay. Writes are relentless and dull — gigabytes an hour, each line worth almost nothing on its own. Reads are rare but desperate: it’s 3 a.m., something is on fire, and someone needs this error, from this service, in the last five minutes, now. You can spend the cost of that bet at write time or at read time, but you cannot escape it. Where you choose to pay decides everything else about the system you end up with.

The honest way to feel that choice is to make the laziest one possible and watch it break. So that’s where we’ll start: we’ll build the simplest log store that still genuinely works, use it, and let it fail on its own terms. Everything VictoriaLogs does afterward will read not as cleverness for its own sake, but as a considered answer to a failure we felt with our own hands.

The laziest store that works

Call it naive-log. One rule governs it: it must not be a strawman. If it were, every move VictoriaLogs makes later would feel staged — a magician palming the card he’s about to “discover.” naive-log has to be something you’d actually reach for at small scale, something whose appeal is real. Only then is its breaking point real too.

Here it is, in full — small enough to hold in your head:

// Append: O(1) amortized. Just bytes on the end of a file.
func Append(f *os.File, fields map[string]string) error {
	b, _ := json.Marshal(fields)
	_, err := f.Write(append(b, '\n'))
	return err
}

// Query: read the WHOLE file, parse EVERY line, filter in memory.
func Scan(path string, min, max time.Time, match map[string]string) ([]map[string]string, error) {
	f, _ := os.Open(path)
	defer f.Close()
	var out []map[string]string
	sc := bufio.NewScanner(f)
	for sc.Scan() {                              // (1) no skipping — reads every byte
		var row map[string]string
		json.Unmarshal(sc.Bytes(), &row)        // (2) parses every field to check one
		t, _ := time.Parse(time.RFC3339, row["_time"])
		if t.Before(min) || t.After(max) {
			continue
		}
		ok := true
		for k, v := range match {
			if row[k] != v {                    // (3) compares after full parse — no value index
				ok = false
				break
			}
		}
		if ok {
			out = append(out, row)
		}
	}
	return out, sc.Err()
}

A line comes in; we append it to the end of a file on disk. That is the entire write side. There is no index, no sort, no bookkeeping — which means that when a question arrives, there is also no shortcut. To answer anything at all, we have to read the whole file, because nothing in it offers a sublinear path to what we’re looking for.

svc=apisvc=workersvc=dbnaive-log.jsonl_time=09:01 svc=api level=info_time=09:04 svc=db level=warn_time=09:02 svc=api level=error_time=09:05 svc=worker level=info_time=09:03 svc=db level=info_time=09:02 svc=api (late arrival)every queryreads ALLN linesno index⇒ O(N)append to the end — O(1), no sort, no coordination
naive-log is one flat file. Every source appends its JSON line to the end (O(1), no coordination), so lines land in arrival order — which is only roughly time-ordered (note the late 09:02 at the bottom). There is no index of any kind, so every read has to march through all N lines.

Why the lazy choice is so tempting

Look at the write path and you’ll understand why anyone would build this on purpose: an incoming log becomes JSON, and those bytes plus a newline go on the end of the file. That’s all.

incoming logfieldsencode JSONMarshalbytes + ‘\n’Write()fileend⟵O(1) amortized — just bytes on the endall the cost is deferred to read time
The entire write path: take the incoming fields, encode them to JSON, append the bytes plus a newline to the end of the file. Three steps, no bookkeeping — this is why appending feels free. The whole cost of that simplicity is pushed to read time.

That is the whole seduction. Writing is trivial, fast, and free of coordination — three steps and no thinking. But nothing is actually free; the cost didn’t vanish, it moved. Every bit of work we skipped at write time is now waiting at read time, with interest. That deferred bill is, quite literally, the rest of this series.

Where the bill comes due

Now send the kind of question people actually ask: give me svc=api and level=error between 09:00 and 09:05. Watch what the store has to do to answer it.

svc=api AND level=error, t in [09:00,09:05]naive-log.jsonlsvc=db level=warnsvc=api level=error ✓svc=api level=infosvc=worker level=errorsvc=api level=error ✓svc=db level=infocursor: no skip1 · no skippingreads every byte2 · parse every fieldto check one3 · compare after parse
The query svc=api AND level=error in [09:00,09:05] touches every line: the cursor cannot skip (no order), it fully parses each line to read any field, and it compares only after parsing. Two lines match — but the work was the whole file. The answer is tiny; the cost is O(N).

Two lines match. To find them, we read all of them. And this is the cruel part: the cost has nothing to do with the size of the answer and everything to do with the size of the history. A one-row result in a billion-row file still reads a billion rows. The store doesn’t get slow under load — it gets slower forever, a little more with every line ever written.

The tax hiding inside a single line

It’s tempting to think the problem is just “too many lines.” But the waste goes deeper than that: even reading one field out of one line is expensive, because of how the line is stored.

_timesvcpodleveltrace_id_msgreq.pathreq.msthe only field the query wantsone opaque JSON blob — fields have no offsetsto read level, parse everything before it (and after, to find the end)row-oriented ⇒ pay for every field to read one
One line, stored row-oriented as opaque JSON. The query only needs `level`, but the fields are glued together in one blob — so to reach `level` the parser must walk and decode every field before it (and every field after, to know where the line ends). You pay for the whole row to read one column.

So the tax compounds in two directions at once. Across the file you pay for every line, even the ones that don’t match. Within each line you pay for every field, even the ones the query never mentions. A filter on level drags the entire row — timestamps, trace IDs, request paths, message bodies — through the parser just to glance at one word.

Why it’s doomed

Six walls

The slowness isn’t a bug we could patch. It falls out of six structural choices baked into “just append JSON to a file” — and each one is a wall the rest of this series spends its time climbing. It’s worth naming them precisely, because every later move VictoriaLogs makes is aimed squarely at one of these.

123456No orderingmatches anywhere;no binary searchNo groupingno per-streambucketsRow-orientedparse all fieldsto read oneNo value indextoken search scansall valuesUnbounded growthdelete = fullrewrite; no holesNo compressiongzip killsrandom access
The six structural limitations of naive-log, each one a wall the rest of the series climbs. Every later move VictoriaLogs makes — sorting, per-stream grouping, columnar blocks, bloom filters, immutable parts, dictionary encoding — exists to knock out one of these panels.

1 · No ordering. Lines sit in the file in whatever order they happened to arrive, which is nobody’s idea of sorted. A matching row could be anywhere, so there’s no binary search to reach for, no range to seek to — only the march from the top.

2 · No grouping. The file knows nothing about which lines belong together. “All the logs from the api service” isn’t a place you can jump to; it’s a verdict you reach one line at a time, after testing every line in the file.

3 · Row-oriented. Each line is one glued-together blob. To read a single field you must parse the whole row — every field before it to find it, every field after it to know where the line ends.

4 · No value index. Even if you’ve reached the right field, there’s no index over its values. A search for an error code or a trace ID reads every value in the file; the data can’t tell you where its own contents live.

5 · Unbounded growth. Append-only cuts both ways. You can’t punch a hole in the middle of a file, so deleting old data means rewriting the whole thing. Retention by age — the one thing every logging system needs — is impossible without copying everything you’re keeping.

6 · No compression that survives reads. Logs are wildly repetitive and should compress beautifully. But plain JSONL only compresses generically, and gzipping the whole file destroys random access: to read line N you have to decompress from byte zero. Compression and seekability are at war, and naive-log loses both.

What logs are actually like

Before we can judge any fix, we need to know the material we’re working with. naive-log fails not just because it’s lazy, but because it ignores everything particular about logs — and logs are a peculiar kind of data. A good log engine isn’t built in spite of these quirks; it’s built to exploit every one of them. Hold these six next to the six walls above: each wall is one of these properties left unused, and every design choice still ahead is one of them finally turned to advantage.

123456Append-onlywrite once, thenquery — no updatesHuge volumeGB–TB per day;each row low-valueNear-orderedarrives ~sorted,never exactlyRepetitivelevel, svc, host,k8s labels repeatNarrow queriestime + a few filters+ free-text searchShort retentionaged out quickly;no long horizons
The six properties of a log workload — the raw material every good log store is built around. Append-only writes, torrential volume with cheap rows, near-time-ordering, heavy repetition, a narrow query shape, and short retention. Hold these next to the six limitations of naive-log: each limitation is one of these levers left unused, and every VictoriaLogs design choice ahead is one of them finally pulled.

1 · Write-once. A log line is never updated in place. It is written once and from then on only read. A storage engine that never has to modify a row in the middle of a file is free to do things a general-purpose database cannot — most of all, to treat its files as immutable.

2 · Torrential, but cheap. Logs arrive by the gigabyte or terabyte a day, yet any single row is worth almost nothing. That ratio rules out any design that spends real money per row — a precise index on every field, say — because you’d pay a fortune to catalogue data you’ll likely never look at.

3 · Nearly ordered, never exactly. When a line shows up at the ingest endpoint, its _time is usually close to now, and usually later than the line before it. But usually is not always — clocks skew, buffers flush late, a batch arrives out of order. The whole gap between “usually” and “always” is where real design decisions get made; an engine that assumed perfect order would corrupt itself on the first late arrival.

4 · Drowning in repetition. Look at a thousand consecutive log lines and you’ll see the same level, the same service, the same host, the same Kubernetes labels, over and over. The information is in what changes; the rest is near-constant. An engine that stores those repeats naively is paying again and again for a value it already knows.

5 · Narrow, predictable questions. For all their volume, logs get asked a surprisingly narrow set of questions: a time range, a few field filters, and some free-text search. You almost never join logs or aggregate across all of history. That narrowness is a gift — you can optimize hard for the shape of query that actually shows up.

6 · Short-lived. Unlike the records in a business database, logs age out fast — days or weeks, not years. The engine doesn’t need to keep data forever; it needs to drop it cheaply and constantly, which pulls retention and deletion to the center of the design instead of the margins.

The four fixes you’re already shouting

If you’ve read this far, you’re probably arguing with the screen — just do X, obviously. Good. There are four obvious fixes, and a careful reader thinks of all of them. The interesting thing is that none of them is simply wrong; each is half-right, and the exact way each one fails is the motivation for a specific thing VictoriaLogs does later. So let’s take the objections seriously, one at a time.

✗✗✗✗Index every fieldKeep it sorted by timeOne file per dayJust gzip the fileElasticsearch’s move — ~30× RAM, ~15× disk.Ignores volume: rows too many, too cheap each.Destroys the O(1) append; one order can’t servestream + field. → LSM: many small sorted parts.Half-right — that’s partitioning, and VL does it.Bounds time & deletion; within a day, full scan.Kills random access — line N ⇒ decompress frombyte 0. Compress per block, so you read only that.each rejected fix is the motivation for a real VictoriaLogs design choice
The steelman. Each obvious fix a sharp reader would shout (left, crossed out) is answered by the reason it fails (right) — and that reason is the motivation for a concrete VictoriaLogs design choice. Indexing everything ignores volume; a single sort order both kills the O(1) append and still can't serve stream+field queries (hence the LSM move); one-file-per-day is real but only coarse; whole-file gzip trades away random access.

“Just index every field.” This is roughly what Elasticsearch does, and it works — at a price. Indexing everything means logs are too high-volume and too cheap-per-row to afford it: you build an expensive, precise catalogue of data that mostly never gets queried. Property 2 (torrential but cheap) says no. The bill shows up as hardware.

VictoriaLogs VS Elasticsearch
~30×
less RAM
~15×
less disk

Index everything and you pay for everything. VictoriaLogs indexes the few fields that identify a stream and leaves the rest as cheap, compressed columns — which is a large part of why it runs the same log workload on a fraction of the RAM and disk. The naive “index it all” instinct isn’t free; it’s just a bill paid in machines.

“Just keep the file sorted by time.” The moment you insist on sorted order, you destroy the O(1) append that made naive-log attractive in the first place — every insert now has to find its place. And even then, one sort order can only answer questions about that order; sorting by time still can’t jump to “this stream’s fields.” VictoriaLogs’ answer is the LSM move, and it’s the backbone of Post 2: never re-sort one giant file. Instead, write many small, immutable, already-sorted parts and merge them in the background, so you get sorted data without ever paying to sort the whole history at once.

“Just use one file per day.” This one is genuinely half-right — it’s partitioning, and VictoriaLogs really does split data by time for exactly this reason. It bounds the time range a query must consider, and it makes deleting old data as cheap as dropping a file (there’s wall 5 handled). But it’s coarse: within a day you’re back to a full scan. Necessary, not sufficient.

“Just gzip the file.” Compression is the right instinct — logs are absurdly repetitive — but whole-file gzip is the wrong shape for it. It trades away random access: to read line N you must decompress from byte 0, so you’ve swapped one scan for another. The fix isn’t don’t compress, it’s compress in blocks, so you decompress only the block you actually need to read. That tension — compression versus seekability — is resolved on disk in Post 4.

The front door

Those four fixes fail in four different ways, but they share one blind spot: each tries to get clever after the data is already lying on disk as opaque lines. That’s the instinct to resist. You cannot optimize your way out of a bad decision about what you’re storing — an index over meaningless blobs is a meticulous catalogue of the wrong thing.

So VictoriaLogs’ first real move isn’t an index, or a sort, or a compressor. It’s imposing structure at the front door — deciding, in the instant a line arrives, what this thing is before it’s ever written down. At ingest it parses the raw log into typed fields, picks out the few fields that say where the line came from (its stream fields), computes a compact streamID from them, and lays the row into a column-friendly shape — all before storage is touched.

That’s the wager restated in one sentence each. naive-log decides it is storing “a line of text.” VictoriaLogs decides it is storing “a typed row that already knows which stream it belongs to.” Everything later in the series — sorting, columns, bloom filters, the inverted index — is only possible because of that one decision at the door. So let’s stand at the door and watch a single line come through, slowed right down.

The first thing the door has to handle is chaos at the threshold. Logs arrive in roughly ten different wire formats — Loki, Elasticsearch, OpenTelemetry, syslog, journald, Datadog, Splunk, jsonline, and VictoriaLogs’ own native protocol — and no sane storage engine wants to know about all of them. So the entire ingest layer exists to do one job: melt every one of those formats down into a single neutral shape — (timestamp, []Field) — and hand it across one narrow interface. Everything protocol-specific lives and dies on the ingest side of that boundary; the storage engine beneath never grows a branch for “is this Loki?”

The funnel — many doors, one convergence

The router ( app/vlinsert/main.go 42 ) sends each URL path to its own protocol handler. But every handler, however different its input, does the same job and ends at the same call:

jsonlineElasticsearchLokiOpenTelemetryDatadogSplunkjournaldnative~10 wire formats(timestamp, []Field)one neutral shapelmp.AddRow
The funnel. Around ten wire protocols enter through their own URL paths, but every handler does the same job: boil its format down to one neutral shape — (timestamp, []Field) — and hand it to a single call, lmp.AddRow. Past that convergence point nothing downstream knows which protocol a row came from.

The neutral shape is deliberately dumb — rows.go 18 :

type Field struct {
	Name  string
	Value string
}

Both fields are strings. At ingest, everything is stringly-typed — even req.ms=12 arrives as the string “12”.

jsonline, traced end to end

We’ll pick one handler and follow it all the way down, because the others are variations on its theme. jsonline is the plainest of them — a newline-delimited stream of JSON objects — which makes it the clearest place to see the shared skeleton every protocol wears.

POSTbodyGetCommonParamsNewLogMessageProcessorper line (loop)readLineAddRowMustClose
The life of one jsonline request. Read the request's settings once (GetCommonParams), open one processor wrapping one buffer, then loop: each line is parsed and handed to AddRow. At the end MustClose flushes whatever is left. The per-line work (readLine → AddRow) is the only part that repeats.

The heart is readLine ( jsonline.go 98 ):

p := logstorage.GetJSONParser()
defer logstorage.PutJSONParser(p)

if err := p.ParseLogMessage(line, preserveKeys, ""); err != nil {   // ① parse + flatten
	return true, fmt.Errorf("%w; line contents: %q", err, line)
}
ts, err := insertutil.ExtractTimestampFromFields(timeFields, p.Fields) // ② pull out the timestamp
if err != nil {
	return true, fmt.Errorf("%w; line contents: %q", err, line)
}
logstorage.RenameField(p.Fields, msgFields, "_msg")                    // ③ normalize the message field
lmp.AddRow(ts, p.Fields, -1)         

Note the loop’s error policy in processStreamInternal: a bad line is logged and skipped; the request only fails if every line failed ( jsonline.go 88 ). One malformed log doesn’t poison a batch.

The three transformations — where the data model is born

Here is the conceptual heart of ingestion, and it’s worth slowing down for. The wire knows nothing of _time or _msg; those are VictoriaLogs’ ideas, not the sender’s. The ingest layer invents them, per request, out of whatever arbitrary JSON showed up. Three small moves turn a stranger’s object into VictoriaLogs’ own data model:

123parse + flattennested to dottedextract timestamppull it out of fieldsrename messageto _msgreq ▸ ms = 12ts = 09:00:01message = boomreq.ms = “12”_time · ts removed_msg = boom
The three transformations that manufacture the data model at the door. Nested keys are flattened to dotted names; the chosen time field is pulled out of the fields and into a real timestamp; the chosen message field is renamed to _msg. What enters as arbitrary JSON leaves as VictoriaLogs' flat, special-field data model.

Flattening matters: {"req":{"path":"/x"}} becomes a field literally named req.path. That’s why VL’s data model is flat even though JSON is nested — the flattening happens right here at the door.

CommonParams — how the door knows what you meant

Those three transformations, and the streamID step just ahead, all depend on something the handler cannot guess: which incoming field is the timestamp, which is the message, which fields are the identity. Is the time in ts, in timestamp, in @timestamp? A generic handler has no way to know. So every request has to say — it declares the mapping — and GetCommonParams ( common_params.go 44 ) gathers that declaration into a single struct the rest of the pipeline reads from.

Each of these settings can be given either as a query arg or as a VL-* header — getArray checks both — so someone poking at it with curl and a production log shipper sending headers drive the exact same code path.

?_stream_fields=…VL-Stream-Fields:…arg OR headerCommonParamsper requestTimeFields → _timeMsgFields → _msgStreamFields → streamID
CommonParams is the one place the request's intent is gathered. Each setting can be given as a URL arg or an equivalent header, and each one feeds a specific downstream step. The generic handler holds no knowledge like 'ts means time' — it all arrives here, per request.

The settings that matter for this post:

Settingarg / headerFeeds
TimeFields_time_field / VL-Time-Fieldtransformation ② — extract _time
MsgFields_msg_field / VL-Msg-Fieldtransformation ③ — rename _msg
StreamFields_stream_fields / VL-Stream-Fieldsthe streamID (next section)
IgnoreFields, ExtraFields, Debug, …variousdrop fields · stamp constant fields · dry-run

The seam — one interface, injected once

The ingest layer never imports the storage code. The moment a batch is ready, it pushes it through a two-method interface ( common_params.go 173 ):

type LogRowsStorage interface {
	MustAddRows(lr *logstorage.LogRows)   // store this batch
	CanWriteData() error                   // can you accept writes right now?
}

Think of it as a wall socket. Ingest holds a plug that fits the socket and nothing more — it has no idea what’s behind the wall. Exactly one line, at startup, decides what’s plugged in ( app/victoria-logs/main.go 50 ):

insertutil.SetLogRowsStorage(&vlstorage.Storage{})   // single-node: the local-disk storage
ingest sidegeneric · one plugMustAddRowsthe socketsingle-node → local diskcluster → network nodesplugged in once, at boot
The seam. Ingest talks only to a two-method socket. One line at boot decides what's plugged in: the local-disk storage for single-node, or a network forwarder for cluster. Every ingest file upstream is identical in both — the entire single-node-vs-cluster fork collapses to this one injection.

That one injection is the whole of the single-node-versus-cluster decision. Every file we’ve read, and every file left in this post, only ever presses that socket — none of them branch on “am I a cluster?”

Inside the buffer

Our line has been parsed, normalized, and handed to AddRow. It is now inside VictoriaLogs — but not yet stored, not yet given its stream identity, not yet laid out. The rest of ingest happens here, in the space between “a request handed us fields” and “a batch crossed into storage.” Three things happen to our line in that space: it waits in a buffer, it earns a streamID, and it gets laid down in a deliberately strange shape. We’ll take them in that order.

Buffering — one processor, one cart, three triggers

AddRow does not store a row. A single request can carry thousands of lines, and writing each one to storage individually would recreate the naive store’s tiny-write problem. So for each request the handler opens one processor owning one reusable buffer — call it the cart — and every line is dropped into that same cart until something decides to wheel it away.

The cart isn’t freshly allocated; it’s borrowed from a pool (GetLogRows), already labelled with this request’s CommonParams so it knows how to shape each line without being re-told. At terabytes a day, make-and-discard would bury the garbage collector; reuse is the Post-0 “cheap per row” economics showing up in code.

AddRow, per line ( common_params.go 253 )

lmp.mu.Lock(); defer lmp.mu.Unlock()                     // claim the cart
lmp.unflushedRows++; lmp.unflushedBytes += EstimatedJSONRowLen(fields)  // tally (for metrics)
if len(fields) > *MaxFieldsPerLine { drop + warn; return }   // guard: default 1000
lmp.lr.MustAdd(cp.TenantID, timestamp, fields, streamFieldsLen)   // into the cart → streamID + layout
if lmp.cp.Debug { render; ResetKeepSettings; return }    // dry-run: show, don't keep
if lmp.lr.NeedFlush() { lmp.flushLocked() }              // size trigger

The lock exists because a background timer goroutine can also reach for the same cart (next). The two counters are for metrics (vl_rows_ingested_total and friends) — the “is the cart full?” decision is a separate measurement the cart keeps on itself (NeedFlush: more than 1.75 MB of bytes, or more than ~20,971 rows).

The three triggers

① SIZE (full)② TIME (~1s)③ CLOSE (ends)flushLockedthe cartMustAddRows → socketResetKeepSettings → reuse
One cart, three ways to wheel it out: it fills past the size limit, a ~1s jittered timer fires, or the request closes. All three run flushLocked — push the whole batch through the socket, then empty the cart but keep its label, so the same cart is reused for the next lines.

flushLocked ( common_params.go 319 ) is the drain: MustAddRows(lr) pushes the whole batch through the socket, then ResetKeepSettings() empties the cart but keeps its label — so it’s the same cart start to finish, emptied and refilled, never swapped. MustClose does a final flush and returns the cart to the pool, so no line is ever stranded in a half-full cart.

The visibility gap — two buffers in series

Here’s the landmine, and it’s one the verification pass corrected. A flush does not make a row searchable. Crossing the socket only lands the batch in the storage layer’s own buffer, which runs its own ~1-second timer before it builds the in-memory part that queries actually read.

ingest cartnot searchabledatadb bufferstill notin-memory partsearchable ✓diskdurable ✓MustClose flushAfterFunc(~1s)◂ Post 1 (ingest) ▸◂ Post 2 / 3 (storage) ▸
Why a freshly POSTed row isn't searchable for a second or two. The ingest cart flushes to the storage layer the instant the request closes — but the storage layer buffers ~1s more (a time.AfterFunc timer in datadb) before building the in-memory part queries read. The delay lives past the socket, in the storage layer — a Post 3 mechanism, not the ingest buffer.

So the honest phrasing for the post: a freshly ingested row isn’t searchable for ~1–2s — the ingest buffer hands it to storage immediately on request close, but the storage layer buffers it ~1s more before building the in-memory part queries read. That timer is a storage-layer (Post 3) mechanism, not the ingest cart. Tests skip the wait with DebugFlush.

streamID — four doors, one formula

This is the step that gives the post its name — the one thing naive-log never does. naive-log appends an opaque line; VictoriaLogs appends a row that already knows which stream it belongs to. Everything downstream — the per-stream grouping, the cheap sort, the skippable blocks — is bought here, at the cost of one hash computed on the way in.

Post 0 taught the recipe: a streamID is tenant + hash128 of the chosen stream fields, sorted into a canonical form so order never matters. What Post 0 left open is the ingest-time question — when a raw row arrives, how does VictoriaLogs even know which fields are the stream fields? The protocols disagree about where that information lives, so MustAdd ( log_rows.go 359 ) offers four doors into the same recipe.

A · first N fieldsB · configured namesC · embedded _streamnative · precomputedinside MustAddMarshalCanonicalsort → byteshash128 → streamID
Four doors, one formula. Three ways a row inside MustAdd declares its stream fields — by position, by configured name, or by an embedded _stream label — plus a native door where the id was already computed upstream in a cluster. All of them converge on the same recipe: sort to canonical bytes, hash, prepend the tenant.

Inside MustAdd, which door is taken is a plain if / else-if / else:

  • Door A — positional (streamFieldsLen >= 0): the first N fields are the stream fields. OpenTelemetry and Loki, where the protocol already groups identity fields up front.
  • Door B — configured names: walk all fields, keep the ones whose name is in the request’s _stream_fields list. jsonline, Elasticsearch, Datadog, Splunk — this is the _stream_fields=service route.
  • Door C — embedded _stream: the row already carries a {service="api"} label (re-importing VL’s own exported logs); parse it back. No label either → the single empty stream {}.
  • The native door (MustAddInsertRow): in a cluster the id was computed once at the edge and shipped precomputed, so this door skips the recipe entirely.

Then all of them converge:

bb.B = st.MarshalCanonical(bb.B)   // sort tags → canonical bytes (order-independent)
sid.tenantID = tenantID
sid.id = hash128(bb.B)             // the 128-bit hash

The gate and the layout

The streamID done, the row reaches mustAddInternal ( log_rows.go 428 ) — the last stop in this post. It does two jobs: check the row is allowed in, then lay it into the cart in a deliberately odd shape.

The gate — three limits

Fail any one, and the whole row is dropped with a throttled warning:

CheckLimitWhy
fields per row2000bounds the columns in a block
field-name length128 bytesnames become column metadata — keep it small
estimated row size2 MiBone row must fit inside one block

A normal log — a handful of short-named fields, a few KB — sails straight through. The gate only exists to refuse the pathological row that would wreck the on-disk format later.

Structure of arrays

You’d expect the cart to keep a list of rows, each row a self-contained bundle of its time, id, and fields. It does the opposite: it keeps several parallel columns, and reconstructs row i by reading index i from each.

i=0i=1i=2streamIDstimestampsstreamTagsrows (field windows)S_apiS_apiS_db09:0109:0209:03service=api(reuse ↑)service=dbwindow →window →window →one index ties a row across all columns
The cart stores columns, not rows. streamIDs, timestamps, the canonical stream tags, and each row's field-window are separate parallel arrays joined by one index. This shape is chosen now so Post 2's sort by (streamID, timestamp) is cheap — you shuffle small keys, not bulky field text — and Post 4's transpose into on-disk columns is natural.

The payoffs are both downstream, but the shape is locked in now: separated key arrays make Post 2’s sort by (streamID, timestamp) cheap (shuffle small keys and slice headers, not field bytes), and starting from columns makes Post 4’s transpose to on-disk columns natural.

Store once — the repetition trick

The field text itself gets one more optimization, and it exploits the Post-0 fact that logs are massively repetitive. When a field is identical to the field in the same column of the row directly above, VictoriaLogs doesn’t write the text again — it reuses it, like a ditto mark in a hand-written ledger (addFieldsInternal, log_rows.go 489 ).

row0row1row2servicelevel_msgapiinfostarted〃〃okdberrortimeout〃 = reuse the value above — zero new bytes
Logs repeat, so repeated field values are stored once. A field identical to the one in the same column of the previous row is reused (a ditto mark), not rewritten. Here only the message changes line to line, so service and level cost nothing after the first row — foreshadowing the dictionary and const-column encodings of Post 4.

Before that, each field runs a quick filter — empty values are dropped (empty = missing), _time is dropped (it lives in the timestamps column), _stream/_stream_id are rejected, and _msg is canonicalized to the empty name. And one guarantee: every stored row ends up with a _msg, falling back to -defaultMsgValue if the line had none.

Be precise about the limits of what we’ve built: this is not yet columnar (each row’s fields are still grouped in its window), and not yet sorted. The sort by (streamID, timestamp), the transpose into real columns, and the freeze into an immutable part all happen past the socket — and that’s where Post 2 opens.

Where this leaves us

The whole path, and the line we won’t cross

HTTP POST /insert/jsonline (bytes)
  │ parse+flatten · extract _time · rename _msg      
  ▼ (timestamp, []Field) ──▶ lmp.AddRow
  │ accumulate in one pooled cart; flush on size/time/close  
  ▼ MustAdd
  │ pick a door → MarshalCanonical → hash128 → streamID       
  ▼ mustAddInternal
  │ gate (2000/128B/2MiB) · drop empties · store-once · SoA    
  ▼ [ cart: arrival order · not sorted · not columnar ]
  ═══ the socket: MustAddRows ═══  ← Post 1 ends here
  ▼ storage buffers ~1s → in-memory part (searchable)        

Prove it to yourself

None of this has to be taken on faith — the whole chapter is visible from a terminal. Spin up the server from Post 0, send one line naming service as the stream field, then ask for it back:

curl -X POST 'http://localhost:9428/insert/jsonline?_time_field=ts&_msg_field=message&_stream_fields=service' \
  -H 'Content-Type: application/json' \
  --data-binary '{"ts":"2026-11-01T09:00:00Z","message":"boom","service":"api","level":"error","req":{"path":"/x","ms":12}}'

# wait ~1-2s (the visibility gap), then read it back
curl http://localhost:9428/select/logsql/query -d 'query=service:api | sort by (_time)'

What comes back proves every transformation at once:

{ "_msg":"boom", "_stream":"{service=\"api\"}",
  "_stream_id":"0000000000000000 06ea53c6272f075b2cc7f8692a30c236",
  "_time":"2026-11-01T09:00:00Z",
  "level":"error", "req.ms":"12", "req.path":"/x", "service":"api" }

req was flattened to req.path / req.ms; req.ms is the string "12"; ts was extracted to _time and is gone as a field; message was renamed to _msg; the _stream_id is tenant(0,0) + hash128; and service=api is in _stream and still a stored field — identity and stored.

And a probe for the streamID rules — ingest a few variants and count the distinct ids:

Ingested_stream_idProves
app, pod, msg:a…114142fbb…baseline
pod, app, msg:b (reversed)…114142fbb… (same)order-independent (canonical sort)
app, pod, msg:c, level:debug…114142fbb… (same)non-stream field ignored by the hash
app, pod:p9, msg:d…e1a0fd79… (different)a new stream-field value → a new stream

What you now know

  1. Before you can be fast you must decide what you store. The naive store says “a line of text”; VictoriaLogs says “a typed row that already knows its stream” — and it decides this at the front door.
  2. ~10 protocols funnel into one neutral shape, (timestamp, []Field), via three transformations — flatten, extract _time, rename _msg — all driven by the per-request CommonParams.
  3. Rows pile into one pooled buffer per request, flushed by size, a ~1s timer, or close — and pushed through a two-method socket that is the entire single-node-vs-cluster fork.
  4. The streamID is one recipe reached by four doors; a stream field is identity and stored, and only stream fields are hashed.
  5. In the buffer, a row is laid out as parallel columns, not a bundle, with repeated field values stored once — a shape chosen to make the next post’s sort and columns cheap.

The row is now in memory: stamped with its stream, in arrival order, not sorted and not yet a file. Next we cross the socket and watch that pile of rows become a sorted, columnar, immutable part — the first real on-disk artifact of the engine.

One line, the whole journey — follow the line ▸ strip down the page and watch the same log change shape at every stage, while the mechanism around it (counters, flush triggers, the hash recipe, the gate) explains why:

123456Raw bytes arrive on the wirevlinsert/jsonlinenested JSON · stringly-typed · ~10 wire protocols funnel to one callline ▸{ time, msg:boom, level:error, service:api, req:{path:/x, ms:12} }Three transformations → the neutral shapeParseLogMessage · RenameFieldflatten req:{path} → req.path · extract time → _timerename msg → _msg (nesting gone; flat, ordered Fields)line ▸_time · _msg:boom · level:error · service:api · req.path:/x · req.ms:12Tallied into one pooled bufferLogRows.MustAddunflushedRows += 1 · unflushedBytes += estflush when: > 1.75 MB · ~20,971 rows · ~1 s · closeline ▸buffer row #2 · _msg:boom · service:api · req.path:/x · …Each row is stamped with a streamIDMarshalCanonical · hash128{ service:api } → canonical bytes (sorted) → hash128a stream field is identity AND stored — only stream fields hashedline ▸streamID …2a30c236 ⊕ _msg:boom · service:api · req.path:/xLaid down as parallel columnsgate + structure-of-arraysservice [ api, ↑, ↑ ] ← adjacent duplicate stored once (ditto)gate: ≤ 2000 fields · ≤ 2 MiB/row · columns joined by index iline ▸col[i=2] · _msg[2]:boom · service[2]:api↑ · req.path[2]:/xPushed across the socketPost 1 ends herelogRowsStorage.MustAddRows(lr) · lr.ResetKeepSettings()storage sorts & writes a part — searchable ~1 s later (Post 2/3)line ▸lr ▸ [ …, row#2(boom · api · /x), … ] → shipped↓ into lib/logstorage ↓
The whole of Post 1, following one log line. Read the accented 'line ▸' strip in each card — that is the same record, transformed stage by stage: raw nested JSON on the wire; then flattened and renamed into the neutral _time/_msg/field form; then tallied as a numbered row in the pooled buffer; then stamped with a streamID hashed from its stream fields; then scattered across parallel columns at its index i (a repeat value kept once as a ditto ↑); and finally shipped inside the LogRows across the two-method socket. The text above each strip is the concept at that stage. The spotlight on the left spine marks where the line is right now; crossing the socket is where ingest ends — the storage layer (Post 2/3) sorts and writes it, and it turns searchable about a second later.