Engineering Atlas › System Design Fundamentals

JSON, Protobuf and other wire formats

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Text formats

JSON encodes every record with its field names, numbers as decimal text, and strings with escaping. A record like {"user_id":123456789,"active":true} spends most of its bytes on keys and punctuation. Numbers have no fixed type: many parsers read them as 64-bit floats, so integers above 2^53 silently lose precision. The JSON standard itself only promises that integers between -(2^53)+1 and 2^53-1 travel safely between implementations (RFC 8259). Parsing is branchy byte-by-byte work, typically 200 to 1,000 MB/s with a good library. In return you get universal tooling, readability in logs, and no shared schema needed to decode.

Binary schema formats

Protocol Buffers writes each field as a tag (field number plus wire type) followed by the value. Integers use varints: 7 bits per byte, high bit set when more bytes follow, so 300 takes 2 bytes instead of 8. A negative plain int32 or int64 always takes ten bytes, so the sint32 and sint64 types use zigzag encoding instead, which turns -1 into 1 and keeps small negatives small (Protobuf encoding). Field names never appear on the wire, only numbers, which is why you must never reuse a field number.

Avro takes a different approach: no tags at all. Values are written in schema order, and the reader needs the writer's schema to decode, then resolves it against its own reader's schema by field name. Fields are matched by name, and a reader field the writer lacks must have a default or decoding fails (Avro spec). That makes Avro compact and well suited to files and Kafka topics with a schema registry, but useless without the writer's schema at hand.

Schema evolution

Compatibility is the real design axis. Backward compatible means new readers can read old data; forward compatible means old readers can read new data. Rolling deployments need both, because old and new versions run side by side. With Protobuf: add fields with new numbers, never change a field's type or number, and treat removal as "reserve the number" (Protobuf dos and don'ts). With Avro: new fields need defaults. With JSON there is no enforcement, so compatibility lives in code reviews and contract tests.

The Protobuf language itself has moved on. Editions replace the proto2 and proto3 syntax lines: a file declares edition = "2023" or "2024", and behaviour such as field presence becomes a feature with a default you can override per file or field. In editions, fields track presence by default, as proto2 optional fields did, and the wire format does not change, so old and new files interoperate (Protobuf Editions).

Columnar formats

Parquet stores each column contiguously in row groups, with dictionary and run-length encoding, plus min/max statistics per chunk. A query that reads 3 of 200 columns reads roughly 1.5% of the bytes and can skip row groups whose statistics exclude the filter. Writes are batch-oriented, so it is for analytics, not for RPC.

Down the stack

Layer 1The contract

A format promises round-tripping: decode(encode(x)) equals x for every value the schema allows. Schema formats add a compatibility contract: under stated rules (only add fields with new numbers, give new Avro fields defaults), a reader built against schema version N can read data written with version N-1 or N+1, skipping or defaulting what it does not know. JSON promises much less: RFC 8259 leaves number precision to the implementation and says only that names SHOULD be unique, so a receiver's behaviour on duplicate keys is unpredictable and two conforming parsers can disagree on the same bytes (RFC 8259).

Layer 2The mechanism

Protobuf writes each field as a key varint, (field_number << 3) | wire_type, followed by the payload. Wire type 0 is a varint, 1 is 8 fixed bytes, 2 is length-delimited (strings, bytes, nested messages, packed repeated numbers) and 5 is 4 fixed bytes. A parser that meets an unknown field number still knows how to skip it from the wire type, and modern runtimes keep those bytes so a proxy can pass them through. Avro container files start with the magic bytes Obj plus 0x01, store the writer schema as JSON in the header metadata, and separate compressed blocks with a 16-byte sync marker so files can be split. With a schema registry, each Kafka message is classically prefixed with a magic byte 0 and a 4-byte big-endian schema id; since Confluent Platform 8.1.1 a producer can instead put a 16-byte schema GUID in a record header and leave the payload unprefixed (Confluent docs). Parquet files hold row groups (128 MB is a common writer default, while the Parquet docs recommend 512 MB to 1 GB for large sequential reads), each with one column chunk per column, divided into pages (Parquet configurations), and a footer with schema and per-chunk min/max statistics, bracketed by the magic PAR1.

Layer 3The machine

Text parsing is a branch-heavy, byte-at-a-time loop, and branch mispredictions cost 15 to 20 cycles each. SIMD parsers such as simdjson classify 64 bytes per instruction batch to find quotes and braces, reaching gigabytes per second on one core with a quarter or fewer of the instructions of RapidJSON (Langdale and Lemire), but still pay to convert decimal text to floats, which is surprisingly costly in both directions. A DOM-style JSON parser allocates per node, so garbage collection follows throughput. Protobuf varints are also branchy, while fixed-width fields decode with one little-endian load. Columnar layouts put the same field's values in contiguous memory, so a scan touches only the needed bytes, uses all of every 64-byte cache line, and lets the CPU apply SIMD comparisons to many values at once; similar values side by side also compress far better.

Why it's built this way

  1. Bytes cost network, storage and cache space, so high-volume formats drop field names and text numbers.
  2. Producers and consumers deploy independently, so formats need a way to evolve without lockstep releases.
  3. Tags make each record self-delimiting at a byte or two per field; Avro instead moves the schema out of band to save those bytes.
  4. Analytical queries read few columns over many rows, and disks and CPUs reward sequential access, so warehouses store columns, not rows.
  5. Humans still debug at system boundaries, so readable JSON survives at the edges.

Diagram

flowchart LR
    A["Record: id=300, name=ann"] --> B{"Format"}
    B --> C["JSON: keys + text, 23 bytes"]
    B --> D["Protobuf: tag+varint, tag+len+bytes, 8 bytes"]
    B --> E["Avro: values only, needs writer schema, 6 bytes"]
    D --> F["Reader skips unknown tags"]
    E --> G["Reader resolves writer vs reader schema"]

Worked example

An event pipeline ingests 200,000 events per second. Each event has 12 fields: two 64-bit IDs, a timestamp, an enum, four small integers, and four short strings averaging 10 characters.

At 200,000 events per second, JSON produces 56 MB/s (4.8 TB/day) versus about 17 MB/s (1.5 TB/day) for Protobuf. Compressed with zstd, the gap narrows to perhaps 2x, because compression removes repeated keys. CPU does not narrow: parsing 56 MB/s of JSON at 400 MB/s per core costs about 0.14 cores per consumer group, several times the binary cost, and 50 consumer groups spend 7 cores just parsing. Retained for 7 days with replication factor 3, the raw difference is about 70 TB of Kafka storage.

The decision is not "binary always". The public API keeps JSON for client ergonomics; the internal event bus uses Protobuf with a schema registry enforcing backward and forward compatibility checks in CI.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
JSONUniversal, readable, no schema neededLarge, slow, weak numeric typesPublic APIs, configs, low volume
Protocol BuffersCompact, fast, tagged evolution, codegenNeeds .proto files and toolingRPC, internal events, mobile clients
AvroSmallest rows, schema resolution by nameNeeds writer schema at read timeKafka with registry, Hadoop files
MessagePack/CBORBinary JSON, no schemaField names still on the wireDrop-in speedup for JSON payloads
ParquetColumnar scans, great compressionBatch writes, no point lookupsAnalytics and data lakes

Pitfalls in production

Leadership lens

Check yourself

  1. Why does Protobuf let old readers ignore new fields while Avro needs the writer's schema?

    Show answer

    Protobuf tags each value with its number and wire type, so a reader can skip unknown tags. Avro writes no tags, so the reader cannot even find field boundaries without the writer's schema.

  2. Why does compression narrow the size gap between JSON and Protobuf but not the CPU gap?

    Show answer

    Compression removes repeated key names, but you still pay to compress, decompress and then parse text.

  3. What makes adding a field to an Avro schema backward compatible?

    Show answer

    Giving it a default, so a new reader can fill it in when reading old data that lacks it.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →