ProtoWire: schema-backed serialization for humans and machines

Motivation

Configs and messages deserve types.

JSON and YAML are easy to read but tell you nothing about shape. Textproto is tied to one tool. FlatBuffers, MessagePack, and CBOR drop the schema at the edge. ProtoWire reuses the .proto definitions you already have, so every byte across every encoding stays type-checked, versioned, and wire-compatible.

Schema-backed

The parser always knows each field's type. No ambiguity, no out-of-band validation, no surprise null.

Three formats, one schema

PXF (text), pb (protobuf wire), and SBE (FIX Simple Binary Encoding) are generated from the same .proto.

Three field states

Distinguish set, null, and absent, which is essential for PATCH semantics and config inheritance.

Well-known type sugar

Inline RFC 3339 timestamps and Go-style durations. Write 2024-01-15T10:30:00Z or 1h30m directly, no nested blocks required.

PXF in practice

Reads like config. Validates like protobuf.

The text format mixes assignments, blocks, lists, and maps in one consistent syntax. Comments use #, //, or /* */. Triple-quoted strings preserve raw multi-line content.

Server config: assignments and blocks

// Schema-pinned to infra.v1.ServerConfig
@type infra.v1.ServerConfig

hostname = "web-01.prod.example.com"
port     = 8443
enabled  = true
status   = STATUS_SERVING

created_at = 2024-01-15T10:30:00Z
timeout    = 30s

tls {
  cert_file = "/etc/ssl/cert.pem"
  key_file  = "/etc/ssl/key.pem"
  verify    = true
}

tags = ["production", "us-east", "frontend"]

labels = {
  env: "production"
  team: "platform"
  "hello world": "quoted keys supported"
}

Repeated messages and integer maps

endpoints = [
  {
    path   = "/api/v1/users"
    method = GET
  }
  {
    path   = "/health"
    method = GET
  }
]

error_codes = {
  404: "Not Found"
  500: "Internal Error"
}

nullable_name = "present"

Oneof with nested choices

event_id = "evt-456"

user {
  user_id = "u-123"
  login {
    ip  = "192.168.1.1"
    mfa = true
  }
}

Performance

Built for the hot path.

ProtoWire is engineered for speed at every level of the stack. PXF reads about twice as fast as JSON and roughly five times as fast as YAML on the same payload, and the binary tier is a different conversation entirely. The numbers below are from a single representative Config message (one TLS block, three endpoints, four labels, and a timestamp + duration) on the Go reference implementation; every port runs the same harness against the same canonical testdata.

Encoded size

Lower is better. pb is the baseline.

Format	Bytes	vs `pb`
`pb` (protobuf wire)	301	1.00×
JSON	606	2.01×
YAML	606	2.01×
PXF	662	2.20×

PXF carries a small premium over JSON because it spells out enum names, type directives, and timestamps as text. That cost buys you a file you can review, comment, and edit by hand.

Encode and decode time

Lower is better. Per-message, single thread.

Format	Encode	Decode	Allocs (encode / decode)
PXF	4.5 µs	6.1 µs	14 / 80
`pb`	6.5 µs	7.4 µs	49 / 115
JSON	8.2 µs	10.9 µs	75 / 151
YAML	32.0 µs	35.5 µs	192 / 460

PXF beats JSON by roughly 1.8× on both encode and decode, and YAML by an order of magnitude. Allocations matter at scale: PXF cuts encode allocations by 5× against JSON and 13× against YAML.

Same harness, every port

The cross-port harness (scripts/cross_pxf_bench.sh and cross_sbe_bench.sh) decodes the same canonical fixtures into descriptor-driven dynamic messages, so the comparison reflects codec dispatch rather than generated-message ergonomics. C++ leads on PXF and SBE; Go and Rust trail by a small constant; the JVM and TypeScript ports are within a 2× band on encode.

Port	PXF unmarshal	PXF marshal	SBE unmarshal	SBE marshal
C++	3.78 µs	3.13 µs	382 ns	236 ns
Go	5.52 µs	3.48 µs	1.05 µs	378 ns
Rust	5.88 µs	5.26 µs	576 ns	432 ns
Java	9.32 µs	3.32 µs	865 ns	276 ns
TypeScript	11.78 µs	4.65 µs	1.59 µs	946 ns

PXF fixture: 624-byte bench.v1.Config (mixed scalars, lists, maps, Timestamp, Duration). SBE fixture: 94-byte bench.v1.Order (10 scalars + a 2-entry repeating group). Apple M1, 3-second window per op. Numbers are not yet collected for Python, C#, Swift, and Dart.

Sub-microsecond reads with the SBE tier

When latency is the budget, SBE drops the parser entirely. The Order payload encodes in 378 ns in Go and 236 ns in C++; a typed View reads fields with zero allocations. Same data, same schema, traded for read-side cost.

SBE is wire-compatible with FIX SBE 1.0 and shares the same .proto definitions as the rest of ProtoWire. Switching tiers is a flag, not a rewrite.

Cross-format numbers (top tables) are from the Go reference implementation; cross-port numbers are aggregated from scripts/cross_pxf_bench.sh and cross_sbe_bench.sh. All measurements on Apple M1, single core, ProtoWire 0.73.x (post-hardening with strict UTF-8 validation and MaxNestingDepth=100 enforced on decode). Allocation counts include the dynamic-message scaffolding used for parity across encodings; production deployments with generated code will see further improvements on the binary tiers.

Pick your stack

Ten implementations, one wire contract.

Every port is verified against the same canonical testdata fixtures, so messages written in one language round-trip cleanly through any other.

Go github.com/trendvidia/protowire-go Reference implementation → C++ libprotowire Native libprotobuf codec → Rust cargo add protowire prost-reflect backend → Java com.trendvidia:protowire JDK 21 + Gradle → Kotlin org.protowire:protowire-kotlin Coroutine wrappers, DSL builders, sealed Result → TypeScript npm i @trendvidia/protowire Pure TS on protobuf-es → Python pip install protowire nanobind C++ wrapper → C# dotnet add package Protowire .NET 10.0+ → Swift SwiftPM Apple platforms + Linux → Dart pub add protowire Flutter-friendly →

Beyond the codecs

protoregistry: where your schemas live.

The codecs read and write bytes; protoregistry stores and serves the .proto files those bytes are typed against. It is a multi-namespace schema registry with versioning, two-phase staging, and backward-compatibility enforcement, built for the same stack as the rest of ProtoWire.

Multi-namespace by design

Each namespace is a self-contained scope. Imports resolve only inside the namespace, so two teams shipping different versions of common.proto never collide.

Two-phase publish & promote

Publish compiles and stages a candidate set; promote atomically swaps every staged version to current, so coordinated multi-schema changes land together.

Compatibility enforced at promote time

Field deletion, type changes, cardinality changes are rejected before traffic sees them. Rollback is a pointer move with zero data duplication.

Hot-swap, lock-free

Readers grab compiled descriptors via atomic.Pointer. Schema swaps are instant; no draining, no restarts.

gRPC + CLI + Go SDK

The gRPC service in proto/protoregistry/v1/ is the durable integration point. A protoregistry CLI runs the server and manages namespaces; the Go client exposes a remote-backed protoreflect.MessageTypeResolver.

Custom built-in types

Extend Google's well-known types with your own shared protos via the reserved __builtins__ namespace. Shadowing real WKTs is rejected unless you opt in.

protoregistry on GitHub → Stability contract

Stability

Wire-compatible since 0.73.0.

ProtoWire commits to backwards-compatible wire formats until the next major release. Read the full contract in STABILITY.md, or jump straight to the EBNF and railroad diagrams.

One .proto, three wire formats. Built for humans and machines.