ProtoWire spec: PXF v1.0 concrete syntax (EBNF + railroad diagrams)

Document

A PXF document is zero or more directives followed by zero or more field entries. Directives carry document-level metadata or alternative payload shapes — typing, side-channel annotations, tabular rows, embedded schemas. Field entries are the conventional message body. The directive prefix may be empty (a body-only document); the body may be empty (a directive-only document).

document      = { directive } , { field_entry } ;

Map entries (the ':' form) only appear inside a { … } block, where the parser cannot statically tell whether the surrounding field is a message or a map<K, V> — both forms are accepted there and disambiguated by the schema layer.

Directives

v1.0 freezes four directive shapes. @type, @dataset, and @proto each have their own production; everything else is a @<name> named directive that the application's runtime interprets. Directives may appear in any order. An implementation that doesn't recognize an application directive name MAY skip the directive (parsing its block for syntactic well-formedness only) or MAY error.

directive = type_directive
          | dataset_directive
          | proto_directive
          | named_directive ;

`@type` — body's message type

Pins the document's body to a fully-qualified message type. A decoder that expects a specific type refuses a document whose @type doesn't match.

type_directive = '@type' , identifier ;

@type infra.v1.ServerConfig
hostname = "web-01.prod.example.com"

`@<name>` & `@entry` — side-channel directives

Named directives carry document-level data that lives alongside the body. The grammar accepts zero or more identifier prefixes after the name and an optional inline block. The two registrations the spec defines are the conventional one-prefix form (@header pkg.Header { … }) and the spec-registered @entry bundle form (@entry [label] [type] { … } — both prefixes optional).

named_directive = '@' , directive_name , { identifier } , [ block_tail ] ;

// Side-channel header — runtime interprets the name and prefixes:
@header chameleon.v1.LayerHeader { id = "x" }

// Bundle form — @entry with optional label + type:
@entry mylabel pkg.MsgType { x = 1 }

`@dataset` — bulk rows (the CSV replacement)

Carries many instances of one message type in a single document — the protowire-native CSV. The header lists the row's message type and the columns; each subsequent parenthesised tuple is one row, with cells positionally bound to the columns. An empty cell denotes an absent field; the literal null denotes a present-but-null field.

dataset_directive = '@dataset' , [ identifier ] , '(' , column_list , ')' , { row } ;
column_list       = identifier , { ',' , identifier } ;
row               = '(' , row_cell , { ',' , row_cell } , ')' ;
row_cell          = [ row_value ] ;

@dataset trades.v1.Trade ( symbol, price, qty )
( "AAPL", 188.42, 100 )
( "MSFT", 415.10,  50 )
( "AAPL", 188.55,  75 )

A document containing any @dataset MUST NOT also carry @type or top-level field entries: the @dataset header IS the document's type declaration. Cells are scalar-shaped — lists ([ … ]) and blocks ({ … }) inside cells are rejected. The row message type MAY be omitted when an anonymous @proto precedes the directive (see below).

`@proto` — embedded schema

Carries the protobuf schema for the document's payload, making PXF self-describing. Four body shapes are lexically distinguished:

proto_directive       = '@proto' , proto_body ;
proto_body            = proto_anonymous_body
                      | proto_named_body
                      | proto_source_body
                      | proto_descriptor_body ;

proto_anonymous_body  = '{' , ?protobuf message body? , '}' ;
proto_named_body      = identifier , '{' , ?protobuf message body? , '}' ;
proto_source_body     = triple_string ;
proto_descriptor_body = bytes ;

Shape	Body interpretation
Anonymous	Message body source; binds to the next typeless directive in document order.
Named	Message body source; registers the message under the supplied dotted name.
Source	Triple-quoted full `.proto` file source.
Descriptor	Base64-encoded `google.protobuf.FileDescriptorSet`.

// Named — schema travels with the data:
@proto trades.v1.Trade {
  string symbol = 1;
  double price  = 2;
  int64  qty    = 3;
}
@dataset trades.v1.Trade ( symbol, price, qty )
( "AAPL", 188.42, 100 )

The contents of brace-bounded @proto bodies are protobuf source (or descriptor bytes) and are NOT decoded as PXF — the parser captures the raw bytes between the matching braces and hands them to a downstream protobuf consumer.

Reserved directive names

Thirteen names are reserved by the spec and forbidden as directive_name productions. Four are value keywords that the lexer already routes to their value form (null, true, false); three have their own production (type, dataset, proto); entry is the registered named-directive shape; and six are reserved for future allocation so applications cannot squat names before the spec defines them.

directive_name = identifier - ( 'type'        | 'dataset'    | 'proto'
                              | 'entry'       | 'table'      | 'datasource'
                              | 'view'        | 'procedure'  | 'function'
                              | 'permissions' | 'null'       | 'true'
                              | 'false' ) ;

Entries

An entry inside the body is either a field assignment / block (the message form) or a map entry (the map<K, V> form). The top level of a document permits only field entries; the map form is reserved for nested blocks where the surrounding field's schema disambiguates.

entry         = field_entry | map_entry ;

field_entry   = identifier , ( assignment_tail | block_tail ) ;
map_entry     = map_key , map_tail ;

assignment_tail = '=' , value ;
map_tail        = ':' , value ;
block_tail      = '{' , { entry } , '}' ;

map_key       = identifier | string | integer ;

Values

Values are scalars, lists, or block values. Lists accept comma- or newline-separated elements and may freely mix the two; the comma is consumed if present.

value       = string
            | integer
            | float
            | bool
            | null
            | bytes
            | timestamp
            | duration
            | identifier
            | list
            | block_value ;

list        = '[' , [ value , { [ ',' ] , value } ] , ']' ;
block_value = '{' , { entry } , '}' ;

Identifiers

Identifiers carry enum values, message-type names, and bare keys. They begin with a letter or underscore and may contain dots, which is useful for fully-qualified type names like infra.v1.ServerConfig.

identifier  = ident_start , { ident_part } ;
ident_start = letter | '_' ;
ident_part  = letter | digit | '_' | '.' ;

bool        = 'true' | 'false' ;
null        = 'null' ;

A protobuf schema bound to PXF MUST NOT declare a message field, oneof, or enum value whose name case-sensitively equals null, true, or false: the lexer always routes such names to their value branch, so the declared element would be unreachable from PXF surface syntax. pxf lint surfaces these violations.

Numbers

Decimal integers and IEEE-754 floats. Floats accept either a decimal point with optional exponent, or an exponent alone.

integer  = [ '-' ] , digit , { digit } ;

float    = [ '-' ] , digit , { digit } ,
           ( '.' , { digit } , [ exponent ]
           | exponent ) ;

exponent = ( 'e' | 'E' ) , [ '+' | '-' ] , digit , { digit } ;

Timestamps & durations

The lexer recognizes a four-digit year followed by - as an RFC 3339 timestamp, and a digit run followed by a time unit as a Go-style duration. Negative integers and identifiers that begin with a letter take precedence over those forms.

(* RFC 3339 date-time. e.g. 2024-01-15T10:30:00Z, 2024-01-15T10:30:00.123456789+02:00 *)
timestamp        = ?RFC 3339 date-time? ;

(* Go time.ParseDuration. e.g. 30s, 1h30m, 500ms, 1.5h *)
duration         = duration_segment , { duration_segment } ;
duration_segment = digit , { digit } ,
                   [ '.' , digit , { digit } ] , time_unit ;
time_unit        = 'ns' | 'us' | 'µs' | 'ms' | 's' | 'm' | 'h' ;

Strings

Single-quoted simple strings honor C-style escapes plus 2-digit hex, 3-digit octal, and 4/8-digit Unicode escapes. Triple-quoted strings preserve raw content with no escape interpretation; the leading newline is stripped, and the closing line's indent is removed from each preceding line.

string        = simple_string | triple_string ;

simple_string = '"' , { string_char | escape_seq } , '"' ;
triple_string = '"""' , ?any text not containing """? , '"""' ;

escape_seq    = '\' , ( simple_escape
                          | hex_escape
                          | octal_escape
                          | unicode_4_escape
                          | unicode_8_escape ) ;

Bytes

Byte literals carry standard or raw base64 with optional padding. Backslashes inside b"…" are not interpreted.

bytes       = 'b' , '"' , { base64_char } , '"' ;
base64_char = letter | digit | '+' | '/' | '=' ;

Comments

Three flavors, freely mixed. Block comments do not nest.

comment       = line_comment | block_comment ;
line_comment  = ( '#' | '//' ) , { ?any byte except LF? } ;
block_comment = '/*' , { ?any byte? } , '*/' ;

Full railroad diagram

The full diagram covers every production above and a few lexical helpers (character classes, hex/octal digits). It's tall; scroll inside the frame, or open it in a new tab.

Concrete syntax. v1.0