zidl Runtime + Benchmarks (Phases 1 and 2) Implementation Plan

Historical document. Written on 2026-11-12, when the project was called zidl (later Graven). It is the plan that built the first runtime and the benchmarks against FlatBuffers and Cap'n Proto. Paths, names and some APIs changed afterwards; the normative text is the format specification.

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: A Rust runtime for the zidl format (zero-copy reads, forward writing with a pool of BytesMut), hand-written "generated" code for 4 scenarios, evolution tests, and benchmarks against FlatBuffers and Cap'n Proto.

Architecture: A Cargo workspace with zidl (runtime), zidl-layout (the layout algorithm, reused by the compiler in phase 3), zidl-schemas (the code the compiler would generate, written by hand) and zidl-bench (criterion + flatbuffers + capnp). The format is validated by benchmark before investing in the compiler.

Tech Stack: Rust 2024 (rustc 1.94), bytes, proptest, criterion, flatbuffers, capnp/capnpc. External tools: capnp 1.5 (installed), flatc (not installed — sudo pacman -S flatbuffers; the generated code is committed, so only whoever regenerates it needs it).

Spec: design/2026-11-12-zerocopy-idl-design.md

Global Constraints

• Little-endian everywhere; no alignment padding; reads via from_le_bytes over slices.
• Table = [u16 fixed_len][fixed part]. Slot offsets are relative to the start of the fixed part (the byte after the u16).
• Pointer = i32 LE, relative to the absolute position of the slot itself (target - slot_pos); 0 = absent; may be negative.
• string/bytes = [u32 len][bytes]; [scalar|struct] = [u32 count][elements]; [table] = [u32 count][u16 elem_fixed_len][count × fixed part] (elements without a header of their own); [string]/[bytes] = [u32 count][count × i32].
• Union slot = [u16 tag][i32 offset] (6 bytes); tag 0 = absent; variants are tables only.
• Every scalar field (including enums and inline structs) has 1 presence bit; bool has 2 bits in the same byte (presence = bit k, value = bit k+1). 1 = present. A new fixed part = all zeros.
• Read rule: slot (or bit byte) outside fixed_len ⇒ absent; slot partially inside ⇒ Err(Truncated); otherwise the bit/offset/tag decides.
• Reading a malformed buffer: never a panic, never UB — always Err. No unsafe in the runtime, except str_unchecked, explicitly an unsafe fn.
• Hot write path with a warm pool: zero allocations per message.
• zidl builds with --no-default-features (no_std + alloc); the std feature (default) enables SimplePool.
• Message ≤ i32::MAX bytes; exceeding it ⇒ Err(TooLarge) on write (not a panic).
• verify defaults: max_depth = 64, budget of bytes visited = max(buf.len() * 8, 1 MiB).

Review Focus

1. Random/truncated bytes / an offset pointing outside or before 0 / len = u32::MAX → Err, never a panic (proptest in Task 5; fixed cases in Task 2).
2. An abandoned streaming list (drop without close, or ? exiting halfway) and a Builder dropped without finish → no panic, buffers return to the pool (Tasks 3 and 4).
3. Bytes of a previous message still alive while the same pool builds the next one → the old message stays intact (Task 3).
4. An empty string present vs absent; an empty list present vs absent → distinguishable (Task 3).
5. A streaming list element with a string + a nested table + a nested streaming list → correct offsets after the fix-up (Task 4).

File Structure

Cargo.toml                         # [workspace] members = ["crates/*"]; remove src/ and the root [package]
crates/zidl/Cargo.toml
crates/zidl/src/lib.rs             # #![no_std], re-exports
crates/zidl/src/error.rs           # Error
crates/zidl/src/le.rs              # trait Le
crates/zidl/src/read.rs            # TableRef, Bit, read lists
crates/zidl/src/pool.rs            # BufPool, SimplePool
crates/zidl/src/write.rs           # Builder, TableW, TableListW, StreamListW, ScalarListW
crates/zidl/src/verify.rs          # VerifyCtx
crates/zidl/tests/fuzz_props.rs    # proptest: arbitrary bytes
crates/zidl-layout/src/lib.rs      # FieldKind, compute_layout
crates/zidl-schemas/schemas/*.zidl # bank, game, iot, stream (source for phase 3)
crates/zidl-schemas/src/{bank,game,iot,stream}.rs
crates/zidl-schemas/src/evo/{bank_v1,bank_v2,bank_v3}.rs
crates/zidl-schemas/tests/{layout_consts,roundtrip,evolution}.rs
crates/zidl-bench/schemas/{bank,game,iot,stream}.{fbs,capnp}
crates/zidl-bench/src/fbs_gen/*.rs # generated by flatc, committed
crates/zidl-bench/build.rs         # capnpc
crates/zidl-bench/src/{lib,data,zidl_impl,fbs_impl,capnp_impl}.rs
crates/zidl-bench/src/bin/{sizes,allocs}.rs
crates/zidl-bench/benches/codec.rs
docs/benchmarks/2026-11-12-baseline.md

Task 1: Workspace, Le, Error, reading scalars and presence

Files:

• Modify: Cargo.toml (becomes a workspace; delete src/lib.rs)
• Create: crates/zidl/{Cargo.toml,src/lib.rs,src/error.rs,src/le.rs,src/read.rs}

Interfaces:

• Produces:

  • pub enum Error { OutOfBounds, Truncated, BadOffset, InvalidUtf8, Missing { field: &'static str }, DepthExceeded, BudgetExceeded, TooLarge } (derive Debug, Clone, Copy, PartialEq, Eq)
  • pub trait Le: Copy + Default { const SIZE: usize; fn read_le(b: &[u8]) -> Self; fn write_le(self, out: &mut [u8]); } for u8..u64, i8..i64, u128, i128, f32, f64 (b.len() == SIZE is a precondition; the caller slices).
  • #[derive(Clone, Copy)] pub struct Bit { pub byte: u16, pub shift: u8 }
  • #[derive(Clone, Copy)] pub struct TableRef<'a> { buf: &'a [u8], fixed_start: usize, fixed_len: u16 }
    • TableRef::at(buf: &'a [u8], pos: usize) -> Result<Self, Error> — reads the header at pos; requires pos + 2 + fixed_len <= buf.len().
    • TableRef::inline(buf: &'a [u8], fixed_start: usize, fixed_len: u16) -> Result<Self, Error> — a list element (no header).
    • fn buf(&self) -> &'a [u8], fn fixed_len(&self) -> u16
    • fn bit(&self, b: Bit) -> bool — byte outside fixed_len ⇒ false.
    • fn slot_state(&self, off: u16, size: u16) -> Result<bool, Error> — fully inside ⇒ Ok(true), fully outside ⇒ Ok(false), partial ⇒ Err(Truncated).
    • fn scalar<T: Le>(&self, off: u16, presence: Bit) -> Result<Option<T>, Error> — applies the read rule.
    • fn scalar_or<T: Le>(&self, off: u16, presence: Bit, default: T) -> T — for already validated required fields; never panics (outside ⇒ default).
    • fn bool(&self, presence: Bit) -> Option<bool> — value at shift + 1.

• [ ] Step 1: Workspace: root Cargo.toml with [workspace] resolver = "3" members = ["crates/*"] and [workspace.package] edition = "2024"; crates/zidl with bytes = { version = "1", default-features = false }, features default = ["std"], std = ["bytes/std"]; lib.rs with #![cfg_attr(not(feature = "std"), no_std)] extern crate alloc;.

• [ ] Step 2: Failing tests in read.rs:

// table: fixed_len=7: [bits][u32 a @1][u16 b @5]; presence a=bit0, b=bit1, bool=bits2/3
fn sample() -> Vec<u8> { vec![7,0, 0b0000_1101, 42,0,0,0, 0,0] } // a present=42, b absent, bool present=true
#[test] fn reads_present_scalar() { let t = TableRef::at(&sample(),0).unwrap();
  assert_eq!(t.scalar::<u32>(1, Bit{byte:0,shift:0}).unwrap(), Some(42)); }
#[test] fn absent_bit_is_none() { /* b at off 5, Bit{0,1} */ assert_eq!(.., None); }
#[test] fn bool_two_bits() { assert_eq!(t.bool(Bit{byte:0,shift:2}), Some(true)); }
#[test] fn slot_beyond_fixed_len_is_none() { /* off 7 size 4 */ assert_eq!(t.scalar::<u32>(7, Bit{byte:0,shift:4}).unwrap(), None); }
#[test] fn bit_byte_beyond_fixed_len_is_none() { /* Bit{byte:9,..} */ }
#[test] fn partial_slot_is_truncated() { /* off 5 size 4 */ assert_eq!(t.slot_state(5,4), Err(Error::Truncated)); }
#[test] fn header_past_end_is_err() { assert_eq!(TableRef::at(&[9,0,1],0).err(), Some(Error::OutOfBounds)); }
#[test] fn le_roundtrip_all_types() { /* write_le→read_le for each type, incl. f64::NAN bits, i128::MIN */ }

• [ ] Step 3: Run cargo test -p zidl → FAIL (does not compile).
• [ ] Step 4: Implement error.rs, le.rs (a macro for the types), read.rs per the Interfaces. All indexing via get(..) — no [] that could panic.
• [ ] Step 5: cargo test -p zidl → PASS; cargo build -p zidl --no-default-features → OK.
• [ ] Step 6: Commit feat(zidl): workspace, Le, Error and reading of scalars/presence.

Task 2: Reading variable-size data

Files: Modify crates/zidl/src/read.rs

Interfaces:

• Consumes: TableRef, Error (Task 1)

• Produces (methods of TableRef<'a>):

  • fn ptr(&self, off: u16) -> Result<Option<usize>, Error> — slot outside ⇒ None; 0 ⇒ None; absolute target slot_abs + rel outside [0, buf.len()) ⇒ Err(BadOffset). Arithmetic in i64.
  • fn bytes(&self, off: u16) -> Result<Option<&'a [u8]>, Error>
  • fn str(&self, off: u16) -> Result<Option<&'a str>, Error> (validates UTF-8); unsafe fn str_unchecked(&self, off: u16) -> Result<Option<&'a str>, Error>
  • fn table(&self, off: u16) -> Result<Option<TableRef<'a>>, Error>
  • fn union(&self, off: u16) -> Result<Option<(u16, TableRef<'a>)>, Error> — tag at off, pointer at off+2; tag 0 ⇒ None.
  • fn scalars<T: Le>(&self, off: u16) -> Result<Option<ScalarList<'a, T>>, Error>; ScalarList { len(), get(i) -> Option<T>, iter() } (also used for [struct] with T = a generated struct implementing Le).
  • fn tables(&self, off: u16) -> Result<Option<TableList<'a>>, Error>; TableList { len(), elem_fixed_len(), get(i) -> Result<TableRef<'a>, Error>, iter() }
  • fn strs(&self, off: u16) -> Result<Option<StrList<'a>>, Error>; StrList { len(), get(i) -> Result<&'a str, Error> }
  • Lengths: len * elem_size computed in usize with checked_mul; overflow ⇒ Err(OutOfBounds).

• [ ] Step 1: Failing tests (buffers built by hand with a helper fn table_with_ptr(rel: i32, tail: &[u8]) -> Vec<u8>):

  • string_forward_offset → Some("oi")
  • negative_offset_into_buffer → reads a target before the slot
  • offset_before_zero_is_bad_offset, offset_past_end_is_bad_offset
  • len_u32_max_is_out_of_bounds (no panic, no allocation)
  • invalid_utf8_is_err
  • zero_offset_is_none
  • table_list_stride_uses_elem_fixed_len (two elements with elem_fixed_len=5; a reader with a field at off 0 reads both)
  • union_unknown_tag_still_returns_tag (tag 99 ⇒ Some((99, t)))

• [ ] Step 2: cargo test -p zidl → FAIL.

• [ ] Step 3: Implement.

• [ ] Step 4: cargo test -p zidl → PASS.

• [ ] Step 5: Commit feat(zidl): reading strings, bytes, tables, unions and lists.

Task 3: Pool, Builder and TableW (without table lists)

Files: Create crates/zidl/src/pool.rs, crates/zidl/src/write.rs

Interfaces:

• Consumes: Le, Bit, Error; TableRef only in the tests.

• Produces:

  • pub trait BufPool { fn take(&self) -> BytesMut; fn give(&self, b: BytesMut); }
  • #[cfg(feature="std")] pub struct SimplePool — SimplePool::new(initial_capacity: usize); take reuses (clear() + reserve) or creates; give stores.
  • pub struct Builder<'p> { pool: &'p dyn BufPool, buf: BytesMut }
    • Builder::new(pool: &'p dyn BufPool) -> Self
    • fn root(&mut self, fixed_len: u16) -> TableW<'_> — writes the header + fixed_len zeros at 0.
    • fn finish(mut self) -> Result<Bytes, Error> — buf.split().freeze(), returns the rest of the BytesMut to the pool (the space is reclaimed by reserve when the Bytes dies); Err(TooLarge) if len > i32::MAX.
    • impl Drop — if not finished, returns buf to the pool.
  • pub struct TableW<'b> (internals free; it must write the fixed part into a "fixed" buffer that can be the main one or the auxiliary buffer of a streaming list, and variable-size data always into the main one). Methods:
    • fn set<T: Le>(&mut self, off: u16, presence: Bit, v: T)
    • fn set_bool(&mut self, presence: Bit, v: bool)
    • fn set_str(&mut self, off: u16, s: &str), fn set_bytes(&mut self, off: u16, b: &[u8])
    • fn set_scalars<T: Le>(&mut self, off: u16, v: &[T])
    • fn set_strs<'s>(&mut self, off: u16, v: impl ExactSizeIterator<Item = &'s str>)
    • fn table(&mut self, off: u16, fixed_len: u16) -> TableW<'_> — a child in the main buffer.
    • fn union(&mut self, off: u16, tag: u16, fixed_len: u16) -> TableW<'_>
  • Writing a pointer: value = target_pos - slot_pos, where slot_pos is the position of the slot in the buffer that holds the fixed part (in the auxiliary buffer, it is the index in the auxiliary buffer — corrected by the fix-up in Task 4).

• [ ] Step 1: Failing tests (round trips via TableRef):

  • roundtrip_scalars_bool_str_bytes
  • nested_table_and_union_roundtrip
  • empty_string_present_vs_absent: set_str(off, "") ⇒ Some(""); not setting ⇒ None. Same for set_scalars::<u32>(off, &[]) ⇒ Some(len 0).
  • old_bytes_survive_pool_reuse: SimplePool::new(64); build msg1 → Bytes b1 (keep it); build a different msg2 from the same pool; assert_eq!(b1, expected1).
  • dropped_builder_returns_buffer: a pool counting give calls (a test pool implementing BufPool); a Builder dropped without finish ⇒ give called once.
  • steady_state_zero_allocs: a counting global allocator in tests/allocs.rs (its own test file, so the #[global_allocator] does not affect the other tests); warm up 10 messages with the Bytes dropped; measure the 11th build+finish+drop ⇒ 0 allocations.

• [ ] Step 2: cargo test -p zidl → FAIL.

• [ ] Step 3: Implement pool.rs and write.rs.

• [ ] Step 4: cargo test -p zidl → PASS.

• [ ] Step 5: Commit feat(zidl): pool, Builder and forward-only TableW.

Task 4: Table lists (known count and streaming) with fix-up

Files: Modify crates/zidl/src/write.rs

Interfaces:

• Consumes: TableW, BufPool (Task 3)

• Produces (methods of TableW<'b>):

  • fn tables(&mut self, off: u16, n: u32, elem_fixed_len: u16) -> TableListW<'_> — reserves the header + n zeroed fixed parts in the main buffer. TableListW::push(&mut self) -> TableW<'_> (a programming panic if it exceeds n is acceptable: that is a writer bug, not external data). On Drop: corrects count to the real one (the rest becomes dead bytes).
  • fn tables_streaming(&mut self, off: u16, elem_fixed_len: u16, ptr_slots: &'static [u16]) -> StreamListW<'_> — ptr_slots = the offsets (in the fixed part) of every i32 pointer of the element, including union_off + 2. Takes an auxiliary buffer from the pool. StreamListW::push(&mut self) -> TableW<'_> (fixed part in the auxiliary buffer). close(self) and Drop do:
    1. append [u32 count][u16 elem_fixed_len] to the main buffer; B = the next position;
    2. extend_from_slice(aux) (one memcpy);
    3. for each element and each ptr_slot with a value ≠ 0: value -= B (in i64, checked);
    4. patch the parent's slot; aux.clear() and pool.give(aux).
  • fn scalars_streaming<T: Le>(&mut self, off: u16) -> ScalarListW<'_, T> — appended directly to the main buffer; push(v: T); Drop corrects count. It cannot coexist with another open variable-size datum (guaranteed by &mut).

• [ ] Step 1: Failing tests:

  • known_count_list_roundtrip (3 elements, each with a string)
  • known_count_closed_short_fixes_count (n=5, push 2 ⇒ len()==2)
  • streaming_list_with_strings_offsets_fixed — elements with a memo string; negative offsets; read them all.
  • streaming_elem_with_nested_table_and_union (Review Focus 5)
  • streaming_inside_streaming — an element of the outer list opens another tables_streaming; read everything.
  • streaming_list_dropped_without_close — exiting via ? halfway ⇒ finish() still produces a readable buffer; the auxiliary buffer is returned (counting give calls).
  • streaming_scalars_roundtrip

• [ ] Step 2: cargo test -p zidl → FAIL.

• [ ] Step 3: Implement.

• [ ] Step 4: cargo test -p zidl → PASS.

• [ ] Step 5: Commit feat(zidl): known and streaming table lists with fix-up.

Task 5: VerifyCtx and properties with arbitrary bytes

Files: Create crates/zidl/src/verify.rs, crates/zidl/tests/fuzz_props.rs

Interfaces:

• Produces:
  • pub struct VerifyCtx { depth: u32, max_depth: u32, budget: usize }
  • VerifyCtx::new(buf_len: usize) -> Self (the defaults from the Global Constraints); VerifyCtx::with_limits(max_depth: u32, budget: usize)
  • fn enter(&mut self) -> Result<(), Error> / fn leave(&mut self) (depth); fn charge(&mut self, bytes: usize) -> Result<(), Error> (budget).
  • Generated code (Task 7) implements fn verify(&self, ctx: &mut VerifyCtx) -> Result<(), Error> with these.
• [ ] Step 1: Failing tests:
  • depth_limit_hits_on_self_cycle — a table whose table field points to itself (rel = -(slot_pos)); a recursive test verifier written in the test itself ⇒ Err(DepthExceeded).
  • budget_limit_hits_on_shared_subtree — with_limits(64, 100) with a list of 50 offsets to the same 10-byte string ⇒ Err(BudgetExceeded).
  • proptest arbitrary_bytes_never_panic: for an arbitrary Vec<u8> (0..512), call TableRef::at(&b,0) and, if Ok, every read method with off in 0..32 and arbitrary Bits; the test only requires no panic.
• [ ] Step 2: cargo test -p zidl → FAIL.
• [ ] Step 3: Implement verify.rs; fix any panic the proptest finds in read.rs.
• [ ] Step 4: cargo test -p zidl → PASS (proptest with cases = 4096).
• [ ] Step 5: Commit feat(zidl): VerifyCtx and robustness tests.

Task 6: zidl-layout — the layout algorithm

Files: Create crates/zidl-layout/{Cargo.toml,src/lib.rs}

Interfaces:

• Produces:
  • pub enum FieldKind { Scalar { size: u16 }, Bool, Ptr, Union } (inline enum/struct = Scalar with its size; Ptr = string/bytes/list/table, 4 bytes; Union = 6 bytes)
  • pub struct FieldLayout { pub slot: Option<u16>, pub presence: Option<(u16, u8)> } ((byte, shift))
  • pub struct TableLayout { pub fixed_len: u16, pub fields: Vec<FieldLayout> }
  • pub fn compute_layout(fields: &[FieldKind]) -> TableLayout — fields in ID order, including the reserved ones with their original type.
• Algorithm (fixes spec §4.3):

cursor = 0; open = None  // (byte_pos, next_shift)
for f in fields:
  need = match f { Scalar => 1, Bool => 2, Ptr | Union => 0 }
  if need > 0:
    if open is Some((p, s)) and s + need <= 8: presence = (p, s); open = (p, s+need)
    else: presence = (cursor, 0); open = (cursor, need); cursor += 1
  if f is not Bool: slot = cursor; cursor += size(f)
fixed_len = cursor

• [ ] Step 1: Failing tests:
  • telemetry_layout: [Scalar 4, Scalar 8, Scalar 4, Scalar 4, Scalar 1] ⇒ fixed_len == 22, slots [1, 5, 13, 17, 21], presence (0,0)..(0,4).
  • bool_does_not_split_across_bytes: 7× Scalar 1 + Bool ⇒ the bool in a new byte (cursor, 0).
  • ptr_has_no_bit: [Ptr, Scalar 2] ⇒ ptr slot 0, bits at 4, scalar slot 5.
  • append_is_stable: the layout of v1 is a prefix of v1 + [Scalar 8] (all v1 fields equal).
• [ ] Step 2: cargo test -p zidl-layout → FAIL.
• [ ] Step 3: Implement.
• [ ] Step 4: cargo test -p zidl-layout → PASS.
• [ ] Step 5: Commit feat(zidl-layout): stable layout algorithm.

Task 7: zidl-schemas — hand-written generated code for the 4 scenarios

Files:

• Create crates/zidl-schemas/schemas/{bank,game,iot,stream}.zidl, crates/zidl-schemas/src/{lib,bank,game,iot,stream}.rs
• Test crates/zidl-schemas/tests/{layout_consts,roundtrip}.rs

Schemas (syntax of spec §6; IDs in order):

// bank.zidl
enum Currency : u16 { BRL = 0, USD = 1, EUR = 2 }
table Entry { account @0: u64; amount @1: i64; memo @2: string?; }
table Transaction { id @0: u64; ts @1: u64; currency @2: Currency; from_shard @3: u32;
                    to_shard @4: u32; entries @5: [Entry]; note @6: string?; }
// game.zidl
struct Vec3 { x: f32, y: f32, z: f32 }
table Entity { id @0: u32; pos @1: Vec3; vel @2: Vec3; hp @3: u16; alive @4: bool; }
table Frame { tick @0: u64; entities @1: [Entity]; }
// iot.zidl
table Telemetry { device @0: u32; ts @1: u64; temp @2: f32; humidity @3: f32?; battery @4: u8; }
// stream.zidl
table Chunk { stream_id @0: u64; seq @1: u32; data @2: bytes; }

Interfaces (the pattern the phase 3 compiler will reproduce — fixed here):

• Per table X: a module with pub const FIXED_LEN: u16, a slot/Bit const per field, pub(crate) const PTR_SLOTS: &[u16].

• pub struct X<'a>(TableRef<'a>):

  • X::read(buf: &'a [u8]) -> Result<Self, Error> (root at 0) and X::from_ref(t: TableRef<'a>) -> Result<Self, Error> — both validate required fields ⇒ Err(Missing { field: "name" }).
  • required scalar: fn id(&self) -> u64; optional: fn humidity(&self) -> Option<f32>; enum: fn currency(&self) -> Currency with Currency::Unknown(u16); optional string: fn note(&self) -> Result<Option<&'a str>, Error>; required list: fn entries(&self) -> Result<impl Iterator<Item = Result<Entry<'a>, Error>> + ExactSizeIterator, Error> (absent ⇒ empty); bytes: fn data(&self) -> Result<&'a [u8], Error>.
  • fn verify(&self, ctx: &mut VerifyCtx) -> Result<(), Error>.

• pub struct XW<'b>(TableW<'b>) with XW::root(b: &'b mut Builder) -> Self, setters fn id(&mut self, v: u64) -> &mut Self, and for lists fn entries(&mut self, n: u32) -> ListW<'_, EntryW> and fn entries_streaming(&mut self) -> StreamW<'_, EntryW> (thin wrappers over TableListW/StreamListW that return EntryW).

• Vec3 implements Le (12 bytes).

• Required: every field without ?.

• [ ] Step 1: Failing tests:

  • layout_consts.rs: for each table, compute_layout(&[...kinds]) ⇒ FIXED_LEN, every slot and every Bit equal to the hand-written constants; Telemetry::FIXED_LEN == 22.
  • roundtrip.rs: per scenario, build with SimplePool, read and compare every field; bank built once with entries(n) and once with entries_streaming() ⇒ equal reads; verify ⇒ Ok.
  • missing_required_is_err: a Transaction without to_shard ⇒ Err(Missing { field: "to_shard" }).
  • unknown_enum_value: write currency with raw 7 via TableW::set::<u16> ⇒ Currency::Unknown(7).

• [ ] Step 2: cargo test -p zidl-schemas → FAIL.

• [ ] Step 3: Write the modules.

• [ ] Step 4: cargo test -p zidl-schemas → PASS.

• [ ] Step 5: Commit feat(zidl-schemas): hand-written code for bank/game/iot/stream.

Task 8: Evolution matrix

Files: Create crates/zidl-schemas/src/evo/{mod,bank_v1,bank_v2,bank_v3}.rs, crates/zidl-schemas/tests/evolution.rs

Versions (only Transaction changes; Entry is the same):

• v1 = bank.rs above (re-export or copy).

• v2: adds fee @7: i64?; note @6 becomes reserved (the v2 writer does not set it); from_shard @3 becomes u32?.

• v3: v2 + to_shard @4 becomes reserved (it was required in v1 ⇒ the future compiler only warns).

• [ ] Step 1: Failing tests:

  • v1_reads_v2: v1 reads a v2 message ⇒ Ok, note() == None, the extra bytes of fee ignored.
  • v2_reads_v1: fee() == None, from_shard() == Some(x).
  • v2_reads_v1_zero_is_present: v1 writes from_shard = 0 ⇒ v2 reads Some(0) (0 ≠ absent).
  • v1_reads_v3_missing_required: v1 reads a v3 message ⇒ Err(Missing { field: "to_shard" }) (never a silent 0).
  • v2_reads_v3: Err(Missing { field: "to_shard" }) too (v2 still requires it).
  • layouts_are_prefix_stable: compute_layout(v1) is a prefix of compute_layout(v2).

• [ ] Step 2: cargo test -p zidl-schemas --test evolution → FAIL.

• [ ] Step 3: Write the version modules.

• [ ] Step 4: → PASS.

• [ ] Step 5: Commit test(zidl-schemas): v1/v2/v3 evolution matrix.

Task 9: Equivalent implementations in FlatBuffers and Cap'n Proto

Files: Create crates/zidl-bench/{Cargo.toml,build.rs}, schemas/*.fbs, schemas/*.capnp, src/fbs_gen/*.rs, src/{lib,data,zidl_impl,fbs_impl,capnp_impl}.rs

Interfaces:

• data.rs: plain Rust types BankTx { id, ts, currency: u16, from_shard, to_shard, entries: Vec<(u64, i64, Option<String>)>, note: Option<String> }, Frame { tick, entities: Vec<Ent> }, Telemetry {..}, Chunk { stream_id, seq, data: Vec<u8> }; deterministic generators bank(seed: u64, n_entries: usize), frame(seed, n: usize), telemetry(seed), chunk(seed, size: usize).

• Each *_impl.rs exposes, per scenario (bank, frame, telemetry, chunk):

  • fn write_<s>(ctx: &mut Ctx, d: &<Data>) -> &[u8] — Ctx reused between iterations (zidl: SimplePool; fbs: FlatBufferBuilder with reset(); capnp: message::Builder with a reusable ScratchSpace/allocator). For zidl, write_bank uses entries_streaming (the hard case) and write_bank_known uses entries(n).
  • fn read_all_<s>(buf: &[u8]) -> u64 — a sum/xor of every field (strings: len + first byte) — an identical checksum across formats.
  • fn read_one_<s>(buf: &[u8]) -> u64 — a single field (to_shard, entities[n/2].hp, temp, seq).
  • Variants with verification: read_all_<s>_verified (fbs root::<T> vs root_unchecked; capnp with the default ReaderOptions vs traversal_limit_in_words: None; zidl verify + read vs read only).

• fbs strings: create_string in the order the backward builder requires (that is part of what is measured).

• build.rs: capnpc::CompilerCommand::new().src_prefix("schemas").file(...).run().

• fbs: flatc --rust -o src/fbs_gen schemas/*.fbs, output committed.

• [ ] Step 0: Ask the user to run ! sudo pacman -S flatbuffers (if flatc is still missing).

• [ ] Step 1: Failing test tests/equivalence.rs: for seeds 0..20 and each scenario, read_all of zidl, fbs and capnp give the same checksum as a checksum_<s>(&Data) computed directly from the struct; same for read_one.

• [ ] Step 2: cargo test -p zidl-bench → FAIL.

• [ ] Step 3: Write the schemas, generate code, implement.

• [ ] Step 4: → PASS.

• [ ] Step 5: Commit feat(zidl-bench): equivalent scenarios in zidl, flatbuffers and capnp.

Task 10: Benchmarks, sizes, allocations and report

Files: Create crates/zidl-bench/benches/codec.rs, src/bin/sizes.rs, src/bin/allocs.rs, docs/benchmarks/2026-11-12-baseline.md

Interfaces: Consumes everything from Task 9.

• [ ] Step 1: benches/codec.rs (criterion, harness = false): groups write/<scenario>, read_all/<scenario>, read_one/<scenario>, read_all_verified/<scenario>, each with zidl, fbs, capnp (and zidl_known for bank). Sizes: bank with 1, 10 and 100 entries; frame with 10 and 500 entities; chunk with 64 B and 64 KiB. Use black_box.
• [ ] Step 2: src/bin/sizes.rs prints a Markdown table: bytes per message per format and scenario, capnp packed and unpacked (capnp::serialize_packed).
• [ ] Step 3: src/bin/allocs.rs with a counting #[global_allocator]: after 100 warm-up iterations, allocations per message (write) per format/scenario.
• [ ] Step 4: Run cargo bench -p zidl-bench, cargo run --release -p zidl-bench --bin sizes, cargo run --release -p zidl-bench --bin allocs. Expected: everything runs; zidl with 0 allocations/message on write.
• [ ] Step 5: Write docs/benchmarks/2026-11-12-baseline.md: machine (lscpu | head, rustc), time tables (median), size, allocations, and for each winning criterion of spec §8 (win on writing, win on size, tie/win on reading, zero allocations) an honest verdict per scenario — including where we lose and the hypothesis for why (e.g. unaligned reads).
• [ ] Step 6: Commit bench: zidl vs flatbuffers vs capnp baseline.