Graven

zidl v1 — Format: envelope, Verified, spec and vectors (Plan 1 of 3)

Historical document. Written on 2026-11-12, when the project was called zidl (later Graven). It is the plan that froze format v1. Names and some details changed afterwards (for example, the magic became GRVN and namespaces no longer carry versions); the normative text is the format specification.

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Freeze format v1: new errors, CRC32C, the to_bytes/from_bytes envelope (with and without the Builder), an API without public unsafe (verified views), a normative spec and conformance vectors generated by a dynamic reader/writer independent of the generated code.

Architecture: Everything about the envelope and the CRC lives in the zidl runtime (no_std + alloc). The verified views are written by hand in zidl-schemas following the same pattern the compiler (Plan 2) will generate. Conformance lives in a new crate zidl-conformance with a dynamic interpreter (a schema described as data → read/write via TableRef/TableW + zidl-layout) that generates and checks the frozen vectors; Plan 2 checks the generated code against the same vectors.

Tech Stack: Rust 2024, bytes, crc32c 0.6.9 (only with the std feature), proptest, serde + serde_json (only in the conformance crate), criterion.

Spec: design/2026-11-12-zidl-v1-hardening-design.md (parts A, B, C and the review items of part E). Following plans: 2 = compiler + lockfile (part D); 3 = fuzzing, generated matrix, Miri, report (part E).

Global Constraints

Review Focus

  1. payload_len = u32::MAX (or any value that overflows 20 + N + 4 on 32 bits) ⇒ Incomplete, never a panic/overflow. → Task 3, decode_huge_len_is_incomplete.
  2. Several concatenated envelopes with a corrupt one in the middle ⇒ the iterator stops with ChecksumMismatch and does not silently "resynchronize". → Task 4, envelopes_stop_at_corruption.
  3. from_bytes with an intact envelope of another root type ⇒ SchemaMismatch before looking at the payload. → Task 5, from_bytes_wrong_schema.
  4. A verified view built from unverified bytes by safe code ⇒ does not compile. → Task 5, compile_fail doctest.
  5. An envelope with an empty payload (payload_len = 0) ⇒ decode accepts it, from_bytes returns OutOfBounds from reading the root, no panic. → Task 3, decode_empty_payload.

File Structure

File Responsibility
crates/zidl/src/error.rs (mod) new variants, #[non_exhaustive]
crates/zidl/src/crc.rs (new) crc32c(&[u8]) -> u32: software slicing-by-8; accelerated via a crate with std
crates/zidl/src/envelope.rs (new) constants, encode, decode, Envelope, envelopes(), schema_id()
crates/zidl/src/write.rs (mod) envelope mode in the Builder; list overflow
crates/zidl/src/read.rs (mod) str_unchecked becomes #[doc(hidden)]
crates/zidl/src/lib.rs (mod) #![deny(unsafe_code)], re-exports
crates/zidl-schemas/src/lib.rs (mod) FromVerified, VListR
crates/zidl-schemas/src/{bank,game,iot,stream}.rs (mod) SCHEMA_ID, verified(), from_bytes(), *Verified views
crates/zidl-bench/src/zidl_impl.rs, benches/codec.rs (mod) trusted via the view; envelope group
crates/zidl-conformance/ (new crate, publish = false) dynamic schema model, interpreter, vector generator, test
crates/zidl-conformance/schemas/conformance.zidl (new) coverage schema (the textual source of truth; Plan 2 compiles it)
testvectors/v1/ (new) frozen .bin + .json, README.md
docs/spec/zidl-format-v1.md (new) normative spec
docs/benchmarks/2026-11-12-envelope.md (new) cost of the envelope

Task 1: New errors and list overflow in the writer

Files:

Interfaces:

Task 2: CRC32C

Files:

Interfaces:

Task 3: Envelope encode / decode

Files:

Interfaces:

Task 4: Copy-free envelope in the Builder and iteration

Files:

Interfaces:

Task 5: Verified views and from_bytes (no public unsafe)

Files:

Decision (a deviation from the design's name): a generic Verified<T> in the runtime does not work: methods with trusted access would have to be impls on a foreign type (the orphan rule), and any public FromTable would allow building it without verifying. So each table gets a view XVerified<'a> with a private field, built only by verified()/from_bytes() of its own module. The only unsafe sits in the string accessors of those views (generated code), with // SAFETY: pointing to the verify() that built them.

Interfaces:

Task 6: The zidl-conformance crate — dynamic schema and interpreter

Files:

schemas/conformance.zidl (the textual source; model.rs describes it by hand and Plan 2 compiles it):

namespace conformance.v1;
enum Color : u8 { Red = 0, Green = 1, Blue = 2 }
struct Point { x: i32, y: i32 }
table Leaf { n @0: u32; s @1: string?; }
union Choice { Leaf @1: Leaf }
table Root {
  a_u8 @0: u8;           a_i16 @1: i16?;        a_u32 @2: u32 = 7;
  a_i64 @3: i64?;        a_u128 @4: u128?;      a_i128 @5: i128?;
  a_f32 @6: f32?;        a_f64 @7: f64?;
  flag @8: bool?;        flag2 @9: bool;
  color @10: Color?;     point @11: Point?;
  name @12: string?;     blob @13: bytes?;
  nums @14: [u16]?;      points @15: [Point]?;  leaves @16: [Leaf]?;
  names @17: [string]?;  blobs @18: [bytes]?;
  child @19: Leaf?;      choice @20: Choice?;
  @21: reserved;         // was `old @21: u32?`
}

Plus RootV2 = Root + extra @22: u64? (for the evolution vectors).

Interfaces:

Task 7: Frozen test vectors

Files:

Interfaces:

Minimum cases (file name = name):

Task 8: Normative v1 spec

Files:

Mandatory sections (normative, MUST/MUST NOT), describing the implemented behavior (check each rule against read.rs/write.rs/zidl-layout; differences from the original design go in a "Differences from the design" section at the end):

  1. Scope, conventions (LE, no alignment, i32 offsets relative to the slot, 0 = absent, the i32::MAX limit).
  2. Table: the u16 fixed_len header, the fixed part, the layout algorithm in pseudocode (a faithful copy of compute_layout), slot kinds.
  3. Presence and the 3-step read rule; required × optional × default; reserved.
  4. Variable-size data: string, bytes, [scalar], [struct], [table] (with elem_fixed_len), [string]/[bytes].
  5. Enum, union (tag 0 = absent, unknown = Unknown), frozen struct.
  6. Verification: what a reader MUST reject (the list = the Error variants except the envelope ones), default limits (depth 64, budget max(len*8, 1 MiB)), unreferenced bytes allowed.
  7. Envelope: layout table, schema_id (64-bit FNV-1a of the qualified name, fixed by the lockfile), CRC32C (polynomial, check value 0xE3069283), check order, Incomplete × ChecksumMismatch, concatenation.
  8. Schema compatibility (§6.1 of the design, normative).
  9. Conformance: an implementation conforms if it passes testvectors/v1/.

Task 9: Cost of the envelope and wrap-up

Files: