Twilic — Weaving Repeated JSON into One Wire
- Rust
- TypeScript
- MessagePack
- WebAssembly
- N-API
Twilic is a documentation-first specification and multi-language implementation for a compact binary format aimed at structured data. The name comes from Old English twilic, the root of modern twill: repeated threads woven into one fabric—same idea for repeated object shapes, keys, and strings in a single wire representation.
I am building Twilic because JSON and MessagePack are easy for one-off objects, but the same schema sent hundreds of times still pays for field names and repeated strings on every row. Protobuf fixes that with a fixed schema; Twilic targets the middle ground—schema-less to start, then automatic interning and optional bound profiles when you know the shape.
What it does
- Self-describing encode/decode without a upfront schema, like MessagePack
- Learned compression via
shape_def,key_ref, andstr_refwhen structure repeats (v2 wire in SPEC.md) - Bound profile (
encodeWithSchema) to omit field names when a schema is known - Batch (row/column) and session patch in the same format family—send deltas instead of full JSON every tick
Reference implementations live in twilic-rust, @twilic/core (N-API + WASM), twilic-go, and twilic-zig. Hono middleware is available for Content-Type-aware bodies.
Benchmark
Sizes below use the same fixtures as twilic/benchmark (local @twilic/core via N-API on Node 24). Throughput is in the same order of magnitude as MessagePack for encode/decode; the main win is wire size on repeated data.
| Payload | Twilic | MessagePack | CBOR | BSON | vs MessagePack |
|---|---|---|---|---|---|
| Single record | 140 B | 140 B | 144 B | 198 B | 0% |
| 256 homogeneous records | 5,316 B | 19,505 B | 20,633 B | 28,241 B | 72.8% smaller |
| 256 mixed records | 9,799 B | 20,893 B | 22,204 B | 32,956 B | 53.1% smaller |
| Session patch (first) | 89 B | 89 B | 93 B | 120 B | 0% |
On a single small object, Twilic is designed not to be worse than MessagePack. On a 256-record homogeneous batch, encoded size drops to roughly one quarter of MessagePack in this harness—the case where key names and repeated strings dominate.
Reproduce locally:
git clone https://github.com/twilic/benchmark.git
cd benchmark
pnpm install
pnpm --dir ../twilic-js build
pnpm bench -- --twilic-vs-msgpack-only
Compare formats in the browser via the playground. A longer write-up (in Japanese) is on Zenn.
Good fit / poor fit
Good fit: telemetry or log batches, WebSocket events with stable shape, tabular columns (userId, active, country), edge-to-cloud links where bandwidth matters, teams already on MessagePack who only struggle with array batches.
Poor fit (explicit non-goals): replacing every JSON protocol, defining application semantics, or optimizing streams of tiny one-off payloads where MessagePack is already enough.
Links
- Specification: github.com/twilic/twilic
- Benchmark harness: github.com/twilic/benchmark
- npm: @twilic/core