Serialization: Turning Structures into Bytes

Serialization encodes in-memory data structures into bytes for storage or transport; deserialization reconstructs them.

Crossing the boundary

Data in a running program lives as structured objects in memory. To store it, send it over a network, or hand it to another program, it must be turned into a flat sequence of bytes, a process called serialization. Deserialization reverses it, reconstructing the structure. Every pipeline, message, and file crosses this boundary, so the encoding chosen shapes size, speed, and compatibility.

Text versus binary encodings

• Text (JSON, XML, YAML): human-readable, self-describing, universal, but verbose and slower to parse.
• Binary (Protobuf, Avro, MessagePack): compact and fast, needs a schema or library to read.
• Each trades transparency against efficiency.

Schema-based formats

Formats such as Protocol Buffers and Avro use a schema that defines the message structure. The schema enables compact encoding (field names need not be repeated in every record) and safe evolution: rules govern how a schema can change while old readers still work. This makes them well suited to long-lived data and to streaming between systems. See schemas and contracts.

Forward and backward compatibility

In a living system, producers and consumers upgrade at different times, so serialized data must survive version mismatches. Backward compatibility lets new code read old data; forward compatibility lets old code read new data by ignoring unknown fields. Designing for both is what keeps a distributed data system from breaking on every change.

Choosing for the job

Use text formats where transparency and tooling matter, such as configuration and small human-inspected exports. Use schema-based binary formats for high-volume streams and durable records where size and evolution matter. For large numeric arrays, dedicated scientific containers like HDF5 serialize structured data more efficiently than general formats. Matching the format to the data is a small decision that compounds across a whole program.