Data Systems

Avro and the Schema Registry

Avro pairs compact binary data with an explicit schema, and a schema registry governs how those schemas may change across a streaming pipeline.

Schema-carrying data

Avro is a serialization format in which data is always accompanied by a schema describing its fields and types. The schema is not embedded in every record, which would be wasteful, but is referenced so that any reader can resolve the bytes. This separation makes Avro compact on the wire while keeping the data self-describing through its schema.

Schema resolution

Kronos motion — data assimilation

Avro's defining feature is schema resolution: a reader can use a schema different from the one the writer used, and Avro reconciles them field by field. If the reader's schema adds a field with a default, missing values are filled; if the writer added a field the reader lacks, it is ignored. These rules are exactly what make backward and forward compatibility mechanical rather than manual.

The schema registry

In a streaming system, embedding a full schema in every message is still too heavy. A schema registry stores schemas centrally and assigns each a small integer id. A producer registers its schema, gets an id, and prefixes each message with just that id. A consumer reads the id and fetches the matching schema. Messages stay tiny while remaining fully resolvable.

Compatibility enforcement

The registry checks each new schema against a configured compatibility mode
Backward mode rejects changes that would break existing readers
Forward mode rejects changes that would break existing writers
Full mode enforces both directions
An incompatible registration fails before any bad data is produced

Why this matters operationally

Without a registry, a producer can deploy a breaking schema change and silently corrupt every downstream consumer. With one, the breaking change is rejected at registration time, turning a production incident into a caught error at deploy time. This is the enforcement mechanism behind schema evolution as a governed process. See schema evolution and event streaming.