Data Lakes
A data lake stores raw data of any format at scale, deferring structure until the data is read rather than imposing it on write.
Store first, structure later
A data lake is a large repository that holds raw data in its native form: arrays, logs, images, tables, and unstructured files together. Its defining principle is schema-on-read: rather than forcing data into a fixed schema before storing it, the lake keeps the original and applies structure only when something reads it. This preserves flexibility and the raw record.
Schema-on-read versus schema-on-write
A traditional warehouse uses schema-on-write: data must conform to a defined structure before it is loaded. A lake reverses this. The benefit is that you never lose information by pre-filtering, and new questions can be asked of old data. The cost is that a lake without discipline becomes a swamp: undocumented, undiscoverable, untrustworthy.
What keeps a lake usable
- A catalog that records what each dataset is and where it lives.
- Consistent metadata and naming so data can be found.
- Provenance so the origin of each file is known.
- Access controls and clear zones (raw, curated, published).
Zones
Lakes are usually divided into zones: a raw zone holds untouched ingested data; a curated zone holds cleaned, conformed datasets; a published zone holds vetted, documented outputs ready for consumers. Data flows one direction through these zones, and the raw zone is immutable.
Storage foundation
Lakes are typically built on object storage, which scales cheaply and holds arbitrary files. Analytical formats such as Parquet are common in the curated zone. When a lake gains transactional table management, it becomes a lakehouse. For a science program, the lake holds the full raw simulation and, later, diagnostic record, from which curated and published datasets are derived without ever discarding the original.