Data Discovery and Metadata
Data discovery is how people locate and evaluate datasets through search, ranking, and metadata signals rather than by knowing where to look.
From catalog to discovery
A catalog is the inventory; discovery is the act of finding the right entry in it. The two are related as a library's shelves are to actually locating a useful book. Discovery combines search over metadata, ranking by relevance and trust, and enough context on each result for a person to decide quickly whether a dataset fits their need.
Metadata drives ranking
Good discovery ranks results using signals beyond a name match. Popularity (how many queries a table gets), freshness (when it was last updated), certification (whether a steward has endorsed it), and lineage centrality (how many pipelines depend on it) all indicate trustworthiness. A rarely used, stale, uncertified table should rank below a heavily used, fresh, certified one even if both match the search term.
Kinds of metadata
- Descriptive: names, descriptions, tags that aid search
- Structural: schema, partitioning, relationships to other datasets
- Administrative: ownership, access class, retention
- Operational: freshness, row counts, quality scores
- Social: usage, endorsements, and documented queries from other users
Semantic search over data
Keyword search fails when a user's vocabulary differs from the schema's. Embedding-based search over dataset descriptions and column documentation lets a query like "plasma confinement time series" find a relevant table even if its columns are named tersely. This is the same vector-similarity technique used for document retrieval, applied to metadata. It works only when the metadata is rich enough to embed.
Discovery as a feedback loop
Every search, click, and query is a signal that can improve future discovery. Tables that users repeatedly choose after a given search should surface faster; tables nobody selects should fall. Capturing this usage closes a loop that makes the catalog progressively more useful without additional manual curation. See data catalogs, vector databases, and data lineage.