> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/home/v1/readme-1.md). # DataOS Philosophy You are probably here because someone decided your team would adopt **Data Products**, or adopt DataOS, and the reasoning did not arrive with the decision. This page is the reasoning. It is written for the engineer who has to build the thing, not the executive who approved it, and it assumes you are skeptical, because you should be. Data engineering patterns do not change for fashion. They change when an old constraint lifts, or a new need outgrows them. One has been lifted, and one has outgrown. Here is the chain. *** ## Your numbers are right. The table still is not usable. You load raw data and transform it in place. This is ELT, and it is correct. Storage is cheap, compute is elastic, and the raw layer survives, so a wrong join is an edit, not a re-ingestion. None of that is the problem. The problem is what ELT was never built to do. It solves the problem of *producing* a transformed table. It does nothing for *consuming* one. The numbers are right, and the table is still half a product. {% hint style="info" %} A table carries its rows and its schema. It does not carry how to find it, whether today's data is fresh enough to trust, who may see which rows, what a column actually means, or where to report it when it breaks. That half lives in your head, in a wiki, and in the one analyst who knows `revenue` means net of refunds. {% endhint %} *** ## You stack tools to fix that. They drift. The usual answer is to surround the table with tools: a catalog for discovery, a quality platform for freshness, a governance tool for access, and a semantic layer for definitions. Each is a real product solving a real gap. Each is also a separate system with its own copy of your table and its own clock. That creates one failure that integration cannot remove: **the tools drift away from the table they describe.** The table is the thing that changes, and not always in shape. The transform team ships a corrected "`revenue`: net of refunds," where it used to be gross. Nothing breaks: same column, same type, every query still runs and still returns a number. That is what makes it dangerous. The other four systems keep describing the old meaning, each on its own schedule. {% code title="one definition change, four systems, no shared clock" %} ```shellscript Monday transform revenue corrected: gross → net of refunds (same column, same type) Monday → semantics layer still defines revenue as gross; every query returns the old meaning, green Monday → quality freshness and row counts pass; checks watch shape, not meaning Monday → governance policy still guards the deprecated gross table, not the live one Thursday catalog re-crawls, still labels revenue "gross sales" until someone edits the description ``` {% endcode %} For three days the tools describe a `revenue` that no longer means what they say, and none of them say so. Drift does not fail loudly. It waits until someone trusts a stale definition and ships a wrong decision. You can fight drift, but only with people. Reconciliation stops being a task and becomes a standing program: tickets, sprints, and a recurring meeting whose only job is keeping four tools agreeing with each other. Every change you make spawns work in three other systems, and it never quite converges. *** ## An agent reads the drift and does not flinch. A human absorbs drift without noticing. The analyst who asks for Q3 revenue knows the team moved it to net of refunds last week, so when the catalog still says gross, they quietly use the corrected figure. That context — whose source is authoritative, that the gross table was deprecated — lives in their head, not in any table. An AI agent carries none of it. It resolves the same `revenue` the catalog still blesses, the stale gross one, at machine speed, with no one in the loop to wince. |

analyst → "what was Q3 revenue?"

$4.21M

net of refunds · authoritative source

|

agent → "what was Q3 revenue?"

$4.88M

gross · deprecated table · no flag

| | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | The model is not the problem. The data underneath it is. An agent pointed at a drifted stack returns the wrong answer with the same confidence as the right one. The half-product problem was merely expensive while only humans consumed data. It turns dangerous the moment a machine does. *** ## The fix is not another layer. It is one object. Drift is not caused by bad tools. It is caused by *separation*. Semantics, quality, and access each live in their own system on their own clock, and nothing binds them to the table. A fifth tool adds a fifth clock. The only way to remove drift is to remove the separation. Author the semantics, the contract, the quality rules, and the access policy *with* the data, as one object. Then they cannot fall out of sync, because there is nothing to sync. One model cannot disagree with itself. That one object is a **Data Product**. {% hint style="info" %} A Data Product is data you can hand to someone, or something, that never met you, and trust them to use it correctly. **Formally**: A self-contained, managed unit of data with a named owner, a published contract, named consumers both human and machine, and a versioned lifecycle. {% endhint %} Eight properties define one. Each earns its place by what it removes from your week.
The property that does the real work is the one you cannot see: a change to the data is a change to its definition, because they are the same edit. Drift never gets a window to open. To be precise about the boundary, a Data Product is **not** a raw table dumped in the warehouse, not a dashboard, not a pipeline, and not a query alias that federates sources no one owns. The pipeline is plumbing. The product is the owned, contracted output built on top of it. *** ## This is the next pattern, not a new tool. The Data Product is not a feature. It is the third pattern in a sequence you already work inside.
PatternWhat it didWhere it stopped
ETLTransformed before load, to spare a scarce warehouse.Discarded the raw. Every transform was bet-the-pipeline; a wrong join meant re-ingesting from a source that may be gone.
ELTLoaded raw, then transformed in place once storage got cheap.Produced a correct table that only its author can safely use. Everything else gets bolted on, and drifts.
ELPLoads raw, then productizes: folds the transform into one authored object.The current pattern. Semantics, contract, policy, and serving are part of the object, not layers around it.
The shift in one line: **Productize absorbs Transform.** The transform stops being a phase you surround with tools and becomes a property of the product. ```mermaid %%{init: "themeVariables": { "background": "#FFFFFF", "primaryTextColor": "#242422", "lineColor": "#242422", "fontFamily": "Neue Montreal, sans-serif" } }}%% flowchart LR E[Extract] --> L[Load raw ] L --> P subgraph P["Productize: one authored object "] direction TB SEM[Semantic model
measures, contracts] PHY[Physical assets
tables, views] GOV[Governance, SLAs, serving
API + AI] SEM <--> PHY PHY --- GOV end classDef stage fill:#FFFFFF,stroke:#242422,color:#242422,stroke-width:1.5px; classDef product fill:#EDE9E5,stroke:#54DED1,color:#242422,stroke-width:1.5px; classDef semantic fill:#EDE9E5,stroke:#009293,color:#242422,stroke-width:1.5px; classDef physical fill:#D6CDC6,stroke:#35505B,color:#242422,stroke-width:1.5px; classDef governance fill:#EDE9E5,stroke:#733635,color:#242422,stroke-width:1.5px; class E,L stage; class SEM semantic; class PHY physical; class GOV governance; style P fill:#FFFFFF,stroke:#54DED1,stroke-width:2px,color:#242422; linkStyle 0 stroke:#242422,stroke-width:2px; linkStyle 1 stroke:#242422,stroke-width:2px; linkStyle 2 stroke:#009293,stroke-width:1.8px; linkStyle 3 stroke:#35505B,stroke-width:1.6px; ``` ELP is the pattern. DataOS is one implementation of it. *** ## What it asks of you, and what it costs. The discipline is not new. It is software engineering, applied to data, and you already use it on code.
You already do this in softwareA Data Product asks the same of data
Write a spec before the code.Write a product spec first: consumers, the questions it answers, what is out of scope.
Publish an interface, hide the implementation.Declare schema, semantics, and SLA first. Consumers bind to the contract, not to storage paths.
Tests gate the merge.Quality runs in the build and blocks publication. Bad data is caught, not alerted on afterward.
Version the API; deprecate, do not break.Schema and meaning change through versions, so consumers, including models in production, are never surprised.
It is fair to be told the cost, not just the benefit. * **It is more work up front.** A spec and a contract come before the pipeline. There is no product without them. The payoff is later and larger: the reconciliation program disappears, because there are no longer four systems to reconcile. * **It does not rescue a bad model.** Modeling is still the craft. A product wrapped around a confused schema is a confused product with a contract. * **It will reject things you call Data Products today.** A federated alias over sources you do not own has nothing durable for a contract to bind to. That is a query, not a product. * **A product built for a dashboard is not automatically fit for an agent.** Data summarized for human reading often lacks the detail a model needs. Fitness is per consumer, and it is designed, not inherited. *** ## How DataOS builds them DataOS produces and runs Data Products from the data where it already lives. It runs over the warehouse, lakehouse, or catalog you already own, so you can start with one product on one source rather than a replatform. Building a Data Product in DataOS follows three stages.

01 Understand

Find what data exists, inspect its quality and lineage, and decide whether to build on existing sources or connect new ones.

/spaces/bCA0p0BwgdQAP5IwnaO4/pages/L7i5I15WZlCTTkMn8gM0

02 Productize

Build it: transformations, validation, semantic models, versioned outputs, and CI/CD, with data and outputs kept in the engine.

/spaces/bCA0p0BwgdQAP5IwnaO4/pages/p6G6skDvC6fgEbY8pXno

03 Consume

Expose one contract across SQL, REST, GraphQL, BI protocols, Python SDKs, and MCP tools for agents.

/spaces/cL04JUTJPL73kRrjaBSa/pages/pAL4eAwPscVp8ARv3npk
*** You were not asked to adopt a tool. You were asked to ship the contract along with the table, so the next person, or the next agent, can use your work without first finding you. ## Next steps * [Data Product Journey](/home/v1/data-product-journey-v2.md): Follow one business question through all three stages as a worked example. * [For Builders](https://v2.dataos.info/build): Build the Orders Analytics product end to end. * [For Consumers](https://v2.dataos.info/consume): Consume a product across SQL, APIs, BI, and agents. * [For Operators](https://v2.dataos.info/operate): Operate products with ongoing trust, quality, and governance. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/home/v1/readme-1.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.