> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/v1/resources/lakehouse/architecture.md). # Architecture A Lakehouse combines cloud object storage, an Iceberg REST Catalog, Sherpa, and a Spark Cluster. ## Lakehouse building blocks The Lakehouse consists of the following building blocks ### Object storage Object storage serves as the physical storage layer for all Iceberg table data. Lakehouse supports these object storage backends: * Amazon S3 * ABFSS * GCS Data files are stored in Apache Parquet, table metadata is managed with Apache Iceberg, and Secrets hold the required credentials. ### REST Catalog The REST Catalog is the Lakehouse metastore. It stores Iceberg table metadata, including schema, snapshots, views, and file locations, using a JDBC-backed catalog with PostgreSQL as the persistence layer. This enables reliable metadata storage and Iceberg view support. The REST Catalog is addressed by two fields, `metastoreUrl` and `metastoreRelativePath`. You'll use these when you connect a Depot to this Lakehouse. See **Create a Lakehouse Depot** for where these values come from and how to look them up for your own Lakehouse. ```yaml metastoreUrl: ..svc.cluster.local:80 metastoreRelativePath: / ``` ### Lakehouse Catalog Runtime (`lakehouse-cruntime`) The Catalog Runtime exposes the Iceberg REST Catalog and Lakehouse governance APIs. It handles: * Iceberg REST Catalog requests from clients. * Catalog metadata reads, writes, and updates. * Governance and access operations for Lakehouse schemas and tables. Spark, Trino, Data Products, maintenance runtimes, and Lakehouse Manager use this service. The Catalog Runtime is the entry point for catalog and governance operations. ### Spark Cluster DataOS always provisions an embedded Spark Cluster (`sparkCluster`) alongside the Lakehouse. It's the query and maintenance engine and handles: * Query execution and direct access through port-forwarding * Maintenance operations such as compaction, manifest rewrite, and snapshot expiry The Spark Cluster has three configurable components: * **Server:** manages the cluster lifecycle * **Driver:** coordinates the Spark job * **Executor:** runs the actual computation {% hint style="info" %} In the manifest, `spec.iceberg.sparkCluster` is an **optional block**: optional to configure, not optional to run. If you omit it, the Spark Cluster is still provisioned, using platform defaults for replicas and resource sizing. Include the block only when you need to override those defaults (for example, to size the driver or executor for your workload). {% endhint %} ### Sherpa Server Sherpa runs alongside the REST Catalog and like the Spark Cluster, is always provisioned as part of the Lakehouse. It handles maintenance job orchestration, delegating execution to the Spark Cluster. The execution flow is: ``` User → REST Catalog → Sherpa Server → Spark Cluster ``` Sherpa queues requests and runs them one at a time to prevent race conditions, and it retries failed operations and stores job status for later inspection. ### Sherpa Worker Sherpa uses a master-worker model. The server delegates queued work to the worker, and the Spark Cluster processes the actual job execution. Same rule as Spark Cluster: `spec.iceberg.sherpa` in the manifest is optional to **configure** (replica count, CPU/memory). If not defined explicitly, Sherpa still comes up with default replicas and sizing. ## Consumer Architecture The Lakehouse is intentionally separated from query execution.
Consumers interact with the Lakehouse through compute engines such as Trino or Spark, which use the Iceberg REST Catalog for metadata and object storage for data access. Data Products built on DataOS consume the Lakehouse through these compute engines rather than querying the Lakehouse directly. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/v1/resources/lakehouse/architecture.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.