AWS-backed
Configure an Amazon S3 Lakehouse destination for Nilus batch and CDC pipelines.
The AWS-backed DataOS Lakehouse destination writes Apache Iceberg-backed datasets into a Lakehouse Depot whose storage layer is Amazon S3. Nilus uses the same Lakehouse connector for every cloud. The Lakehouse Depot selects the storage backend, not a separate connector type. AWS-backed, Azure-backed, and GCP-backed Lakehouse Depots all share the same pipeline shape, sink address pattern, and table behavior. This page covers the AWS / S3 variant. The Azure-backed DataOS Lakehouse and GCP-backed DataOS Lakehouse variants are documented separately.
Requirements
Connectivity and credentials must both be in place before the pipeline can run.
Connectivity
The Nilus runtime must reach the configured S3 endpoint and the Iceberg REST catalog identified by
METASTORE_URL.The Lakehouse Depot must be defined in DataOS with
storageType: s3and aspec.s3block, and must be reachable from the runtime cluster.An AWS region is required. Provide it via
aws_regionin the sink options or on the Depot resource.
Lakehouse depot shape
For an S3-backed Lakehouse the Depot resource carries:
spec:
storageType: s3
s3:
bucket: my-iceberg-bucket
region: us-east-1
relativePath: warehouse/ # optional sub-prefix
endpoint: https://s3.us-east-1.amazonaws.com # optional, for S3-compatible storesStorage credentials projected from the Depot's secret:
aws_access_key
Yes
Maps to AWS_ACCESS_KEY_ID at runtime.
aws_secret_key
Yes
Maps to AWS_SECRET_ACCESS_KEY at runtime.
aws_session_token
No
Use for temporary credentials (e.g. assumed-role flows).
Permissions
The Depot's IAM principal must be able to read, write, and delete objects under the configured bucket / prefix.
The runtime must be authorized to register tables in the Iceberg REST catalog. The Lakehouse Depot secret must include an
apikey(a DataOS API token). Nilus reads it asLAKEHOUSE_APIKEYand passes it to the Iceberg REST catalog as the connection token. If the secret has noapikey, the pipeline fails at startup with'apikey' is required in Lakehouse depot secret.
Sink address
Reference the Lakehouse Depot by name; the address format is the same for every cloud backend:
Providing a lakehouse:// URI directly in a manifest is not supported. Always go through the Depot.
Sink options
dest_table
Yes
Target Iceberg table in <schema>.<table> form. Exactly two dot-separated parts.
incremental_strategy
Yes
One of replace, append, merge.
aws_region
Conditional
Required when the region is not already on the Depot or in AWS_REGION.
aws_endpoint
No
Optional endpoint override for S3-compatible object stores.
iceberg_metadata_compression
No
Compression codec for Iceberg metadata files. gzip (default) or none.
partition_by
No
Iceberg partition spec (list of column / transform pairs). See Optimize sink datasets.
A custom S3 endpoint can be set either as the aws_endpoint sink option or as endpoint on the Depot's spec.s3 block; it is honored consistently across batch, CDC, stream, and metadata pipelines. When neither is set, Nilus defaults to https://s3.<region>.amazonaws.com.
Iceberg metadata compression
Nilus compresses Iceberg metadata files with gzip by default. To change this, set the iceberg_metadata_compression sink option:
Valid values are "gzip" and "none" and must be lowercase. Other casings such as "Gzip" or "None" are rejected by schema validation. Changing this setting between runs is safe: it does not rewrite or invalidate the metadata of existing tables, and pipelines that never set it continue to use the gzip default.
Sample Nilus configs
Each example below is self-contained and uses the current Nilus pipeline shape.
Batch ingestion
CDC ingestion
Behavior and capabilities
Table format: Apache Iceberg. Nilus writes Parquet data files to S3 and registers / updates the resulting tables in the Iceberg REST catalog.
Object model: Iceberg tables addressed as
<schema>.<table>.Supported pipeline types:
batchandcdc.Supported incremental strategies:
replace,append,merge.Storage layer: Amazon S3 or S3-compatible object stores (when
aws_endpointis configured on the Depot or in sink options).Decimal types: Table reads from SQL batch sources preserve source precision and scale. A
query:extract stores decimals as Icebergdecimal(38,9). Do not switch extract styles for the samedest_table. See Schema evolution.
Troubleshooting
METASTORE_URL environment variable is required for Lakehouse destination
The Depot did not project METASTORE_URL into the runtime.
Re-check the Lakehouse Depot configuration; verify the Iceberg REST catalog endpoint is set on the Depot resource.
Unsupported scheme: <X>. Expected 'lakehouse://'
The manifest authored a non-dataos:// URI that did not resolve to a Lakehouse Depot.
Always reference the Lakehouse via dataos://<lakehouse-depot>.
Table name must be in the format <schema>.<table>
dest_table is missing the schema prefix or has more than two parts.
Use exactly <schema>.<table>.
S3 write fails with 403 / AccessDenied
Storage credentials lack write or delete permission on the configured prefix.
Re-check the Depot secret and the IAM / role bindings on the bucket.
NoSuchBucket or region mismatch
The Depot's region is not set, or the Depot region disagrees with the runtime AWS_REGION.
Set aws_region in sink options or on the Depot resource.
Iceberg commit fails with 409 Conflict
Concurrent writers from another pipeline are competing on the same target table.
Serialize writes per Iceberg table, or use distinct destination tables per pipeline.
ValueError: Cannot serialize value, scale of value does not match type decimal(38, 9): 4
A query: extract created the table with decimal(38,9), then a table read tried to write decimals with the source scale, or the reverse.
Keep one extract style per table. Use a new dest_table, or drop and reload the existing table. See Schema evolution.
Related docs
Optimize sink datasets: guidance on
incremental_strategyand dataset-shape tuning for Iceberg targets.
Last updated
Was this helpful?