> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/concepts/resources/nilus/destinations/cloud-warehouses/bigquery.md).

# BigQuery

[Google BigQuery](https://cloud.google.com/bigquery/docs) is a managed analytical warehouse on Google Cloud. Nilus writes into BigQuery datasets and tables for both batch and CDC pipelines using a service-account-authenticated connection, with optional GCS staging for large loads.

## Requirements

Connectivity and credentials must both be in place before the pipeline can run.

### Connectivity

* The Nilus runtime must reach `https://bigquery.googleapis.com` over HTTPS.
* For staged loads, the runtime must also reach `https://storage.googleapis.com`.
* The destination project (encoded as the URI hostname) and the BigQuery dataset must already exist; Nilus does not auto-create datasets.

### Required parameters

| Parameter            | Required    | Default                | Description                                                                                                                                                                         |
| -------------------- | ----------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `project`            | Yes         | -                      | GCP project ID that owns the destination dataset. Encoded as the URI hostname.                                                                                                      |
| `credentials_path`   | Conditional | -                      | Path to a Google service-account JSON key file. Mutually exclusive with `credentials_base64`.                                                                                       |
| `credentials_base64` | Conditional | -                      | Base64-encoded contents of the service-account JSON key. Mutually exclusive with `credentials_path`. When neither is supplied, Nilus falls back to Application Default Credentials. |
| `location`           | No          | runtime default (`US`) | BigQuery location for the destination dataset, for example `US`, `EU`, `asia-southeast1`. Must match the dataset's actual location.                                                 |

### Permissions

The service account behind the credentials needs at minimum:

* `roles/bigquery.dataEditor` on the target dataset (or finer `dataset.tables.create`, `dataset.tables.updateData`, `dataset.tables.delete`).
* `roles/bigquery.jobUser` on the project (to run load and merge jobs).
* If `staging_bucket` is configured: `roles/storage.objectAdmin` (or finer `objects.create`, `objects.delete`) on the staging bucket.

### URI format

```
bigquery://<project>?credentials_path=<path-to-service-account-json>&location=<location>
```

Inline credentials variant:

```
bigquery://<project>?credentials_base64=<base64-service-account-json>&location=<location>
```

## Sink options

| Option                 | Required | Description                                                                                          |
| ---------------------- | -------- | ---------------------------------------------------------------------------------------------------- |
| `dest_table`           | Yes      | Target table in `<dataset>.<table>` form, or `<project>.<dataset>.<table>` for cross-project writes. |
| `incremental_strategy` | Yes      | One of `replace`, `append`, or `merge`.                                                              |
| `staging_bucket`       | No       | `gs://`-prefixed staging bucket for large loads. Validated to start with `gs://`; raises otherwise.  |

## Sample Nilus configs

Each example below is self-contained and uses the current Nilus pipeline shape.

### Batch ingestion

```yaml
name: nilus-bigquery-batch
version: v1alpha
type: nilus
spec:
  type: batch
  compute: universe-compute
  source:
    address: dataos://mysql-depot
    options:
      source_table: sales.orders
  sink:
    address: dataos://bigquery-destination
    options:
      dest_table: analytics.orders
      incremental_strategy: replace
```

### CDC ingestion with GCS staging

```yaml
spec:
  type: cdc
  compute: universe-compute
  source:
    address: dataos://mongodb-cdc
    cdc:
      collection.include.list: "retail.orders"
      topic.prefix: "orders_cdc"
  sink:
    address: dataos://bigquery-destination
    options:
      dest_table: analytics.orders_cdc
      incremental_strategy: merge
      staging_bucket: gs://nilus-bq-staging
```

## Behavior and capabilities

* **Compute model**: Nilus connects with a service account and submits BigQuery jobs; BigQuery executes the load / merge.
* **Object model**: BigQuery datasets and tables. `dest_table` accepts either two-part (`dataset.table`) or three-part (`project.dataset.table`) names.
* **Supported pipeline modes**: `batch` and `cdc`.
* **Authentication fallback**: when neither `credentials_path` nor `credentials_base64` is supplied, the runtime falls back to Application Default Credentials. Useful when running the pipeline on a GCE / GKE instance that already has a workload identity attached.
* **Region awareness**: the dataset, the staging bucket (if any), and the `location` query parameter must all be in the same region or BigQuery will reject the load with `Location: <X>` mismatch errors.

## Troubleshooting

| Symptom                                                                             | Likely cause                                                                                 | Resolution                                                                                           |
| ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `Table name must be in the format <dataset>.<table> or <project>.<dataset>.<table>` | `dest_table` was supplied as a bare table name.                                              | Use the qualified `<dataset>.<table>` form (or three-part for cross-project writes).                 |
| `Staging bucket must start with gs://`                                              | The `staging_bucket` value uses `gcs://` or omits the scheme.                                | Use the canonical `gs://<bucket>` form.                                                              |
| `403 Permission … denied on resource …`                                             | Service account lacks dataset or job-running permissions.                                    | Grant `roles/bigquery.dataEditor` on the dataset and `roles/bigquery.jobUser` on the project.        |
| `Cannot read and write in different locations`                                      | Dataset, staging bucket, and `location` query param are not all in the same BigQuery region. | Align the dataset region, the staging bucket region, and the `location` value.                       |
| Application Default Credentials unexpectedly used                                   | Both credential parameters were omitted; the runtime fell back to ADC.                       | Pass `credentials_path` or `credentials_base64` explicitly when ADC is not the intended auth method. |

## Related docs

* [Google BigQuery](/concepts/resources/nilus/batch/batch-sources/google-bigquery.md): companion batch source connector.
* [Optimize Sink Datasets](/concepts/resources/nilus/pipeline-optimization/optimize-sink-datasets.md): guidance on `incremental_strategy`, `partition_by`, `cluster_by`, and other dataset-shape settings.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/concepts/resources/nilus/destinations/cloud-warehouses/bigquery.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.