> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/concepts/resources/secret/data-sources/bigquery.md). # BigQuery ## Prerequisites To create a Secret for securing BigQuery credentials, you must have the following information: ### **Source system requirements** * **Project ID**: The BigQuery project ID used to identify your Google Cloud project. You can retrieve this from the Google Cloud Console by navigating to the project dashboard and locating the Project ID under project information. * **Email ID**: The email address associated with the service account used to access BigQuery. This can be found in the **IAM & Admin > Service Accounts** section of the Google Cloud Console by selecting the relevant service account and viewing its details. * **JSON Key File**: The path to the JSON key file for the service account, used for authentication. To obtain this, go to \**IAM & Admin > Service Accounts*- in the Google Cloud Console, select the service account, click **Keys**, and download the JSON key file by selecting **Add Key > JSON**. Ensure the file is securely stored and note its path. Ensure you have these credentials ready before proceeding with the Secret creation process. ## Create a Secret for securing BigQuery credentials BigQuery is a data warehouse that serves as a centralized repository for structured data, supporting efficient query and analysis. To create a BigQuery Secret in DataOS, ensure you have access to the DataOS Command Line Interface (CLI) and the required permissions. Follow the steps below to complete the creation process efficiently and securely. ### **Step 1: Create a manifest file** Begin by creating a manifest file to hold the configuration details for your BigQuery Secret. ```yaml name: ${{bigquery-secret-name}} version: v2alpha type: secret tags: - ${{tag-1}} - ${{tag-2}} description: "Credentials for bigquery depot" layer: user secret: type: key-value files: gcp_json_key: ${{bigquery-gcs-key-json}} ``` For more information about each attribute, refer to the [configurations section.](/concepts/resources/secret/manifest-configuration.md) ### **Step 2: Apply the manifest** To create the BigQuery Secret within DataOS, use the `apply` command. ```bash dataos-ctl resource apply -f ${{manifest-file-path}} ``` **Example Usage:** ```bash dataos-ctl resource apply -f secret.yaml #output INFO[0000] 🛠 apply... INFO[0000] 🔧 applying bigquery-cred:v2alpha:secret... INFO[0004] 🔧 applying bigquery-cred:v2alpha:secret...created INFO[0004] 🛠 apply...complete ``` ### **Step 3: Validate the Secret** To validate the proper creation of the BigQuery Secret in DataOS, use the `get` command. ```bash dataos-ctl resource get -t secret ``` **Expected Output:** ```bash INFO[0000] 🔍 get... INFO[0000] 🔍 get...complete NAME | VERSION | TYPE | WORKSPACE | STATUS | RUNTIME | OWNER -----------------|---------|-----------------|-----------|--------|-----------|------------------------------ bigquery-cred | v2alpha | secret | | active | | iamgroottmdcio ``` To get the list of all the Secrets within the Dataos environment, execute the following command. ```bash dataos-ctl resource get -t secret -a ``` Expected Output: ```bash time="2026-03-25T15:34:17+05:30" level=info msg="🔍 resource get..." time="2026-03-25T15:34:17+05:30" level=info msg="🔍 resource get...complete" NAME | VERSION | TYPE | STATUS | RUNTIME | OWNER --------------------------------+---------+--------+--------+---------+------------------------- bigquery-cred | v2alpha | secret | active | | iamgroottmdcio azureconnection-testing | v2alpha | secret | active | | iamgroottmdcio azuresecretnilus | v2alpha | secret | active | | iamgroottmdcio bitbucket-secrets | v2alpha | secret | active | | iamgroottmdcio ``` ## Delete the Secret {% hint style="warning" %} Before you can delete a Secret, you need to make sure there are no other Resources dependent on it. For example, if a Depot has a dependency on a Secret, trying to delete that Secret will cause an error. So, you'll need to remove the Depot first, and then you can delete the Secret. This rule applies not just to Depot but also to all dependent Resources, such as Workflow, Service, Worker, etc. The following error will be thrown if any Resource has a dependency on a Secret, as shown below. **Example usage:** ```bash dataos-ctl resource delete -t secret -n postgres-cred time="2026-03-25T15:46:12+05:30" level=info msg="🗑 delete..." time="2026-03-25T15:46:12+05:30" level=info msg="🗑 deleting postgres-cred:v2alpha:secret..." time="2026-03-25T15:46:13+05:30" level=info msg="🗑 deleting postgres-cred:v2alpha:secret...error" time="2026-03-25T15:46:13+05:30" level=warning msg="🗑 delete...error for resource postgres-cred" time="2026-03-25T15:46:13+05:30" level=error msg="Invalid Parameter - failure deleting tenant resource : cannot delete resource, it is a dependency of 'depot:v2alpha:postgresconnection'" ``` {% endhint %} To delete the BigQuery Secret, use one of the following commands: {% tabs %} {% tab title="Command 1" %} ```bash dataos-ctl resource delete -t secret -n ${{secret-name}} ``` {% endtab %} {% tab title="Command 2 " %} ```bash dataos-ctl resource delete -i "${{secret-name}}|v2alpha|secret" ``` {% endtab %} {% tab title="Command 3" %} ```bash dataos-ctl resource delete -f ${{manifest-file-path}} ``` {% endtab %} {% endtabs %} Specify the Resource type and Secret name in the `delete` command. **Example Usage:** {% tabs %} {% tab title="Command 1" %} ```bash dataos-ctl resource delete -t secret -n testsecret #output time="2026-03-25T15:53:55+05:30" level=info msg="🗑 delete..." time="2026-03-25T15:53:55+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret..." time="2026-03-25T15:53:56+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret...deleted" time="2026-03-25T15:53:56+05:30" level=info msg="🗑 delete...complete" ``` {% endtab %} {% tab title="Command 2" %} ```bash dataos-ctl resource delete -i "testsecret|v2alpha|secret" #output time="2026-03-25T15:55:37+05:30" level=info msg="🗑 delete..." time="2026-03-25T15:55:37+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret..." time="2026-03-25T15:55:37+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret...deleted" time="2026-03-25T15:55:37+05:30" level=info msg="🗑 delete...complete" ``` {% endtab %} {% tab title="Command 3" %} ```bash dataos-ctl resource delete -f ${{manifest-file-path}} #output time="2026-03-25T15:53:55+05:30" level=info msg="🗑 delete..." time="2026-03-25T15:53:55+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret..." time="2026-03-25T15:53:56+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret...deleted" time="2026-03-25T15:53:56+05:30" level=info msg="🗑 delete...complete" ``` {% endtab %} {% endtabs %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/concepts/resources/secret/data-sources/bigquery.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.