> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/consume/recipes/retrieve-large-result-sets-with-pagination.md).

# Retrieve large result sets with pagination

Page through Semantic REST result sets larger than 50,000 records using limit, offset, and a deterministic order.

Your Semantic REST query returns more than **50,000 records**, so you need to retrieve it in batches. Semantic REST queries are asynchronous: you submit the query, poll the statement until it completes, then fetch the result. This recipe adds `limit`, `offset`, and a deterministic `order` so each batch picks up exactly where the last one ended. The journey is submit a batch → poll the statement → retrieve the result → repeat with the next offset.

**What this enables:**

* [ ] Retrieve result sets larger than 50,000 records in predictable batches
* [ ] Avoid duplicate or missing records between batches
* [ ] Choose JSON batches or a single Parquet file for the result

{% hint style="info" %}
**Prerequisites:** a DataOS API token and the Data Product's API base URL. See [Authentication](https://v2.dataos.info/references/interfaces/apis/data-product-apis/getting-started/authentication).
{% endhint %}

## Steps

### 1. Submit the first batch

Add `limit`, `offset`, and `order` to the **POST request body**:

* `limit` defines the maximum number of records in each batch. Set it based on your application's throughput requirements.
* `offset` defines where the batch begins.
* `order` must stay the same across all batches.

The following request retrieves the first 50,000 records:

{% code title="Paginated Semantic REST query" %}

```bash
curl --request POST \
  'https://<DATAOS_FQDN>/vulcan/tenants/<DATAOS_TENANT_ID>/data-products/<DATAOS_TENANT_ID>-<resource-name>/api/v1/query/semantic/rest' \
  --header 'Content-Type: application/json' \
  --header "Authorization: Bearer ${DATAOS_TOKEN}" \
  --data '{
    "query": {
      "measures": [
        "customer_profile.count"
      ],
      "dimensions": [
        "customer_profile.region_id"
      ],
      "order": {
        "customer_profile.region_id": "asc"
      },
      "limit": 50000,
      "offset": 0,
      "timezone": "UTC"
    }
  }'
```

{% endcode %}

### 2. Poll the statement

Each POST request returns a statement ID:

```json
{
  "id": "01KYP6V3G1JTNSRARCN0HAF43B",
  "status": "QUEUED",
  "strategy": "EXECUTE"
}
```

Poll the statement until the status becomes `SUCCESS`:

{% code title="Poll statement status" %}

```bash
curl --request GET \
  'https://<DATAOS_FQDN>/vulcan/tenants/<DATAOS_TENANT_ID>/data-products/<DATAOS_TENANT_ID>-<resource-name>/api/v1/query/statement/<statement-id>' \
  --header "Authorization: Bearer ${DATAOS_TOKEN}"
```

{% endcode %}

Keep polling while the status is `ACCEPTED`, `QUEUED`, or `IN_PROGRESS`.

### 3. Retrieve the result

{% tabs %}
{% tab title="JSON" %}
{% code title="Fetch result as JSON" %}

```bash
curl --request GET \
  'https://<DATAOS_FQDN>/vulcan/tenants/<DATAOS_TENANT_ID>/data-products/<DATAOS_TENANT_ID>-<resource-name>/api/v1/query/statement/<statement-id>/result?format=json' \
  --header "Authorization: Bearer ${DATAOS_TOKEN}"
```

{% endcode %}
{% endtab %}

{% tab title="Parquet" %}
{% code title="Download result as Parquet" %}

```bash
curl --location \
  'https://<DATAOS_FQDN>/vulcan/tenants/<DATAOS_TENANT_ID>/data-products/<DATAOS_TENANT_ID>-<resource-name>/api/v1/query/statement/<statement-id>/result?format=parquet' \
  --header "Authorization: Bearer ${DATAOS_TOKEN}" \
  --output result.parquet
```

{% endcode %}

Parquet downloads the complete result as a single file. `limit`, `offset`, and `columns` are ignored for the Parquet format. For large exports, use Parquet instead of retrieving many JSON batches.
{% endtab %}
{% endtabs %}

### 4. Fetch the next batch

Keep the same `order` and increase only the `offset`, then repeat steps 1 to 3:

```
Batch 1 → limit=50000, offset=0
Batch 2 → limit=50000, offset=50000
Batch 3 → limit=50000, offset=100000
```

Continue until a returned batch contains fewer records than the requested `limit`.

## Troubleshooting

<table><thead><tr><th width="272.60467529296875">Issue</th><th>Resolution</th></tr></thead><tbody><tr><td>The statement status is <code>FAILED</code></td><td>Inspect the <code>error</code> field in the statement response.</td></tr><tr><td>Batches contain duplicate records, or records are missing</td><td>Use the same <code>order</code> in every request and change only the <code>offset</code> between batches.</td></tr><tr><td><code>limit</code> or <code>offset</code> has no effect on the result</td><td>The Parquet format ignores <code>limit</code>, <code>offset</code>, and <code>columns</code> and returns the complete result. Use JSON to retrieve batches.</td></tr></tbody></table>

## Outcome

You retrieved a result set larger than 50,000 records in consistent batches, with no duplicates or gaps, or downloaded it as a single Parquet file.

## References

* [Query API reference](https://v2.dataos.info/references/interfaces/apis/data-product-apis/api-reference/query): full statement submission, polling, and result endpoints
* [Build applications with Semantic GraphQL](https://v2.dataos.info/consume/recipes/query-semantic-model-with-graphql.md): a synchronous alternative for smaller, structured result sets


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/consume/recipes/retrieve-large-result-sets-with-pagination.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
