> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/july2026/recipes/spark-ml-python-model.md).

# Cluster data with a Spark ML Python model

Write a Vulcan Python model that clusters data with Spark MLlib when Spark is available, and falls back to scikit-learn when it isn't.

A fixed rule like "more than 5 issues means high risk" only looks at one column at a time. Clustering groups rows by several features at once, so rows that behave similarly end up in the same group even if no single feature crosses a threshold.

Use this recipe when a Python model needs to run a real machine learning algorithm (KMeans clustering, in this case) and needs to work in two places: a Spark cluster in production, and a local environment where Spark or its MLlib JAR might not be available. The model below scores devices into risk clusters from four engineered features (`issues_7d`, `open_issues_30d`, `days_since_last_issue`, `risk_score`), using Spark MLlib when it can and scikit-learn otherwise.

{% hint style="info" %}
Every Python model function receives a `context: ExecutionContext` argument. On a Spark-backed engine, `context.spark` is a live `SparkSession`. This is what lets a Python model build MLlib pipelines instead of being limited to pandas.
{% endhint %}

## Steps

### 1. Ship the JAR that MLlib needs

Spark MLlib's estimators (`KMeans`, `LogisticRegression`, and others) are backed by JVM classes. Place the JAR under `dependencies/java/` at the project root (flat, no nested folders) and register it in `config.yaml`.

```yaml
# config.yaml (Spark gateway)
gateways:
  spark:
    connection:
      type: spark
      config:
        spark.jars: dependencies/java/spark-mllib_2.12-3.5.1.jar
        # or, for a Maven-hosted dependency instead of a local JAR:
        # spark.jars.packages: org.apache.spark:spark-mllib_2.12:3.5.1
```

Installing `pyspark` alone is not enough. PySpark ships the Python API, but MLlib's implementation classes still need to be resolvable on the Spark classpath, and that's what the JAR (or Maven package) provides. If the JAR isn't on the classpath, `KMeans.fit(...)` fails at runtime with a class-loading error even though the Python import succeeded.

### 2. Try the Spark path first, and fail back safely

Guard every step of the Spark path: check for `context.spark`, import `pyspark.ml` inside a `try/except`, and wrap the whole fit/transform sequence in its own `try/except` that returns `None` on any failure.

```python
def _run_spark_kmeans_if_available(context, base_df, feature_cols):
    spark = getattr(context, "spark", None)
    if spark is None:
        return None
    try:
        from pyspark.ml.clustering import KMeans as SparkKMeans
        from pyspark.ml.feature import VectorAssembler
    except Exception:
        return None
    try:
        spark_df = spark.createDataFrame(base_df[["snapshot_date", "device_id", "risk_score"] + feature_cols])
        assembler = VectorAssembler(inputCols=feature_cols, outputCol="features")
        train_df = assembler.transform(spark_df).select("snapshot_date", "device_id", "risk_score", "features")
        row_count = train_df.count()
        if row_count == 0:
            return None
        k = int(min(3, row_count))
        kmeans = SparkKMeans(k=k, seed=42, featuresCol="features", predictionCol="risk_cluster")
        fitted = kmeans.fit(train_df)
        out = fitted.transform(train_df).select("snapshot_date", "device_id", "risk_score", "risk_cluster").toPandas()
        out["engine_used"] = "spark_ml_kmeans"
        return out
    except Exception:
        return None
```

This model runs in at least two environments: a Spark cluster in production, and possibly a non-Spark engine in local development. Returning `None` (instead of raising) from this function is what lets the caller fall through cleanly to the scikit-learn path. The broad `except Exception: return None` is deliberate here: it's what makes "run distributed when possible, degrade gracefully otherwise" work.

Cap the cluster count at the row count of the batch (`k = int(min(3, row_count))`): KMeans fails if `k` exceeds the number of points it's clustering, which is a real risk on a small or empty batch.

### 3. Fall back to scikit-learn

Add a plain scikit-learn path that runs when Spark isn't available, when the MLlib import fails, or when the Spark path raises.

```python
def _run_sklearn_kmeans(base_df, feature_cols):
    row_count = len(base_df)
    if row_count < 2:
        fallback = base_df[["snapshot_date", "device_id", "risk_score"]].copy()
        fallback["risk_cluster"] = 0
        fallback["engine_used"] = "sklearn_fallback_single_cluster"
        return fallback
    from sklearn.cluster import KMeans as SklearnKMeans
    k = int(min(3, row_count))
    model = SklearnKMeans(n_clusters=k, random_state=42, n_init=10)
    labels = model.fit_predict(base_df[feature_cols].to_numpy())
    out = base_df[["snapshot_date", "device_id", "risk_score"]].copy()
    out["risk_cluster"] = labels.astype("int64")
    out["engine_used"] = "sklearn_kmeans"
    return out
```

This guarantees the model always produces a valid output table, even in a local or dev environment without a Spark cluster.

### 4. Record which engine ran, and normalize the schema

Add an `engine_used` column to the declared schema, and cast every output column to the exact dtype `columns` declares before the function yields, regardless of which path produced the result.

```python
@model(
    "s3depot.device_meb.device_issue_ml_workflow1",
    columns={
        "snapshot_date": "timestamp",
        "device_id": "string",
        "risk_score": "double",
        "risk_cluster": "int",
        "engine_used": "string",
        "model_version": "string",
    },
    kind=dict(name=ModelKindName.FULL),
    grains=["snapshot_date", "device_id"],
    depends_on=["s3depot.device_meb.device_issue_features1"],
    cron="@daily",
)
def execute(context, start, end, execution_time, **kwargs):
    base_df = context.fetchdf("SELECT * FROM s3depot.device_meb.device_issue_features")
    if base_df.empty:
        yield from ()
        return

    feature_cols = ["issues_7d", "open_issues_30d", "days_since_last_issue"]
    result = _run_spark_kmeans_if_available(context, base_df, feature_cols)
    if result is None:
        result = _run_sklearn_kmeans(base_df, feature_cols)

    result["model_version"] = "v1.0-issue-ml-workflow"
    result["risk_cluster"] = result["risk_cluster"].astype("int64")
    result["risk_score"] = result["risk_score"].astype("float64")
    yield result[["snapshot_date", "device_id", "risk_score", "risk_cluster", "engine_used", "model_version"]]
```

Silent fallback is only safe if it's observable. Recording `engine_used` directly in the output table means anyone reading the data can tell which engine actually produced a given row, without checking logs. Vulcan validates whatever is yielded against the `columns` schema regardless of which internal function produced it, so casting dtypes once, centrally, avoids duplicating that logic in both branches.

{% hint style="warning" %}
Return an empty generator (`yield from (); return`), never an empty DataFrame with the wrong dtypes. Python models must `yield`, not `return`, so Vulcan can tell "no data this run" apart from an error.
{% endhint %}

## Best practices

* Guard the Spark ML path with a fallback rather than letting the whole model fail when Spark or MLlib isn't available. This keeps the same model portable between local development and a production Spark cluster.
* Record which engine actually ran directly in the output schema. Silent fallback is only safe if it's observable.
* Cap cluster or model parameters like `k` against the actual row count of the batch being processed.
* Normalize dtypes once, centrally, right before yielding, instead of duplicating casts across every code path that can produce the result.
* Seed any stochastic ML algorithm (`seed=42` here) so results are reproducible across runs.

## Troubleshooting

| Symptom                                                            | Likely cause                                                | Fix                                                                                           |
| ------------------------------------------------------------------ | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Model always falls back to scikit-learn, even on the Spark cluster | An exception inside the Spark path is being silently caught | Temporarily narrow the `except Exception` to log the real error, then restore the broad catch |
| `KMeans` fit fails with a cluster-count error                      | `k` exceeds the number of rows in a small batch             | Confirm the `k = int(min(3, row_count))` guard is in place                                    |
| Output schema mismatch on write                                    | The Spark and scikit-learn branches return different dtypes | Cast dtypes once, centrally, right before `yield`, not separately in each branch              |
| Model errors instead of skipping on empty input                    | Missing empty-data guard                                    | Check `base_df.empty` before running either clustering path                                   |
| JAR-related class-loading error at runtime                         | The MLlib JAR isn't on the classpath                        | Confirm `spark.jars` (or `spark.jars.packages`) is set in the Spark gateway config            |

## Related recipes

* [Build a Python model for AI/ML](https://v2.dataos.info/build/july2026/recipes/build-a-python-model.md) - the general pattern this recipe specializes for clustering.

## References

* [Python models](https://v2.dataos.info/references/resources/vulcan/models/data-models/types/python) - the `@model` decorator, execution context, and dependency handling.
* [Model kinds](https://v2.dataos.info/references/resources/vulcan/models/data-models/model-kinds) - supported `kind` values for Python models.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/build/july2026/recipes/spark-ml-python-model.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
