External models
How `EXTERNAL` models describe schemas for tables Vulcan does not manage, and how to create and validate them.
Your models sometimes need to query tables that exist outside your Vulcan project: a third-party data source, a table managed by another system, or a read-only database. These are external tables.
Vulcan does not manage external tables, but it can use metadata about them. When you define external models, you give Vulcan column names and types. Even though Vulcan doesn't manage these tables, knowing their schema helps with:
Column-level lineage (see how data flows through external tables)
Query optimization (Vulcan can make better decisions)
Documentation (your data catalog knows what is in those tables)
Vulcan stores this metadata as EXTERNAL models.
How external models work
EXTERNAL models are metadata-only. They describe a table's schema (column names and types). There is no query for Vulcan to run, and Vulcan does not manage the data.
Important limitations:
Vulcan does not know what data is in the table (or if it exists)
If someone alters the external table, Vulcan doesn't detect it
If all data is deleted, Vulcan doesn't know
Vulcan never modifies external tables
When external tables get queried: only when a Vulcan model references them. Vulcan never proactively queries external tables on its own. The referencing model's kind, cron, and previously loaded time intervals determine exactly when that query happens.
Creating external models
You define external models in YAML files. You have two options:
Let Vulcan generate it (easiest): use the
create_external_modelsCLI command.Write it yourself: hand-craft the YAML if you need more control.
The main file is inputs.yaml in your project root. You can also add more files in the inputs/ directory.
Here's what an external model definition looks like, for three tables in a demo project:
Vulcan never runs this file; it's metadata only. The SQL model below is what actually queries these tables:
The following sections show how to create external models like these: generated automatically, written by hand, or a mix of both.
Using CLI
Instead of writing inputs.yaml manually, Vulcan can generate it for you with the create_external_models CLI command:
What it does:
Scans your project for references to external tables
Fetches column information from your SQL engine's metadata
Writes everything to
inputs.yaml, or toinputs.ymlif that's the file your project already uses
create_external_models only queries SQL engine metadata, not the external tables themselves, so it's fast and safe to run.
If Vulcan cannot access a table's metadata: it omits that table from the file and issues a warning. Define it manually instead (see "Writing YAML by hand" below).
Writing YAML by hand
Sometimes you need to define external models manually, when Vulcan cannot access the metadata or you want more control. Here is the structure:
What you need:
name: Fully qualified table name (with quotes if needed for case sensitivity)columns: Dictionary of column names to data types
Optional fields:
description: Human-readable descriptiongateway: Gateway name (for gateway-specific tables)remote_engine: The engine or source system the table actually lives in, such asredshift
Use remote_engine when the query engine is Trino or Spark and the table is reached through a catalog that points at another system. It records where the data really comes from, so consumers can see the source system in the model's metadata:
remote_engine is metadata only and applies to external models. It's free text: Vulcan stores the value as you write it and doesn't validate it. If you leave it out, no source system is recorded.
Tip: use triple-quoted names if your table names have special characters or need case sensitivity. The exact format depends on your SQL engine.
Using the inputs directory
A common problem: you run vulcan create_external_models and it generates inputs.yaml. But some tables need manual definitions (when Vulcan cannot access their metadata). If you add them to inputs.yaml and run the command again, the command overwrites your manual changes.
Solution: put manual definitions in the inputs/ directory:
How it works:
Vulcan loads
inputs.yaml(orinputs.yml) firstThen it loads all
.yamlfiles frominputs/It merges everything together
Best practice: use create_external_models to manage the main file, and put any tables that need manual definitions in the inputs/ directory. You can then regenerate the main file without losing your manual work.
Gateway-specific external models
If you use isolated systems with multiple gateways, you may have external tables that only exist on specific gateways.
Example: your model uses a gateway variable to select different databases:
When you run with --gateway dev, it queries dev_db.customers. When you run with --gateway prod, it queries prod_db.customers. These are different tables with potentially different schemas.
Solution: run create_external_models with the --gateway flag:
This sets gateway: dev on the external model, so it only loads when that gateway is active. Do this for each gateway that has different external tables.
Validating external data
inputs.yaml is a pure contract file: name, dialect, grains, and columns. It doesn't carry validation rules. To validate upstream data from an external source, use one of these instead:
Standalone audits in
audits/*.sql: write SQL audit rules that select bad rows and reference the external table by name, then attach them as assertions on downstream models. See Assertions for the full syntax.Data Quality checks in
dq/*.yml(kind: dq): non-blocking checks that run after the dependent model builds successfully. See Data Quality for the full syntax.
Both run when the dependent Vulcan model executes, so quality issues in the upstream source are caught before they propagate downstream.
Last updated
Was this helpful?