Configure dbt Core with OpenLineage

This guide shows you how to configure dbt Core to send OpenLineage events to your Ataccama ONE Agentic orchestrator connection.

Before you begin, ensure you have generated an API key and copied your endpoint URL.

Install the OpenLineage integration

pip install openlineage-dbt

When pinning dependencies, add openlineage-dbt==<VERSION> to your requirements.txt alongside dbt-core.

Configure the connection details

Set the OpenLineage endpoint and API key using environment variables:

export OPENLINEAGE_URL=<YOUR_OPENLINEAGE_ENDPOINT_URL>
export OPENLINEAGE_API_KEY=<YOUR_API_KEY>

Optionally, specify a namespace to organize your lineage events:

export OPENLINEAGE_NAMESPACE=<YOUR_NAMESPACE>

For production environments, load the API key from your vault or CI/CD secret store rather than committing it to source control.

Run dbt through the OpenLineage wrapper

Replace dbt commands with dbt-ol to send OpenLineage events:

dbt-ol run

The dbt-ol wrapper supports all standard dbt CLI commands (run, test, build, etc.) and sends OpenLineage events once execution completes.

Enable real-time event streaming (optional)

By default, dbt-ol sends events only after the entire dbt run completes. To send events in real time as each model finishes executing, use the --consume-structured-logs flag:

dbt-ol run --consume-structured-logs

This provides immediate visibility into pipeline progress and captures richer failure context if models fail during execution.

The --consume-structured-logs flag requires openlineage-dbt version 1.26.0 or later.

Verify the connection

Test your configuration with a lightweight model selection:

dbt-ol run --select <MODEL_NAME>

You should see START and COMPLETE events sent to your OpenLineage endpoint in the command output. Check the Overview tab of your connection in Ataccama ONE Agentic to confirm the run appears in the job details.

Troubleshooting

No events appear: Verify OPENLINEAGE_URL and OPENLINEAGE_API_KEY are set correctly in your environment. Check dbt logs for connection errors.
Authentication fails: Ensure your API key is correct and has not been deleted from the connection settings.
Unsupported adapter: The OpenLineage dbt integration currently supports: BigQuery, Snowflake, Spark, Redshift, Athena, Glue, Postgres, ClickHouse, Trino, Databricks, SQL Server, Dremio, and DuckDB.

Additional resources

For additional configuration options and advanced usage, see:

Was this page useful?