Configure Apache Airflow with OpenLineage

This guide shows you how to configure Apache Airflow to send OpenLineage events to your Ataccama ONE Agentic orchestrator connection.

Before you begin, ensure you have generated an API key and copied your endpoint URL.

Install the OpenLineage provider

Choose the appropriate package based on your Airflow version:

For versions older than 2.7

pip install openlineage-airflow

For version 2.7 and newer (recommended)

pip install apache-airflow-providers-openlineage

Enable OpenLineage

Ensure OpenLineage is enabled in your airflow.cfg:

[openlineage]
disabled = False

disabled = False is the default value, so you can also omit this setting entirely if you want OpenLineage enabled.

Configure the transport

The transport defines how Airflow sends OpenLineage events to Ataccama ONE Agentic. You can configure it through:

External YAML/JSON file - Cleaner approach, recommended for managing secrets
Inline JSON in airflow.cfg - Direct configuration in the [openlineage] transport key
Environment variable - Using AIRFLOWOPENLINEAGETRANSPORT, useful for containerized deployments

Below are environment-agnostic configuration methods - use whichever is most suitable your deployment approach.

For production environments, avoid committing API keys to source control. Use external configuration files with restricted access permissions or environment variables managed by your secrets management system.

Method 1: External configuration file (verified)

Create an openlineage.yml file:

transport:
  type: http
  url: https://<YOUR_INSTANCE>.ataccama.one/
  endpoint: gateway/openlineage/<CONNECTION_ID>/events
  auth:
    type: api_key
    apiKey: <YOUR_API_KEY>

Split your OpenLineage endpoint URL into url (base) and endpoint (path). For example, <YOUR_INSTANCE>.ataccama.one/gateway/openlineage/<CONNECTION_ID>/events becomes url: <YOUR_INSTANCE>.ataccama.one/ and endpoint: gateway/openlineage/<CONNECTION_ID>/events.

You can use this file in two ways:

Option A: Automatic detection
Place openlineage.yml directly in your Airflow config directory. Many Airflow images/providers will automatically detect and load it.

Option B: Explicit path
Reference the file explicitly in airflow.cfg:

[openlineage]
disabled = False
transport = /opt/airflow/openlineage.yml

Method 2: Inline configuration in airflow.cfg

The following methods are documented in Apache Airflow but have not been verified with Ataccama ONE Agentic.

[openlineage]
disabled = False
transport = {"type": "http", "url": "https://<YOUR_INSTANCE>.ataccama.one/", "endpoint": "gateway/openlineage/<CONNECTION_ID>/events", "auth": {"type": "api_key", "apiKey": "<YOUR_API_KEY>"}}

Method 3: Environment variables

The following methods are documented in Apache Airflow but have not been verified with Ataccama ONE Agentic.

export AIRFLOW__OPENLINEAGE__TRANSPORT='{"type": "http", "url": "https://<YOUR_INSTANCE>.ataccama.one/", "endpoint": "gateway/openlineage/<CONNECTION_ID>/events", "auth": {"type": "api_key", "apiKey": "<YOUR_API_KEY>"}}'

# Optionally, explicitly enable OpenLineage (disabled=False is the default)
export AIRFLOW__OPENLINEAGE__DISABLED=False

Restart Airflow

After configuration, restart your Airflow scheduler and workers for the changes to take effect.

Troubleshooting

No events appear: Ensure disabled = False in the [openlineage] section. Check Airflow scheduler and worker logs for OpenLineage errors.
Authentication fails: Verify your API key is correct and the transport format matches your provider version.
Provider not found: The official Airflow Docker images may already include the provider. Otherwise, install apache-airflow-providers-openlineage explicitly.

Additional resources

For additional configuration options and methods, see:

Was this page useful?