Use Data Quality Gates Locally in Python
Use this page when you want to validate records in native Python code without sending them over the network.
Supported ONE versions
DQ Gates are supported in the following versions of Ataccama ONE:
-
Ataccama ONE 16.3.0 and later.
Before you begin
Make sure you’ve reviewed prerequisites and limitations before continuing.
The exported firewall definition ZIP file contains the firewall definition together with other assets needed for local evaluation, such as reference data and metadata files. Make sure the ZIP file is accessible to the Python runtime at evaluation time.
How it works
The Python flow has four steps:
-
Install the ONE Python SDK with Python DQ Engine.
-
Authenticate to Ataccama ONE.
-
Download and export the firewall definition to a ZIP file.
-
Evaluate records locally in Python.
Installation
Install the ONE Python SDK and Python DQ Engine packages:
pip install ataccama_one-<version>-py3-none-any.whl ataccama_one_expressions-<version>-py3-none-any.whlConfiguration
Create an authenticated client
Choose one of the authentication options described in Authentication setup.
Then create a client:
import os
from dotenv import find_dotenv, load_dotenv
from ataccama_one import Client, OpenIdConnectAuth
def get_env_var(name: str) -> str:
value = os.getenv(name)
if value is None:
raise ValueError(f"Environment variable {name} is required but not set.")
return value
load_dotenv(find_dotenv(".env"))
client = Client(
platform_version=get_env_var("ATACCAMA_PLATFORM_VERSION"),
url=get_env_var("ATACCAMA_INSTANCE_URL"),
auth=OpenIdConnectAuth(
client_id=get_env_var("ATACCAMA_CLIENT_ID"),
client_secret=get_env_var("ATACCAMA_CLIENT_SECRET"),
keycloak_host=get_env_var("ATACCAMA_KEYCLOAK_HOST"),
keycloak_realm=get_env_var("ATACCAMA_KEYCLOAK_REALM"),
),
)If you use custom API endpoints, see Custom URL resolution for self-managed deployments. If you run into SSL or certificate issues while connecting to Ataccama ONE, see Data Quality Gates Troubleshooting.
Fetch and export the firewall definition
Local Python execution works with exported DQ firewall definitions.
firewall_id = get_env_var("YOUR_DQ_FIREWALL_ID")
client.export_firewall_definition(firewall_id, save_to="firewall_definition.zip")If you want to fetch multiple firewalls at once, use client.get_firewall_definitions(), which returns a Python iterator:
# Adjust `firewall_filter` to limit which firewalls are fetched from Ataccama ONE
firewall_filter = None
for firewall in client.get_firewall_definitions(filter=firewall_filter):
client.export_firewall_definition(
firewall.firewall_id,
save_to=f"{firewall.name}.zip",
)For details about filtering fetched firewalls, see Firewall filtering.
Evaluate data locally
Load the exported firewall definition ZIP file into the local Python runtime. The ZIP contains the firewall definition together with other assets needed for local evaluation, such as reference data and metadata files.
After the ZIP is loaded, validate_records(…) evaluates the input records locally in Python without sending them to Ataccama ONE.
from ataccama_one.local import DqFirewall
firewall = DqFirewall.from_file("continent_firewall_definition.zip")
results = firewall.validate_records([
["invalid"],
["africa"],
["america"],
["prague"],
["south america"],
["europe"],
])
print(results.record_results)Was this page useful?