Ataccama ONE Agentic ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
Downloads

Run DQ Evaluation from Pipelines

The REST API is currently in Early Access Preview. The API specification and endpoints might change before being marked as stable.

We recommend testing thoroughly and being prepared to adapt to potential changes in future releases.

This guide shows you how to run DQ evaluation as a step in your data pipeline or orchestration workflow: triggering DQ evaluation automatically and stopping the pipeline if quality thresholds are not met.

Before using the Data Quality API, configure authentication as described in API Authentication.

How DQ evaluation works in a pipeline

In the API, DQ evaluation runs as one part of DQ monitor processing. A processing is a single run of a DQ monitor that can include DQ evaluation, profiling, or both, depending on the monitor configuration. The run also includes any dependent tasks configured on the monitor, such as anomaly detection, the export of invalid records, or the collection of invalid record samples.

The decision signals used in this guide come from the DQ evaluation part. The API does not provide profiling results, so for a monitor that runs profiling only, your pipeline can check only that the processing completed.

The processing runs asynchronously: you start it, it runs in the background, and your pipeline polls for completion before retrieving the results.

Ataccama ONE evaluates the data and reports the results, including any threshold breaches. You implement in your orchestration tool whether the pipeline should stop or continue based on those results. Running DQ evaluation from a pipeline follows this workflow:

  1. Identify the DQ monitor for your catalog item (only needed for secondary monitors).

  2. Start the processing on the monitor.

  3. Poll for completion to know when results are ready.

  4. Retrieve the results of the evaluation.

  5. Decide whether to proceed or stop the pipeline based on the results.

Step 1: Identify the DQ monitor (secondary monitors only)

Each catalog item has a primary DQ monitor that controls how its processing runs. For primary monitors, you can skip this step and use primary as the dqMonitorUrn in the following steps.

For secondary monitors, provide the monitor’s URN, in the format urn:ata:{your-tenant}:data-quality:dq-monitor:{dq-monitor-uuid}.

To obtain the URN, list the monitors of the catalog item: see List monitors. For more information about URNs, see URN Reference.

Step 2: Start the DQ monitor processing

Start the processing by sending a POST request to the processings endpoint. The operation runs asynchronously: the API responds immediately with 202 Accepted and links for tracking the processing.

Only one processing can run at a time for a given DQ monitor. If a processing is already running, the request returns 409 Conflict.

To run the primary monitor, use primary as the dqMonitorUrn. To run a secondary monitor, use the monitor’s URN.

Request: Start the DQ monitor processing
curl -X POST "https://{your-environment}.ataccama.one/api/data-quality/v1/catalog-items/{catalogItemUrn}/dq-monitors/{dqMonitorUrn}/processings" \
  -H "Authorization: Bearer {access_token}"

The response contains:

  • polling.href: Link to poll for the processing state. The same link is returned in the Location response header.

  • result.href: Link where the DQ results are available after the processing finishes.

    This link is missing if DQ evaluation is not enabled in the DQ monitor (the processing then runs profiling only), or if the job initialization takes longer than usual. If the link is missing because of slow initialization, the processing has still started: poll for completion as usual, then retrieve the results directly from the results endpoint (see Step 4: Retrieve the results).

Response: Links for tracking the processing
{
  "polling": {
    "href": "https://{your-environment}.ataccama.one/api/processing/v1/workflows/{workflowUrn}"
  },
  "result": {
    "href": "https://{your-environment}.ataccama.one/api/data-quality/v1/catalog-items/{catalogItemUrn}/dq-monitors/{dqMonitorUrn}/processings/{processingUrn}/dq-results"
  }
}

Response codes

The request to start the processing returns these status codes:

Status code Description

202 Accepted

Processing started successfully (returns polling and result links).

400 Bad Request

Invalid request parameters or body (for example, a malformed URN). The response body includes a type, title, detail, and optionally an errors array with specific validation failures and JSON Pointer references to the problematic fields.

401 Unauthorized

Invalid or expired access token.

403 Forbidden

Insufficient permissions to access the requested resource.

404 Not Found

Catalog item or DQ monitor does not exist.

409 Conflict

A processing is already running for the given DQ monitor.

500 Internal Server Error

Unexpected error. Retry later.

Step 3: Poll for completion

Poll the processing workflow

Poll the link from the polling.href field (or the Location header) to track the processing. The link leads to the processing workflow resource (/processing/v1/workflows/{workflowUrn}), which reports its state in the status field.

Continue polling while status is PENDING or RUNNING, and stop when it changes to any other value. The response includes a Retry-After header with the number of seconds to wait before polling again; the header is present until the workflow reaches a terminal state.

The status field can have one of the following values:

  • PENDING: The processing has not started yet. Keep polling.

  • RUNNING: The processing is running. Keep polling.

  • FINISHED: The processing finished successfully. Results are ready to retrieve.

  • FAILED: The processing did not complete. No results are available for this run.

  • CANCELED: The processing was canceled. No results are available for this run.

  • SKIPPED: The processing was skipped (for example, because the monitor has no rules assigned). No results are available for this run.

Response: State of the processing workflow
{
    "urn": "urn:ata:{your-tenant}:processing:workflow:f783ee2b-b990-4ee1-aef7-53186433fe54",
    "type": "DQ-JOB",
    "startedAt": "2026-07-15T08:24:21Z",
    "status": "FINISHED",
    "name": "CUSTOMERS / Primary Monitor",
    "owner": "jane.doe",
    "finishedAt": "2026-07-15T08:26:07Z",
    "duration": 106,
    "errors": []
}
The list of possible values is extensible, so a future release might introduce a status that your pipeline does not handle. Continue to the results only when the status is FINISHED. This way, an unexpected value stops the pipeline instead of passing as a success (see Handle failed processing runs).

Alternative: list recent processings

You can also check the state by listing recent processings.

The processings list reports the state in a different field with different values: processingState, with values such as SUCCESS, FAILED, CANCELED, IN_PROGRESS, and UNKNOWN. When using this method, continue polling while processingState is IN_PROGRESS and proceed only on SUCCESS.

Request: List the most recent processing
curl -X GET "https://{your-environment}.ataccama.one/api/data-quality/v1/catalog-items/{catalogItemUrn}/dq-monitors/{dqMonitorUrn}/processings?size=1&sort=-finishedAt" \
  -H "Authorization: Bearer {access_token}"
Response: The most recent processing and its state
{
  "meta": {
    "prev": null,
    "next": null
  },
  "data": [
    {
      "processingUrn": "urn:ata:{your-tenant}:data-quality:processing:789e4567-e89b-12d3-a456-426614174000",
      "dqMonitorUrn": "urn:ata:{your-tenant}:data-quality:dq-monitor:123e4567-e89b-12d3-a456-426614174000",
      "catalogItemUrn": "urn:ata:{your-tenant}:catalog:catalog-item:456e4567-e89b-12d3-a456-426614174000",
      "processingState": "SUCCESS",
      "startedAt": "2026-01-20T14:30:00Z",
      "finishedAt": "2026-01-20T14:35:00Z"
    }
  ]
}

To sort, filter, and paginate the processings list, use the query parameters shown in the example request (such as size and sort). For the full list of query parameters, see Browse processing history.

Step 4: Retrieve the results

After the processing completes successfully (either workflow status is FINISHED, or processingState is SUCCESS), retrieve the results from the result.href link returned in Step 2: Start the DQ monitor processing, or use the results endpoint directly. See Retrieve DQ Results for External Systems for detailed information on retrieving results.

An example to get the latest results:

Request: Retrieve the latest DQ results
curl -X GET "https://{your-environment}.ataccama.one/api/data-quality/v1/catalog-items/{catalogItemUrn}/dq-monitors/{dqMonitorUrn}/processings/latest/dq-results" \
  -H "Authorization: Bearer {access_token}"
Use latest as the processingUrn to get results from the most recently completed processing.

The response includes the signals your pipeline can base the decision on: for example, overallQuality (pass/fail counts), overallDqFindings (threshold breaches), or the more detailed dimensionResults and dqAttributeResults. For the complete response structure, see Retrieve DQ Results for External Systems or the DqResults schema in the Data Quality API specification.

Response: DQ results used for the pipeline decision
{
  "processingUrn": "urn:ata:{your-tenant}:data-quality:processing:789e4567-e89b-12d3-a456-426614174000",
  "overallQuality": {
    "passedCount": 141750,
    "failedCount": 8250
  },
  "dimensionResults": [...],
  "overallDqFindings": [
    {
      "findingUrn": "urn:ata:{your-tenant}:data-quality:dq-finding:184f6c1d-98cd-4823-b51a-c4fa5dad2a96",
      "thresholdUrn": "urn:ata:{your-tenant}:data-quality:dq-threshold:232ca0c0-0000-7000-0000-000008ce37f5",
      "status": "ACTIVE",
      "summary": "CUSTOMERS DQ 67% (threshold: 80%)"
    }
  ],
  "dqAttributeResults": [...]
}

Step 5: Decide whether to proceed or stop the pipeline

Stop the pipeline when the results show that data quality does not meet your standards.

ONE evaluates the data and reports threshold breaches in the results but does not stop your pipeline. You implement the decision logic (such as checking the results and halting downstream tasks) in your orchestration tool.

Depending on how quality standards are defined in your organization, the decision can be based on any signal in the results, from the overall pass and fail counts to findings on individual rule instances (dqAttributeResults[].ruleInstanceResults[].dqFindings). The following sections describe two common approaches.

DQ thresholds are configured on the DQ monitor: the monitor-level overallDqThresholds and per-rule dqThresholds each define a percentage of valid records (from 0 to 100). When DQ evaluation results in fewer valid records than the configured percentage, the evaluation triggers an alert finding, and the finding appears in the DQ results. For details on configuring thresholds, see Manage DQ Monitors via API.

In your pipeline, check the results for findings with the status ACTIVE:

  • overallDqFindings: Threshold breaches at the monitor level.

  • dqAttributeResults[].ruleInstanceResults[].dqFindings: Threshold breaches for individual rule instances, if the decision needs to consider specific rules.

Stop the pipeline only on ACTIVE findings. A finding with the status CLOSED means the threshold is no longer breached.

An example implementation in an orchestrator task (for example, an Airflow task or a dbt Cloud job step):

Example: Stop the pipeline on an active finding
results = get_dq_results()  # GET .../processings/latest/dq-results

active_findings = [
    f for f in results.get("overallDqFindings", [])
    if f["status"] == "ACTIVE"
]

if active_findings:
    summaries = "; ".join(f["summary"] for f in active_findings)
    raise Exception(f"DQ threshold breached, blocking downstream tasks: {summaries}")

Failing the task this way stops the downstream tasks in most orchestrators. The summary of each finding is human-readable (for example, CUSTOMERS DQ 67% (threshold: 80%)), so it can be logged or sent to alerting without modification.

Stop the pipeline on the overall pass rate

If no thresholds are configured on the monitor, or if the pipeline has its own quality limit, compute the pass rate from overallQuality and compare it against the limit:

Example: Stop the pipeline below a pass rate limit
quality = results["overallQuality"]
pass_rate = quality["passedCount"] / (quality["passedCount"] + quality["failedCount"])

if pass_rate < 0.95:
    raise Exception(f"DQ pass rate {pass_rate:.1%} below the 95% limit, blocking downstream tasks")

Handle failed processing runs

Stop the pipeline also when the processing itself does not complete, since no trustworthy results are available in that case.

Block downstream tasks on any terminal state other than a success (see Step 3: Poll for completion):

  • When polling the processing workflow, block on any status other than FINISHED (FAILED, CANCELED, or SKIPPED).

  • When listing recent processings, block on any processingState other than SUCCESS (FAILED, CANCELED, or UNKNOWN).

API reference

For detailed API reference, including all endpoints, parameters, request and response schemas, and examples, see Data Quality API specification. For the processing workflow resource used in Step 3: Poll for completion, see Processing API specification.

Next steps

Was this page useful?