If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Workflow Basics

Technical Workflow is an out-of-the-box orchestration tool that lets you automate data management processes via a set of predefined tasks that cover a variety of scenarios. Any workflow can be easily initiated on demand or scheduled via the Scheduler component.

This article describes the main concepts related to workflows to help you get started.

When to use workflows?

In general, workflows are meant to fulfill the need for automated processes that surround running a ONE plan. A typical use case for using workflows would be downloading data, processing it, and then uploading it back to the server, all as a matter of running a single workflow, capable of taking care of all these operations in the order specified and reporting whether all operations have been successfully executed.

For more workflow examples, see Workflow Tutorials.

Tasks

Workflows consist of tasks and, if necessary, links between them. Each task represents one operation to be run while links provide the dependencies so a task won’t be run before its predecessors.

There is no limit to the number of tasks in one workflow: there can be a single one or several tasks running in the workflow at a time since workflows allow parallelism in tasks that do not depend on each other (that is, there are no links between them).

Some of the tasks require that ONE Runtime Server is running. This is indicated by a yellow icon in the upper-left corner of the task. See the Workflow Tasks section in the documentation for reference information about tasks.

Links

Tasks can be connected to each other by links, creating dependencies between tasks.

A task is run only after all of the connected predecessors have finished running.

Endpoints

Every task has two endpoints by which it connects to other tasks:

normal endpoint: Linked tasks that follow are executed only if the task has been executed successfully.
error endpoint: Linked tasks that follow are executed only if task execution failed.

Error links require that the Continue On Failure global option is enabled (it is turned off by default) as the run is stopped immediately when a failure is encountered.

Linking styles

A task can be connected to as many other tasks as possible by links, as a predecessor or a successor, and more than one link can lead from or to a single endpoint.

At the same time, a task does not have to be connected to any other tasks at all. In this case, tasks are run at the same time unless the workflow has limited resources (see Workflow Resource Management).

Dependent

When the following workflow is executed, 10_build_tld_lkp and 11_build_name_lkp tasks run simultaneously, and 12_wait_for_build_processes waits until both those tasks are done.

Independent

When the following workflow is executed, 001_hello_world and 10_run_plan tasks start and run simultaneously as they have no predecessors to wait for. The 15_report_success task waits for 10_run_plan to finish successfully.

Conditioned links

It is possible to set a condition (a Boolean expression) on a link. In other words, the consecutive task is run only if the condition is satisfied, otherwise, it is skipped.

For example, a conditional link can be used to fork the process into two—given the result from the source task, a different branch of subsequent tasks can be chosen based on the condition. In this case, two links lead from the source task.

In the following example, the Read SQL Result task checks the database for existing tables. The condition on the link then checks whether a particular table is found. If the condition is satisfied, the Execute SQL task drops the table.

Accept mode

In case there are several links leading to a certain task and some of them are not accepted (because the source task fails or the condition on the link is not met), the Accept Mode (located on the General tab of task settings) determines whether the task is run or not.

There are two accept modes:

ALL_VALID - All incoming links must be accepted.
AT_LEAST_ONE - At least one incoming link must be accepted.

The following example demonstrates when the two accept modes can be useful:

In order to report success (20_report_success task), all plans must be successfully executed (links highlighted in green leading from normal endpoints).
In order to report failure (21_report_failure task), at least one plan needs to fail (gray links leading from error endpoints).

For detailed information, see Workflow States and Evaluation.

Using expressions

Workflows support the ONE expression syntax and most of its functions and operators. See Expressions in Workflows.

Using variables

To pass external information to the workflow or pass a parameter to a ONE plan, you can use variables. See Workflow Variables for more information.

Run a workflow

It is possible to execute workflows in a variety of ways from different interfaces, including ONE Desktop, command line, and ONE Runtime Server Admin. See Execute Workflows.

Schedule a workflow

It is also possible to schedule workflows for regular execution with minimal maintenance via the Scheduler component. The Scheduler interface in the ONE Runtime Server Admin lets you see all scheduled jobs along with their scheduling plans, descriptions, the last and next executions, and their statuses.

You can also run, enable, and disable jobs from the ONE Runtime Server Admin. For information about setting up and using the scheduling component, see Scheduler.

Was this page useful?