If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Run DPM Job

Starts processing of a plan or component using remote Data Processing Module (DPM).

Configuration parameters

Name Type Description

Main Plan

mandatory

Points to the main ONE plan that should be executed.

Main Plan Path

optional

The path to the main plan at the remote site. If not defined, the local path is used by default.

Runtime Configuration

optional

Points to the runtime configuration that should be applied.

ONE Platform

mandatory

The name of your instance of ONE Platform.

Priority

optional

Sets the priority of the job in the DPM job queue. The default value is 0.

The higher the value, the higher the priority.

Zip

optional

If set to true, when running a job locally, processing results are zipped and stored in the temporary workflow directory (starting with wfinst).

Async

optional

If set to true, DPM waits until the job is finished and downloads the results. Otherwise, DPM only submits the job.

By default, jobs are executed synchronously (async=false).

Force Cluster Launch

optional

If set to true, the workflow task is always processed using ONE Spark DPE.

Cluster Name

optional

The name of the Spark cluster on which the job should be executed.

To get a list of available clusters, run the command datasourceConfigList (no parameters required).

For example:

$ ./start.sh datasourceConfigList -h localhost -p 8531 -u admin -p admin

Cluster User

optional

Sets the Spark cluster credentials. Used if the credentials are not provided in the default configuration or if you want to use another set of credentials.

If Spark credentials are not provided, the authentication is skipped.

Cluster Password

optional

Sets the password for the Spark user. If not provided, the password file is checked instead.

Cluster Password File

optional

Points to the file that contains the password for the Spark cluster.

Working Dir

optional

The absolute path of the working directory from which relative paths are computed. If not set, it defaults to the current working folder. If set to /, all relative paths are resolved to the root.

Must be defined as an absolute path.

DPE Label

optional

Assigns Data Processing Engines (DPE) with the matching label to the DPM job. For more information, see Set a DPE label.

Root Dir

optional

For Unix-like paths, if not specified, the system default is used (/).

Files

optional

Points to additional input files, such as lookups, components, CSV files:

Path (optional): Path to the file on ONE Platform.
Resource: Enter the local path. You can also use wildcards here.

Links

optional

Points to files from ONE Object Storage. Consists of two values, the first one being the path that Executor uses when running the job, the second one a link that refers to the actual location of the file in the Object Storage.

Path Vars

optional

Defines additional path variables. For more information about workflow variables, see Expressions in Workflows. Paths and other string values to files need to be enclosed in double quotes, for example: "path".

Arguments

optional

If using parametrized components, this parameter is used to configure those parameters. Each argument takes two values: a key and a value.

Mount Drivers

optional

Used if the job needs access to a database. DPM and DPE add this database driver into the runtime configuration, in the <drivers> element.

If a connection is already defined, it should point to the driver name (<datasource driverName="<driver_name>" …>).

Set a DPE label

A DPE label is a label that is used to group together DPEs that share the same configuration. Or, you can use the label to assign the same configuration to the grouped DPEs.

When creating a Run DPM Job task, you have the option to set a DPE label that is then matched against labels assigned to available DPEs. To set a DPE label to the workflow task, access the Run DPM Job Configuration.

If no DPE label is set, DPM uses the known capabilities of DPEs and weighs them against the requirements to determine which DPEs run the job. For more information, see Constraints Configuration.

The DPE Configurations section of the DPM Admin Console displays a list of labels and the configurations that they have assigned. From the list, you can view and manage DPEs that are associated with each label.

For more information, see dpm-admin-console:dpm-and-dpe-configuration.adoc, section Configure DPE labels.

Was this page useful?