Remote Plan Execution
The remote plan execution lets you run any ONE plan on the Ataccama ONE Platform. Plans are typically developed and tested on small data sets or on dedicated environments. This helps you validate the plans and assess how well they perform so that you can make the necessary adjustments before applying them on your actual data. Once the plan works as expected, you can run it remotely on production data in ONE.
Jobs are first sent to the Data Processing Manager (DPM) module, which forwards the job information to a suitable Data Processing Engine (DPE). Actual data processing takes place only in DPE instances.
Configure remote plan execution
Remote plan execution leverages existing controls and concepts of ONE Desktop. Once you connect to the Ataccama ONE Platform, you can choose between the following launch options:
-
Local: A suitable DPE is selected automatically depending on the plan. The plan is first preprocessed in order to detect any steps that require using ONE Spark DPE.
-
Spark: All jobs are sent only to ONE Spark DPE.
To create a new environment for the Ataccama ONE Platform, follow the instructions in Environments, section Add a new environment. In this case, the new environment must be configured to use your Ataccama ONE Platform server, with the Launch type set to ONE Platform launch.
Remote execution on selected DPEs
From 15.4.0, you can choose which engine will execute your plan remotely by specifying a DPE label in ONE Desktop. This is particularly useful when you work with multiple DPE instances, such as hybrid DPEs and standard DPEs, enabling you to direct your plan execution to a specific engine.
As an alternative to configuring the engine for specific plans, you can instead configure a label for the Run DPM Job workflow task and use it to run these plans in an automated way. For more details, see Run DPM Job.
When to use DPE labels
Consider using DPE labels in the following scenarios:
-
Connecting to private on-premise data sources: If your organization has confidential on-premise databases that are accessible only within a private network, you can use a hybrid DPE with a specific label to securely connect and process data without exposing endpoints to the public internet.
-
Optimizing resource allocation: For resource-intensive or complex jobs, you might have a hybrid DPE configured with higher resources (memory, CPU, storage). By specifying a DPE label, you can direct such jobs to the hybrid DPE to benefit from better performance.
Prerequisites
Before configuring DPE labels in ONE Desktop, ensure the following:
-
You are using ONE Desktop version 15.4.0 or later.
-
The DPE label you intend to use must exist in the dpm-admin-console:dpm-admin-console.adoc. To verify you’re using the correct label, open the DPM Admin Console and go to Configuration > DPE Configurations.
-
The DPE engine associated with the label must be running.
-
ONE Desktop must be able to connect to your Ataccama ONE Platform environment.
-
The database drivers are configured consistently between your DPE instance and ONE Desktop, as described in Drivers used for remote execution.
|
For instructions about how to configure a self-managed DPE for an Ataccama Cloud environment and set DPE labels, see Self-Managed DPE Deployment in Ataccama Cloud. For instructions about how to configure DPE labels in the DPM Admin Console, see DPM and DPE Configuration in DPM Admin Console. |
Configure DPE labels for remote execution
To specify a DPE label in ONE Desktop for remote plan execution:
-
In ONE Desktop, select the environment that you want to use from the toolbar. The environment must be configured for Ataccama ONE Platform.
-
Open the plan you wish to execute remotely and in the toolbar, select Run configuration.
-
On the Ataccama DQC Launch tab, in Execution Parameters, provide the name of your DPE in the DPE Label field.
By default, this field is empty. The name you provide must exactly match the label as it appears in the DPM Admin Console.
For example, if your DPE label is
hybrid-dpe, enterhybrid-dpein the field.
-
Select Run to execute your plan.
The plan is then sent to DPM, which forwards it to the DPE associated with the label you provided.
| If you leave the DPE Label field empty, the plan is executed using the default DPE configuration. |
Verify plan execution in DPM Admin Console
After you run the plan, you can check whether it was executed on the DPE you selected.
-
Open the DPM Admin Console.
-
Switch to the Jobs tab.
-
Locate your job in the list and check the Engine column to verify that it is running on the correct DPE instance. The engine label should match the label you specified in ONE Desktop.
Troubleshooting
When using DPE labels, you might encounter the following error: Cannot submit job because there is no (compatible/available) engine.
This occurs if the specified DPE label is not registered in the DPM Admin Console or if the DPE with that label is not running.
To solve this issue, try the following:
-
Verify that the DPE label is correctly entered in ONE Desktop.
-
Ensure the label is configured in the DPM Admin Console (Configuration > DPE Configurations tab).
-
Confirm that the DPE with this label is running without issues.
Remote execution on ONE Spark DPE
| ONE Metadata Reader and ONE Metadata Writer steps are not supported to run on Spark. |
If you want all plans to be executed on the Spark engine, set the run configuration as follows:
-
In ONE Desktop, select the environment that you want to use from the toolbar. The environment must be configured for Ataccama ONE Platform.
-
Open your plan and in the toolbar, select Run configuration.
-
In Data Processing Parameters, select SPARK and provide the name of your cluster.
-
Select Run to execute your plan on ONE Spark DPE. You can track the progress and the outcome of the run on the Run Results tab.
Was this page useful?