Iterate
This task is used to implement a for-each cycle. It sequentially runs child workflows for each input value (for example, a file or database row).
The task writes the set of retrieved values (files, database rows) into the memory immediately after being launched and runs child workflows based on that set, ignoring any filesystem or database changes that might affect the original set of values.
Properties
Name | Type | Description | Expression support |
---|---|---|---|
Iterable |
mandatory |
Implementation and configuration of additional iterators. Possible implementations: File Iterator, Set Iterator, SQL Row Iterator. |
none |
Workflow Id |
mandatory |
ID of a workflow to run. Workflow ID specification depends on the workflow execution context:
|
semi-expression |
Iteration Type |
mandatory |
SERIAL or PARALLEL run. |
none |
Break On Error |
optional;
Default value: |
If an error occurs, the iteration is either terminated immediately ( |
none |
The task finishes with the result OK
, unless any of the child workflows does not start correctly or finishes with a wrong result.
The Break On Error parameter has no impact on the task result.
When the iterator has no values, the task finishes with the result OK
.
File Iterator
The task runs child workflows for all files in the Directory.
When Iteration Type is set to SERIAL, the task iterates in the order corresponding to the alphabetical order of files specified in Directory and Mask. |
Properties
Name | Type | Description | Expression support | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Directory |
mandatory |
Location of files.
The directory can be local, on Amazon S3 or Azure Data Lake Storage Gen1 server, and on HDFS (if your product contains ONE Spark DPE).
The remote resources are accessible with the |
semi-expression |
||||||||||||||||||||
Mask |
mandatory |
File name mask used to select files to iterate over. Supports the following wildcards:
|
semi-expression |
||||||||||||||||||||
Iterable Item |
mandatory |
Name of the variable that is passed to the child workflow (that is, a variable with the same name should be defined in the child workflow as an input variable). The variable value would be the processed file name (including full path) in each iteration. |
none |
||||||||||||||||||||
Continue on error Flag |
optional Default value: |
When file or folder information cannot be used, the task reports to log and continues, it does not fail. |
none |
||||||||||||||||||||
Recursive |
optional Default value: |
Files in all subdirectories of the directory are either used for iteration too ( |
none |
||||||||||||||||||||
Condition |
optional |
Expression evaluated for each candidate entry to check if it should be included in the output or not. Supports the following variables:
|
expression |
||||||||||||||||||||
Parameter Mapping |
optional |
Set of user defined parameters to pass to the workflow as global variable values. |
Parameter mapping properties
Name | Type | Description | Expression support |
---|---|---|---|
Parameters |
optional |
Set of user defined parameters to pass to the workflow as global variable values. |
none |
Example
<!-- File iterator with the mask property defined using a constant string -->
<iterable>
<directory>./inputs</directory>
<iterableItem>IterationValue</iterableItem>
<mask>*.csv</mask>
<parameterMapping>
<parameters>
<parameter expression="'some_text'" name="childrenVar01"/>
<parameter expression="var01" name="childrenVar02"/>
</parameters>
</parameterMapping>
<recursive>true</recursive>
</iterable>
<!-- File iterator with the mask property defined using a variable -->
<iterable>
<directory>./inputs</directory>
<iterableItem>IterationValue</iterableItem>
<mask>${FILE_MASK}</mask> <!-- if the FILE_MASK variable contains for example a value "*.txt", the task will iterate over all .txt files found in the defined directory -->
<parameterMapping>
<parameters>
<parameter expression="'some_text'" name="childrenVar01"/>
<parameter expression="var01" name="childrenVar02"/>
</parameters>
</parameterMapping>
<recursive>true</recursive>
</iterable>
Set iterator
The task runs child workflows for all items defined in the Set.
When Iteration Type is set to SERIAL, the task iterates in the order corresponding to the specified order of items defined in Set. |
Properties
Name | Type | Description | Expression support |
---|---|---|---|
Set |
mandatory |
Set to be used for iteration. |
semi-expression |
Separator |
mandatory |
Value of separator that is used. |
none |
Iterable Item |
mandatory |
Name of the variable that is passed to the child workflow (that is, a variable with the same name should be defined in the child workflow as an input variable). The value of the variable is taken from the Set in each iteration. For example, if you define Set as |
none |
Parameter Mapping |
mandatory |
Set of user defined parameters to pass to the workflow as global variable values. |
Parameter mapping properties
Name | Type | Description | Expression support |
---|---|---|---|
Parameters |
optional |
Set of user defined parameters to pass to the workflow as global variable values. |
none |
Example
<iterable>
<iterableItem>IterationValueName</iterableItem>
<parameterMapping>
<parameters>
<parameter expression="'some_text'" name="childrenVar01"/>
<parameter expression="var01" name="childrenVar02"/>
</parameters>
</parameterMapping>
<separator>,</separator>
<set>1,2,5</set>
</iterable>
SQL row iterator
The task runs child workflows for each row returned by an SQL query.
When Iteration Type is set to SERIAL, the iteration order corresponds to the order of results returned by Query (so ORDER BY clause can be used to affect the iteration order).
|
Properties
Name | Type | Description | Expression support |
---|---|---|---|
Connection Name |
mandatory |
Connection name to use for query invocation. |
semi-expression |
Query |
mandatory |
Query to execute. |
semi-expression |
Mapping |
optional |
Set of parameters derived from the result set to pass to the workflow as global variable values. |
none |
Parameter Mapping |
optional |
Set of user defined parameters to pass to the workflow as global variable values. |
Mapping properties
Name | Type | Description | Expression support |
---|---|---|---|
Result Set |
optional |
Defines a set of output parameters to be read from the query’s result set. Applicable when running SQL queries. |
none |
Result set properties
The value is read from the first row of the first available result set. |
Name | Type | Description | Expression support |
---|---|---|---|
Name |
mandatory |
Defines the name of the mapping. The value read from the mapping is available under this name in the condition and saved in the task variables. |
none |
Index |
mandatory |
Defines the index of the value in the result set.
Indexed from |
none |
Parameter mapping properties
Name | Type | Description | Expression support |
---|---|---|---|
Parameters |
optional |
Set of user defined parameters to pass to the workflow as global variable values. |
none |
Example
<iterable>
<connectionName>derby</connectionName>
<mapping>
<resultSet>
<resultSetMapping name="IterationValue" index="2"/>
</resultSet>
</mapping>
<parameterMapping>
<parameters>
<parameter expression="'some_text'" name="childrenVar01"/>
<parameter expression="var01" name="childrenVar02"/>
</parameters>
</parameterMapping>
<query>select * from Table</query>
</iterable>
Was this page useful?