Operate On File
Provides 10 potential file operations: Copy, Delete, Exists, Not_exists, Mkdir, Move, Info, Unzip, Zip, and List, which are defined in the Operation parameter.
The task evaluates both /data/tmp and /data/tmp/ paths as a tmp folder, so to operate the folder contents without the folder itself, we recommend using the /data/tmp/* mask.
|
Since version 12, this task contains the functionality of HDFS Operate on File, HDFS Download File, and HDFS Upload File tasks. |
The task can use remote resources (accessible with resource://<resourceName>/<path>/<inputFile>
syntax):
-
HDFS (if your product contains ONE Spark DPE).
-
Amazon S3 and Azure Data Lake Storage Gen1 servers.
Amazon S3 does not operate with a folder structure the way other systems do; it operates only with files.
It allows having a file with forward slash (/
) characters in its filename, which is represented by the browser as a folder.
In other words, when you create a folder on Amazon S3, you create a file with a forward slash in its filename.
The resulting file and filename cause issues with operations such as Delete or Move in the Operate on File workflow task when removing the last file from folder. Therefore, we strongly recommend using the operation Mkdir in all of your workflow tasks whenever it is necessary.
When using the Mkdir operation, your folder is still available even after its last file is deleted. All operations respect the permissions on the filesystem as expected (that is, in case of insufficient permissions on the specified source or target, the task fails).
Copy
Copies source file or folder to the destination folder. The operation automatically preserves target files timestamps.
Name | Type | Description | Expression support | ||
---|---|---|---|---|---|
Operation |
mandatory, must be "COPY" |
Operation name. |
none |
||
Source File |
mandatory |
Path to the source mask on files or folders. The task fails when no file or directory is found to mask. Supports using wildcards. |
semi-expression |
||
Target File |
mandatory |
Path to the destination folder or file:
|
semi-expression |
||
Recursive Flag |
optional Default value: |
Permission to copy subdirectories. |
none |
||
Overwrite Flag |
optional Default value: |
Permission to overwrite files in the destination folder.
The task fails when Overwrite Flag is |
none |
||
Keep Dir Tree Flag |
optional Default value: |
Permission to copy the source file system hierarchy.
Applicable only in case the Source File contains wildcards on multiple path levels (for example, If
|
none |
Delete
Deletes a target file or folder. The task fails when it does not have permissions to delete Target File.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "DELETE" |
Operation name. |
none |
Target File |
mandatory |
Path to the mask on the target file or folder. The task fails when no file or directory is found to mask. Supports using wildcards. |
semi-expression |
Recursive Flag |
optional Default value: |
Permission to delete subdirectories. |
none |
Exists
Verifies the existence of a source file or folder.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "EXISTS" |
Operation name. |
none |
Source File |
mandatory |
Path to the source file or directory. The task fails when the source does not exist. |
semi-expression |
Not exists
Verifies the absence of a source file or folder.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "NOT_EXISTS" |
Operation name. |
none |
Source File |
mandatory |
Path to the source file or directory. The task fails when the source exists. |
semi-expression |
Mkdir
Creates a folder.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "MKDIR" |
Operation name. |
none |
Target File |
mandatory |
Path to the destination folder.
The task fails when the target is a multi-level directory structure from which some of the parent folders are missing and Recursive Flag is |
semi-expression |
Recursive Flag |
optional Default value: |
Permission to make parent directories as needed. |
none |
Move
Moves a source file or folder to the destination folder. The operation automatically preserves target files timestamps.
The task fails when it does not have permissions to delete Source File.
Name | Type | Description | Expression support | ||
---|---|---|---|---|---|
Operation |
mandatory, must be "MOVE" |
Operation name. |
none |
||
Source File |
mandatory |
Path to the source mask on files or folders. The task fails when no file or directory is found to mask. Supports using wildcards. |
semi-expression |
||
Target File |
mandatory |
Path to the destination folder or file:
|
semi-expression |
||
Overwrite Flag |
optional Default value: |
Permission to overwrite files in the destination folder.
The task fails when Overwrite Flag is |
none |
||
Keep Dir Tree Flag |
optional Default value: |
Permission to copy the source file system hierarchy. Applicable only in case the Source File contains wildcards on multiple path levels (for example, If
|
none |
File info
Saves information about a source file or folder to Workflow Variables.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "INFO" |
Operation name. |
none |
Source File |
mandatory |
Path to the source file or folder. The task fails when the source does not exist. |
semi-expression |
Task variable
Name | Description | ||
---|---|---|---|
Name |
Absolute path to the file or folder. |
||
LocalName |
Name of the file or folder. |
||
Size |
Size of the file or folder in bytes. |
||
Timestamp |
Last modification date in
|
Zip
Source files or folders are zipped to a ZIP file.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "ZIP" |
Operation name. |
none |
Source File |
mandatory |
Path to source mask on files or folders. The task fails when no file or directory is found to mask. |
semi-expression |
Target File |
mandatory |
Path to the target ZIP file. The task fails when Target File is an existing directory (that is, the target ZIP file name cannot be the same as an existing directory name). |
semi-expression |
Exclude Parameters |
optional |
Set of mask parameters that are excluded from zipping. |
semi-expression |
Overwrite Flag |
optional Default value: |
Permission to overwrite the output ZIP file.
The task fails when Overwrite Flag is |
none |
Recursive Flag |
optional Default value: |
Permission to zip subdirectories. |
none |
Unzip
Extracts a source ZIP file to the destination folder.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "UNZIP" |
Operation name. |
none |
Source File |
mandatory |
Path to the source file. The task fails when the source does not exist. |
semi-expression |
Target File |
mandatory |
Path to the destination folder. If the target folder structure does not exist, it is created automatically, with respect to the permissions on the filesystem. |
semi-expression |
Overwrite Flag |
optional, Default value: |
Permission to overwrite ZIP file.
The task fails when Overwrite Flag is |
none |
List
Saves information of a source file or folder to the task variable list
.
The values are separated by the specified separator string.
There is also extended listing supported that contains additional information for each file or folder separated by an extra separator.
In that case, values are in the order full name, type ('D' for directory, 'F' for file), last modified time in the format "yyyy-MM-dd HH:mm:ss", and size in bytes.
For example, C:\data\test.txt|F|2023-10-08 12:30:50|45120;C:\data\images|D|2023-05-11 05:00:11|0
.
Name | Type | Description | Expression support | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Operation |
mandatory, must be "LIST" |
Operation name. |
none |
||||||||||||||||||||
Source File |
mandatory |
Path to the source file. The task fails when the source does not exist. |
semi-expression |
||||||||||||||||||||
Mask |
mandatory |
File name mask used to select files and/or folders to list. |
semi-expression |
||||||||||||||||||||
List File Flag |
optional Default value: |
Include files in listing. |
none |
||||||||||||||||||||
List Folders Flag |
optional Default value: |
Include folders in listing. |
none |
||||||||||||||||||||
Recursive Flag |
optional Default value: |
Also list contents of any subdirectories. All subdirectories are scanned regardless of the mask or condition. |
none |
||||||||||||||||||||
Separator |
mandatory |
String separating the items found in the resulting variable value. |
none |
||||||||||||||||||||
Extended Listing Flag |
optional Default value: |
Generates extended listing with full name, type ('D' for directory, 'F' for file), last modified time in "yyyy-MM-dd HH:mm:ss" format, and size in bytes.
For example: |
none |
||||||||||||||||||||
Extended Info Separator |
mandatory |
String separating the extended information items for each file or folder in the result. |
none |
||||||||||||||||||||
Continue On Error flag |
optional Default value: |
When specific file or folder information cannot be used, it is reported to the log and the task continues without failing. |
none |
||||||||||||||||||||
Condition |
optional |
Expression evaluated for each candidate entry to check if it should be included in the output or not. Supports the following variables:
|
expression |
Was this page useful?