Lead your team forward
OCT 24 / 9AM ET Register nowOperate On File
Provides nine potential file operations: Copy, Delete, Exists, Not_exists, Mkdir, Move, Info, Unzip, and Zip, which are defined in the Operation parameter.
The task evaluates both /data/tmp and /data/tmp/ paths as a tmp folder, so to operate the folder contents without the folder itself, we recommend using the /data/tmp/* mask.
|
Since version 12, this task contains the functionality of HDFS Operate on File, HDFS Download File, and HDFS Upload File tasks. |
The task can use remote resources (accessible with resource://<resourceName>/<path>/<inputFile>
syntax):
-
HDFS (if your product contains ONE Spark DPE).
-
Amazon S3 and Azure Data Lake Storage Gen1 servers.
Amazon S3 does not operate with a folder structure the way other systems do; it operates only with files.
It allows having a file with forward slash (/
) characters in its filename, which is represented by the browser as a folder.
In other words, when you create a folder on Amazon S3, you create a file with a forward slash in its filename.
The resulting file and filename cause issues with operations such as Delete or Move in the Operate on File workflow task when removing the last file from folder. Therefore, we strongly recommend using the operation Mkdir in all of your workflow tasks whenever it is necessary.
When using the Mkdir operation, your folder is still available even after its last file is deleted. All operations respect the permissions on the filesystem as expected (that is, in case of insufficient permissions on the specified source or target, the task fails).
Copy
Copies source file or folder to the destination folder. The operation automatically preserves target files timestamps.
Name | Type | Description | Expression support | ||
---|---|---|---|---|---|
Operation |
mandatory, must be "COPY" |
Operation name. |
none |
||
Source File |
mandatory |
Path to the source mask on files or folders. The task fails when no file or directory is found to mask. Supports using wildcards. |
semi-expression |
||
Target File |
mandatory |
Path to the destination folder or file:
|
semi-expression |
||
Recursive Flag |
optional Default value: |
Permission to copy subdirectories. |
none |
||
Overwrite Flag |
optional Default value: |
Permission to overwrite files in the destination folder.
The task fails when Overwrite Flag is |
none |
||
Keep Dir Tree Flag |
optional Default value: |
Permission to copy the source file system hierarchy.
Applicable only in case the Source File contains wildcards on multiple path levels (for example, If
|
none |
Delete
Deletes a target file or folder. The task fails when it does not have permissions to delete Target File.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "DELETE" |
Operation name. |
none |
Target File |
mandatory |
Path to the mask on the target file or folder. The task fails when no file or directory is found to mask. Supports using wildcards. |
semi-expression |
Recursive Flag |
optional Default value: |
Permission to delete subdirectories. |
none |
Exists
Verifies the existence of a source file or folder.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "EXISTS" |
Operation name. |
none |
Source File |
mandatory |
Path to the source file or directory. The task fails when the source does not exist. |
semi-expression |
Not exists
Verifies the absence of a source file or folder.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "NOT_EXISTS" |
Operation name. |
none |
Source File |
mandatory |
Path to the source file or directory. The task fails when the source exists. |
semi-expression |
Mkdir
Creates a folder.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "MKDIR" |
Operation name. |
none |
Target File |
mandatory |
Path to the destination folder.
The task fails when the target is a multi-level directory structure from which some of the parent folders are missing and Recursive Flag is |
semi-expression |
Recursive Flag |
optional Default value: |
Permission to make parent directories as needed. |
none |
Move
Moves a source file or folder to the destination folder. The operation automatically preserves target files timestamps.
The task fails when it does not have permissions to delete Source File.
Name | Type | Description | Expression support | ||
---|---|---|---|---|---|
Operation |
mandatory, must be "MOVE" |
Operation name. |
none |
||
Source File |
mandatory |
Path to the source mask on files or folders. The task fails when no file or directory is found to mask. Supports using wildcards. |
semi-expression |
||
Target File |
mandatory |
Path to the destination folder or file:
|
semi-expression |
||
Overwrite Flag |
optional Default value: |
Permission to overwrite files in the destination folder.
The task fails when Overwrite Flag is |
none |
||
Keep Dir Tree Flag |
optional Default value: |
Permission to copy the source file system hierarchy. Applicable only in case the Source File contains wildcards on multiple path levels (for example, If
|
none |
File info
Saves information about a source file or folder to Workflow Variables.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "INFO" |
Operation name. |
none |
Source File |
mandatory |
Path to the source file or folder. The task fails when the source does not exist. |
semi-expression |
Task variable
Name | Description | ||
---|---|---|---|
Name |
Absolute path to the file or folder. |
||
LocalName |
Name of the file or folder. |
||
Size |
Size of the file or folder in bytes. |
||
Timestamp |
Last modification date in
|
Zip
Source files or folders are zipped to a ZIP file.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "ZIP" |
Operation name. |
none |
Source File |
mandatory |
Path to source mask on files or folders. The task fails when no file or directory is found to mask. |
semi-expression |
Target File |
mandatory |
Path to the target ZIP file. The task fails when Target File is an existing directory (that is, the target ZIP file name cannot be the same as an existing directory name). |
semi-expression |
Exclude Parameters |
optional |
Set of mask parameters that are excluded from zipping. |
semi-expression |
Overwrite Flag |
optional Default value: |
Permission to overwrite the output ZIP file.
The task fails when Overwrite Flag is |
none |
Recursive Flag |
optional Default value: |
Permission to zip subdirectories. |
none |
Unzip
Extracts a source ZIP file to the destination folder.
Name | Type | Description | Expression support |
---|---|---|---|
Operation |
mandatory, must be "UNZIP" |
Operation name. |
none |
Source File |
mandatory |
Path to the source file. The task fails when the source does not exist. |
semi-expression |
Target File |
mandatory |
Path to the destination folder. If the target folder structure does not exist, it is created automatically, with respect to the permissions on the filesystem. |
semi-expression |
Overwrite Flag |
optional, Default value: |
Permission to overwrite ZIP file.
The task fails when Overwrite Flag is |
none |
Was this page useful?