ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
latest (16.1.0) 16.0.0 15.4.0 15.3.0 15.2.0 15.1.0 14.5.x 13.9.x 13.8.x 13.7.x 13.6.x 13.5.0 13.4.x 13.3.x 13.2.x
User Community Service Desk Downloads
If you can't find the product or version you're looking for, visit support.ataccama.com/downloads

Apache Avro

Apache Avro assets are supported by the following data sources:

  • Google Cloud Storage

  • Azure Data Lake Storage Gen2

  • Microsoft OneLake

  • Amazon S3

There is no limit to the size of the Avro asset you can import. However, only individual Avro files up to 6 GB can be profiled.

How Apache Avro assets are imported

You can import the following Apache Avro assets into ONE:

  • Avro files

  • Avro tables

  • Partitioned Avro tables

Avro file listing screen

When an Avro file is imported, a catalog item is created with the attributes from the file.

Avro file detail

However, when Avro assets other than files are imported, ONE analyzes the asset then creates a catalog item with the attributes based on the Avro asset. For example, an Avro folder that contains a partitioned table creates a catalog item with the attributes from the Avro files and the partitioned columns.

Avro partitioned table detail

An Avro folder that contains an Avro table creates a catalog item with the attributes from the Avro table.

Avro table detail

Avro file import can only recognize:

  • A folder with Avro files. For example: ./Orders/{Avro Part Files}.

  • A folder with partitioned Avro files, where data is partitioned by specific columns like year and month. For example: ./Orders/{Year=2024}/{Month=01}/{Avro Part Files}.

Profiling

Profiling can only process individual Parquet files up to 6 GB.

  • Full profiling is supported for individual Avro files, Avro tables, and partitioned Avro tables.

  • Sample profiling is supported for Avro tables, but only runs on the first Avro file found in the folder. Partitioned Avro tables do not support sample profiling.

Browsing

You can preview the data only for small Avro files - up to 5 MB by default. The maximum file size can be configured using the property ataccama.one.avro.preview-maximum-size in dpe/etc/application.properties.

Was this page useful?