Create a Data Slice

Data slices let you work with a subset of a catalog item based on selected criteria, rather than processing the entire item. This is optimal both from computational and cost perspectives.

Data slices are especially helpful when dealing with large tables which are updated frequently. Instead of processing millions or even billions of records, you can restrict operations to a relevant subset—such as data from the last year or month.

While it is still possible to create SQL catalog items or virtual catalog items, the data slice functionality facilitates additional use cases which cannot be achieved using these methods.

Using data slices you can:

Slice a large volume of data by a selected attribute and run analyses only on this relevant subset of data.
Easily define the subset criteria—for example, based on an attribute containing date information (to filter out old or irrelevant data), or defined by country. This can be dynamic (for example, the previous month or the last seven days) or ad hoc (based on a specific range).
See results for the data subset.

To check whether you can use data slices with your database, see Supported databases.

Create a data slice

To create a data slice:

In Data Catalog > Catalog Items, select the catalog item for which you want to create a data slice.
Using the three dots menu, select Create data slice.

Alternatively, you can also create data slices from the Data Slices section of the catalog item Data Structure tab or in monitoring projects where data slices are already active.
A Create data slice dialog opens. Define the slice:
1. Provide a name for the new data slice.
2. Select the Slice by attribute. This is the attribute by which the data will be filtered.
  
  Only attributes of Date, Datetime, String, Integer, and Long data types can be selected.
3. Specify the Slice by criteria. For example, the values you are interested in (for`String` attributes), or a date range (for Date and Datetime attributes).
  
  If your date information is stored in a non-date attribute (such as a string), see Slice by date stored as a non-date attribute.
  Date data type example
  
  String data type example
  
  No range is preselected, so select one before you save the slice. You can select a predefined range, such as Previous month.
  
  To enter a start and an end date, select Fixed dates. To define a window relative to the date on which the slice is used, select Relative dates. See Slice by a relative date window.
  
  If you are using is from list or is not from list, use the plus icon to add additional values to the list.
Save and publish.

Slice by a relative date window

With Relative dates, the slice covers whole days, weeks, months, quarters, or years relative to the date on which the slice is used, for example, the last seven days or the previous month. The window is recalculated every time the slice is used, so you don’t need to update it.

You can define the window in two editors:

Basic: Select Previous, Current, or Next, and a unit. The window covers one whole period, for example, the previous month.
Advanced: Enter the From and To offsets and select a unit. 0 is the period in which the slice is used, negative numbers are the periods before it, and positive numbers are the periods after it. Both the From and the To periods are included in the window.

To switch between the editors, select Advanced or Basic below the dates.

The following table shows examples of windows in the advanced editor:

Window From To Unit

Window	From	To	Unit
Today	`0`	`0`	Days
Yesterday	`-1`	`-1`	Days
Today and the seven days before it	`-7`	`0`	Days
Yesterday, today, and tomorrow	`-1`	`1`	Days
The next 30 days, without today	`1`	`30`	Days
The previous week	`-1`	`-1`	Weeks
The current month	`0`	`0`	Months
The last three whole months	`-3`	`-1`	Months
The previous quarter	`-1`	`-1`	Quarters

Today

0

Days

Yesterday

-1

Days

Today and the seven days before it

-7

0

Days

Yesterday, today, and tomorrow

-1

1

Days

The next 30 days, without today

1

30

Days

The previous week

-1

Weeks

The current month

0

Months

The last three whole months

-3

-1

Months

The previous quarter

-1

Quarters

When you define a relative window, keep in mind that:

Offsets count whole periods. For example, 0 to 0 months covers the entire current month, not only the days up to today.
Weeks run from Monday to Sunday.
For Datetime attributes, the last day of the window is included in full.
The dates under the fields, marked As of today, show the window for today’s date. The slice details in the Select data slice dialog, and when you hover over the slice name in Applied Data Slice, also show the calculated dates. The monitoring project results show the dates used for each run.
Depending on the database, the window is calculated from the current date of the database or from the current date in UTC. For SAP OData sources, it’s always calculated from UTC. The dates under the fields are calculated from the date in your browser, so close to midnight they can differ by one day from the window that is used.

You can’t save the slice if:

From or To is empty.
From is later than To.
The window falls outside the years 1 to 9999.

A slice opens in the editor in which it was saved. If you need a window that the basic editor doesn’t cover, for example, one that spans several periods, use the advanced editor. In that case, you can’t switch back to the basic editor.

Slice by date stored as a non-date attribute

If you have date information saved as, for example, a string data type, it is still possible to slice by date.

To do this:

In Slice by, select the attribute containing the data information.
Select the Contains date option.
From the list provided, choose the date format used in the selected attribute.
You can now specify the time range of the data you are interested in.

Where to use data slices

After creating a data slice, you can use it in two ways:

Apply it to a catalog item: The slice filters data for all screens and operations on that catalog item, including profiling, DQ evaluation, and exports. See Apply a data slice to a catalog item.
Use it in a monitoring project: The slice filters data only within the context of that monitoring project. See Use data slices in the monitoring projects.

Apply a data slice to a catalog item

You can apply one data slice per catalog item. Once applied, the slice is used for all subsequent profiling, DQ evaluation, and other supported operations.

To apply a data slice:

Navigate to the catalog item Overview tab.
In the Summary widget, find the Applied Data Slice field and click Select.
In the Select data slice dialog:
1. Select the slice you want to apply from the list.
2. Review the slice definition on the right.
Select slice to confirm.

When a data slice is applied:

The slice name is displayed next to the catalog item name in the header, making it immediately visible that you are working with filtered data.
All supported operations use the sliced data instead of the full catalog item.
The Data tab displays only the sliced data.

To change or clear the applied data slice:

Click Select in the Applied Data Slice field.
In the Select data slice dialog, you can:
- Select a different slice.
- Edit slice to modify the current slice.
- Clear slice to remove the applied slice and return to using the full, unfiltered catalog item.
Publish the changes.

Data slices on SQL catalog items

If you create an SQL catalog item from a catalog item that has a data slice applied, the slice is ignored, and the SQL catalog item contains the full, unfiltered data. To filter the SQL catalog item, apply a data slice to it directly.

When you apply a data slice to an SQL catalog item, the query of the catalog item is used as a subquery, and the slice filters its results. This also applies to the record count and to data samples of the catalog item.

Because the slice filters the results of the query, keep in mind that:

If the query contains LIMIT, the slice is applied only to the rows that LIMIT returns. If none of these rows match the slice, the sliced catalog item contains no records, even if the underlying table contains matching rows.
If the slice filters on GROUP BY keys or on the partition keys of window functions, the database can apply it before running the rest of the query. The database can’t apply the slice before aggregated values are calculated, or before ROLLUP, CUBE, or GROUPING SETS, so in these cases the query processes all rows first.

Use data slices in monitoring projects

You can use data slices within monitoring projects to evaluate only a subset of data in that specific context, without affecting the catalog item itself.

For more information, see Use data slices in monitoring projects.

View and manage data slices

To view, create, edit, or delete data slices, navigate to the Data Structure tab and find the Data Slices section.

Use the three dots menu to edit or delete slices.

If you edit a data slice that is currently applied to the catalog item or in used in a monitoring project, you will see a warning message. It is not possible to delete data slices that are applied to catalog items or used in a monitoring project.

You can also see a summary of data slices on the catalog item Overview tab, in the Summary section.

Where data slices have been created in catalog items, you can see this information on the catalog item Overview tab.

Data slice permissions

The access level you need depends on the action:

Create a data slice: at least View metadata access on the catalog item.
Use a data slice in a monitoring project: at least Editing access on the monitoring project.

For the full description of the access levels, see Access Levels.

Supported databases

Data slices are enabled by default for the following databases:

Amazon Aurora MySQL
Amazon Aurora PostgreSQL
Amazon Redshift
Azure Data Explorer
Azure Synapse Analytics
Azure SQL Database
Big Query
Databricks
IOMETE
MariaDB
Microsoft Fabric (Lakehouse SQL endpoint)
MSSQL Server
MySQL
Oracle
PostgreSQL
SAP HANA
SAP OData
Snowflake
Teradata

You can also use data slices on other JDBC connections, including custom drivers, if the driver configuration declares the SQL dialect of the database in the sql-dialect property. Data slices on these connections are experimental and turned off by default.

To turn them on, set the data-slice.enabled property of the driver to true in the DPM Admin Console. The change takes effect immediately, without restarting DPE. For details, see Configure drivers. After you turn them on, verify the results against your own data.

Turn off data slices for a driver

Setting data-slice.enabled to false doesn’t affect data slices that are already applied to catalog items or used in monitoring projects. After you turn off data slices, Create data slice isn’t available and the other data slice options are hidden. The slice name is still displayed in the catalog item header, and the data is still filtered by the slice.

To stop using data slices on a connection completely:

Clear the data slices on all catalog items from the connection and in all monitoring projects that use them. Do this while data slices are still enabled for the driver. You don’t need to delete the slices.
In the DPM Admin Console, set data-slice.enabled to false for the driver and save your changes. The change takes effect immediately, without restarting DPE. See Configure drivers.

Create data slice is not available

If Create data slice isn’t available, its tooltip says why:

The connection doesn’t support data slices.: Data slices aren’t enabled for the driver, or the driver configuration doesn’t declare an SQL dialect. See Supported databases.
The connection’s driver is not currently available.: The connector is turned off in the DPM Admin Console, or no DPE currently provides it. If you turn off the whole connector, not only data slices, this tooltip appears only after the DPE is restarted.

Was this page useful?