Connect to a Source
Connect to a data source to import asset metadata and data, catalog and profile your assets, and monitor their data quality.
ONE can work with a number of sources. For a full list of supported options, see Supported Data Sources.
|
To be able to use a particular data source type, it must first be configured in Data Processing Engine (DPE). DPE needs to have suitable drivers for each data source and different instances of DPE can be configured to handle only jobs coming from a specific data source. If none of your DPE instances can work with a specific data source type, that type is not available when adding a new source in ONE. |
Overview
To view all connected sources, go to Knowledge Catalog > Sources.
Here, you can also add new sources or edit existing ones. A data source must have at least one connection set up, but it is possible to have several connections for a single source. In that case, catalog items from all connections are grouped under that source.
| To add a new source and set up a connection, follow the corresponding guide for your source type in this section of the documentation. |
To view a source connection, open the source and switch to the Connections tab. Here you can view the connection details, import catalog item metadata, edit or delete a connection.
Browse connection
To explore the connection metadata, on the source Connections tab, select Browse. This displays the items that the connection contains.
| In some cases, browsing can take a long time as not all data source types support metadata paging. For example, if there are thousands of tables in a schema, ONE has to read them all to display a single page. |
Next, you can search for a particular item or use the filters provided, preview the data (the eye icon), or import the catalog item metadata (the arrow icon).
You can also select one or more items and start profiling. To learn more about profiling options, see Running Data Discovery and Profiling.
To view connection details, select the connection name (or, use the three dots menu) and then Show details. This displays:
-
The statistics about the documented catalog items, specifically the number of catalog items on which metadata import, sample or full profiling, or DQ evaluation were performed respectively.
-
Connection details.
-
Connection credentials.
-
The Test Connection option to verify that the connection is working.
Edit source metadata
To change the source name, description, or the deployment tag, go to Knowledge Catalog > Sources. Open the source you want to modify and in the three dots menu select Edit.
| You can add or remove deployment tags from the source Overview tab as well. |
Once you’re done editing, select Save and publish.
| In addition, you can change the source Location from the Overview tab. |
Create or edit location
Use locations to further categorize assets within the source. This is then used in the catalog item path (for example, Analytics Snowflake DWH [source] > DEMO_ENV [location] > BUSINESS_OPS [catalog item]).
| If needed, locations can be nested. To do this, open the location details and create a new location. |
To add a new location, select Add Location and enter the location type. Optionally, provide the location name, origin path, and description.
To modify an existing location, in the three dots menu select Edit.
| Select Show details in the location three dots menu to see which assets have this location assigned, as well information about glossary terms. |
| For more information about the Add Catalog item option on the location Overview tab, see Create or edit folders. In general, we do not recommend using this to create new catalog items. |
Create or edit folders
Folders in Workspace are intended for storing SQL catalog items and virtual catalog items (VCIs). For example, this might be useful when you want to manually document some assets within a source but don’t intend to profile them later.
However, we do not recommend creating new catalog items this way as it limits what you can do with such assets (for instance, you cannot add data to them). In addition, you currently cannot add existing catalog items to a folder.
To add a new folder, select Add Folder and enter a name and description. Select Save. Next, in the three dots menu of the folder, select Publish.
To modify an existing folder, in the three dots menu select Edit.
| If needed, folders can be nested. To do this, open the folder details and create a new folder. |
How to add a catalog item to a folder or a location
Use one of the following options:
-
On the source Overview tab, create or edit a folder or location, depending on your use case.
-
Select the folder or the location (or, in the three dots menu, select Show details) and then Add Catalog item.
-
Provide the following:
-
Name: A unique name of the catalog item.
-
Connection: Choose one from the list.
-
Description (Optional): A description of the catalog item.
-
Origin path (Optional): Specify this if you need to track the asset origin, for example, if you plan on uploading data to it from another catalog item later.
-
Owner and Steward: The owner and data steward for the asset.
Number of attributes and Number of records fields are filled in later, once the catalog item data is loaded. -
Type: The type of catalog item. Choose one from the list provided. Depending on the selected type, further details might be required.
-
-
If available, select Add Attribute to define catalog item attributes.
-
Provide the following information:
-
Name: A unique name of the attribute.
-
Column type (Optional).
-
Data type (Optional): The data type of the attribute.
-
Supported: Select if the attribute should be included in profiling, DQ evaluation, and other analyses.
-
Order: The order of the attribute in the catalog item.
-
Type: Select one from the list depending on the selected catalog item type. For some attribute types you might have to provide further details.
-
-
Repeated steps 4 and 5 as many times as needed.
-
Select Save.
You can create as many catalog items as required. -
In the three dots menu of the new catalog item, select Publish to make the catalog item available to other users.
Delete source
| This action cannot be reversed. |
To delete a source, go to Knowledge Catalog > Sources. Select the source you want to delete and then Instant delete. Confirm your choice.
Or, open the source and in the three dots menu select Instant delete.
Edit connection
To edit an existing connection, open the source and switch to the Connections tab. In the three dots menu for the connection you want to modify, select Edit.
To add a connection to an existing source, on the source Connections tab select Add Connection.
| For more information about how to add or edit the connection for your source type, refer to the corresponding guide in this section of the documentation. |
Delete connection
To delete a connection, go to the source Connections tab and in the three dots menu of the connection select Delete. Confirm your choice.
To remove the connection for all users, use the three dots menu again and select Confirm delete. To undo your action, select Restore instead.
Calculate lineage
To calculate lineage on all available data sources, click the Calculate Lineage button on the Sources tab. Lineage graphs are available in the Lineage tab of each Catalog Item
| This action might cause a reduction in the data source performance. Notify other users of the potential disruption. |
Was this page useful?