Cloud Portal ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
latest (16.2.0) 16.1.0 16.0.0 15.4.0 15.3.0 15.2.0 15.1.0 14.5.x 13.9.x 13.8.x 13.7.x 13.6.x 13.5.0 13.4.x 13.3.x 13.2.x
User Community Service Desk Downloads

BigQuery Connection

Before you can create a BigQuery connection, you need to configure dpe/etc/application.properties according to the BigQuery configuration section in Data Sources Configuration.

Create a source

To connect to BigQuery:

  1. Navigate to Data Catalog > Sources.

  2. Select Create.

  3. Provide the following:

    • Name: The source name.

    • Description: A description of the source.

    • Deployment (Optional): Choose the deployment type.

      You can add new values if needed. See Lists of Values.

    • Stewardship: The source owner and roles. For more information, see Stewardship.

Alternatively, add a connection to an existing data source. See Connect to a Source.

Add a connection

  1. Select Add Connection.

  2. In Select connection type, choose Relational Database > BigQuery.

  3. Provide the following:

    • Name: A meaningful name for your connection. This is used to indicate the location of catalog items.

    • Description (Optional): A short description of the connection.

    • Dpe label (Optional): Assign the processing of a data source to a particular data processing engine (DPE) by entering the DPE label assigned to the engine. For more information, see DPM and DPE Configuration in DPM Admin Console.

    • JDBC: A JDBC connection string pointing to the IP address or the URL where the data source can be reached.

      The format of your connection string required depends on your chosen authentication method. See Google service account key JDBC and OAuth Google user JDBC respectively.

  4. In Pushdown processing, select Profile data using pushdown processing and Evaluate data quality using pushdown processing respectively if you want to use pushdown processing for profiling or data quality evaluation.

    1. Follow the instructions in BigQuery Pushdown Processing to configure pushdown processing.

  5. In Additional settings:

    • Select Enable exporting and loading of data if you want to export data from this connection and use it in ONE Data or outside of ONE.

      If you want to export data to this source, you also need to configure write credentials. See Connection credentials.

      Consider the security and privacy risks of allowing the export of data to other locations.

Add credentials

Different sets of credentials can be used for different tasks. One set of credentials must be set as default for each connection.

To determine whether you need to configure more than a single set of credentials, see Connection credentials.

  1. Select Add Credentials.

  2. Choose an authentication method and proceed with the corresponding step:

Integrated credentials

  1. Provide the following:

    • Name (Optional): A name for this set of credentials.

    • Description (Optional): A description for this set of credentials.

  2. If you want to use this set of credentials by default when connecting to the data source, select Set as default.

    See also Connection credentials.

  3. Proceed with Test the connection.

If you select integrated credentials you need to provide OAuthPvtKeyPath via driver properties.

  1. Select Add Driver property.

    • Property name: OAuthPvtKeyPath.

    • Driver property type: File.

    • Content: Browse and upload your service account key file (.json or .p12 format).

OAuth Google user credentials

  1. Select OAuth Google user credentials and provide the following:

    • Name (Optional): A name for this set of credentials.

    • Description (Optional): A description for this set of credentials.

    • Select a secret management service (optional): If you want to use a secret management service to provide values for the following fields, specify which secret management service should be used. After you select the service, you can enable the Use secret management service toggle and provide instead the names the values are stored under in your key vault. For more information, see Secret Management Service.

    • Client ID: The OAuth 2.0 client ID.

    • Client secret: The OAuth 2.0 client secret. Alternatively, enable Use secret management service and provide the name this value is stored under in your selected secret management service.

  2. If you want to use this set of credentials by default when connecting to the data source, select Set as default.

    See also Connection credentials.

OAuth Google user JDBC

Connect to the server using a connection URL written in the following format:

jdbc:bigquery://<Host>:<Port>;ProjectId=<Project>;OAuthType=1;

The variables are defined as follows:

  • <Host> is the DNS or IP address of the server.

  • <Port> is the number of the TCP port to connect to. Specifying the port number is optional if you are connecting to port 443.

  • <Project> is the name of your BigQuery project.

Example:

jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=MyProject;OAuthType=1;

The connector opens a new browser window to Google sign-in page. Sign in using the new browser window and the connector obtains the refresh token.

Google service account key credentials

  1. Select Google service account key credentials and provide the following:

    • Name (Optional): A name for this set of credentials.

    • Description (Optional): A description for this set of credentials.

    • Service account key: Upload the service account key (in JSON file format) associated with the project where the Google Cloud Storage bucket you want to connect to is located.

      To learn how to create a service account and obtain the service account key from your Google Cloud project, see the official Google documentation:

  2. If you want to use this set of credentials by default when connecting to the data source, select Set as default.

    See also Connection credentials.

Google service account key JDBC

Connect to the server using a connection URL written in the following format:

jdbc:bigquery://<Host>:<Port>;ProjectId=<Project>;OAuthType=0;OAuthServiceAcctEmail=<ServiceAccountEmail>;

The variables are defined as follows:

  • <Host> is the DNS or IP address of the server.

  • <Port> is the number of the TCP port to connect to. Specifying the port number is optional if you are connecting to port 443.

  • <Project> is the name of your BigQuery project.

  • <ServiceAccountEmail> is the service account email address for authentication.

Example:

jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=MyProject;OAuthType=0;OAuthServiceAcctEmail=my-service-account@my-project.iam.gserviceaccount.com;

The connector uses the service account to authenticate the connection.

Test the connection

To test and verify whether the data source connection has been correctly configured, select Test Connection.

If the connection is successful, continue with the following step. Otherwise, verify that your configuration is correct and that the data source is running.

Save and publish

Once you have configured your connection, save and publish your changes. If you provided all the required information, the connection is now available for other users in the application.

In case your configuration is missing required fields, you can view a list of detected errors instead. Review your configuration and resolve the issues before continuing.

Next steps

You can now browse and profile assets from your connection.

In Data Catalog > Sources, find and open the source you just configured. Switch to the Connections tab and select Document. Alternatively, opt for Import or Discover documentation flow.

Or, to import or profile only some assets, select Browse on the Connections tab. Choose the assets you want to analyze and then the appropriate profiling option.

Was this page useful?