Secret Management Services

ONE Runtime Server can read secrets from an external secret management service, also called a vault, instead of storing them in the runtime configuration file. You define each vault once and then reference the secrets it holds from other contributed configurations and from data sources.

Supported providers

ONE Runtime Server supports the following providers:

Azure Key Vault: Authentication with Azure AD client credentials or Azure AD managed identity.
AWS Secrets Manager: Authentication with access key credentials, optionally combined with an assume role configuration.
HashiCorp Vault: Authentication with Google service account key credentials, Google Compute Engine credentials, or HashiCorp JWT credentials.
CyberArk Secrets Manager: Authentication with a PFX or Cert client certificate.

The same providers can also be integrated with Ataccama ONE. See Secret Management Services.

Define vaults in the runtime configuration

Vault definitions belong to the VaultContributor contributed configuration. See Contributed configurations.

Sample vault contributor configuration

<contributedConfigs>
  <config class="com.ataccama.dqc.auth.vault.VaultContributor">
    <vaults>
      <!-- Vault definitions -->
    </vaults>
  </config>
</contributedConfigs>

Each <vault> element defines a connection to one provider. The name attribute identifies the vault when you reference a secret stored in it. The <auth> element holds the credentials used to authenticate to the provider.

You can encrypt these credentials, such as clientKey, secretKey, serviceKey, or passphrase, in the same way as any other password in the runtime configuration. See Encrypt Passwords.

For the full definition of each provider, see Azure Key Vault, AWS Secrets Manager, HashiCorp Vault, or CyberArk Secrets Manager.

How secret references work

Instead of writing a secret into the runtime configuration file, you write a reference to it. A reference has the form vault:<provider>:<vault name>:<secret>, where the provider determines the rest of the reference.

You can use a reference in any attribute that takes a secret, such as clientKey, secretKey, accessKey, or the password of a data source. For example, clientKey="vault:azure:test_azure_key_vault:adls-browsing-secret-integration" retrieves the adls-browsing-secret-integration secret from the vault named test_azure_key_vault.

Provider Reference template Description

Provider	Reference template	Description
Azure Key Vault	`vault:azure:<vault name>:<secret key name>`	`<vault name>`: The `name` attribute of the vault definition. `<secret key name>`: The name of the secret in the key vault.
AWS Secrets Manager	`vault:aws:<secret name>:<secret key name>`	`<secret name>`: The `secretName` attribute of the vault definition. `<secret key name>`: The key within that secret.
HashiCorp Vault	`vault:hashi:<vault name>:<secret path>:$.<key>`	`<vault name>`: The `name` attribute of the vault definition. `<secret path>`: The path to the secret in the secrets engine, for example, `kv_testvault/qa`. `$.<key>`: A JSONPath expression that selects the key within the secret, for example, `$.s3-secret-key`.
CyberArk Secrets Manager	`vault:cyberArk:<vault name>:<safe name>:<object name>:<field name>`	`<vault name>`: The `name` attribute of the vault definition. `<safe name>`: The safe that holds the account, for example, `Partner-Ataccama`. `<object name>`: The account object, for example, `Database-MySQL-dpp`. `<field name>`: The property of the account object to retrieve, for example, `Content`.

Azure Key Vault

vault:azure:<vault name>:<secret key name>

<vault name>: The name attribute of the vault definition.
<secret key name>: The name of the secret in the key vault.

AWS Secrets Manager

vault:aws:<secret name>:<secret key name>

<secret name>: The secretName attribute of the vault definition.
<secret key name>: The key within that secret.

HashiCorp Vault

vault:hashi:<vault name>:<secret path>:$.<key>

<vault name>: The name attribute of the vault definition.
<secret path>: The path to the secret in the secrets engine, for example, kv_testvault/qa.
$.<key>: A JSONPath expression that selects the key within the secret, for example, $.s3-secret-key.

CyberArk Secrets Manager

vault:cyberArk:<vault name>:<safe name>:<object name>:<field name>

<vault name>: The name attribute of the vault definition.
<safe name>: The safe that holds the account, for example, Partner-Ataccama.
<object name>: The account object, for example, Database-MySQL-dpp.
<field name>: The property of the account object to retrieve, for example, Content.

PostgreSQL connection using an Azure Key Vault secret

The following runtime configuration defines an Azure Key Vault and a PostgreSQL data source whose password is read from that vault:

Example runtime configuration with a data source password read from a vault

<?xml version='1.0' encoding='UTF-8'?>
<runtimeconfig>
  <parallelismLevel>1</parallelismLevel>
  <contributedConfigs>
    <config class="com.ataccama.dqc.auth.vault.VaultContributor">
      <vaults>
        <vault name="test_azure_key_vault_di" vaultUrl="<Vault URL>" class="com.ataccama.dqc.azure.auth.vault.AzureKeyVault">
          <auth clientId="*****" clientKey="*****" tenantId="******" authType="AAD_CLIENT_CREDENTIAL" class="com.ataccama.dqc.azure.auth.vault.AzureKeyVaultAuth"/>
        </vault>
      </vaults>
    </config>
  </contributedConfigs>
  <dataSources>
    <dataSource password="vault:azure:test_azure_key_vault_di:aws-postgresql-password" name="PostgreSQL" driverName="PostgreSQL" user="dpp" url="<JDBC connection string>">
      <properties/>
      <propertiesEncrypted/>
      <propertiesFile/>
      <propertiesFileEncrypted/>
    </dataSource>
  </dataSources>
  <databaseDrivers>
    <databaseDriver driverClass="org.postgresql.Driver" name="PostgreSQL">
      <classpath>
        <classpathEntry path="<Path to jdbc driver>/postgresql-42.7.11.jar"/>
      </classpath>
    </databaseDriver>
  </databaseDrivers>
  <runtimeComponents/>
  <pathVariables/>
</runtimeconfig>

Azure Key Vault

Azure Key Vault supports two authentication types: Azure AD client credentials and Azure AD managed identity. For the meaning of each parameter, see Azure Key Vault Integration.

Azure AD client credentials

Azure Key Vault definition using Azure AD client credentials

<vault name="azure-client"
       vaultUrl="https://vault.vault.azure.net/"
       class="com.ataccama.dqc.azure.auth.vault.AzureKeyVault">
  <auth authType="AAD_CLIENT_CREDENTIAL"
        clientId="client"
        clientKey="secret"
        tenantId="tenant"
        class="com.ataccama.dqc.azure.auth.vault.AzureKeyVaultAuth"/>
</vault>

Azure AD managed identity

Azure Key Vault definition using Azure AD managed identity

<vault name="azure-managed"
       vaultUrl="https://vault.vault.azure.net/"
       class="com.ataccama.dqc.azure.auth.vault.AzureKeyVault">
  <auth authType="AAD_MANAGED_IDENTITY"
        clientId="client"
        class="com.ataccama.dqc.azure.auth.vault.AzureKeyVaultAuth"/>
</vault>

S3 connection using an Azure Key Vault secret

In the following runtime configuration, the secretKey of an S3 connection is read from the vault named test_azure_key_vault:

Example runtime configuration with an S3 connection using an Azure Key Vault secret

<?xml version='1.0' encoding='UTF-8'?>
<runtimeconfig>
  <parallelismLevel>1</parallelismLevel>
  <contributedConfigs>
    <config class="com.ataccama.dqc.auth.vault.VaultContributor">
      <vaults>
        <vault name="test_azure_key_vault" vaultUrl="https://vault.vault.azure.net/" class="com.ataccama.dqc.azure.auth.vault.AzureKeyVault">
          <auth clientId="***********" clientKey="***********" tenantId="***********" authType="AAD_CLIENT_CREDENTIAL" class="com.ataccama.dqc.azure.auth.vault.AzureKeyVaultAuth"/>
        </vault>
      </vaults>
    </config>
    <config class="com.ataccama.dqc.s3.config.S3Contributor">
      <s3Connections>
        <s3Connection s3Compatible="AWS" secretKey="vault:azure:test_azure_key_vault:s3-secret-key" accessKey="***********" name="S3test" region="eu-central-1" sseType="NONE" url="s3://<bucket_name>"/>
      </s3Connections>
    </config>
  </contributedConfigs>
  <databaseDrivers/>
  <runtimeComponents/>
  <pathVariables/>
</runtimeconfig>

AWS Secrets Manager

AWS Secrets Manager uses access key credentials, optionally combined with an assume role configuration. The sessionToken attribute is optional and can be omitted. For the meaning of each parameter, see AWS Secrets Manager Integration.

Access key credentials

AWS Secrets Manager definition using access key credentials

<vault endpoint="<endpoint of Secrets Manager>"
       secretName="<Secret Name In AWS Secret Manager>"
       name="<Name of Vault>"
       region="<Region>"
       class="com.ataccama.dqc.aws.auth.vault.AwsSecretsManagerVault">
  <auth secretKey="<Secret Key>"
        accessKey="<Access Key>"
        sessionToken=""
        authType="AWS_ACCESS_KEY"
        class="com.ataccama.dqc.aws.auth.vault.AwsVaultAuth"/>
</vault>

Access key credentials with assume role

AWS Secrets Manager definition using access key credentials with assume role

<vault endpoint="<AWS Secret Management endpoint>"
       secretName="dev-test-credentials"
       name="aws_secrets_access_key_with_assume_role"
       region="<region>"
       class="com.ataccama.dqc.aws.auth.vault.AwsSecretsManagerVault">
  <auth secretKey="<Secret Key>"
        accessKey="<Access Key>"
        sessionToken=""
        authType="AWS_ACCESS_KEY"
        class="com.ataccama.dqc.aws.auth.vault.AwsVaultAuth">
    <assumeRole roleArn="arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
                sessionName="test-session"
                externalId="ASSUME_ROLE_EXTERNAL_ID"
                region="<region>"
                class="com.ataccama.dqc.aws.auth.runtime.config.AwsAssumeRoleAuthConfig"
                enabled="true"/>
  </auth>
</vault>

Azure Data Lake Storage connection using an AWS Secrets Manager secret

In the following runtime configuration, the clientKey of an Azure Data Lake Storage Gen2 connection is read from the vault named aws_secrets_access_key:

Example runtime configuration with an Azure Data Lake Storage Gen2 connection using an AWS Secrets Manager secret

<?xml version='1.0' encoding='UTF-8'?>
<runtimeconfig>
  <parallelismLevel>1</parallelismLevel>
  <contributedConfigs>
    <config class="com.ataccama.dqc.auth.vault.VaultContributor">
      <vaults>
        <vault endpoint="https://secretsmanager.eu-central-1.amazonaws.com" secretName="dev-test-credentials" name="aws_secrets_access_key" region="eu-central-1" class="com.ataccama.dqc.aws.auth.vault.AwsSecretsManagerVault">
          <auth secretKey="***********" accessKey="***********" sessionToken="" authType="AWS_ACCESS_KEY" class="com.ataccama.dqc.aws.auth.vault.AwsVaultAuth"/>
        </vault>
      </vaults>
    </config>
    <config class="com.ataccama.dqc.azure.config.AzureGen2Contributor">
      <azureGen2Connections>
        <azureGen2Connection endpoint="dfs.core.windows.net" clientId="***********" authenticateUser="false" clientKey="vault:aws:dev-test-credentials:adls-client-secret" containerName="ataccama-browsing-dev" name="ataccamabrowsingdev" tenantId="***********" authType="AAD_CLIENT_CREDENTIAL" storageAccount="ataccamadatalakegen2" authTokenEndpoint="***********"/>
      </azureGen2Connections>
    </config>
  </contributedConfigs>
  <dataSources/>
  <databaseDrivers/>
  <runtimeComponents/>
  <pathVariables/>
</runtimeconfig>

HashiCorp Vault

HashiCorp Vault supports three authentication types: Google service account key credentials, Google Compute Engine credentials, and HashiCorp JWT credentials. The namespace attribute is required. For the meaning of each parameter, see HashiCorp Vault Integration.

Google service account key credentials

HashiCorp Vault definition using Google service account key credentials

<vault name="hashi-google-service-auth"
       vaultUrl="http://vault.url"
       namespace="namespace"
       class="com.ataccama.dqc.hashicorp.auth.vault.HashiVault">
  <auth authType="GOOGLE_SERVICE_ACCOUNT_KEY"
        serviceKey="serviceKey"
        role="role"
        class="com.ataccama.dqc.google.auth.vault.GoogleKeyVaultAuth"/>
</vault>

Google Compute Engine credentials

HashiCorp Vault definition using Google Compute Engine credentials

<vault name="hashi-google-compute-auth"
       vaultUrl="http://vault.url"
       namespace="namespace"
       class="com.ataccama.dqc.hashicorp.auth.vault.HashiVault">
  <auth authType="GOOGLE_COMPUTE_ENGINE_TOKEN"
        role="role"
        class="com.ataccama.dqc.google.auth.vault.GoogleKeyVaultAuth"/>
</vault>

HashiCorp JWT credentials

HashiCorp Vault definition using JWT credentials

<vault name="hashi-jwt-auth"
       vaultUrl="http://vault.url"
       namespace="namespace"
       class="com.ataccama.dqc.hashicorp.auth.vault.HashiVault">
  <auth authType="HASHICORP_JWT_TOKEN"
        hashiJwt="jwt"
        role="role"
        class="com.ataccama.dqc.hashicorp.auth.vault.HashiVaultAuth"/>
</vault>

S3 connection using HashiCorp Vault secrets

In the following runtime configuration, both the accessKey and the secretKey of an S3 connection are read from a HashiCorp Vault:

Example runtime configuration with an S3 connection using HashiCorp Vault secrets

<?xml version='1.0' encoding='UTF-8'?>
<runtimeconfig>
  <parallelismLevel>1</parallelismLevel>
  <contributedConfigs>
    <config class="com.ataccama.dqc.auth.vault.VaultContributor">
      <vaults>
        <vault name="69bef018-0000-7000-0000-0000001f02ea" namespace="admin" vaultUrl="<HashiCorp Vault URL>" class="com.ataccama.dqc.hashicorp.auth.vault.HashiVault">
          <auth role="dev" serviceKey="*******" authType="GOOGLE_SERVICE_ACCOUNT_KEY" class="com.ataccama.dqc.google.auth.vault.GoogleKeyVaultAuth"/>
        </vault>
      </vaults>
    </config>
    <config class="com.ataccama.dqc.s3.config.S3Contributor">
      <s3Connections>
        <s3Connection s3Compatible="AWS" secretKey="vault:hashi:69bef018-0000-7000-0000-0000001f02ea:kv_testvault/qa:$.s3-secret-key" accessKey="vault:hashi:69bef018-0000-7000-0000-0000001f02ea:kv_testvault/qa:$.s3-access-key" name="S3Hashicorp" region="eu-central-1" sseType="NONE" url="s3://<bucket_name>"/>
      </s3Connections>
    </config>
  </contributedConfigs>
  <databaseDrivers/>
  <runtimeComponents/>
  <pathVariables/>
</runtimeconfig>

CyberArk Secrets Manager

CyberArk Secrets Manager supports two authentication types: PFX client certificate and Cert client certificate. The applicationId attribute is required, and the vaultUrl must point to the Central Credential Provider (CCP) and end with /Accounts. For the meaning of each parameter, see CyberArk Secrets Manager Integration.

PFX client certificate

CyberArk Secrets Manager definition using a PFX client certificate

<vault name="cyberark-pfx-client-certificate-auth"
       vaultUrl="http://vault.url"
       applicationId="applicationId"
       class="com.ataccama.dqc.cyberark.auth.vault.CyberArkVault">
  <auth authType="PFX_CLIENT_CERTIFICATE"
        pfxFile="myPfxFile"
        passphrase="myPassphrase"
        class="com.ataccama.dqc.cyberark.auth.vault.CyberArkVaultAuth"/>
</vault>

Cert client certificate

CyberArk Secrets Manager definition using a Cert client certificate

<vault name="cyberark-cert-client-certificate-auth"
       vaultUrl="http://vault.url"
       applicationId="applicationId"
       class="com.ataccama.dqc.cyberark.auth.vault.CyberArkVault">
  <auth authType="CERT_CLIENT_CERTIFICATE"
        certFile="myCertFile"
        keyFile="myKeyFile"
        passphrase="myPassphrase"
        class="com.ataccama.dqc.cyberark.auth.vault.CyberArkVaultAuth"/>
</vault>

PostgreSQL connection using a CyberArk Secrets Manager secret

In the following runtime configuration, the password of a PostgreSQL data source is read from a CyberArk Secrets Manager vault that authenticates with a PFX client certificate:

Example runtime configuration with a data source password read from CyberArk Secrets Manager

<?xml version='1.0' encoding='UTF-8'?>
<runtimeconfig>
  <parallelismLevel>1</parallelismLevel>
  <contributedConfigs>
    <config class="com.ataccama.dqc.auth.vault.VaultContributor">
      <vaults>
        <vault name="69bef018-0000-7000-0000-0000001d000a" vaultUrl="https://<CyberArk CCP host>/AIMWebService/api/Accounts" applicationId="CCP_Ataccama_AppID" class="com.ataccama.dqc.cyberark.auth.vault.CyberArkVault">
          <auth passphrase="*******" pfxFile="*******" authType="PFX_CLIENT_CERTIFICATE" class="com.ataccama.dqc.cyberark.auth.vault.CyberArkVaultAuth"/>
        </vault>
      </vaults>
    </config>
  </contributedConfigs>
  <dataSources>
    <dataSource password="vault:cyberArk:69bef018-0000-7000-0000-0000001d000a:Partner-Ataccama:Database-MySQL-dpp:Content" name="PostgreSQL" driverName="PostgreSQL" user="dpp" url="<JDBC connection string>">
      <properties/>
      <propertiesEncrypted/>
      <propertiesFile/>
      <propertiesFileEncrypted/>
    </dataSource>
  </dataSources>
  <databaseDrivers>
    <databaseDriver driverClass="org.postgresql.Driver" name="PostgreSQL">
      <classpath>
        <classpathEntry path="<Path to jdbc driver>/postgresql-42.7.13.jar"/>
      </classpath>
    </databaseDriver>
  </databaseDrivers>
  <runtimeComponents/>
  <pathVariables/>
</runtimeconfig>

Was this page useful?