External Secret Store for Self-Managed DPE

From version 17.2.0, a self-managed Data Processing Engine (DPE) can read its secrets from an external secret store, such as Azure Key Vault, instead of the Helm values file. This page describes the setup with Azure Key Vault. For a general description of self-managed DPE installation, see Kubernetes-Based Hybrid DPE Deployment Guide.

How the external secret store works

By default, the Helm values file generated for a self-managed DPE contains the DPE secrets in plain text, and the DPE Helm chart stores them in Kubernetes Secrets. When you set global.hybridDeployment.csiConfig in the Helm values file, DPE reads its secrets from the secret store through the Secrets Store CSI Driver instead:

The secrets are kept only in the secret store.
The Secrets Store CSI Driver mounts them as read-only files into the DPE pod (/mnt/secrets-store) when the pod starts.
The DPE Helm chart does not create the dpe-secret, dpe-encryption-keys-secret, and dpe-encryption-password-secret Kubernetes Secrets, so the secrets are not stored in Kubernetes Secrets or etcd.
The Helm values file no longer contains any secrets.

Prerequisites

A self-managed DPE on Kubernetes, version 17.2.0 or later, and the Helm values file generated for it in Ataccama Cloud Portal. See Local installation.
kubectl and Helm access to the cluster.
The Secrets Store CSI Driver and the provider for your secret store installed in the cluster. See Prepare the cluster.
An identity that the provider uses to read the secrets. The identity needs the get and list permissions on the secrets in the vault. See Azure authentication options.

Prepare the cluster

The Secrets Store CSI Driver and the Azure Key Vault provider are two separate components. Install both before you deploy DPE with csiConfig set.

Install the Secrets Store CSI Driver:

Install the Secrets Store CSI Driver

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo update
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
    --namespace kube-system \
    --set syncSecret.enabled=false

syncSecret.enabled=false makes the driver only mount the secrets as files and not copy them to Kubernetes Secrets. This is the mode the DPE Helm chart uses.

Install the Azure Key Vault provider:

Install the Azure Key Vault provider

helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm repo update
helm install azure-provider csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
    --namespace kube-system

If the Secrets Store CSI Driver is already installed as a separate Helm release, as in the previous step, the provider chart conflicts with it on the shared ServiceAccount. In that case, skip the driver bundled with the provider chart:

Install the Azure Key Vault provider without the bundled driver

helm install azure-provider csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
    --namespace kube-system \
    --set secrets-store-csi-driver.install=false

Verify that both components are running:

Verify the Secrets Store CSI Driver and the Azure Key Vault provider

kubectl get daemonset -n kube-system -l app=secrets-store-csi-driver
kubectl get daemonset -n kube-system -l app=csi-secrets-store-provider-azure
kubectl get pods -n kube-system | grep -E "secrets-store|csi-azure"

All pods must be in the Running state before you deploy DPE.

The provider field of csiConfig also accepts aws, gcp, and vault. In that case, install the corresponding provider for the Secrets Store CSI Driver and set parameters according to that provider’s documentation. The migration script described on this page supports only Azure Key Vault.

DPE secrets and where to find them

The following table lists the secrets that DPE reads from the secret store. Each secret is identified by a key in global.hybridDeployment.csiConfig.objectNames. The second column shows where you can find the value in the Helm values file generated in Ataccama Cloud Portal.

Key in objectNames Field in the generated values file Notes

Key in `objectNames`	Field in the generated values file	Notes
`dpeJWTPrivateKey`	`JWT.key_private`	DPE JWT private key. Store the value as it is.
`dpmJWTKeyContent`	`dpm.JWT.key_content`	Data Processing Module (DPM) JWT key. Store the value as it is.
`dpmJWTKeyFingerprint`	`dpm.JWT.key_fp`	DPM JWT key fingerprint. Store the value as it is.
`minioRootUser`	`global.minio.rootUser`	ONE Object Storage (MinIO) user. Store the value as it is.
`minioRootPassword`	`global.minio.rootPassword`	ONE Object Storage (MinIO) password. Store the value as it is.
`encryptionKeystore`	`global.dataEncryption.peripheralKeystore` or `global.dataEncryption.keystore`	Encryption keystore (JCEKS). The values file contains it base64-encoded. Store the base64 string and set `objectEncoding: base64` for this secret in `parameters.objects`, so that the driver mounts the decoded binary file.
`encryptionPassword`	`global.dataEncryption.password`	Encryption keystore password. The values file contains it base64-encoded. Store the decoded password.
`licenseFile`	-	Optional. The contents of a DPE license file, mounted as `/opt/ataccama/license.plf`. From version 17.2.0, self-managed DPEs don’t need a license file, so you can usually leave this out.

dpeJWTPrivateKey

JWT.key_private

DPE JWT private key. Store the value as it is.

dpmJWTKeyContent

dpm.JWT.key_content

Data Processing Module (DPM) JWT key. Store the value as it is.

dpmJWTKeyFingerprint

dpm.JWT.key_fp

DPM JWT key fingerprint. Store the value as it is.

minioRootUser

global.minio.rootUser

ONE Object Storage (MinIO) user. Store the value as it is.

minioRootPassword

global.minio.rootPassword

ONE Object Storage (MinIO) password. Store the value as it is.

encryptionKeystore

global.dataEncryption.peripheralKeystore or global.dataEncryption.keystore

Encryption keystore (JCEKS). The values file contains it base64-encoded. Store the base64 string and set objectEncoding: base64 for this secret in parameters.objects, so that the driver mounts the decoded binary file.

encryptionPassword

global.dataEncryption.password

Encryption keystore password. The values file contains it base64-encoded. Store the decoded password.

licenseFile

Optional. The contents of a DPE license file, mounted as /opt/ataccama/license.plf. From version 17.2.0, self-managed DPEs don’t need a license file, so you can usually leave this out.

All keys except licenseFile are required. If one of them is missing, Helm fails with the error global.hybridDeployment.csiConfig.objectNames.<key> is required when csiConfig is set.

`csiConfig` reference

Configure the secret store in global.hybridDeployment.csiConfig. From this configuration, the DPE Helm chart creates a SecretProviderClass resource named dpe-secret-provider and mounts it into the DPE pod.

Field Description

Field	Description
`provider`	Required. The provider of the Secrets Store CSI Driver: `azure`, `aws`, `gcp`, or `vault`.
`parameters`	Provider-specific parameters. They are passed as they are to `spec.parameters` of the `SecretProviderClass`. For Azure Key Vault, they include the vault name, the tenant ID, the authentication settings, and the list of secrets (`objects`). See the Azure Key Vault provider documentation.
`nodePublishSecretRef`	Optional. The name of a Kubernetes Secret with the credentials the provider uses to access the secret store, for example, the client ID and client secret of an Azure service principal. The Secret must exist in the DPE namespace.
`objectNames`	The file names under which the secrets are mounted. Each value must match the `objectAlias` of a secret in `parameters.objects`. See DPE secrets and where to find them for the list of keys.

provider

Required. The provider of the Secrets Store CSI Driver: azure, aws, gcp, or vault.

parameters

Provider-specific parameters. They are passed as they are to spec.parameters of the SecretProviderClass. For Azure Key Vault, they include the vault name, the tenant ID, the authentication settings, and the list of secrets (objects). See the Azure Key Vault provider documentation.

nodePublishSecretRef

Optional. The name of a Kubernetes Secret with the credentials the provider uses to access the secret store, for example, the client ID and client secret of an Azure service principal. The Secret must exist in the DPE namespace.

objectNames

The file names under which the secrets are mounted. Each value must match the objectAlias of a secret in parameters.objects. See DPE secrets and where to find them for the list of keys.

When you configure the secrets, follow these rules:

Each value in objectNames must be equal to the objectAlias of the corresponding entry in parameters.objects. We recommend using the key itself as the alias, for example, dpeJWTPrivateKey: dpeJWTPrivateKey.
The encryptionKeystore entry in parameters.objects needs objectEncoding: base64.
global.hybridDeployment.imagePullSecrets must stay in the values file. See Limitations of the external secret store.

The following example uses an Azure VM managed identity:

Example csiConfig for Azure Key Vault

global:
  hybridDeployment:
    enabled: true
    csiConfig:
      provider: azure
      parameters:
        usePodIdentity: "false"
        useVMManagedIdentity: "true"
        keyvaultName: "<KEY_VAULT_NAME>"
        tenantId: "<TENANT_ID>"
        objects: |
          array:
            - |
              objectName: ataccama--dpe-prod--dpeJWTPrivateKey
              objectType: secret
              objectAlias: dpeJWTPrivateKey
            - |
              objectName: ataccama--dpe-prod--dpmJWTKeyContent
              objectType: secret
              objectAlias: dpmJWTKeyContent
            - |
              objectName: ataccama--dpe-prod--dpmJWTKeyFingerprint
              objectType: secret
              objectAlias: dpmJWTKeyFingerprint
            - |
              objectName: ataccama--dpe-prod--minioRootUser
              objectType: secret
              objectAlias: minioRootUser
            - |
              objectName: ataccama--dpe-prod--minioRootPassword
              objectType: secret
              objectAlias: minioRootPassword
            - |
              objectName: ataccama--dpe-prod--encryptionKeystore
              objectType: secret
              objectAlias: encryptionKeystore
              objectEncoding: base64
            - |
              objectName: ataccama--dpe-prod--encryptionPassword
              objectType: secret
              objectAlias: encryptionPassword
      objectNames:
        dpeJWTPrivateKey: dpeJWTPrivateKey
        dpmJWTKeyContent: dpmJWTKeyContent
        dpmJWTKeyFingerprint: dpmJWTKeyFingerprint
        minioRootUser: minioRootUser
        minioRootPassword: minioRootPassword
        encryptionKeystore: encryptionKeystore
        encryptionPassword: encryptionPassword

Azure authentication options

The Azure Key Vault provider supports the following ways to authenticate to the vault. Whichever option you choose, the identity needs the get and list permissions on the secrets in the vault.

Option Settings in csiConfig

Option	Settings in `csiConfig`
VM system-assigned managed identity	`usePodIdentity: "false"`, `useVMManagedIdentity: "true"`
User-assigned managed identity	`usePodIdentity: "false"`, `useVMManagedIdentity: "true"`, `userAssignedIdentityID: <CLIENT_ID>`
Service principal	`usePodIdentity: "false"`, `useVMManagedIdentity: "false"`, and `nodePublishSecretRef` set to the name of a Kubernetes Secret with the service principal credentials
Azure AD Pod Identity (legacy)	`usePodIdentity: "true"`, `useVMManagedIdentity: "false"`

VM system-assigned managed identity

usePodIdentity: "false", useVMManagedIdentity: "true"

User-assigned managed identity

usePodIdentity: "false", useVMManagedIdentity: "true", userAssignedIdentityID: <CLIENT_ID>

Service principal

usePodIdentity: "false", useVMManagedIdentity: "false", and nodePublishSecretRef set to the name of a Kubernetes Secret with the service principal credentials

Azure AD Pod Identity (legacy)

usePodIdentity: "true", useVMManagedIdentity: "false"

All options also need keyvaultName, tenantId, and objects in parameters.

Service principal credentials

For a service principal, create a Kubernetes Secret with the client ID and client secret in the DPE namespace before you deploy DPE:

Service principal credentials Secret

apiVersion: v1
kind: Secret
metadata:
  name: dpe-akv-sp-creds
type: Opaque
stringData:
  clientId: "<CLIENT_ID>"
  clientSecret: "<CLIENT_SECRET>"

Apply the Secret to the DPE namespace:

Apply the service principal credentials Secret

kubectl apply -f dpe-akv-sp-creds.yaml -n <DPE_LABEL>

Replace <DPE_LABEL> with the label of your DPE, which is also the name of the DPE namespace.

Then set nodePublishSecretRef: dpe-akv-sp-creds in csiConfig.

Put the client ID only in this Kubernetes Secret. Do not add clientID to parameters: this switches the provider to Workload Identity mode instead of service principal authentication.

Do not commit the Secret file to version control.

Migrate an existing self-managed DPE

To move the secrets of an existing self-managed DPE to Azure Key Vault, use the setup-csi-secrets.ps1 PowerShell script. In one run, the script:

Uploads the secrets from the existing values file to Azure Key Vault.
Generates a new values file, values-csi.yml by default, with the secrets removed and csiConfig filled in.
For service principal authentication, also generates the template of the Kubernetes Secret with the service principal credentials.

Script prerequisites

PowerShell 5.1 or later.
Azure CLI, logged in with az login, with permission to set secrets in the target vault.
The powershell-yaml module. If it is missing, the script installs it for the current user.
The values file of the existing DPE installation.

Script parameters

Parameter Required Description

Parameter	Required	Description
`-VaultName`	Yes	The name of the Azure Key Vault.
`-Prefix`	Yes	The first part of the secret names in the vault. Identifies the deployment group, for example, `ataccama` or `prod-eu`.
`-TenantId`	Yes	The Azure tenant ID. Written to `parameters.tenantId` in the generated values file.
`-AuthMethod`	Yes	How the provider authenticates to the vault: `VMManagedIdentity`, `UserManagedIdentity`, `ServicePrincipal`, or `PodIdentity`. See Azure authentication options.
`-ClientId`	For `UserManagedIdentity` and `ServicePrincipal`	The client ID of the user-assigned managed identity or of the service principal.
`-SpSecretName`	No	The name of the Kubernetes Secret with the service principal credentials. Used only with `ServicePrincipal`. Default: `dpe-akv-sp-creds`.
`-Namespace`	No	The second part of the secret names in the vault: the Kubernetes namespace DPE runs in. If not set, the script uses `global.namespace` from the values file. If that is not set either, the script takes the namespace from `dpm.host`, which must start with `dpm-grpc-<namespace>.`. If the namespace can’t be determined, the script fails.
`-ValuesFile`	No	The existing values file with the secrets. Default: `values.yml`.
`-OutputFile`	No	The generated values file. Default: `values-csi.yml`.
`-WhatIf`	No	Dry run. Shows which secrets would be uploaded without changing the vault.

-VaultName

Yes

The name of the Azure Key Vault.

-Prefix

Yes

The first part of the secret names in the vault. Identifies the deployment group, for example, ataccama or prod-eu.

-TenantId

Yes

The Azure tenant ID. Written to parameters.tenantId in the generated values file.

-AuthMethod

Yes

How the provider authenticates to the vault: VMManagedIdentity, UserManagedIdentity, ServicePrincipal, or PodIdentity. See Azure authentication options.

-ClientId

For UserManagedIdentity and ServicePrincipal

The client ID of the user-assigned managed identity or of the service principal.

-SpSecretName

The name of the Kubernetes Secret with the service principal credentials. Used only with ServicePrincipal.

Default: dpe-akv-sp-creds.

-Namespace

The second part of the secret names in the vault: the Kubernetes namespace DPE runs in. If not set, the script uses global.namespace from the values file.

If that is not set either, the script takes the namespace from dpm.host, which must start with dpm-grpc-<namespace>.. If the namespace can’t be determined, the script fails.

-ValuesFile

The existing values file with the secrets.

Default: values.yml.

-OutputFile

The generated values file.

Default: values-csi.yml.

-WhatIf

Dry run. Shows which secrets would be uploaded without changing the vault.

Secret names in the vault

The script names the secrets <Prefix>--<Namespace>--<key>, for example, ataccama—dpe-prod—dpeJWTPrivateKey. The keys are those listed in DPE secrets and where to find them. In the generated csiConfig, the vault secret name is the objectName, and the key is the objectAlias.

What the script changes

The script uploads the seven required secrets as described in DPE secrets and where to find them. It does not upload licenseFile.

In the generated values file, the script:

Removes JWT.key_private, JWT.key_content, dpm.JWT.key_content, dpm.JWT.key_fp, the whole global.minio block, global.dataEncryption.password, and global.dataEncryption.peripheralKeystore or global.dataEncryption.keystore.
Adds global.hybridDeployment.csiConfig with provider: azure, the authentication parameters for the chosen -AuthMethod, keyvaultName, tenantId, objects, and objectNames. For ServicePrincipal, it also sets nodePublishSecretRef.
Keeps all other values unchanged.

For ServicePrincipal, the script also writes <SpSecretName>.yaml next to the generated values file, with clientId filled in.

Run the migration

Check what the script would upload with a dry run:

Dry run of the migration script

.\setup-csi-secrets.ps1 `
    -VaultName  <KEY_VAULT_NAME> `
    -Prefix     ataccama `
    -TenantId   <TENANT_ID> `
    -AuthMethod VMManagedIdentity `
    -ValuesFile values.yml `
    -WhatIf

Run the script without -WhatIf. For example:

VM system-assigned managed identity

.\setup-csi-secrets.ps1 `
    -VaultName  <KEY_VAULT_NAME> `
    -Prefix     ataccama `
    -TenantId   <TENANT_ID> `
    -AuthMethod VMManagedIdentity

User-assigned managed identity

.\setup-csi-secrets.ps1 `
    -VaultName  <KEY_VAULT_NAME> `
    -Prefix     ataccama `
    -TenantId   <TENANT_ID> `
    -AuthMethod UserManagedIdentity `
    -ClientId   <CLIENT_ID>

Service principal

.\setup-csi-secrets.ps1 `
    -VaultName  <KEY_VAULT_NAME> `
    -Prefix     ataccama `
    -TenantId   <TENANT_ID> `
    -AuthMethod ServicePrincipal `
    -ClientId   <CLIENT_ID>

Review the generated values-csi.yml file. Make sure it contains global.hybridDeployment.imagePullSecrets and any other environment-specific settings from your original values file.
If you use ServicePrincipal, set up the service principal credentials:
1. Fill in clientSecret in the generated dpe-akv-sp-creds.yaml file.
2. Apply the file to the DPE namespace:
  Apply the service principal credentials Secret
  kubectl apply -f dpe-akv-sp-creds.yaml -n <DPE_LABEL>
Make sure the managed identity or service principal used by the Azure Key Vault provider has the get and list permissions on the secrets in the vault.
Upgrade the DPE with the generated values file. Use the same helm upgrade --install command as for the original installation, with values-csi.yml instead of the original values file.
Store the original values file securely or delete it, as it still contains the secrets.

What happens during the upgrade

Helm deletes the Kubernetes Secrets that the DPE chart created before: dpe-secret, dpe-encryption-keys-secret, and dpe-encryption-password-secret.
The Secrets Store CSI Driver mounts the secrets when the DPE pod starts. The DPE pod must restart to use them. If it doesn’t restart on its own, restart it:
Restart the DPE deployment
```
kubectl rollout restart deployment -n <DPE_LABEL>
```

DPE reads the secrets only when the pod starts. If you change a secret in the vault later, restart the DPE pod again.

Limitations of the external secret store

global.hybridDeployment.imagePullSecrets must stay in the values file. Kubernetes needs the image pull secrets to download the DPE image before any volume is mounted, so they can’t come from the secret store.

Troubleshooting

Error `provider not found: provider "azure"`

The Azure Key Vault provider is not installed in the cluster. Run the verification commands from Prepare the cluster and install the provider if it is missing.

Error `csiConfig.objectNames.<key> is required`

One of the required keys is missing in csiConfig.objectNames. Add all keys listed in DPE secrets and where to find them.

Was this page useful?