Ataccama ONE Agentic ONE Desktop ONE DQ&C ONE MDM ONE RDM ONE Runtime Server
latest (17.1.0) 17.0.0 16.3.x 16.2.0 16.1.0 16.0.0 15.4.0 15.3.0 15.2.0 15.1.0 14.5.x 13.9.x 13.8.x 13.7.x 13.6.x 13.5.0 13.4.x 13.3.x 13.2.x
Downloads

Network Connectivity and HTTP Proxy Configuration

This article describes how to configure network connectivity and HTTP proxy settings in hybrid and self-managed (on-premise) deployments, both VM- and Kubernetes-based.

In Ataccama Cloud, DPE network connectivity is managed by Ataccama.

Overview of DPE outbound connections

A Data Processing Engine (DPE) instance and the jobs it launches open several kinds of outbound connections.

If your network routes outbound traffic through an HTTP(S) proxy, each type of connection must be configured separately, as outlined in the following table. Setting the proxy only for DPE does not configure it for jobs or for all JDBC drivers.

Jobs do not connect directly to Data Processing Module (DPM), ONE Object Storage, or Keycloak. Instead, DPE downloads job inputs and uploads job results on their behalf.

From To Configured by Proxy settings used

DPE

DPM (gRPC)

ataccama.client.connection.dpm.*; see DPM connection

DPE JVM system properties. See Connect DPE to ONE through a proxy.

DPE

ONE Object Storage (MinIO)

ataccama.one.object-storage.storages[n].connectionProperties.url

DPE JVM system properties.

DPE

Keycloak (optional)

ataccama.authentication.keycloak.server-url; see Keycloak authentication

DPE JVM system properties.

DPE

Data sources (browsing, metadata import, data preview, pushdown processing)

The data source connection in ONE and the JDBC driver in ataccama.one.dpe.drivers.path

JDBC driver properties of the connection (recommended), or DPE JVM system properties if the driver supports them. See Connect DPE to data sources through a proxy.

Local jobs (DQ evaluation, profiling, and other jobs run by DPE)

Data sources

The same data source connection in ONE

JDBC driver properties of the connection (recommended), or job JVM system properties. See Connect jobs to data sources through a proxy.

Local jobs

DPE (gRPC on localhost, port ataccama.server.grpc.port)

Set automatically

Must not go through the proxy. Keep localhost in http.nonProxyHosts.

Spark job launcher (for example, Databricks)

Spark cluster and its APIs

application-SPARK_<type>.properties; see Databricks Configuration

plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.env.JAVA_OPTS.

Proxy system properties

DPE uses the standard Java networking system properties. You pass them as -D options in JAVA_OPTS, not in DPE application.properties.

Property Data type Description

http.proxyHost

String

The host name or IP address of the proxy used for http:// connections.

http.proxyPort

Number

The port of the proxy used for http:// connections.

Default value: 80.

https.proxyHost

String

The host name or IP address of the proxy used for https:// connections and for gRPC connections to DPM.

https.proxyPort

Number

The port of the proxy used for https:// and gRPC connections.

Default value: 443.

http.nonProxyHosts

String

The hosts that are accessed directly, bypassing the proxy. The setting applies to both HTTP and HTTPS.

Separate the values with a pipe (|). You can use as a wildcard at the beginning or at the end of a value, for example, .example.com|10.*.

Default value: localhost|127.*|[::1].

If you set this property, the value replaces the default. Always include localhost|127.*|[::1], otherwise jobs can’t report back to DPE.

Proxy authentication

Authenticating proxies are not supported for DPE’s own connections to ONE (DPM, ONE Object Storage, Keycloak). http.proxyUser and http.proxyPassword are not standard Java properties, and DPE ignores them. Configure the proxy to allow DPE’s traffic without authentication, for example, by allowlisting the DPE host.

For data sources, use the proxy authentication options of the JDBC driver, if available. See Connect DPE to data sources through a proxy.

Connect DPE to ONE through a proxy

  1. Set the FIREWALL_FRIENDLY_REGISTRATION connection mode in the dpe/etc/application.properties file (see DPM connection).

    In the default NORMAL_REGISTRATION mode, DPM opens connections into DPE on port 8532, and that traffic can’t go through a forward proxy. In the firewall-friendly mode, DPE opens the connection to DPM itself, so only outbound traffic is needed.

    ataccama.one.dpe.service.dpm.connection.mode=FIREWALL_FRIENDLY_REGISTRATION

  2. Add the proxy system properties to the DPE JVM options. Set https.proxyHost and https.proxyPort even if DPM uses gRPC without TLS because the gRPC client always looks up the HTTPS proxy.

    • VM-based deployments

    • Kubernetes-based deployments

    1. Edit /etc/systemd/system/dpe.service.d/dpe.conf and append the options to the existing JAVA_OPTS. Put the http.nonProxyHosts value in single quotes, because the start script evaluates JAVA_OPTS in a shell and | would otherwise be treated as a pipe.

      Environment="JAVA_OPTS=<existing options> -Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxy.example.com -Dhttps.proxyPort=3128 -Dhttp.nonProxyHosts='localhost|127.*|[::1]|*.internal.example.com'"

      If you deploy with Ansible, set the same options in the dpe_java_opts variable instead, so that they aren’t overwritten on the next deployment. The environment_vars (http_proxy, https_proxy) variable in the Ansible inventory applies only to the Ansible tasks (for example, downloads), not to DPE itself.

    2. Reload and restart the service:

      sudo systemctl daemon-reload
sudo systemctl restart dpe

    1. Add the following options to extraJavaOpts in your values override file.

      extraJavaOpts: "-Dhttps.proxyHost=proxy.example.com -Dhttps.proxyPort=3128 -Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=3128 -Dhttp.nonProxyHosts='localhost|127.*|[::1]|*.svc.cluster.local'"

      Do not use javaOpts for this, because it replaces the default JVM options (including memory settings) instead of adding to them.

  3. If the proxy inspects TLS traffic (SSL bumping), verify and configure the following:

    • Add the proxy’s CA certificate to the DPE truststore.

      On Kubernetes, use initContainers.extraTrustCerts. See also TLS Configuration.

    • The proxy must support HTTP/2 and long-lived bidirectional gRPC streams to the DPM gRPC host. If it doesn’t, exclude the DPM gRPC host from inspection.

    • The proxy’s idle timeout must be longer than ataccama.one.dpe.service.dpm.connection.firewall-friendly.max-connection-inactivity.

Connect DPE to data sources through a proxy

How a data source connection uses a proxy depends on the JDBC driver:

  • Drivers that connect over HTTP(S) (for example, Databricks, Snowflake, Google BigQuery) usually have their own proxy settings. Configure them as driver properties of the connection in ONE: in the connection’s Driver properties section, select Add Driver Property.

    This is the recommended option because the settings apply only to that connection and support proxy authentication. In addition, they are sent with the connection, so they apply both to DPE and to the jobs that use that connection.

    Examples (see the driver vendor’s documentation for the full list)
    Driver Driver properties

    Databricks

    UseProxy=1 ProxyHost=proxy.example.com ProxyPort=3128 ProxyAuth=1 ProxyUID=<user> ProxyPWD=<password>

    Snowflake

    useProxy=true proxyHost=proxy.example.com proxyPort=3128 proxyUser=<user> proxyPassword=<password> nonProxyHosts=<hosts>

    Some HTTP-based drivers instead support the JVM system properties described in Proxy system properties. They then use the DPE JVM options from Connect DPE to ONE through a proxy for DPE, and the job JVM options from Connect jobs to data sources through a proxy for jobs.

  • Drivers that use a native database protocol over TCP (for example, PostgreSQL, Oracle, Microsoft SQL Server, MySQL) do not use HTTP proxies. DPE and the jobs need a direct network route to the database.

  • Non-JDBC connectors have their own proxy configuration. See Power BI Connection, Power BI Report Server Connection, Tableau Connection, and Salesforce Connection.

Connect jobs to data sources through a proxy

Jobs run in their own JVM processes and do not inherit the DPE JVM options. If jobs need JVM-level proxy settings (for example, because a driver doesn’t have its own proxy properties), set them separately for each launch type.

Local jobs

Set the options in plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.env.JAVA_OPTS.

This property takes effect only when plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.exec points to the launch script (see Executor). Without the launch script, DPE starts jobs with a plain java command that ignores JAVA_OPTS.

The JAVA_OPTS values of jobs must not contain spaces other than the ones that separate options.

  • VM-based deployments

  • Kubernetes-based deployments

In dpe/etc/application.properties (or dpe_additional_config if you deploy with Ansible), set the following:

plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.exec=/opt/dpe/bin/local/exec_local.sh
plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.env.JAVA_OPTS=-Dhttps.proxyHost=proxy.example.com -Dhttps.proxyPort=3128 -Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=3128 -Dhttp.nonProxyHosts=localhost|127.*|[::1]

Do not use quotes here. Unlike the DPE start script, the job launch script passes the value to Java unchanged, so quotes would become part of the value.

Setting LOCAL.env.JAVA_OPTS replaces the JAVA_OPTS that jobs would otherwise take from DPE’s environment, so repeat any other options the jobs need, such as memory settings.

Add the following options to extraJavaOptsDqcLocal in your values override file. The Helm chart already sets LOCAL.exec.

extraJavaOptsDqcLocal: "-Dhttps.proxyHost=proxy.example.com -Dhttps.proxyPort=3128 -Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=3128 -Dhttp.nonProxyHosts=localhost|127.*|[::1]"

Spark jobs (Databricks, Hadoop)

For Spark launch types, plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.env.JAVA_OPTS (in dpe/etc/application-SPARK_<type>.properties) configures the launcher process that runs on the DPE host. This process submits the job and communicates with the cluster.

Add the proxy system properties here if the DPE host reaches the cluster through a proxy:

plugin.executor-launch-model.ataccama.one.launch-type-properties.SPARK.env.JAVA_OPTS=<existing options> -Dhttps.proxyHost=proxy.example.com -Dhttps.proxyPort=3128 -Dhttp.nonProxyHosts=localhost|127.*|[::1]

The Spark processing itself runs on the cluster. The cluster’s connectivity to data sources is governed by the cluster’s own network configuration, or by spark.driver.extraJavaOptions / spark.executor.extraJavaOptions in the cluster Spark configuration. See Databricks Configuration and Hadoop Configuration.

Proxy troubleshooting

DPE doesn’t appear in ONE or keeps reconnecting

Check that:

  • https.proxyHost is set.

  • The proxy allows CONNECT requests to the DPM gRPC host on port 443.

  • The proxy doesn’t require authentication for DPE.

Jobs fail right after start or hang without reporting progress

http.nonProxyHosts probably doesn’t include localhost, so the job’s connection back to DPE is sent to the proxy. Verify and update the value accordingly (see Proxy system properties).

Browsing a data source works but jobs on it fail

The proxy is configured only in the DPE JVM options. Move it to the connection’s driver properties, or also set the job JVM options (see Connect jobs to data sources through a proxy).

Was this page useful?