Logging Configuration

ONE Runtime Server can log information about the execution of ONE plans and the behavior of server components.

You can configure where the logging information is displayed. It can be sent to the console (standard or error output) or to a file with up to four levels of logging messages (listed in order of increasing severity):

DEBUG
INFO
WARN
ERROR

Logging configuration is specified by a series of XML elements in a configuration file. The default logging configuration file is <ATACCAMA_HOME>/server/etc/logback-extension.xml. The location of the configuration file can be specified by the Logging Component in case ONE Runtime Server is used in the server mode. For the batch mode, logging configuration is specified in Runtime Configuration. If no logging configuration file is supplied, the default configuration is used: log records with the severity level INFO and higher are written to the console (standard output).

Configuration

Default settings

The following configuration represents the default logging configuration, which is used when no logging configuration is provided for ONE Runtime Server. The configuration contains one logger (root), which catches all log records with the severity level INFO and higher and sends them to the Console appender named STDOUT. The Console appender is configured to send log records to the standard output (the console).

logback-extension.xml

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%d{yyy.MM.dd HH:mm:ss} %-7([%level])   %logger %msg%n</pattern>
    </encoder>
</appender>

<root level="info">
    <appender-ref ref="STDOUT" />
</root>
...
<!-- silence jetty a bit, because it throws too many messages on INFO. This can be overridden in included files -->
<logger name="org.eclipse.jetty" level ="warn" />

Update logging settings in the server context

You can modify the logging configuration by editing the configuration file located in <ATACCAMA_HOME>/server/etc/logback-extension.xml. By default, the configuration is scanned for changes every 10 seconds, so a server restart is not required.

Appenders

The following basic types of log appenders specified in the class attribute are:

ch.qos.logback.core.ConsoleAppender
ch.qos.logback.core.FileAppender
ch.qos.logback.classic.db.DBAppender
ch.qos.logback.core.rolling.RollingFileAppender

Console Appender

This log appender writes log records to the standard and/or error output. In addition to name, this appender has the following attributes:

encoder - Determines the manner in which an event is written (defined within the pattern attribute).
- The pattern attribute sets the pattern of the log messages. In the following example, the pattern attribute sets the timestamp, aligns the LEVEL (severity) to seven characters, prints the logger, prints the message itself, and ends with a system-specific endline.

Example configuration

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%d{yyy.MM.dd HH:mm:ss} %-7([%level])   %logger %msg%n</pattern>
    </encoder>
</appender>

File Appender

This log appender writes log records to the file specified by configuration parameters. In addition to name, this appender has the following attributes:

file - Name of the file where log records are stored.
append - If the file already exists, new events are appended at the end. Alternatively, the log file is cleared after restart.
encoder - Determines the manner in which an event is written (defined within the pattern attribute).

Example configuration

<appender name="server-log" class="ch.qos.logback.core.FileAppender">
   <file>../storage/server.log</file>
   <append>true</append>
   <encoder>
      <pattern>%d{yyy.MM.dd HH:mm:ss} %-7([%level])   %logger %msg%n</pattern>
   </encoder>
</appender>

DB Appender

This appender inserts logging events into four database tables in a format independent of the Java programming language. It is very useful in a high availability multi-node setup as it can write logs from several nodes into one table, that is, it can store logs from the whole cluster in one place. For more information, see Advanced Database Appender Configuration.

First, create tables according to the scripts that can be found in the /runtime/lib/logback-classic.jar. In the JAR file, the scripts are located in /ch/qos/logback/classic/db/script.
Next, copy the necessary drivers to the classpath.
Finally, set up the appender according to the following information. The following attributes are used for the appender:
- name - Defines the name of the appender.
- connectionSource - Defines how to acquire the JDBC driver.
- driverClass - Specific name of the driver class.
- url - Connection string.
- user - User with permission to write to the database.
- password - User’s password.

Example configuration

<appender name="DB" class="ch.qos.logback.classic.db.DBAppender">
  <connectionSource class="ch.qos.logback.core.db.DriverManagerConnectionSource">
    <driverClass>com.microsoft.sqlserver.jdbc.SQLServerDriver</driverClass>
    <url>jdbc:sqlserver://HOST:1433;databaseName=MY_DB</url>
    <user>MY_USER</user>
    <password>STRONG_PASSWORD</password>
  </connectionSource>
</appender>


 <root level="info">
    <appender-ref ref="DB" />
 </root>

Rolling File Appender

This appender extends the File Appender with the capability to roll over log files. It can log to a file named log.txt and, once a certain condition is met, the file contents are archived and the file is cleared for new logs. For more information about configuring the rolling file appender, see Advanced Rolling File Appender Configuration.

The RollingFileAppender subcomponent, RollingPolicy, is responsible for undertaking the actions required for a rollover. In addition to name, it is necessary to specify:

file - Name of the file where the current logs are stored.
fileNamePattern - Defines the naming pattern of the archived log files.
maxFileSize - Defines the maximum size of the file before creating a new log file.
maxHistory - Defines the number of archived log files preserved.
totalSizeCap - Controls the maximum size of all the archived and the current log file. It requires the maxHistory to be applied first.
encoder - Determines the manner in which an event is written.

Example configuration

<configuration>
  <appender name="ROLLING" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>mylog.txt</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <!-- rollover daily -->
      <fileNamePattern>mylog-%d{yyyy-MM-dd}.%i.txt</fileNamePattern>
       <!-- each file should be at most 100MB, keep 60 days worth of history, but at most 20GB -->
       <maxFileSize>100MB</maxFileSize>
       <maxHistory>60</maxHistory>
       <totalSizeCap>20GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>%msg%n</pattern>
    </encoder>
  </appender>
  <root level="DEBUG">
    <appender-ref ref="ROLLING" />
  </root>
</configuration>

Loggers

Defining a logging rule consists of specifying the name which is used to process logs from a particular component, severity level for which the rule is triggered, and the appender to which matched log records should be sent.

Simple example

The root logger is the default logger that processes all the logs. It is necessary to specify it in the configuration. Every other logger has the logger name.

The following logger named org.apache.activemq processes all logs from the org.apache.activemq class with the WARN severity level and then sends them to the stdout and server-log appenders. Attributes that must be defined are:

name - The name of the logger. It is usually the name of the class you wish to log.
level - The severity level for which the rule is triggered. Possible levels are DEBUG, INFO, WARN, ERROR.
additivity - Defines whether the ancestor logger (or root) logs the messages from the defined logger. To turn this behavior off, set additivity to false. If true, the event can be logged multiple times (by parent loggers).
appender-ref - Specifies which appenders should be used. It needs to be specified with the attribute ref and the appender name from the appender defined in the configuration.

<root level="info">
   <appender-ref ref="server-log" />
</root>
<logger name="org.apache.activemq" level="warn" additivity="false">
   <appender-ref ref="server-log" />
   <appender-ref ref="STDOUT" />
</logger>

This root logger would catch a log record like the following:

2019.05.17 18:36:15 [INFO]    NmeServerComponent.sor_consent_validate.comp Starting plan...

Full configuration example

The following example shows several appenders specified and commented out. In order to use them, uncomment the appender you wish to configure.

When defining where logs should be stored, you need to provide absolute file paths. Otherwise, the log files are not created.

<?xml version="1.0" encoding="UTF-8"?>
<included>
    <!-- Configure internal Ataccama logging -->

    <!-- Use predefined loggers and appenders provided by Ataccama, configured with the following properties -->
    <property name="logging.useDefaultConfig" value="true" />

    <!-- Configure the root logger level, and STDOUT appender filter for third-party software -->
    <property name="root.level" value="INFO" />
    <property name="stdout.level" value="DEBUG" />

    <!-- Configure the logging level only for the Ataccama engine -->
    <property name="ataccama.level" value="DEBUG" />
    <property name="ataccama.stdout.level" value="DEBUG" />
    <property name="ataccama.additivity" value="false" />

    <!--Configure the Ataccama engine to log into the file -->
    <!-- <appender name="ATACCAMA_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>../logs/ataccama.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>ataccama.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} %-7([%level])   %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="ataccama" additivity="false">
        <appender-ref ref="ATACCAMA_FILE" />
    </logger>
    -->

    <!--Configure the HttpDispatcher.accessLog to log into the file -->
    <!--
    <appender name="ACCESS_LOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>../logs/access.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>../logs/access.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} %-7([%level])   %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="ataccama.HttpDispatcher.accessLog" level="INFO">
        <appender-ref ref="ACCESS_LOG" />
    </logger>
    -->

    <!--  Debug Spring or Keycloak security -->
    <!--
    <appender name="SECURITY_LOG" class="ch.qos.logback.core.FileAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>DEBUG</level>
        </filter>
        <file>logs/security.log</file>
        <append>true</append>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} %-7([%level])   %logger %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="org.keycloak" level="DEBUG">
        <appender-ref ref="SECURITY_LOG" />
    </logger>
    <logger name="org.springframework.security" level="DEBUG">
        <appender-ref ref="SECURITY_LOG" />
    </logger>
    -->
</included>

Useful hints

For more information, see the official Logback documentation.

Configuration

Basic logging configuration (like in the previous example) can be found bundled within the product Java libraries.

To extend the basic configuration, put the logback-extension.xml file into the workdir of the application.
To use a custom location for the extension file, you can use Java option -Dlogging.logbackExtensionFile=<custom_location>/logback-extension.xml.
To create a new configuration instead of using the default one, use -Dlogback.configurationFile=/path/to/config.xml.

Debugging

Use the Java opt -Dlogback.debug=true to find useful information for debugging such as which configuration files are used, which were not found, and so on. This option is very useful during the initial logging setup.

Enable logging SQL queries

You can log the SQL queries and their parameters used in JDBC Writer step and Execute SQL tasks in plans and workflows. The SQL statements are logged separately from the parameters that they take.

To enable this option, the following parameters need to be passed to the Java Virtual Machine:

sql.logStatements=true: Enables logging SQL queries.
sql.logMaxParams=20: Defines how many parameters are logged. The default value is 20.
sql.logSubstParams=true: If set to true, the runtime logs all queries with a full SQL statement that includes the parameters. If set to false, the runtime logs a short version of the queries that consists only of parameters without the SQL statement.

There are several ways to provide these parameters depending on how you start jobs. For more information, see the articles JVM Configuration and Runtime Properties.

The following example shows the expected output in the log file of the task when inserting new rows in a table:

25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] INSERT INTO party (src_name,src_gender,src_birth_date,src_sin,src_card,src_address,src_email,src_primary_key,meta_last_update) VALUES (?,?,?,?,?,?,?,?,?)
25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] [Dr.
John Smith, M, 12/16/1978, 000000000, 88682239496, 14618 110 Ave Surrey V3R2A9, john.smith@.com, 1, 2006-12-06 00:00:00.0]
25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] [Smith W.
John, M, 16.12.1978, 095-242-434, 266805807984498, Surrey 14618 110 Ave, john.w.smith@gmail.com, 2, 2007-04-15 00:00:00.0]
25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] [John William Smith, 12, 781216, SIN095242434, 12, 25 Linden Str Toronto M4X 1V5, john.will.smith@gmail.com, 3, 2009-06-28 00:00:00.0]
25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] [Dr.
J.W.
Smith, M, 11/16/78, 095242433, 4334338874158390, 12, dr.john.smith@gmail.c, 4, 2005-03-20 00:00:00.0]
25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] [John Smith, 12, 16.11.1978, 095252433, 431768161757282, 8500 Leslie L3T 7M8 Toronto, smith.john@gmail.com, 5, 2008-11-03 00:00:00.0]

Was this page useful?