Lead your team forward
OCT 24 / 9AM ET Register nowLogging Configuration
ONE Runtime Server can log information about the execution of ONE plans and the behavior of server components.
You can configure where the logging information is displayed. It can be sent to the console (standard or error output) or to a file with up to four levels of logging messages (listed in order of increasing severity):
-
DEBUG
-
INFO
-
WARN
-
ERROR
Logging configuration is specified by a series of XML elements in a configuration file.
The default logging configuration file is <ATACCAMA_HOME>/server/etc/logback-extension.xml
.
The location of the configuration file can be specified by the Logging Component in case ONE Runtime Server is used in the server mode.
For the batch mode, logging configuration is specified in Runtime Configuration.
If no logging configuration file is supplied, the default configuration is used: log records with the severity level INFO
and higher are written to the console (standard output).
Configuration
Default settings
The following configuration represents the default logging configuration, which is used when no logging configuration is provided for ONE Runtime Server.
The configuration contains one logger (root
), which catches all log records with the severity level INFO
and higher and sends them to the Console
appender named STDOUT
.
The Console appender is configured to send log records to the standard output (the console).
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yyy.MM.dd HH:mm:ss} %-7([%level]) %logger %msg%n</pattern>
</encoder>
</appender>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
...
<!-- silence jetty a bit, because it throws too many messages on INFO. This can be overridden in included files -->
<logger name="org.eclipse.jetty" level ="warn" />
Appenders
The following basic types of log appenders specified in the class
attribute are:
-
ch.qos.logback.core.ConsoleAppender
-
ch.qos.logback.core.FileAppender
-
ch.qos.logback.classic.db.DBAppender
-
ch.qos.logback.core.rolling.RollingFileAppender
Console Appender
This log appender writes log records to the standard and/or error output.
In addition to name
, this appender has the following attributes:
-
encoder
- Determines the manner in which an event is written (defined within thepattern
attribute).-
The
pattern
attribute sets the pattern of the log messages. In the following example, thepattern
attribute sets the timestamp, aligns theLEVEL
(severity) to seven characters, prints the logger, prints the message itself, and ends with a system-specific endline.
-
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yyy.MM.dd HH:mm:ss} %-7([%level]) %logger %msg%n</pattern>
</encoder>
</appender>
File Appender
This log appender writes log records to the file specified by configuration parameters.
In addition to name
, this appender has the following attributes:
-
file
- Name of the file where log records are stored. -
append
- If the file already exists, new events are appended at the end. Alternatively, the log file is cleared after restart. -
` encoder` - Determines the manner in which an event is written (defined within the
pattern
attribute).
<appender name="server-log" class="ch.qos.logback.core.FileAppender">
<file>../storage/server.log</file>
<append>true</append>
<encoder>
<pattern>%d{yyy.MM.dd HH:mm:ss} %-7([%level]) %logger %msg%n</pattern>
</encoder>
</appender>
DB Appender
This appender inserts logging events into four database tables in a format independent of the Java programming language. It is very useful in a high availability multi-node setup as it can write logs from several nodes into one table, that is, it can store logs from the whole cluster in one place. For more information, see Advanced Database Appender Configuration.
-
First, create tables according to the scripts that can be found in the
/runtime/lib/logback-classic.jar
. In the JAR file, the scripts are located in/ch/qos/logback/classic/db/script
. -
Next, copy the necessary drivers to the classpath.
-
Finally, set up the appender according to the following information. The following attributes are used for the appender:
-
name
- Defines the name of the appender. -
` connectionSource` - Defines how to acquire the JDBC driver.
-
driverClass
- Specific name of the driver class. -
` url` - Connection string.
-
` user` - User with permission to write to the database.
-
password
- User’s password.
-
appender name="DB" class="ch.qos.logback.classic.db.DBAppender">
<connectionSource class="ch.qos.logback.core.db.DriverManagerConnectionSource">
<driverClass>com.microsoft.sqlserver.jdbc.SQLServerDriver</driverClass>
<url>jdbc:sqlserver://HOST:1433;databaseName=MY_DB</url>
<user>MY_USER</user>
<password>STRONG_PASSWORD</password>
</connectionSource>
</appender>
<root level="info">
<appender-ref ref="DB" />
</root>
----
=== Rolling File Appender
This appender extends the File Appender with the capability to roll over log files.
It can log to a file named `log.txt` and, once a certain condition is met, the file contents are archived and the file is cleared for new logs.
For more information about configuring the rolling file appender, see xref:advanced-rolling-file-appender-configuration.adoc[].
The RollingFileAppender subcomponent, `RollingPolicy`, is responsible for undertaking the actions required for a rollover.
In addition to `name`, it is necessary to specify:
* `file` - Name of the file where the current logs are stored.
* `fileNamePattern` - Defines the naming pattern of the archived log files.
* `maxFileSize` - Defines the maximum size of the file before creating a new log file.
* `maxHistory` - Defines the number of archived log files preserved.
* `totalSizeCap` - Controls the maximum size of all the archived and the current log file.
It requires the `maxHistory` to be applied first.
* `encoder` - Determines the manner in which an event is written.
.Example configuration
[source,xml]
<configuration> <appender name="ROLLING" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>mylog.txt</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- rollover daily -→ <fileNamePattern>mylog-%d{yyyy-MM-dd}.%i.txt</fileNamePattern> <!-- each file should be at most 100MB, keep 60 days worth of history, but at most 20GB -→ <maxFileSize>100MB</maxFileSize> <maxHistory>60</maxHistory> <totalSizeCap>20GB</totalSizeCap> </rollingPolicy> <encoder> <pattern>%msg%n</pattern> </encoder> </appender> <root level="DEBUG"> <appender-ref ref="ROLLING" /> </root> </configuration>
== Loggers Defining a logging rule consists of specifying the `name` which is used to process logs from a particular component, severity `level` for which the rule is triggered, and the `appender` to which matched log records should be sent. === Simple example The `root` logger is the default logger that processes all the logs. It is necessary to specify it in the configuration. Every other logger has the `logger` name. The following logger named `org.apache.activemq` processes all logs from the `org.apache.activemq` class with the `WARN` severity level and then sends them to the `stdout` and `server-log` appenders. Attributes that must be defined are: * `name` - The name of the logger. It is usually the name of the class you wish to log. * `level` - The severity level for which the rule is triggered. Possible levels are `DEBUG`, `INFO`, `WARN`, `ERROR`. * `additivity` - Defines whether the ancestor logger (or root) logs the messages from the defined logger. To turn this behavior off, set additivity to `false`. If `true`, the event can be logged multiple times (by parent loggers). * `appender-ref` - Specifies which appenders should be used. It needs to be specified with the attribute `ref` and the appender name from the appender defined in the configuration. [source,xml]
<root level="info"> <appender-ref ref="server-log" /> </root> <logger name="org.apache.activemq" level="warn" additivity="false"> <appender-ref ref="server-log" /> <appender-ref ref="STDOUT" /> </logger>
This root logger would catch a log record like the following: [source,xml]
2019.05.17 18:36:15 [INFO] NmeServerComponent.sor_consent_validate.comp Starting plan…
== Full configuration example The following example shows several appenders specified and commented out. In order to use them, uncomment the appender you wish to configure. IMPORTANT: When defining where logs should be stored, you need to provide absolute file paths. Otherwise, the log files are not created. [source,xml]
<?xml version="1.0" encoding="UTF-8"?> <included> <!-- Configure internal Ataccama logging -→
<!-- Use predefined loggers and appenders provided by Ataccama, configured with the following properties --> <property name="logging.useDefaultConfig" value="true" />
<!-- Configure the root logger level, and STDOUT appender filter for third-party software --> <property name="root.level" value="INFO" /> <property name="stdout.level" value="DEBUG" />
<!-- Configure the logging level only for the Ataccama engine --> <property name="ataccama.level" value="DEBUG" /> <property name="ataccama.stdout.level" value="DEBUG" /> <property name="ataccama.additivity" value="false" />
<!--Configure the Ataccama engine to log into the file --> <!-- <appender name="ATACCAMA_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>../logs/ataccama.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>ataccama.%d{yyyy-MM-dd}.log</fileNamePattern> <maxHistory>30</maxHistory> </rollingPolicy> <encoder> <pattern>%d{yyyy-MM-dd HH:mm:ss} %-7([%level]) %msg%n</pattern> </encoder> </appender>
<logger name="ataccama" additivity="false"> <appender-ref ref="ATACCAMA_FILE" /> </logger> -->
<!--Configure the HttpDispatcher.accessLog to log into the file --> <!-- <appender name="ACCESS_LOG" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>../logs/access.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>../logs/access.%d{yyyy-MM-dd}.log</fileNamePattern> <maxHistory>30</maxHistory> </rollingPolicy> <encoder> <pattern>%d{yyyy-MM-dd HH:mm:ss} %-7([%level]) %msg%n</pattern> </encoder> </appender>
<logger name="ataccama.HttpDispatcher.accessLog" level="INFO"> <appender-ref ref="ACCESS_LOG" /> </logger> -->
<!-- Debug Spring or Keycloak security --> <!-- <appender name="SECURITY_LOG" class="ch.qos.logback.core.FileAppender"> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>DEBUG</level> </filter> <file>logs/security.log</file> <append>true</append> <encoder> <pattern>%d{yyyy-MM-dd HH:mm:ss} %-7([%level]) %logger %msg%n</pattern> </encoder> </appender>
<logger name="org.keycloak" level="DEBUG"> <appender-ref ref="SECURITY_LOG" /> </logger> <logger name="org.springframework.security" level="DEBUG"> <appender-ref ref="SECURITY_LOG" /> </logger> --> </included>
== Useful hints For more information, see the official https://logback.qos.ch/documentation.html[Logback documentation^]. === Configuration Basic logging configuration (like in the previous example) can be found bundled within the product Java libraries. * To extend the basic configuration, put the `logback-extension.xml` file into the `workdir` of the application. * To use a custom location for the extension file, you can use Java option `-Dlogging.logbackExtensionFile=<custom_location>/logback-extension.xml`. * To create a new configuration instead of using the default one, use `-Dlogback.configurationFile=/path/to/config.xml`. === Debugging Use the Java opt `-Dlogback.debug=true` to find useful information for debugging such as which configuration files are used, which were not found, and so on. This option is very useful during the initial logging setup. === Enable logging SQL queries You can log the SQL queries and their parameters used in JDBC Writer step and Execute SQL tasks in plans and workflows. The SQL statements are logged separately from the parameters that they take. To enable this option, the following parameters need to be passed to the Java Virtual Machine: * `sql.logStatements=true`: Enables logging SQL queries. * `sql.logMaxParams=20`: Defines how many parameters are logged. The default value is `20`. * `sql.logSubstParams=true`: If set to `true`, the runtime logs all queries with a full SQL statement that includes the parameters. If set to `false`, the runtime logs a short version of the queries that consists only of parameters without the SQL statement. There are several ways to provide these parameters depending on how you start jobs. For more information, see the articles xref:jvm-configuration.adoc[] and xref:runtime-properties.adoc[]. The following example shows the expected output in the log file of the task when inserting new rows in a table: [source,bash]
25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] INSERT INTO party (src_name,src_gender,src_birth_date,src_sin,src_card,src_address,src_email,src_primary_key,meta_last_update) VALUES (?,?,?,?,?,?,?,?,?) 25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] [Dr. John Smith, M, 12/16/1978, 000000000, 88682239496, 14618 110 Ave Surrey V3R2A9, john.smith@.com, 1, 2006-12-06 00:00:00.0] 25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] [Smith W. John, M, 16.12.1978, 095-242-434, 266805807984498, Surrey 14618 110 Ave, john.w.smith@gmail.com, 2, 2007-04-15 00:00:00.0] 25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] [John William Smith, 12, 781216, SIN095242434, 12, 25 Linden Str Toronto M4X 1V5, john.will.smith@gmail.com, 3, 2009-06-28 00:00:00.0] 25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] [Dr. J.W. Smith, M, 11/16/78, 095242433, 4334338874158390, 12, dr.john.smith@gmail.c, 4, 2005-03-20 00:00:00.0] 25.05.2020 18:33:04 [INFO] [Jdbc Writer][c2][s1] [John Smith, 12, 16.11.1978, 095252433, 431768161757282, 8500 Leslie L3T 7M8 Toronto, smith.john@gmail.com, 5, 2008-11-03 00:00:00.0]
Was this page useful?