How to use Logback SDK v2

Prev Next

Available in Classic and VPC

This chapter explains how to use the Effective Log Search & Analytics Logback SDK. NELO is the project code name for Effective Log Search & Analytics.

Add the NELO2 Logback SDK dependency

Add the dependency as follows.

After extracting the compressed file, add the paths of the core module and the logback module to <systemPath>.

<dependencies>
        <dependency>
            <groupId>nelo2-java-sdk-core</groupId>
            <artifactId>nelo2-java-sdk-core</artifactId>
            <version>1.6.6</version>
            <scope>system</scope>
            <systemPath>/nelo2-java-sdk-core-1.6.6.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>nelo2-java-sdk</groupId>
            <artifactId>nelo2-java-sdk-logback</artifactId>
            <version>1.6.6</version>
            <scope>system</scope>
            <systemPath>/nelo2-java-sdk-logback-1.6.6.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.1.7</version>
        </dependency>
        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libthrift</artifactId>
            <version>0.9.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.2.6</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.3.1</version>
        </dependency>
</dependencies>

Note

  • This SDK includes and provides logback-classic, the SLF4J binding library of Logback.
    Since SLF4J supports only one binding at a time, you must not use it together with other SLF4J binding libraries.

  • Among Logback libraries, the SLF4J binding library is logback-classic.
    If the existing reference library overlaps with those referenced by the NELO2 Logback SDK, problems may occur.
    In this case, it is recommended to use the later version.

Effective Log Search & Analytics Logback appender settings and options

Logback searches for configuration files in the following order: logback.groovy → logback-test.xml → logback.xml.

This document uses logback.xml as the reference. A common logback.xml configuration is as follows:

<configuration>
    <!-- Console Appender -->
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%-5level]\(%file:%line\) %msg%n</pattern>
        </encoder>
    </appender>
    <!-- Logger -->
    <logger name="com" additivity="false">
        <level value="debug"/>
        <appender-ref ref="STDOUT" />
    </logger>
    <!-- Root Logger -->
    <root level="warn">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

The NELO2 Logback appender can be configured as follows:

<appender name="nelo" class="com.naver.nelo2.logback.ThriftAppender">
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
        <level>TRACE</level>
    </filter>
    <param name="projectName" value="프로젝트 키"></param>
    <param name="collectorUrl" value="elsa-col.ncloud.com"></param>
    <param name="encoding" value="utf-8"></param>
    <param name="timeout" value="1200"></param>
    <param name="enable" value="true"></param>
    <param name="debug" value="false"></param>
    <param name="version" value="1.0.0"></param>
    <param name="errorCodeType" value="default"></param>
    <param name="port" value="10006"></param>
    <param name="isBulkEnabled" value="true"></param>
</appender>

You can set the following options:

  • Appender: Select the appender class name based on the transfer protocol.

    • Thrift Appender : com.naver.nelo2.logback.ThriftAppender

    • Http Appender : com.naver.nelo2.logback.HttpAppender

  • projectName: Project key. You can check the project key in the project information of Effective Log Search & Analytics.

  • version: Project version. (Only English letters, numbers, -, _, . are allowed, and it should start with an English letter, number, or _.)

  • collectorUrl: URL of the Effective Log Search & Analytics log collector server.

    • ThriftAppender: elsa-col.ncloud.com

    • HttpAppender: http://elsa-col.ncloud.com/_store

  • port: Collector server port.

    • Thrift Appender : 10006

    • Http Appender : 80

  • enable: Use status. The default value is true.

  • logType: Set logType.

  • logSource: Set logSource.

  • errorCodeType: Error code type.

    • default: Uses exception information from Logback basic information. If exception information is not delivered (for example, when the log is recorded in the log.error(message) format), the entire error message is used as the error code.

      Example: When a NullPointerException occurs → NullPointerException

    • message: Uses only the part from the beginning of the error message up to the first whitespace.

      Example: Download has failed. → Download error

    • mdc: Set the value of the "errorCode" item in SLF4J MDC to use.

      Example: MDC.put("errorCode", "Login") → Login

  • debug: Prints debugging messages for the NELO2 appender. The default value is false.

    • This option applies globally, and "true" takes precedence over "false." If multiple appenders are declared and only one of them has the debug value set to "true," debug logs are output to the console from all appenders.

  • timeout: Socket timeout used for transmission. The default value is 5000 ms (5 seconds).

  • keepAlive: Socket keepAlive timeout used for transmission. The default value is 60,000 ms (1 minute), and the maximum value is 180,000 ms (3 minutes).

  • isBulkEnabled: Specifies whether to use bulk mode. The default value is true. If set to false, each log is sent individually.

  • bulkSize: Maximum number of logs to be sent in a single bulk when using bulk mode. The default value is 1000, and the maximum value is 100,000.

  • bulkInterval: Cycle to call bulk requests when using bulk mode. The default value is 1000 ms (1 second), and the maximum value is 10,000 ms (10 seconds).

  • alwaysIncludeLocation: Specifies whether the SDK adds the Location field to all logs. The default value is true.

    • false: Check and set the Location field only for logs with logLevel set to "ERROR."
    • true: Check and set the Location field for all logs. This may negatively affect logging performance compared to false.

  • mdcConversionRule: A rule for renaming MDC keys.

    • format: key1:newKey1;key2:newkey2;...

      Example: If you set mdcConversionRule to time:date;fullname:name and the keys exist, the MDC key time is renamed to "date" and "fullname" is renamed to "name."

bulk/single

Effective Log Search & Analytics Logback SDK supports both single mode, which sends logs individually, and bulk mode, which sends logs in batches.

Set isBulkEnabled to true or false in the XML appender configuration to use bulk or single mode (default: true).

For performance reference, see the table below according to the protocol.

Throughput when sending 1 KB logs from a single thread for 10 minute

  • thrift
    • single mode : 2587.93 logs/sec
    • bulk mode : 6492.99 logs/sec
  • http
    • single mode : 595.86 logs/sec
    • bulk mode : 4617.98 logs/sec

The specifications for the equipment used in the performance test above are as follows:

  • Log transfer server: 2 GHZ 12 core CPU, 48 G memory
  • Log collecting server: 2.26 GHZ 12 core CPU, 48 G memory

Note

  • The transfer performance may differ depending on the load.
    The test was run with no load, and it shows relatively low performance when transferring with the actual instance in use.
    Different performance due to the instance load can be perceived more clearly in the single mode than the bulk mode.
    Therefore, it is recommended to use the default bulk mode.

  • The maximum size of the packet that the collecting server allows is 30 MB.
    Configure an appropriate bulkSize based on the client server's log pattern. (Default: 1000)

AsyncAppender

You can get the same type of results by using the AsyncAppender provided by Logback, as shown below.

Pay attention to the following configuration values. queueSize has a default value of 128, but this may not be sufficient for some applications.

includeCallerData has a default value of false. In this case, the AsyncAppender ignores the location information of the log.

See the AsyncAppender configuration example below:

<appender name="nelo-logback-async" class="ch.qos.logback.classic.AsyncAppender">
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
        <level>ERROR</level>
    </filter>
    <param name="neverBlock" value="true" />
    <param name="includeCallerData" value="true" />
    <param name="queueSize" value="2048" />
    <appender-ref ref="nelo-logback" />
</appender>

For details on configuration values, see the Official documentation.

How to use AsyncAppender

In addition to the default ThriftAppender, you can use the LogbackAsyncAppender to perform log transmission in a separate thread.

To do this, set the XML file as follows:

<configuration>
    <!-- Console Appender -->
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%-5level]\(%file:%line\) %msg%n</pattern>
        </encoder>
    </appender>

    <!-- NELO2 Logback Appender -->
    <appender name="nelo-logback" class="com.naver.nelo2.logback.ThriftAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>ERROR</level>
        </filter>
        <param name="projectName" value="프로젝트 키"/>
        <param name="collectorUrl" value="elsa-col.ncloud.com"/>
        <param name="port" value="10006"/>
        <param name="timeout" value="1200" />
        <param name="enable" value="true" />
        <param name="errorCodeType" value="default" />
    </appender>

    <!-- NELO2 Logback Async Appender -->
    <appender name="nelo-logback-async" class="ch.qos.logback.classic.AsyncAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>ERROR</level>
        </filter>
        <param name="neverBlock" value="true" />
        <param name="IncludeCallerData" value="true" />
        <param name="queueSize" value="2048" />
        <appender-ref ref="nelo-logback" />
    </appender>

    <!-- Logger -->
    <logger name="com" additivity="false">
        <level value="debug"/>
        <appender-ref ref="STDOUT" />
        <appender-ref ref="nelo-logback-async" />
    </logger>

    <!-- Root Logger -->
    <root level="warn">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="nelo-logback-async" />
    </root>
</configuration>

Effective Log Search & Analytics Logback SDK usage example

The following is a sample code that demonstrates the actual use of the Effective Log Search & Analytics Logback SDK:

...
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

private static Logger logger = LoggerFactory.getLogger(LogbackTest.class);
...
    logger.debug("Effective Log Search & Analytics Logback SDK Debug Message");
    try {
        String npe = null;
        npe.toString();
    } catch(Exception e) {
        logger.error("Effective Log Search & Analytics Logback SDK Exception", e);
    }

Service limits

  • When using the AsyncAppender, if the log generation speed exceeds the transfer speed, logs that exceed the queue size are not transferred.
  • It is recommended to select between the AsyncAppender and the default appender according to the following criteria:
    • If you want to minimize log loss: Use default appender.
    • If you are concerned about application performance degradation in case of an Effective Log Search & Analytics system failure: Use AsyncAppender.

Troubleshooting

1. When logs are transferred to the Effective Log Search & Analytics but you cannot view them from the web

Check whether the project ID is valid. If the project ID is invalid, you cannot view logs in the web dashboard.

Then check whether actual error data is being sent. Check if the following transfer log is exported after setting the NELO2 appender's debug attribute to true in the configuration file (logback.xml).

<!-- NELO2 Logback Appender -->
<appender name="nelo-logback" class="com.naver.nelo2.logback.ThriftAppender">
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
        <level>ERROR</level>
    </filter>
    <param name="projectName" value="%YOUR_PROJECT_KEY%"/>
    <param name="collectorUrl" value="elsa-col.ncloud.com"/>
    <param name="port" value="10006"/>
    <param name="timeout" value="1200" />
    <param name="enable" value="true" />
    <param name="errorCodeType" value="default" />
    <param name="debug" value="true" />
</appender>

[NELO2] Log Append : sent event, return value :
…

FAQ

1. How to use AsyncAppender in a batch program (or a simple test program)

You need to add code at the end of the batch program to wait for a few seconds.

try {
    Thread.sleep(3000L);
} catch (InterruptedException ignore){}

AsyncAppender transfers logs asynchronously through a separate daemon thread that records logs internally.

In a Java batch program, the main thread terminates immediately, causing the batch application to exit before the Log4j AsyncAppender’s daemon thread can be created and send logs.

The JVM terminates immediately if there are no non-daemon threads running, regardless of daemon threads.

Therefore, add a wait statement at the end of the program as described above, so that all logs are sent before the application exits.

2. How to log Java stack traces in Logback

To print a stack trace in Action, BO, DAO, or Java batch programs using Logback, use the form log.error(e.getMessage(), e);.

The SLF4J Logger does not support logging methods that take only a Throwable as an argument.

String[] aa = null;
try {
    aa[0] = "111";
} catch (NullPointerException e) {
//	log.error(e); //Method not supported in SLF4J.
    log.error(e.getMessage(), e); ///Prints the stack trace
}

How to minimize performance degradation caused by Logback logging

You can maximize filtering by using name and level in the logger settings of logback.xml.

For example, if you set com or org to the DEBUG level in the logger settings, a large number of ILoggingEvent (Logback) instances are unnecessarily generated.

Even if the threshold in the NELO Logback appender is set to ERROR and the logs are not actually transmitted, the ILoggingEvent instances are still generated by the logger and delivered to the appender.

Settings that degrade performance (for development use only)

<!-- Logger -->
<logger name="com" additivity="false">
    <level value="debug"/>
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-logback" />
</logger>

<!-- Logger -->
<logger name="org" additivity="false">
    <level value="debug"/>
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-logback" />
</logger>

<!-- Root Logger -->
<root level="warn">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-logback" />
</root>

Settings that consider performance (for production use)

<!-- Logger -->
<logger name="com" additivity="false">
    <level value="error"/>
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-logback" />
</logger>

<!-- Root Logger -->
<root level="warn">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-logback" />
</root>

4. How to ensure a graceful shutdown when using a WAS

If the WAS (e.g., Tomcat) is shut down while error logs are being transmitted, the following exception may occur, and the WAS may fail to terminate properly.

To prevent this, call the stop() method on the LoggerContext instance when shutting down WAS, and close the NELO2 Java appender. This ensures a graceful shutdown.

Note

Spring provides org.springframework.web.util.Log4jConfigListener for Log4j, but it does not provide a listener for Logback to support graceful shutdown.

The NELO2 Logback SDK provides its own LogbackShutdownListener. By adding the following configuration to web.xml, you can ensure a graceful shutdown even if error logs are being transmitted when the WAS is stopped:

<listener>
    <listener-class>
        com.naver.nelo2.logback.LogbackShutdownListener
    </listener-class>
</listener>

5. What to do if a timeout occurs during Thrift bulk transfer

If logs are not successfully sent to the collector server, you may see logs similar to the following:

[NELO2] sendMessage (1426319665440) sendBulk failed..  Error occur : java.net.SocketTimeoutException: Read timed out

In this case, increase the timeout in the XML appender settings and decrease the bulkSize to reduce the amount of data included in a single packet.

6. Cautions on memory usage

The provided SDK uses bulk transfer mode by default, and the default value of bulkSize is 1000.

As each Effective Log Search & Analytics log includes multiple fields such as projectName, it occupies about 1 KB of memory even when the log body is very short.

Therefore, with the default settings, a bulk occupies approximately (log size + 1 KB) × 1000 of memory.

When starting a Java process, you can specify the maximum heap size using the -Xmx option. At this point, consider additional memory usage described above.

Be more cautious when you set the bulkSize option even larger.

7. Warning messages related to httpClient logging

When using the HTTP protocol, the following log appears when the application starts.

This occurs because the httpClient library uses the Apache logging framework, and the logs generated here remain in the root logger.

SLF4J: The following loggers will not work because they were created
SLF4J: during the default configuration phase of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#substituteLogger
SLF4J: org.apache.http.impl.conn.PoolingClientConnectionManager
SLF4J: org.apache.http.impl.conn.DefaultClientConnectionOperator
SLF4J: org.apache.http.impl.client.DefaultHttpClient
Business Registration Number: 129-86-31394 E-commerce Permit Number:No. 2009-GyeonggiSeongnam-0510
CEO: Kim Yuwon Address: NAVER Green Factory 17th–26th Floors, Buljeong-ro 6, Bundang-gu, Seongnam-si, Gyeonggi-do 13561, Republic of Korea. Customer Support Number: 1544-5876
© NAVER Cloud Corp. All Rights Reserved.