How to use Log4j2 SDK v2

Prev Next

Available in Classic and VPC

This chapter explains how to use the Effective Log Search & Analytics Log4j2 SDK. NELO is the project code name for Effective Log Search & Analytics.

Add the NELO2 Log4j2 SDK dependency

Add the dependency as follows.
After extracting the compressed file, add the paths of the core module and the log4j2 module to <systemPath>.

<dependencies>
        <dependency>
            <groupId>nelo2-java-sdk-core</groupId>
            <artifactId>nelo2-java-sdk-core</artifactId>
            <version>1.6.6</version>
            <scope>system</scope>
            <systemPath>/nelo2-java-sdk-core-1.6.6.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>nelo2-java-sdk</groupId>
            <artifactId>nelo2-java-sdk-log4j2</artifactId>
            <version>2.8.5</version>
            <scope>system</scope>
            <systemPath>/nelo2-java-sdk-log4j2-2.8.5.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.8.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.8.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>2.8.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libthrift</artifactId>
            <version>0.9.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.2.6</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.3.1</version>
        </dependency>
</dependencies>

Note

  • This SDK includes log4j-slf4j-impl, which is the SLF4J binding library for Log4j2.
    Since SLF4J supports only one binding at a time, you must not use it together with other SLF4J binding libraries.

  • If the existing reference library overlaps with those referenced by the NELO2 Log4j2 SDK, problems may occur.
    In this case, it is recommended to use the later version.

Effective Log Search & Analytics Log4j2 appender settings and options

The following is an example of a log4j2.xml configuration file that includes the Effective Log Search & Analytics Log4j2 appender:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" shutDownHook="disable">
    <Appenders>
        <Console name="console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <ThriftAppender>
            <name>nelo</name>
            <collectorUrl>COLLECTOR_HOST</collectorUrl>
            <port>COLLECTOR_PORT</port>
            <projectName>PROJECT_KEY</projectName>
            <version>PROJECT_VERSION</version>
            <logType>LOG_TYPE</logType>
            <logSource>LOG_SOURCE</logSource>
        </ThriftAppender>
        <HttpAppender>
            <name>http</name>
            <collectorUrl>COLLECTOR_HOST</collectorUrl>
            <port>COLLECTOR_PORT</port>
            <projectName>PROJECT_KEY</projectName>
            <version>PROJECT_VERSION</version>
            <logType>LOG_TYPE</logType>
            <logSource>LOG_SOURCE</logSource>
        </HttpAppender>
    </Appenders>
    <Loggers>
        <Logger name="nelo" level="error" additivity="false">
            <AppenderRef ref="nelo"/>
            <AppenderRef ref="console"/>
        </Logger>
        <Root level="warn">
            <AppenderRef ref="nelo"/>
            <AppenderRef ref="console"/>
        </Root>
    </Loggers>
</Configuration>

You can set the following options:

  • Appender: Select the appender class name based on the transfer protocol.

    • Thrift Appender : ThriftAppender

    • Http Appender : HttpAppender

  • projectName: Project key. You can check the project key in the project information of Effective Log Search & Analytics.

  • version: Project version. (Only English letters, numbers, -, _, . are allowed, and it should start with an English letter, number, or _.)

  • collectorUrl: URL of the Effective Log Search & Analytics log collector server.

    • ThriftAppender: elsa-col.ncloud.com

    • HttpAppender: (http://elsa-col.ncloud.com/_store

  • port: Collector server port.

    • Thrift Appender : 10006

    • Http Appender : 80

  • enable: Use status. The default value is true.

  • logType: Set logType.

  • logSource: Set logSource.

  • errorCodeType: Error code type.

    • Default: Uses exception information from Log4j basic information. If exception information is not delivered (for example, when the log is recorded in the log.error(message) format), the entire error message is used as the error code.

      Example: When a NullPointerException occurs → NullPointerException

    • message: Uses only the part from the beginning of the error message up to the first whitespace.
      Example: Download has failed. → Download error

    • mdc: Set the value of the "errorCode" item in SLF4J MDC to use.
      Example: MDC.put("errorCode", "Login") → Login

  • debug: Displays debugging information of Effective Log Search & Analytics. The default value is false.

    • This option applies globally, and "true" takes precedence over "false." If multiple appenders are declared and only one of them has the debug value set to "true," debug logs are output to the console from all appenders.

  • timeout: Socket timeout used for transmission. The default value is 5000 ms (5 seconds).

  • keepAlive: Socket keepAlive timeout used for transmission. The default value is 60,000 ms (1 minute), and the maximum value is 180,000 ms (3 minutes).

  • isBulkEnabled: Specifies whether to use bulk mode. The default value is true. If set to false, each log is sent individually.

  • bulkSize: Maximum number of logs to be sent in a single bulk when using bulk mode. The default value is 1000, and the maximum value is 100,000.

  • bulkInterval: Cycle to call bulk requests when using bulk mode. The default value is 1000 ms (1 second), and the maximum value is 10,000 ms (10 seconds).

  • alwaysIncludeLocation: Specifies whether the SDK adds the Location field to all logs. The default value is true.

    • false: Check and set the Location field only for logs with logLevel set to "ERROR."
    • true: Check and set the Location field for all logs. This may negatively affect logging performance compared to false.

  • mdcConversionRule: A rule for renaming MDC keys.

    • format: key1:newKey1;key2:newkey2;...
      Example: If you set mdcConversionRule to time:date;fullname:name and the keys exist, the MDC key time is renamed to "date" and "fullname" is renamed to "name."

Bulk and single modes

The NELO2 Log4j2 SDK supports both single mode, which sends logs individually, and bulk mode, which sends logs in batches.

Set isBulkEnabled to true or false in the XML appender configuration to use bulk or single mode (default: true).

For performance reference, see the table below according to the protocol.

Throughput when sending 1 KB logs from a single thread for 1 minute

  • thrift
    • single mode: 2636.00 logs/sec
    • bulk mode: 6369.10 logs/sec
  • http
    • single mode: 583.36 logs/sec
    • bulk mode: 4618.90 logs/sec

The specifications for the equipment used in the performance test above are as follows:

  • Log transfer server: 2 GHZ 12 core CPU, 48 G memory
  • Log collecting server: 2.26 GHZ 12 core CPU, 48 G memory

Note

  • The transfer performance may differ depending on the load.
    The test was run with no load, and it shows relatively low performance when transferring with the actual instance in use.
    Different performance due to the instance load can be perceived more clearly in the single mode than the bulk mode.
    Therefore, it is recommended to use the default bulk mode.

  • The maximum size of the packet that the collecting server allows is 30 MB.
    Configure an appropriate bulkSize based on the client server's log pattern. (Default: 1000)

How to use AsyncAppender

In addition to the default ThriftAppender, you can configure Log4j2's AsyncAppender to perform log transmission in a separate thread.
See below to configure. Set includeLocation of the AsyncAppender to "true." Otherwise, an error occurs during the transfer.
For details on configuring the AsyncAppender, see the Log4j2 manual.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" shutDownHook="disable">
    <Appenders>
        <Console name="console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <ThriftAppender>
            <name>nelo</name>
            <collectorUrl>COLLECTOR_HOST</collectorUrl>
            <port>COLLECTOR_PORT</port>
            <projectName>PROJECT_KEY</projectName>
            <version>PROJECT_VERSION</version>
            <logType>LOG_TYPE</logType>
            <logSource>LOG_SOURCE</logSource>
        </ThriftAppender>
        <HttpAppender>
            <name>http</name>
            <collectorUrl>COLLECTOR_HOST</collectorUrl>
            <port>COLLECTOR_PORT</port>
            <projectName>PROJECT_KEY</projectName>
            <version>PROJECT_VERSION</version>
            <logType>LOG_TYPE</logType>
            <logSource>LOG_SOURCE</logSource>
        </HttpAppender>
        <Async name="async">
            <AppenderRef ref="nelo"/>
            <includeLocation>true</includeLocation>
        </Async>
    </Appenders>
    <Loggers>
        <Logger name="nelo" level="error" additivity="false">
            <AppenderRef ref="async"/>
            <AppenderRef ref="console"/>
        </Logger>
        <Root level="warn">
            <AppenderRef ref="nelo"/>
            <AppenderRef ref="console"/>
        </Root>
    </Loggers>
</Configuration>

Effective Log Search & Analytics Log4j2 SDK usage example

The following is a sample code that demonstrates the actual use of the NELO2 Log4j2 SDK:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

private static final Logger logger = LoggerFactory.getLogger("nelo");
...
    logger.debug("Effective Log Search & Analytics log4j2 SDK Debug Message");
    try {
        String npe = null;
        npe.toString();
    } catch(Exception e) {
        logger.error("Effective Log Search & Analytics log4j2 SDK Exception", e);
    }

Service limits

  • When using the AsyncAppender, if the log generation speed exceeds the transfer speed, logs that exceed the queue size are not transferred.
  • It is recommended to select between the AsyncAppender and the default appender according to the following criteria:
    • If you want to minimize log loss: Use default appender.
    • If you are concerned about application performance degradation in case of a NELO2 system failure: Use AsyncAppender.

Troubleshooting

1. When error data occurs but cannot be viewed in Admin

Check if the projectName is correct when there is no error in the result message after transferring the logs to Effective Log Search & Analytics' collecting server.

Check whether actual error data is being sent. Check if the following transfer log is exported after setting the Effective Log Search & Analytics NELO appender's debug attribute to true in the configuration file (log4j2.xml).

<!-- define nelo appender -->
<ThriftAppender>
    <name>nelo</name>
    <collectorUrl>COLLECTOR_HOST</collectorUrl>
    <port>COLLECTOR_PORT</port>
    <projectName>PROJECT_KEY</projectName>
    <version>PROJECT_VERSION</version>
    <logType>LOG_TYPE</logType>
    <logSource>LOG_SOURCE</logSource>
    <debug>true</debug>
</ThriftAppender>

[NELO2] Log Append : sent event, return value :
…

FAQ

1. How to use AsyncAppender in a batch program (or a simple test program)

You need to add code at the end of the batch program to wait for a few seconds.

try {
    Thread.sleep(3000L);
} catch (InterruptedException ignore){}

AsyncAppender transfers logs asynchronously through a separate daemon thread that records logs internally.

In a Java batch program, the main thread terminates immediately, causing the batch application to exit before the Log4j AsyncAppender’s daemon thread can be created and send logs.

The JVM terminates immediately if there are no non-daemon threads running, regardless of daemon threads.

Therefore, add a wait statement at the end of the program as described above, so that all logs are sent before the application exits.

2. How to log Java stack traces in Log4j (including NELO2)

To print a stack trace in Action, BO, DAO, or Java batch programs using Log4j, use the form log.error(e.getMessage(), e);.

The SLF4J Logger does not support logging methods that take only a Throwable as an argument.

String[] aa = null;
try {
    aa[0] = "111";
} catch (NullPointerException e) {
//    log.error(e); //Method not supported in SLF4J.
    log.error(e.getMessage(), e); ///Prints the stack trace
}

3. How to minimize performance degradation caused by Log4j2 logging

Maximize filtering by using name and level in the logger settings of log4j2.xml.

For example, if you set com or org to the DEBUG level in the logger settings, a large number of ILoggingEvent (Log4j) instances are unnecessarily generated.

Even if the threshold in the NELO Log4j2 appender is set to ERROR and the logs are not actually transmitted, the ILoggingEvent instances are still generated by the logger and delivered to the appender.

Settings that degrade performance (for development use only)

<Loggers>
    <Logger name="nelo" level="debug" additivity="false">
        <AppenderRef ref="nelo"/>
        <AppenderRef ref="console"/>
    </Logger>
    <Root level="debug">
        <AppenderRef ref="nelo"/>
        <AppenderRef ref="console"/>
    </Root>
</Loggers>

Settings that consider performance (for production use)

<Loggers>
    <Logger name="nelo" level="error" additivity="false">
        <AppenderRef ref="nelo"/>
        <AppenderRef ref="console"/>
    </Logger>
    <Root level="warn">
        <AppenderRef ref="nelo"/>
        <AppenderRef ref="console"/>
    </Root>
</Loggers>

4. What to do if a timeout occurs during Thrift bulk transfer

If logs are not successfully sent to the collector server, you may see logs similar to the following:

[NELO2] sendMessage (1426319665440) sendBulk failed..  Error occur : java.net.SocketTimeoutException: Read timed out

In this case, increase the timeout in the XML appender settings and decrease the bulkSize to reduce the amount of data included in a single packet.

5. Cautions on memory usage

The provided SDK uses bulk transfer mode by default, and the default value of bulkSize is 1000.

As each Effective Log Search & Analytics log includes multiple fields such as projectName, it occupies about 1 KB of memory even when the log body is very short.

Therefore, with the default settings, a NELO2 bulk occupies approximately (log size + 1 KB) × 1000 of memory.

When starting a Java process, you can specify the maximum heap size using the -Xmx option. At this point, consider additional memory usage described above.

Be more cautious when you set the bulkSize option even larger.

Business Registration Number: 129-86-31394 E-commerce Permit Number:No. 2009-GyeonggiSeongnam-0510
CEO: Kim Yuwon Address: NAVER Green Factory 17th–26th Floors, Buljeong-ro 6, Bundang-gu, Seongnam-si, Gyeonggi-do 13561, Republic of Korea. Customer Support Number: 1544-5876
© NAVER Cloud Corp. All Rights Reserved.