Using the Log4j2 SDK v2

Article Summary

Share feedback

Thanks for sharing your feedback!

Available in Classic and VPC

This chapter describes how to use Effective Log Search & Analytics NELO Log4j2 SDK. NELO is the project code name for Effective Log Search & Analytics.

Add Effective Log Search & Analytics Log4j2 SDK dependency

Add Dependency as below.
After unzip the file, add the path to the core module and log4j2 module to <systemPath>.

Download log4j2 SDK

<dependencies>
        <dependency>
            <groupId>nelo2-java-sdk-core</groupId>
            <artifactId>nelo2-java-sdk-core</artifactId>
            <version>1.6.6</version>
            <scope>system</scope>
            <systemPath>/nelo2-java-sdk-core-1.6.6.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>nelo2-java-sdk</groupId>
            <artifactId>nelo2-java-sdk-log4j2</artifactId>
            <version>2.8.5</version>
            <scope>system</scope>
            <systemPath>/nelo2-java-sdk-log4j2-2.8.5.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.8.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.8.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>2.8.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libthrift</artifactId>
            <version>0.9.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.2.6</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.3.1</version>
        </dependency>
</dependencies>

Remarks

This SDK includes the log4j-slf4j-impl, which is the slf4J binding library for the log4j2.
Because the slf4j supports only one binding at a time, you cannot use any libraries for other slf4j bindings.
Problems may occur if an existing reference library is duplicated with the library referenced by the nelo2 log4j2 SDK.
We recommend using a higher version in this case.

Effective Log Search & Analytics log4j2 appender settings and options

The below is an example the log4j2.xml configuration file containing Effective Log Search & Analytics log4j2 appender.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" shutDownHook="disable">
    <Appenders>
        <Console name="console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <ThriftAppender>
            <name>nelo</name>
            <collectorUrl>COLLECTOR_HOST</collectorUrl>
            <port>COLLECTOR_PORT</port>
            <projectName>PROJECT_ID</projectName>
            <version>PROJECT_VERSION</version>
            <logType>LOG_TYPE</logType>
            <logSource>LOG_SOURCE</logSource>
        </ThriftAppender>
        <HttpAppender>
            <name>http</name>
            <collectorUrl>COLLECTOR_HOST</collectorUrl>
            <port>COLLECTOR_PORT</port>
            <projectName>PROJECT_ID</projectName>
            <version>PROJECT_VERSION</version>
            <logType>LOG_TYPE</logType>
            <logSource>LOG_SOURCE</logSource>
        </HttpAppender>
    </Appenders>
    <Loggers>
        <Logger name="nelo" level="error" additivity="false">
            <AppenderRef ref="nelo"/>
            <AppenderRef ref="console"/>
        </Logger>
        <Root level="warn">
            <AppenderRef ref="nelo"/>
            <AppenderRef ref="console"/>
        </Root>
    </Loggers>
</Configuration>

The following can be set as options.

Appender (Select appender's class name according to a transmission protocol.)
- Thrift Appender : ThriftAppender
- Http Appender : HttpAppender
projectName: Project ID. You can view a project ID created in Effective Log Search & Analytics's project information.
version : Project version (Only English alphabets, numbers, "-" and "" are allowed. It must start with an alphabetic character/number/"").
collectorUrl: Effective Log Search & Analytics log collector server URL
- ThriftAppender: elsa-col.ncloud.com
- HttpAppender: http://elsa-col.ncloud.com/_store
port: Collector server port
- Thrift Appender : 10006
- Http Appender : 80
enable: Whether to use (Default: true)
logType : logType settings
logSource : logSource settings
errorCodeType: Error code type
- default: Use the exception information of the log4j default information. If the exception information is not delivered (if a log is recorded in the log.error(message) format), an entire error message is used as an error code.
  
  e.g.) If a NullPointerException occurs => NullPointerException
- message: Use only from the beginning until the blank of an error message.
  
  e.g.) Download error Download failed. => Download error
- mdc: Set the “errorCode” metric value of SLF4J MDC to use.
  
  e.g.)MDC.put(“errorCode”, “Login”) => Login
debug: Display the debugging information for Effective Log Search & Analytics. The default value is false.
- This option applies globally, and "true" overrides "false". That is, if multiple appenders are declared and one of them has a debug value of true, every appender will output a debug log to the console.
timeout: Timeout of socket used to send. Default: 5000ms (5s)
keepAlive: keepAlive timeout of socket used to send. Default: 60000ms (1m), Maximum: 180000ms (3m)
isBulkEnabled : Whether to use the bulk mode. Default: true. If false, send each log individually.
bulkSize : The maximum number of logs to be sent in a bulk when using the bulk mode. Default: 1000, Maximum: 100000
bulkInterval : interval to call bulk request, defaults to 1000ms(1 sec), maximum 10000ms(10 sec)
alwaysIncludeLocation : whether SDK add Location fields to all logs, defaults to true.
- false : check and set Location fields for logs which logLevel is ERROR.
- true : check and set Location field for all logs. This may bring bad effects to logging performance compare to false.
mdcConversionRule : Rules to rename MDC key
- format: key1:newKey1;key2:newkey2;...
  
  e.g.) Set mdcConversionRule to time:date;fullname:name, this rule will rename MDC key time to date and fullname to name if they are exists.

bulk/single mode

NELO2 log4j2 SDK supports the single mode for sending logs per request and the bulk mode for sending it in batches.

You can set isBulkEnabled to true/false in the xml appender settings to use the bulk/single mode. (Default: true).

Refer to the table below for a performance reference, depending on a protocol.

Throughput when sending a log of 1kb size in a single thread for 1 minute

thrift
- single mode: 2636.00 logs/sec
- bulk mode: 6369.10 logs/sec
http
- single mode: 583.36 logs/sec
- bulk mode: 4618.90 logs/sec

The specifications of the equipment used for the above performance test are as follows.

Log transmission server: 2GHZ 12core cpu, 48G mem
Log collector server: 2.26GHZ 12core cpu, 48G mem

Remarks

Transmission performance may vary depending on load.
Tests were run in a no-load situation and show relatively low performance when sending logs to the real instances in use.
The perceived performance according to instance loads is greater in the single mode than in the bulk mode.
Therefore, we recommend using the default bulk mode.
The collector server allows the packet size up to 30MB.
You must set the appropriate bulkSize considering a client server log pattern. (Default: 1000)

How to use AsyncAppender

In addition to NeloAppender you use by default, you can use Async appender supported in Log4j2 to set the actual log transmission to be performed on a separate thread.
Please refer to the following for configuration. The includeLocation of the async appender must be set to true. Otherwise, an error occurs during transmission.
For the detailed settings of AsyncAppender, please refer to the following log4j2 manual.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" shutDownHook="disable">
    <Appenders>
        <Console name="console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <ThriftAppender>
            <name>nelo</name>
            <collectorUrl>COLLECTOR_HOST</collectorUrl>
            <port>COLLECTOR_PORT</port>
            <projectName>PROJECT_ID</projectName>
            <version>PROJECT_VERSION</version>
            <logType>LOG_TYPE</logType>
            <logSource>LOG_SOURCE</logSource>
        </ThriftAppender>
        <HttpAppender>
            <name>http</name>
            <collectorUrl>COLLECTOR_HOST</collectorUrl>
            <port>COLLECTOR_PORT</port>
            <projectName>PROJECT_ID</projectName>
            <version>PROJECT_VERSION</version>
            <logType>LOG_TYPE</logType>
            <logSource>LOG_SOURCE</logSource>
        </HttpAppender>
        <Async name="async">
            <AppenderRef ref="nelo"/>
            <includeLocation>true</includeLocation>
        </Async>
    </Appenders>
    <Loggers>
        <Logger name="nelo" level="error" additivity="false">
            <AppenderRef ref="async"/>
            <AppenderRef ref="console"/>
        </Logger>
        <Root level="warn">
            <AppenderRef ref="nelo"/>
            <AppenderRef ref="console"/>
        </Root>
    </Loggers>
</Configuration>

Effective Log Search & Analytics log4j2 SDK usage example

The following is an example of the codes that actually uses Nelo2 log4j2 SDK.

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

private static final Logger logger = LoggerFactory.getLogger("nelo");
...
    logger.debug("Effective Log Search & Analytics log4j2 SDK Debug Message");
    try {
        String npe = null;
        npe.toString();
    } catch(Exception e) {
        logger.error("Effective Log Search & Analytics log4j2 SDK Exception", e);
    }

Limitations

When using the async appender, occurring logs that exceed the queue size will not be sent if the log occurrence rate is faster than the transmission rate.
We recommend you to choose the async appender and the default appender according to the criteria below.
- To minimize log loss: Default appender
- If concerned about the application performance degradation in the event of an nelo2 system failure: asnyc appender

Troubleshooting

1. When error data occurred, but cannot view it on Admin

If there are no errors in the result message after sending logs to Effective Log Search & Analytics collector server, check if the "projectName" is correct.

Confirm if the actual error data is sent. In the configuration file(log4j2.xml), set the debug property of Effective Log Search & Analytics nelo appender to true and execute it to check if the following transmission logs are output.

<!-- define nelo appender -->
<ThriftAppender>
    <name>nelo</name>
    <collectorUrl>COLLECTOR_HOST</collectorUrl>
    <port>COLLECTOR_PORT</port>
    <projectName>PROJECT_ID</projectName>
    <version>PROJECT_VERSION</version>
    <logType>LOG_TYPE</logType>
    <logSource>LOG_SOURCE</logSource>
    <debug>true</debug>
</ThriftAppender>

[NELO2] Log Append : sent event, return value :
…

FAQ

1. How do I use AsyncAppender in a batch program (or a simple test program)?

You need to add some codes to wait for a few seconds at the end of the batch program.

try {
    Thread.sleep(3000L);
} catch (InterruptedException ignore){}

AsyncAppender has a separate daemon thread to record logs internally, thus, it delivers the logs asynchronously.

In Java batch program, the main thread is terminated immediately, so the daemon thread of the log4jAsyncAppender is created and the batch application is shut down before sending logs.

JVM is terminated immediately if there are no live general threads, regardless of the daemon thread.

Therefore, add the codes for waiting at the end of the program as above so that it can be terminated after sending all logs.

2. How can I log Java stack traces to the log4j (including NELO2)?

In order to output a stack trace using the log4j in Action, BO, DAO, and Java batch program, use the form of the log.error(e.getMessage(), e);.

SLF4J Logger is a method factor and does not support logging methods that only accept Throwable.

String[] aa = null;
try {
    aa[0] = "111";
} catch (NullPointerException e) {
//    log.error(e); //Methods not supported by SLF4J.
    log.error(e.getMessage(), e); ///Output stacktrace
}

3. How can I minimize performance degradation caused by the log4j logging?

Use a name and level in the log4j.xml logger settings to maximize filtering.

If you set com or org to DEBUG level in the logger settings as below, lots of ILoggingEvent(log4j) is unnecessarily generated in the logger.

Threshold is set to ERROR in the nelo log4j2 appender, so no log is actually sent, but ILoggingEvent is created in the logger and passed to the appender.

Settings that degrade performance (for developing only)

<Loggers>
    <Logger name="nelo" level="debug" additivity="false">
        <AppenderRef ref="nelo"/>
        <AppenderRef ref="console"/>
    </Logger>
    <Root level="debug">
        <AppenderRef ref="nelo"/>
        <AppenderRef ref="console"/>
    </Root>
</Loggers>

Settings that consider performance (for operating)

<Loggers>
    <Logger name="nelo" level="error" additivity="false">
        <AppenderRef ref="nelo"/>
        <AppenderRef ref="console"/>
    </Logger>
    <Root level="warn">
        <AppenderRef ref="nelo"/>
        <AppenderRef ref="console"/>
    </Root>
</Loggers>

4. If a timeout occurs during the thrift bulk transmission

If a log was not sent normally to a collector server, you can check the following logs.

[NELO2] sendMessage (1426319665440) sendBulk failed..  Error occur : java.net.SocketTimeoutException: Read timed out

In this case, you need to increase the timeout and lower the bulkSize in the xml appender settings to reduce the data to be included in one packet.

5. Note for memory usage

SDK currently offers the bulk transmission mode by default, and the default bulkSize is 1000.

As one Effective Log Search & Analytics log contains several fields, such as projectName, it occupies about 1kb of a memory even though a log body is very short.

Therefore, by default, Nelo2 bulk occupies (log size+1kb)*1000 of a memory.

When you run a Java process, you can specify the maximum heap with the -Xmx option. Please consider the additional memory usage mentioned above.

Be especially careful if you specify a larger bulkSize option.

Was this article helpful?

What's Next

Network Traffic Monitoring Overview

Table of contents

Add Effective Log Search & Analytics Log4j2 SDK dependency
Effective Log Search & Analytics log4j2 appender settings and options