How to use Log4j SDK v2

Prev Next

Available in Classic and VPC

This chapter explains how to use the Effective Log Search & Analytics NELO Log4j SDK. NELO is the project code name for Effective Log Search & Analytics.

Add the Effective Log Search & Analytics Log4j SDK dependency

Add the dependency as follows.
After extracting the compressed file, add the paths of the core module and the log4j module to <systemPath>.

<dependencies>
        <dependency>
            <groupId>nelo2-java-sdk-core</groupId>
            <artifactId>nelo2-java-sdk-core</artifactId>
            <version>1.6.6</version>
            <scope>system</scope>
            <systemPath>/nelo2-java-sdk-core-1.6.6.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>nelo2-java-sdk-log4j</groupId>
            <artifactId>nelo2-java-sdk-log4j</artifactId>
            <version>1.6.6</version>
            <scope>system</scope>
            <systemPath>/nelo2-java-sdk-log4j-1.6.6.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libthrift</artifactId>
            <version>0.9.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.2.6</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.3.1</version>
        </dependency>
</dependencies>

Note

  • This SDK includes slf4j-log4j12, which is the SLF4J binding library for Log4j.
    Since SLF4J supports only one binding at a time, you must not use it together with other SLF4J binding libraries.

  • If the existing reference library overlaps with those referenced by the NELO2 Log4j SDK, problems may occur.
    In this case, it is recommended to use the later version.

Effective Log Search & Analytics Log4j appender settings and options

Enter the common configuration in log4j.xml as follows:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration PUBLIC "" "log4j.dtd">
<log4j:configuration xmlns:log4j='http://jakarta.apache.org/log4j/'>
    <!-- // define appenders // -->
    <appender name="STDOUT" class="org.apache.log4j.ConsoleAppender">
        <param name="Threshold" value="DEBUG"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%m%n"/>
        </layout>
    </appender>
    <!-- // define loggers // -->
    <logger name="com" additivity="false">
        <level value="INFO"/>
        <appender-ref ref="STDOUT"/>
    </logger>
    <!-- // define root // -->
    <root>
        <level value="WARN"/>
        <appender-ref ref="STDOUT"/>
    </root>
</log4j:configuration>

Enter the Effective Log Search & Analytics NELO2 Log4j appender as follows. For projectName, enter the project key registered in Effective Log Search & Analytics:

<!-- define nelo appender -->
<appender name="nelo" class="com.naver.nelo2.log4j.ThriftAppender">
    <param name="Threshold" value="ERROR"/>
    <param name="projectName" value="%YOUR_PROJECT_KEY%"/>
    <param name="collectorUrl" value="elsa-col.ncloud.com"/>
    <param name="port" value="10006"/>
    <param name="timeout" value="1200"/>
    <param name="enable" value="true"/>
    <param name="errorCodeType" value="default"/>
</appender>

You can set the following options:

  • Appender: Select the appender class name based on the transfer protocol.

    • Thrift Appender : com.naver.nelo2.log4j.ThriftAppender

    • Http Appender : com.naver.nelo2.log4j.HttpAppender

  • projectName: Project key. You can check the project key in the project information of Effective Log Search & Analytics.

  • version: Project version. (Only English letters, numbers, -, _, . are allowed, and it should start with an English letter, number, or _.)

  • collectorUrl: URL of the Effective Log Search & Analytics log collector server.

    • ThriftAppender: elsa-col.ncloud.com

    • HttpAppender: http://elsa-col.ncloud.com/_store

  • port: Collector server port.

    • Thrift Appender : 10006

    • Http Appender : 80

  • enable: Use status. The default value is true.

  • logType: Set logType.

  • logSource: Set logSource.

  • errorCodeType: Error code type.

    • Default: Uses exception information from Log4j basic information. If exception information is not delivered (for example, when the log is recorded in the log.error(message) format), the entire error message is used as the error code.

      Example: When a NullPointerException occurs → NullPointerException

    • message: Uses only the part from the beginning of the error message up to the first whitespace.

      Example: Download has failed. → Download error

    • action: Path information of the URL. (Lucy interceptor must be applied.)

      Example: http://xxx.com/board/list?id=100 → /board/list

    • mdc: Set the value of the "errorCode" item in SLF4J MDC to use.

      Example: MDC.put("errorCode", "Login") → Login

  • debug: Displays debugging information of Effective Log Search & Analytics. The default value is false.

    • This option applies globally, and "true" takes precedence over "false." If multiple appenders are declared and only one of them has the debug value set to "true," debug logs are output to the console from all appenders.

  • timeout: Socket timeout used for transmission. The default value is 5000 ms (5 seconds).

  • keepAlive: Socket keepAlive timeout used for transmission. The default value is 60,000 ms (1 minute), and the maximum value is 180,000 ms (3 minutes).

  • isBulkEnabled: Specifies whether to use bulk mode. The default value is true. If set to false, each log is sent individually.

  • bulkSize: Maximum number of logs to be sent in a single bulk request when using bulk mode. The default value is 1000, and the maximum value is 100,000.

  • bulkInterval: Cycle to call bulk requests when using bulk mode. The default value is 1000 ms (1 second), and the maximum value is 10,000 ms (10 seconds).

  • alwaysIncludeLocation: Specifies whether the SDK adds the Location field to all logs. The default value is true.

    • false: Check and set the Location field only for logs with logLevel set to "ERROR."
    • true: Check and set the Location field for all logs. This may negatively affect logging performance compared to false.

  • mdcConversionRule: A rule for renaming MDC keys.

    • format: key1:newKey1;key2:newkey2;...

      Example: If you set mdcConversionRule to time:date;fullname:name and the keys exist, the MDC key time is renamed to "date" and "fullname" is renamed to "name."

bulk / single

The NELO2 Log4j SDK supports both single mode, which sends logs individually, and bulk mode, which sends logs in batches.

Bulk mode is supported starting from version 1.4.0. Set isBulkEnabled to true or false in the XML appender configuration to use bulk or single mode (default: true).

For performance reference, see the table below according to the protocol.

Throughput when sending 1 KB logs from a single thread for 1 minute

  • thrift

    • single mode : 2615.54 logs/sec
    • bulk mode : 6642.97 logs/sec

  • http

    • single mode : 592.97 logs/sec
    • mode : 4665.26 logs/sec

The specifications for the equipment used in the performance test above are as follows:

  • Log transfer server: 2 GHZ 12 core CPU, 48 G memory, and located in Gasan IDC.
  • Log collecting server: 2.26 GHZ 12 core CPU, 48 G memory, and located in Gasan IDC.

Note

  • The transfer performance may differ depending on the load.
    The test was run with no load, and it shows relatively low performance when transferring with the actual instance in use.
    Different performance due to the instance load can be perceived more clearly in the single mode than the bulk mode.
    Therefore, it is recommended to use the default bulk mode.

  • The maximum size of the packet that the collecting server allows is 30 MB.
    Configure an appropriate bulkSize based on the client server's log pattern. (Default: 1000)

AsyncAppender

You can get the same type of results by using the AsyncAppender provided by Log4j, as shown below.

Pay attention to the following configuration values. bufferSize has a default value of 128, but this may not be sufficient for some applications.

locationInfo has a default value of false. In this case, the AsyncAppender ignores the location information of the log.

See the AsyncAppender configuration example below:

<appender name="nelo-async" class="org.apache.log4j.AsyncAppender">
    <param name="Threshold" value="ERROR"/>
    <param name="blocking" value="false"/>
    <param name="locationInfo" value="true"/>
    <param name="bufferSize" value="2048"/>
    <appender-ref ref="nelo"/>
</appender>

For details on configuration values, see the Official documentation.

How to use AsyncAppender

In addition to the default ThriftAppender, you can use the Nelo2AsyncAppender to perform log transmission in a separate thread.

To do this, set the XML file as follows:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration PUBLIC "" "log4j.dtd">
<log4j:configuration xmlns:log4j='http://jakarta.apache.org/log4j/'>
    <!-- // define appenders // -->
    <appender name="STDOUT" class="org.apache.log4j.ConsoleAppender">
        <param name="Threshold" value="DEBUG"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%m%n"/>
        </layout>
    </appender>
    <!-- define nelo appender -->
    <appender name="nelo" class="com.naver.nelo2.log4j.ThriftAppender">
        <param name="Threshold" value="ERROR"/>
        <param name="projectName" value="프로젝트 키"/>
        <param name="collectorUrl" value="elsa-col.ncloud.com"/>
        <param name="port" value="10006"/>
        <param name="timeout" value="1200"/>
        <param name="enable" value="true"/>
        <param name="errorCodeType" value="default"/>
    </appender>
    <!-- define nelo-async appender -->
    <appender name="nelo-async"
              class="org.apache.log4j.AsyncAppender">
        <param name="Threshold" value="ERROR"/>
        <param name="blocking" value="false"/>
        <param name="locationInfo" value="true"/>
        <param name="bufferSize" value="2048"/>
        <appender-ref ref="nelo"/>
    </appender>
    <!-- // define loggers // -->
    <logger name="com" additivity="false">
        <level value="ERROR"/>
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="nelo-async"/>
    </logger>
    <!-- // define root // -->
    <root>
        <level value="WARN"/>
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="nelo-async"/>
    </root>
</log4j:configuration>

Effective Log Search & Analytics Log4j SDK usage example

The following is a sample code that demonstrates the actual use of the Effective Log Search & Analytics Log4j SDK:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

private static Logger logger = LoggerFactory.getLogger(log4jTest.class);
...
    logger.debug("Effective Log Search & Analytics log4j SDK Debug Message");
    try {
        String npe = null;
        npe.toString();
    } catch(Exception e) {
        logger.error("Effective Log Search & Analytics log4j SDK Exception", e);
    }

Service limits

  • The contents of the above guide are not officially supported in Log4j version 2.0.
  • When using the AsyncAppender, if the log generation speed exceeds the transfer speed, logs that exceed the queue size are not transferred.
  • It is recommended to select between the AsyncAppender and the default appender according to the following criteria:
    • If you want to minimize log loss: Use default appender.
    • If you are concerned about application performance degradation in case of an Effective Log Search & Analytics system failure: Use AsyncAppender.

Troubleshooting

1. When logs are transferred to the Effective Log Search & Analytics but you cannot view them from the web

Check if the projectName is correct when there is no error in the result message after transferring the logs to Effective Log Search & Analytics' collecting server. As the projectName is the project key, you can check them in project management.

Then check whether actual error data is being sent. Check if the following transfer log is exported after setting the Effective Log Search & Analytics NELO appender's debug attribute to true in the configuration file (log4j.xml).

<!-- define nelo appender -->
<appender name="nelo" class="com.naver.nelo2.log4j.ThriftAppender">
    <param name="Threshold" value="ERROR"/>
    <param name="projectName" value="%YOUR_PROJECT_KEY%"/>
    <param name="collectorUrl" value="elsa-col.ncloud.com"/>
    <param name="port" value="10006"/>
    <param name="timeout" value="1200"/>
    <param name="enable" value="true"/>
    <param name="errorCodeType" value="default"/>
    <param name="debug" value="true"/>
</appender>

[NELO2] Log Append : sent event, return value :
…

FAQ

1. How to use AsyncAppender in a batch program (or a simple test program)

You need to add code at the end of the batch program to wait for a few seconds.

try {
    Thread.sleep(3000L);
} catch (InterruptedException ignore){}

AsyncAppender transfers logs asynchronously through a separate daemon thread that records logs internally.

In a Java batch program, the main thread terminates immediately, causing the batch application to exit before the Log4j AsyncAppender’s daemon thread can be created and send logs.

The JVM terminates immediately if there are no non-daemon threads running, regardless of daemon threads.

Therefore, add a wait statement at the end of the program as described above, so that all logs are sent before the application exits.

2. How to include a Java stack trace in Log4j

To print a stack trace in Action, BO, DAO, or Java batch programs using Log4j, use the form log.error(e.getMessage(), e);.

The SLF4J Logger does not support logging methods that take only a Throwable as an argument.

String[] aa = null;
try {
    aa[0] = "111";
} catch (NullPointerException e) {
//	log.error(e); //Method not supported in SLF4J.
    log.error(e.getMessage(), e); ///Prints the stack trace
}

3. How to minimize performance degradation caused by Log4j (including Effective Log Search & Analytics) logging

Maximize filtering by using name and level in the logger settings of log4j.xml.

For example, if you set com or org to the DEBUG level in the logger settings, a large number of ILoggingEvent (Log4j) instances are unnecessarily generated.

Even if the threshold in the NELO Log4j appender is set to ERROR and the logs are not actually transmitted, the ILoggingEvent instances are still generated by the logger and delivered to the appender.

Settings that degrade performance (for development use only)

<!-- Logger -->
<logger name="com" additivity="false">
    <level value="debug"/>
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-log4j" />
</logger>

<!-- Logger -->
<logger name="org" additivity="false">
    <level value="debug"/>
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-log4j" />
</logger>

<!-- Root Logger -->
<root level="warn">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-log4j" />
</root>

Settings that consider performance (for production use)

<!-- Logger -->
<logger name="com" additivity="false">
    <level value="error"/>
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-log4j" />
</logger>

<!-- Root Logger -->
<root level="warn">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-log4j" />
</root>

4. How to ensure a graceful shutdown when using a WAS

If the WAS (e.g., Tomcat) is shut down while error logs are being transferred, the WAS may occasionally fail to terminate properly.

To prevent this, call the stop() method on the LoggerContext instance when shutting down WAS, and close the NELO2 Java appender. This ensures a graceful shutdown.

Note

Spring provides org.springframework.web.util.Log4jConfigListener for Log4j, but it does not provide a listener for Logback to support graceful shutdown.

If you need this behavior, implement it by adding the following code in the application:

public class YOUR_CLASS_NAME implements ServletContextListener {
    public void contextDestroyed(ServletContextEvent arg0) {
        // shutdown log4j, at destroy time
        LogManager.shutdown();
    }
    public void contextInitialized(ServletContextEvent arg0) {
        // do not any operation, at initial time
    }
}

5. What to do if a timeout occurs during Thrift bulk transfer

If logs are not successfully sent to the collector server, you may see logs similar to the following:

[NELO2] sendMessage (1426319665440) sendBulk failed..  Error occur : java.net.SocketTimeoutException: Read timed out

In this case, increase the timeout in the XML appender settings and decrease the bulkSize to reduce the amount of data included in a single packet.

6. Cautions on memory usage

The provided SDK uses bulk transfer mode by default, and the default value of bulkSize is 1000.

As each Effective Log Search & Analytics log includes multiple fields such as projectName, it occupies about 1 KB of memory even when the log body is very short.

Therefore, with the default settings, a NELO2 bulk occupies approximately (log size + 1 KB) × 1000 of memory.

When starting a Java process, you can specify the maximum heap size using the -Xmx option. At this point, consider additional memory usage described above.

Be more cautious when you set the bulkSize option even larger.

Business Registration Number: 129-86-31394 E-commerce Permit Number:No. 2009-GyeonggiSeongnam-0510
CEO: Kim Yuwon Address: NAVER Green Factory 17th–26th Floors, Buljeong-ro 6, Bundang-gu, Seongnam-si, Gyeonggi-do 13561, Republic of Korea. Customer Support Number: 1544-5876
© NAVER Cloud Corp. All Rights Reserved.