Login
      Contents x
      Using the Log4j SDK v2
        • Print
        • PDF
        Contents

        Using the Log4j SDK v2

          • Print
          • PDF
          Article Summary

          Available in Classic and VPC

          This chapter describes how to use Effective Log Search & Analytics NELO Log4j SDK. NELO is the project code name for Effective Log Search & Analytics.

          Add Effective Log Search & Analytics Log4j SDK dependency

          Add Dependency as below.
          After unzip the compressed file, add the path to the core module and log4j module to <systemPath>.

          <dependencies>
        <dependency>
            <groupId>nelo2-java-sdk-core</groupId>
            <artifactId>nelo2-java-sdk-core</artifactId>
            <version>1.6.6</version>
            <scope>system</scope>
            <systemPath>/nelo2-java-sdk-core-1.6.6.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>nelo2-java-sdk-log4j</groupId>
            <artifactId>nelo2-java-sdk-log4j</artifactId>
            <version>1.6.6</version>
            <scope>system</scope>
            <systemPath>/nelo2-java-sdk-log4j-1.6.6.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libthrift</artifactId>
            <version>0.9.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.2.6</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.3.1</version>
        </dependency>
</dependencies>

          Remarks

          • This SDK includes the slf4j-log4j12, which is the slf4J binding library for the log4j.
            Because the slf4j supports only one binding at a time, you cannot use any libraries for other slf4j bindings.

          • Problems may occur if an existing reference library are duplicated with the library referenced by the nelo2 log4j SDK.
            We recommend using a higher version in this case.

          Effective Log Search & Analytics log4j appender settings and options

          The common configuration in the log4j.xml is as follows.

          <?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration PUBLIC "" "log4j.dtd">
<log4j:configuration xmlns:log4j='http://jakarta.apache.org/log4j/'>
    <!-- // define appenders // -->
    <appender name="STDOUT" class="org.apache.log4j.ConsoleAppender">
        <param name="Threshold" value="DEBUG"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%m%n"/>
        </layout>
    </appender>
    <!-- // define loggers // -->
    <logger name="com" additivity="false">
        <level value="INFO"/>
        <appender-ref ref="STDOUT"/>
    </logger>
    <!-- // define root // -->
    <root>
        <level value="WARN"/>
        <appender-ref ref="STDOUT"/>
    </root>
</log4j:configuration>

          Enter Effective Log Search & Analytics NELO2 log4j appender as follows. The projectName must be the project ID registered in Effective Log Search & Analytics.

          <!-- define nelo appender -->
<appender name="nelo" class="com.naver.nelo2.log4j.ThriftAppender">
    <param name="Threshold" value="ERROR"/>
    <param name="projectName" value="%YOUR_PROJECT_ID%"/>
    <param name="collectorUrl" value="elsa-col.ncloud.com"/>
    <param name="port" value="10006"/>
    <param name="timeout" value="1200"/>
    <param name="enable" value="true"/>
    <param name="errorCodeType" value="default"/>
</appender>

          The following can be set as options.

          • Appender (Select appender's class name according to transmission protocols.)

            • Thrift Appender : com.naver.nelo2.log4j.ThriftAppender

            • Http Appender : com.naver.nelo2.log4j.HttpAppender

          • projectName: Project ID. You can view a project ID created in Effective Log Search & Analytics's project information.

          • version : Project version (Only English letters, numbers, "-", and "" are allowed. It must start with an alphabetic character/number/"".)

          • collectorUrl: Effective Log Search & Analytics log collector server URL

            • ThriftAppender: elsa-col.ncloud.com

            • HttpAppender: http://elsa-col.ncloud.com/_store

          • port: Collector server port

            • Thrift Appender : 10006

            • Http Appender : 80

          • enable: Whether to use (Default: true)

          • logType : logType settings

          • logSource : logSource settings

          • errorCodeType: Error code type

            • default: Use the exception information of log4j default information. If the exception information is not delivered (if a log is recorded in the log.error(message) format), the entire error message is used as an error code.

              e.g.) If NullPointerException occurs => NullPointerException

            • message: Use only from the beginning until the blank of an error message.

              e.g.) Download error Download failed. => Download error

            • action: URL Path information (Lucy intercepter must be enabled.)

              e.g.) http://xxx.com/board/list?id=100 => /board/list

            • mdc: Set the “errorCode” metric value of SLF4J MDC to use.

              e.g.)MDC.put(“errorCode”, “Login”) => Login

          • debug: Display the debugging information for Effective Log Search & Analytics. The default value is false.

            • This option applies globally, and "true" overrides "false". That is, if multiple appenders are declared and one of them has the debug value as true, every appender will output debug logs to the Console.

          • timeout: Timeout of socket used to send. Default: 5000ms (5s).

          • keepAlive: keepAlive Timeout of socket used to send. Default: 60000ms (1m), Maximum 180000ms (3m).

          • isBulkEnabled : Whether to use the bulk mode. Default: true. If false, send each log individually.

          • bulkSize : The maximum number of logs to be sent in a single bulk request when using the bulk mode. Default: 1000, Maximum 100000.

          • bulkInterval : interval to call bulk request, defaults to 1000ms(1 sec), maximum 10000ms(10 sec)

          • alwaysIncludeLocation : whether SDK add Location fields to all logs, defaults to true.

            • false : check and set Location fields for logs which logLevel is ERROR.
            • true : check and set Location field for all logs. This may bring bad effects to logging performance compare to false.

          • mdcConversionRule : Rules to rename MDC key

            • format: key1:newKey1;key2:newkey2;...

              e.g.) Set mdcConversionRule to time:date;fullname:name, this rule will rename MDC key time to date and fullname to name if they are exists.

          bulk / single

          NELO2 log4j SDK supports the single mode for sending a log per request and the bulk mode for sending it in a batch.

          The bulk mode is a feature supported since version 1.4.0, and you can set the isBulkEnabled to true/false in the xml appender settings to use the bulk/single mode. (Default: true).

          Refer to the table below for a performance reference depending on a protocol.

          Throughput when sending a log of 1kb size in a single thread for 1 minute

          • thrift

            • single mode : 2615.54 logs/sec
            • bulk mode : 6642.97 logs/sec

          • http

            • single mode : 592.97 logs/sec
            • mode : 4665.26 logs/sec

          The specifications of the equipment used for the above performance test are as follows.

          • Log transmission server: 2GHZ 12core cpu, 48G mem, Located at Gasan IDC
          • Log collector server: 2.26GHZ 12core cpu, 48G mem, Located at Gasan IDC

          Remarks

          • Transmission performance may vary depending on a load.
            The tests were run under a no-load situation, and it shows relatively low performance when being sent to the real instances in use.
            The perceived performance according to the instance load is greater in the single mode than in the bulk mode.
            Therefore, we recommend using the default bulk mode.

          • The collector server allows a packet size up to 30mb.
            You must set the appropriate bulkSize considering a client server log pattern. (Default: 1000)

          AsyncAppender

          You can get the same result by using AsyncAppender, which is supported by the log4j as below.

          The following are the settings you should be aware of. The default bufferSize is 128, but it may not be sufficient for some applications.

          The default locationInfo is false, in which case AsyncAppender will ignore the log occurrence location information.

          Please refer to AsyncAppender setting example below.

          <appender name="nelo-async" class="org.apache.log4j.AsyncAppender">
    <param name="Threshold" value="ERROR"/>
    <param name="blocking" value="false"/>
    <param name="locationInfo" value="true"/>
    <param name="bufferSize" value="2048"/>
    <appender-ref ref="nelo"/>
</appender>

          For more information on the settings, refer to Official Document.

          How to use AsyncAppender

          In addition to ThriftAppender you use by default, you can use Nelo2AsyncAppender to set the actual log transmission to be performed on a separate thread.

          To do this, set the xml file as shown below.

          <?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration PUBLIC "" "log4j.dtd">
<log4j:configuration xmlns:log4j='http://jakarta.apache.org/log4j/'>
    <!-- // define appenders // -->
    <appender name="STDOUT" class="org.apache.log4j.ConsoleAppender">
        <param name="Threshold" value="DEBUG"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%m%n"/>
        </layout>
    </appender>
    <!-- define nelo appender -->
    <appender name="nelo" class="com.naver.nelo2.log4j.ThriftAppender">
        <param name="Threshold" value="ERROR"/>
        <param name="projectName" value="Project ID"/>
        <param name="collectorUrl" value="elsa-col.ncloud.com"/>
        <param name="port" value="10006"/>
        <param name="timeout" value="1200"/>
        <param name="enable" value="true"/>
        <param name="errorCodeType" value="default"/>
    </appender>
    <!-- define nelo-async appender -->
    <appender name="nelo-async"
              class="org.apache.log4j.AsyncAppender">
        <param name="Threshold" value="ERROR"/>
        <param name="blocking" value="false"/>
        <param name="locationInfo" value="true"/>
        <param name="bufferSize" value="2048"/>
        <appender-ref ref="nelo"/>
    </appender>
    <!-- // define loggers // -->
    <logger name="com" additivity="false">
        <level value="ERROR"/>
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="nelo-async"/>
    </logger>
    <!-- // define root // -->
    <root>
        <level value="WARN"/>
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="nelo-async"/>
    </root>
</log4j:configuration>

          Effective Log Search & Analytics log4j SDK usage example

          The following is an example of the codes that actually use Effective Log Search & Analytics log4j SDK.

          import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

private static Logger logger = LoggerFactory.getLogger(log4jTest.class);
...
    logger.debug("Effective Log Search & Analytics log4j SDK Debug Message");
    try {
        String npe = null;
        npe.toString();
    } catch(Exception e) {
        logger.error("Effective Log Search & Analytics log4j SDK Exception", e);
    }

          Limitations

          • The log4j 2.0 version does not officially support contents of the above guide.
          • When using the async appender, occurring logs that exceed the queue size will not be sent if the log occurrence rate is faster than the transmission rate.
          • We recommend that you select between the async appender and the default appender according to the criteria below.
            • To minimize log loss: Default appender
            • If concerned about application performance degradation in the event of Effective Log Search & Analytics system failure: async appender

          Troubleshooting

          1. When a log is sent to Effective Log Search & Analytics, but cannot be viewed on the web

          If there are no errors in the result message after sending a log to Effective Log Search & Analytics collector server, check if the "projectName" is correct. The “projectName” is the project ID, which can be viewed in the project management.

          Next, confirm if the actual error data is sent. In the configuration file(log4j2.xml), set the debug property of Effective Log Search & Analytics NELO appender to true and execute it to check if the following transmission logs are output.

          <!-- define nelo appender -->
<appender name="nelo" class="com.naver.nelo2.log4j.ThriftAppender">
    <param name="Threshold" value="ERROR"/>
    <param name="projectName" value="%YOUR_PROJECT_ID%"/>
    <param name="collectorUrl" value="elsa-col.ncloud.com"/>
    <param name="port" value="10007"/>
    <param name="timeout" value="1200"/>
    <param name="enable" value="true"/>
    <param name="errorCodeType" value="default"/>
    <param name="debug" value="true"/>
</appender>

[NELO2] Log Append : sent event, return value :
…

          FAQ

          1. How do I use AsyncAppender in a batch program (or a simple test program)?

          You need to add some codes to wait for a few seconds at the end of a batch program.

          try {
    Thread.sleep(3000L);
} catch (InterruptedException ignore){}

          AsyncAppender has a separate daemon thread to record logs internally, thus it delivers logs asynchronously.

          In Java batch program, the main thread is terminated immediately, so the daemon thread of the log4jAsyncAppender is created and the batch application is shut down before sending logs.

          JVM is terminated immediately if there are no live general threads regardless of a daemon thread.

          Therefore, add the codes for waiting at the end of the program as above so that it can be terminated after sending all logs.

          2. How can I include Java stack trace in Log4j?

          In order to output a stack trace using the log4j in Action / BO / DAO / Java batch program, use the form of the log.error(e.getMessage(), e);.

          SLF4J Logger is a method factor and does not support logging methods that only accept Throwable.

          String[] aa = null;
try {
    aa[0] = "111";
} catch (NullPointerException e) {
//	log.error(e); // Methods not supported by SLF4J.
    log.error(e.getMessage(), e); ///stacktrace outputted.
}

          3. How can I minimize performance degradation caused by the log4j (including Effective Log Search & Analytics) logging?

          Use a name and level in the log4j.xml logger settings to maximize filtering.

          If you set com or org to DEBUG level in the logger settings as below, lots of ILoggingEvent (log4j) is unnecessarily generated in the logger.

          Threshold is set to ERROR in the nelo log4j appender, so logs are actually not sent, but ILoggingEvent is created in the logger and passed to the appender.

          Settings that degrade performance (for developing only)

          <!-- Logger -->
<logger name="com" additivity="false">
    <level value="debug"/>
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-log4j" />
</logger>

<!-- Logger -->
<logger name="org" additivity="false">
    <level value="debug"/>
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-log4j" />
</logger>

<!-- Root Logger -->
<root level="warn">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-log4j" />
</root>

          Settings that consider performance (for operating)

          <!-- Logger -->
<logger name="com" additivity="false">
    <level value="error"/>
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-log4j" />
</logger>

<!-- Root Logger -->
<root level="warn">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="nelo-log4j" />
</root>

          4. How can I terminate stably when using in WAS?

          If WAS (Tomcat, etc.) is terminated while error logs are being sent, sometimes WAS may not be shut down normally.

          To prevent this, you can stably terminate by calling the stop() method for LoggerContext instance to close the nelo2 java appender when finishing WAS.

          Remarks

          Spring provides org.springframework.web.util.Log4jConfigListener for Log4J, but it does not offer Listener to support stable termination for Logback.

          If you need to do this, you can implement by adding the following codes in your application.

          public class YOUR_CLASS_NAME implements ServletContextListener {
    public void contextDestroyed(ServletContextEvent arg0) {
        // shutdown log4j, at destroy time
        LogManager.shutdown();
    }
    public void contextInitialized(ServletContextEvent arg0) {
        // do not any operation, at initial time
    }
}

          5. If a timeout occurs during the thrift bulk transmission

          If a log was not sent normally to a collector server, you can check the following logs.

          [NELO2] sendMessage (1426319665440) sendBulk failed..  Error occur : java.net.SocketTimeoutException: Read timed out

          In this case, you need to increase the timeout and lower the bulkSize to reduce the data to be included in one packet from the xml appender settings.

          6. Note for memory usage

          SDK currently offers the bulk transmission mode by default, and the default bulkSize is 1000.

          As one Effective Log Search & Analytics log contains several fields, such as projectName, it occupies about 1kb of the memory even though a log body is very short.

          Therefore, by default, Nelo2 bulk occupies (log size+1kb)*1000 of the memory.

          When you run Java process, you can specify the maximum heap with the -Xmx option. Please consider the additional memory usage mentioned above.

          Be especially careful if you specify a larger bulkSize option.

          Was this article helpful?
          What's Next
          Change password!
          Changing your password will log you out immediately. Use the new password to log back in.
          Change profile
          Success!
          First name must have atleast 2 characters. Numbers and special characters are not allowed.
          Last name must have atleast 1 characters. Numbers and special characters are not allowed.
          Enter a valid email
          Enter a valid password
          Your profile has been successfully updated.
          Logout