Using OpenTSDB

The latest service changes have not yet been reflected in this content. We will update the content as soon as possible. Please refer to the Korean version for information on the latest updates.

Available in VPC

OpenTSDB (Time Series DataBase) is a distributable and scalable time series database powered by HBase.
OpenTSDB was created to store, index, and deliver metrics information collected from servers, networks, and applications.

OpenTSDB architecture

• Time Series Daemon (TSD): stores the entered metric information in the HBase storage.
• Input sources: metric information sent from servers, networks, applications, and other sources.
• Storage: store time series data using HBase as the default storage.

This guide explains how to build OpenTSDB with Cloud Hadoop.

For information about the OpenTSDB open source used in this guide, see OpenTSDB

Preparations

Create a server to install OpenTSDB. The example describes how to use it on a Linux server.

• For more information about creating a server, see the Getting Started with Server guide.
• To access a Linux server, see the Connecting to the Server guide.

Using OpenTSDB

First, check the requirements for installing the OpenTSDB open source on the server. The installation requirements are as follows:

<Installation requirements>

• Java SE (JDK)
• Gnuplot
• OpenTSDB (open source)

1. Install OpenTSDB

To install OpenTSDB on a server, follow these steps:

1. Use the following command to install OpenJDK on the server:

   yum install java-1.8.0-openjdk-devel.x86_64
   

2. Use the yum package manager to install the gnuplot package required to create graphs for OpenTSDB.

   yum -y install gnuplot
   

3. Create a user account.

   useradd -m  -d /home/opentsdb -u 10001 opentsdb
   

4. Create a log directory.

   mkdir /var/log/opentsdb && chown opentsdb:opentsdb /var/log/opentsdb
   

5. Install the OpenTSDB open source.

   # root user
   su - opentsdb
   
   # opentsdb user
   git clone https://github.com/OpenTSDB/opentsdb
   cd opentsdb
   

6. Build the OpenTSDB source.

   # opentsdb user
   ./build.sh
   

2. Install the HBase Client

To install the HBase client, follow these steps:

1. Use the following command to download the HBase binary:

   • Cloud Hadoop 2.0 or later versions

   # opentsdb user
   wget https://archive.apache.org/dist/hbase/2.2.6/hbase-2.2.6-bin.tar.gz
   tar xvfz hbase-2.2.6-bin.tar.gz
   ln -s hbase-2.2.6 hbase
   

   • Cloud Hadoop 1.9 or earlier versions

   # opentsdb user
   wget https://archive.apache.org/dist/hbase/2.0.2/hbase-2.0.2-bin.tar.gz
   tar xvfz hbase-2.0.2-bin.tar.gz
   ln -s hbase-2.0.2 hbase
   

   Note

   For Cloud Hadoop Classic, download HBase 1.1.2

2. In Ambari UI, click [ACTIONS] > Download Client Configs to transfer the downloaded "HBase client configuration information" to the server where you installed OpenTSDB.

   • For more information about accessing and using the Ambari UI, see the Ambari UI guide.
     

3. Unzip the HBase client configuration information.

   # opentsdb user
   tar xvfz HBASE_CLIENT-configs.tar.gz -C ~/opentsdb/hbase/conf
   sed -i -e 's|\(^export JAVA_HOME=.*\)|#\1|' ~/opentsdb/hbase/conf/hbase-env.sh
   

4. Edit the opentsdb user's .bash_profile.

   # opentsdb user
   echo 'export HBASE_HOME=~/opentsdb/hbase'                 >> ~/.bash_profile
   echo 'export PATH=$PATH:${HBASE_HOME}/bin'                >> ~/.bash_profile
   echo 'export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk'   >> ~/.bash_profile
   logout
   

5. Check the Cloud Hadoop cluster's hostname and private IP information in the Ambari UI > Hosts menu.
   

6. Use the following commands to add /etc/hosts information:

   • Add the hostname and private IP information of the Cloud Hadoop cluster you identified in the previous step.

   # root user
   # echo 'IP            hostname'     >> /etc/hosts
   echo  '1**.**.*.*  e-001-*****-**'  >> /etc/hosts
   echo  '1**.**.*.*  m-001-*****-**'  >> /etc/hosts
   echo  '1**.**.*.*  m-002-*****-**'  >> /etc/hosts
   echo  '1**.**.*.*  d-001-*****-**' >> /etc/hosts
   echo  '1**.**.*.*  d-002-*****-**' >> /etc/hosts
   

7. Use the following command to test the hbase shell access:

   su - opentsdb
   # opentsdb user
   hbase shell
   
   HBase Shell; enter 'help<RETURN>' for list of supported commands.
   Type "exit<RETURN>" to leave the HBase Shell
   Version 1.1.2, rcc2b70cf03e3378800661ec5cab11eb43fafe0fc, Wed Aug 26 20:11:27 PDT 2015
   hbase(main):001:0>
   

If you fail to access HBase Shell, add an OpenTSDB server to the ACG of Cloud Hadoop. For more information about setting up ACG, see the Firewall Settings (ACG) guide.

3. Create an OpenTSDB table

To create an OpenTSDB table, follow these steps:

1. Create a TSDB table using the following command:

   # opentsdb user
   export COMPRESSION=SNAPPY
   export DATA_BLOCK_ENCODING=FAST_DIFF
   export TSDB_TTL=2147483647
   sh ~/opentsdb/src/create_table.sh
   

2. Use the following command to view the list of tables:

   # opentsdb user
   hbase shell
   hbase(main):001:0> list
   TABLE
   tsdb
   tsdb-meta
   tsdb-tree
   tsdb-uid
   4 row(s) in 0.1620 seconds
   
   => ["tsdb", "tsdb-meta", "tsdb-tree", "tsdb-uid"]
   hbase(main):002:0>
   

4. Activate OpenTSDB

Use the following command to start the TSD on port 8888:

# opentsdb user
 cd opentsdb
 ./build/tsdb tsd --port=8888 --staticroot=build/staticroot --
 cachedir=`pwd`/build/tmp --zkquorum=m-001-*****-**,m-002-*****-**,d-001-*****-**:
 2181 --zkbasedir=/hbase-unsecure &

5. Store Time Series Data in OpenTSDB

To store time series data in OpenTSDB, follow these steps:

1. Use the following command to create a cpu.util metric:

   # opentsdb user
   ./build/tsdb mkmetric cpu.util --zkquorum=m-001-*****-**,m-002-*****-**,d-001-*****-**:
   2181 --zkbasedir=/hbase-unsecure metrics cpu.util: [0, 0, 1]
   

2. Use the following command to create a cpu.util metrics collection script: (sar-data-input.sh)

   • This script is referenced from the original article Effective Monitoring and Alerting.

   #!/bin/bash 
   # Ignore sar's header.  
   LC_ALL=en sar -u 1 | sed -u -e '1,3d' | 
   while read time cpu usr nice sys io steal idle; 
   do
       NOW=$(date +%s)
       echo put cpu.util $NOW $usr time=user
       echo put cpu.util $NOW $sys time=system
       echo put cpu.util $NOW $io time=io
       echo put cpu.util $NOW $idle time=idle
       # Report values to standard error.
       echo timestamp:$NOW user:$usr sys:$sys io:$io idle:$idle >&2 
   done | nc -w 30 localhost 8888
   

3. Use the following command to transfer metrics data to OpenTSDB:

   sh sar-data-input.sh
   

6. Check the OpenTSDB Graph

To view the OpenTSDB graphs, follow these steps:

1. Check the public IP of the OpenTSDB server in the server console.
   
2. Access port 8888 on the OpenTSDB server and check the collected graphs.