Using OpenTSDB

Available in Classic

OpenTSDB (Time Series Database), which operates based on HBase, is a distributed and scalable time series database.
OpenTSDB is created for processing tasks such as saving, indexing, and delivering metric information collected from servers, networks, and applications.

OpenTSDB architecture

• TSD (Time Series Daemon): It handles the role of saving entered metric information to the HBase storage.
• Input source: It is the metric information delivered from servers, networks, and applications.
• Storage: It uses HBase as the default storage to save the time series data.

This page explains the method for building OpenTSDB using Cloud Hadoop.

Refer to OpenTSDB for information about OpenTSDB open sources used in this page.

Preparations

Create a server to install OpenTSDB. This example describes the method to use for a Linux server.

• For more information about creating servers, please refer to Getting started with Server.
• Refer to Linux server connection guide for how to access a Linux server.

Use OpenTSDB

First of all, check the following requirements for installing OpenTSDB open source in a server. <Installation requirements> is:

Installation requirements

• Java SE (JDK)
• Gnuplot
• OpenTSDB (open source)

1. Install OpenTSDB

The following describes how to install OpenTSDB on a server.

1. Download JDK from the Oracle website using the following command, and install it on the server.

   rpm -ivh jdk-8u192-linux-x64.rpm
   

   Note

   • i: Install package
   • v: Print details
   • h: Print the installation progress status

2. Use the yum package manager to install the gnuplot package required for generating OpenTSDB graphs.

   yum -y install gnuplot
   

3. Create a user account.

   useradd -m  -d /home/opentsdb -u 10001 opentsdb
   

4. Create a log directory.

   mkdir /var/log/opentsdb && chown opentsdb:opentsdb /var/log/opentsdb
   

5. Install the OpenTSDB open source.

   # root user
   su - opentsdb
   
   # opentsdb user
   git clone https://github.com/OpenTSDB/opentsdb
   cd opentsdb
   

6. Build the OpenTSDB source.

   # opentsdb user
   ./build.sh
   

2. Install HBase client

The following describes how to install the HBase client.

1. Use the following command to download HBase Binary.

   # opentsdb user
   wget https://archive.apache.org/dist/hbase/1.1.2/hbase-1.1.2-bin.tar.gz
   tar xvfz hbase-1.1.2-bin.tar.gz
   ln -s hbase-1.1.2 hbase
   

2. Download the HBase client settings information from the Ambari UI page.

   • Please refer to Ambari UI for more information about accessing and using the Ambari UI.
     

3. Decompress the HBase client settings information.

   # opentsdb user
   tar xvfz HBASE_CLIENT-configs.tar.gz -C ~/opentsdb/hbase/conf
   sed -i -e 's|\(^export JAVA_HOME=.*\)|#\1|' ~/opentsdb/hbase/conf/hbase-env.sh
   

4. Edit the opentsdb user's .bash_profile.

   # opentsdb user
   echo 'export HBASE_HOME=~/hbase'                   >> ~/.bash_profile
   echo 'export PATH=$PATH:${HBASE_HOME}/bin'         >> ~/.bash_profile
   echo 'export JAVA_HOME=/usr/java/jdk1.8.0_192-amd64' >> ~/.bash_profile
   logout
   

5. Check the Cloud Hadoop cluster's host name and IP information from the Ambari UI's Hosts page.
   

6. Add the /etc/hosts information by using the following command.

   • Add the host name and IP information of the Cloud Hadoop cluster identified in the previous step.

   # root user
   # echo 'IP               host name'      >> /etc/hosts
   echo  '1*.**.**.237  e-001-*****-**'  >> /etc/hosts
   echo  '1*.**.**.109  m-001-*****-**'  >> /etc/hosts
   echo  '1*.**.**.142  m-002-*****-**'  >> /etc/hosts
   echo  '1*.**.**.169  dn-001-*****-**' >> /etc/hosts
   echo  '1*.**.**.186  dn-002-*****-**' >> /etc/hosts
   echo  '1*.**.**.221  dn-002-*****-**' >> /etc/hosts
   

7. Test the access to hbase shell by using the following command.

   su - opentsdb
   # opentsdb user
   hbase shell
   
   HBase Shell; enter 'help<RETURN>' for list of supported commands.
   Type "exit<RETURN>" to leave the HBase Shell
   Version 1.1.2, rcc2b70cf03e3378800661ec5cab11eb43fafe0fc, Wed Aug 26 20:11:27 PDT 2015
   hbase(main):001:0>
   

If the HBase Shell access has failed, then add the OpenTSDB server to Cloud Hadoop's ACG.
Please refer to Set firewall (ACG) for more information about setting up ACG.

3. Create OpenTSDB table

The following describes how to create an OpenTSDB table.

1. Use the following command to create a TSDB table.

   # opentsdb user
   export COMPRESSION=SNAPPY
   export DATA_BLOCK_ENCODING=FAST_DIFF
   export TSDB_TTL=2147483647
   sh ~/opentsdb/src/create_table.sh
   

2. View the table list with the following command.

   # opentsdb user
   hbase shell
   hbase(main):001:0> list
   TABLE
   tsdb
   tsdb-meta
   tsdb-tree
   tsdb-uid
   4 row(s) in 0.1620 seconds
   
   => ["tsdb", "tsdb-meta", "tsdb-tree", "tsdb-uid"]
   hbase(main):002:0>
   

4. Start OpenTSDB

Use the following command to start the Time Series Daemon (TSD) with Port 8888.

# opentsdb 사용자
cd opentsdb
./build/tsdb tsd --port=8888 --staticroot=build/staticroot --
cachedir=`pwd`/build/tmp --zkquorum=m-001-*****-**,m-002-*****-**,dn-001-*****-**:
2181 --zkbasedir=/hbase-unsecure &

5. Save time series data to OpenTSDB

The following describes how to save time series data to OpenTSDB.

1. Use the following command to create the cpu.util metric.

   # opentsdb user
   .build/tsdb mkmetric cpu.util --zkquorum=m-001-*****-**,m-002-*****-**,dn-001-*****-**:
   2181 --zkbasedir=/hbase-unsecure metrics cpu.util: [0, 0, 1]
   

2. Use the following command to create a cpu.util metric collection script. (sar-data-input.sh)

   • Script source: Referred to the original copy of Effective Monitoring and Alerting

   #!/bin/bash 
   # Ignore sar's header.  
   LC_ALL=en sar -u 1 | sed -u -e '1,3d' | 
   while read time cpu usr nice sys io steal idle; 
   do
       NOW=$(date +%s)
       echo put cpu.util $NOW $usr time=user
       echo put cpu.util $NOW $sys time=system
       echo put cpu.util $NOW $io time=io
       echo put cpu.util $NOW $idle time=idle
       # Report values to standard error.
       echo timestamp:$NOW user:$usr sys:$sys io:$io idle:$idle >&2 
   done | nc -w 30 localhost 8888
   

3. Use the following command to send the metric data to OpenTSDB.

   sh sar-data-input.sh
   

6. View OpenTSDB graph

The following describes how to view the OpenTSDB graph.

1. Check the public IP of the OpenTSDB server from the server console.
   

2. Connect to Port 8888 of the OpenTSDB server to view the collected graph.