Using OpenTSDB

Available in VPC

OpenTSDB (Time Series Database), which operates on the basis of HBase, is a distributable and scalable time series database.
OpenTSDB is created for processing tasks such as saving, indexing, and delivering metric information collected from servers, networks, and applications.

OpenTSDB architecture

• TSD (Time Series Daemon): saves entered metric information to the HBase storage.
• Input source: metric information delivered from servers, networks, and applications.
• Storage: uses HBase as the default storage to save the time series data.

This guide explains how to build OpenTSDB using Cloud Hadoop.

For the information on the OpenTSDB open source used in this guide, see OpenTSDB.

Preparations

Create a server where you want to install OpenTSDB. This example describes the method to use for a Linux server.

• For more information about creating servers, see Getting started with Server guide.
• For information on accessing Linux servers, see Accessing server guide.

Use OpenTSDB

First, check the requirements for installing OpenTSDB open source on the server. <Installation requirements> are as follows:

Installation requirements

• Java SE (JDK)
• Gnuplot
• OpenTSDB (open source)

1. Install OpenTSDB

The following describes how to install OpenTSDB on a server.

1. Download JDK from the Oracle website using the following command, and install it on the server.

   rpm -ivh jdk-8u192-linux-x64.rpm
   

   Note

   • i: Install package
   • v: Print details
   • h: Print the installation progress status

2. Use the yum package manager to install the gnuplot package required to create OpenTSDB graphs.

   yum -y install gnuplot
   

3. Create a user account.

   useradd -m  -d /home/opentsdb -u 10001 opentsdb
   

4. Create a log directory.

   mkdir /var/log/opentsdb && chown opentsdb:opentsdb /var/log/opentsdb
   

5. Install the OpenTSDB open source.

   # root user
   su - opentsdb
   
   # opentsdb user
   git clone https://github.com/OpenTSDB/opentsdb
   cd opentsdb
   

6. Build the OpenTSDB source.

   # opentsdb user
   ./build.sh
   

2. Install HBase client

The following describes how to install the HBase client.

1. Use the following command to download HBase Binary.

   # opentsdb user
   wget https://archive.apache.org/dist/hbase/1.1.2/hbase-1.1.2-bin.tar.gz
   tar xvfz hbase-1.1.2-bin.tar.gz
   ln -s hbase-1.1.2 hbase
   

2. Click [ACTIONS] > Download Client Configs in the Ambari UI, and then download the HBase client settings information.

   • For more information about accessing and using Ambari UI, see Ambari UI guide.
     

3. Decompress the HBase client settings information.

   # opentsdb user
   tar xvfz HBASE_CLIENT-configs.tar.gz -C ~/opentsdb/hbase/conf
   sed -i -e 's|\(^export JAVA_HOME=.*\)|#\1|' ~/opentsdb/hbase/conf/hbase-env.sh
   

4. Edit the opentsdb user's .bash_profile.

   # opentsdb user
   echo 'export HBASE_HOME=~/hbase'                   >> ~/.bash_profile
   echo 'export PATH=$PATH:${HBASE_HOME}/bin'         >> ~/.bash_profile
   echo 'export JAVA_HOME=/usr/java/jdk1.8.0_192-amd64' >> ~/.bash_profile
   logout
   

5. Check the host name and private IP information of the Cloud Hadoop cluster in the Ambari UI > Hosts menus.
   

6. Add the /etc/hosts information by using the following command.

   • Add the host name and Private IP information of the Cloud Hadoop cluster checked in the previous step.

   # root user
   # echo 'IP            host name'     >> /etc/hosts
   echo  '1**.**.*.*  e-001-*****-**'  >> /etc/hosts
   echo  '1**.**.*.*  m-001-*****-**'  >> /etc/hosts
   echo  '1**.**.*.*  m-002-*****-**'  >> /etc/hosts
   echo  '1**.**.*.*  d-001-*****-**' >> /etc/hosts
   echo  '1**.**.*.*  d-002-*****-**' >> /etc/hosts
   

7. Test the access to hbase shell by using the following command.

   su - opentsdb
   # opentsdb user
   hbase shell
   
   HBase Shell; enter 'help<RETURN>' for list of supported commands.
   Type "exit<RETURN>" to leave the HBase Shell
   Version 1.1.2, rcc2b70cf03e3378800661ec5cab11eb43fafe0fc, Wed Aug 26 20:11:27 PDT 2015
   hbase(main):001:0>
   

If the connection to HBase Shell fails, add the OpenTSDB server to ACG of Cloud Hadoop. For more information on ACG settings, see Set firewall (ACG) guide.

3. Create OpenTSDB table

The following describes how to create an OpenTSDB table.

1. Use the following command to create a TSDB table.

   # opentsdb user
   export COMPRESSION=SNAPPY
   export DATA_BLOCK_ENCODING=FAST_DIFF
   export TSDB_TTL=2147483647
   sh ~/opentsdb/src/create_table.sh
   

2. View the table list with the following command.

   # opentsdb user
   hbase shell
   hbase(main):001:0> list
   TABLE
   tsdb
   tsdb-meta
   tsdb-tree
   tsdb-uid
   4 row(s) in 0.1620 seconds
   
   => ["tsdb", "tsdb-meta", "tsdb-tree", "tsdb-uid"]
   hbase(main):002:0>
   

4. Start OpenTSDB

Use the following command to start TSD with Port 8888.

# opentsdb user
 cd opentsdb
 ./build/tsdb tsd --port=8888 --staticroot=build/staticroot --
 cachedir=`pwd`/build/tmp --zkquorum=m-001-*****-**,m-002-*****-**,d-001-*****-**:
 2181 --zkbasedir=/hbase-unsecure &

5. Store time series data in OpenTSDB

The following describes how to save time series data to OpenTSDB.

1. Use the following command to create the cpu.util metric.

   # opentsdb user
   .build/tsdb mkmetric cpu.util --zkquorum=m-001-*****-**,m-002-*****-**,d-001-*****-**:
   2181 --zkbasedir=/hbase-unsecure metrics cpu.util: [0, 0, 1]
   

2. Use the following command to create a cpu.util metric collection script. (sar-data-input.sh)

   • Script source: see Effective Monitoring and Alerting

   #!/bin/bash 
   # Ignore sar's header.  
   LC_ALL=en sar -u 1 | sed -u -e '1,3d' | 
   while read time cpu usr nice sys io steal idle; 
   do
       NOW=$(date +%s)
       echo put cpu.util $NOW $usr time=user
       echo put cpu.util $NOW $sys time=system
       echo put cpu.util $NOW $io time=io
       echo put cpu.util $NOW $idle time=idle
       # Report values to standard error.
       echo timestamp:$NOW user:$usr sys:$sys io:$io idle:$idle >&2 
   done | nc -w 30 localhost 8888
   

3. Use the following command to send the metric data to OpenTSDB.

   sh sar-data-input.sh
   

6. Check OpenTSDB graph

The following describes how to check a OpenTSDB graph.

1. Check the public IP of the OpenTSDB server from the server console.
   
2. Connect to Port 8888 of the OpenTSDB server to view the collected graph.