Documentation Index

Fetch the complete documentation index at: https://guide.ncloud-docs.com/llms.txt

Use this file to discover all available pages before exploring further.

Using OpenTSDB (Secure Hadoop)

Prev Next

Available in VPC

OpenTSDB (Time Series DataBase) is a distributable and scalable time series database powered by HBase.
OpenTSDB was created to store, index, and deliver metrics information collected from servers, networks, and applications.

Unlike in a standard Cloud Hadoop environment, using OpenTSDB in Secure Hadoop requires Kerberos authentication, an LDAP account, HBase permission settings, and JAAS settings for the OpenTSDB JVM. Since OpenTSDB accesses HBase and ZooKeeper through the AsyncHBase client, even if HBase Shell access is possible, Kerberos authentication may fail when running TSD (Time Series Daemon) without separate JAAS settings.

OpenTSDB architecture

This guide explains how to set up OpenTSDB using Secure Hadoop and run TSD on port 8888.

For information about the OpenTSDB open source used in this guide, see OpenTSDB.

  • Time Series Daemon (TSD): Stores the entered metric information in the HBase storage.
  • Input sources: Metric information sent from servers, networks, applications, and other sources.
  • Storage: Uses HBase as the default storage to save the time series data.

hadoop-chadoop-use-ex1_1-1

Note

For more information about OpenTSDB, see OpenTSDB website.

Preliminary task

Create a server to install OpenTSDB. The example describes how to use it on a Linux server.

Caution

In Secure Hadoop, simply creating a Linux OS account using useradd does not grant HBase access permissions. The OpenTSDB run-as account must have the Kerberos principal, keytab, LDAP user, and HBase permissions prepared.

Using OpenTSDB

The requirements for installing the OpenTSDB open source on a server are as follows:

  • Java SE 8(JDK)
  • Gnuplot
  • OpenTSDB open source
  • Kerberos principal and keytab
  • LDAP user
  • HBase client settings
  • HBase table creation permissions

1. Create Secure Hadoop account

To use OpenTSDB in Secure Hadoop, you must create a Kerberos principal and an LDAP user. We recommend configuring the user IDs registered in Kerberos and LDAP to be the same.

  1. Access the Kerberos admin shell on the master node.

    kadmin.local

  2. Create a principal for OpenTSDB.

    addprinc -randkey opentsdb@USER.GUIDE

  3. Create a keytab file for OpenTSDB.

    ktadd -k /etc/security/keytabs/opentsdb.keytab opentsdb@USER.GUIDE

  4. Configure the keytab file permissions.

    chown opentsdb:hadoop /etc/security/keytabs/opentsdb.keytab
chmod 400 /etc/security/keytabs/opentsdb.keytab
    Note

    If you are running OpenTSDB on a server outside the Secure Hadoop cluster, copy the created keytab file to a secure path on the OpenTSDB server.

  5. Check the principal included in the keytab.

    klist -kte /etc/security/keytabs/opentsdb.keytab

    Output example:

    Keytab name: FILE:/etc/security/keytabs/opentsdb.keytab
KVNO Timestamp           Principal
---- ------------------- ----------------------
   1 06/12/2026 13:00:00 opentsdb@USER.GUIDE
    Caution

    For JAAS settings and kinit commands, you must enter the principal exactly as it is included in the keytab. If the keytab contains a headless principal for the cluster such as hbase-<cluster>@USER.GUIDE, you will not be able to set it to principal="opentsdb@USER.GUIDE".

  6. Create an LDAP user.

    ldif=/tmp/add-opentsdb.ldif
HASH_USER_PW=$(slappasswd -h "{SSHA}" -s "new_password")

cat <<EOF | tee $ldif
dn: uid=opentsdb,ou=users,dc=USER,dc=GUIDE
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
cn: opentsdb
sn: opentsdb
uid: opentsdb
uidNumber: 30001
gidNumber: 20004
homeDirectory: /home/opentsdb
loginShell: /bin/bash
userPassword: ${HASH_USER_PW}
EOF

ldapadd \
  -H ldap://localhost:389 \
  -D "cn=root,dc=USER,dc=GUIDE" \
  -w "Kerberos_password" \
  -f "$ldif"

  7. Add the opentsdb user to the required group.

    ldif=/tmp/modify-opentsdb-group.ldif

cat <<EOF | tee $ldif
dn: cn=group name,ou=groups,dc=USER,dc=GUIDE
changetype: modify
add: memberUid
memberUid: opentsdb
EOF

ldapmodify \
  -H ldap://localhost:389 \
  -D "cn=root,dc=USER,dc=GUIDE" \
  -w "Kerberos_password" \
  -f "$ldif"

  8. Check that the LDAP user was registered correctly.

    ldapsearch \
  -H ldap://localhost:389 \
  -D "cn=root,dc=USER,dc=GUIDE" \
  -w "Kerberos_password" \
  -b "dc=USER,dc=GUIDE" \
  "(uid=opentsdb)"

2. Install OpenTSDB

To install OpenTSDB on a server:

  1. Use the following command to install OpenJDK on the server:

    sudo -s
yum install java-1.8.0-openjdk-devel.x86_64

  2. Use the yum package manager to install the gnuplot package required to create graphs for OpenTSDB.

    yum -y install gnuplot

  3. Create the OpenTSDB executing user and install the source.

    useradd opentsdb
su - opentsdb
git clone https://github.com/OpenTSDB/opentsdb
cd opentsdb

  4. Build the OpenTSDB source.

    ./build.sh

3. Install HBase client

To install the HBase client:

  1. Use the following command to download the HBase binary:

    • Cloud Hadoop 2.0 or later versions
    wget https://archive.apache.org/dist/hbase/2.2.6/hbase-2.2.6-bin.tar.gz
tar xvfz hbase-2.2.6-bin.tar.gz
ln -s hbase-2.2.6 hbase
    • Cloud Hadoop 1.9 or earlier versions
    wget https://archive.apache.org/dist/hbase/2.0.2/hbase-2.0.2-bin.tar.gz
tar xvfz hbase-2.0.2-bin.tar.gz
ln -s hbase-2.0.2 hbase
    Note

    For Cloud Hadoop Classic, download HBase 1.1.2.

  2. Carry out the following steps in order in the Ambari UI:

    1. Navigate to the [ACTIONS] > Download Client Configs menu.
    2. Check the filename of the HBase client configuration information.
    3. Run a command in the OpenTSDB server terminal to download the configuration file (HBASE_CLIENT-configs.tar.gz).
    • For more information about accessing and using the Ambari UI, see the Ambari UI guide.
      hadoop-chadoop-use-ex1_2-2_ko

  3. Unzip the HBase client configuration information.

    tar xvfz HBASE_CLIENT-configs.tar.gz -C ~/opentsdb/hbase/conf
sed -i -e 's|\(^export JAVA_HOME=.*\)|#\1|' ~/opentsdb/hbase/conf/hbase-env.sh

  4. Edit the opentsdb user's .bash_profile.

    echo 'export HBASE_HOME=~/opentsdb/hbase'                 >> ~/.bash_profile
echo 'export PATH=$PATH:${HBASE_HOME}/bin'                >> ~/.bash_profile
echo 'export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk'   >> ~/.bash_profile
source ~/.bash_profile

  5. Navigate to the Ambari UI > Hosts menu, then check the hostname and private IP information of the Secure Hadoop cluster.

    hadoop-chadoop-use-ex1_2-5_ko

  6. Use the following commands to add /etc/hosts information:

    • Add the hostname and private IP information of the Cloud Hadoop cluster you identified in the previous step.
    # root user
# echo 'IP            hostname'     >> /etc/hosts
echo  '1**.**.*.*  e-001-*****-**'  >> /etc/hosts
echo  '1**.**.*.*  m-001-*****-**'  >> /etc/hosts
echo  '1**.**.*.*  m-002-*****-**'  >> /etc/hosts
echo  '1**.**.*.*  d-001-*****-**'  >> /etc/hosts
echo  '1**.**.*.*  d-002-*****-**'  >> /etc/hosts

  7. Check the ZooKeeper quorum and znode parent values ​​in the HBase client settings.

    grep -E "hbase.zookeeper.quorum|hbase.zookeeper.property.clientPort|zookeeper.znode.parent|hbase.security.authentication" \
  $HBASE_HOME/conf/hbase-site.xml
    Caution

    Be sure to check the zookeeper.znode.parent value of the hbase-site.xml by navigating to Ambari Web UI > HBase > Configs or within the Secure Hadoop VM.

  8. Perform Kerberos authentication using the OpenTSDB run-as account.

    kinit -kt /etc/security/keytabs/opentsdb.keytab opentsdb@USER.GUIDE
klist

  9. Use the following command to test the hbase shell access:

    hbase shell

HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
hbase(main):001:0> whoami
hbase(main):002:0> exit
    • If you fail to access HBase Shell, add the OpenTSDB server to the ACG of Secure Hadoop.
    • For more information about setting up ACG, see the Firewall Settings (ACG) guide.

4. Grant HBase permissions

The OpenTSDB account requires HBase table creation permissions to create an OpenTSDB table.

  1. Access HBase Shell.

    sudo -u hbase hbase shell

    Alternatively, access HBase shell after running kinit.

    kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-<CLUSTER_NAME>@USER.GUIDE

  2. Grant permissions to the OpenTSDB account.

    grant 'opentsdb@USER.GUIDE', 'RWXCA'

  3. If OpenTSDB is configured to use a headless principal for the cluster, grant permissions to the actual principal included in the keytab as well.

    grant 'hbase-<CLUSTER_NAME>', 'RWXCA'
grant 'hbase-<CLUSTER_NAME>@USER.GUIDE', 'RWXCA'
    Note

    HBase permissions are applied to the username recognized by HBase after Kerberos authentication, not the Linux executing username. Grant permissions based on the whoami results of klist -kte <KEYTAB> and hbase shell.

  4. Review the permissions.

    user_permission '.*'
exit

    READ, WRITE, EXEC, CREATE, and ADMIN permissions must be displayed on opentsdb, opentsdb@USER.GUIDE, or the actual usage principal.

Note

If you are in an environment where you are managing HBase permissions through Ranger, you must also grant HBase table creation permissions to the same user in the Ranger policy.

5. Create OpenTSDB table

To create an OpenTSDB table:

  1. Perform Kerberos authentication using the OpenTSDB run-as account.

    kinit -kt /etc/security/keytabs/opentsdb.keytab opentsdb@USER.GUIDE
klist

  2. Create a TSDB table using the following command:

    export HBASE_HOME=~/opentsdb/hbase
export COMPRESSION=SNAPPY
export DATA_BLOCK_ENCODING=FAST_DIFF
export TSDB_TTL=2147483647
sh ~/opentsdb/src/create_table.sh
    Note

    The OpenTSDB default table creation script can use LZO as the default compression codec. If the LZO codec is missing in a Secure Hadoop environment, a ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec error may occur. In this case, use either COMPRESSION=SNAPPY or COMPRESSION=NONE.

  3. Use the following command to view the list of tables:

    hbase shell
hbase(main):001:0> list
TABLE
tsdb
tsdb-meta
tsdb-tree
tsdb-uid
4 row(s)

=> ["tsdb", "tsdb-meta", "tsdb-tree", "tsdb-uid"]
hbase(main):002:0> exit

6. Configure OpenTSDB

Configure the OpenTSDB configuration file and JAAS configuration file before running TSD.

  1. Create an OpenTSDB configuration directory and file.

    mkdir /home1/opentsdb/opentsdb/conf
vi /home1/opentsdb/opentsdb/conf/opentsdb.conf

  2. Configure the following items according to your environment:

    hbase.security.auth.enable = true
hbase.security.authentication = kerberos
hbase.kerberos.regionserver.principal = hbase/_HOST@USER.GUIDE
hbase.sasl.clientconfig = Client
tsd.network.port = 8888
tsd.http.staticroot = build/staticroot
tsd.http.cachedir = /home1/opentsdb/opentsdb/build/tmp
tsd.storage.hbase.zk_quorum = m-001-*****-**:2181,m-002-*****-**:2181,d-001-*****-**:2181
tsd.storage.hbase.zk_basedir = /hbase-unsecure
    Caution

    For tsd.storage.hbase.zk_basedir, you must input the zookeeper.znode.parent value from the HBase client settings. Do not guess the value and fix it to /hbase-secure or /hbase-unsecure.

  3. Create a JAAS configuration file.

    vi /home1/opentsdb/opentsdb/conf/opentsdb-jaas.conf

    3-1. If you are using a headless keytab for a cluster, configure it as follows:

    Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  doNotPrompt=true
  useTicketCache=false
  keyTab="/etc/security/keytabs/hbase.headless.keytab"
  principal="hbase-<CLUSTER_NAME>@USER.GUIDE";
};

    3-2. If you are using a keytab specifically for OpenTSDB, check the principal included in the keytab and enter the output value exactly as it is.

    klist -kte /etc/security/keytabs/opentsdb.keytab
    • Write the JAAS settings if the output example is as follows:
    KVNO Timestamp           Principal
---- ------------------- ----------------------
   1 06/09/2026 21:42:13 opentsdb@USER.GUIDE
    • Write the JAAS settings as follows:
    Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  doNotPrompt=true
  useTicketCache=false
  keyTab="/etc/security/keytabs/opentsdb.keytab"
  principal="opentsdb@USER.GUIDE";
};

  4. Check that Kerberos authentication works with the keytab.

    4-1. To check authentication when using hbase.headless.keytab:

    kdestroy
kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-<CLUSTER_NAME>@USER.GUIDE
klist

    4-2. To check authentication when using opentsdb.keytab:

    kdestroy
kinit -kt /etc/security/keytabs/opentsdb.keytab opentsdb@USER.GUIDE
klist
    Caution

    If kinit fails, TSD cannot run either. In this case, check the keytab file, principal value, and KDC time synchronization status first, rather than the OpenTSDB settings.

7. Run OpenTSDB

Use the following commands to run TSD on port 8888:

  1. Switch to the OpenTSDB run-as account.

    su - opentsdb
cd /home1/opentsdb/opentsdb

  2. Deliver the JAAS settings to the OpenTSDB JVM.

    export JVMARGS="-Djava.security.auth.login.config=/home1/opentsdb/opentsdb/conf/opentsdb-jaas.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dzookeeper.sasl.client=true -Dzookeeper.sasl.clientconfig=Client"

  3. Run TSD.

    ./build/tsdb tsd --config=/home1/opentsdb/opentsdb/conf/

  4. To have it run in the background, execute the following command:

    nohup ./build/tsdb tsd --config=/home1/opentsdb/opentsdb/conf/opentsdb.conf > /var/log/opentsdb/opentsdb.log 2>&1 &

  5. Check that port 8888 has opened correctly.

    ss -lntp | grep 8888

  6. Check the OpenTSDB API response.

    curl -s http://localhost:8888/api/version

    Example of a normal response:

    {
  "version": "2.5.0"
}

8. Store Time Series Data in OpenTSDB

To store time series data in OpenTSDB:

  1. Use the following commands to create a cpu.util metric:

    export JVMARGS="-Djava.security.auth.login.config=/home1/opentsdb/opentsdb/conf/opentsdb-jaas.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dzookeeper.sasl.client=true -Dzookeeper.sasl.clientconfig=Client"
./build/tsdb mkmetric cpu.util --config=/home1/opentsdb/opentsdb/conf/opentsdb.conf

  2. Use the following commands to create a cpu.util metrics collection script: (sar-data-input.sh)

    • Script source: This script is referenced from the original article Effective Monitoring and Alerting.
    #!/bin/bash
# Ignore sar's header.
LC_ALL=en sar -u 1 | sed -u -e '1,3d' |
while read time cpu usr nice sys io steal idle;
do
    NOW=$(date +%s)
    echo put cpu.util $NOW $usr time=user
    echo put cpu.util $NOW $sys time=system
    echo put cpu.util $NOW $io time=io
    echo put cpu.util $NOW $idle time=idle
    # Report values to standard error.
    echo timestamp:$NOW user:$usr sys:$sys io:$io idle:$idle >&2
done | nc -w 30 localhost 8888

  3. Use the following command to transfer metrics data to OpenTSDB:

    sh sar-data-input.sh

9. Check OpenTSDB Graph

To view the OpenTSDB graphs:

  1. Check the public IP of the OpenTSDB server in the server console.
    hadoop-chadoop-use-ex1_6-1_ko
  2. Access port 8888 on the OpenTSDB server and check the collected graphs.
    hadoop-chadoop-use-ex1_6-2_ko

Troubleshoot

Kerberos TGT error

If the following error occurs, Kerberos authentication is not complete:

Failed to find any Kerberos tgt

Execute the following commands to carry out authentication:

kinit -kt /etc/security/keytabs/opentsdb.keytab opentsdb@USER.GUIDE
klist

No keytab file

If the following error occurs, the keytab file does not exist or the path is incorrect:

Key table file '/etc/security/keytabs/opentsdb.keytab' not found

Create a keytab on the master node and deploy it to the OpenTSDB execution server.

ktadd -k /etc/security/keytabs/opentsdb.keytab opentsdb@USER.GUIDE

No HBase table creation permissions

If the following error occurs, you have insufficient HBase permissions:

AccessDeniedException: Insufficient permissions for user 'opentsdb@USER.GUIDE' (action=create)

Grant permissions using the HBase administrator account.

grant 'opentsdb', 'RWXCA'
grant 'opentsdb@USER.GUIDE', 'RWXCA'

If only the short name opentsdb has permissions and the actual requesting user is recognized as opentsdb@USER.GUIDE, permission matching may fail. In this case, you must grant permissions to the full principal as well.

LZO codec error

If the following error occurs, the LZO codec is not in the HBase classpath:

ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec

Change the compression codec before creating a table.

export COMPRESSION=SNAPPY

Alternatively, do not use compression.

export COMPRESSION=NONE

HBase Shell runs, but creation fails

If hbase shell runs, that means that Kerberos authentication was successful. However, separate HBase permissions are required to create tables.

Check the following items:

klist

whoami
user_permission '.*'

If the whoami result is opentsdb@USER.GUIDE, but permissions are granted only to opentsdb, table creation may fail. In this case, add permissions to the full principal.

grant 'opentsdb@USER.GUIDE', 'RWXCA'

No LoginModules configured for Client error when running TSD

If the following error occurs, the OpenTSDB JVM failed to read the JAAS configuration file:

javax.security.auth.login.LoginException: No LoginModules configured for Client

When this error occurs, check the following items:

  • Ensure that the /home1/opentsdb/opentsdb/conf/opentsdb-jaas.conf file exists.
  • Ensure that the JAAS file contains the Client { ... }; section.
  • Before running TSD, ensure that the -Djava.security.auth.login.config option is set in JVMARGS.

Run example:

export JVMARGS="-Djava.security.auth.login.config=/home1/opentsdb/opentsdb/conf/opentsdb-jaas.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dzookeeper.sasl.client=true -Dzookeeper.sasl.clientconfig=Client"
./build/tsdb tsd --config=/home1/opentsdb/opentsdb/conf/opentsdb.conf

No password provided error when running TSD

If the following error occurs, the JAAS file was read but keytab authentication failed, and the state has switched to password prompt:

javax.security.auth.login.LoginException: No password provided

When this error occurs, check the following items:

  1. Check the principal included in the keytab.

    klist -kte /etc/security/keytabs/hbase.headless.keytab

  2. Check that the principal value of JAAS exactly matches the klist -kte output.

  3. Ensure that doNotPrompt=true is included in JAAS.

  4. Check if direct authentication is possible using the keytab.

    kdestroy
kinit -kt /home1/opentsdb/opentsdb/conf/hbase.headless.keytab hbase-<CLUSTER_NAME>@USER.GUIDE
klist

  5. If kinit fails, the keytab file or principal value is incorrect. Either recreate the keytab correctly, or edit the keyTab and principal value in JAAS.

ZooKeeper AuthFailed error

If the following error occurs, ZooKeeper SASL authentication has failed:

WatchedEvent state:AuthFailed type:None path:null

When this error occurs, check the following items:

  • Check that the -Dzookeeper.sasl.client=true option is set.
  • Check that the -Dzookeeper.sasl.clientconfig=Client option is set.
  • Ensure that the Client section of the JAAS file is using the correct keytab and principal.
  • Check that kinit -kt <KEYTAB> <PRINCIPAL> is successful.

TSD is running, but port 8888 is not open

Check the TSD process and port using the following commands:

ps -ef | grep '[t]sdb tsd'
ss -lntp | grep 8888

If the TSD process is missing, check the OpenTSDB log.

tail -200 /var/log/opentsdb/opentsdb.log

HBase meta region is not online

OpenTSDB must be able to access the hbase:meta region of HBase. If the HBase Master is being initialized or the number of HBase regions is displayed as 0, OpenTSDB may also fail to run.

Check the HBase status using the following command:

echo "status 'simple'" | hbase shell -n

In a normal state, the live region server is displayed, and the hbase:meta region at least must be online. If the following logs repeat, have an HBase administrator or service representative restore the HBase meta region status first.

hbase:meta,,1.1588230740 is NOT online
Master startup cannot progress
PleaseHoldException: Master is initializing
Caution

Directly modifying or deleting ZooKeeper HBase znodes can affect the HBase service status. Perform the HBase meta region recovery task according to the instructions of the HBase administrator or service representative.

Business Registration Number: 129-86-31394 E-commerce Permit Number:No. 2009-GyeonggiSeongnam-0510
CEO: Kim Yuwon Address: NAVER Green Factory 17th–26th Floors, Buljeong-ro 6, Bundang-gu, Seongnam-si, Gyeonggi-do 13561, Republic of Korea. Customer Support Number: 1544-5876
© NAVER Cloud Corp. All Rights Reserved.