Using Hive

Available in VPC

Data Forest supports building an independent Apache HiveServer2 (HS2) service environment for each user. Hive Metastore is required, and Data Forest provides the Hive Metastore. Hive is a SQL-based data warehouse solution for analyzing and processing large volumes of data stored in data storage systems.

Check HIVESERVER2-LDAP app details

Once the app is created, you can view its details. If the Status in the app details is Stable, the app is running normally.
To view app details:

1. In the VPC environment on the NAVER Cloud Platform console, navigate to > Services > Big Data & Analytics > Data Forest.
2. Click Data Forest > Apps on the left.
3. Select the account that owns the app.
4. Click the app to view its details.
5. Review the app details.
   
   • Quick links
     • supervisor-hs2-auth-ldap-0: Supervisor URL for managing HS2
     • shell-hs2-auth-ldap-0: HS2 web shell URL
     • webui-hs2-auth-ldap-0: URL for accessing the HS2 Web UI
       • home: View running sessions and recently executed Hive queries.
       • configuration: View HS2 configuration in XML format.
       • Metrics Dump: View real-time JMX metrics in JSON format.
       • Stack Trace: View stack traces of all active threads.
       • LLAP Daemon: View the status of Hive LLAP daemons.
       • local logs: View local logs. Operator access only.
   • Connection String: URL for accessing the HS2 app
     • JDBC connection string (inside-of-cluster): Connection string used for connections via JDBC from Beeline, Zeppelin, Hue, and user-defined programs. This address is used when accessing HS2 from the Data Forest internal network.
     • JDBC connection string: Address used when accessing HS2 from the Data Forest external network. The JDBC connection string(inside-of-cluster) cannot be accessed from a user’s PC. When it is difficult to distinguish between internal and external networks, you can use JDBC connection string.
     • JDBC connection string(inside-of-cluster) Example: Before using the example link, replace the changeme in the password parameter with the user account password.
     • JDBC connection string Example: Before using the example link, replace the changeme in the password parameter with the user account password.
   • Component: The default values are the recommended resources. The HIVESERVER2-LDAP-3.1.0 type is composed of 1 hs2-auth-ldap component.
     • hs2-auth-ldap: Component to process the authentication in the LDAP method.

Example:

The following shows the HS2 access interface.

Access the HiveServer2 app with Beeline

To access HS2 with the beeline -u {JDBC connection string} -n {username} -p {password} command:

# Keberos authentication
$ curl -s -L -u test01:$PASSWORD -o df.test01.keytab "https://sso.kr.df.naverncp.com/gateway/koya-auth-basic/webhdfs/v1/user/test01/df.test01.keytab?op=OPEN"
$ ls -al
total 20
drwxr-s--- 4 test01 hadoop  138 Dec 16 17:57 .
drwxr-s--- 4 test01 hadoop   74 Dec 16 17:44 ..
-rw-r--r-- 1 test01 hadoop  231 Dec 16 17:36 .bashrc
-rw------- 1 test01 hadoop  302 Dec 16 17:36 container_tokens
-rw-r--r-- 1 test01 hadoop  245 Dec 16 17:57 df_beta.test01.keytab
lrwxrwxrwx 1 test01 hadoop  101 Dec 16 17:36 gotty -> /data1/hadoop/yarn/local/usercache/test01/appcache/application_1607671243914_0024/filecache/10/gotty
-rwx------ 1 test01 hadoop 6634 Dec 16 17:36 launch_container.sh
drwxr-S--- 3 test01 hadoop   19 Dec 16 17:53 .pki
drwxr-s--- 2 test01 hadoop    6 Dec 16 17:36 tmp
$ kinit example -kt df.example.keytab 

# Access HS2
$ beeline -u "jdbc:hive2://hs2-auth-ldap.new-hiveserver2.test01.kr.df.naverncp.com:10001/;transportMode=http;httpPath=cliservice" -n test01 -p '{password}'

If the access is made successfully, the following appears.

Connected to: Apache Hive (version 3.1.0.3.1.0.0-78)
Driver: Hive JDBC (version 3.1.0.3.1.0.0-78) 
Transaction isolation: TRANSACTION_REPEATABLE_READ

Use Hive in Zeppelin

1. Access Zeppelin, click the account name in the top right of the interface, and then click the interpreter menu.
   

2. Search for the JDBC interpreter.
   

3. Add the settings in hive url Properties as follows by referring to Connection String > JDBC connection string Example in the app details.
   

   Note

   If the password set when creating the account includes any special characters, you must replace them using URL encoding before entering it.

Access the shell in the container

You can access the shell (/bin/bash) through the web browser in the HS2 app container. You can access the shell to easily view the container's status and perform writing tasks such as changing the configuration file or downloading files in the shell.

1. Access shell-hs2-auth-ldap-0 from the Quick links list.
2. Log in to access the shell.
   • Username: Enter the account name of the user who has run the HS2 app.
   • Password: Enter the account password.

Cautions for using Hive

Hive rules

Unlike public HS2, you can print all database lists of other users with the SHOW DATABASES; command. However, you cannot access other users' databases without permission.
When using the HS2 app, read the Using public Hive guide and comply with the rules.

Database naming rules

The same database naming rules apply as in Hive. In the HS2 app, the CREATE DATABASE command does not fail even if the name violates these rules. Because the HS2 app cannot enforce database naming rules at the system level, make sure to follow the rules when creating a database. If database naming rules are violated, databases created in the HS2 app are not visible in the public HS2.

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.reflect.UndeclaredThrowableException)

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.reflect.UndeclaredThrowableException) (state=08S01,code=1)