Using Hive

Available in VPC

This guide describes a method to save data in an Object Storage bucket and run a simple Hive query through Hue and Beeline.

Architecture example

Save data that needs to be kept in an Object Storage bucket, and use the Cloud Hadoop cluster only when necessary.

When running the Hive query, the query is run to the next step.

1. Submit query to Hive server in Cloud Hadoop cluster in Hive client
2. The server processes the query and requests metadata from the Hive metadate DB (MySQL) installed on the master server
3. Server loads the data stored in Object Storage bucket
4. Hive server returns result to the client

Create Hive table

<example> Upload a sample data file to Object Storage bucket of NAVER Cloud Platform, and create Hive External Table to allow the data to be used on Hive

fs.s3a.access.key=<API-ACCESS-KEY>
fs.s3a.connection.ssl.enabled=false
fs.s3a.endpoint=http://kr.objectstorage.ncloud.com
fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
fs.s3a.secret.key=<API-SECRET-KEY>

The method to create a Hive table is as follows:

1. Download sample data. Unzip the data and upload the AllstarFull.csv file to your Object Storage bucket.

   • Since Hive reads the location of the data file by folder, save one data item per folder. (Recommended)
   Note

   The provided sample data is a portion of Lahman's Baseball Database Version 2012, and all copyrights of the data belong to Sean Lahman.

   

2. Create Hive External Table with the following phrase in Hue Hive Editor.

   • location: designate a bucket path with a data set file saved

   DROP table allstarfull;
   
   CREATE external TABLE if not EXISTS `allstarfull` (
           `playerID` VARCHAR(20),
           `yearID` INT,
           `gameNum` INT,
           `gameID` VARCHAR(30),
           `teamID` VARCHAR(4),
           `lgID` VARCHAR(4),
           `GP` INT,
           `startingPos` INT
   )
   ROW FORMAT DELIMITED
   FIELDS TERMINATED BY ','
   location 's3a://deepdrive-hue/input/lahman2012/allstarfull';
   

Run Hive query

You can run a Hive query using various tools provided by Cloud Hadoop. This guide describes how to run a Hive query with the following <Example of Hive query running tools>.

• <Example of Hive query running tools>
  • Hive interpreter of Hue
  • Beeline based on SQLLine CLI among Hive clients

Hive interpreter of Hue

The method to run Hive query using Hive interpreter of Hue is as follows:

1. Log in to Hue and run HiveQL query in Hive interpreter.

   SELECT * FROM allstarfull LIMIT 10;
   

2. Check if the file uploaded to s3 is connected to the Hive table through the result page.
   

Beeline

The method to run Hive query using Beeline is as follows:

1. Directly access the host with the Hive client using SSH and start a Beeline session with the following command:

   • Enter the master node (m-00x) host name into [HIVE-SERVER2-SERVER].

   beeline -u "jdbc:hive2://[HIVE-SERVER2-SERVER]:10000" -n hive
   

   Note

   For how to access the host, see the Connecting to cluster nodes through SSH guide.

2. When the Beeline prompt is displayed, run HiveQL query.

   • The following query does not print the result but saves it in Object Storage.

   beeline> INSERT OVERWRITE DIRECTORY 's3a://deepdrive-hue/output/' ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
   SELECT playerid, sum(gp) from allstarfull group by playerid;
   

   Note

   Before saving the query, make sure the specified location (directory) exists in the Object Storage bucket.

3. Run the following command to check if the result was saved successfully.

   hadoop fs -cat s3a://deepdrive-hue/output/000000_0
   

   

4. The result is as follows:

   aaronha01,24
   aasedo01,1
   abreubo01,2
   adamsac01,0
   adcocjo01,2
   ageeto01,2
   aguilri01,3
   aguirha01,1
   alexado01,0
   alfoned01,1
   allendi01,6
   allenjo02,1
   alleyge01,1
   allisbo01,2
   alomaro01,12
   ...
   

Activation of LLAP of Hue

The method to activate LLAP in the editor list of Hue is as follows:

1. Access Ambari and activate Hive LLAP in [Hive] > [CONFIGS] > [SETTINGS]. Add settings, press [SAVE] to save, and press [RESTART] to restart.

2. Activate Hue Hive LLAP Module in [Hue] > [CONFIGS] > [Hue Service Module]. Add settings, press [SAVE] to save, and press [RESTART] to restart.

3. You can see the activated LLAP editor in the editor list of Hue.