Linking Object Storage + Sub Account

Available in Classic

This page introduces the method for linking and using the Cloud Hadoop, Object Storage, and Sub Account of NAVER Cloud Platform.

Bigdata Service

• Cloud Hadoop: You can easily establish open source frameworks such as Hadoop, HBase, Spark, Hive and Presto.
  Please refer Cloud Hadoop overview for more information about Cloud Hadoop.

• Object Storage: It can be used for data that requires safe storage or large in volume.
  It can also be used for backup and restoration of server data with the provided APIs.
  Please refer Object Storage overview for more information about Object Storage.

• Sub Account: You can register users as sub accounts and grant permissions for specific services.
  The registered users can use the service for which they have been authorized in the same manner as their main account.
  Please refer Sub Account Guide for more information about Sub Account.

Link and use Hadoop + Object Storage + Sub Account

Create Sub Account and add policies

You can register users as sub accounts and grant permissions for specific services. The registered users can use the service, for which they have been authorized, in the same manner as their main account.

Please refer to the Sub Account Guide for more information about Sub Account.

The following describes how to create Sub Account and add policies.

1. Create the Sub Accounts service from the NAVER Cloud Platform console.

   • When you create Sub Accounts, select and set API access under the access type item so the API keys can be used for accessing Object Storage.
     

2. Select the Sub Account created from the list of Sub Accounts, and set the usage policy in the [Policy] tab.

   • Add NCP_CLOUD_HADOOP_MANAGER policy from Cloud Hadoop.
   • Add the NCP_OBJECT_STORAGE_MANAGER policy of Object Storage.
   • Add NCP_SERVER_MANAGER policy for ACG change management.
     

3. Create and check the API authentication key for the Object Storage access from the [API Key] tab.
   

4. Log in with the Sub Account from Sub Account > Dashboard > Sub account login page.

   • The sub account ID is the Login ID set when creating the sub account.
   • The password is the Login password set when creating the sub account.
     

• You can only use the services allowed by the policies set for the account.
  

Create Object Storage bucket

Create a bucket to save data in Object Storage. The created bucket can be viewed from the Bucket Management menu.

Create Cloud Hadoop cluster

Create a Cloud Hadoop cluster.
Please refer to Getting started with Cloud Hadoop for more information about creating Cloud Hadoop.

Upload the sample data to Object Storage

1. Download the dataset from here.
   • Decompress the downloaded file.
2. Upload the decompressed file to the bucket created at Object Storage > Bucket Management.
   

Data query

You can search the sample data from web UIs such as Ambari, HUE, Zeppelin, etc.

1. Once the Cloud Hadoop creation is complete, please add ports to connect to Ambari, HUE, and Zeppelin on ACG.
   • For ACG setting method, refer to ACG rule setting.
2. Connect to HUE (Port 8000), create a mart database with a Hive query, and then create an Orders table.

CREATE DATABASE mart;

3. Look up the data.

CREATE EXTERNAL TABLE `orders2` (
        `order_id` BIGINT,
        `order_number` BIGINT,
        `days_since_prior_order` DOUBLE

)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
location 's3a://ncp-bigdata/';

4. Connect to Zeppelin (Port 9995).

SELECT * FROM orders2;

5. Connect to Zeppelin (port 9995).
6. Click [Notebook] > Create new note and create a new Zeppelin Notebook.
   
7. After setting the Default Interpreter to jdbc, click the [Create] button.
   
8. Search the orders2 table registered on the Hive metastore.

%jdbc(hive)
SELECT order_number, AVG(days_since_prior_order)
FROM mart.orders2
WHERE 1=1
AND order_number IS NOT NULL
GROUP BY order_number
ORDER BY order_number

• Proceed with a simple analysis of the CSV file, which was uploaded as an example.
• With the assumption that more orders result in shorter reorder terms, proceed with verification.

<Orders2 table>

• order_id: order number
• order_number: number of orders
• days_since_prior: time since last order

9. Search the Hive table (orders2) created from HUE using Cloud Hadoop from Zeppelin, and then visualize it into a chart as follows.
   

When you take a look at the chart, you can assume that customers with higher number of orders have shorter time before the next reorder.