Connect Object Storage and Sub Account

Available in Classic

This chapter describes how to connect and use the Cloud Hadoop, Object Storage, and Sub Account of NAVER Cloud Platform.

Bigdata Service

The services integrated in this example are as follows:

• Cloud Hadoop: You can easily build open source frameworks such as Hadoop, HBase, Spark, Hive, and Presto. For more information on Cloud Hadoop, see the Cloud Hadoop overview guide.

• Object Storage: You can use it to store data that requires secure storage or to store large volumes of data.
  You can also use the provided APIs for server data backup and recovery purposes. For more information on Object Storage, see the Object Storage overview guide.

• Sub Account: You can register users as sub accounts and grant permissions for specific services.
  The registered internal users can use the services for which they have been authorized in the same manner as the main account. For more information on Sub Accounts, see the Sub Account user guide.

Integrate and use Hadoop, Object Storage, and Sub Account

The process of analyzing data stored in Object Storage using Cloud Hadoop by integrating the three services is as follows:

Create and add policies for Sub Accounts

You can register users as sub accounts and grant permissions for specific services. The registered users can use the services for which they have been authorized in the same manner as the main account. For more information on Sub Accounts, see the Sub Account user guide.

To create and add policies for Sub Accounts:

1. Create the Sub Account service from the NAVER Cloud Platform console.
   • To access Object Storage, you must use an API Key. When creating Sub Accounts, select API Access in the Access Type field.
     
2. Click the login ID of the Sub Account created in the Sub Account menu, and then add a usage policy under the Policy tab.
   • Add the Cloud Hadoop NCP_CLOUD_HADOOP_MANAGER policy.
   • Add the Object Storage NCP_OBJECT_STORAGE_MANAGER policy.
   • Add the NCP_SERVER_MANAGER policy for ACG change management.
     
3. Click [Add] in the [Access Key] tab menu to add an API authentication key for accessing Object Storage, and then confirm.
   
4. Set the Sub Account login page access key in Sub Account > Dashboard.
   
5. Access the URL of the Sub Account login page, and then log in to the sub account as follows:
   • The sub account ID is the Login ID set when creating the sub account.
   • The password is the Login password set when creating the sub account.
     

• You can only use the services allowed by the policies set for the account.
  

Create Object Storage bucket

Create an ncp-bigdata bucket in Object Storage to store data. You can check the created bucket in Bucket Management. For more information on creating buckets, see the Object Storage user guide.

Create Cloud Hadoop cluster

Create a Cloud Hadoop cluster. For more information about how to create a Cloud Hadoop cluster, see Getting started guide.

Upload sample data onto Object Storage

1. Download the dataset.
   • Unzip the downloaded file.
2. Click on the ncp-bigdata bucket in the Object Storage > Bucket Management menu, and then upload the unzipped file.
   

View data

You can view sample data using Web UIs, including Ambari, Hue, and Zeppelin. For more information about how to connect to the Web UI, see the UI access and password settings by service guide.

1. When you complete creating your Cloud Hadoop cluster, add the ports required for Ambari, Hue, and Zeppelin access your ACG.
   • For how to set ACG, see Set ACG rules.
2. Access Hue (Port 8000), and then create a mart database with a Hive query.

CREATE DATABASE mart;

3. Create an orders2 table as follows:

CREATE EXTERNAL TABLE `orders2` (
        `order_id` BIGINT,
        `order_number` BIGINT,
        `days_since_prior_order` DOUBLE

)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
location 's3a://ncp-bigdata/';

4. View the data of the orders2 table.

SELECT * FROM orders2;

5. Access Zeppelin (port 9995).
6. Click [Notebook] > Create new note, and then create a new Zeppelin Notebook.
   
7. Designate Default Interpreter as jdbc, and click the [Create].
   
8. View the orders2 table registered on Hive Metastore.

%jdbc(hive)
SELECT order_number, AVG(days_since_prior_order)
FROM mart.orders2
WHERE 1=1
AND order_number IS NOT NULL
GROUP BY order_number
ORDER BY order_number

• Proceed with a simple analysis of the uploaded CSV file, which was uploaded as an example.
• It is assumed that as the number of orders increases, the time taken for an order to be repeated shortens.
• Orders2 table
  • order_id: Order number.
  • order_number: Number of orders.
  • days_since_prior_order: Number of days since last order.
• In the Orders2 table, days_since_prior_order is a column that displays how long it has been since the last order up to 30 days, and order_number is the number of orders by the user.
• In Zeppelin, view the Hive table (orders2) created in Hue using Cloud Hadoop, and then visualize it into a chart as follows:
  
• When you take a look at the chart, you can assume that customers with a higher number of orders have a shorter time before the next reorder.