Integrate Cloud DB for MySQL with external Hive Metastore (optional)

Available in VPC

This guide introduces how to integrate NAVER Cloud Platform Cloud DB for MySQL with Cloud Hadoop's external Hive Metastore.

Preliminary task

1. Create a Cloud Hadoop cluster.
   • For more information about how to create a Cloud Hadoop cluster, see Getting started.
2. Create Cloud DB for MySQL.
   • Make sure to create MySQL 8.0.x version, as integrations are only available in MySQL 8.0.x version.
   • For more information about how to create Cloud DB for MySQL, see Cloud DB for MySQL server creation guide.
   • After the creation is complete, to integrate with Cloud Hadoop, navigate to Database > Cloud DB for MySQL > DB management > DB Config management and set the sql_require_primary_key configuration option to OFF.
     
3. Set up an ACG.
   • In order to access Cloud DB for MySQL from Cloud Hadoop, enter Cloud Hadoop ACG in the access source of Cloud DB for MySQL ACG and allow all communication ports (1-65535).
     

External Hive Metastore

Integrate with external Hive Metastore

Hive Metastore server uses DB storage to manage metadata. The default settings are configured to use MySQL installed within Cloud Hadoop.
By using the external Hive Metastore integration method, Cloud DB for MySQL can be used by changing it to the metadata storage in the Hive Metastore server.
You need to check whether there is a Hive Job in progress as the Hive Metastore server is automatically restarted according to the Metastore storage replacement task.
The replacement of metadata storage does not include the transfer of existing metadata.

To change to the external Metastore:

1. In the VPC environment on the NAVER Cloud Platform console, navigate to > Services > Big Data & Analytics > Cloud Hadoop.
2. After selecting a cluster to be integrated from the cluster list, click [Manage clusters] > Integrate with external Hive Metastore.
3. After entering the user_ID and password, click [Test connection].
   • There may be restrictions on usage depending on the password plugin settings of Cloud DB for MySQL.
     You need to manage and integrate the Cloud DB for MySQL account in accordance with the Cloud Hadoop external Hive Metastore account policy.
4. After checking whether the test connection is normal, click [Yes].
   • For a successful test connection, you need to perform the following items in advance:
   • After running the test connection, you can check the content of the console message to see whether it was successful.
     • Normal: The connection test result to Cloud DB for MySQL is normal.
     • ACG blocked: Access to the DB Server from Cloud Hadoop is blocked. Edit the ACG settings in Cloud DB for MySQL.
     • Incorrect account ID and password: The account ID or password has been entered incorrectly. Enter the correct account name and password.
     • Failure to comply with password restrictions: User password must contain a minimum of 8 letters and a maximum of 20 letters and must contain at least 1 English uppercase and lowercase letter, special character, and number. Special characters such as ', ", `, |, /, &, +, or space are not allowed.
     • Account DDL not in possession: The account you entered cannot perform DDL commands. Edit the DB permission as DDL.
     • When there is no DB creation procedure in Cloud DB for MySQL: Database control is not smooth. Check whether the sys.ncp_create_db procedure is working properly.
     • When Cloud DB for MySQL is already integrated: The requested DB Service is currently integrated with Cloud Hadoop. Select another DB service.
     • When the master node is down: The status of Cloud Hadoop's master node is not normal. A server inspection is needed.
     • Unknown error: Contact Customer service.
     • When Hive is deleted from the customer's cluster: Hive is not installed in Cloud Hadoop. Check the cluster.
5. Once the Hive Metastore integration task starts, the Status field in the cluster list or details becomes Configuring and then displays again as Running once the task is completed.
   
6. Check the MySQL integration result.
   • After accessing the Rocky server created under the Server service, access the MySQL domain as shown below and verify that the hive_***_CHadoop database and internal tables have been created successfully.

   $ yum -y install mysql
   $ mysql --version
   $ mysql -u [Account name] -p -h [mysql domain]
   
   mysql> show databases;	# When integrated with the Cloud Hadoop cluster, confirm that the hive_***_CHadoop DB is listed
   mysql> use hive_***_CHadoop;	# hive_***_Use Chadoop DB
   mysql> show tables; 	# Confirm that tables are listed