AI Forest scenario

Available in VPC

The following shows a scenario where AI Forest is used to upload data to HDFS and execute jobs.

Create workspace

The following describes how to create a workspace.

1. From the NAVER Cloud Platform console, click the Services > Big Data & Analytics > Data Forest menus, in that order.

2. Click AI Forest > Workspace > [Create workspace] > [Advanced workspace].

3. Enter the workspace settings information.

   • Account name: df-test
   • Workspace name: ws
   • Workspace type: Singlebatch

4. Select TensorFlow for the Docker image, and r2.3.1-py3 for the image version.
   

   Note

   PyTorch is an open source machine learning library for Python programs. For more information, refer to the PyTorch website.

5. Select the default value for the GPU model name, number of GPU cores, and memory capacity.
   

6. Enter the input items in the data settings area, and then click the [Add] button.

   • Input
     • Input Path: path for input data to be copied from HDFS to the container Enter "/user/{username}/data_in".
     • Input Container Local Path: path for data saved after being copied to the container Enter "data_in".
   • Output
     • Output Path: path for output data to be copied from the container to HDFS Enter "/user/{username}/data_out".
     • Output Container Local Path: path for output data saved in the container Enter "data_out".
       

7. Click the [Next] button. The workspace creation is completed.

Upload data to HDFS

If the user uses data sets for manual model learning, then the data set must be in prepared in Input Path entered at the time when the workspace was created for Data Forest.

Upload the data to the HDFS path below. Uploaded data sets are copied to the Input Container Local Path entered at the time when the workspace was created.

• hdfs://koya/user/{Data Forest account}/data_in
  

Write code in workspace

The following describes how to write code in a workspace.

1. From the NAVER Cloud Platform console, click the Services > Big Data & Analytics > Data Forest menus, in that order.
2. Click AI Forest > Workspace browser.
3. Select the account and workspace, and then click the [Upload] button.
4. When the upload window appears, drag the run.sh file to the upload window.

   • run.sh input example

     #!/bin/bash
     ls -l /data_in
     

5. Click the [Start upload] button.
6. Click the [OK] button when the upload is completed.

Submit GPU batch jobs

1. From the NAVER Cloud Platform console, click the Services > Big Data & Analytics > Data Forest menus, in that order.
2. Click AI Forest > Workspace browser.
3. Select an account, and then a workspace.
4. Select the checkbox of the run.sh file which is a shell script file, and then click the [Run] button.
5. Enter ./run.sh in the Command field when the Run window appears, and then enter aiforest in App name.
6. Click the [Run] button.
   
7. You can see the executed jobs and their details from the AI Forest > DL app menu.

Check execution logs

The following describes how to check execution logs.

1. From the NAVER Cloud Platform console, click the Services > Big Data & Analytics > Data Forest > App menus, in that order.
2. Click Quick links > AppMaster URL under the details of the app currently running.
   
3. Click the Cluster > Application menu.
4. Find and click the ID of the application with a name value that begins with "aiforest."
   
5. Click Logs of the application ID. You can check the logs of the executed app.