Using the Monitoring Console
  • PDF

Using the Monitoring Console

  • PDF

Available in VPC environments.

Monitoring supports two types of dashboards for you to check various monitoring information on Cloud Hadoop performance and history. As Monitoring is a service included in NAVER Cloud Platform's Cloud Hadoop, it can be used without additional charges.

The types of dashboards provided by Monitoring are as follows.

  • HADOOP dashboard: monitoring information related to Cloud Hadoop in operation
  • OS dashboard: hardware and network information of each Cloud Hadoop server in operation

Through these two types of dashboards, you can check the information related to Cloud Hadoop from the last 2 months, as well as each server's hardware and network indicators. Each dashboard consists of graphic charts, and you can increase work efficiency by printing specific graphic charts or downloading them to your PC as files in various extensions.

Note

You can set for the user to be notified when monitoring results show that a certain indicator has exceeded its threshold or met specific conditions by having them recognized as events. For more details about using event and notification settings, refer to Monitoring Cloud Hadoop with Cloud Insight.

Getting started with Monitoring

  1. In order, click the Services > Big Data & Analytics > Cloud Hadoop menus on the NAVER Cloud Platform console.
  2. Click the [Create cluster] button, and then create a Cloud Hadoop cluster.
  3. Click the Cloud Hadoop > Monitoring menus on the left.
  4. Click the clusters to be monitored from the Cloud Hadoop cluster list.

Monitoring page

The basics of using Monitoring are as follows.

chadoop-vpc-monitoring1_ko

  • You can select the currently operating Cloud Hadoop clusters and the servers for each cluster from the area on the left.
  • Click the cluster name to display the HADOOP dashboard on the area on the right, and then click the server under the cluster name to display the OS dashboard.

Check Monitoring dashboards

The dashboards provided by Monitoring consist of multiple graphic charts. Users can display and intuitively check only the information they want from the dashboard of each cluster. The following describes how to use dashboards.

HADOOP Dashboard

chadoop-vpc-monitoring2_ko

  • By clicking the cluster you want from the Cloud Hadoop cluster list on the left, you can view the HADOOP dashboard as shown on the right.
    • The HADOOP dashboard collects data every minute.
    • Monitoring information are based on average values, and the frequency of queries varies, depending on the selected period type.
  • The indicators you can check for each group are as follows.
Group Indicator Unit Explanation
Apps apps_completed num number of applications submitted to YARN that have completed
apps_failed num number of applications submitted to YARN that have failed to complete
apps_killed num number of applications submitted to YARN that have been killed
apps_pending num number of applications submitted to YARN that are in a pending state
apps_running num number of applications submitted to YARN that are running
apps_submitted num number of applications submitted to YARN
Blocks corrupt_blocks num number of blocks that HDFS reports as corrupted
missing_blocks num number of blocks in which HDFS has no replicas
pending_deletion_blocks num number of blocks marked for deletion
pending_replication_blocks num status of block replication: blocks being replicated, age of replication requests, and unsuccessful replication requests
under_replicated_blocks num number of blocks that need to be replicated one or more times
Containers allocated_container num number of resource containers allocated by the ResourceManager
pending_containers num number of containers in the queue that have not yet been allocated
reserved_containers num number of containers reserved
HDFS capacity(GB) capacity_remaining_gb GB amount of remaining HDFS disk capacity
HDFS read/write(bytes) hdfs_bytes_read num number of bytes read from HDFS
hdfs_bytes_written num number of bytes written to HDFS
HDFS utilization(%) hdfs_utilization % percentage of HDFS storage currently used
Memory(MB) allocated_mb MB amount of memory allocated to the cluster
available_mb MB amount of memory available to be allocated
reserved_mb MB amount of memory reserved
total_mb MB total amount of memory in the cluster
Nodes num_live_data_nodes num number of data nodes that are receiving work from Hadoop
unhealthy_nodes num number of nodes available to MapReduce jobs marked in an UNHEALTHY state
active_nodes num number of nodes presently running MapReduce tasks or jobs
decommissioned_nodes num number of nodes allocated to MapReduce applications that have been marked in a DECOMMISSIONED state
lost_nodes num number of nodes allocated to MapReduce that have been marked in a LOST state
rebooted_nodes num number of nodes available to MapReduce that have been rebooted and marked in a REBOOTED state
total_nodes num number of nodes presently available to MapReduce jobs
V_cores allocated_v_cores num number of core nodes working
pending_v_cores num number of core nodes waiting to be assigned
Data transfers total_load num total number of concurrent data transfers
YARN memory(%) yarn_memory_available_percentage % percentage of remaining memory available to YARN (= available_mb / total_mb)
  • You can monitor the changes in the cluster's indicators in real time.
    • This is how indicators change when the number of cluster data nodes is reduced.
      chadoop-vpc-monitoring3_ko
  • Users can zoom in or out of graphs by placing the mouse cursor directly over the graph, as well as view indicators from the desired period on the dashboard by setting the search period.
    chadoop-vpc-monitoring4_ko
  • You can expand the chart extension by clicking on chadoop-vpc-monitoring-icon_ko, and you can download the graph as an expandable file. Choose a format to download your data.
    chadoop-vpc-monitoring5_ko

OS Dashboard

chadoop-vpc-monitoring6_ko

  • On the Monitoring page, select the server under the cluster instead of the cluster name. You can view the OS dashboard.
    • The OS dashboard collects data every minute.
    • Monitoring information are based on average values, and the frequency of queries varies, depending on the selected period type.
  • You can check the master node, edge node, and data node that make up the Cloud Hadoop cluster, as well as check individual CPU usage, load average, memory, disk I/O, disk usage, and network I/O indicators.

Was this article helpful?