Available in VPC
The Monitoring service supports 2 types of dashboards, allowing you to view various monitoring information about Cloud Hadoop performance and history. The Monitoring service is included in NAVER Cloud Platform's Cloud Hadoop, so it can be used without any additional fees.
Monitoring provides the following types of dashboards:
- HADOOP Dashboard: Monitoring information related to Cloud Hadoop in operation
- OS Dashboard: Hardware and network information for each operating Cloud Hadoop server
The 2 dashboards provide information about Cloud Hadoop over the past 2 months, as well as hardware and network metrics for each server. Each dashboard consists of graph charts, and you can print specific graph charts or download them to your PC as files with various extensions to improve work efficiency.
Note
If a specific metric exceeds a threshold or meets a specific condition as a result of monitoring, it can be set to be recognized as an event and notify the user with a notification. For more information on setting up events and notifications, see the Cloud Hadoop monitoring with Cloud Insight guide.
Getting started
- From the NAVER Cloud Platform console's VPC environment, navigate to
> Services > Big Data & Analytics > Cloud Hadoop. - Click [Create cluster] to create a Cloud Hadoop cluster.
- For more information on creating a cluster, see the Getting started with Cloud Hadoop guide.
- Click Cloud Hadoop > Monitoring on the left.
- From the list of Cloud Hadoop clusters, click on the cluster you want to monitor.
Monitoring interface
The basics of using Monitoring are as follows:
- In the left component, you can select the currently running Cloud Hadoop clusters and the servers for each cluster.
- When you click on a cluster name, the HADOOP Dashboard is displayed in the right component, and when you click on a server below a cluster name, the OS Dashboard is displayed.
Check monitoring dashboard
The dashboards provided by Monitoring consist of several graphical charts. Users can intuitively view only the information they need by displaying it on the dashboard for each cluster. To use each dashboard:
HADOOP Dashboard
- Clicking on the cluster of interest from the list of Cloud Hadoop clusters on the left will display the HADOOP Dashboard as shown on the right.
- Data is collected on the HADOOP Dashboard every minute.
- Monitoring information is based on average values, and the view cycle varies depending on the selected type of time period.
- The metrics that can be viewed for each group are as follows:
| Group | Metric name | Unit | Description |
|---|---|---|---|
| Apps | apps_completed | num | number of applications submitted to YARN that have completed |
| apps_failed | num | number of applications submitted to YARN that have failed to complete | |
| apps_killed | num | number of applications submitted to YARN that have been killed | |
| apps_pending | num | number of applications submitted to YARN that are in a pending state | |
| apps_running | num | number of applications submitted to YARN that are running | |
| apps_submitted | num | number of applications submitted to YARN | |
| Blocks | corrupt_blocks | num | number of blocks that HDFS reports as corrupted |
| missing_blocks | num | number of blocks in which HDFS has no replicas | |
| pending_deletion_blocks | num | number of blocks marked for deletion | |
| pending_replication_blocks | num | status of block replication: blocks being replicated, age of replication requests, and unsuccessful replication requests | |
| under_replicated_blocks | num | number of blocks that need to be replicated one or more times | |
| Containers | allocated_container | num | number of resource containers allocated by the ResourceManager |
| pending_containers | num | number of containers in the queue that have not yet been allocated | |
| reserved_containers | num | number of containers reserved | |
| HDFS capacity(GB) | capacity_remaining_gb | GB | amount of remaining HDFS disk capacity |
| HDFS read/write(bytes) | hdfs_bytes_read | bytes | number of bytes read from HDFS |
| hdfs_bytes_written | bytes | number of bytes written to HDFS | |
| HDFS utilization(%) | hdfs_utilization | % | percentage of HDFS storage currently used |
| Memory(MB) | allocated_mb | MB | amount of memory allocated to the cluster |
| available_mb | MB | amount of memory available to be allocated | |
| reserved_mb | MB | amount of memory reserved | |
| total_mb | MB | total amount of memory in the cluster | |
| Nodes | num_live_data_nodes | num | number of data nodes that are receiving work from Hadoop |
| unhealthy_nodes | num | number of nodes available to MapReduce jobs marked in an UNHEALTHY state | |
| active_nodes | num | number of nodes presently running MapReduce tasks or jobs | |
| decommissioned_nodes | num | number of nodes allocated to MapReduce applications that have been marked in a DECOMMISSIONED state | |
| lost_nodes | num | number of nodes allocated to MapReduce that have been marked in a LOST state | |
| rebooted_nodes | num | number of nodes available to MapReduce that have been rebooted and marked in a REBOOTED state | |
| total_nodes | num | number of nodes presently available to MapReduce jobs | |
| V_cores | allocated_v_cores | num | number of core nodes working |
| pending_v_cores | num | number of core nodes waiting to be assigned | |
| Data transfers | total_load | num | total number of concurrent data transfers |
| YARN memory(%) | yarn_memory_available_percentage | % | percentage of remaining memory available to YARN (= available_mb / total_mb) |
- You can monitor changes in the metrics of your cluster in real time.
- This is how the metrics change when the number of cluster data nodes is reduced.
- This is how the metrics change when the number of cluster data nodes is reduced.
- Users can zoom in and out of the graph by hovering their mouse cursor over it as shown below, and after selecting the time period to view, they can view the metrics for that period on the dashboard.
- You can print charts by clicking
as shown below and download graphs as files with various extensions. Select the desired format to download the data.
OS Dashboard
- On the Monitoring page, select a server under a cluster rather than a cluster name. The OS Dashboard will be displayed.
- Data is collected on the OS Dashboard every minute.
- Monitoring information is based on average values, and the view cycle varies depending on the selected type of time period.
- You can check the master nodes, edge nodes, and data nodes that make up the Cloud Hadoop cluster, as well as the CPU Usage, LoadAverage, Memory, Disk I/O, Disk Usage, and Network I/O metrics for each.