Cloud Hadoop troubleshooting

Available in VPC

You might run into the following problems when using Cloud Hadoop. Find out causes and possible solutions.

Out of Memory (OOM) event

An OOM event caused the server to hang.

Cause

If system memory usage increases rapidly, the kernel's OOM Killer is triggered and terminates processes that consume large amounts of memory. If a kernel process is terminated as a result, the server may hang.

Solution

What to do when the server hangs
Contact Customer Support to request a VM reboot.

How to prevent server hangs

• Configure ping checks and process monitoring to periodically check the node status.
• Scale out edge nodes or master nodes that run jobs to distribute the load.
• Scale up edge nodes or master nodes that run jobs to distribute the load.

Change the node specifications to scale up the memory capacity.

The cluster does not operate normally after changing the configuration in Ambari.

Cause

When you change the configuration through Ambari, related settings may be unintentionally affected, causing the cluster to malfunction.

Solution

Ambari stores cluster configurations and version numbers in order of modification time. You can roll back to a version from before the cluster malfunctioned and restart the cluster.

The following instructions describe how to roll back the HDFS configuration to a previous version.

1. In Ambari, navigate to All Services > HDFS > Configs.
2. Click to compare it with the previous version.
   
3. Check the comparison between Version 2 (working configuration) and Version 3 (malfunctioning configuration).
   
4. Click Version 2, the version to apply.
5. Click [MAKE CURRENT] to roll back to the previous settings.
   
6. Check that a new version number has been assigned, and then click [RESTART].
   

Lost password for Ambari account

The password for the Ambari account has been forgotten.

Cause

The cluster administrator account or password entered when the cluster was created has been forgotten.

Solution

See Reset cluster administrator password to reset the password.

Cannot access Zeppelin Notebook

Zeppelin Notebook cannot be accessed while using a Spark cluster.

Cause

• Zeppelin Notebook is not running.
• SSH tunneling is configured incorrectly.

Solution

• Access the Ambari Web UI and check that Zeppelin Notebook is running properly.
  • See Web UI.
• If Zeppelin Notebook is running properly but cannot be accessed, check the tunneling settings.
  • See Access the Web UI with tunneling.

Report security vulnerabilities

The security.datanode.protocol.acl setting is set to *, which has been reported as a security vulnerability.

Cause

security.datanode.protocol.acl is a property key that specifies the users and groups that can access DataNodes. By default, "*" is set to allow access by all users, but you can change the permissions manually.

Solution

Starting with Cloud Hadoop 2.3, the security.datanode.protocol.acl setting is provided as hdfs hadoop.

For clusters created with an earlier version of Cloud Hadoop or clusters currently running with *, you can modify the rules for specifying users and groups that can access DataNodes as follows.

• Separate users and groups with a space ( ).
• Separate users in the user list and groups in the group list with commas (,).

Example:
The following rule allows the users alice and bob and the groups users and wheel.
alice,bob users,wheel

Users automatically created for each component in Cloud Hadoop are included in the hadoop group, so you can configure the setting as follows.
security.datanode.protocol.acl=hdfs,custom_user1,custom_user2 hadoop,custom_group1,custom_group2

Learning resources

We offer various materials for you to explore. To learn more about Cloud Hadoop, check out these helpful links:

• FAQs: Cloud Hadoop FAQs
• API guide: Cloud Hadoop API guide
• NAVER Cloud Platform's blog: Cloud Hadoop - Apache Iceberg and Apache NiFi available
• NAVER Cloud Platform's blog: Cloud Hadoop, a big data analytics platform made even more powerful with Notebook features!