Set custom alerts

Available in VPC

Alerts generated within the Cloud Hadoop cluster can be forwarded according to user settings. This guide describes two methods for receiving alerts generated from Oozie workflows and Ambari within the Cloud Hadoop cluster by mail or message, depending on the user's settings.

1. How to directly build a local SMTP server to receive alerts in Oozie workflow by mail
2. How to receive alerts in Ambari as messages using NAVER Cloud SENS service

Mail Alerts using SMTP Server

In this guide, sendmail is installed and used as an SMTP server on the edge node of the Cloud Hadoop cluster to receive alerts within the Oozie workflow by mail. If you build a separate STMP server and use outside the cluster, you should set ACG that the cluster and SMTP server can communicate.

a. Build an SMTP server

1. Connect to the Cloud Hadoop (VPC) edge node via ssh. For instructions on how to access the edge node, see Access a cluster node with SSH.
2. Execute the following command below to check if sendmail is installed.

   rpm -qa |grep sendmail
   

3. If it is not installed, execute the command below to download sendmail.

   sudo yum install -y sendmail sendmail-cf
   

4. To change the sendmail-related settings, execute the command below to read the sendmai.mc file.

   cd /etc/mail
   sudo vi sendmail.mc
   

5. Change the contents of the sendmail.mc file as follows.

• Remove dnl in lines 56 and 57
• Change 127.0.0.1 in line 121 to 0.0.0.0

Before change

[52] dnl TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
[53] dnl define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
[118] DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl

After change

[52] TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
[53] define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
[118] DAEMON_OPTIONS(`Port=smtp,Addr=0.0.0.0, Name=MTA')dnl

6. Execute the following command below to create sendmail.mc file with sendmail.cf.

sudo sh -c "sudo m4 sendmail.mc > sendmail.cf" 

7. Execute the command below to add the IPs of the m-001 node and m-002 node where Oozie is running to the /etc/mail/access file.

sudo vi /etc/mail/access

Before change

# By default we allow relaying from localhost...
Connect:localhost.localdomain       RELAY
Connect:localhost                            RELAY
Connect:127.0.0.1                           RELAY

After change

# By default we allow relaying from localhost...
Connect:localhost.localdomain       RELAY
Connect:localhost                            RELAY
Connect:127.0.0.1                           RELAY
Connect:10.0.3.16                           RELAY
Connect:10.0.3.17                           RELAY

8. Run the following command to apply the access ip.

sudo makemap hash access < access

9. Run sendmail and test it to see if it works properly. After that, check the sent email.

sudo systemctl start sendmail
sudo yum -y install mailx
mail test123@navercorp.com
Subject: test
test
test
.
EOT

b. Set up Oozie email notification in Ambari

1. After accessing Ambari, click [Oozie] > [CONFIGS] > [ADVANCED] > [Custom oozie-site] > [Add Property ...] and add the 2 settings below, press [SAVE] to save, and then press [RESTART] to restart.

• Key: oozie.email.from.address, Value: Address to use as sender (e.g., oozie@localhost.com)
• Key: oozie.email.smtp.host, Value: IP of the server where the SMTP server is installed (In this guide, the SMTP server is installed on the edge node, so the IP of the edge node.)
  

c. Test Oozie email notification on Hue

In this guide, to reproduce the Oozie workflow's failure, we connected to Hue and executed a simple workflow like the one below. After performing the shell action, we arbitrarily set the conversion state to KILL. You can apply the same to complex workflows by referring to Using Sqoop and Hive in Hue Oozie Workflow.

1. Access [Hue] and click the left menu [Oozie editor] > [Workflow]. After that, add an arbitrary Shell Script to the workflow as shown below and click [Settings].
   
2. To reproduce the Oozie workflow's failure situation, set all transition states to KILL as shown below.
   
3. In the error message settings within the workflow, check EMAIL when an error occurs and enter the email address, subject, and content to receive.
   
4. Submitting the workflow fails after a while. After that, check if the mail was sent properly.

Message Alerts using SENS service

In this guide, NAVER Cloud's SENS service is used to implement the message alert feature for alerts in Ambari. For the description of SENS service, see Simple & Easy Notification Service Introduction
guide.

a. Create a SENS project

1. To use the SENS service, create a SENS project and query key values. For an explanation of this, see Common guide.
2. Register the sender number to use the SENS SMS service. For an explanation of this, see Calling number guide.

b. Ambari Alert Notifications Settings

1. After accessing Ambari, click [Alerts] > [ACTIONS] > [Manage Notifications] > [+] to register an alert target as shown below.
   
2. Connect to the Cloud Hadoop (VPC) edge node via ssh. For instructions on how to access the edge node, see Access a cluster node with SSH.
3. Execute the command below to create the Custom Alerts file in the /var/lib/ambari-server/resources/scripts path. This guide utilizes an API to sending requests message to the SENS service. For more information about the SENS service API, see SMS API guide. For more information about creating Ncloud API Signature, see Ncloud API guide.

sudo vi /var/lib/ambari-server/resources/scripts/alert.py

#!/usr/bin/env python

import sys
import urllib2
import json
import logging
from datetime import datetime

import hashlib
import hmac
import base64
import time

SENS_API_ENDPOINT = "https://sens.apigw.ntruss.com/sms/v2/services/{serviced issued from the SENS console}/messages"
LOG_PATH_FILE      = "/var/log/ambari-server/custom_notification.log"
TIMESTAMP = int(time.time() * 1000)
TIMESTAMP = str(TIMESTAMP)

def test_notification():
    definition_name = sys.argv[1]
    definition_label = sys.argv[2]
    service_name = sys.argv[3]
    alert_state = sys.argv[4]
    alert_text = sys.argv[5]
    send_message_from_sens(definition_name, definition_label, service_name, alert_state, alert_text)

def send_message_from_sens(definition_name, definition_label, service_name, alert_state, alert_text):
    try:
        log_data = str(TIMESTAMP + " -- " + 'Alert Dispatcher' + " -- " + definition_name + " -- " + definition_label + " -- " + service_name + " -- " + alert_state + " -- " + alert_text + " -- ")
        request_body = { "type":"SMS", "from":{calling number}, "content":log_data, "messages":[ { "to":{received number} } ] }

        headers={"Content-Type": "application/json; charset=utf-8", "x-ncp-apigw-timestamp": TIMESTAMP, "x-ncp-iam-access-key": {Access key ID issued from the portal or sub account}, "x-ncp-apigw-signature-v2": make_signature()}
        data = json.dumps(request_body, ensure_ascii=False)
        req = urllib2.Request(SENS_API_ENDPOINT, data, headers)
        f = urllib2.urlopen(req)
        response = f.read()
        f.close()

        file = open(LOG_PATH_FILE, "a+")
        file.write(log_data)
        file.write(response)
        file.close()
    except Exception as err:
        logging.warning(err)

def make_signature():
    access_key = {Access key ID issued from the portal or sub account}
    secret_key = {Secret key issued from the portal or sub account}
    secret_key = bytes(secret_key)

    method = "POST"
    uri = "/sms/v2/services/{serviced issued from the SENS console}/messages"

    message = method + " " + uri + "\n" + TIMESTAMP +  "\n" + access_key
    message = bytes(message)
    signingKey = base64.b64encode(hmac.new(secret_key, message, digestmod=hashlib.sha256).digest())
    return signingKey

if __name__ == '__main__':
    test_notification()

4. Execute the following command to create a log file for Ambari Alerts in the /var/log/ambari-server/ path.

sudo vi /var/log/ambari-server/custom_notification.log

5. Execute the command below to set permissions for the alert.py and custom_notification.log files.

sudo chmod -R 700 /var/lib/ambari-server/resources/scripts/alert.py
sudo chmod -R 766 /var/log/ambari-server/custom_notification.log

6. Execute the command below and add the path of the Alert Script file to the ambari.properties file.

sudo vi /etc/ambari-server/conf/ambari.properties

notification.dispatch.alert.script=/var/lib/ambari-server/resources/scripts/alert.py

7. Execute the command below to restart ambari-server.

sudo ambari-server restart

8. Connect to the SENS service console and check whether the message transmission has been processed properly.