Set Custom Alerts

Available in VPC

Alerts generated within the Cloud Hadoop cluster can be forwarded according to user settings. This guide describes two methods for receiving alerts generated from Oozie workflows and Ambari within the Cloud Hadoop cluster by mail or message, depending on the user's settings.

1. How to directly build a local SMTP server to receive alerts in Oozie workflow by mail
2. How to receive alerts in Ambari as messages using NAVER Cloud SENS service

Mail Alerts using SMTP Server

In this guide, sendmail is installed and used as an SMTP server on the edge node of the Cloud Hadoop cluster to receive alerts within the Oozie workflow by mail. If you build a separate STMP server and use outside the cluster, you should set ACG that the cluster and SMTP server can communicate.

a. Build an SMTP server

1. Connect to the Cloud Hadoop (VPC) edge node via ssh. For instructions on how to access the edge node, see Connect cluster nodes vis SSH guide.
2. Execute the following command below to check if sendmail is installed.

   rpm -qa |grep sendmail
   

3. If it is not installed, execute the command below to download sendmail.

   sudo yum install -y sendmail sendmail-cf
   

4. To change the sendmail-related settings, execute the command below to read the sendmai.mc file.

   cd /etc/mail
   sudo vi sendmail.mc
   

5. Change the contents of the sendmail.mc file as follows.

• Remove dnl in lines 52 and 53
• Change 127.0.0.1 in line 118 to 0.0.0.0

<Before change>

[52] dnl TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
[53] dnl define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
[118] DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl

<After change>

[52] TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
[53] define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
[118] DAEMON_OPTIONS(`Port=smtp,Addr=0.0.0.0, Name=MTA')dnl

6. Execute the following command below to create sendmail.mc file with sendmail.cf.

sudo sh -c "sudo m4 sendmail.mc > sendmail.cf" 

7. Execute the command below to add the IPs of the m-001 node and m-002 node where Oozie is running to the /etc/mail/access file.

sudo vi /etc/mail/access

<Before change>

# By default we allow relaying from localhost...
Connect:localhost.localdomain       RELAY
Connect:localhost                            RELAY
Connect:127.0.0.1                           RELAY

<After change>

# By default we allow relaying from localhost...
Connect:localhost.localdomain       RELAY
Connect:localhost                            RELAY
Connect:127.0.0.1                           RELAY
Connect:10.0.3.16                           RELAY
Connect:10.0.3.17                           RELAY

8. Run the following command to apply the access ip.

sudo makemap hash access < access

9. Run sendmail and test it to see if it works properly. After that, please check the sent email.

sudo systemctl start sendmail
sudo yum -y install mailx
mail test123@navercorp.com
Subject: test
test
test
.
EOT

b. Set up Oozie Email Notification in Ambari

1. After connecting to Ambari, click [Oozie] > [CONFIGS] > [ADVANCED] > [Custom oozie-site] > [Add Property ...] add the two settings below, press the [SAVE] button to save , and then press the [RESTART] button to restart.

• Key: "oozie.email.from.address", Value: "Address to use as sender (e.g. oozie@localhost.com")
• Key: "oozie.email.smtp.host", Value: "IP of the server where the SMTP server is installed (In this guide, the SMTP server is installed on the edge node, so the IP of the edge node)
  

c. Test Oozie Email Notification on Hue

In this guide, to reproduce the Oozie workflow's failure, we connected to Hue and executed a simple workflow like the one below. After performing the shell action, we arbitrarily set the conversion state to KILL. You can apply the same to complex workflows by referring to the Sqoop and Hive usage guides in Hue Oozie Workflow.

1. Access [Hue] and click the left menu [Oozie editor] > [Workflow] button. After that, add an arbitrary Shell Script to the workflow as shown below and click the [Settings] button.
   
2. To reproduce the Oozie workflow's failure situation, set all transition states to KILL as shown below.
   
3. In the error message settings within the workflow, check EMAIL when an error occurs and enter the email address, subject, and content to receive.
   
4. Submitting the workflow fails after a while. After that, check if the mail was sent properly.

Message Alerts using SENS service

In this guide, NAVER Cloud's SENS service is used to implement the message alert function for alerts in Ambari. For the description of SENS service, see Simple & Easy Notification Service Introduction Guide.

a. Create a SENS project

1. To use the SENS service, create a SENS project and query key values. For an explanation of this, see Common Guide.
2. Please register the sender number to use the SENS SMS service. For an explanation of this, see Calling Number Guide.

b. Ambari Alert Notifications Settings

1. After connecting to Ambari, click [Alerts] > [ACTIONS] > [Manage Notifications] > [+] button to register an alert target as shown below.
   
2. Connect to the Cloud Hadoop (VPC) edge node via ssh. For instructions on how to access the edge node, see Connect cluster nodes vis SSH.
3. Execute the command below to create the Custom Alerts file in the /var/lib/ambari-server/resources/scripts path. This guide utilizes an API to sending requests message to the SENS service. For a description of the SENS service API, see SMS API Guide. For instructions on creating Ncloud API Signature, see Ncloud API Guide.

sudo vi /var/lib/ambari-server/resources/scripts/alert.py

#!/usr/bin/env python

import sys
import urllib2
import json
import logging
from datetime import datetime

import hashlib
import hmac
import base64
import time

SENS_API_ENDPOINT = "https://sens.apigw.ntruss.com/sms/v2/services/{serviced issued from the SENS console}/messages"
LOG_PATH_FILE      = "/var/log/ambari-server/custom_notification.log"
TIMESTAMP = int(time.time() * 1000)
TIMESTAMP = str(TIMESTAMP)

def test_notification():
    definition_name = sys.argv[1]
    definition_label = sys.argv[2]
    service_name = sys.argv[3]
    alert_state = sys.argv[4]
    alert_text = sys.argv[5]
    send_message_from_sens(definition_name, definition_label, service_name, alert_state, alert_text)

def send_message_from_sens(definition_name, definition_label, service_name, alert_state, alert_text):
    try:
        log_data = str(TIMESTAMP + " -- " + 'Alert Dispatcher' + " -- " + definition_name + " -- " + definition_label + " -- " + service_name + " -- " + alert_state + " -- " + alert_text + " -- ")
        request_body = { "type":"SMS", "from":{calling number}, "content":log_data, "messages":[ { "to":{received number} } ] }

        headers={"Content-Type": "application/json; charset=utf-8", "x-ncp-apigw-timestamp": TIMESTAMP, "x-ncp-iam-access-key": {issued from portal or sub account Access Key ID}, "x-ncp-apigw-signature-v2": make_signature()}
        data = json.dumps(request_body, ensure_ascii=False)
        req = urllib2.Request(SENS_API_ENDPOINT, data, headers)
        f = urllib2.urlopen(req)
        response = f.read()
        f.close()

        file = open(LOG_PATH_FILE, "a+")
        file.write(log_data)
        file.write(response)
        file.close()
    except Exception as err:
        logging.warning(err)

def make_signature():
    access_key = {This is the access key ID issued by the portal or Sub Account}
    secret_key = {This is the secret key issued by the portal or Sub Account}
    secret_key = bytes(secret_key)

    method = "POST"
    uri = "/sms/v2/services/{serviced issued from the SENS console}/messages"

    message = method + " " + uri + "\n" + TIMESTAMP +  "\n" + access_key
    message = bytes(message)
    signingKey = base64.b64encode(hmac.new(secret_key, message, digestmod=hashlib.sha256).digest())
    return signingKey

if __name__ == '__main__':
    test_notification()

4. Execute the following command to create a log file for Ambari Alerts in the /var/log/ambari-server/ path.

sudo vi /var/log/ambari-server/custom_notification.log

5. Execute the command below to set permissions for the alert.py and custom_notification.log files.

sudo chmod -R 700 /var/lib/ambari-server/resources/scripts/alert.py
sudo chmod -R 766 /var/log/ambari-server/custom_notification.log

6. Execute the command below and add the path of the Alert Script file to the ambari.properties file.

sudo vi /etc/ambari-server/conf/ambari.properties

notification.dispatch.alert.script=/var/lib/ambari-server/resources/scripts/alert.py

7. Execute the command below to restart ambari-server.

sudo ambari-server restart

8. Connect to the SENS service console and check whether the message transmission has been processed properly.