Submit Spark Scala Job

Available in VPC

This guide explains how to create a Spark Scala Job and submit it to a Cloud Hadoop cluster.

Writing and compiling Scala code

There are 2 ways to write a Spark application in Scala and package it into a jar:

1. Use Scala in the terminal
2. Use the IntelliJ SBT plugin

1. Use Scala in the terminal

The following is an example of writing Scala code that outputs HelloScala in the terminal, compiles it, and then packages it into a jar:

Download Scala binary file

Download Scala binary and unzip it.

If you are using homebrew on macOS, you can install it as follows:

brew install scala

Set environment variables

Using the following commands, set the SCALA_HOME environment variable in the executable file (e.g., .bashrc) and add $SCALA_HOME to PATH.

export SCALA_HOME=/usr/local/share/scala
export PATH=$PATH:$SCALA_HOME/

Write Spark application

To write a Spark application:

1. Run scala REPL to execute scala.

❯ scala
# Welcome to Scala version ...
# Type in expressions to have them evaluated.
# Type :help for more information.
# scala>

2. As shown below, write HelloWorld.scala class and save it.

object HelloWorld {
  def main(args: Array[String]): Unit = {
      println("Hello, world!")
  }
}

scala> :save HelloWorld.scala
scala> :q

3. Use the following command to compile with scalac:

❯ scalac  HelloWorld.scala

4. Use the ls command to check the .class file to see if it was compiled successfully.

❯ ls HelloWorld*.class
HelloWorld$.class HelloWorld.class

Create jar file

To create a jar file:

1. Go to the directory containing the HelloWorld*.class file, and then package the class file into a jar using the following commands:

❯ jar cvfe HelloWorld.jar HelloWorld HelloWorld*.class
added manifest
adding: HelloWorld$.class(in = 670) (out= 432)(deflated 35%)
adding: HelloWorld.class(in = 645) (out= 524)(deflated 18%)

2. Check in MANIFEST.MF whether the HelloWorld class in the packaged jar file is set as the application's entry point.

❯ unzip -q -c HelloWorld.jar META-INF/MANIFEST.MF
Manifest-Version: 1.0
Created-By: 1.8.0_181 (Oracle Corporation)
Main-Class: HelloWorld # entry point

2. Use the IntelliJ SBT plugin

This guide explains how to set up an environment in IntelliJ for developing and debugging Spark applications and how to build a WordCount Job called Hello Scala using an example.

• Build manager: SBT
• Example writing environment: Windows OS, IntelliJ Ultimate 2022.1.4

Create project

To create a project:

Run IntelliJ.

1. Search for Scala in the Plugins menu on the left and install it.
   

2. A restart is required to apply the plugin. Click [Restart IDE] to restart IntelliJ.
   

3. Click on Projects in the left-hand menu on the Home screen, and then click New Project.
   

4. As shown below, select Scala and sbt, then click [Create].

   • Project name: Set as "WordCount."
   • Select Scala and sbt version.
     

5. Check that the project has been created successfully.

   • When a project is created, you can check directory and file information with the following structure by default:
     • .idea: IntelliJ configuration file.
     • project: File used for compiling.
     • src: Source code. Most of the application code must be in src/main. src/test is a space for test scripts.
     • target: Location where the project is saved when compiled.
     • build.sbt: SBT configuration file.
       

Import SBT library

For IntelliJ to recognize Spark code, you need to import the spark-core library and documentation.

1. Check the compatible Scala version along with the spark-core library and Artifact ID in the mvn repository.

   

2. Click Target > build.sbt, and then add the following content to the script window:

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.0"

3. Use the Build console to check whether the library has been successfully imported.
   

Group Id %% Artifact Id % Revision

Write Spark application

This section explains how to write an example WordCount application that counts the words contained in a sonnet using a Shakespeare sonnet text file (shakespeare.txt) as the dataset.

1. Download shakespeare.txt and save the file in src/main/resources.

   • When running this application on a Cloud Hadoop cluster, upload the dataset to an S3 bucket or HDFS.
     

2. Select src > main to expand the directory, right-click on the scala directory, and then click New > Scala Class.

3. Create a new class under WordCount/src/main/scala.

   • Kind: Object
     

4. Write the following sample code in WordCount.scala and run it to check whether it has been set up correctly:

object WordCount {
    def main(args: Array[String]): Unit = {
      println("This is WordCount application")
    }
}

5. Ensure that the result is output correctly.
   

6. Delete the sample code applied to WordCount.scala, then write out the code to count the words in the Shakespeare sonnet text file as follows:

import org.apache.spark.{SparkConf, SparkContext}

object WordCount {

def main(args: Array[String]) : Unit = {

  //Create a SparkContext to initialize Spark
  val conf = new SparkConf()
  conf.setMaster("local")
  conf.setAppName("Word Count")
  val sc = new SparkContext(conf)

  // Load the text into a Spark RDD, which is a distributed representation of each line of text
  val textFile = sc.textFile("src/main/resources/shakespeare.txt")

  //word count
  val counts = textFile.flatMap(line => line.split(" "))
    .map(word => (word, 1))
    .reduceByKey(_ + _)

  counts.foreach(println)
  System.out.println("Total words: " + counts.count())
  counts.saveAsTextFile("/tmp/sonnetWordCount")
}

}

7. Run WordCount.scala and check the output result.
   

Create jar file

1. After uploading the dataset to the Object Storage bucket, change the resource file path in the source code as follows:
   • To upload the dataset to HDFS, use hdfs:// instead of s3a://.

// FROM
conf.setMaster("local")
// TO
conf.setMaster("yarn-cluster")

// FROM
val textFile = sc.textFile("src/main/resources/shakespeare.txt")
// TO
val textFile = sc.textFile("s3a://deepdrive-hue/tmp/shakespeare.txt")

// FROM
counts.saveAsTextFile("/tmp/sonnetWordCount");
// TO
counts.saveAsTextFile("s3a://deepdrive-hue/tmp/sonnetWordCount");

2. Use the following commands in the terminal console to package the updated code into a compiled jar so that it can be submitted to the Cloud Hadoop cluster:
   • The jar file includes the application code and all dependencies defined in build.sbt.
   • The sbt package command creates wordcount_2.11-0.1.jar under $PROJECT_HOME/target/scala-2.11.

> cd ~/IdeaProjects/WordCount # PROJECT_HOME
> sbt package

Submit Spark Job to Cloud Hadoop cluster

This section explains how to deploy and submit a Spark application (.jar) written locally to Cloud Hadoop.

Upload jar to Object Storage

Copy shakespeare.txt and jar to the Object Storage bucket using Hue's S3 browser or the Object Storage console.

• For more information about accessing and using Hue, see the Using Hue guide.
• For more information about the Object Storage bucket, see the Object Storage overview guide.
  

Submit job

This section outlines 2 methods for submitting jar files to a cluster.

spark.hadoop.fs.s3a.access.key <OBJECT-STORAGE-ACCESS-KEY>
spark.hadoop.fs.s3a.endpoint   kr.objectstorage.ncloud.com
spark.hadoop.fs.s3a.secret.key <OBJECT-STORAGE-SECRET-KEY>

• Use Hue's Spark Submit Jar
  

• Submit from Spark client node

1. Run the spark-submit command as follows on the cluster node where the Spark client is installed:

spark-submit --class WordCount --master yarn-cluster --deploy-mode cluster s3a://deepdrive-hue/tmp/wordcount_2.11-0.1.jar

2. Once the job has been completed, check that the results are stored in the specified bucket path as shown below.