--- title: "Submit Spark Scala Job" slug: "hadoop-vpc-16" tags: ["VPC", "Cloud Hadoop"] updated: 2026-09-17T09:17:45Z published: 2026-09-17T09:17:45Z canonical: "guide.ncloud-docs.com/hadoop-vpc-16" stale: true --- > ## Documentation Index > Fetch the complete documentation index at: https://guide.ncloud-docs.com/llms.txt > Use this file to discover all available pages before exploring further. # Submit Spark Scala Job Available in VPC This guide explains how to create a Spark Scala Job and submit it to a Cloud Hadoop cluster. ## Writing and compiling Scala code There are 2 ways to write a Spark application in Scala and package it into a jar: 1. [Use Scala in the terminal](/docs/en/hadoop-vpc-16#1-%ED%84%B0%EB%AF%B8%EB%84%90%EC%97%90%EC%84%9C-scala-%EC%82%AC%EC%9A%A9) 2. [Use the IntelliJ SBT plugin](/docs/en/hadoop-vpc-16#2-intellij-sbt-%ED%94%8C%EB%9F%AC%EA%B7%B8%EC%9D%B8-%EC%82%AC%EC%9A%A9) ### 1. Use Scala in the terminal The following is an example of writing Scala code that outputs `HelloScala` in the terminal, compiles it, and then packages it into a jar: #### Download Scala binary file Download [Scala binary](https://www.scala-lang.org/download/) and unzip it. If you are using homebrew on macOS, you can install it as follows: ``` brew install scala ``` #### Set environment variables Using the following commands, set the `SCALA_HOME` environment variable in the executable file (e.g., `.bashrc`) and add `$SCALA_HOME` to `PATH`. ``` export SCALA_HOME=/usr/local/share/scala export PATH=$PATH:$SCALA_HOME/ ``` #### Write Spark application To write a Spark application: 1. Run scala REPL to execute `scala`. ``` ❯ scala # Welcome to Scala version ... # Type in expressions to have them evaluated. # Type :help for more information. # scala> ``` 1. As shown below, write HelloWorld.scala class and save it. ``` object HelloWorld { def main(args: Array[String]): Unit = { println("Hello, world!") } } ``` ``` scala> :save HelloWorld.scala scala> :q ``` 1. Use the following command to compile with `scalac`: ``` ❯ scalac HelloWorld.scala ``` 1. Use the `ls` command to check the .class file to see if it was compiled successfully. ``` ❯ ls HelloWorld*.class HelloWorld$.class HelloWorld.class ``` #### Create jar file To create a jar file: Note To run the jar command, you must have Java SE and JRE installed. 1. Go to the directory containing the HelloWorld*.class file, and then package the class file into a jar using the following commands: ``` ❯ jar cvfe HelloWorld.jar HelloWorld HelloWorld*.class added manifest adding: HelloWorld$.class(in = 670) (out= 432)(deflated 35%) adding: HelloWorld.class(in = 645) (out= 524)(deflated 18%) ``` 1. Check in MANIFEST.MF whether the HelloWorld class in the packaged jar file is set as the application's entry point. ``` ❯ unzip -q -c HelloWorld.jar META-INF/MANIFEST.MF Manifest-Version: 1.0 Created-By: 1.8.0_181 (Oracle Corporation) Main-Class: HelloWorld # entry point ``` ### 2. Use the IntelliJ SBT plugin This guide explains how to set up an environment in IntelliJ for developing and debugging Spark applications and how to build a WordCount Job called Hello Scala using an example. - Build manager: SBT - Example writing environment: Windows OS, IntelliJ Ultimate 2022.1.4 #### Create project To create a project: Run IntelliJ. 1. Search for Scala in the **Plugins** menu on the left and install it. ![chadoop-4-6-002_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-002_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) 2. A restart is required to apply the plugin. Click **[Restart IDE]** to restart IntelliJ. ![chadoop-4-6-restart_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-restart_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) 3. Click on **Projects** in the left-hand menu on the Home screen, and then click **New Project**. ![chadoop-4-6-003_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-003_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) 4. As shown below, select **Scala** and **sbt**, then click **[Create]**. - Project name: Set as "WordCount." - Select Scala and sbt version. ![chadoop-4-6-004_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-004_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) 5. Check that the project has been created successfully. - When a project is created, you can check directory and file information with the following structure by default: - **.idea**: IntelliJ configuration file. - **project**: File used for compiling. - **src**: Source code. Most of the application code must be in src/main. src/test is a space for test scripts. - **target**: Location where the project is saved when compiled. - **build.sbt**: SBT configuration file. ![chadoop-4-6-006_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-006_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) #### Import SBT library For IntelliJ to recognize Spark code, you need to import the spark-core library and documentation. Note - Since the spark-core library is compatible with specific versions of Scala, be sure to check both the spark-core and Scala versions when importing the library. 1. Check the compatible Scala version along with the spark-core library and Artifact ID in the [mvn repository](https://mvnrepository.com/artifact/org.apache.spark/spark-core). ![chadoop-4-6-008_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-008_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) 2. Click **Target > build.sbt**, and then add the following content to the script window: ``` libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.0" ``` 1. Use the Build console to check whether the library has been successfully imported. ![chadoop-4-6-007_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-007_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) Note When importing libraries from SBT, use the following syntax: ``` Group Id %% Artifact Id % Revision ``` #### Write Spark application This section explains how to write an example WordCount application that counts the words contained in a sonnet using a Shakespeare sonnet text file (shakespeare.txt) as the dataset. 1. Download [shakespeare.txt](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/shakespeare.txt?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) and save the file in src/main/resources. - When running this application on a Cloud Hadoop cluster, upload the dataset to an S3 bucket or HDFS. ![chadoop-4-6-009_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-009_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) 2. Select **src** > **main** to expand the directory, right-click on the **scala** directory, and then click **New** > **Scala Class**. 3. Create a new class under WordCount/src/main/scala. - Kind: Object ![chadoop-4-6-010_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-010_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) 4. Write the following sample code in WordCount.scala and run it to check whether it has been set up correctly: ``` object WordCount { def main(args: Array[String]): Unit = { println("This is WordCount application") } } ``` 1. Ensure that the result is output correctly. ![chadoop-4-6-011_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-011_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) 2. Delete the sample code applied to WordCount.scala, then write out the code to count the words in the Shakespeare sonnet text file as follows: ``` import org.apache.spark.{SparkConf, SparkContext} object WordCount { def main(args: Array[String]) : Unit = { //Create a SparkContext to initialize Spark val conf = new SparkConf() conf.setMaster("local") conf.setAppName("Word Count") val sc = new SparkContext(conf) // Load the text into a Spark RDD, which is a distributed representation of each line of text val textFile = sc.textFile("src/main/resources/shakespeare.txt") //word count val counts = textFile.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.foreach(println) System.out.println("Total words: " + counts.count()) counts.saveAsTextFile("/tmp/sonnetWordCount") } } ``` Note Master URLs The Master URL varies depending on the Spark deployment environment. - Local (pseudo-cluster): `local`, `local[N]`, `local[*]` (Divided by the number of threads used; "*" uses as many threads as the maximum number of processors available in the JVM.) - Clustered Spark Standalone: `spark://host:port,host1:port1...` Spark on Hadoop YARN: `yarn` Spark on Apache Mesos: `mesos://` 1. Run WordCount.scala and check the output result. ![chadoop-4-6-012_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-012_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) #### Create jar file 1. After uploading the dataset to the Object Storage bucket, change the resource file path in the source code as follows: - To upload the dataset to HDFS, use `hdfs://` instead of `s3a://`. ``` // FROM conf.setMaster("local") // TO conf.setMaster("yarn-cluster") // FROM val textFile = sc.textFile("src/main/resources/shakespeare.txt") // TO val textFile = sc.textFile("s3a://deepdrive-hue/tmp/shakespeare.txt") // FROM counts.saveAsTextFile("/tmp/sonnetWordCount"); // TO counts.saveAsTextFile("s3a://deepdrive-hue/tmp/sonnetWordCount"); ``` Note This guide is based on Spark 1.6, so you must specify the `yarn-cluster` in `conf.setMaster()`. As of Spark 2, `yarn` may be used. 1. Use the following commands in the terminal console to package the updated code into a compiled jar so that it can be submitted to the Cloud Hadoop cluster: - The jar file includes the application code and all dependencies defined in build.sbt. - The `sbt package` command creates wordcount_2.11-0.1.jar under `$PROJECT_HOME/target/scala-2.11`. ``` > cd ~/IdeaProjects/WordCount # PROJECT_HOME > sbt package ``` ![chadoop-4-6-terminal_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-terminal_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) ## Submit Spark Job to Cloud Hadoop cluster This section explains how to deploy and submit a Spark application (.jar) written locally to Cloud Hadoop. ### Upload jar to Object Storage Copy shakespeare.txt and jar to the Object Storage bucket using Hue's S3 browser or the Object Storage console. - For more information about accessing and using Hue, see the [Using Hue](/docs/en/hadoop-vpc-18) guide. - For more information about the Object Storage bucket, see the [Object Storage overview](/docs/en/storage-storage-6-1) guide. ![chadoop-4-6-013_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-013_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) ### Submit job This section outlines 2 methods for submitting jar files to a cluster. Note The following properties must be properly set in spark-defaults.conf: ``` spark.hadoop.fs.s3a.access.key spark.hadoop.fs.s3a.endpoint kr.objectstorage.ncloud.com spark.hadoop.fs.s3a.secret.key ``` - Use Hue's Spark Submit Jar ![chadoop-4-6-014_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-014_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) - Submit from Spark client node 1. Run the `spark-submit` command as follows on the cluster node where the Spark client is installed: ``` spark-submit --class WordCount --master yarn-cluster --deploy-mode cluster s3a://deepdrive-hue/tmp/wordcount_2.11-0.1.jar ``` 1. Once the job has been completed, check that the results are stored in the specified bucket path as shown below. ![chadoop-4-6-015_ko](https://cdn.document360.io/6998976f-9d95-4df8-b847-d375892b92c2/Images/Documentation/chadoop-4-6-015_en.png?sv=2026-02-06&spr=https&st=2026-11-03T18%3A23%3A27Z&se=2026-11-03T19%3A23%3A27Z&sr=c&sp=r&sig=Rlzw2kjB0lWLccbjQ0jjNyDLVUdJYT3kU0Nw3pjNla0%3D) Note The deploy mode is determined by the location where the driver (SparkContext) runs in the deployment environment. The possible modes are as follows: - `client` (default): The driver runs on the machine where the Spark application is running. - `cluster`: The driver runs on a random node within the cluster. You can change this to the `--deploy-mode` CLI option of the `spark-submit` command, or to `spark.submit.deployMode` in the Spark property configuration.