<\/p>\n
We’ll first go through Spark installation on MacOS. Next we go through the Windows installation as well.<\/p>\n
<\/p>\n
1. Install Homebrew<\/strong><\/h4>\nHomebrew is a package manager for MacOS.\u00a0 You use it to install packages. So go ahead to install it using the command<\/p>\n
\/<\/span>bin\/<\/span>bash -<\/span>c \"$(curl -fsSL https:\/\/raw.githubusercontent.com\/Homebrew\/install\/master\/install.sh)\"<\/span>\r\n<\/pre>\nThis command would install XCode command line tools and Homebrew<\/p>\n
<\/p>\n
2. Install Java<\/strong><\/h4>\nSpark using Java tools for performing various operations. Therefore, you need to have Java installed in your system.<\/p>\n
Install Java using the command:<\/p>\n
brew install openjdk\r\n<\/pre>\n <\/p>\n
Check Java installation using the command:<\/p>\n
java --version<\/span><\/pre>\n <\/p>\n
<\/p>\n
3. Install Scala<\/strong><\/h4>\nSpark was actually written in Scala. So we would need Scala installed as well.<\/p>\n
brew install Scala<\/span>\r\n<\/pre>\n <\/p>\n
You can find Scala tutorials here<\/a><\/p>\n4. Install Apache Spark<\/strong><\/h4>\nYou now need to install Spark using this command<\/p>\n
brew install apache-spark<\/span><\/pre>\n <\/p>\n
This command would install Apache Spark. You can then launch the Spark shell using the command below:<\/p>\n
spark-shell<\/span><\/pre>\n <\/p>\n
5. Verify Apache Spark Installation<\/strong><\/h4>\nLet’s now validate Spark installation by writing a simple command that creates a Spark DataFrame<\/p>\n
import<\/span> spark.implicits._<\/span>\r\nval<\/span> data =<\/span> Seq<\/span>((<\/span>\"Java\"<\/span>,<\/span> \"Programmin\"<\/span>,<\/span> \"60800\"<\/span>),<\/span> (<\/span>\"Python\"<\/span>,<\/span> \"Analysis\"<\/span>,<\/span> \"150000\"<\/span>),<\/span> (<\/span>\"Scala\"<\/span>,<\/span> \"Coding\"<\/span>,<\/span> \"3500\"<\/span>))<\/span>\r\nval<\/span> df =<\/span> data.<\/span>toDF()<\/span> \r\ndf.<\/span>show()<\/span>\r\n<\/pre>\nNote: You can type the codes on line at a time<\/p>\n
The output screen is as shown below:<\/p>\n![\"Creating](\"https:\/\/kindsonthegenius.com\/apache-spark\/wp-content\/uploads\/2022\/05\/Screenshot-2022-05-31-at-22.13.29-e1654028167147-1024x390.png\")
Creating a DataFrame in Spark<\/figcaption><\/figure>\n <\/p>\n
6. Spark Installation on Window<\/strong><\/h4>\nFollow the steps below to install Apache Spark on Window.<\/p>\n
Step 1<\/strong> – Navigate the Apache Spark download page using this link – https:\/\/spark.apache.org\/downloads.html<\/a><\/p>\nStep 2<\/strong> – Download the tgz file<\/p>\nStep 3<\/strong> – Unzip the file into a local directory<\/p>\nStep 4<\/strong> – Create a directory path in your C drive and copy the Spark folder content into the directory<\/p>\nStep 5<\/strong> – Set the JAVA_HOME, SPARK_HOME, HADOOP_HOME an PATH environment variables. Use the data below:<\/p>\nJAVA_HOME = C:\\Program Files\\Java\\jdk18.0.1.1\r\nPATH = %PATH%;%JAVA_HOME%\r\n\r\nSPARK_HOME = C:\\apps\\opt\\spark-3.0.0-bin-hadoop2.7\r\nHADOOP_HOME = C:\\apps\\opt\\spark-3.0.0-bin-hadoop2.7\r\nPATH=%PATH%;%SPARK_HOME%\r\n<\/pre>\nNote<\/strong>: The directory path should be exactly the location on you local drive<\/p>\nStep 6<\/strong> – Download winutils.exe<\/strong> for Hadoop and copy it to the bin folder in the spark path. Get it from here https:\/\/github.com\/cdarlint\/winutils<\/a><\/p>\n
This command would install XCode command line tools and Homebrew<\/p>\n
<\/p>\n
2. Install Java<\/strong><\/h4>\nSpark using Java tools for performing various operations. Therefore, you need to have Java installed in your system.<\/p>\n
Install Java using the command:<\/p>\n
brew install openjdk\r\n<\/pre>\n <\/p>\n
Check Java installation using the command:<\/p>\n
java --version<\/span><\/pre>\n <\/p>\n
<\/p>\n
3. Install Scala<\/strong><\/h4>\nSpark was actually written in Scala. So we would need Scala installed as well.<\/p>\n
brew install Scala<\/span>\r\n<\/pre>\n <\/p>\n
You can find Scala tutorials here<\/a><\/p>\n4. Install Apache Spark<\/strong><\/h4>\nYou now need to install Spark using this command<\/p>\n
brew install apache-spark<\/span><\/pre>\n <\/p>\n
This command would install Apache Spark. You can then launch the Spark shell using the command below:<\/p>\n
spark-shell<\/span><\/pre>\n <\/p>\n
5. Verify Apache Spark Installation<\/strong><\/h4>\nLet’s now validate Spark installation by writing a simple command that creates a Spark DataFrame<\/p>\n
import<\/span> spark.implicits._<\/span>\r\nval<\/span> data =<\/span> Seq<\/span>((<\/span>\"Java\"<\/span>,<\/span> \"Programmin\"<\/span>,<\/span> \"60800\"<\/span>),<\/span> (<\/span>\"Python\"<\/span>,<\/span> \"Analysis\"<\/span>,<\/span> \"150000\"<\/span>),<\/span> (<\/span>\"Scala\"<\/span>,<\/span> \"Coding\"<\/span>,<\/span> \"3500\"<\/span>))<\/span>\r\nval<\/span> df =<\/span> data.<\/span>toDF()<\/span> \r\ndf.<\/span>show()<\/span>\r\n<\/pre>\nNote: You can type the codes on line at a time<\/p>\n
The output screen is as shown below:<\/p>\nCreating a DataFrame in Spark<\/figcaption><\/figure>\n <\/p>\n
6. Spark Installation on Window<\/strong><\/h4>\nFollow the steps below to install Apache Spark on Window.<\/p>\n
Step 1<\/strong> – Navigate the Apache Spark download page using this link – https:\/\/spark.apache.org\/downloads.html<\/a><\/p>\nStep 2<\/strong> – Download the tgz file<\/p>\nStep 3<\/strong> – Unzip the file into a local directory<\/p>\nStep 4<\/strong> – Create a directory path in your C drive and copy the Spark folder content into the directory<\/p>\nStep 5<\/strong> – Set the JAVA_HOME, SPARK_HOME, HADOOP_HOME an PATH environment variables. Use the data below:<\/p>\nJAVA_HOME = C:\\Program Files\\Java\\jdk18.0.1.1\r\nPATH = %PATH%;%JAVA_HOME%\r\n\r\nSPARK_HOME = C:\\apps\\opt\\spark-3.0.0-bin-hadoop2.7\r\nHADOOP_HOME = C:\\apps\\opt\\spark-3.0.0-bin-hadoop2.7\r\nPATH=%PATH%;%SPARK_HOME%\r\n<\/pre>\nNote<\/strong>: The directory path should be exactly the location on you local drive<\/p>\nStep 6<\/strong> – Download winutils.exe<\/strong> for Hadoop and copy it to the bin folder in the spark path. Get it from here https:\/\/github.com\/cdarlint\/winutils<\/a><\/p>\n
<\/p>\n
Check Java installation using the command:<\/p>\n
java --version<\/span><\/pre>\n <\/p>\n
<\/p>\n
3. Install Scala<\/strong><\/h4>\nSpark was actually written in Scala. So we would need Scala installed as well.<\/p>\n
brew install Scala<\/span>\r\n<\/pre>\n <\/p>\n
You can find Scala tutorials here<\/a><\/p>\n4. Install Apache Spark<\/strong><\/h4>\nYou now need to install Spark using this command<\/p>\n
brew install apache-spark<\/span><\/pre>\n <\/p>\n
This command would install Apache Spark. You can then launch the Spark shell using the command below:<\/p>\n
spark-shell<\/span><\/pre>\n <\/p>\n
5. Verify Apache Spark Installation<\/strong><\/h4>\nLet’s now validate Spark installation by writing a simple command that creates a Spark DataFrame<\/p>\n
import<\/span> spark.implicits._<\/span>\r\nval<\/span> data =<\/span> Seq<\/span>((<\/span>\"Java\"<\/span>,<\/span> \"Programmin\"<\/span>,<\/span> \"60800\"<\/span>),<\/span> (<\/span>\"Python\"<\/span>,<\/span> \"Analysis\"<\/span>,<\/span> \"150000\"<\/span>),<\/span> (<\/span>\"Scala\"<\/span>,<\/span> \"Coding\"<\/span>,<\/span> \"3500\"<\/span>))<\/span>\r\nval<\/span> df =<\/span> data.<\/span>toDF()<\/span> \r\ndf.<\/span>show()<\/span>\r\n<\/pre>\nNote: You can type the codes on line at a time<\/p>\n
The output screen is as shown below:<\/p>\nCreating a DataFrame in Spark<\/figcaption><\/figure>\n <\/p>\n
6. Spark Installation on Window<\/strong><\/h4>\nFollow the steps below to install Apache Spark on Window.<\/p>\n
Step 1<\/strong> – Navigate the Apache Spark download page using this link – https:\/\/spark.apache.org\/downloads.html<\/a><\/p>\nStep 2<\/strong> – Download the tgz file<\/p>\nStep 3<\/strong> – Unzip the file into a local directory<\/p>\nStep 4<\/strong> – Create a directory path in your C drive and copy the Spark folder content into the directory<\/p>\nStep 5<\/strong> – Set the JAVA_HOME, SPARK_HOME, HADOOP_HOME an PATH environment variables. Use the data below:<\/p>\nJAVA_HOME = C:\\Program Files\\Java\\jdk18.0.1.1\r\nPATH = %PATH%;%JAVA_HOME%\r\n\r\nSPARK_HOME = C:\\apps\\opt\\spark-3.0.0-bin-hadoop2.7\r\nHADOOP_HOME = C:\\apps\\opt\\spark-3.0.0-bin-hadoop2.7\r\nPATH=%PATH%;%SPARK_HOME%\r\n<\/pre>\nNote<\/strong>: The directory path should be exactly the location on you local drive<\/p>\nStep 6<\/strong> – Download winutils.exe<\/strong> for Hadoop and copy it to the bin folder in the spark path. Get it from here https:\/\/github.com\/cdarlint\/winutils<\/a><\/p>\n
![\"Spark](\"https:\/\/kindsonthegenius.com\/apache-spark\/wp-content\/uploads\/2022\/05\/Screenshot-2022-06-01-at-18.59.09-1024x620.png\")
<\/a>