Spark - Setting up Spark Dev Environment using SBT and Eclipse

Setting up Apache Spark Development Environment using SBT and Eclipse

Spark: Setting up Dev Environment (Spark : SBT + Eclipse)
--------------------------------------------------

STEP 1(Prereq): Install /Setup Software
Install Spark

STEP 2:
mkdir -p src/{main,test}/{java,resources,scala}
mkdir lib project target

STEP 3:
Run sbt, sbt eclipse

STEP 4:
Import project into Workspace
Add code for Spark

[sumanthanumula]

Thanks for the help. U saved lot of time..! To the point.

[kajalmishra]

[info] [launcher] getting org.scala-sbt sbt 1.8.0 (this may take some time)...
[error] [launcher] java.lang.RuntimeException: Couldn't find pwsh.exe or powershell.exe on path or in default system locations
[error] [launcher] could not retrieve sbt 1.8.0

Any clue why am I receiving this error on windows server 2016?

Even I have powershell.exe path setup in system variable.

[nithingangadharan]

Great video, I was stuck for last 7-8 days for various reasons. I followed steps shown in this video and bingo. Thanks Melvin

[geenafernandez]

Hi,

I got this error when i executed the program. Please help.

Error: Could not find or load main class SimpleApp

[jnehlyl]

thanks sir, please what about spark version 3.0.0 and scala 2.12.1.

[SpiritOfIndiaaa]

Thank you, when i try to import an existing sbt project I am getting below error : F:\BigData\oratocass>sbt eclipse
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Loading settings for project oratocass-build from assembly.sbt ...
[info] Loading project definition from F:\BigData\oratocass\project
[info] Loading settings for project oratocass from build.sbt ...
[info] Set current project to oratocass (in build file:/F:/BigData/oratocass/)
[error] Not a valid command: eclipse (similar: client, help, alias)
[error] Not a valid project ID: eclipse
[error] Expected ':'
[error] Not a valid key: eclipse (similar: deliver, licenses, clean)
[error] eclipse
[error]

[hushenglang]

great tutorial, thanks so much. very helpful and easy to follow

[mamadoubamb]

fine for Mac. that is a great video. Now I have my first running program on Eclipse; Thank you.

[ParthibanKarunanithi]

Great Video, I followed all the steps - I am stuck when i am running the application: Run --> Run as Scala Application.
Getting the following message - Error: Could not find or load main class - Can you please help on this?
However I was able to create jar file using sbt package.