Spark multinode environment setup on yarn

Spark Multinode environment setup in yarn mode

[ash]

One of best videos out there...thanks buddy. Looking for videos eagerly...Specially Flink and ML with R

[abhishekn]

Thank you so much sir. It's really helpful

[arjunchatterjee]

Super! loved the video. Very educative

[thilinadimonto]

Very helpful tutorial. Thank you very much

[ManuEalias]

Thank you for the video. It was really helpful. But after specifying m2 for secondary node, the secondary daemon didn't show in your m2. Also could you please specify the details about the spark-defaults.conf file

[jhendric]

You did a great job installing hdfs in 80% of your video. Then you setup a Spark cluster in the last 20% of the video. However, nowhere in your video did you cover running Spark on Yarn. You actually setup a standalone Spark cluster.

[arslanshakeel]

I have followed the steps and also my i can see master and worker on jps but no workers are shown in url,

[faisalnajmuddin]

my spark 3.0 doesn't have work folder, what should i do then, should i recreate the folder by my self

[sharmisthabhattacharjee]

Any reason for adding property in hadoop 2?
I think it is replaced with resourcemanager in v2.Please correct me if I am wrong.

[djibb.]

Thx for this article.
However i would like to know how to set up hdfs to enable all workers and master to share the same repository?
I installed a Spark-Cluster with 3 workers and i would like to save a dataframe along all workers. I created on each worker the repository ” home/data/”. It saves but if i read it back, i am geting “lost files error: file part XXXX does not exist”.
I tried also using parquet and using partitions by column y but i still get the same kind of error “file footer not found ”

Any suggestions please?
Thx

[houssemguidara]

One beginner's question: Isn't Yarn involved at any point in installing Hadoop and Spark? I have read somewhere that Yarn should be installed as an intermediate layer between Hadoop and Spark.

[hamzahabdulrahmanmjamel]

How can we increase the workers cores ?

[jatinprakash]

Hi, I am not able to get through ssh password. Which password is it asking for?

[bhanuprakashvattikuti]

Thank u for the tutorial, I sucked at the following point please help me
When i am trying hadoop dfsadmin -reports
i am encountering
DEPRECATED: use of this script to execute hdfs command is deprecated.
instead use hdfs command for it

report: call from master/192.168.100.2 to master:9000 failed on connection exception:
java.net.connectionException: connection refused;