Hadoop Certification - CCA - Setup Spark 1.2.1 on Quickstart VM

Connect with me or follow me at

[itversity]

For any technical discussions or doubts, please use our forum - discuss.itversity.com
For practicing on state of the art big data cluster, please sign up on - labs.itversity.com
Lab is under free preview until 12/31/2016 and after that subscription
charges are 14.99$ per 31 days, 34.99$ per 93 days and 54.99$ per 185 days

[veerui.s]

hello Sir,
when i try to start spark shell 1.2.1 after configuration setup using bin/pyspark --master local getting below response.

16/02/24 19:24:08 INFO BlockManagerMaster: Registered BlockManager
16/02/24 19:24:11 INFO EventLoggingListener: Logging events to
must be set!
at

Could you please help me out in resolving this?

[prabhumuthiayan]

Hello Durga, I have been following you for taking up this certification. You havent yet taken the exam? Why i am not seeing any blogs explaining the CCA175 exam experience? Also i would like to know if all tasks can be done either in terminal or GUI? Or they will emphasize us to use anyone of these?

Planning to take up the exam in near future, please let me know if just following your this 84 videos & performing hands on on the same would be suffice.

Please answer!

Thanks,

[sayalimahajan]

do we have to perform all these steps while using big data labs ?

[neerajbhatt]

Thanks for video. I have a query. I set up Apache hadoop 2.6 . Please tell me where to get /etc/spark files or how to configure them manually.
I have no folder in my system named /etc/spark

[praveenv]

Hi Sir, I am using cloudera quickstart 5.7. Here also do i need to install spark 1.2.1?
In the certification exam, what version will be there in VM? For practice is it mandatory to install spark on my machine?

[tomhuffman]

I had a a problem where I needed to reload my VirtualBox VM. I reinstalled quickstart and was going through the setup and noticed I don't see the classpath.txt file. Also the spark-dafaults.conf doesn't have the content described in video #26 for spark 1.2.1 install. Is there a step I may have missed?

[muralirachakonda]

Hello Durga Sir, I am using cloudera VM 5.5. It has spark 1.5. My question is when I am using hive context sqlContext = sqlContext.sql("select * from departments"). I am getting error saying department table doesn't exists. What could be the reason? I am able to access file from hdfs but not the table. Please help.

[sureshpathak]

Hi Durga, I am planning for CCA 175.Bit confused about which version of spark-shell will be integrated in CCA 175 VM and is it ok If I practice with already integrated spark version spark 1.6.0.

[sampatht]

Thanks Durga for all your efforts. As already posted the spark version 1.2.x is not available for download and in v1.3.x spark-defaults.conf is empty(just comments) and classpath.txt file is not available. Not sure if I'm missing something here.
If possible could you upload v1.2.x to a common repository ?

[shivakonar]

Hi Durga,

I am running Cloudera Quickstart VM 5.4 . Under /etc/spark/conf there is no directory called yarn-conf and also there is no classpath.txt file. Is this normal? Will it cause any issue later on?

[shankarbokka]

Hi Durga,

I am using cloudera VM 5.5.0, comes with spark 1.5.0. I followed your video to setup spark 1.2.1, but I am getting "error: not found: value sc" error for scala and pyspark. I am not able to even launch spark-sql

*** I watched your video 3 times, to make sure I have everything right still, NO Luck ):

Thanks
Shankar

[prpramod]

For some reason i can not use the Latest Cloudera5.4.2 VM i am using Cloudera 4.1.1 VM there was no spark preinstalled, So I have downloaded the spark-1.2.1-bin-cdh4.tgz i unzipped and placed it in /usr/lib and i am able to launch spark-shell & pyspark of 1.2.1 .There is no /etc/spark/conf directory in cloudera 4.1.1 so How can i make it hadoop aware?.Could you please let me know?

[natesanyuvaraj]

Hi Durga,

I am trying to download spark 1.2.1 version from apache website. Only version 1.3.0 and above is available for download. Which version do i use for practice ?
Thanks.

[sureshtalks]

Spark already installed in quickstart VM why again we need to install it . is there any difference . I am using Qucik start VM 5.4

[abhishekkar]

Hi Durga/All, Which version of spark should I install in Cloudera VM to do practice for CCA 175? Kindly reply.

[ravitejaravipati]

Hi Sir,

I am using spark version 1.5.0 . I have successfully created soft link in spark. But when I run the sqlContext.sql("select * from my spark-shell is saying remote shuttingdown and also exiting from spark-shell. Can you please resolve this issue? Thanks.

[gadginir]

spark 1.2.1 is no more available for download on their site and the spark version available on my latest Cloudera VM is 1.6.0. How to go ahead? I believe I don't have any option than practising on whatever Spark deployed on my Cloudera VM.

[surajkarale]

can not able to find ls -ltr /etc/spark/conf

[sureshpathak]

Anybody recently appear for CCA 175 in January 2017, If you guys can tell me which version of Spark is integrated for exam it would be great help!