Hadoop Streaming in Python, hadoop streaming tutorial

Hadoop Streaming in Python, hadoop streaming tutorial

AMAZON:

Hadoop Streaming in Python - hadoop streaming tutorial

This video describes steps about how to write MapRedue produce programs in Python and test it using Hadoop streaming. This is one of the popular methods of writing MR code in non-java languages.

[ksjirafe]

Excellent video. Thanks for sharing and making the complex concept simple.
The error in the command executed was - the option should be -reducer and not -reduce, hence the log says - invalid option -reduce.

[akaktak]

HI, what configuration files should be (mapred-site.xml, yarn-site.xml )?

[kirankoshy]

Is the mapper already sorted when tested in the hdfs?or how can we sort the output of the mapper to send as an input to reducer?

[yellankisanthan]

What is -k1, 1 in the sort command. I need to sort with composite keys and how do i do that?

[tura]

How I can run a diferent input format (for example, csv format) with python and streaming jar?

[tusharaggarwal]

I tried this in windows 10 powershell and it is not working. Any help the -k1 thing is giving me error.

[afx]

Thank you. This video helped me. Thank you so much.

[qhfl]

If I run Hadoop with YARN cluster, does Hadoop streaming split the job and operate in all nodes?

[nathhari]

Excellent Video, I appreciate your simple method of explaining a complex concept. Also the issue you got while executing hadoop streaming command first time was, you have used keyword reduce instead of reducer.
Thank You!

[MamtaSharma-esfc]

I am getting Error in 30L....invalid statement in reducer.py

[ahmadmaroof]

Thanks for your nice explanation.
Also, can you please tell me how can we use combiner in this and is it possible to use input file from local disk instead of hdfs system.

[bhishanpoudel]

Nice tutorial. But i got a small error. cat wordcount_in.txt | python mapper1.py | sort -k1, 1| python reducer.py gives me nothing, it runs good but outputs nothing. If possible can you share the code in github /svn/pastebin or something? It would be nice for other noobies like me.

[jaspreetsinghsaini]

how can we apply this python code for video streaming and storage, , , ????

[YKUZ]

Hi, nice tutorial video! I would like to ask you for a email contact. I would like to discuss with you about something, it is possible? Thanks Jan.