Hadoop Basic Commands for beginners

Показать описание

This tutorial is for practicing hadoop basic commands. Below mentioned are some commands with instructions :

# Open a terminal window to the current working directory.
/home/hadoop
# 1. Print the Hadoop version
hadoop version
# 2. List the contents of the root directory in HDFS
hadoop fs -ls /
# 3. Report the amount of space used and available on currently mounted file system
hadoop fs -df hdfs:/
# 4. Count the number of directories, files and bytes under the paths that match the specified file pattern
hadoop fs -count hdfs:/
# 5. Run a DFS file system checking utility
hadoop fsck – /
# 6. Run a cluster balancing utility
hadoop balancer
# 7. Create a new directory
hadoop fs -mkdir /home/hadoop/work/sample

# 8. Add a sample text file from the local directory
# 9. List the contents of this new directory in HDFS.
hadoop fs -ls /home/hadoop/work/sample
# 10. Add the entire local directory called “tchadoop2” to the /home/hadoop/work/.
hadoop fs -put /home/hadoop/work/sample/tchadoop2 /home/hadoop/work/sample
# 11. Since /home/hadoop/work is your home directory in HDFS, any command that does not have an absolute path is interpreted as relative to that directory. The next command will therefore list your home directory, and should show the items you’ve just added there.
hadoop fs -ls
# 12. See how much space this directory occupies in HDFS.
hadoop fs -du -s -h /home/hadoop/work /pragim
# 13. Delete a file ‘customers’ from the “retail” directory.
hadoop fs -rm /home/hadoop/work /pragim/class2
# 14. Ensure this file is no longer in HDFS.
hadoop fs -ls /home/hadoop/work /pragim/class2
# 15. Delete all files from the “pragim” directory using a wildcard.
hadoop fs -/home/hadoop/work /pragim /*
# 16. To empty the trash
hadoop fs -expunge
# 17. Finally, remove the entire pragim directory and all of its contents in HDFS.
hadoop fs -rm -r /home/hadoop/work /pragim
# 18. List the hadoop directory again
hadoop fs -ls hadoop

# 22. cp is used to copy files between directories present in HDFS
hadoop fs -cp /home/hadoop/work /sample/*.txt /home/hadoop/work/sample/cp
# 23. ‘-get’ command can be used alternaively to ‘-copyToLocal’ command

# 25. Default file permissions are 666 in HDFS
# Use ‘-chmod’ command to change permissions of a file

# 26. Move a directory from one location to other
hadoop fs -mv hadoop apache_hadoop
# 27. Default replication factor to a file is 3.
# Use ‘-setrep’ command to change replication factor of a file
# 28. Copy a directory from one node in the cluster to another
# Use ‘-distcp’ command to copy,
# -overwrite option to overwrite in an existing files
# -update command to synchronize both directories
hadoop fs -distcp hdfs://namenodeA/apache_hadoop hdfs://namenodeB/hadoop
# 29. Command to make the name node leave safe mode
hadoop fs -expunge
sudo -u hdfs hdfs dfsadmin -safemode leave
# 30. List all the hadoop file system shell commands
hadoop fs
# 31. Last but not least, always ask for help!
hadoop fs -help