How to write a bash script for your reproducible data analysis pipeline (CC247)

Bash and Zsh are tremendously powerful tools for doing file operations from the command line. You can combine multiple bash programs in a script to automate your data analysis pipelines. In this Code Club, Pat uses bash scripts and a driver script to download files from a website and will share some tricks for using wget and tar within his bash scripts. The goal of this project is to highlight reproducible research practices using a number of tools. The specific output from this project will be a map-based visual that shows the level of drought across the globe.

#bash #wget #tar #R #Rstats

Support Riffomonas by becoming a Patreon member!

You can also find complete tutorials for learning R with the tidyverse using...

0:00 Introduction
1:09 Navigating NOAA's GHCN data portal
3:47 Creating an executable bash script
6:27 Creating a bash script to download data with wget
14:15 Creating command line arguments for bash scripts
17:43 Creating a driver script for pipeline
19:57 Add another step to pipeline with tar

[haraldurkarlsson]

Very interesting. I tend to simply just run curl in the Mac terminal rather than a script. But I only upload these occationally.

[leonelemiliolereboursnadal]

Thanks again!! you are the best man!, 🙌🙌

[musicspinner]

Great stuff. Looking forward to seeing snakemake in action. 👍🏽

[metameles]

-O lets you overwrite an existing file. Have a good day.

[mdeeaonetwothree]

This tutorial uses the Windows based Visual Studio to demonstrate Snakemake. It gives some great tips on reproducibility and gives good examples on interpreting and fixing error messages as part of a small project. Shows how to test rules individually. Does not demonstrate parallelisation. The creation of the DAG was interesting.

[markrandall]

I have been trying this and some URL exit without the URL in single quotes and some need single quote URL to exit. Gone to encapsulating all URL in single quotes.

[dubhsith]

Odd that someone would load Tidyverse and then use bash to script the download of files :)