How I run my Python scripts everyday in the cloud

➡ WEB

➡ COMMUNITY

➡ PROXIES

➡ HOSTING (Digital Ocean)

If you are new, welcome. I'm John, a self taught Python developer working in the web and data space. I specialize in data extraction and automation. If you like programming and web content as much as I do, you can subscribe for weekly content.

⚠ DISCLAIMER
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.

This video was sponsored by ProxyScrape.

[hugohoyzer]

super slick! been following you since a year now. your content and skill is evolving constantly. its hard to be able to keep up with you :D

[graczew]

Hehe finally get that "devops" for scrap the world. Good job as always. If I get this film year ago this will save me many hours. However this is perfect guide for someone who want to run scraper in cloud.

[guruware]

14:20 - about that 2>&1...
1 is the file descriptor for stdout (which is the default when using some command >> outfile)
descriptor 2 is stderrr where the errors (usually) go,
"usually" because some coders are too lazy, they output everything to stdout by using printf(...) instead of using fprintf(stderr, ...) for error output
by 2>&1 you are redirecting stderr to stdout and errors go to you log - messages AND errors
you could do something like 2 > error.log to get a separate error-logfile

[skillswithsid]

Loving the content John, keep em coming.

[milosZcr]

Cool. I think you (or your viewers) may also want to explore Functions product that D.O .offers. I see that you can also use cron-like features to run the python (or other languages) code.

Just like GCP (cloud functions) there is a free layer, so you can run many runs before they start to charge you. But, in this case, for the log file thing, I guess you would also have to buy D.O. storage to store that file there. I see its $5 / mo for 250GB, which would be good if all functions one manages need much more than the basic droplets 10GB.

I have to say the method of doing this a bit more manually like you are showing us, does offer much more flexibility about what you can do, and how to do it.

I am more a GCP guy, so I will sure also try your method, depending on the use case I have.

Thanks for the good info, as usual.

Cheers

[dinobaurier]

Don‘t know about zsh but usually you can just type „cd“ to navigate to the users home dir. Also, at apt install if it asks for Y/n the capital letter is the default. So you can just press Enter if the Y is capitalized.

[thghtfl]

thanks for the video mate! one question after watching this though: why didn’t you wrap all that up in a docker composer to set up your environment quickly?

[domitorid]

Use "tail" for watching logs. Instead of "cat". That way you would get a live log.

[majestif]

Many thanks for an amazing tutorial! It would be great to see how to push logs to a remote service like Sentry.

[azamatbagatov]

Great video! The videos you make are so much more transparent than other coding channels who just assume familiarity with this and that technology. (I’ve been watching for years and this is the first time I’ve commented!) Been running very similar workflows to this and Digital Ocean seems so much easier and cheaper than EC2, etc. Ansible is also great for automating the command lines, installs, etc. and gives full reproducibility if you want to set up multiple similar instances. One question: where do you recommend keeping your object store and/or SQL database for storing the scraped data? On something backed up by Digital Ocean, or your local machine, or a server in your homelab, or…? Cheers!

[oludelehalleluyah]

Wow... I've learnt something new that I'll use with another project

[oludelehalleluyah]

I've really learnt a lot from you...

Thank you

[MrZinchyk]

Question: if you frequently work with similar tasks, why not make a Bash script?

[Divyv]

Hey john, really nice video ! I was wondering if I could help you with more Quality Editing in your videos and also make a highly engaging Thumbnail and also help you with the overall youtube strategy and growth ! Pls let me know what do you think ?

[Archepter]

My man, cron jobs to run scripts exist since the 80's, way before the cloud. I have never heard anyone running daily scripts from their PC manually, wth?!?!?

[abiodun]

i use screen to manage multiple instances