These 5 Things Help my make better Web Scrapers

Let me share with you 5 useful tips for web scraping content. Extracting data from the web to analyze is a common need for modern businesses and finding the quickest and most efficient ways of doing so has become a useful skill.

This video shares 5 tips for helping you on your way to building better spiders and scrapers.

If you would like to support me and my work:

-------------------------------------

Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases

-------------------------------------

# timestamps
00:00 intro
00:21 Response Object
00:58 Custom Headers
02:10 Proxies
03:17 Network Tab
03:50 Render JS
04:42 Bonus
05:30 Outro

[xA]

Self-throttled webscraping avoids the need for proxies and can better simulate non-automated interaction with websites, especially in conjunction with a user-agent string for an actual web browser. The developer tools within modern web browsers can be used to determine if there are any "backdoor" URLs to retrieve raw data, often in JSON format, to avoid scraping and parsing HTML.

When webscraping be a good data citizen by tracking which data has been retrieved previously if possible so you can avoid requesting it next time. This is particuarly important for websites which are updated with new content on a frequent or regular basis.

[CodePhiles]

very informative, thank you for these tips

[sohaibrahman]

Your videos inspired me to enter in web scraping field.

[senthilsds]

Thank you for your guidance. I am able to do freelancing with your tutorials.

[Achiesamablog]

I need to get hired, where to go if my only skill is webscraping? Thanks

[dhaferfree]

Hello, you are doing a great job thanks so much, I have a question how we can get the first Href only ?

[droidumar]

Please make a video for handling csrf tokens

[obiterdictum]

I'm coding a scraper, trouble with parsing long list of keys to search box and capture results in file

[AkshayKaushik]

Hey John
I have learned all web scraping libraries from you.
Love your videos.
I have a question: Where to find good proxies? I am using free ones after testing there response time.
Any suggestions from your side, Which website provide more number of proxies as per bundle price?

[DenzelHooke]

Any good residential proxy you'd recomend?

[homerdus]

would be interesting to know if there is a script or command line tool on linux e.g. ubuntu that I can extract in text format similar as "network tab" on chrome.
I would like to lynx dump css element (response) or even just the headers/cookies data.
In Chrome I can : Developer console (F12) -> Network -> Name -> Response .... or -> Header -> Request URL / Cookie
Would there be similar possible using lynx (dump or wget) on ubuntu linux ?

[XO-cqgx]

Do you mean webscrap with JS is not effective?

[javierjdaza]

men i got a big problem en my work.
im need to automate a work flow in a .xbap application that just run in internet explorer.
i dont know what use for this, because if i right click in any place of the webapp doesnt show me "inspect element".

what do you recommend for me? i was thinking to use pywinauto but the app run in my web browser (ie) its a link.


thank you men, i love ur channel

[adnankattekaden]

How can i connect with you Can You Create An Telegram Group