Is web scraping legal? 🫢😳

Courses for Data Nerds
==================================

Build a Portfolio
==================================
Rebate Code: "LUKE"

Books for Data Nerds
==================================

Tech for Data Nerds
==================================

Social Media / Contact Me
======================

As a member of the Amazon, Coursera, Hostinger, and Parallels Affiliate Programs, I earn a commission from qualifying purchases on the links above. It costs you nothing but helps me with content creation.

#dataanalyst #datascience

[carlosalba]

Alternative Title: “Dude discovers TOS” lmao

[NicEeEe]

So companies won’t let us scrape their info but they’ll happily sell ours?

[kardz]

Alternative title: "data scientist tries to find job by collecting data(gone wrong)."

[JenOween]

Imagine if LinkedIn took phishing job posts and scam posts as seriously as they take scraping.

[lachee]

In Australia, if it is publically available it's fair game as long as it's not a detriment to the service and other users.

[tjdjultima]

I’ve done similar tasks professionally. Rotate your IPs, purchased leases to residential IPs work well, and you can set request headers to better imitate a “real” browser instead of whatever webdriver you’re using. A lot of times you can isolate the data call without having to render a bunch of images and just fire that as it’s own request through postman or whatever and then only get the json for every listing. LinkedIn is pretty notoriously tough to do thoroughly though.

[RidingWithGerdas]

Next time when you scrape, add some randomness to your process to look less like a bot

[MmeHyraelle]

And thats why i need an account to view linkedin now... Thanks.

[UlrichTonmoy]

MS be like only we are allowed to scrape public data and steal private one but not the other way around

[volterkeg]

It's not illegal, but it can to lead to some extremely overwhelming situations for the site if left unregulated. Whether or not a website is ok with it, you should time your bots. Don't run your bots with uncapped speed. Some websites even require you to follow some guidelines like one page per sec. The benefit of a bot should be automated consistency not speed.

[eliasb]

3 things:

- proxy pools
- rotate IP addresses
- randomize sleeps between requests

[Pod-Z]

Scraping actual useful stuff is prob my second favorite programming activity, forget the law do it anyway and if they want to come for you barricade yourself in a log cabin and let the k go

[ssherwood]

So when you scrape schedule the read to occur at a random time and with day spread. Also if you occasionally use the account to comment it will confuse their system

[Benexdrake]

I have my own Web Scraper, for Crunchyroll, Imdb, Pokémon, Pokémon Tcg, Magic Tcg and Honda Parts in C#, this project makes much fun.
I use Selenium and Httpagility for it.

[gorillaz]

When i built my first web scraper, i already noticed that it probably illegal becuase i need to bypass the "I'm not a robot" chapta.

[kizhissery]

No huge website allow scraping data, last thing to do is settimeout between each mouse movement but then scraping would take ages.
If I would scrape I might directly fetch backend REST api, providing headers and dynamically updating cookie every 12hrs, also huge apps like fb uses gql, so may not feasible or learn gql endpoint which provide entire data.(only happen if you know all the queries for gql)

[jithendra.k.sfirst_yr_b.sc]

I'm into this...


Did some illegal stuff, by being ignorant....😅

[peterbauer]

It shouldn’t be illegal, public information should be public information. But like... I get why LinkedIn doesnt want bots running rampant on their website

[sauce]

You should have made or bought dummy linked in accounts, used those as scrapers as well

[kexec.]

for the sake of your time, linkedin lost the battle since it was public data