Learn Python - Web scraping a private API - Questions from the comments episode 2

It's questions from the comments time! In this episode, we explore how to scrape a website by using its private API. You will learn about the requests library, functions, for loops and a little bit of pandas. If you ever wonder "Why should I learn programming" I hope this video helps!

[danielderma]

I am taking two of the most highly rated courses on udemy about scrape and they do not have half of your production, and teaching you are great. Success for the future!, Éxito.

[rafidrahman]

Helloooo, thank you so so much for literallyy making a video about my comment. Learnt so so much about python and API requests. You are one of the best teachers in youtube period. This certainly gave me a head start in my project and I can wait to complete it !

However, there is this one issue I am facing. According to calculations for 1 million job listing, there should be = 1 000 000 / 30 = 33 333 pages, given 1 page has 30 listing. But whenever I cross the 332 page mark and hover into pages like 400/500 I get the following message,

" Result window is too large, from + size must be less than or equal to: [10000] but was [1000020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.", "error":{"status":500}} "

This is a major problem I am facing. Given the genius you are, I am sure you will come up with an idea to fix this lol. Thanks in advance loll !

[tadashi_hamada]

Loved the enthusiasm when you were checking the website for data! This was a great course. Just what i needed. You got a new subscriber :)

[mrhide]

Amazing value. Wish you make vids like this all the time(make money with python)! Thank you!

[trentemollerules]

Great production and knowledge. Made it look real tight. Also like the way you explained it all. Subscribed!

[jcinthailand]

Hey!! yes, exporting to SQL would be a very nice thing to know

[Kioba]

This was really good content! I was able for follow along on my system and got the same results.

[thewilltejeda]

Is there any way to send a request in order to find potential acceptable parameters? (Once you've already found an useful api curl)

[KoldbyTheEye]

Awesome video! Subscribed! Quick question for you though. On the scraping project I'm working on, when I go to copy the cURL bash into the converter as you did, mine has a cookies section as well as the headers, params, data, and python request code. What do you think that means about the site I'm scraping? Should I delete the cookies section of the conversion? Cheers, Joe

[christoph]

Really good video on web scraping.
Please do more of that kind of videos.
If you could do the video of how to feed the data into a spl database that would be awesome - thx.

[richardfitzgerald]

Thank you for the great content! I'm wondering if there is any way to get this approach to not fail if there is javascript, or at least be accepted as a real and current browser. I'm aware that copying out the curl provides all the headers/user agents etc but some websites seem to still be able to tell that it is not a real browser, perhaps it is because javascript is not rendering properly and it gives it away? any thoughts would be much appreciated!

[golamrahman]

great lesson !!! It is really good for slow learner. can someone tell me why the data df has only one row and one column?

[DeepDiveinUniverse]

Great man !! I will be using this in near future 😀😀

[DeepDiveinUniverse]

One question it's may be related .. I do have user id and password for a website now I want to scrap the data from there ?? How to use this technique ( the one you showed in video) on scraping those data???
Probably if you will give some hints or direction that would be good if you will get time you may make a video may 😀😀 . Thanks 👍

[velvetcasuat]

Hey man ... what is the tool that you use to work with the data ?
Edit: It's jupyter notebook ... I figured it out watching the first episode of "making money with python" .

[alenjose]

shouldn't lat be first and then lng??

[arfankhan]

can use while loop for the page scraping automatically

[barrsido]

Hi, it is Barrsido from Reddit. I'm having another problem with my code. Most of the game is done, but for some reason the 'bal' variable does not update so during the betting, results, and scoring phases, it messes up on the second run. I put it back into the codeshare. Please msg me on reddit if you see this.

[dextm]

im high af and that intro made me laugh

[pupukdspadb]

May i have your email address sir? I need your guidance how to scrape webiste that required username password and entering captcha to login?