Golang web scraper with Colly | Tutorial

Learn how to build a web scraper using Go and the colly package in this quick and easy tutorial.

If you found this video helpful, please consider liking it. Also subscribe if you'd like to see more of this type of content.

⌛ Timestamps
00:00 - Intro and project description
01:11 - Analyse the page we will be scraping
01:27 - Create Fact struct type
01:55 - Create slice of Fact
02:05 - Import colly package and create new collector
02:30 - Examine HTML structure of rhino facts page
02:45 - Use collector.OnHTML method to traverse the DOM
04:45 - Use collector.OnRequest to print when collector makes requests
05:11 - Print scraped data to the terminal as JSON
06:08 - Create writeJSON function to save rhino facts in a file

⭐ Go Packages

🔗 Useful Links

[iliatalebzade]

Awesome content
straight to the point explanation
and LOVELY thumbnails lol
thanks!

[jatindersinghaujla]

Thanks very useful video in very short time. It helps me to create web scraper in few minutes

[isharmaharjan]

This was so informative and understandable. Thank you!

[brimmed]

Thank you. I'm trying to learn go and i finally had a real world thing i need to code up so started looking here after finishing the basic tutorials. Normally i would just whip something up in python as the pages i'm scraping is maybe 10 pages and trying to imput that date into json format to import into another software. hopefully i can get it working this weekend.

[natanael]

you helped me a lot!
Greetings from Brazil!
-Nate

[hendrikbonthuys]

Thanks for this useful video. I look forward to seeing more Go tutorials from you in future. Best, HB

[barnabassolomon]

Why did u stop making content. Your videos are amazing.

[ammiekoenigstein]

Hello, the tutorials are especially great, first of all, thank you for sharing; 
Chinese users can't access YouTube channel in China due to network proxy problem, I want to share your tutorials to Chinese users, can I do that, ?
looking forward to your reply, thanks.

[AmitKumar-qvor]

mam can you say how to scrape data from a SPA site, which is built on vanillaJS and cant run on js disabled mode??? how to set headers and get the xht links from network tabs???

thank you mam

[perceptiondigitale]

Bonjour, merci beaucoup, est ce que la façon de faire a changé en 2022 ?

[utkarshagrawal]

Hey I am very new to Golang, just one question:-

Why did you put collector. visitor at the end? First, we are making the request to it, then we are parsing it with a collector.OnHTM? So it should be at the top?

Or it's just first, we are defining the structure of everything that what we need, what tag we need to parse and then we define the vision URL?

[abiodun]

Thank you, do you think you can create a course for beginners?

[thinkcrypto]

hello, how would i extract specified data from a website and store in Excel

[oceansblue]

Great Video, except i am running into errors that not too sure how to resolve.

1= .\main.go:25:4: undefined: log
collector.OnHTML(".factList li", func(element *colly.HTMLElement){
factID, err :=
if err != nil {
log.Println("Could not get id")

2= .\main.go:31:8: undefined: factId
factDesc := element.Text

fact := Fact{
ID: factId,
Description: factDesc,
}
Hope i can hear back from you.

[abdurashidgaybullayev]

do you have a video tutorial on authorization

[adithyarajendran]

helo mam, i have doubt in making a web scrapper and write it to csv file ..is there any way i can message you privetly..it would be of great help for me

[LEKIPE]

How to not overwrite the json file content each time