Scaling Web Scraping: Concurrency vs Parallelism

Do you want to extract more data in a shorter time? In this video, you’ll learn the difference between these two, what to use when, and some practical examples. Let’s begin.

To better understand concurrency, let’s think of multitasking. A Central Processing Unit (CPU or simply a processor) can work on only one task at a time. If you give it multiple tasks, such as playing a song and writing code, it simply switches between them.

As for parallelism, this is a type of computation in which multiple processors carry out many processes at the same time. In parallel programming, the code is written to utilize multiple CPU Cores. In this case, more than one process is actually executed in parallel.

You can use both concurrency and parallelism to speed up web scraping.

In this video, you’ll learn:
What is concurrency?
How to use concurrency to speed up scraping
What is parallelism?
How to use parallelism to speed up scraping
Concurrency vs. parallelism: the differences
When to use what?

Oxylabs is a world-leading proxy service provider, offering trustworthy proxy services for companies, in addition to 24/7 monitoring systems and a 102M+ proxy pool in nearly 200 locations. We help companies access big data essential for effective business operations.

© 2023 Oxylabs. All rights reserved.

[apah]

Good video. Is it possible and efficient to combine both approaches ?