How to Process Streams Sequentially with async/await in Node.js

---

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Node js / javascript: how to process streams sequentially with pipe and async/await when parsing csv and calling webservices with got?

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---

The Challenge

Consider the scenario where you're parsing a CSV file, and for each line, you need to call different web services using the got library. The challenge here lies in ensuring that each line is processed one at a time. While using the pipe method in streams, it can lead to unexpected behavior where the program continues executing without waiting for the previous async operation to complete.

Here’s what the initial implementation looked like:

[[See Video to Reveal this Text or Code Snippet]]

This code snippet demonstrated a major flaw: the processing was happening in parallel, which was contrary to our goal of sequential execution.

The Solution

To achieve the desired sequential processing, we need to utilize the for await ... of syntax, which allows us to treat our stream as an AsyncIterator. This will ensure that each line is processed one at a time, fully waiting for asynchronous operations to complete before moving to the next line.

Step-by-Step Implementation

Setup Your Environment: Ensure you have the necessary dependencies installed, primarily csv, fs, and got.

[[See Video to Reveal this Text or Code Snippet]]

[[See Video to Reveal this Text or Code Snippet]]

Key Points

Sequential Execution: The use of for await ... of allows each line of the CSV to be processed in the order it is read from the stream, ensuring that all previous asynchronous calls complete before the next iteration begins.

Understanding Async/Await: Note that async/await is built upon promises, allowing for cleaner and more readable asynchronous code than traditional promise-based syntax.

What You Should See

When you run the refactored code, the output should reflect the following sequence, indicating that each web service call is completed before proceeding to the next line in the CSV:

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

Do not shy away from exploring other libraries or approaches should your project require even more flexibility, but the solution provided here should serve you well in most scenarios involving CSV data processing.