How to Fetch and Parse CSV Files in Node.js Using CSV-Parser

---

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: NodeJs Fetch and parse CSV file using csv-parser

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---

In this guide, I will guide you through the process of fetching a CSV file from AWS S3 using axios and parsing it using the csv-parser library. This approach allows you to store the parsed data in your local database efficiently.

The Problem: Fetching and Parsing CSV Files

You might encounter the task of fetching a CSV file and parsing it only to find that the parsing utility you're using isn't extracting the data correctly. For instance, you might see the CSV file contents displayed in plain text instead of structured data.

In the question we are addressing, the developer successfully fetched a CSV file from a public S3 bucket using axios, but was unable to parse it correctly with csv-parser. This mainly occurred because csv-parser operates with streams rather than accepting plain text directly.

Understanding CSV-Parser

Firstly, it's important to understand how csv-parser works:

Solution: Correctly Fetching and Parsing a CSV File

To implement a successful solution, follow these steps:

Step 1: Fetching the CSV as a Stream

Instead of fetching the CSV data as plain text, you should instruct axios to fetch it as a stream. Modify your .get method like so:

[[See Video to Reveal this Text or Code Snippet]]

Step 2: Piping the Stream to csv-Parser

Once you have the response as a stream, the next step is to pipe that stream into the csv-parser. Here’s how to do that:

[[See Video to Reveal this Text or Code Snippet]]

Final Implementation

Bringing it all together, your code should look like this:

[[See Video to Reveal this Text or Code Snippet]]

Explanation of the Code

Importing Libraries: Start by importing the necessary libraries: csv-parser for parsing and axios for making HTTP requests.

The Async Function: Define your asynchronous function parseCSVFile, which receives the CSV file path.

Piping and Parsing: Pipe the fetched data directly into csv-parser, then handle the parsed data with .on('data'), .on('end'), and .on('error') to capture any errors that occur during parsing.

Conclusion

Remember, the key takeaway is to always pipe your streams into csv-parser as it is designed to work with them. This allows you to seamlessly convert raw CSV data into a structured format that can be utilized in your applications.

Happy coding!