How to use read and write streams in node.js for BIG CSV files

An overview on how to use a read and write stream in node when you want to read in a very large csv file and process it quickly.

------------

[sameerbharadwaj]

Thank You! This helped a lot with writing 20mil records to solr.

[lucanthony]

Hello @vJunkie,

Thanks for the great content.

CAn you make a video using csv-parser & fs libraries to read & write csv files ?

[pranjaldoorwar]

This might be helpful to me i am trying to read csv data and insert it into mongodb first before parsing it

[PhuTran-nbme]

Hello Sir, I have a question, much appreciated If you can address my doubt. Do chunks come in sequence, in other words, does the read stream wait for the current chunk to finish the operations before it emit the "data" event to process the next chunk? You have a potentially long running iteration there and more importantly, each chunk is interdependent to each other as the last If statement is used to assign the last incomplete row to "unprocessed" variable for the next chunk to use.

[vpetryniak]

How to use Web stream API in node js to do the same thing?

[Tony.Nguyen]

What if I want to read data of users - User{age:20………height:7“} - and save it into a database, but that chunk of data miss some properties that a user have like height:7“ ? Could this happen if yes how to solve it

[martinmingosuarez]

there is a core module called readline that works with streams to read one line at a time, no need to write the logic yourself, otherwise great explanation and video. Cheers

[himaliamit]

if you show us the format of the csv file it would have made much more sense.

[ViktosB]

Hi, I really enjoy your videos. One of the most productive publisher on YouTube. But in that video the name "overWatermark" is highly misleading. I struggled understand the whole structure only because of that naming. This would be better if you use instead "canWrite". canWrite is true when we can write and false when the buffer is full. Then we stop -> !canWrite.
const canWrite = writeStream.write(`${i}, \n`);
if (!canWrite) {
await new Promise((resolve) => writeStream.once('drain', resolve));
}
}
I might be wrong of course.
Other than that - great job again. Continue watching you content 🙂