C Programming: Loading large text files into memory

Learn how to load large text file into memory using dynamically-sized arrays. We'll be using fopen, fgets, malloc, and realloc.

[wicqedeyebot]

You are an amazing teacher, you help many people like me who is trying to learn the language. I myself just started to code C in a few months ago, I think it was in Jan 2017. C is a huge advantage over micro-controller programming and os programming compare to any high level languages today, and that's made me wanted to learn it. Sadly I had no teacher who could teach me to code since I am still a high-school student. These day I could find anything on the internet with the click of my fingers and I could find awesome people like you. Anyway Thank You for publishing the video on the youtube.

[ferassalous]

you saved me from jumping off a roof much love

[naboulsikhalid]

an other great tutor who really put the programmer's thinking into practice. Thanks a lot. u got both like and sab. I will follow your teaching religiously.

[haiderkhan]

This is better than any programming lecture I've been to

[hammerheadlemon]

Really enjoyed watching this. I'm learning C and learning a lot from your videos. Please keep (or return to) making C videos! Thanks a lot for your work.

[johnsteve]

thank you very much, sir, it's really helpful. Hope you've been well!

[fourJ_]

Wow, you are best teacher in the world!!

[flaviocampos]

2021 and this helped me a lot.
thank you!

[shlomopfeuffer]

Hi Berry, I think you made a small mistake 16:40, if you want to get rid of the '\n' character it should be buf[strlen(buf) - 1] = '\0';

btw great video

EDIT: You fixed it later in the video 33:08 my bad

[mohamedhany]

A great video, thanks.
I have definitely learned many concepts. Awesome :)

[notalexj]

This is a fantastic explanation thank you!

[dsandhya]

I guess u need to close the file pointer too????

[bonbonpony]

06:49 This part is a bit misleading, because there's no "grow" in memory allocation. There might be some data already allocated after the array, blocking it from growing, even if there's still plenty of free memory after that data. Therefore, such "growing" usually means reallocation of the entire array into a new location in memory that is big enough to accommodate the new size, and copying the old data into that new place that has still more room for new data. Unless you use some other data structure, e.g. a chunked array in which every chunk is fixed size, but it contains a pointer to the next chunk when there's more chunks (kinda like a linked list of arrays), or some tree-like multi-level indexing structure, like a B-tree.
13:45 Whoa, that would be a lot of copying back and forth :q Not to mention the O(n) complexity of calculating the string lengths.
16:22 No it isn't. It's 6. The new line character is part of the string. So in 16:50 you basically replace the null character with another null character, which is pointless.
17:16 Another length calculation? That's 2·O(n) already :P You could as well calculate it one time before that "trimming" part and reuse the length in it. Too bad that `fgets` doesn't return the length of the string that it read, because then `strlen` wouldn't be needed altogether. But how about checking the file pointer's position after each read and see how far it advanced? (which would be equal to the string length, without the need to recalculate it).

[tayyipadguzel]

Thanks for everything. It was very helpful for us.You are great man!

[redonebig]

do you have any tutorial for numbers, I have only been programming for three months and I am having a hard time :(

[panchiux]

How can I load a very large file into a 2d array. For example these 5 colum by 18000 rows.
Each colum and row consist of;
country_code country_name date total_cases new_cases
ABW Aruba 3/13/2020 2 2
ABW Aruba 3/14/2020 4 2
ABW Aruba 3/15/2020 12 8
ABW Aruba 3/15/2020 17 5

I also want to be able to modify the cases by getting the average of each month into a new colum written into a new file
I need help please.

the list goes on.

[videos]

I ´m trying to work with a huge text file (80mb of text) and i´m using the fopen to prepare the file for reading but.... the program always BROKES or freezes !! Using debbuger can i see the problem is with the fopen, when the program start fopen function, he freezes os crash.... Some people says the fopen have a limit, but it is up to 2gigabytes !!! ... Why my program brokes when i´m try to prepare it to read with fopen (fopen on "r" mode)?

Someone ?

[lizcarey]

Great video, your explanation is excellent. I do have a question though, I am just beginning to learn C so may be wrong, but should we free the malloc and realloc memory. If so how do we do that?

[kpratik]

What's the name of the IDE that you are using? Thanks

[xyugi]

A problem I found with your implementation is that if the last line doesn't end with a newline then the last string gets its last character cut. I fixed it by adding

if(buffer[strlen(buffer)] == '\n')
buffer[strlen(buffer)] = '\0';

before you cut the newline char

Also, you never free the memory or close the file, is this intentional?