Python: How to open Big Data Files Buffering Tutorial

This tutorial video covers how to open big data files in Python using buffering. The idea here is to efficiently open files, or even to open files that are too large to be read into memory.

[tymothylim]

Thank you very much for this video. It was very educational and helps me see how to read in big files without impacting my RAM too much :)

[xtodazxzibit]

Nice, but the performance wont be enhanced unless you increase the buffer size for the file to be written, since f.write is writing line by line to the output

[coreyk]

Thanks for the vid. Do you ever use SQL or do you find python to be enough by itself?

[adelmahfooz]

Great Video..Watched more of your videos.. Examples and explanations are very Comprehensible. Subscribed definitely...

[jaimeandrescatano]

Great video, could you do something similar with a file that is not only text? let's say a .iso or a .mp4?

[sinamirmahmud]

Hi sir, i wonder if you could tell me where you acquired those tick data? Can you please help me with acquiring high resolution (5~10) seconds tick datas?

[ThinkGrowIndia]

Thanks for the great video :) I have a 450MB json file. I am not sure how to load it in my ipython notebook as it requires memory. Do you have any advice?

[wetbadger]

When I change the buffer size, the processing time doesn't change. So I tried this without the buffering argument and again got the same result. Are you sure this code does anything?

[jagannathsahu]

how to use this buffering for sending response from request means after making rest call using request library

[tilmahutli]

Hello, one question. You can unzip 5gb files?, you know any code in python?

[karanamkaushik]

Thanks that, this video is really useful to understand stuff. But I am really stuck on something. I actually have a very large CSV file rather than SQL Database and I would like to perform SQL GROUP BY operations on it. Could you suggest me something, please?

[dfrusdn]

Harrison look in to using pandas library. Think of it as a excel with more freedom in python. It is also BSD license so you can use it for your charting.

[vinitrinh]

Why is it that the output file is way bigger than the input file? *for exampleoutput 1*

[cocofortin]

I am doing EDA on Google cust revenue prediction
File is 24Gb .
Can you help me with that how to do that without system lag ..or any tutorial related to that.

[alexandrevalente]

IT should be more useful to show timing while starting the script.
In Linux or any other UNIX system I would do: time command parameters.
For Windows there are apparently similar commands: I found a post in Stack-Overflow mentioning timetit in Windows resource kit (2003) ... Is it ported to new Windows verisons ? Or with PowerShell it mentions "Measure-Command".
Here is the link:

[UpYours]

Does the buffer on the write file matter?

[jackdavidvillegas]

can i ask if you can help me how to open this big file data.evp file about 1.927.743
can you help me.

[__sagar_shah__]

Mine only works till 92 then it throws some error

Traceback (most recent call last):
File "d:\sagar0\python proj git\nfsu project\sentdex.py", line 4, in <module>
for line in f:
File "C:\Users\sagar\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input, self.errors, decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6044: character maps to <undefined>

Any help?

[kishorekumar]

how to use insert instead of replace in

[roadblock]

uh its saying the number of bytes to buffer isnt reconised