Memory Efficient Dataframes from MASSIVE SQL Queries!!! #python #datascience #pandas #sql #coding

Have you ever wondered how to create memory-efficient pandas dataframes with python? In this short I demonstrate just how easy it is by using panda's chunksize argument.

Here is the source code:

# Libraries
import pandas as pd
import sqlalchemy
from sqlalchemy import create_engine

# Creating SQL Alchemy engine object

# Initiailizing connection to database

sql = "SELECT * FROM players"

# Demonstrating individual chunks
print(chunk)

# Function to query in chunks and concat results into a final dataframe
def chunkDF(sql, conn, chunksize):
return final_df

# Passing the results of the function to a dataframe
df = chunkDF(sql, conn, 100)

[naisanza]

But when you read one large file, you can't chunk 🥲