Pandas Limitations - Pandas vs Dask vs PySpark - DataMites Courses

Pandas data size limitation and other packages (Dask and PySpark) for large Data sets.

#PandasLimitations
#PandasvsDaskvsPySpark

[karmawangchuk]

Sir, thank you so much. I am following DataMites closely at heart and looking forward to seeing tutorials on Big Data Technologies.

[radheshyammohapatra]

Clearly explained and waiting for your pyspark series

[aliyananwar]

An awesome comparison video for a data science student like me about importance of big data.

[mrmuranga]

Thanks for the clear explanation.
I would like to practice more on dask and pyspark...would you be in a position in to recommend some tutorials..thanks

[tatatabletennis]

nice video sir...please make complete videos for Dask and Pyspark sir

[Navinneroth]

Excellent explanation .. thank you. Look forward for more.

[skogafoss]

Waiting for your pyspark playlist 😁 you are the best

[abhbk]

Thank you very much for explaining this.

[MrMLBson]

@DataMites help me understand why we have to stop using Dask around 200GB of data. Couldn't Dask handle terabytes of data if the script was run on a multi-node cluster that could scale up or down to meet data ingestion sizes on the fly?

[st-]

Thanks for the info, I have used pandas data frame for fetching and performing metric calculations on ~25 million records on daily basis.

Question : can I use pandas data frame even though I have 200+ gb data with more memory processing without using dask.

[petertreit]

Really awesome presentation! Any pointers between how we could convert dask dataframes to pyspark dataframes?

[Sharan_R]

Thank you but can we expect PySpark Series sooner ? It is very clear!

[dhanalakotamohan]

What is the system specification which you have considered for generating these benchmarks. Pandas - 1to5 GB for system with 32GB RAM?

[haneulkim]

Thanks! Dask does not upload all data into memory like pandas? I'm not fully understanding difference btwn pandas with chunksize and dask.

[lekithraj]

Can you tell me about the UI u r using for showing the programming. or is it an IDE? please explain

[ravulapallivenkatagurnadha]

Try to do more videos on big data and data scince

[purvidholakia]

Thanks for the info. I am reading the SQL table with the help of pandas dataframe but when the table is very large such as with 14, 320, 316 rows pandas is not working. How to connect to SQL with dask or pyspark.

[adityasadhukhan]

When are your pyspark videos be available on YouTube

[prabhakarvadakattu]

Sir, Can you please tell me the software you have used to record the video.

[coolmantej]

I think dask also supports distributed system for processing much like spark so why dask is not able to support more than 100 gb data or say 1 tb data just like spark???