Working with larger-than-memory datasets with Polars

preview_player
Показать описание
Polars makes it easy to work with very large datasets even on a regular laptop. In this video I show you just how easy on a typical data science query.

This video comes from my Data Analysis with Polars course on Udemy - check out the course here with a half price discount:

Want to know more about Polars for high performance data science and ML? Then you can:
- subscribe to my channel
  1. Rho-Signal Data Analytics
Рекомендации по теме
Working with larger-than-memory 0:02:04

Working with larger-than-memory datasets with Polars

Python Pandas Tutorial 0:05:43

Python Pandas Tutorial 15. Handle Large Datasets In Pandas | Memory Optimization Tips For Pandas

Handling kaggle large 0:08:10

Handling kaggle large datasets on 16Gb RAM | CSV | Yashvi Patel

Working with Large 1:07:01

Working with Large Datasets as a Data Scientist (with Python)

Polars vs Spark 0:11:11

Polars vs Spark - Larger Than Memory Datasets

Peter Hoffmann - 0:38:33

Peter Hoffmann - Using Pandas and Dask to work with large columnar datasets in Apache Parquet

Read Giant Datasets 0:15:17

Read Giant Datasets Fast - 3 Tips For Better Data Science Skills

Using Pandas and 0:38:33

Using Pandas and Dask to work with large columnar datasets in Apache Parquet

handling large datasets 0:03:04

handling large datasets in python

Why and How 0:11:18

Why and How to use Dask (Python API) for Large Datasets ?

Will Polars replace 0:00:53

Will Polars replace Pandas for Data Science?

Using the {arrow} 0:18:18

Using the {arrow} and {duckdb} packages to wrangle medical datasets that are Larger than RAM

Tools for Working 0:31:21

Tools for Working with Large Datasets

This INCREDIBLE trick 0:12:54

This INCREDIBLE trick will speed up your data processes.

Algorithm Researcher explains 0:08:10

Algorithm Researcher explains how Pytorch Datasets and DataLoaders work

Vector databases are 0:03:22

Vector databases are so hot right now. WTF are they?

Handling Large Datasets 0:03:55

Handling Large Datasets in Pandas | #42 of 53: The Complete Pandas Course

Spatial Statistics for 1:18:26

Spatial Statistics for Huge Datasets and Best Practices

Financial Modeling on 0:46:24

Financial Modeling on Large, Streaming Datasets

Financial Modeling on 0:55:59

Financial Modeling on Large, Streaming Datasets

Managing Large Datasets 0:11:37

Managing Large Datasets

Try limiting rows 0:05:01

Try limiting rows when creating reporting for big data in Power BI

Very Large Datasets 0:10:59

Very Large Datasets with the GPU Data Frame

Data Visualization Fundamentals 0:07:56

Data Visualization Fundamentals - Ingesting large datasets

Комментарии
Автор

Great video. I still want that workstation tho...

efilson
Автор

Question for you all: what kind of problems could this kind of streaming solve for you?

rho-signaldataanalytics
Автор

Hi @Rho-Signal Data Analytics, what is the difference between polars.read_csv vs polars.read_csv_batched vs polars.scan_csv

vg_
Автор

hi, how to use sqlcontext on polars? i wonder how to use sql syntax to lazydataframe

SugengWahyudi
Автор

hi, cool proyect, this has suport for geodataframes ?

Автор

When streaming, what is the size of the batch?
When streaming, it means that it will handle any data size, but cost is just time?
---
I was playing with a 18 million dataset, try to inner join with some other dataset in a aws lambda at full memory.. lambda crashed. I will try again with this streaming..

davido
Автор

I've never - such SHT'(intel)

paulhetherington