Itamar Turner-Trauring - Measuring memory: Python memory profilers and when to use them

Measuring memory: Python memory profilers and when to use them
[EuroPython 2021 - Talk - 2021-07-30 - Optiver]
[Online]

By Itamar Turner-Trauring

If your program is using too much memory, it can crash, thrash, or just run more slowly. To reduce memory usage, you need to measure it somehow.

There are a number of tools to measure Python memory usage, however. Which one should you pick? This depends heavily on your use case.

In this talk you will learn about the different ways memory problems manifest in Python, the memory usage patterns of different application types, and which tools to use in each situation.

For web applications, the typical problem is memory leaks: memory that is never freed adds up if your program runs forever. That means you need tools to help you identify the source of memory leaks, whether in Python or in C extensions.

For data processing applications like data science or scientific computing, the problem is typically caused by processing large amounts of data. Here you need to find the peak memory usage, and what code what responsible for it.