Python lists remember what you did to them

Two equal lists can take differrent amounts of memory!

Two Python lists of the same size can take different amounts of memory, even if they contain exactly the same elements. A larger list can even take less memory than a smaller list. How can this be? Lists appear to remember more than just their elements, and somehow contain some information of their history. In this video we investigate this phenomenon. We see what the system getsizeof function returns and how the internal structure of a list is represented in order to find the answer.

SUPPORT ME ⭐
---------------------------------------------------

Top patrons and donors: Jameson, Laura M, Dragos C, Vahnekie, John Martin, Casey G

BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------

CHAPTERS
---------------------------------------------------
0:00 Intro
1:01 Let's investigate
3:04 Under the hood of a list
5:06 Capacity
6:13 Why capacity != length
7:08 Geometric resizing
7:34 The answer in terms of capacities
9:19 Reality check
9:49 Thanks

[scoates]

As requested: I just worked on a micropython ("embedded python") project where it's the actual [re]allocation that took a LOT of time (or so it seems without great profiling tools). I wasn't running out of RAM, but it was the changing of capacity/allocation (in my case, in strings, not lists, but probably similar; maybe just not with spare capacity) that was the bottleneck. I like this deep-dive stuff. Even if it's not immediately practical, it's still useful to know how things work.

[jeroenritmeester]

This guy really went and took an elaborate deep dive for 10 minutes, only to debunk the relevance of his own video at the end. Well, at least you're humble!

It should be said that these are CPython specific internals and other implementations like PyPy or even subsequent versions of CPython can behave completely differently.

[voxelfusion]

list a: "I know what you did last runtime."

[megaing]

One interesting extra detail: For the last example with `for x in range` it probably doesn't reallocated at all, but only allocates once. This is what the `length_hint` special function is for. It's supposed to return a guess of how many elements the iterator returns

[callbettersaul]

So many people ask that question about the list size differences, but this king actually answered it.

[Ba_Dashi]

I love these deep dives, but I feel like you could add on your description which OS and Python interpreter version showed this behavior. Both so that we can compare and for future reference

[nio]

I think understanding the internals at some level (even if your details aren't 100% accurate) is *extremely* useful because having that vague idea makes it easier to recognize situations where you do need to dig in deeper.

It's good to always have an idea of what's happening, even if in most cases, to borrow from physics, your "cow" is perfectly round and frictionless.

Basically, when you're using an abstraction, you should at least understand the limits of that abstraction.

[comedyclub]

Well, in the past I was always complaining about shit in javascript like "[ ] == [ ] is false but ![ ] == [ ] is true" and "NaN === NaN is false" and praised python for its predictability. You prove me wrong every time you upload a video.

[davewagler]

With Python3.8.10 in Linux Mint, the sizes of the three original lists are, respectively, 80, 80, 88.

[hagen-p]

Thanks for the insight! (And yes, in embedded systems, or slow platforms, small differences in memory consumption/runtime can add up, so knowing about this can be very relevant.)

[andrewglick]

This was very interesting! Somewhat related, I have heard that Python dicts have faster lookup times than lists, but this doesn't really make sense to me. I would love to see your take on deconstructing the memory and speed differences between using lists and dicts. Great content as always!

[Vijwal]

"python lists remember what you did to them" Sounds like a threat

[thefekete]

As an embedded c programmer, well, I just statically allocate list sizes... Using malloc() is too much work. Not programming work, having to deal with code reviewers and their inherent fear of that function is a real pain.

[kaninchengaming-inactive-]

"Python lists remember what you did to them" rather sounds like a threat

[XCanG]

I just today speaking with my friend who code on Swift about python lists, generators and list comprehensions, what an ideal time to release a video!

[spenceabeen]

The code knows what you did
The code knows what you did
The code knows what you did

[Rudxain]

My guess for the number-literal "quirk" is that it makes a deep copy of each arbitrary-precision int and therefore each one has a different address and unique pointers to be stored. That explains why when using variables it has the same capacity as when explicitly repeating the element. Maybe because it uses Run Length Encoding to internally repeat the same pointer without making copies of it

[TuMadre]

y'know, the more i learn about python's inner workings, the more i begin to question it.
i like the idea of code being as simple to read as python does it, but seeing what it takes to get there makes me have second thoughts.

[XxyehezkelxX]

I know this was more on the informative side of videos, but honestly I was mostly laughing at your comments on screen lol.
In terms of the vid itself, pretty much what I expected in terms of memory management, but awesome vid nevertheless!