Watch out for this (async) generator cleanup pitfall in Python

Watch out for this common generator pitfall.

What happens to a try/finally, with block, or async with block inside a generator if the generator isn't exhausted? Does the cleanup code still run? When and how does it run? In this video we take a look at the answers to these questions and learn how to avoid a common situation where cleanup code doesn't run when you want it to.

SUPPORT ME ⭐
---------------------------------------------------
Sign up on Patreon to get your donor role and early access to videos!

Feeling generous but don't have a Patreon? Donate via PayPal! (No sign up needed.)

Want to donate crypto? Check out the rest of my supported donations on my website!

Top patrons and donors: Laura M, Neel R, Dragos C, Jameson, Matt R, Pi, Vahnekie, Johan A, Mark M, Mutual Information

BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------

CHAPTERS
---------------------------------------------------
0:00 Intro
1:21 The cleanup question
5:30 The async case

[amarasa]

So basically, don't open and close resources in your generators, but give handles to those resources to your generators?
Like don't let your generator open a file itself, but give it a file handle?

[marckiezeender]

How about instead of "don't use generators" we just say "don't use 'with' or 'finally' blocks inside of generators"

Edit: what I mean is that you should avoid context and resource management inside of generators. If the generator needs access to, say, a file, then the generator's caller should be responsible for opening and closing that file.

[motbus]

You can also use a context manager which encloses both generator and resources

[danwellington]

Obligatory Rust comment that completely ignores the 30-odd years of design choices Python has to do its best to not break

[PeterZaitcev]

what *could* be done with the mentioned PEP is to add support for these iterclose methods but do not activate them in any default classes, instead provide a wrapper in contextlib that you can attach to your generator functions (or classes) which adds the missing method.

[lpt_]

python developers trying to not depend on the implementation details:

[idoben-yair]

The ACTUAL solution is to write your code such that it doesn't rely on exactly when resources get cleaned up, and if the exact time of the cleanup is important, then express it EXPLICITLY in the code by calling a cleanup method. RAII-style context managers are not the right tool when the order and timing of cleanup is important. Generators are not the issue here. The engineering approach is the issue.

To elaborate: RAII means Resource Acquisition Is Initialization. The converse, i.e., Reference Loss Is Resource Disposal, does NOT hold. This is an important implication of the reference counting style of resource management, unlike C++-style RAII where you can engineer your classes to enforce both RAII and RLIRD.

[xelaxander]

The resource existing at least as long as the generator seems like to me like the natural solution. Imagine if it wasn’t, I.e. the generator was accessible while the resource isn’t. Suddenly it’s even more cursed, since you can try to get an element from the generator, but it indeterminately fails to yield.

[aaliboyev]

As a conclusion "Wrap all your with statements with for loop inside into separate function which only holds this statement to guarantee gc"

[RogerValor]

Not sure how often I care about when resources are cleaned up, as long as they are cleaned up. But I guess this is good to know for the case when I encounter it. So thx!

[fyaa]

Why do you expect garbage collection to behave like RAII in other languages? The point is that you don't have to take care to clean it up, but you can hardly control when it happens.

[tchd]

No problem was ever solved by adding 'async' in front of it

[jenaf]

Id have the generator need the resource to be construced, and have the context manager outside of the generator

[black-snow]

And then there's the annoying bug with async generators and context vars. I feel like aGens are in such bad shape that I don't dare to use them.

[anon_y_mousse]

Do you think async in general is a good idea? Is it actually better than just using proper threads?

[tophat]

Sorry, beyond tired and ever so slightly drunk. Want to listen but need to sleep and tomorrow I won't remember this video exists.

[MrAlanCristhian]

That is the type of behavior that is detected with unit testing.

[SpeedingFlare]

Just use C and do if(error) goto cleanup; where cleanup cleans up anything that's non-null in whatever order you want.

For async, I don't know

[dirtdart]

I'm only up to 8:30 but why can't you just do a `async for x in contextlib.aclosing(gen()):` ?