Lambda in a Loop is a Code Smell

Watch out for smelly code!

A common mistake in Python is to use a lambda function inside of a loop. While it's not always a problem, it should set off your senses to take a second look when you see a lambda inside a loop. If the lambda references a loop variable, you might not realize that the way that closures (or cell variables) work in Python means the lambda doesn't store the latest value of a variable that was present when the lambda is created. In this video I walk through a simplified real-world-code example where I made this mistake myself, including how to fix it.

SUPPORT ME ⭐
---------------------------------------------------
Sign up on Patreon to get your donor role and early access to videos!

Feeling generous but don't have a Patreon? Donate via PayPal! (No sign up needed.)

Want to donate crypto? Check out the rest of my supported donations on my website!

Top patrons and donors: Jameson, Laura M, Dragos C, Vahnekie, Neel R, Matt R, Johan A, Casey G, Mark M, Mutual Information, Pi

BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------

CHAPTERS
---------------------------------------------------
0:00 Intro
0:12 What is a code smell?
0:43 Sponsoring myself!
1:11 Example setup
2:49 The symptoms
4:02 Why the lambda in a loop is bad
5:00 The partial fix
5:59 Exceptions to the rule
7:15 Prevent this before it ever happens
8:00 Thanks

[Erdnussflipshow]

Literally had this problem 2 days ago. Was trying to add callbacks to buttons, all buttons did the same thing. scratched my head for a while before I figured it out

Great video and explanation

[salamonandris]

You can do this with keyword binding: `lambda i=i: print(i)`, where i is the loop variable. Partial application is nice but not necessary.

[unpythonic]

What a wonderfully subtle bug! Great example, and I'll be on the lookout now.

[terra_creeper]

One big factor of this bug, at least to me, is how lambdas are percieved. If I used a named function in this example, it would seem obvious to me that the value is only looked up when the function is called. But for some reason a lambda doesn't feel like a normal function to me, which is why I almost always make this mistake and forget to use partial.

[bttlemastr]

So this is basically just capturing a variable by reference vs by value. In C++ you have to specify this when using a lambda expression, whereas in Python people probably aren't aware of this and the two ways of capturing require completely different syntax.

[Cookie-mvhg]

I haven't got this kind of error YET.
It's just interesting that every once in a while I come back to see the problems that I don't understand at first and recognized that you have make a video explaining that. I see this as a sign of my improvements

[NaifAlqahtani]

I LITERALLY had pylint call me out on this yesterday and couldn’t for the life of me figure out why. Even after googling and reading online. It just didnt make sense.

What a coincidence to see you upload a video on this exact topic one day later

[MilChamp]

I usually call myself self-taught coder. However, when I am pushed to think about it, I honestly feel that this channel has taught me more of what I know than anything or anyone else (other than perhaps StackOverflow). Every video is interesting, and every old video still somehow maintains relevance. Thanks James.

[opticalreticle]

this is definitely not a smelly python channel. keep up the high quality work

[kyletech]

I discovered this exact issue 2 weeks ago and thought I must have just missed the mCoding video covering it.

[PhilmannDark]

My workaround for this was lambda fut, l_task_it=task_id: ... which makes a copy of the value of task_id when the lambda is being created (and not when it's being evaluated). In the lambda, it's then safe to use l_task_id. But it looks a bit weird and there is a chance that a coworker will think this was a mistake and delete the "useless" lambda parameter.

[mika]

This looks like the same problem we had in javascript before the addition of "let" and "const", filling the arguments at lambda creation is a workaround to a problem created by the language that should not exist.

[mayureshkedari]

I would love to see more content in the form of series on
1. threading vs multiprocessing
2. distributed computing in python using ray framework or its equivalent

[shahaffrsshahaffrs]

Once again, great content and a very deep understanding of python. I had this problem years ago, and I couldn't find any solution to this "lambda function in loop" problem. At the and I finally understood the problem and how to solve it. after years of watching your videos and gaining so much from your content - thank you for making the process of learning to program better and easier!

[yxh]

Great video!
I also appreciate you breaking away from overuse of memes and keeping the 💯 content unsullied

[abadhaiku]

OH, so _that's_ why the Java compiler says that "values used in lambda expressions must be final or effectively-final" and makes you copy it to a new variable before using it.

[pants]

This is why I like C++ lambdas. They let you choose whether to capture by value or by reference, so this probably wouldn't be a problem in the first place and it certainly wouldn't need a library to fix.

[asdfghyter]

i feel like this is as much of a python footgun as a code smell. it’s a bad choice in the design of the language that runs counter to how people would expect it to work. in many other languages you would not have this issue and calling lambdas in a loop is perfectly fine.

for example, in rust with its explicitness about lifetime would make this bug impossible, since you are not allowed to have mutable aliasing. and in many other languages the loop variable is fresh and immutable, so one iteration of the loop can never affect another.

i believe that if you loop using a lambda, e.g. a forEach loop in js, it’s fine in most situations, except if the lambda catches a variable from an outer scope, which is modified in the loop, in which case you’ll run into the same issue again.

[fionnbracken]

Golang has same issue but they have just accepted a proposal to change the behaviour of lambdas in loops lol.

[dwarftoad]

Another way to understand this problem. Consider replacing the lambda with a reference to a normal function defined elsewhere (with the exact same parameters as the lambda!) and see what would happen -- and how you would fix it -- you must either add an argument, i.e. the value is supplied when called, or do some kind of partial application of the function that captures a copy of the value that is used later when invoked.

(e.g. use a function taking the task_id and returns the lambda `def return lambda fut: print_task_completed(fut, task_id) ..., or the equivalent `add_done_callback( (lambda id: lambda fut: print_task_completed(fut, id))(task_id) )` or the IIFE pattern mentioned.)