Python itertools - The key to mastering iteration

The key to iteration in Python

A key feature of Python is it's incredible support for lazy iteration. Defining and consuming lazy sequences in Python are easy. You can even define an infinite sequence like powers of 2 or prime numbers! To make working with iterables in Python even easier, Python provides some basic building blocks to help you compose iteration primitives in different ways, which is what the itertools module is for. This module comes with 21 different primitives (and many more recipes for how to combine them), which we go over in this video. Will you use them?

SUPPORT ME ⭐
---------------------------------------------------
Sign up on Patreon to get your donor role and early access to videos!

Feeling generous but don't have a Patreon? Donate via PayPal! (No sign up needed.)

Want to donate crypto? Check out the rest of my supported donations on my website!

Top patrons and donors: Laura M, Jameson, Dragos C, Vahnekie, Neel R, Matt R, Johan A, Casey G, Mark M, Mutual Information, Pi

BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------

CHAPTERS
---------------------------------------------------
0:00 Intro
1:47 An initial warning
2:30 Lookahead motivating multi-accumulate
17:22 multi_accumulate example
19:43 Thanks

[mCoding]

Errata:
0:00 filter(x) isn't valid; to filter out falsy values, use filter(None, x)
2:33 & 17:22: the example call in multi_accumulate's docstrings yields an additional (1, 1) at the beginning
4:40: the min and max should be arguments to itertools.accumulate, not list

[dcmayo]

I use the combinatorial ones a lot, often for testing things. I use all four of them.

I have 5 versions of networking device firmware, and I want to make sure they're all compatible with each other, so I iterate over all the pairs of

I have 10 devices networked together, and I want to test throughput for all possible paths. That's combinations, or permutations if I want to test both directions.

I have 5 versions of firmware and 3 models of devices, and I want to make sure all firmwares work on all devices, so I iterate over the product of firmwares and models

[QuantumHistorian]

The combinatorics functions are pretty damn useful IMO. Sure, it might "just" be for maths stuff, but the range of computational maths problems that involves them is vast! It's essentially the answer to the question "what are all the ways to put these inputs together?" which is super generic.

[Liam_The_Great]

"iterable" doesn't sound like a word anymore

[isodoubIet]

Combinations with replacement are very useful for implementing the statistical bootstrap.

[Labs]

at 4:41 I'm assuming "min" and "max" are passed to "accumulate" and not the "list" constructor

[guidodinello]

Ive found those functions useful while doing simple grid searching where you test combinations of hyperparameters to tune ml models.

[jacanchaplais]

I think there's a bug in your multi_accumulate example. You'd want the first argument of itertools.accumulate to be iterator, not iterable, right? For non exhaustable iterables, like range or sequences, you will end up counting the first element twice. Not an issue for min or max, but you'll definitely see an issue if you use the running sum example. iterator will have that initial value removed, though, so using that instead should solve the problem.

[traal]

I use batched all the time at work, for sysadmin type stuff, or API queries that slow down when you give too many search terms.

I’ve used the combinatoric functions to solve programming challenges, e.g. Advent of Code

[BenGroebe]

I used zip_longest recently. I needed to vertically display two lists side by side in a GUI, and there was actually very little chance they'd be the same length. zip_longest with fill="", then '\n'.join

[fabiolean]

I was part of a project that did analysis on proposed firewall rules. Since the rules could be subnet-to-subnet we used the combinatoric functions to ensure we analyzed every possible unique combination of source and destination addresses in the proposed rules.

[jevandezande]

filterfalse makes sense when you have a function defined elsewhere that you are using (e.g. filter(my_func, it)). To reverse the conditional would require writing a new function or adding a lambda (e.g. filterfalse(lambda x: not my_func(x), it)), which is ugly.

[Jagi]

I've used permutations and combinations in genomics. Probably pretty useful for big data in general.

[klmcwhirter]

I disagree "slightly" with your comment about using a for-loop instead of chaining the utility functions.

First, always test the performance of the algorithm which you are conceiving. In general, the built-in utilities (especially the ones implemented in C) almost always perform better than something written in Python. But, performance in Python is not always intuitive - especially when chaining things together where boxing / un-boxing occurs. Test!

I definitely agree with your statement about "showing off". I usually frame my comment about this as a maintainability problem. Think about the poor soul who will have to troubleshoot / enhance that part of the code 3 years from now. And if I wrote the code, inevitably that poor soul will be me.

My rule of thumb is ... if you have to write more lines of comments to explain what is going on than the actual lines of code themselves - something is wrong.

Thanks for another great video!

[blitzarsun]

I was hoping you would do itertools. Please do functools too!

[DrDeuteron]

filterfalse is necessary b/c sometimes your predicate is "bool", and lambda x: not x is blech.

[agbenfante]

“Try not to get caught up in showing off just how well you know the itertools library”

I feel attacked

[atrus]

Another advantage of product is that it can be a step in an iterator stream, whereas nested for loops can't be.

[FedericoSpada]

14:34 I've used it to print the content of a list inside a tkinter table: I zip_longest the list and the rows already present in the table; if the row item is None (not enough rows), I add a new row, if the list item is None (too many rows), I delete the row; finally if both are present, I update the row with the list item. I've found out that this is faster than always deleting all rows and then re-add them: it's better to update the existing ones.

[JakubYTb]

I was like: pff, itertools. I know those, what new can I learn.
But that last example... very clever :-) yet still just applying the basics. Nice.