How To Design Robust Python Functions

In this video, I’m going to show you the main principles I use to write code that’s less likely to fail, with of course plenty of Python examples. Watch to find out how to avoid writing brittle functions in Python.

🔖 Chapters:
0:00 Intro
0:29 Do Not Check Type Constraints
11:09 Value Constraints
14:21 Optional Values
16:40 Returning None
19:41 Final thoughts
20:33 Outro

#arjancodes #softwaredesign #python

[brainstormbalaclava]

4:08
"what happens if somebody makes a subclass of a list?" (c)
it will not break these code, because `isinstance` will also return true for instance of subclass of list

[rupen]

I recently wrote a math function like that average function, so I'll leave some tips for overengineering the typehints on that kind of function:
You can use the Real abstract base class from the numbers package. "f(x: Real, y: Real) -> Real". That allows for different numeric classes, like Fraction or Decimal, or even user defined classes.


And you can also use generic overrides like: "f[T: Real](x: T, y: T) -> T" (if the inputs are of a type T, the output will be of type T). Use TypeVar if you don't have access to the python 3.12 generics syntax.

This is overkill for most people, you can just do f(x: int, y: int) -> int, or f(x: float, y: float) -> float. But maybe it'll be useful for someone reading the comments.

[michelfug]

I also prefer exceptions to returned null values, but one thing I don't like about that is the fact that the returned exceptions are not part of the signature of a function. So the type checking kinda falls apart there. You really have to read the docs (or the source code) to see what exceptions to look out for :(

[maleldil]

4:14 If someone makes a subclass of list, any variables of that type will still pass `isinstance(my_sublist, list)`.

5:43 You don't need to define those aliases. In the static type system, `int | float` is equivalent to `float`, and there's already a `Collection` type in `collections.abc` that does what you want (your function needs the input to be iterable - so you can use `sum` - and support `__len__`). There's a table in the `collections.abc` documentation that shows the different protocols and what they support. Using `Collection` will make this work with lists, tuples, sets, dicts (keys), etc. I get your point about how generic functions should be, and I find Iterable/Sequence/Collection to be general enough without much effort. It also helps when you want your collections to allow subtypes, as list[T] only accepts objects of type T (list is invariant because it's mutable), but Sequence[T] allows any subtype of T (Sequence is covariant because it's immutable).

19:10 If you're already using type hints, the type checker will give you an error if you try to use a potentially None value without checking first. I can be nicer to handle None values using try/catch everywhere.

[SkielCast]

In the first example, I believe the best way would be to use collections.abc.Collection and type numbers as Collection[float], knowing that all floats are treated as supertypes of int as per PEP 484. One step further will be to define a CanBeAdded(Protocol) with a definition to __add__ and type the function with numbers: Collection[CanBeAdded]. Not really useful in this case but with complex objects this is a typical case for dependency inversion

[milosbulatovic]

I like how the hairstyle changes for the commercial. 😄

[GBOAC]

2:16 maybe good to note that assert as is vulnerable to removal/silencing via optimisation. It isn't meant as an alternative to a manual if + raise condition.

[chijiokejoseph]

You are right. I used to return None values, too, when a function failed or when its arguments were wrong. But this is something that Rust kinda of corrected for me with its Option and Result enums. Returning None means there's only one way to fail, but if there are multiple ways to fail, returning a value and the error like in Go seemed like a good choice.

But I've come to actually use an approach similar to what you do here. I created a custom error class. Then, use it as the superclass of all errors in my project and then handle it usually in the main.py with a single try except that catches the base error class I created. If I need information about where the error occurred, I use the traceback module to print the exception's details.

[rupen]

I hard disagree with raising errors instead of returning None. Errors don't show up in the type signature. How can the caller be sure if your function will throw an error? If I am actually sure that, for example, the user with that id exists, I just use a helper "expect" function that narrows the type from "T | None" to "T" and throws/logs if it gets None.
user = expect(get_user(5))
or user = get_user(5); if user is None: ...

[thepablo]

3:16 here, you don't need to use float | int. With only float is sufficient, because integers are considered a subset of float

[edbolton]

List isn’t the only type of iterable though, hardcoding to just list is exchanging one type of brittleness for another. The original version is way more flexible, pass it a set or a tuple or a custom object that implements sum() and len() and it’ll just work. If you want runtime typechecking you should probably be using a language which has that as a first class concern as opposed to bolting it on manually.

[jamespeterson]

The thing with exceptions in my optinion is that python functions are missing some type of build in exception hinting. If there is no explicit doc-string for that function i could only guess and capture all the exceptions or follow all recursive subcalls this function make to find all the `raise` calls.

[Dyanosis]

When it comes to "get_user" returning null/None or throwing an error, it's definitely a design decision that you need to stand firm on. I like to air on the side of "return null by default and have a separate function for "nullsafe". So "get_user_nullsafe" would throw an Exception (raise an Exception in Python terms) and "get_user" would be allowed to return null. You could also do the inverse where "get_user" raises an Exception and "get_nullable_user" (or "get_user_nullable") would return null if no User.

The other difference is based on framework. For example - Spring Boot will assume, for you, that if "findBy" returns null, that an exception should be thrown and a 404 is automatically returned. You can override/disable this assumption, but that means that if you expect that something won't be there, you should have a "findByNullable" if you want to have a function that explicitly will not be expected to throw an exception.

[SkielCast]

For the zero division error at 14:15, I believe that rather than polluting the code everywhere with exceptions, Pydantic's BaseModel with validator would be a better fit and will by default validate the data the earliest possible

[ArturR-xudt]

Here is a proposition for next episode. Pydantic and complex nested json objects returned from a rest api(that you have no control over). How to properly build pydantic models for that. If the objects can be deeply nested and usually are optional. It's tragic to work with something like that - on every level checking for None... Are there any tips to tackle that? For now i use helper methods for the mostly accessed objects. I know you can set default values instead of optional on the model but that is not always a solution if you have to return that json data to a rest server after modification because it can have unexpected results.

[Michaelzagyt]

I came across BAML recently and I think it’s really interesting and there’s not too much on YouTube about it, at least not on my feed. It has a python implementation and I think it would make a good video.

[Rusterzinho]

Great video! I usually return T |None but this got me thinking about maybe raising exception a bit more?

[Andrumen]

My take on validation is that validation must always be done at the beginning of a program. Meaning that there is a configuration file, and some user values supplied. When the user supplies those values, they must be valid values, at least within the context of the desired input. Once the input has been validated, it should not need more validation because, at least in principle, the program(mer) is smart enough to do the processing based on that. So user input must be validated, but the routines as such should not validate and operate under the conditions that the input data is valid. That being said, every package should have unit tests with edge cases, that is what unit tests are meant to do, anyway.

[thomasbrothaler]

@arjancodes i recently struggled with log propagation, concerning how you should log especially in case of exceptions. I feel like sometimes a long stack trace is more confusing then helping. Would you mind making a video of this topic? Love your videos, thank you so much!

[tjanos]

You might want to check out the "Nothing" class from the typing module, especially when you often raise exceptions.