How To OPTIMIZE Your Code!! #python #programming #coding

This short video explains what pre-allocation is and how it can be used to optimize code.

Background Music:
Attribution 4.0 International (CC BY 4.0)

[b]

Note: you’ll really only notice a performance improvement when working with large data.

[asad-ullahkhan]

Important to note two things:
1. This will give increasingly diminishing returns as the number gets higher as the Big-O is linear for both (amortized)
2. Allocation in a list uses a more sophisticated strategy revolving around doubling the size of the list each time which reduces the copying

This is good advice for optimization though. Always make sure to look at Big-O first and fix those issues before starting with these :). Cool short!

[vieilatome]

Python dev: *allocates a 30 million cells array*
Me, as an assembly dev: "I just wanna talk to him *loads shotgun* "

[pafnutiytheartist]

Now compare it to numpy array 🙃
The real way to optimize python code is to use native libraries

[moderneinstein]

Memory re-allocation occurs at exponential phases, meaning that the length of the new list doubled upon each allocation event .
Secondly, the initiated list object already has pre-allocated memory, just not as much as may be needed later.
What this means is that the memory reallocation will not occur upon the addition of evey element, or else, adding N elements to a list would occur at O (N^2 ) runtime complexity.
In Java, Vectors and Array lists could be extended with the ensureCapacity (int) function .

[lidwinguillermogascagarcia]

At first glance the upper method was less overwhelming, so I thought it was better.
As I listened to your explanation, I realized why is the "Pre-Allocation" actually a whole lot better.

[marklonergan]

... and we've come full circle:

Long ago - you must define the size up-front.

Some time after - dynamic sizing is made for those edge-case scenarios where they're needed.

Soon after - computers just keep getting faster so i'll just use all dynamic-size lists so i don't have to worry about it.

And then - languages made that default everything to dynamic-sizing.

Now - "let me explain pre-allocation"

[mirimjam]

way 3: my_list3 = list(range(30_000))

this is the result of my test:
way 1: appending => 2.67s
way 2: pre-allocation => 1.27s
way 3: builtin function => 0.36s

While pre-allocation might be a tad bit faster, just using the builtin functions correctly saves you way more time.
And if speed is really a concern, you should either be using numpy or just not using python at all

[cracc_baby]

bruh this brings back memories.. i used to code stuff like this in 2nd grade on the old apple II E, except print every number and make the classroom computer crash xD

[chadd_robertson]

This is really cool - been using python for years and never even considered pre-allocation.

[Checker]

It would be interisting to see a comparison to List Comprehension

[delofj]

Note that to create a list with all number from 0 to that_number you just can do "my_list = list(range(that_number))"

Actually in the for loop, "range(that_number)" just create a list with all number from 0 to that_number-1 and the "for i in" is going through all the numbers in this list

So range() is already there for us to do this, and that’s who we use it
We just need to transform it into a list object with list() because otherwise it would be a range object

[PKua]

I always get a feeling when someone tries to optimise the Python code on the level of “which type of loop is the fastest” that they chose the wrong tool for the job

[Root]

You not only save time, but energy and compute time, so optimizing programs with these concepts also has benefits several ways if this code is run often on a server or whatnot.

[almicc]

there's a sort of irony in seeing performance optimization in python

[RunningRunner]

In micropython this is very important in ording to make display update faster

[justinmoore]

Honestly, if we're going to use pre-allocation, then we're not really using the Python List for its advantage of being dynamic. Might as well use a numpy arange.

[kebab-case]

Usually, the array allocation is exponential, meaning if the array doesn't have enough space, it will reallocate 2X e.g. 1, 2, 4, 8, etc.

[snowflakeman]

The Chad way of doing things: my_list = list(range(30_000_000))

[Labicraft]

A few extras:
I replicated this but with a couple extra methods. The results:
Without preallocation (method 1 in this vid): 6.8 seconds
With preallocation: 5.73 seconds
List comprehension: 2.27 seconds
list(range(30_000_000)): 1.27 seconds
While the last one is kind of cheating, remember you have list comprehensions and the map() function which you can use instead of these, which are faster and more understandable (although looking for speed in pure python is generally not a good idea anyway...)