Cython: Blend the Best of Python and C++ | SciPy 2015 Tutorial | Kurt Smith

[jefftam]

11:00 Cython Workflow (compile flow)
Cython source file (pyx) -> parse into c file -> compile into extension module (library, .so) along with other h and c/cpp files

15:08 Compiling with distutils
import cythonize that will take pyx and generate extension (.so) correctly

15:40 Comiling an extension module
how to invoke the CLI to build extension, do setup.py with build_ext parameter

18:30 pyximport: another way to interacting with cython code, simpler case, import cython just like plain python code, and it knows to recompile/use pre-cached code

20:00 iPython notebook: 3rd way to interact, and "%%cython"

21:30 Hello World exercise

22:00 1st approach bulding extension from pyx file (cythonize)
30:30 2nd appraoch pyximport
33:40 iPython notebook

38:20 Cython language: cdef keyword (gateway from C to python), can declare c-type variable, c functions in python
or Extension type (it looks like a python class, but you can use the cdef statement to define attributes. The attributes may be Python objects (either generic or of a particular extension type), or they may be of any C data type. So you can use extension types to wrap arbitrary C data structures and provide a Python-like interface to them.)

43:00 declaration of c-type variables (ctype takes precedences, for e.g. the float is ctype float), list will be a statically declared, so it will be effectively strongly-type variable, similarily for dictionary

45:20 def (python only, can be used in cython too), cdef (pure C function defined in python, take in and output ctype), cpdef (best of both world, can be call from python (wrapper) or cython, but the return and input types have to be compatible in both cython and python)

49:00 question and example of def vs cdef, where to call the cdef function (from normal python code you need a wrapper function)

51:50 if declare as cpdef, then now can be access from python directly (Since it is compiled into both pure C and python code (wrapper function to C function)), but it is more restrictive so you have can't use cpdef everywhere (for e.g. an C pointer type, then have to use cdef exclusively)

59:00 exercise - hamming, speed up (original, compile to cython, modify to cython c-type) speed up compare (b)

1:16:00 from general python (untype) -> python bytes, build -> some improvement but not much. Then modify to char* (c-style string) makes a huge improvement since the compiler now knows it is purely C-string-> much faster

1:25:20 Cython annotation, shows you where the most python-intensive region is

1:30:00 another example to show speeding up (Levenstein distance)

1:36:00 wrapping external C functions (existing C/cpp code to be used in python)
1. cdef extern from "(the header file.h)" (essentially the include in cython)
2. declare the interface of the C function (pick the function we need, no need for others), this function signature is checked against the C file to make sure it is correctly used
3. def a python function (wrapper) to call the C function
4. call that python wrapper from other python file/scope

1:44:20 wrapping external C structures, only declare what you need, even in struct
1:48:00 ctypedef (similar to declaring a new type of data, such as time_t is a long)
1:49:50 a python function returning a python tuple of c-type int
1:51:00 capsule object (like a smart pointer, opaque in python, pass around to extract in C layer later)

1:53:00 an exercise to wrap the Levenshtein.h/c file in levenshetin.pyx
1:55:20 how it is done in setup.py (use Extension (to wrap stuff use this, pyx the interface, c for the guts), compile it with cythonize), compile pyx file along with the c. the library (dynamic) .so file should be produced (elaborated more around 2:11:00)
BREAK
2:00:00 Cython in the wild (different big projects using Cython)
2:10:30 c generated by Cython, put into PyPI for installation (distribution), setup tools to build it for your package. Can also make extension module as binary (then one need to take care of the different platform). Cross-compilation possible.
2:16:50 cython -X to override compiler directive
2:18:30 continue example (Levenshtein)
2:21:10 const qualifier in cython (agnostic to it), but have to match data type
2:27:20 error handling in cython
2:28:30 Cython, Numpy (typed memory view, an Cython array datatype that is similar to Numpy but is developed in Cython)
2:31:30 "double[::1]" - declaration of memoryviews where elements are contiguous (::1 = contiguous 1D)
2:35:10 "int[:. :, :]" declaring a 3D typed memoryview. It is a pointer to a memory storage (buffer), either be a C-array/numpy or another memoryview
2:42:00 MemoryView example
3:00:00 MemoryView and Numby interaction (cdef d as MV, assign a numpy array to it, access d as MV)
3:02:20 declarators on top of the example, cython import inside cython code, compiler directive (boundscheck -> disable out-of-bound error check for peformance), or (wraparound checks if user uses negative indexing)
3:13:00 Cython compares to other alternatives
3:26:20 C++ wrapping, extension type (python class implemented on C level), __cinit__ (C levle initialization to have legitimate object, so cdef data declared would be initialized here, this is done before __init__ so it is ), __dealloc__ undo any thing done in __cinit__ to avoid memory leak
3:32:25 usually cdef members aren't visible in python but we can declare them accessible from python level with "public" qualifier
3:33:00 particle_extern.h class to be wrapped by Cython. (i) 'cdef extern from "<header>', cppclass <alias> <name to use in generated code(optional)>, (so that python class can be particle, and cpp class is _particle)
3:35:30 pointer to the C++ instance, __cinit__ and __dealloc__
3:37:20 Property, getter/setter (already deprecated, look in official documentation for updated version of Properties)
3:38:20 C++ distutil setup
3:39:30 pass-by-pointer, not the instance itself

The End

Hope it helps anyone who stumbles onto this video

[kcvinu]

I tested cython with python version 3.11. I just created a GUI library with cython and it seems a nice language and nice bridge between C and Python. But surprisingly "ctypes" ran faster than cython. Yes! I wrote the same GUI lib in Odin & C3. Then called the functions from python with ctypes. It was 2.5 times faster than cython. Both Odin & C3 are newer languages with manual memory management. Both are aiming to be alternatives to C. Due to this performance diff, I checked my cython code again and again. I realized that type conversion takes more time in Cython. But ctypes module in CPython 311 is marvelous.

[sandipdas]

Is nobody gonna talk about how swiftly he switched from Mac to Windows

[Neuroszima]

how exactly "Cython code" (it's not really the cython code that is running since he claims that it gets translated) can manage to outperform dedicated python.h C interface code? aren't you required to have that python.h interaction of translating python objects into c and vice versa? I can't seem to find the reason why Cython interpreter is better than then probably an optimal solution put into package from outside of python

[robinranabhat]

Updating this as I watch :

11:02 : Cython Workflow
21:17 : Hello World Exercise

[himanshumishra]

How did you switch between OS X and Windows so quick on a MacBook? I didn't get that.

[llama]

I can literally copy the code from the solution into the hamming_cython.pyx file, build it, time it and i always error out with " File "hamming_cython.pyx", line 9, in hamming_cython.hamming_loop
def hamming_loop(char *s0, char *s1):
TypeError: expected bytes, str found"

[giovannidigiannatale]

I have a problem I cannot get rid of.. if in my file "module.pyx" I import math, then the main script that calls module fails!!!! If module.pyx is without "import math" then everithing is fine!