Between pyodide, pyo3, rust-cpython, and rustpython, I think Pyo3 is the best way to drop in rust in a python project for a speed up, if that is your goal. Some of the demos show using python from rust, but to me the biggest feature is without a doubt compiling rust code to native python modules. I'm using it to speed up image manipulation backed by numpy arrays.
There’s a setuptools rust [0] extension package that can be used to hook the compilation of the rust into the wheel building or install from source. Maturin [1] seems to be regarded as the new and improved solution for this, but I found that it’s angled toward the using python from rust.
There’s also the rust numpy [2] package by the same org which is fantastic in that it lets you pass a numpy matrix to a native method written in rust and convert it to the rust equivalent data structure, perform whatever transformation you want (in parallel using rayon [3]), and return the array. When building for release, I was seeing speed ups of 100x over numpy on the most matrix mathable function imaginable, and numpy is no joke.
I think there is a lot of potential for these two ecosystems together. If there’s not a python package for something, there’s probably a rust crate.
If anyone is interested the python package that I'm building with some rust backend, its called pyrogis [4] for making custom image manipulations through numpy arrays.
> Between pyodide, pyo3, rust-cpython, and rustpython, I think Pyo3 is the best way to drop in rust in a python project for a speed up, if that is your goal. Some of the demos show using python from rust, but to me the biggest feature is without a doubt compiling rust code to native python modules. I'm using it to speed up image manipulation backed by numpy arrays.
> There’s a setuptools rust [0] extension package that can be used to hook the compilation of the rust into the wheel building or install from source. Maturin [1] seems to be regarded as the new and improved solution for this, but I found that it’s angled toward the using python from rust.
> There’s also the rust numpy [2] package by the same org which is fantastic in that it lets you pass a numpy matrix to a native method written in rust and convert it to the rust equivalent data structure, perform whatever transformation you want (in parallel using rayon [3]), and return the array. When building for release, I was seeing speed ups of 100x over numpy on the most matrix mathable function imaginable, and numpy is no joke.
What sort of algorithm was that? Generally getting 100x speedup on vectorized code is highly unusual even using handcoded c++. So I suspect it was quite loop heavy? In those cases I have also seen very significant speed ups.
I have been using pythran [1] for speeding up my python code. It generally achieves extremely good performance. I have blogged about it here [2] and recently a member used pythran to speed up some nbody benchmarks [3] which was used in an article to argue for using compiled languages.
Matrix of shape (rows, columns, 3). Average the last dim for each point and change it to [0,0,0] if average less than a value, [255,255,255] if greater. A brightness threshold. May be remembering the speed up factor wrong so take it with a grain of salt - fact of the matter is it was very impressive.
I’m checking out that post later, I’m trying to make my package easy to build on, so being able to write extensions with Pythran would be another great option for speed ups. Thanks
Just for the fun of it I tested what speed up I could get with a naive algorithm and pythran. Based on your description it looks like the I should do the following:
def threshold_pixel(img, thr):
out = np.zeros_like(img)
o = np.mean(img, axis=-1)
out[o>thr] = 255
return out
This runs in ~30ms for a (1024,1024,3) array using numpy on my machine. Using pythran (note I had to explicitely write out the loop for out[o>thr] =255, due to a bug, that I found and just reported), I get a speed of 6.ms (with openmp) and 9ms without (I did not tune the openmp, but this should yield a much higher speedup).
P.S.: Just had a look at your project, very cool, I have to try that
There’s a setuptools rust [0] extension package that can be used to hook the compilation of the rust into the wheel building or install from source. Maturin [1] seems to be regarded as the new and improved solution for this, but I found that it’s angled toward the using python from rust.
There’s also the rust numpy [2] package by the same org which is fantastic in that it lets you pass a numpy matrix to a native method written in rust and convert it to the rust equivalent data structure, perform whatever transformation you want (in parallel using rayon [3]), and return the array. When building for release, I was seeing speed ups of 100x over numpy on the most matrix mathable function imaginable, and numpy is no joke.
I think there is a lot of potential for these two ecosystems together. If there’s not a python package for something, there’s probably a rust crate.
If anyone is interested the python package that I'm building with some rust backend, its called pyrogis [4] for making custom image manipulations through numpy arrays.
[0] https://github.com/PyO3/setuptools-rust
[1] https://github.com/PyO3/maturin
[2] https://github.com/PyO3/rust-numpy
[3] https://github.com/rayon-rs/rayon
[4] https://github.com/pierogis/pierogis