> When we get to larger matrix-matrix operations, such as 100x100 * 100x100, we can effectively write off any overheads due to memory allocations. But we definitely see that there is a potential for some fairly significant performance gains in the lower end! Notice too that these gains are realized by using the pure-Julia LoopVectorization.jl as the standard BLAS tools tend to have extra threading overhead in this region (again, not optimizing as much in this region).
> But, if you have been riding the GPU gospel without looking into the details then this plot may be a shocker! However, GPUs are designed as dumb slow chips with many cores, and thus they are only effective on very parallel operations, such as large matrix-matrix multiplications. It is from this point that assumption (2) is derived for large newtork operations. But again, in the case of small networks such GPU kernels will be outperformed by well-designed CPU kernels due to the lack of parallel opportunities.
"Small" is difficult, and it is actually in many cases, harder than "big". Small needs language support, GC support, runtime support, and is delicate - any one thing can throw off performance. You can't hide behind library calls. In Julia, since we can orchestrate everything, from the abstractions all the way down to the instructions, making small problems work well has been possible.
A single grad student can make large problems work (my thesis was large linear algebra and I could just hack away on enough C and MPI to get it done). In our early days on Julia, we realized that 90% of the world actually needs small linear algebra and it is a tantalizingly difficult problem. The work done in the Julia community over the years has made it all possible through a collaboration across lots of different teams and disciplines.
I think this is a pretty key point- the majority of users haven't been well-served by large-scale compute. I've heard from a number of folks in genomics that all they need is a faster way to invert a "big" matrix- and when they show me the matrix, it's tiny compared to what state of the art supercomputers are working on.
One of the first questions I ask when discussing issues like this, is to define "big" and "small" for me. Every group has their own set of definitions, and the differences can result in interesting conversations.
The performance advantage was still in the ballpark of 4-8x faster for training MNIST on the CPU, which while smaller than most networks people are training on their GPUs, still has more than 40 thousand parameters.
For someone with a statistical background, this is a lot of parameters. John von Neumann could wiggle a lot of elephant trucks.
A lot of practical/useful models fill the range from the tiny ones we may use in UDEs and SciML to this MNIST convnet.
yeah, SimpleChains definitely won't be useful for the next large language model, but there's a lot of very small models in the world from physics, to pharma, to high frequency trading. Having libraries that are optimized for these usecases is a major benefit for a fairly large niche of users.
Neural network renaissance definitely renewed the interests to use neural networks as function approximators in many areas.
For the one I am interested in recently (dynamics control), many networks are small feed-forward ones or two-layer LSTMs. The importance there is about balance robust controller v.s. update frequency. If you can run the same controller at higher frequency, it is basically free performance improvement (the robot will be more stable, more responsive etc.).
And video games, too. Small models may be useful for procedural content and dynamic environment and actor responses. The barrier, so far, had been the performance overhead of using them.
Yeah, also for small networks that perform well in Julia, there's XNNPACK with AVX2, so I wonder how those 2 compare. But as soon as your network is large enough that execution time exceeds PCIe transfer time, GPUs win in any case.
> When we get to larger matrix-matrix operations, such as 100x100 * 100x100, we can effectively write off any overheads due to memory allocations. But we definitely see that there is a potential for some fairly significant performance gains in the lower end! Notice too that these gains are realized by using the pure-Julia LoopVectorization.jl as the standard BLAS tools tend to have extra threading overhead in this region (again, not optimizing as much in this region).
> But, if you have been riding the GPU gospel without looking into the details then this plot may be a shocker! However, GPUs are designed as dumb slow chips with many cores, and thus they are only effective on very parallel operations, such as large matrix-matrix multiplications. It is from this point that assumption (2) is derived for large newtork operations. But again, in the case of small networks such GPU kernels will be outperformed by well-designed CPU kernels due to the lack of parallel opportunities.