"Math on datasets that can be parallelized" is a pretty huge swathe of use cases in the Data Science/Data Engineering world, I've probably spent at least 30% of my time on similar problems.
At my first job we used some special SAS product to handle data larger than memory without getting much parallelism, then it was Hadoop, then Spark. Now I can write Julia code that is agnostic over the CPU or GPU and vastly outperforms for the same types of jobs, and where I can run the same code on my laptop or a cluster. It's a huge advance for my domain! I agree it probably doesn't apply to most engineers though.
Even in NLP/Ad-tech/CV spaces a lot of the time is spent in cpu/io bound tasks such as featurization and reading datasets off of disk and shuttling them to GPU. On my most recent model training jobs in TF w/16GB of working memory I sit at an average of ~40% GPU utilization.
Some of this overhead is language specific, and some is due to shitty code. Never the less I'd bet if I didn't need to shuttle memory to GPU or could do multiple things at a time I'd crush my perf number. ( noting of course that the 40% gpu util is roughly 10x better than a CPU )
Interesting, that makes sense. This inspired me to run some checks. The CPU version of the pipeline I'm working on spends about 90% of its time with ~100% CPU utilization in a massively parallel process and 10% of its time on other stuff, mostly io. With the GPU making the massively parallel part ~10x faster, io is now the dominant portion of my code – Amdahl's law in action!
Tensorflow datasets are pretty great, once you get them really rolling. They do a great job of scaling out worker threads for various parts of the featurization process, keeping an arbitrarily big cache of batches ready to go in ram, etc.
aye - the above job was done with TF datasets. They are limited in that the python gil requires multiprocessing, multiprocessing involves serialization in python, serialization involves dealing with the rather extreme object overhead in python ( 24 bytes for an int! ).
Which all means there's a bunch of CPU bound stuff between your job and the GPU/Cuda kernels. How fast your app can deal with the above will influence overall GPU utilization.
While I appreciate that Python is easy and flexible enough to write, it bugs me immensely every time I run into situations like this.
We go to all this effort to build and write stuff in this optimising, fancy framework only for the whole process to be bottlenecked by some silly performance limitation in Python.
It's usually possible to sidestep the python limitations with a bit of elbow-grease. The usual killer for performance is tf. py_function, which does indeed have to respect the GIL. If you can work out a nice way to handle your data without it, it should be able to stick to cpp in the backend and avoid the GIL. (So, data in a format that tf has a parser for, and write transformations using tf methods where you can.)
At my first job we used some special SAS product to handle data larger than memory without getting much parallelism, then it was Hadoop, then Spark. Now I can write Julia code that is agnostic over the CPU or GPU and vastly outperforms for the same types of jobs, and where I can run the same code on my laptop or a cluster. It's a huge advance for my domain! I agree it probably doesn't apply to most engineers though.