Or someone works out a hardware architecture that's optimized for AI inference in the way you describe, but also good for 3D graphics.

That would fuse 3D graphics and AI accelerators into 1 and the same unit, as far as consumer hardware is concerned.