So how long until someone trains one of these models to complete tasks by interacting directly with network/unix sockets?

At the moment, it seems like the model needs to be trained with each modality of data in mind at the start, but a generalised "supermodality" that can deliver all the others would allow truly generalised learning if the model were still capable of making sense of the input.

You'd obviously still need to finetune on any new modalities, but you wouldn't need to start from scratch.