By the way, why _do_ llms work with tokens and not letters? Is that some kind of optimization, or is there a more fundamental reason for it?

I actually would love to see the output of an LLM that used letters or syllables or phonemes. The output when it makes mistakes would be absolutely wild.