Confabulation is human behavioral phenomena that is not all that uncommon. Have you ever heard a grandpa big fish story? Have you ever pretended to know something you didn't because you wanted approval or to feel confident? Have you answered a test question wrong when you thought you were right? What I find fascinating about these models is they are already more intelligent and reliable than the worst humans. I've known plenty of people who struggle to conceptualize and connect information and are helpless outside of dealing with familiar series of facts or narratives. That these models aren't even as large as human brains makes me suspect that practical hardware limits might still be in play here.

Right, that’s kind of my point. We call it “hallucination” because we don’t understand it, but need a shorthand to convey the concept. Here’s a paper trying to demystify it so maybe we don’t need to make up anthropomorphized theories.