It's pretty easy to craft a prompt that will force the LLM to reply with something like
> The `foobar` is also incorrect. It should be a valid frobozz, but it currently points to `ABC`, which is not a valid frobozz format. It should be something like `ABC`.
Where the two `ABC`s are the exact same string of tokens.
Obviously nonsense to any human, but a valid LLM output for any LLM.
This is just one example. Once you start using LLMs as tools instead of virtual pets you'll find lots more similar.
People say nonsense all the time. LLMs also don't have this issue all the time. They are also often right instead of saying things like this. If this reply was meant to be a demonstration of LLMs not having human level understanding and reasoning, I'm not convinced.
> The `foobar` is also incorrect. It should be a valid frobozz, but it currently points to `ABC`, which is not a valid frobozz format. It should be something like `ABC`.
Where the two `ABC`s are the exact same string of tokens.
Obviously nonsense to any human, but a valid LLM output for any LLM.
This is just one example. Once you start using LLMs as tools instead of virtual pets you'll find lots more similar.