> I read the V1 code this time instead of guessing

Does the LLM even keep a (self-accessible) record of previous internal actions to make this assertion believable, or is this yet another confabulation?

No they do not (to be clear, not internal state, just the transcript). It’s entirely role-play. LLM apologies are meaningless because the models are mostly stateless. Every new response is a “what would a helpful assistant with XYZ prior context continue to say?”

Yes, the LLM is able to see the entire prior chat history including tool use. This type of interaction occurs when the LLM fails to read the file, but acts as though it had.

This seems like the experience I've had with every model I've tried over the last several years. It seems like an inherent limitation of the technology, despite the hyperbolic claims of those financially invested in all of this paying off.

Opus 4.6 pre-nerf was incredible, almost magical. It changed my understanding of how good models could be. But that's the only model that ever made me feel that way.

Yes! I genuinely got a LOT of shit done with Opus 4.6 "pre nerf" with regular old out-of-the-box config, no crazy skills or hacks or memory tweaks or anything. The downfall is palpable. Textbook rugpull.

There was no nerf - this meme needs to die.

What exactly happened then? How did we all have this collective hallucination?

Collective hallucinations are common. Mandela effect, people thinking FB is listening to your microphone because they see relevant ads, etc

This is a common phenomenon that all humans pattern match to things we expect. When we learn a new vocabulary word you see it everywhere for the next two days. When we think Claude might be nerfed, we overindex on every instance of Claude underperforming.

The only way to account for this is credulous, hard data. Like benchmarks over time. To this day no one has provided evidence that Claude Code, when fixed to the same thinking level, has had degraded performance.

Are there any good ways to benchmark models over time that don't fall victim to Goodhart's law? It seems that once the benchmark is defined, the AI will train on it, and it will become effectively meaningless.

I read many articles about AIs doing extremely well on various tests in graduate or PhD level programs. But these tests are well defined. A professor put the same models though his freshman CS class and most of them failed.

These models don't learn continuously, they are a static snapshot one training is finished. You only need a new benchmark once new models are published (or you need a private benchmark, in which case you don't need to update the benchmark at all)

Did they nerf the model or was it changes to Claude code? I agree it got frustrating.

That was better, but still not to the point that I just let it go on my repo.

If it isn’t working for you why don’t you choose an older model? 4.6

Matches what I am experiencing. Makes incredible stupid mistakes.

The weird stuff is yesterday I asked it to test and report back on a 30+ commit branch for a PR and it did that flawlessly.

The docs suggest not using max effort in most cases to avoid overthinking :shrug:

They've jumped the shark. I truly can't comprehend why all of these changes were necessary. They had a literal money printing machine that actually got real shit done, really well. Now it's a gamble every time and I am pulling back hard from Anthropic ecosystem.

it's clearly all in your head. 4.6 is just as capable as it used to be. literally no one on the internet has managed to post credulous and real evidence of a nerf

this is just another trendy conspiracy theory that people reinforce because of selection/recency bias. you hear "nerf", your brain overindexes on the next time Claude does poorly. it is the same phenomenon when you notice a new vocabulary word all the time.

It seems clear that it was a money spending machine, not a money printing machine.