Part of me wonders if there's some subtle behavioral change with it too. Early on we're distrusting of a model and so we're blown away, we were giving it more details to compensate for assumed inability, but the model outperformed our expectations. Weeks later we're more aligned with its capabilities and so we become lazy. The model is very good, why do we have to put in as much work to provide specifics, specs, ACs, etc. So then of course the quality slides because we assumed it's capabilities somehow absolved the need for the same detailed guardrails (spec, ACs, etc) for the LLM.
This scenario obviously does not apply to folks who run their own benches with the same inputs between models. I'm just discussing a possible and unintentional human behavioral bias.
Even if this isn't the root cause, humans are really bad at perceiving reality. Like, really really bad. LLMs are also really difficult to objectively measure. I'm sure the coupling of these two facts play a part, possibly significant, in our perception of LLM quality over time.
Still I don't previously remember Claude constantly trying to stop conversations or work, as in "something is too much to do", "that's enough for this session, let's leave rest to tomorrow", "goodbye", etc. It's almost impossible to get it do refactoring or anything like that, it's always "too massive", etc.
I keep reading about this, but I have never, ever seen it. Daily Claude Max user for ~6 months. Not saying it doesn’t happen, but it’s never once happened to me.
This really is a lot of it, at least trying to help people at work internally. I've discovered a lot of people overly rely on Claude writing directives (always do X, never do Y, remember this every time) to its MEMORY.md, which it does mostly unprompted. The problem is, the few times I've noticed my agent getting "squirrely," some or a lot of the stuff in MEMORY.md was flat out wrong (the agent wrote down the wrong memory), confused or in direct contradiction with its CLAUDE.md, etc.
When I fixed this, it was like magic, working how I wanted again. I now have a skill to periodically audit MEMORY.md and CLAUDE.md according to the conventions I've learned work best for me - which I suppose /dream is supposed to handle eventually, but you're kind of trusting it to audit its own memories, which have, at least to me, already proven to be unreliable.
With so many factors like this, not even to mention context exhaustion, window size, effort, etc. - anecdotal evidence is almost worthless without examining someone's entire local state.
A lot of it, to me, feels like user error, I haven't really noticed much behavioral difference between 4.5, 4.6, and 4.7, at least in my own workflow. I will note though that constantly managing these things is a lot of work that I hope one day becomes less necessary. It's more than I can expect people on my team to manage on their own, and unless I sit down with them 1 on 1 and review their issues, or write some clever agent to help them, I don't really know how I can help people reporting things that I hear posted here a lot.
Even superpowers started dividing things into "phases".
"I think we can postpone this to phase 2 and start with the basics".
Meanwhile using more tokens to make a silly plan to divide tasks among those phases, complicated analysis of dependency chains, deliverables, all that jazz. All unprompted.
100% agree, and I experienced that behaviour first hand. I got confident, started giving less guidelines, and suddenly two weeks have passed and the LLM put me into a state of horrible code that looks good superficially because I trusted it too much.
This scenario obviously does not apply to folks who run their own benches with the same inputs between models. I'm just discussing a possible and unintentional human behavioral bias.
Even if this isn't the root cause, humans are really bad at perceiving reality. Like, really really bad. LLMs are also really difficult to objectively measure. I'm sure the coupling of these two facts play a part, possibly significant, in our perception of LLM quality over time.