In my experience the players who love the gamified aspects of RPGs and the ones who prefer more “pure roleplaying” games are very different people, personality-wise. Without getting too much into stereotypes, I think the former folks tend to be more math-y / engineering-minded and love to “theorycraft” new characters and so on, whereas the latter tend to be more performing arts / literature / drama types.

Of course there are loads of exceptions to every rule and my point here is not to diminish or advantage either style of play, but just to highlight that these preference differences exist. This is important because I think it’s a mistake to assume there should be a one-size-fits-all form of RPGs.

I would also argue that WotC has done a lot of hard work (with several missteps) over the years but with 5E they’ve succeeded in a big way with building a system that’s easy enough to teach to beginners while offering a lot of depth for experienced players. This kind of broad spectrum game design is very difficult to achieve.

Of course the more complicated systems (designed for the mathies) are harder for beginners to learn but I’d also argue that the more open-ended, rules-light systems are hard on beginners in a different way. These games require a level of assertiveness and creativity that many beginners are unable to muster, so they often end up sitting there not participating much. In a way, these players are overwhelmed with options due to the open-endedness of the game.

D&D’s more regimented system gives each player something to do without overwhelming them, kind of like a well-designed board game. It’s a happy medium which I think is responsible for much of the game’s success in recent years.

There were some rules-light games in the 90s, appealing to a different kind of people than D&D. Vampire, Werewolf, and other Storyteller games were the most prominent. But there were also games like Cyberpunk and and Shadowrun, which were kind of similar to D&D in their target audience. But different game mechanics led to different approaches to encounters and ultimately to different outcomes.

Call of Cthulhu was thematically very different. There were no expectations that the characters would grow stronger over time, and fighting was rarely the right answer. Ars Magica was kind of like an RPG for accountants. Resource management became important, when you had tens of characters living in a hidden fortress for decades.

Then there were many niche games. Fading Suns had an interesting setting that combined many genres. The rules were kind of related to the Storyteller system, but there was so much added stuff that it never really worked. Unknown Armies was another interesting game with a very different setting. It was one of the first games I remember with rules designed to support a specific style of storytelling.

And I 100% agree about rules-light systems being possibly harder for newcomers. I've seen it again and again.

It me! I find the systems/rules/design of RPGs fascinating. I've never been interested in playing, though. I don't think I have the personality for it

Post-TSR D&D might have capitalized on this, but it was inevitable either way. By that point, generations of nascent game devs had been raised on D&D, and by the time it rose from the ashes following 4E (which I say as a 3.5 baby), we had come full circle and people were sincerely looking to recapture the feeling of WoW at the tabletop, just as WoW's whole genre was born from looking to recapture D&D on the computer.

Instead of classes and levels, popular RPG systems of the time had you develop each skill independently. Some options could only be taken at character creation. A common solution was a pool of character creation points that could be used for magic, other special abilities, status and wealth, or additional points in abilities and skills. Combat systems focused on injuries over abstract hit points and were often more deadly than in D&D. Mostly to emphasize the narrative impact of combat.

By the late 90s D&D experienced a major revival with the release of Baldur’s Gate. I believe the popularity of JRPGs among western gamers went a long way to prove that there was a market for these D&D-style games. So it may have been the case, as you said, that game designers and players (in the west) lost interest in these games but it was only a temporary dip.

Even if TSR had disappeared without fanfare the JRPG market had already been established, so this style of gameplay was not in any danger of disappearing.

Class-based systems, on the other hand, at least break up the FOO strategies into a few separate lanes which give players multiple opportunities to try different things.

I think in general, open-ended systems suffer when they present the player with too many options all at once and it’s hard to design an open-ended system which doesn’t do this. When players are overwhelmed with options they tend to go into “analysis paralysis” mode which can be a frustrating enough experience that they prefer to use a FOO strategy over revisiting that zone of complexity and confusion.