It's worth noting that this is primarily because older games just... didn't stream much at all. Ultimately a lot of stutter is due to streaming "things"... textures, shaders, meshes, worlds, even game logic. As part of the push for both larger and higher fidelity games, it is no longer possible for these games to just load a level into memory and sit on it until the next load screen. The push to eliminate load screens has also complicated matters, even for relatively linear games. When you go back to the 90s games you do indeed ditch the stutter but you get the load screens back.
None of this is to say that stuttering is acceptable. Nor are - IMO in 2024 - load screens (hi Starfield). But to frame this as "90s games had solved this problem and we somehow forgot how to do it" is misunderstanding the issues; 90s games simply did not have these problems because their scope was much smaller. It's not that they had magical streaming systems that worked great, they simply didn't really have streaming systems at all.
I think we're likely (hopefully?) at a local minimum of a bunch of this though. Folks are transitioning from older style baked, statically loaded stuff to newer much more dynamic systems and streaming, but don't necessarily have a lot of the experience yet necessary to tune (or even in some cases, use) the new systems. Ex. I am flabbergasted every time a new game ships without virtual textures. Texture streaming is basically solved (and no, we don't need sampler feedback
![Wink ;) ;)](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
)... just turn it on. Nanite pretty much solves geometry/mesh streaming, although simpler systems can work fine for that too. PSOs are still a bit of a mess but a workable and understood mess I think. Raytracing is probably the biggest offender as it's just not possible to do efficient fine-grained BLAS/TLAS streaming on PC right now.
That said, I'd wager that the majority of stutter issues that are still happening with games that understand how to use the above features are actually from the gameplay and physics side. In Unreal you can get into a lot of trouble if you just naively use blueprints and spawn and destroy actors all over the place. This is nothing new, and more experienced folks know how to deal with it via more native C++ custom gameplay systems, but newer folks that are heavier on the art department can certainly fall into traps in this area. These systems have seen some improvements at an engine level but fundamentally stuff like blueprints are never going to be appropriate for things with 1000s or 10's of thousands of instances. Systems like MASS (as used in CitySample for the traffic and crowds and so on) are more meant for those kinds of cases, but I would generally recommend and expect AAA folks to roll their own stuff at this level, because gameplay needs and systems are obviously very game dependent.
Anyways the tldr is I think it's completely legit to call it unacceptable to have persistent stutters and other interruptions, although like anything sometimes the hyperbole goes a bit far. But certainly when comparing to past games it's not that previous games had solutions to this stuff, they just did not target the scope required to even encounter the problems. Folks making smaller games with more linear, static content also don't tend to run into these "massive open world" kind of issues to nearly the same extent.
I mean it's not supposed to run and look terrible either, at least as long as your hardware hits the minimum requirements. Stick to your guns here - as a consumer it's always fine to call out stuff that you don't think lives up to your required standards; you no more need to make apologies for framerates than stutter (particularly on consoles).