PD have proven that they can make a great looking game, but not necessarily that they can make a good simulation. A lot of the newer iterations of GT have also been slightly disappointing because they haven't been adding a lot of new things to the game. This isn't to say that they make a bad game, or that they are a bad dev, but their games are consider sim-lite by many.
Havent been adding new things to the game is completely irrelevant with the simulation aspect of the game. The simulation genre doesnt offer huge room for new ideas, features and improvements either.
In terms of physics there were improvements though.
Also AFAIK the Gran Turismo series wasnt famous mainly for its visuals either.
Actually there are even more people that consider it as one of the most complete racers than the "many" that consider it as simply a "sim-litte" good looking game. That includes pro drivers as well. It was also one of the few games that got 10/10 in Edge magazine in 1998 in a time where probably only 3 games won such a score until then (including Super Mario 64). As of today only 5 games managed a perfect score
The biggest things GT lacks in terms of simulation are laws of physics that can not be applied in a game no matter how you see it. And of course the occasional nitpicking. Although I must say that after I played GT4 at a Mitsubishi's booth inside a sim cabinet (applied more physics elements), it was super realistic. It felt like a totally different game.
As far as physics and realism that can be applied in a game for perfection, only if our systems were super computers we would have gotten that perfection.
For a title that features so many types of cars, the amount of cars, the amount of terrains and driving circumstances, PD has actually proven that they are more than good at delivering sim games. There are way too many resources put into various parts of the game, that I find it wondrous how they have managed to make the sim aspect of the game at least as good as they have done.
Now that some sources imply so many features for such a costy to develop racing game, that might even push the release towards the end of 2008 or further, the only thing I can say is that PD are probably the most dedicated and stubborn developers at offering the best possible simulator with as many features as possible.
By no means what we get is a result of laziness or lack of talent. These people live literally inside their offices because of too much work.