It wasn't the PS2's memory size that was a problem - that seemed fine taking into account the machines release date and comapring it to DC and GC. It was the amount of space that textures of a similar quality to DC, GC and Xbox took up in main memory that was the problem. A 16 colour CLUT simply doesn't have the colour depth to replace 2bpp VQ and 4bpp S3TC/DXTC.
I remember that large textures were often used to cover all or most of a model (character body, car, etc) because they were the most efficient way to manage textures and also the store the data (arrange surfaces optimally within the texture to minimise wasted blank space). 16 colours could easily be an appallingly low number of colours to use in a texture for a 24-bit rendered image, depending on what you wanted to reproduce. 16 colours is what the Master System had for backgrounds (and sprites) btw.
That kind of wrap around texturing was bad practice even back then. You don't waste anything if you use texturing to texture single objects or parts of a model, instead of the whole model.
The 8bit consoles had to define the world with the 16 colours, when you had polygons, textures and 16 mil colours to do it on PS2, so no comparison there.
You are not supposed to
draw details when you have that many polys. The rare poster, painting, sign or other object in the games that needs more than 16 colours can be done with 8 bit fine without breaking the bank.
Try doing some conversions of monocrome-ish stuff to 16 colours. Often if there is no large colour gradations (which could be done with vertex colours anyway) but many small details, it's quite hard to spot the difference.
And again, even if we suppose that colourdepth was a bigger problem than I'm making it out to be, there would still be no explaining the difference in resolution. S3TC has the same bits per pixel, and DC had fewer Mb's per frame.
Unless of course you are implying that devs used 8bit textures very often to compensate, which I think we can safely say, based on anecdotal evidence and visual evidence, was not the case.
8bit CLUT textures, while they would have been on par with VQ in colours, or better in fact, would have been a complete waste of resources most of the time and would have been much blurrier.
The DC could only texture from video ram - meaning every game had around 5.5MB of texture memory available all the time. I don't think you could really use it for anything else (at least without copying to main ram first) so that's probably why DC games tended to have such detailed textures compared to the PS2. On PS2 texture memory had to fight for space with everything else. Using a mix of 4 and 8 bit CLUTs, and using Fafalada's rough conversion/comparison, that equates to something like 11 ~ 16.5 MB of PS2 main ram lost if you want to use "DC quality" textures - which would barely leave you enough memory for DC quality models and maps. It's easy to see IMO why PS2 devs chose to go for much higher poly counts (that would also benefit from more and better per vertex lighting).
If that was true, which I don't think it is, it's strange that not a single game chooses to go in the other direction. Even games that was quite spare on geometry and had limited environments, like ICO and Ecco the Dolphin only had slightly better textures than other games.
Incedently Ecco, a DC port, has some of the best textures ever on PS2, even though they are still downgraded from the DC original.
There was one game, Jack and Daxter, that had quite detailed textures, with none of the lowres blur apparent in almost all other games (the sequels didn't impress me as much) the downside was that it was mainly small heavily tiled textures, which was noticable once you saw it.
Point being, that it was not some kind of deficiency in the system that didn't allow a high texel to pixel ratio.
On the GC it would seem that S3TC and the A-ram made a similar difference. Although now I think about it, while I remember GC textures being far more colourful, I'm not sure they were always that much higher resolution than the PS2. Perhaps that depended on how you used the A-ram. Like with the PS2 textures had to fight for space with everything else.
They were more detailed without question. Look at Metroid and RE4 for some good examples. "256x256 and 512x512 textures all over the place.
Transparent textures were rare though, and with large textures (the norm) the space taken by conversion tables should be relatively tiny (unless I'm missing something). And with quite a few 8-bit CLUTs needed in the mix to do a similar looking range of textures I can easily image needing more than double the space.
Well detail-maps that are not supposed to be transparent in the details, water, window, quads where you don't want the hard edges of binary alpha, etc. Depending on the game of course, but they are not that rare.
Even without JPEG decompression, the kind of from-disk texture streaming that we see on current systems would have benefited the PS2 a lot. Check out Rallisport Challenge 2 on the Xbox to see tens or hundreds of MB of data per (full) course being streamed in from HDD - the game is incredible, and the most impressive racing/driving game of last gen by a frikkin * mile.
Some PS2 games did actually stream textures from the DVD. I seem to remember the sequel to Baldurs Gate being one of them.