That will almost certainly be a fillrate issue.
I personally always felt like the N64's memory subsystem was broken, you're supposed to put Z and Color on different pages, to improve performance, it's clearly stated in the docs. In practice it made very little difference. With Z enabled, you'd be lucky to match fillrate with a PS1 (which of course had no Z)
Neither has what you would term a texture cache today.
PS1 has a page of memory "VRAM" that contains all the textures, and the framebuffers, you can get textures in there larger than the maximum supported UV coords.
N64 has a tiny piece of memory 4K, where a texture must be manually loaded before you can render from it. If mipmapping is enabled, this is reduced to 2K because alternate mipmaps must be in opposite pages.
There were a few High res games on PS1, but it wasn't common. Neither platform really had the fill rate to do it well accross a wide variety of genres.
N64 was probably hit a bit harder, because any extra memory pressure was bad.
Wouldn't be of much use, the theoretical numbers were much higher than what you'd see in practice since it was bandwidth limited for the most part. Plus you have to take into account things like stalls while loading the texture into the cache which the PS1 never had to deal with.
Turning off Z made a considerable difference, World Driver runs without a Z buffer for this reason, but it was a significant risk when we made that decision, because it was unclear if Nintendo would bounce a title for excessive Z fighting.
Disclaimer it's been a LONG time since I wrote an N64 game and even longer since I wrote a PS1 one, I responded from memory, so all facts in this post should be considered appropriatly degraded.