Basically I think Quake was the only thing that was really comparable to Mario 64. On the right hardware it was probably better, but I doubt many had access to hardware at the time that was running Quake as fast and looking as good as Mario 64.
Quake undoubtedly evolved the FPS genre by becoming fully 3D and pushing the technology.. however I think Mario 64 was more revolutionary in how it adapted platformers to 3D with 3rd person cameras and truely analog character control.