Well, it's one of the reasons.Guden Oden said:Except, people's ears aren't all created equal, which is why these virtual speaker schemes never work very well.
Another problem lies in the basic premise of trying to recreate a sound space within a different room/environment. The properties of the room will always be superimposed on top of the recording. There have been attempts to remove at least the effects of the first order reflections using mics and DSPs, with dubious success.
With binaural recordings reproduced over headphones, you can avoid this, but these recordings often suffer from being recorded inside a dummy head, causing the reproduced sound to pass through two sets of ears, those of the dummy at recording, and those of the listener at playback. (I believe this penchant for doing binaural recordings the theoretically "ideal" way but in practise less than optimal comes from the strong tradition of dummy head recordings in auditory research.)