Multi Focal HUD

[Emirikol]

No body knows about the existence of a multi focal Head-Mounted Display? Thanks

[mboeller]

you mean holographic HUD's like the one in use in fighters, some business jets and in some commercial jets?

[Emirikol]

Usually VR HUDs have a fixed focus,
I mean a HUD with multi-focal optics. The observer's eyes can focus near objects and far objects freely as in reality

[epicstruggle]

I did a paper for one of my CS classes about the need for a multi focal optics in a HMD unit. I described a system that would accomplish this. In my search for related works, I did not come up with anything like what i proposed. I figure in the last 6 months since i did this paper, nothing will have changed. The main obstacle to a MFO in HMD is the lack of good miniturized lcd displays. Sony which many thought would revolutionazy the industry, has dropped their plans to produce more of their HMDs.

Emirikol, if you do find anything, please PM me. I would be very interested in hearing what you find out.

[Emirikol]

I have found something about Super Multi View Images but it seems to me another holographic picture system, ... simply unconfortable

[MfA]

Usually VR HUDs have a fixed focus,
I mean a HUD with multi-focal optics. The observer's eyes can focus near objects and far objects freely as in reality

As far focusing is concerned there might be one degree of far, but there is a continuous spectrum of near of course ;) Having a mix of multiple focus points might make some simulations a little more realistic, but isnt what we really need variable focus optics? (AFAICS with variable focus a single focal point would be sufficient together with a normal stereoscopic setup too, I doubt it makes a big difference wether something we are not looking at is truly out of focus or merely is made to seem so by the stereoscopic effect and some blurring.)

Marco

PS. I was assuming with multi focal you mean the system mixes images at different fixed focal points (presumably with a projection system in the spirit of "Pepper's Ghost").

[Emirikol]

Marco, the goal is to simulate reality the best we can.
Studies have dimostrated that the fixed focus is the main cause of head and eyes fatigue in wearing a VR o AR HeadMountedDisplay.

In real life eyes change focus tens times per minutes while in VR eyes are hold in almost the same position and at the same focus lenght.

The 3D effect in the human vision is dued to:

1. Stereoscopy
2. Perspective
3. Focusing

In HMD usually we see a stereoscopic perspective correct 3D image focused as if it was a 2-3m distant ahead wallpaper. This is not natural.

To produce an image with a progressive focus from infinite to 30cm is a primary effort of VR designers.

[mboeller]

Marco, the goal is to simulate reality the best we can.
Studies have dimostrated that the fixed focus is the main cause of head and eyes fatigue in wearing a VR o AR HeadMountedDisplay.

In real life eyes change focus tens times per minutes while in VR eyes are hold in almost the same position and at the same focus lenght.

The 3D effect in the human vision is dued to:

1. Stereoscopy
2. Perspective
3. Focusing

In HMD usually we see a stereoscopic perspective correct 3D image focused as if it was a 2-3m distant ahead wallpaper. This is not natural.

To produce an image with a progressive focus from infinite to 30cm is a primary effort of VR designers.


IMHO multifocus HMD is not the answer.
IMO you need an eye-tracker so the system knows where the person looks (updated at >50Hz). This portion of the screen is updated and focused then so everything looks perfect.
The other parts of the screen should look slightly blurry at the same time to enhance the "feeling" to look.
It should be possible to use an low-energy laser to track the eye-movement (IMHO this has been done already, but I'm not 100% sure).

[Basic]

mboller:
That would make the out of focus blur look real, but it wouldn't remove the eye strain from conflicting focusing vs stereoscopy. Because you would still have an incorrect focus.

If the optics are made so the image is in perfect focus no matter how the eye is focused and combine that with what you said, then there's a chance that the eye strain is removed.
But it might not be so easy to construct such optics, even if it's at least theoretically possible. (By focusing all light through a point in the center of the pupil, making a "virtual pinhole camera". )

It might be easier to make optics that can change the focal plane physically with electrical motors, combine it with syntetic bluring for off-focus parts, and control both with an eye tracker.

But my favourite idea is to have optics for the "virtual pinhole camera", use multiple light sources to move the pinhole position over the pupil, and find a LCD display with a few hundred Hz refresh rate (ouch :) ). With such a system you could render one frame from each position of the pinhole, and send it to the eye to the right pinhole position.
The result is kind of like T-buffer (or any other acc-buffer) depth of field, but the mixing will be done in the eye according to how you're actually focusing. Pretty much the real deal.


But to answer your question Emirikol:
No I don't know of any such HMD. :(

[Dave B(TotalVR)]

Ive been harking on (though not on forums) about using LCD displays to produce a holographic diffraction grating for ages. Then all you need to do is shine a laser through the back of the LCD and you have a full 3D image that you can focus on the front and back of perfectly. Not only would this absolutely rule in terms of game immersion it would also not really be much more expensive - if any more expensive at all - than current flat panel displays. All you need is an LCD screen capable of refreshing at 120hz for 60hz effective refresh, or 225hz for 75hz effective as you need to do each colour channel separately. Switching the laser at that speed is no problem.

Imagine the pink demons in Doom 3 actually runnig right up to you face before you knock them a couple of metres back with your dual barrels of justice...

[Basic]

That would need an *ahem* rather high res LCD. :D

[MfA]

Emirikol : ok, so I was assuming wrongly (apparently, you say little explicitly, since you now say progressive focus I assume you meant that all along). Multi suggests to me multiple but discrete, so I thought you meant something else ...

Marco

PS. <A HREF=http://www.google.it/search?hl=it&ie=UTF-8&oe=UTF-8&q=%22Device+and+Method+for+Viewing+Three-dimensional+Computer+Graphic+Displays%22&lr=>a little googeling</A> turned up they did a research project on a MEMS based deformible lens for a variable focus HMD (a system with motors and normal lenses is probably impractical, at least that assumption gave me the query which found the link :) at Berkeley, but I cant find any related publications ... so it might have been found to be utterly infeasible, at least at the time. Or maybe the publications are still pending review, if you are really interested you could always mail mr. Banks.

PPS. couldn't virtual retinal displays "easily" emulate variable focus BTW?

[SA]

Variably focused HMD displays is still an unsolved problem. As was mentioned, it is the leading cause of eye fatigue, headaches, etc. when using HMDs, and probably a major reason they have not become more popular even for professional uses.

Direct retinal scanned HMDs (where a multicolor laser is scanned directly into the eye, see http://www.mvis.com) and eye tracking with conventional HMDs offer the most promising avenues to solve this problem.

[Emirikol]

I have studied the laser scanner display but the only way to obtain variable focus is to change the focus by modulate optics for EVERY pixel during the scanning process....But This meccanism should operate at about 1024x768x30=23MHz!!!, We must exclude sliding lens...MEMS system that deformate a lens may, perhaps, be a possibile solution, I dont know.
Another possibility is to perturbate the angle of the scanned laser beam to simulate, on a pixel basis, the optics' convergence or divergence, but the needed accuracy is very high and I doubt a today MEMS chip can be able to do that.

[MfA]

Wouldnt it be easier to make the laser project the right colour for the spot on the retina the scanner is aiming at at a given time, instead of trying to aim the original pixels at the right spots?

[Nagorak]

Ok if eye fatigue is such a big deal, can someone please explain why there is no such fatigue when staring at a computer monitor (with the exception of one that's running at crap refresh rates)? I mean I know supposedly there is fatigue from looking at a computer monitor, but I've never experienced it and I spend hours in front of mine, much of which is spent reading and writing text, etc.

So that being the case couldn't your eyes just "get used" to VR displays, and if so isn't all this nonsense about making it more realistic on the eyes, just that: nonsense?

[Captain Chickenpants]

Partially because your screen is further away than VR headset would be, and your eye has to work harder to focus on things close up.

Partially because even without thinking about it you will be looking around
at other things.

It is actually recommended to take a short break every so often (about 50 mins I beleive), and look at things at a different distance so as to relax the eyes. (At least thats my excuse and I am sticking to it).

[SA]

There are two primary problems with HMDs.

The first has to do with the problem mentioned, lack of focal variation. The brain uses depth cues (primarily from the stereo disparity) to decide how far away each image point is. It then uses this distance information to automatically focus the eye. As the eye focuses correctly, it correlates the resultant newly focused image as a feedback mechanism with the depth input data to know when the eye is properly in focus. Automatic focus cameras do something similar. The problem with HMDs is that the focused image never matches what the brain expects from the stereo disparity thereby creating the headaches, etc. Stereo glasses used with a monitor cause the same type of problem.

The second problem is head tracking latency and accuracy. This usually causes motion sickness. The brain is very sensitive to slight differences between the motion the inner ear senses and the motion the eyes see. Head tracking latency and accuracy is much easier to solve, and many high end professional HMDs have solved this problem.

[SA]

The simplest solution is to use a combination of eye tracking and pupil size tracking from sensors as input and process the 3d image using the proper depth of field calculations for focusing at the given 3d image point with the given pupil size. To further reduce the disparity between what the brain expects and what it sees, including proper bright light processing such as glare, etc. would also be of some benefit.

[Dio]

I'm one of the ones who has a trouble with conventional HMD's - I tend to get headaches after 5-10 minutes. I find it difficult to cope properly with the different 'stereoscopic focus' vs. 'image focus' - my eyes have a tendency to look in the wrong places.

I found this while experimenting with a HMD about ten years ago now. I suspect I would see an improvement if I got used to it - since I used to have similar problems for the first half-hour or so after I changed from my glasses to my contact lenses, but nowadays there's no switch time or headaches at all.

[Dio]

A second note: I find stereoscopic glasses (w/projector or monitor) approaches don't create the same problem for me, probably because of the aforementioned thing that 'far is far, but there is a lot of near'.

[Emirikol]

There are two primary problems with HMDs.

The first has to do with the problem mentioned, lack of focal variation. The brain uses depth cues (primarily from the stereo disparity) to decide how far away each image point is. It then uses this distance information to automatically focus the eye. As the eye focuses correctly, it correlates the resultant newly focused image as a feedback mechanism with the depth input data to know when the eye is properly in focus. Automatic focus cameras do something similar. The problem with HMDs is that the focused image never matches what the brain expects from the stereo disparity thereby creating the headaches, etc. Stereo glasses used with a monitor cause the same type of problem.


You are right SA.

But I must disagree with your second post.
The observer pupil size is not correlated to focus depth.
To change the deep of field of the image doesn't improve the vision quality. Because the image is always focused as if it was a screen XX cm away from the observer eyes. When the eye see an object apparently near (sensation given by the stereoscopic effect) the brain instantly tries a "near" focus. When the eye see an apparently far object, the brain tries to focus the eye to infinity....but in an usual HMD the image is always at the same focus, so changing the observed object forces the brain to try a wrong focus with conseguent fatigue

[mboeller]

But I must disagree with your second post.
The observer pupil size is not correlated to focus depth.
To change the deep of field of the image doesn't improve the vision quality. Because the image is always focused as if it was a screen XX cm away from the observer eyes. When the eye see an object apparently near (sensation given by the stereoscopic effect) the brain instantly tries a "near" focus. When the eye see an apparently far object, the brain tries to focus the eye to infinity....but in an usual HMD the image is always at the same focus, so changing the observed object forces the brain to try a wrong focus with conseguent fatigue

IMHO it should be possible to track the thickness of the pupil with an laser and use this info to focus the picture in the right distance using an holographic display (or retina display).

[MfA]

I think by the time we have a holographic display we will also have the computational power to use it for full 3D ... just using it to put a 2D image with artificial DOF effects at the correct focus depth seems a bit of a waste :)

[mboeller]

I think by the time we have a holographic display we will also have the computational power to use it for full 3D ... just using it to put a 2D image with artificial DOF effects at the correct focus depth seems a bit of a waste :)

Sorry;

I ment holographic HUD's like the one in use in fighters like the new F16 models or more modern fighters. I didn't ment real holographic displays.
Holographic HUD's project the image so that it appears to be in infinite distance ( or a few meters away, depending on the requirement ).

[Quaz51]

THE solution!
technology VRD (virtual retinal display)
with "depth modulation"

http://www.cs.nps.navy.mil/people/faculty/capps/4473/projects/fiambolis/vrd/vrd_full.html

[SA]

That's right about the optical screen depth. Dynamically changing the depth of field in the image would only make it appear more photorealistic based on what the eye was looking at and would not help with the focal disparity.

As was pointed out, the disparity in focal depth is an optical problem and only a dynamic opitical solution can solve it.

Motorized optics (similar to the optics in an auto focus camera) could be used to solve the problem, but it seems a bit mechanical. You would again use the eye tracker to see what was being viewed in the image, calculate the distance to the 3d surface in the image at that point, and adjust the optical screen depth accordingly using the motorized optics.

Some other possible solutions, a bit less mechanical, would be to use electromechanical optics such as deformable mirrors and lenses.

[SA]

The major problem with immersive stereo HMDs, however, is not technical in my opinion. The market never developed to the point to fund the R&amp;D to solve the technical problems. Nvision and many others diverted R&amp;D efforts to other areas where there was a larger market. Stereo glasses and CAVE systems occupied much of the professional market, and the consumer market has been satisfied without stereo and full immersion.

[Emirikol]

That's right about the optical screen depth. Dynamically changing the depth of field in the image would only make it appear more photorealistic based on what the eye was looking at and would not help with the focal disparity.

As was pointed out, the disparity in focal depth is an optical problem and only a dynamic opitical solution can solve it.

Motorized optics (similar to the optics in an auto focus camera) could be used to solve the problem, but it seems a bit mechanical. You would again use the eye tracker to see what was being viewed in the image, calculate the distance to the 3d surface in the image at that point, and adjust the optical screen depth accordingly using the motorized optics.

Some other possible solutions, a bit less mechanical, would be to use electromechanical optics such as deformable mirrors and lenses.

quite true

[Emirikol]

THE solution!
technology VRD (virtual retinal display)
with "depth modulation"

http://www.cs.nps.navy.mil/people/faculty/capps/4473/projects/fiambolis/vrd/vrd_full.html

I have read the entire documentation but it is not explained what they mean for "Depth Modulation". In any case the document refers to the first microvision retinal scanner prototype...now, after 2 years, they still do not have a HMD capable of multiple focus.

[Chalnoth]

THE solution!
technology VRD (virtual retinal display)
with "depth modulation"

http://www.cs.nps.navy.mil/people/faculty/capps/4473/projects/fiambolis/vrd/vrd_full.html

I think I might have actually seen this one in action.

I think it was about 1996-1997 or so when I went to check out the flight sim facilities at the Pax River Naval base. They had three flight sims:

The first was a large dome, 20ft in diameter, if I remember correctly, with an F-18 cockpit suspended in the center. The display was projected all over the dome (with the highest-resolution display in the front). That was very cool, though the ground detail was pretty low.

The second consisted of three screens arranged in a format similar to what is seen today from Matrox' Triple head tech, though the displays were much larger. It was a simulation of a VTOL aircraft (I forget the name...it had tilt wings and propellers...).

The third was a VR display with a headset that projected the display directly into the eyes. While I only used it for a few minutes, there was absolutely no eye fatigue, and looking at the computer-generated surroundings was quite natural, though the helmet was rather heavy. It was very, very cool, with the only drawbacks being the heavy helmet, the swing arm used for motion detection (it wasn't connected to the helmet directly...but it had to be close, and so was right above my head...apparently it's been broken a few times, too...), and the fact that the calibration for projecting the image directly into the eyes had to be done exactly, and was different for every person.

[Emirikol]

I think to have found a solution

[Basic]

And it is?

[Emirikol]

Sorry I cannot unveil the details
but I've found a way to cast the rays of light at a different angle depending by the single pixel deep. I'm writing a Paper.

[MfA]

Do laser beams stay collimated if you modulate them with a LCD?

If so Basic's pinhole projection idea would be pretty straightforward, backlight a LCD with lasers and focus the resulting beam somewhere between the minimum focal point of the eye and the eye's lens (wouldnt want you to fry your retina the moment you try to watch something too closely :). You would probably want to do it with several lenses one of which very close to the eye to get the necessary arc to be able to cover enough of the retina. You need to apply a warp to the image to counteract for the distortion from eye's lens of course, but that could all be done digitally.

[Chalnoth]

Of course, the only problems would be in attempting to deal with people that have vision that's not 20/20. However, since that's still a requirement of pilots, I don't know if the Navy considers it much of a problem...

[Emirikol]

Do laser beams stay collimated if you modulate them with a LCD?

If so Basic's pinhole projection idea would be pretty straightforward, backlight a LCD with lasers and focus the resulting beam somewhere between the minimum focal point of the eye and the eye's lens (wouldnt want you to fry your retina the moment you try to watch something too closely :). You would probably want to do it with several lenses one of which very close to the eye to get the necessary arc to be able to cover enough of the retina. You need to apply a warp to the image to counteract for the distortion from eye's lens of course, but that could all be done digitally.

I dont' understand how this sistem works...
can you further explain me

[MfA]

Pinhole cameras have a natural focal point because they act somewhat like a fresnel lense ... but mostly the projection stays in focus, albeit distorted, regardless of the distance to the image plane (and thus regardless of lenses you put behind them). I assumed that effect was what Basic was after.

It doesnt really matter what kind of optics you use though, with a normally lit LCD only a minute amount of the energy you put into lighting would actually find its way to the eye with pinhole projection. So collimated light put through a virtual pinhole seems to make more sense.

I dont quite see how he intends to handle when you look sideways though, you could use multiple pinholes like he suggests, covering the full arc the eye can cover ... but because of the discrete number of pinholes the moment one is obscured by the edge of the pupil it can make a big difference in the intensity at the retina. You would need a very large number of pinholes to remedy that (in a way this would be trying to reproduce the lightfield coming towards the eye, instead of trying to reproduce the image ... a poor man's holography so you will).

Maybe you could track the pupil and put a large lens in front of the eye and aim the beam at it according to the direction you are looking (with beam I mean the collection of rays representing the individual pixels). This would still give you the virtual pinhole projection, but allow you to cover a wide viewing arc.

I dont have the necessary experience to intuit if this could actually work, and Im not gonna do the math ... so take it with a pinch of salt ;)

[Basic]

I think you got the idea right MfA. The intention was to track the pupil, and only send light for pinholes that hit the pupil. This could be done by moving optics as you say, or it could be done by switching light sources.

Look at this image (http://hem.passagen.se/basic3/fora/dimension3d/dof_hmd.htm). (Sorry my web space doesn't alow image leeching.)
The url hints on how old the idea is. :D

There are two lightsources in that image, just to show the principle. And they are both hitting the pupil. A real implementation would have lots of light sources. Dense enough to get several pinholes at the pupil, and over large enough area to get possible pinholes on every position the pupil may be.
The lens system shown is the simplest possible, but it would be a good idea to design it so that the final lens is more "curved" around the eye, so you can get a high FOV.

What is the meaning with multiple pinholes on the pupil?
Let's go back to the good ol' T-buffer, and think about how DOF is made with it. You render four images with the camera slightly shifted, but directed so that points at the focal plane coincide. And then blend those images together.
This is equivalent to aproximate the pupil with four pinholes (spread over the pupil). The camera shifts are the different pinhole positions over the pupil, and the different direction of the cameras comes from how the eye is focused. With T-buffers the program have to decide what eye focus to simulate.

Now get back to the "DOF HUD". Since we have the possibility to project diferent images on different pinholes, you could take the four images of the T-buffer and project them on four different pinholes, and let the eye decide where the focus should be.

So the result would be that you calculate an image for one pinhole, send it to the LCD, and light the corresponding lightsource. Calculate the image for next pinhole...

The problems here are:
You have to track the pupil to make sure that the active pinholes are inside it.
You must have nice point shaped light sources. I don't know how small you can make LEDs, but if it isn't small enough you could "enhance" it by putting a mask on top of it. The strength shouldn't be a problem since you actually could get pretty much all of the power into the eye, and even a LED can be quite bright then.
You need a realy fast LCD. It must have refresh rate of at least (eyes flicker limit) times (number of active pinholes). (say 75*4=300Hz)
And then, someone (else than me :) ) need to test if the focused light can damage something if pointed at the wrong place.

PS
I liked the description "poor mans holography".
DS