New Augmented Reality Demo for Gizmondo

[Lazy8s]

Joining that catapult game that was in development for Gizmondo which used augmented reality, a new, little demo showing a detailed character jogging smoothly in place has been made:

http://studierstube.icg.tu-graz.ac.at/handheld_ar/gizmondo.php

http://studierstube.icg.tu-graz.ac.at/handheld_ar/images/Giz_03.jpg

[Guden Oden]

That character sort of looks like Zorro. :D

Weird that someone would actually use a gizmondo for something impressive. What framerate is it running at?

[Shifty Geezer]

25fps. Looks much smoother than the EyeToy monster demo from E3, and I guess shows you don't need stupid amount of processing power to do AR.

Edit : It's actually an AR engine, with Gizmondo just as an example of a handheld console running it. ARToolkit, their library (available free it seems) also runs on phones and PDAs.

[hidefguy]

Nice find, Lazy8s - just loaded it onto my gizmondo and it works really well as an AR demo.

[Skrying]

Nice find, Lazy8s - just loaded it onto my gizmondo and it works really well as an AR demo.

Whoa, people actually have these? Lol.

[Kristof]

That character sort of looks like Zorro. :D


Isn't that a Quake MD2 model ?

K-

[hidefguy]

Whoa, people actually have these? Lol.

Something of a rarity, but I've had one on my shelf for quite a while, hoping to put it to some good use. Presumably there must be warehouses full of these devices somewhere, probably bought up from the liquidation sale. I tried quite hard both in person, at the Gizmondo E3 appearance the year before last, and by mail, to convince the gizmondo outfit of the benefits of opening up development on these devices, to no avail. I guess that they were hoping for PSP-scale sales and the same business model. It's really too bad, because Daniel Wagner's application shows just how powerful this device is.

[drgoldie]

hi this is Daniel, the author of the Giz demo...

i agree, the Gizmondo is a very powerful device.
i've rarely seen such a good camera and the 3D chip is blazin fast (although i don't like that it just has ~800KB texture memory).

we will maybe start some project using these devices...

Daniel

[Shifty Geezer]

Have you any insight into PSP and it's camera? Would you look to try to get your tech onto there too if not already making motions in that direction? And have you any plans for real applications for this tech or is it more a proof of concept, just to show it can be done?

[drgoldie]

i have no insight into any PSP things.
i would like to use it but as far as i know this is completely out of question - at least with the official setups. Sony is very, very strict about keeping the PSP a closed system. looking at what is going on in the PSP homebrew scene i even understand that a little bit. yet, i would have loved to see Sony actively support a homebrew scene instead of fighting it.

i didn't know there was a camera for the PSP. i'll have a look into how well it can be used using homebrew. maybe this is an interesting platform after all. i fear though that Sony would not like us to use it. does anybody have experiences with legal issues in that area?

about using the Gizmondo: this demo is ofcourse just to show that it can be done. we are always searching for new platforms to use for our research purposes and i believe the Gizmondo is definitely interesting. so yes, we do have plans, but this is all in a very early stage (might not go anywhere at all...)

bye,
Daniel

[Shifty Geezer]

The PSP camera isn't out yet; it's set to be released in September. I'm having difficulty thinking of applications for augmented reality as demo'd, cool as AR is!

Another point I notice is your system is far smoother than the PS3 implementation in Eye of Judgement. Do you have any theories as to why that is? Have you just developed a better, faster system? ;) That game would definitely benefit from the smoother updates.

[drgoldie]

i do not know that game, can you post a link to a video showing this effect?

about applications for AR: there are many, starting in medicine and manufactoring over to games (which is what i am concentrating more on) and to advertisement. one of the main problems AR faces today is the lack of inexpensive, robust hardware. that's the reason why we are working on devices such as the Gizmondo or Smartphones. these are inexpensive and ideally people already own them...

bye,
Daniel

[drgoldie]

ok, i looked into the videos at IGN and i'm quite impressed. this is one of the first times that a major game developer makes a nice, high quality (AAA) game that uses AR technology.

yet, i'm not sure how this game would be set up. would players have the TV box next to the table (positioned like a third player) and look at the TV while they place the cards on the table?

about the tracking: it looks pretty smooth to me and it obviously does not require those black borders that are required for ARToolKit. but then ARToolKit is quite old by now and a PS3 has a magnitude (or two...) more power than a Gizmondo has...

Daniel

[Lazy8s]

Strategy-action gameplay and wireless-networking multiplayer were a part of Catapult for Gizmondo.

http://fireant.tv/directory/episode/1108716

http://img69.imageshack.us/img69/6500/indextz6.jpg (http://imageshack.us) http://img69.imageshack.us/img69/3057/indexgu0.jpg (http://imageshack.us)

AR Tennis is a research project using a customized ARToolKit for exploiting the advanced features of smartphones for multiplayer gameplay, translating the movement tracking of the phones into control over the tennis rackets and feeding back tactile response for ball contact to the player with the phones' vibration function.

http://www.hitlabnz.org/route.php?r=prj-view&prj_id=35

[Shifty Geezer]

That game certainly looks different. I don't think it's particularly suited to a mobile though. As a mobile is small and...mobile!, using a camera based based game on a static viewpoint doesn't make much sense versus same game on a static console setup. What would be better is something like 3D objects superimposed on the real-world view and you point and shoot with the mobile device. Perhaps in the city you have virtual ducks being overlaid, as as you move the deivce the camera tracks movement and you've got to shoot the ducks?

[Zaphod]

virtual ducks being overlaid, as as you move the deivce the camera tracks movement and you've got to shoot the ducks?There are several games like that for Symbian smartphones, but thay haven't made much impact yet. Here's one (http://www.yewsoft.com/product.php?type=screens&productid=15). There are also other proof of concept apps (http://virtual.vtt.fi/multimedia/index_dm.html) around using the camera for interaction/AR. Here's another example (http://people.freenet.de/hskopp/wabbellab.html) with source available.

[Shifty Geezer]

There are several games like that for Symbian smartphones, but thay haven't made much impact yet. Here's one (http://www.yewsoft.com/product.php?type=screens&productid=15).I can understand why. Playing a mobile game at a restaurant table or busstop is ordinarily inconspicuous and I expect most would want to keep it that way, rather than waving their phones around!
There are also other proof of concept apps (http://virtual.vtt.fi/multimedia/index_dm.html) around using the camera for interaction/AR. Here's another example (http://people.freenet.de/hskopp/wabbellab.html) with source available.Is the fidelity enough for the motion tracking ball game? I'd have thought a camera would give a pretty rough interpretation of subtle movements, and you'd need unrealistic degrees of movement versus the real physical game.

The non-gaming apps make more sense to me. An idea I had yonks back was virtual museum exhibits, where you stand inside a virtual structure and use your mobile device as a window to view it. Perhaps a large blue room that could fit lots of visitors in with various markers on the wall to track, and the software determines where you are looking and renders the ancient building in that view chromakeyed to the background. That'd be a really nice advance for museums and work very well with large screen portables. Something like PSP ought to be ideal at that.

I'll add that the ARinterior software seemed rather slow and primitive. Once they've got the camera lens matched, they should be able to draw 3D at full rate over the top. They woud benefit greatly from taking a mirror-ball photo to get a spherical illumination map. Then at least the furniture would match the lighting and fit in rather than looking like a composite with 1990's vector graphics!

[drgoldie]

there are several games (one of the first i know is the Siemens Mosquito Hunt) for "low-end" mobile phones that use very simple pixel flow detection for camera tracking. these games usually asume a fixed camera location and thereby translate pixel movement of the camera directly into rotational changes of the camera (simple concept and works great).

a main problem most of these tech demos have is that the performance is often not good enough for real games - especially for fast-paced games such as tennis or shooters.

that's why i like the Gizmondo quite a lot at the moment. this is the very first time i got or saw something like this to work at a really high-fidelity frame rate (constantly above 25 fps). i'm quite sure that the PSP could be a great device for AR games too...

Daniel

[Zaphod]

The non-gaming apps make more sense to me. An idea I had yonks back was virtual museum exhibits, where you stand inside a virtual structure and use your mobile device as a window to view it. Perhaps a large blue room that could fit lots of visitors in with various markers on the wall to track, and the software determines where you are looking and renders the ancient building in that view chromakeyed to the background. That'd be a really nice advance for museums and work very well with large screen portables. Something like PSP ought to be ideal at that.Don't know about the fully virtual museum, but the virtual-assisted museum is certainly being experimented with. The advent of WIFI-enabled mobile devices with a camera and wireless audio will eventually see the replacement of information booths, cassette tours, and whatnot. Just take a picture of a symbol and have auxiliary audio, video, and AR content directly on the mobile. Market penetration for the devices isn't quite here yet, but it's not far off and such systems won't be all that expensive to implement.

[Graham]

AR Tennis is a research project using a customized ARToolKit for exploiting the advanced features of smartphones for multiplayer gameplay, translating the movement tracking of the phones into control over the tennis rackets and feeding back tactile response for ball contact to the player with the phones' vibration function.

http://www.hitlabnz.org/route.php?r=prj-view&prj_id=35

I had a play of this a while ago when it was still a reasearch project at the hitlab. It was interesting, but the hardware was definitely the weak point.
It was based on a modified version of AR Toolkit Plus (http://studierstube.org/handheld_ar/artoolkitplus.php) from memory. ARTK+ is basically a modification of the standard ar toolkit, adding a few (desperately needed) features, and supporting fixed point processing.

From what I know, the AR-tennis project received an nvidia-sponsored grant to pursue commercial development and application.

Overall to be totally honest, the AR toolkit is pretty bad. The code is a shocker, and it's tracking performance can be down right terrible. It is difficult to setup as well. But at least they are finally investing some proper money and time into getting it into a more commercially sound state, however licensing the toolkit is still a struggle (to be kind).There are other alternatives out there. There is the MXR toolkit (http://mxrtoolkit.sourceforge.net/), which is *much* smarter than the ar toolkit, and has *far* better tracking performance, however it's code is probably even worse, is bloated, is no longer updated, is buggy, and has performance issues.
Even the BBC has a secret project to do their own tracking toolkit, but I don't know the details. The BBC actually used the AR toolkit in a medical themed television programme a while back.


I'll add that the ARinterior software seemed rather slow and primitive. Once they've got the camera lens matched, they should be able to draw 3D at full rate over the top. They woud benefit greatly from taking a mirror-ball photo to get a spherical illumination map. Then at least the furniture would match the lighting and fit in rather than looking like a composite with 1990's vector graphics!


There are real-time implementations of this (mirror ball for ambient lighting). They look very good too, however the camera calibration has to be absolutely perfect. The hitlab have yet to update their site with information on the project however. Here is a paper (http://csdl.computer.org/comp/proceedings/ismar/2002/1781/00/17810279.pdf)on the same subject.


The non-gaming apps make more sense to me. An idea I had yonks back was virtual museum exhibits, where you stand inside a virtual structure and use your mobile device as a window to view it. Perhaps a large blue room that could fit lots of visitors in with various markers on the wall to track, and the software determines where you are looking and renders the ancient building in that view chromakeyed to the background. That'd be a really nice advance for museums and work very well with large screen portables. Something like PSP ought to be ideal at that.


Done already a few times. With varying levels of success. example (http://mmgrad.csuhayward.edu/apollobeyond/) that is slightly different but still in a museum - here is a video (http://mmgrad.csuhayward.edu/apollobeyond/tui.html).
I don't have any hand-held examples I can link to, but they are out there. Anything else gets into the 'too expensive' or 'too dangerous' or 'too unhygenic' department. So all the museum based ones are usually fixed.

The exhibits I've programmed have ended up being mostly single user activities, the exception is the AR-starwars project I did last year, which is on tour with the 'star wars: where imagination meets science' exhibit. (was in boston till a month or two back). - picture (http://hungryspoon.com/IMG_1502.jpg) - thats not me btw. This project worked VERY well for user interaction.

[Zaphod]

that's why i like the Gizmondo quite a lot at the moment. this is the very first time i got or saw something like this to work at a really high-fidelity frame rate (constantly above 25 fps).Trouble with the Gizmondo is that it's going the way of the Dodo. Here's a video of similar stuff that you've shown (http://www.hitlabnz.org/fileman_store/ar_tennis_hq.wmv) running on last gen Series 60 (papers available as well). Certainly less impressive graphics wise than your demo (but if Nokia get their way) probably a more viable platform going forward.

Edit: Didn't notice that Lazy8s had already linked that. Silly me...

[Zaphod]

It was based on a modified version of AR Toolkit Plus (http://studierstube.org/handheld_ar/artoolkitplus.php) from memory. ARTK+ is basically a modification of the standard ar toolkit, adding a few (desperately needed) features, and supporting fixed point processing.FWIW: Here's an earlier S60 port of the ARToolkit (http://staffwww.itn.liu.se/~andhe/UMAR/) from the same guys. I think that is the first smartphone AR-implementaion I read about.

[hidefguy]

Trouble with the Gizmondo is that it's going the way of the Dodo. ... Certainly less impressive graphics wise than your demo (but if Nokia get their way) probably a more viable platform going forward.

Yes, in fact it's already "gone", apart from Ebay and other private purchases, since Gizmondo Europe went bankrupt in January. I agree totally that currently, the best hardware bases are in Symbian - e.g. the N93, which we've got on order specifically for this purpose, but Windows mobile-based devices will be appearing within the next six months with more capable GPU's than the GF4500 in the Gizmondo.

[drgoldie]

Overall to be totally honest, the AR toolkit is pretty bad. The code is a shocker, and it's tracking performance can be down right terrible. It is difficult to setup as well. But at least they are finally investing some proper money and time into getting it into a more commercially sound state, however licensing the toolkit is still a struggle (to be kind).There are other alternatives out there. There is the MXR toolkit, which is *much* smarter than the ar toolkit, and has *far* better tracking performance, however it's code is probably even worse, is bloated, is no longer updated, is buggy, and has performance issues.

The ARToolKit source code is not beautiful. ARToolKitPlus is a little bit more cleaned up, but also not perfect. I don't know about the tracker of the MXR toolkit that much, but ARTag is to my knowledge the best marker tracking library at the moment (closed source and commercial though and no PDA/phone versions...).

FWIW: Here's an earlier S60 port of the ARToolkit from the same guys. I think that is the first smartphone AR-implementaion I read about.

The Hitlab's Symbian version of ARToolKit is basically just a very old version of ARToolKitPlus with Symbian specific modifications.

I agree totally that currently, the best hardware bases are in Symbian - e.g. the N93, which we've got on order specifically for this purpose, but Windows mobile-based devices will be appearing within the next six months with more capable GPU's than the GF4500 in the Gizmondo.

Can you back that up? Which Symbian hardware is better than the best current Windows Mobile hardware? I always had the impression that WinCE hardware was more powerful (which is also because WinCE wastes more resources than symbian...)

Which devices with new nVidia chipsets are gonna be released? I know only about the HTC Trilogy which will have a GoForce 5500. Any other?

bye,
Daniel

[Lazy8s]

As a starting reference, the Mobile Data Visualization site that's linked in hidefguy's signature provides an overview of many of the different 3D portable devices for the market.

http://mobile.sdsc.edu/devices.html

[hidefguy]

Can you back that up? Which Symbian hardware is better than the best current Windows Mobile hardware? I always had the impression that WinCE hardware was more powerful (which is also because WinCE wastes more resources than symbian...)

Which devices with new nVidia chipsets are gonna be released? I know only about the HTC Trilogy which will have a GoForce 5500. Any other?

By that I meant handsets with hardware graphics acceleration. There are already several symbian-based devices with hardware graphics acceleration available (or soon available) on the market that can be programmed in GLES (N93, M600, P990, W950i - the OMAP 2420 in the N93 appears to represent an especially powerful development platform), while there are (as far as I know) currently two accelerated devices with CE/WM5 that can be programmed (the X50/51v and the Gizmondo, both of which are EOL'd or defunct). Unfortunately, the Samsung SGH-i310 running WM5 with the GF4800 that's just started shipping has no hardware 3D drivers (neither GLES or D3DM), but the HTC Forseer (called Modeo in the US) will be released sometime later this year, which uses WM5 and a GF5500, though it's unclear, again, whether this will ship with 3D drivers. As far as I'm aware, the Trilogy doesn't use a GF GPU. I've been told to expect more Windows mobile based devices with programmable 3D drivers in the first half of next year.

[drgoldie]

at the moment we strongly concentrate on WM devices, but maybe it is time to look out more for Symbian devices too. thanks for opening my eyes...

Daniel

[Graham]

I'll have a look into ARTag. Looks good.
I know I have seen that site before... Can't remember when.

The major difference between the MXR toolkit and the ARtoolkit is the pattern design. MXR patterns require a black border that is not connected to the internal pattern, and also require a notch in one edge.

http://mxrtoolkit.sourceforge.net/IMAGES/3DLive_part.jpg

The AR toolkit simply finds corner points in the thresholded image, then links them up, trying to find squares. The problem of course is the entire maker must be visible. MXR, on the other hand, tracks the outside edge or internal edge of the border, or both.
The notch means the orientation of the maker is already known. One major limitation of square markers is that the pattern cannot be rotated, or even reflected (AR toolkit takes into account markers visible in a mirror). This greatly reduces the variability of the markers and also slows it down a lot. Simply loops the recognition code 8 times.
The image comparison code in the AR toolkit is also quite poor, it's 'confidence' rating for matching markers is simply a dot product of the corresponding pixels. The default minimum confidence is 0.5 if memory serves :/ MXR toolkit is smarter here too. Both do allow for full colour markers at least.

That said the mxr is just as difficult to work with.

[drgoldie]

I'll have a look into ARTag. Looks good.
I know I have seen that site before... Can't remember when.

ARTag is great. I believe it is currently the most advanced "square-marker" tracking system. unfortunately it is closed source and there is no PDA/Phone version.


The AR toolkit simply finds corner points in the thresholded image, then links them up, trying to find squares.

Actually it is the other way around. ARToolKit first finds the squares, then extracts the edges and uses those to calculate the corners. It then uses the edges for the first (initial guess) and the corners for the 2nd (refinement) pose estimation.


One major limitation of square markers is that the pattern cannot be rotated, or even reflected (AR toolkit takes into account markers visible in a mirror). This greatly reduces the variability of the markers and also slows it down a lot. Simply loops the recognition code 8 times.

This is only true for the standard ARToolKit template patterns. It is only slow when using many markers (e.g. 30+). ARToolKitPlus' id-based markers do not share this problem. One can have 4096 markers at a higher speed than a single template marker. The rotation is simply checked by ruling out bit-masks that are not valid...

Daniel

[Lazy8s]

A new AR application that lets construction and utility workers see the layout of underground gas, power, and cable lines of an area in the city is being prototyped.

http://studierstube.icg.tu-graz.ac.at/handheld_ar/vidente.php