That ambient occlusion did seem neat, until I actually started thinking about it a few hours later. And the problem is that the worst case scenario for that is unsolvable. As in, any and every time your temporal coherence well, isn't, then your game simply crashes performance wise because you are doing 256 rays (or whatever your max is) per pixel and there goes everything.
Examples:
That building collapses, Bugger!
A fast moving vehicle (or anything) eclipses the scene, Bugger!
You turn around really fast, Bugger!
A bunch of anything is moving on the scene almost at all, Bugger!
I.E. anything that moves relative to the camera is a "hole" and crashes performance. Heck, anything moving at all creates holes everywhere, as your previous solution is now potentially invalid. As presented the technique is totally useless for games. But could be neat for modeling scenes and getting results back quickly, and is of course a neat experiment.