Photon Mapping

chris1515

Legend
Supporter
It seems we are on the verge to have one photon mapping GPU maybe inside PS5 or if it is not PS5 maybe on an NVIDIA next iteration of RTX.

http://casual-effects.com/research/Mara2013Photon/Mara13Photon.pdf

Nvidia search about the different photon mapping method on a Geforce 670 in 2013, the best one is 2d tiles on a GPU

https://patents.google.com/patent/US9633471B2/en

Sony patent with hardware 2d tiles photon mapping, reading the patent it seems rasterizing is only used for direct illumination and visibility for shadow. Full indirect photon mapping lightning

http://fileadmin.cs.lth.se/cs/Education/EDAN30/papers/photonMappingTutorial-SIG08.pdf

It goes much faster than path tracing, sometimes two orders of magnitude faster. In SIGGRAPH 2008 they did a rendering demo needing 100 rays per pixel to render in 5 minutes in path tracing to be noise free, took 5 seconds and 10 rays per pixel in photon mapping without blur.

Advantage:
_ Much faster than path tracing/raytracing even simple Whitted raytracing and with better quality because it can render caustic. Photon tracing pass is like raymarching.
_ fully independent of geometry scene complexity
_ In 2D tile method resolution will be equal to what you will be able to render into the direct illumination solution rasterizing or raytracing. The limitation is the number of photons you generate during the photon tracing pass, the more you can trace, the best the quality will be.
_ Easy to integrate inside an existing rasterizing or raytracing pipeline
_ In 2D tile possibility to use tile rendering inside on-chip memory, I believe this what Sony is doing reading the patent
_ No need to do some denoising when the number of photons is too low the light blur, like into the REVISION 2019 raytracing 4k winning demo.

Problem
_ KD-tree construction is slow(probably generation with something like scene generation in Power VR GPU)
_ Can't do glossy effects like eye reflection or certain refraction. Can't do direct illumination. The two effects demand too much photon.
_ Difficult to parameter the number of photon for each surface, it kills photon mapping as a global method for offline rendering. This is only used for some effect like volume rendering where it is great and much faster than path tracing. Importance sampling can be a solution to this problem.

The year after the 2015 hardware 2D tile Photon mapping patent, Sony Interactive Entertainment hired an employee of Imagination/Power VR which did his master thesis on Progressive Photon mapping using CUDA in a GPU. He is a raytracing/Photon mapping specialist...

https://www.linkedin.com/in/carlovloet

https://carlovloet.wordpress.com/

On raytracing gem book there is a chapter about Photon Mapping on RTX with a 2080 TI but it does not use the 2D tile method.

https://github.com/ananthaks/DXR-PhotonMapper

A photon mapper on Titan X, slowest part kd tree construction. I read the Kun Zou 2008 paper about KD-tree construction on GPU I don't think this is fast enough, the GPU is only as good as a multicore CPU.

If someone resolve the KD-tree construction this a much better solution than raytracing and probably the only way to solve it is to use something fast enough and able to build BVH every frame like the scene hierarchy generator of Power VR but for KD-tree.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.118.6575&rep=rep1&type=pdf

photon map is decoupled from the geometric representation of the scene, making the method capable of simulating global illumination in scenes containing complex objects.

These objects do not need to be tessellated; they can be instanced, or even represented by an implicit function. Since
the method is based on a bidirectional simulation, it automatically adapts to illumination and view. Furthermore,
because the use of photon maps reduces noise and aliasing, the method is suitable for rendering of animations.

Photon mapping pass will be slower with more photon on a PS1 scene at same resolution than a scene of the Avatar movie, this is not a joke because this is fully decoupled from scene complexity, you can use tesselated, displacement shading or any form of complex geometry or primitive this will not be the problem and the limitation of the photon mapping pass. It will be the number of photons linked to the number of ray.

2D tile Photon mapping has less dependancy on resolution than other photon mapping method and probably other global illumination method. The resolution will be the one you can reach with the direct illumination rendering pass.

In offline rendering the biggest problem of Photon mapping it was too difficult to parameter automatically to know the number of photon you need to use for each surface and it took too much artist time . But the inventor of the method told it is possible to solve problem with importance sampling but I think offline rendering choose path tracing without doing more research about photon mapping only more power was needed after all. Artist time is too precious.
 
Last edited:
Maybe it's worth to add the incompatibility between PT and Photon Mapping has been resolved: https://cgg.mff.cuni.cz/~jaroslav/papers/2012-vcm/

Photon mapping pass will be slower with more photon on a PS1 scene at same resolution than a scene of the Avatar movie, this is not a joke because this is fully decoupled from scene complexity
But more complex geometry makes tracing photons more expensive as always?
Being independent of screen resultion also is an advantage that can be brought to almost any other technique as well. But i need to read about this 2D tile tech which is new to me.
 
Maybe it's worth to add the incompatibility between PT and Photon Mapping has been resolved: https://cgg.mff.cuni.cz/~jaroslav/papers/2012-vcm/


But more complex geometry makes tracing photons more expensive as always?
Being independent of screen resultion also is an advantage that can be brought to almost any other technique as well. But i need to read about this 2D tile tech which is new to me.

It seems reading all paper I read nothing just the number of photons is the limitation. I hope the next step is too solve the low geometry on realtime graphics from my point of view.

Edit: A guy with enough mathematical and physics knowledge explains me photon tracing pass is like raymarching tracing inside a volume. I was by intuition knowing it is like tracing inside a volume but he was knowing much better than me.
 
I don't know, I am very skeptical about the potential use of that tech for Ray tracing in a console.

Usually in those patents they identify every possible use cases, like for videogames, gaming console, computer, any devices and such. Here they only talk about 'image processing device', 'computer-readable recording medium' and 'computer program'.

Shouldn't it be more suited for camera recording devices and such ?
 
I think different people may have different ideas of what makes PM distinct from other techniques, personally i see it as a mix of multiple ideas for optimization, which where in the original paper:

Can approximate indirect diffuse by gathering nearby photons. This gathering can be done like the initial proposal of a kD tree just for the photons or using any other spatial data structure. If we do it in screenspace the process would be very similar to current denoising if not the same.

Can help Monte Carlo methods to find better ray directions, e.g. with a hint like: Nearby photon knows it is in shadow form light source X, so ignore that one. (At this pint we need somthing like PT and PM, likely that's too much to be interesting for realtime?)

Can do caustics easier than naive PT because it stats at the lights not at the eye. (But if we aim for a generalization of all RT algortihms we end up at works like the paper i've linked above, while 2080 Ti barely manages naive PT in Quake 2 - so likely that's not the focus yet.)



So i conclude it's mainly indirect diffuse that's interesting here for games.

The Sony patent (i do not really understand the language and refuse to try harder :) ) talks about 'an image processing method to construct virtual 3D space'. Maybe this just means 'build AS and do RT' but make it sound as familiar as a traditional draw call.
PM is technically RT, so i don't see anything fancy behind those words other than accelerating RT.
But maybe PM raises the need for flexibility in a good way: You need to bin photons to the scene, so maybe accelerating binning or opening up AS to be reused (adding region queries not just ray intersection). Could be all FF and blackboxed as well ofc. But could also be a set of very nice building blocks.

The point 'detect low luminance regions' is quite uncler to me. Maybe it is used to find regions that require more photons to become stable: Likely i get this wrong.

For the gathering they mention something like getting a reduced set of photons per pixel. I'm confused here too, but it all sounds like focus on the indirect diffuse thing.


A black box to accelerate just PM but nothing else sounds not reasonable to me - much less than RTX, which allows to implement PM, PT and whatever.
But if it would be just the building blocks they would not need to mention PM at all in the patent, so that's all a bit odd.

If someone resolve the KD-tree construction this a much better solution than raytracing and probably the only way to solve it is to use something fast enough and able to build BVH every frame like the scene hierarchy generator of Power VR but for KD-tree.
That's a bit off - firstly it does not matter much which form of spatial acceleration structure you use, and second Photon Mapping is raytracing, not something else. So both Path Tracing and Photon mapping require the same RT, and maybe you want to reuse acceleration structure silicon for both.
 
I think different people may have different ideas of what makes PM distinct from other techniques, personally i see it as a mix of multiple ideas for optimization, which where in the original paper:

Can approximate indirect diffuse by gathering nearby photons. This gathering can be done like the initial proposal of a kD tree just for the photons or using any other spatial data structure. If we do it in screenspace the process would be very similar to current denoising if not the same.

Can help Monte Carlo methods to find better ray directions, e.g. with a hint like: Nearby photon knows it is in shadow form light source X, so ignore that one. (At this pint we need somthing like PT and PM, likely that's too much to be interesting for realtime?)

Can do caustics easier than naive PT because it stats at the lights not at the eye. (But if we aim for a generalization of all RT algortihms we end up at works like the paper i've linked above, while 2080 Ti barely manages naive PT in Quake 2 - so likely that's not the focus yet.)



So i conclude it's mainly indirect diffuse that's interesting here for games.

The Sony patent (i do not really understand the language and refuse to try harder :) ) talks about 'an image processing method to construct virtual 3D space'. Maybe this just means 'build AS and do RT' but make it sound as familiar as a traditional draw call.
PM is technically RT, so i don't see anything fancy behind those words other than accelerating RT.
But maybe PM raises the need for flexibility in a good way: You need to bin photons to the scene, so maybe accelerating binning or opening up AS to be reused (adding region queries not just ray intersection). Could be all FF and blackboxed as well ofc. But could also be a set of very nice building blocks.

The point 'detect low luminance regions' is quite uncler to me. Maybe it is used to find regions that require more photons to become stable: Likely i get this wrong.

For the gathering they mention something like getting a reduced set of photons per pixel. I'm confused here too, but it all sounds like focus on the indirect diffuse thing.


A black box to accelerate just PM but nothing else sounds not reasonable to me - much less than RTX, which allows to implement PM, PT and whatever.
But if it would be just the building blocks they would not need to mention PM at all in the patent, so that's all a bit odd.


That's a bit off - firstly it does not matter much which form of spatial acceleration structure you use, and second Photon Mapping is raytracing, not something else. So both Path Tracing and Photon mapping require the same RT, and maybe you want to reuse acceleration structure silicon for both.

I did not say you don't need ray intersection hardware for doing Photon Tracing efficiently It is a two passes algorithm with a raytracing pass. Maybe they will do the ray intersection with other things. Maybe they will take Power VR technology for example. It is possible to license the technology

Edit: and when I talk about KD tree, It all about the KD tree Photon maps looking like a point cloud. Last year Sony was hiring for R&D people with GI knowledge and Volume rendering knowledge it was the two first point of the job offer.

Edit: In the Nvidia paper, it looks like exactly the same algorithm. The 2D tile part has much more explanation inside the Nvidia study. And this for Nvidia is the best Photon Mapping method on a GPU.

Edit2: The patent only looks like one pass the one describe inside Nvidia study. Nvidia study bypasses all the raytracing part. And there is no raytracing document reference inside the Sony patent. And they talk about another rendering part not describe inside the patent. The patent and the Nvidia study are only about the shading part. I think Sony will let other people doing the raytracing part.
 
Last edited:
I did not say you don't need ray intersection hardware for doing Photon Tracing efficiently It is a two passes algorithm with a raytracing pass. Maybe they will do the ray intersection with other things. Maybe they will take Power VR technology for example. It is possible to license the technology
Ok, it sounded like you would see PM as a faster alternative to RT so i pointed it out - did not want to sound smart.

I don't think they would license tech, but maybe that's the reason the patent only covers PM but not related RT?
Technically it should be possible to accelerate both RT and PM on a small chip because both is about AS and binning, and they had enough time.
 
Ok, it sounded like you would see PM as a faster alternative to RT so i pointed it out - did not want to sound smart.

I don't think they would license tech, but maybe that's the reason the patent only covers PM but not related RT?
Technically it should be possible to accelerate both RT and PM on a small chip because both is about AS and binning, and they had enough time.

Maybe they will try to do it themselves but it is a possibility now at least than Power VR open a licensing program.

EDIT: I think Nvidia will maybe add the possibility in another iteration of RTX because it is an implementation of the 2D tiles describe in Nvidia study...

EDIT2: and when I said decoupled of geometry this the photon map pass...

The photon map algorithm was developed in 1993–1994 and the first papers on the method were published in 1995. It is a versatile algorithm capable of simulating global illumination including caustics, diffuse interreflections, and participating media in complex scenes. It provides the same flexibility as general Monte Carlo ray tracing methods using only a fraction of the computation time. The global illumination algorithm based on photon maps is a two-pass method. The first pass builds the photon map by emitting photons from the light sources into the scene and storing them in a photon map when they hit non-specular objects. The second pass, the rendering pass, uses statistical techniques on the photon map to extract information about incoming flux and reflected radiance at any point in the scene. The photon map is decoupled from the geometric representation of the scene. This is a key feature of the algorithm, making it capable of simulating global illumination in complex scenes containing millions of triangles, instanced geometry, and complex procedurally defined objects.

The raytracing pass is dependant of scene complexity but it seems you can go with less ray than path tracing for example...
 
Last edited:
I think Nvidia will maybe add the possibility in another iteration of RTX because it is an implementation of the 2D tiles describe in Nvidia study...
Unlikely. First they will add HW BVH build to support more dynamic scenes, LOD and compatibility mesh shaders... as i hope. Would be super at least :)
And second they might add ray reordering if they don't have yet.
PM would benefit from both.

The raytracing pass is dependant of scene complexity but it seems you can go with less ray than path tracing for example...
Yeah, but you still need incoherent paths, and again each one provides only incomplete information so you need many.
But that's surely the main argument. Never implemented PM so i don't know how less rays it might be, but...

"It provides the same flexibility as general Monte Carlo ray tracing methods using only a fraction of the computation time."

Is this a claim from the time when the PM paper came out?
Veachs works came later and brought all those improvements on multiple importance sampling, bidirectional PT, metropolis light transport. And they made PT a whole lot faster.
Even if those improvements are not practical for low realtime ray budgets but PM is, PMs advantage over naive PT is that it combines multiple samples (or photons) - exactly what denoising enables as well.

So i remain doubtful. HW RT is one thing, but HW tuned to one certain algorithm out of many? Would be something new. We'll se...
 
Nvidia search about the different photon mapping method on a Geforce 670 in 2013

So Nvidia ditched the idea all the way back to Kepler, after testing and exploring this method, and never developed this tech any further even today, and went with another tech for a reason?
 
Unlikely. First they will add HW BVH build to support more dynamic scenes, LOD and compatibility mesh shaders... as i hope. Would be super at least :)
And second they might add ray reordering if they don't have yet.
PM would benefit from both.


Yeah, but you still need incoherent paths, and again each one provides only incomplete information so you need many.
But that's surely the main argument. Never implemented PM so i don't know how less rays it might be, but...

"It provides the same flexibility as general Monte Carlo ray tracing methods using only a fraction of the computation time."

Is this a claim from the time when the PM paper came out?
Veachs works came later and brought all those improvements on multiple importance sampling, bidirectional PT, metropolis light transport. And they made PT a whole lot faster.
Even if those improvements are not practical for low realtime ray budgets but PM is, PMs advantage over naive PT is that it combines multiple samples (or photons) - exactly what denoising enables as well.

So i remain doubtful. HW RT is one thing, but HW tuned to one certain algorithm out of many? Would be something new. We'll se...

https://aras-p.info/blog/2018/03/21/Random-Thoughts-on-Raytracing/

I have a suspicion that the above is probably not done “because patents”. Maybe Imagination has an imperial ton of patents in the area of ray reordering, and Nvidia has a metric ton of patents in all the raytracing research they’ve been doing for decades by now, and so on. And if that’s true, then indeed “just expose these bits to everyone” is next to impossible, and DXR type approach is “best we can do given the situation”… Sad!

From what I read for HW build BVH and ray reordering, they will probably need license some patent from Power VR. Reading the site of Imgtec patentable is a word used everywhere. I am not sure you can use HW build BVH or Kd tree without needed to pay Imgtec.

https://graphics.stanford.edu/courses/cs348b-00/course8.pdf

So Nvidia ditched the idea all the way back to Kepler, after testing and exploring this method, and never developed this tech any further even today, and went with another tech for a reason?

Because after collecting the photon you need to organize it inside a kd tree and this part is slow... If you can generate it fast enough the problem is solved.

The definition of photon mapping inside road to path tracing from Pixar

The photon mapping method [51, 52] is a three-pass method to com-
pute indirect illumination (including indirect diffuse illumination and
caustics):
1. Emit light particles (photons) from the light sources, trace them
through the scene, and store them in a point cloud when they hit
a surface (or get scattered in a volume).
2. Organize the photon point cloud into a global data structure
called a photon map—typically a kd-tree—which is independent
of surface complexity and parameterization.

3. Render using local photon density and power to determine indirect illumination.

The reason it fail as a general global method in offline rendering
Photon mapping has been used for very specialized effects in a few
movies. It was used to render a caustic from a whisky glass in FinalF Fantasy The Spirits Within (2001)—this was rendered with a plug-into the Maya renderer (and composited on top of RenderMan Reyesi images. Photon mapping was also used for rendering artistic photonb beameffects [50, 88] in volumes in Disney’s Tangled and Frozen

Even though photon mapping was built into renderers such as men-al ray and RenderMan, it was never very successful as a general globali illuminationsolution for movie production. We can only speculate why
it never “took off”, but one possible explanation is that users found it
hard to tune its parameters: how many photons should be emitted, how
many photons should be used for density estimation, and how many
final gather rays should be used?
Photon mapping has recently been incorporated into advanced ren-
dering techniques such as VCM/UPS (see Section 8.3), so photon map-
ping is probably—indirectly—still part of the future of movie rendering.

Edit:But I agree with you for correctness bidirectional path tracing mixed with photon mapping is the best solution after we are in vdieogames I think Photon mapping can be a solution before we have more power.

https://www.ci.i.u-tokyo.ac.jp/~hachisuka/starpm2013a/Combining_PM_and_BPT_annotated.pdf
 
Last edited:
So why Nvidia didn't bother with it since?

Because Imgtec seems to have patent for HW build BVH or KD-tree data structures in hardware you need to find some financial arrangements with Imgtec and it is officially possible since 3 May 2019 if I remember well.

This is probably the reason Nvidia use compute for build it BVH for raytracing currently and doesn't have ray reordering. Nvidia engineer are not stupid, they know they need to reorder the ray for coherent memory access.

Edit: Imgtec launched a raytracing GPU 4 years before Nvidia ;)

Edit: And read the photon mapper they tried to do on Titan X slowest part build this fucking KD-tree.

Edit3: I wait the SIGGRAPH 2008 document about the path tracing demo against photon mapping. We can see the type of path tracing used if this is pre or post Veach 1997 thesis.
 
Last edited:
This is a video from SIGGRAPH 2008 one of the guy Henrik Wann Jensen is the creator of Photon Mapping 02 hours 48 it is midnight here I will see it tomorrow.

 
Last edited:
Edit:But I agree with you for correctness bidirectional path tracing mixed with photon mapping is the best solution after we are in vdieogames I think Photon mapping can be a solution before we have more power.
Makes sense, but then you have to consider other alternatives as well like radiosity method which i am using: No need for a gathering step so no additional kD tree required. Iterative solver is a natural fit for anmiated scenes not just single images. Memory requirements justified by getting infinite bounces for free. Shading just texture lookup. Directionality can be added to support rough glossy specular. No noise or blotches.
It's surely faster, but harder to implement in both engines and tools. (Remember Enlighten implementation in Battlefield 3: They used static light mapped proxy geometry - that's more work than just RTX on.)

There are too much options to be certain, and picking one for FF HW would be a risky and limiting decission.
 
Makes sense, but then you have to consider other alternatives as well like radiosity method which i am using: No need for a gathering step so no additional kD tree required. Iterative solver is a natural fit for anmiated scenes not just single images. Memory requirements justified by getting infinite bounces for free. Shading just texture lookup. Directionality can be added to support rough glossy specular. No noise or blotches.
It's surely faster, but harder to implement in both engines and tools. (Remember Enlighten implementation in Battlefield 3: They used static light mapped proxy geometry - that's more work than just RTX on.)

There are too much options to be certain, and picking one for FF HW would be a risky and limiting decission.

Yes probably on PC side maybe less risky for a console hardware where the most important is performance for the next 5 years. If it is the Sony choice.
 
Makes sense, but then you have to consider other alternatives as well like radiosity method which i am using: No need for a gathering step so no additional kD tree required. Iterative solver is a natural fit for anmiated scenes not just single images. Memory requirements justified by getting infinite bounces for free. Shading just texture lookup. Directionality can be added to support rough glossy specular. No noise or blotches.
It's surely faster, but harder to implement in both engines and tools. (Remember Enlighten implementation in Battlefield 3: They used static light mapped proxy geometry - that's more work than just RTX on.)

There are too much options to be certain, and picking one for FF HW would be a risky and limiting decission.

One point for you this is against naive path tracing and the best case with caustics. But I was knowing this was a scene with caustics because I see some more complex scene with photon mapping this faster than path tracing but only 4 to 10 times times for example. Not bad but not as good than with caustic.
 
What makes PM "independant of scene complexity" is the second pass that sorts out all the photons statistically across many screen pixels. Effectivelly, that is exactly what new denoising techniches are doing in PT now. So, PM does need less rays than naive PT, but in real time, we are already doing sparse PT with a denoising pass, not naive brute-forcey PT, so the amount of rays needed by a hypothetical PM solution can't be much less. In short, modern PT learned a few tricks that were once exclusive to PM, evening them out along the way.
 
What makes PM "independant of scene complexity" is the second pass that sorts out all the photons statistically across many screen pixels. Effectivelly, that is exactly what new denoising techniches are doing in PT now. So, PM does need less rays than naive PT, but in real time, we are already doing sparse PT with a denoising pass, not naive brute-forcey PT, so the amount of rays needed by a hypothetical PM solution can't be much less. In short, modern PT learned a few tricks that were once exclusive to PM, evening them out along the way.

It means you don't need tensor core to doing it but you use other things. You trade tensor core against another hardware.

It is more a question of which one will have the more efficient die size.
 
Last edited:
Back
Top