Machine Learning: WinML/DirectML, CoreML & all things ML

Lurkmass said:
Well it doesn't help your argument that the original paper is coming out against your claims again ...
Click to expand...
In what way does the quote from the paper "Like prior works on neural materials, we forgo explicit constraints on energy conservation and reciprocity relying on the MLP learning these from data." go against my argument "Unless something is very wrong with loss functions or training set, it should not exceed the range of values provided by the original BRDF in a meaningful way and there are methods to enforce these limits"? If anything, it states the same.

Lurkmass said:
I don't know how many I have to repeat this to you but this generated approximation from the neural network can't be mathematically integrated!
Click to expand...
Can you explain why the MLP can't be mathematically integrated, while the papers you quote say exactly the opposite? There are algorithms for numerical integration, but if the task is to get a certain response from the network, this doesn't seem necessary in most of the cases.

In 2D space, you can fit a function using the MLP. With ReLU activations, the MLP would represent a complex piecewise linear approximation of the original function, which you can integrate.

If you want to know the response of the MLP, you can plot your functions alongside the MLP to see how the MLP corresponds to the original function.

Lurkmass said:
Also the normalization workflow they mentioned is unimplemented and would be slower which defeats the main benefit of the original paper which was faster material evaluation ...
Click to expand...
Unimplemented because it wasn't necessary for the correct output, as per your own quote from the neural materials paper) Normalization in inference should cost next to nothing. I guess the paper was focused on the training.
 
Lurkmass said:
I don't understand why you're passing off a working theory as an insignificant subject when it serves a the core basis of what we're doing today in games ...
Click to expand...
The ideological purity of analytically conserving/reciprocating models has no value to me. It's just a sanity check.

If the contortions needed to jury rig purity into heavily compressed originally pure ubershaders produces worse output for the same budget what's the point? Close enough is good enough.

Why do you hate approximately conserving/reciprocating models so much when the errors can probably only be numerically detected with a full unbiased Monte Carlo render to convergence? Even then, they might not be visible.
 
OlegSH said:
In what way does the quote from the paper "Like prior works on neural materials, we forgo explicit constraints on energy conservation and reciprocity relying on the MLP learning these from data." go against my argument "Unless something is very wrong with loss functions or training set, it should not exceed the range of values provided by the original BRDF in a meaningful way and there are methods to enforce these limits"? If anything, it states the same.
Click to expand...
That's just a hand wavy confession of saying that your data driven (AI) model is incapable of providing a generalized solution in comparison to a theory (science) driven model ...
OlegSH said:
Can you explain why the MLP can't be mathematically integrated, while the papers you quote say exactly the opposite? There are algorithms for numerical integration, but if the task is to get a certain response from the network, this doesn't seem necessary in most of the cases.
Click to expand...
The advertised "Real-Time Neural Appearance Models" paper doesn't cover the implementation of mathematical integration. At best the paper makes a passing reference to another past work on that topic ...
OlegSH said:
In 2D space, you can fit a function using the MLP. With ReLU activations, the MLP would represent a complex piecewise linear approximation of the original function, which you can integrate.

If you want to know the response of the MLP, you can plot your functions alongside the MLP to see how the MLP corresponds to the original function.
Click to expand...
The difference being is that you can directly integrate this function while there's not many ways to apply a similar operation for a neural network's output ... (mathematically modeled equation vs predictive set of values)
OlegSH said:
Unimplemented because it wasn't necessary for the correct output, as per your own quote from the neural materials paper) Normalization in inference should cost next to nothing. I guess the paper was focused on the training.
Click to expand...
The normalization flow described in the other paper had evaluation timings measured on the order of HOURS .There's a reason why this new paper isn't a straightforward extension on neural control variates because it would otherwise defeat the main benefit of the new results which was higher performance ...

There's other problems too with neural BRDFs such as no support for displacement mapping (tessellation) and no support for scattering properties (translucency/participating media) in materials. PBR models are simply easier to go about reasoning (no black box) and extending (more complex material properties) too as well ...
 
Lurkmass said:
There's other problems too with neural BRDFs such as no support for displacement mapping (tessellation) and no support for scattering properties (translucency/participating media) in materials.
Click to expand...
Don't use displacement mapping, voxels suit neural models much better. Neural implementations for BSDF/BSSRDF all over the place.
PBR models are simply easier to go about reasoning (no black box) and extending (more complex material properties) too as well ...
Click to expand...
On the other hand for prefiltering the pure models need a specialized solution for every shader. The simplicity is only really there for Monte Carlo.

A trained black box can be trivially trained for prefiltering AND bake in real/physical geometry. Best pure solutions can do for geometry minification is try to estimate new parameters. A poor fit at best, a real object is more than a NDF, the neural solution is in some ways more physical then.

PS. you are right they found a tractable AA'd shader for glint rendering in Unreal Engine, not trivial.
 
MfA said:
There is no regression, the purity of PBR isn't what created progress, it was the predictability for the artists. The paying audience don't care if a PBR renderer can analytically go to white in a white furnace test at convergence. As long as it's white enough, it will be predictable enough for the artists and look good enough to the audience.

For some effects like glitter, PBR is still completely unfeasible with practical budgets. Nothing is sacred.
Click to expand...

Not anymore for glitter! There's a paper on how to do this pretty well in realtime (like actual realtime game today, not "20ms @1080p on a 4090" realtime). But I need to remember the actual word BRDF folk use for sparkly stuff if I'm going to find it (should you be interested).

It's not the one Unreal put in recently either as it's newer, the glints are stable across mips and converge to GGX as it gets minified, very cool looking.
 
MfA said:
Don't use displacement mapping, voxels suit neural models much better. Neural implementations for BSDF/BSSRDF all over the place.
Click to expand...
You do realize that supporting virtual geometry for skinned meshes is somewhat contingent on the requirement of displacement mapping support because it would be computationally prohibitive to manually transform that amount of data, right ? Hence why the biggest AAA console game engine (Unreal) is going about reimplementing displacement mapping support (Nanite Tessellation) and one of the advertised techniques with mesh shading involved a software tessellation scheme too ...
 
Last edited:
BTW, just so we are clear, when they say they don't support displacement they mean that they can't currently do relief mapping inside the shader. Displaced geometry is irrelevant to the shader, it affects the normal input and derivatives thereof but that happens outside the shader.

Pretty sure relief mapping isn't pure PBR to begin with.
 
MfA said:
BTW, just so we are clear, when they say they don't support displacement they mean that they can't currently do relief mapping inside the shader. Displaced geometry is irrelevant to the shader, it affects the normal input and derivatives thereof but that happens outside the shader.

Pretty sure relief mapping isn't pure PBR to begin with.
Click to expand...
@Bold That's not true ...

If you change the topology of an object from that of a cube to a sphere you also change the normal vectors as well for their surfaces! You now have infinitesimally many more normal vectors depending on the point of a sphere's surface as opposed to just having 6 normal vectors in total for a cube before. You encounter surface normals everywhere in PBR and you can't just use direct normal maps either lest you risk having mismatched normals which will generate incorrect results during BRDF evaluation. If displacement mapping truly never mattered for our materials then we never would've encoded this information as a texture in the first place or even if we did we would soon abandon such in favour of tighter encodings like a procedural algorithm or a function but that is not the case since we need to derive our surface normals from these displacement map textures for our BRDFs too ...

If they had truly meant that they couldn't support things like relief mapping or parallax occlusion mapping then they would've more precisely stated that their results aren't compatible with explicit derivatives or procedural texturing techniques ...
 
Lurkmass said:
If they had truly meant that they couldn't support things like relief mapping or parallax occlusion mapping then they would've more precisely stated that their results aren't compatible with explicit derivatives or procedural texturing techniques ...
Click to expand...
They kinda did, they said they tried to implement "Rendering Neural Materials on Curved Surfaces", which is just that. Then they said "we found that this approach is always outperformed by fixed-function ray marching", which I interpret as just ray tracing the displaced geometry and then shading.

PS. oops, they of course meant doing traditional texture space "displacement mapping" ala relief mapping and only then shading, instead of trying to roll it all into the neural network like the mentioned paper. Otherwise they wouldn't have said ray marching.
 
Last edited:
MfA said:
They kinda did, they said they tried to implement "Rendering Neural Materials on Curved Surfaces", which is just that. Then they said "we found that this approach is always outperformed by fixed-function ray marching", which I interpret as just ray tracing the displaced geometry and then shading.

PS. oops, they of course meant doing traditional texture space "displacement" mapping ala relief mapping and only then shading, instead of trying to roll it all into the neural network like the mentioned paper. Otherwise they wouldn't have said ray marching.
Click to expand...
That wasn't the extension they've implemented. It's this research paper (NeuMIP: Multi-Resolution Neural Materials) that they've attempted to build upon. Based on the wording, they're floating the idea of a neural based parallax occlusion mapping technique as an alternative to displacement mapping itself but that approach doesn't work for skinned virtual geometry and Nanite doesn't support explicit LoDs either to be compatible with POM ...
 
Last edited:
Pretty clearly a citation error, they mixed up the two papers (same primary author, different year). The one I mentioned is actually about displacement mapping.

Regardless, the fact they mention raymarching means they can do texture space "displacement mapping" before shading.
 
Last edited:
MfA said:
Pretty clearly a citation error, they mixed up the two papers (same primary author, different year). The one I mentioned is actually about displacement mapping.

Regardless, the fact they mention raymarching means they can do texture space "displacement mapping" before shading.
Click to expand...
Either way their conclusion was that none of these current approaches are fast enough whether you chalk this up to a citation error or not ...
 
It was slower than doing traditional texture space "displacement mapping" first (aka. ray marching). Is that a problem though? You can just do that, get your UV for intersection and throw it into the neural shader.

They have no pressing need for purity :p
 
MfA said:
It was slower than doing traditional texture space "displacement mapping" first (aka. ray marching). Is that a problem though? You can just do that, get your UV for intersection and throw it into the neural shader.

They have no pressing need for purity :p
Click to expand...
I assume our discussion is moot now since there's no standardized gfx API to do explicit matrix operations anymore. ;)
 
gamengen.github.io

GameNGen

Diffusion Models Are Real-Time Game Engines
gamengen.github.io gamengen.github.io
Abstract
We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality. GameNGen can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation. GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions. Conditioning augmentations enable stable auto-regressive generation over long trajectories.
Click to expand...
 
What surprises me is that it's possible to train a neural network to learn what "DOOM" looks like, and supposely generate convincing images with necessary arguments (I'd imagine with a history of operations, e.g. moving and firing).
It's not clear which they use to produce the video (only "a single TPU") but even TPUv5e is quite beefy, not to mention TPUv5p. Anway, this seems to me that Jen-Sun Huang's "every pixel could be AI generated in 10 years" comment is no longer a pipe dream.
 
You must log in or register to reply here.
Back
Top