Machine Learning: WinML/DirectML, CoreML & all things ML

April 1, 2024
It comes as little surprise that Windows continues to be a popular choice for professional workstations, and in 2023, about 90% of Puget Systems customers purchased a Windows-based system. Today, we’ll discuss the benefits and drawbacks of Windows-based workstations compared to Linux-based systems, specifically with regard to Stable Diffusion workflows, and provide performance results from our testing across various Stable Diffusion front-end applications.
...
We will discuss some other considerations with regard to the choice of OS, but our focus will largely be on testing performance with both an NVIDIA and an AMD GPU across several popular SD image generation frontends, including three forks of the ever-popular Stable Diffusion WebUI by AUTOMATIC1111.
...
A new option for AMD GPUs is ZLUDA, which is a translation layer that allows unmodified CUDA applications on AMD GPUs. However, the future of ZLUDA is unclear as the CUDA EULA forbids reverse-engineering CUDA elements for translation targeting non-NVIDIA platforms.

We decided to run some tests, and surprisingly, we found several instances where ZLUDA within Windows outperformed ROCm 5.7 in Linux, such as within the DirectML fork of SD-WebUI. Compared to other options, ZLUDA does not appear to be meaningfully impacted by the presence of HAGS.
 
They cant forbid "Clean Room Design" can they ?

The Google vs Oracle suit kind of established that using an API counts as fair use. So I think it's fair to make something completely compatible with CUDA, unless if there are some patents involved. One thing is that you'll need to make everything, including the toolings (e.g. the compiler and other support tools), otherwise you'll be under the purview of NVIDIA's licensing terms.
 
nVidia announced a new partnership with Microsoft:
Windows Copilot Runtime to Add GPU Acceleration for Local PC SLMs
Microsoft and NVIDIA are collaborating to help developers bring new generative AI capabilities to their Windows native and web apps. This collaboration will provide application developers with easy application programming interface (API) access to GPU-accelerated small language models (SLMs) that enable retrieval-augmented generation (RAG) capabilities that run on-device as part of Windows Copilot Runtime.

SLMs provide tremendous possibilities for Windows developers, including content summarization, content generation and task automation. RAG capabilities augment SLMs by giving the AI models access to domain-specific information not well represented in ‌base models. RAG APIs enable developers to harness application-specific data sources and tune SLM behavior and capabilities to application needs.

These AI capabilities will be accelerated by NVIDIA RTX GPUs, as well as AI accelerators from other hardware vendors, providing end users with fast, responsive AI experiences across the breadth of the Windows ecosystem.

The API will be released in developer preview later this year.
 
Neural BRDFs potentially not being "physically based" majorly misses the mark in terms of graphics technology integration. There's no compromising the very cornerstones of physically based shading models like energy conservation and reciprocity since not having these properties can contribute to over brightening/darkening of a rendered scene ...
 
It can just be a Disney Ubershader but with more dimensions and a different way of computing the output. Still physically based, just distilled to a MLP.
 
Neural BRDFs potentially not being physically based
It is physically based.

By exposing hardware accelerated tensor operations to ray tracing shaders, we show that it is possible to inline and execute the neural decoders efficiently inside a real-time path tracer. We analyze scalability with increasing number of neural materials and propose to improve performance using code optimized for coherent and divergent execution.

Our neural material shaders can be over an order of magnitude faster than non-neural layered materials. This opens up the door for using film-quality visuals in real-time applications such as games and live previews.

 
It can just be a Disney Ubershader but with more dimensions and a different way of computing the output. Still physically based, just distilled to a MLP.
It's about as much "physically based" as those other reflectance models much like Blinn-Phong where lighting artists are forced to constantly tweak the scene lighting to get their desired reflection output prior to the widespread introduction of the more physically grounded shading models during the last generation ...
It is not since the published paper immediately concedes this point at the very beginning of section 9 ...
Energy conservation and reciprocity. Because the neural material is only an approximate fit of the input material, it is not guaranteed to be energy conserving. Although we have not observed this to be a problem in our tests, this could become an issue for high albedo materials with high orders of bounces (e.g. white fur). Enforcing energy conservation would require the network to output in a form that is analytically integrable, or integrates to a known value. The latter can be achieved with normalizing flows (as in [Müller et al.2020]) at an increased evaluation cost. Our BRDF model is currently not reciprocal, but reciprocity could be enforced with the modified Rusinkiewicz encoding of directions [Zheng et al. 2021]. We opted for the Cartesian parameterization of directions that was more numerically stable in our experiments and yielded better visuals.
Just because the result is ray traced does not necessarily mean it is qualified to be "physically based" much less physically accurate. In fact doing ray tracing exposes the non-reciprocity flaw of the neural reflectance model more easily since the incoming/outgoing direction of the rays means that the lighting results are directionally dependent(!) in a scene. In a physically based shading model, all reflectance models are supposed to respect the Helmholtz reciprocity principle ...
 
It's about as much "physically based" as those other reflectance models much like Blinn-Phong where lighting artists are forced to constantly tweak the scene lighting to get their desired reflection output prior to the widespread introduction of the more physically grounded shading models during the last generation ...
It's as "physically based" as the shader graphs from which the MLP was derived. Loss functions ensure it adheres to the law of energy conservation, and even if they didn't, one could simply apply clamping to the network's output to maintain the energy conservation. This comparison to Blinn-Phong has no basis in reality and and is made up.
 
Just because the result is ray traced does not necessarily mean it is qualified to be "physically based" much less physically accurate. In fact doing ray tracing exposes the non-reciprocity flaw of the neural reflectance model more easily since the incoming/outgoing direction of the rays means that the lighting results are directionally dependent(!) in a scene. In a physically based shading model, all reflectance models are supposed to respect the Helmholtz reciprocity principle ...
What traditional shading algorithm has complete reciprocity taking into account numerical precision? Given the mountain of hacks rendering will always be, complaining about reciprocal approximation errors of a compression/interpolation system seems a bit premature without comparing it to all the other error sources. I doubt it matters.
 
It's as "physically based" as the shader graphs from which the MLP was derived. Loss functions ensure it adheres to the law of energy conservation, and even if they didn't, one could simply apply clamping to the network's output to maintain the energy conservation. This comparison to Blinn-Phong has no basis in reality and and is made up.
Shader Graphs have absolutely nothing to do in relation with physically based rendering as they're an artist material shading composition system whereby shaders are constructed with a node-based graph interface instead of an explicit coding interface. Also your idea of 'clamping' the neural network won't really achieve anything since their main problem is that there very few methods to integrate their outputs from them ...
What traditional shading algorithm has complete reciprocity taking into account numerical precision? Given the mountain of hacks rendering will always be, complaining about reciprocal approximation errors of a compression/interpolation system seems a bit premature without comparing it to all the other error sources. I doubt it matters.
If you're going to marginalize the lack of both energy conservation and reciprocity properties in comparison to numerical precision issues then I have the impression that you don't seem to understand the fundamental theory behind physically based rendering. An inaccurate model will generate incorrect results for some given inputs as opposed to an imprecise model where the least significant set of digits contribute to a much smaller amount of error in the result. The industry has moved on to physically based models for a very good reason since no amount of precision can fix prior problems with the older reflectance models ...

At that point if you have a wrong model, does it really matter if the results included ray tracing or not since we understand them to be inaccurate either way ? Going back to non-physically grounded material shading models would be a massive regression in terms of rendering consistency and correctness ...
 
An inaccurate model will generate incorrect results for some given inputs as opposed to an imprecise model where the least significant set of digits contribute to a much smaller amount of error in the result.
What's the limit on error (and bias thereof) which turns it from imprecise to incorrect?

The artists will still specify a physically based model. Why do they need to care that behind the scenes the MLP approximation has some error in reciprocity when on any practical budget they are swimming in hacks creating far more in the final image?
 
What's the limit on error (and bias thereof) which turns it from imprecise to incorrect?

The artists will still specify a physically based model. Why do they need to care that behind the scenes the MLP approximation has some error in reciprocity when on any practical budget they are swimming in hacks creating far more in the final image?
Artists can still specify inputs similar to that of physically based models but the lighting/material interaction evaluation is inherently invalid on deeper grounds because of it's very theory being wrong. Just because we use all sorts of approximations in real-time rendering doesn't serve as an excuse to straight up regress on our currently more rigorous and correct theories with physically based rendering as opposed to some euphemism like neural rendering ...
 
Shader Graphs have absolutely nothing to do in relation with physically based rendering as they're an artist material shading composition system whereby shaders are constructed with a node-based graph interface instead of an explicit coding interface. Also your idea of 'clamping' the neural network won't really achieve anything since their main problem is that there very few methods to integrate their outputs from them ...
They do, because graphs also define the emissive materials, so any neural network training on emissive materials would require the NN to learn the emissive properties of the material as well. This cannot be considered as breaking the conservation of energy.

However, you're grasping at straws with your claims that neural materials break the conservation of energy. First, the MLP would learn to approximate the BRDF from a training set based on this function. Unless something is very wrong with loss functions or training set, it should not exceed the range of values provided by the original BRDF in a meaningful way and there are methods to enforce these limits. Second, in the neural materials paper you mentioned, the authors explicitly stated that energy conservation wasn't a problem with the materials they tested. They also discussed the training methods, such as normalization, used to address the corner cases. The quote you highlighted above contradicts your claim that people with academic degrees are unaware of the basic laws of physics typically studied in high school.
 
Artists can still specify inputs similar to that of physically based models but the lighting/material interaction evaluation is inherently invalid on deeper grounds because of it's very theory being wrong. Just because we use all sorts of approximations in real-time rendering doesn't serve as an excuse to straight up regress on our currently more rigorous and correct theories with physically based rendering as opposed to some euphemism like neural rendering ...
There is no regression, the purity of PBR isn't what created progress, it was the predictability for the artists. The paying audience don't care if a PBR renderer can analytically go to white in a white furnace test at convergence. As long as it's white enough, it will be predictable enough for the artists and look good enough to the audience.

For some effects like glitter, PBR is still completely unfeasible with practical budgets. Nothing is sacred.
 
They do, because graphs also define the emissive materials, so any neural network training on emissive materials would require the NN to learn the emissive properties of the material as well. This cannot be considered as breaking the conservation of energy.
An emissive material is just a surface that also serves as a light source, nothing more and nothing less. They pose no relation to a neural network's guarantee that their generated output is Integrable or not ...

All of your attempts to overcomplicate your explanations isn't helpful to the public in understanding the problem statement at hand and trying to draw out relationships between the different terms when there aren't any only furthers veers you more off way ...
However, you're grasping at straws with your claims that neural materials break the conservation of energy. First, the MLP would learn to approximate the BRDF from a training set based on this function. Unless something is very wrong with loss functions or training set, it should not exceed the range of values provided by the original BRDF in a meaningful way and there are methods to enforce these limits. Second, in the neural materials paper you mentioned, the authors explicitly stated that energy conservation wasn't a problem with the materials they tested. They also discussed the training methods, such as normalization, used to address the corner cases. The quote you highlighted above contradicts your claim that people with academic degrees are unaware of the basic laws of physics typically studied in high school.
@Bold Well it doesn't help your argument that the original paper is coming out against your claims again ...
Our main focus is on developing a system that fits the aforementioned criteria. Like prior works on neural materials, we forgo explicit constraints on energy conservation and reciprocity relying on the MLP learning these from data.
I don't know how many I have to repeat this to you but this generated approximation from the neural network can't be mathematically integrated! Also the normalization workflow they mentioned is unimplemented and would be slower which defeats the main benefit of the original paper which was faster material evaluation ...

LOL @ the idea that some high school physics class is comparable to the more rigorous post-secondary institution treatment of classical optical physics involving multidimensional integrals ...
There is no regression, the purity of PBR isn't what created progress, it was the predictability for the artists. The paying audience don't care if a PBR renderer can analytically go to white in a white furnace test at convergence. As long as it's white enough, it will be predictable enough for the artists and look good enough to the audience.

For some effects like glitter, PBR is still completely unfeasible with practical budgets. Nothing is sacred.
The foundations of PBR such as energy conservation and reciprocity did indeed make art more predictably physically grounded. I don't understand why you're passing off a working theory as an insignificant subject when it serves a the core basis of what we're doing today in games ...

More complex material features like glints can absolutely be supported in a PBR framework like UE5's Substrate system. If you truly mean that nothing is uncompromising then I assume that includes ray tracing as well too then ? If we're going to put limits inherent model accuracy limits to material shading then I presume it's fine to do the same in other areas of rendering such as global illumination ?
 
Back
Top