ok about reconstruction techniques why cant game engines simply transform the final rendered frames or raster images into vectors simply since vectors are infinite it means the end of aliasing, is it really that hard nowadays we already have special raytracing cores and upscaling hardware units or ai cores, isnt it the obvious thing to do, have a special hardware unit that transforms frame buffer images to vectors with the help of ai,No cause that's framewide. I'd expect some local preservation vs. current methods something along lines of
Temporally Reliable Motion Vectors for Better Use of Temporal Information
We present temporally reliable motion vectors that aim at deeper exploration of temporal coherence, especially for the generally believed difficult applications on shadows, glossy reflections, and occlusions. We show that our temporally reliable motion vectors...link.springer.com
With "reliability" / "deeper exploration" lost if frameskip would occur so instead you lose out on local features.
Proposed upscaling technique - render vectors to pixels, convert to vectors, upscale and re-raster *spawn
- Thread starter msaki
- Start date
So you want something like adaptive upscaling, e.g. kind of DLSS, but with variable render / output resolution per tile?
That's a good idea, and would be possible already. VRS implements the necessary flexibility on variable resolution HW rasterization, and DLSS would need an update to respect this.
It adds some complexity to handle resolution mismatch on both temporal accumulation buffers and neighboring screen regions, but sounds not like a very hard problem.
But just to avoid confusion about semantics, 'GPU driven rendering' really only means to move gfx engine functionality from CPU to GPU. The goal is to do things like batching draw calls or culling entirely on GPU.
If we do some compute rasterizer which can render tiles at variable resolutions, many views, or something like that, the term 'compute' makes more sense because we go to solve fixed function limitations with a software alternative.
The current shortcomings are listed in the paper, in short there's a lot of smearing and arbitrary thresholds for this to work "reliably".
im not talking about motion vectors im talking about transforming an aliased raster image on the frame buffer into a vector image.. in realtime
Attachments
As far current thinking goes afaict offline renderers tried to do this "deeper exploration" to lessen load and omit artifacts with reprojection and I think I even saw disney's patent on it but lost sight.
You're aware that's exactly how GPUs currently work, right? The graphics are vectors, triangle models, which are rasterised to the display. No matter how much you zoom into the edge of a model, the edge is drawn smoothly, with whatever AA method is or isn't applied. If you want the rasterised version as vectors, just don't rasterise the projection of the 3D geometry. I don't know what one is supposed to do to then visualise that though if we're not rasterising.
Your proposal would give us this pipeline:
Vector data (triangles) to rasterized data (pixels)
Try to reconstruct vector data from pixels (which will be faulty)
Rasterize the reconstructed vector data again, but at higher resolution.
The faster pipeline will always be:
Initial vector data to final rasterized image, but already at the higher resolution we need.
So we're using the faster pipeline already. And if that's too slow we try to get away with lower resolutions.
Potential optimizations are variable resolution, so we use lower res where good enough (e.g. VRS), and spatial / temporal upscaling (DLSS).
The upscaling approach has the same goal than your idea, but it does not work on vector data but in image space.
Vector data is much more complex and is not cache friendly, opposed to image data. And since we're reconstructing from image data anyway, there is no reason to explore alternatives based on vectors.
were not talking the same thing, im not talking about geometry triangles or edges im talking about the final rendered image on a frame buffer that is stored as a bitmap to be transformed to a vector image instead this will kill aliasing since vectors have infinite resolutionsYou're aware that's exactly how GPUs currently work, right? The graphics are vectors, triangle models, which are rasterised to the display. No matter how much you zoom into the edge of a model, the edge is drawn smoothly, with whatever AA method is or isn't applied. If you want the rasterised version as vectors, just don't rasterise the projection of the 3D geometry. I don't know what one is supposed to do to then visualise that though if we're not rasterising.
To correct myself: I have forgotten about jittered samples. Each resolution level of tiles would need its own jittered projection if we still want temporal reconstruction.
The current HW can't do this, and it may be not worth to add. Extra costs on vertex processing and caching, and triangle setup.
Try to reconstruct vector data from pixels (which will be faulty) why?
Rasterize the reconstructed vector data again, why rasterise again cant it just be viewed/stored as a vector image why the need to raster? ive played flash games that used vectors and had infinite resolutions no matter how you zoomed they where just clean, not much of them this days though but it is possible i just dont understand the obsession to rasterize
yes ofcourse but ive seen alot of examples of raster to vector transformations that resolve to a cleaner vector image 1080p is enough really, with the age of ai and machine learning this is surely achievable if nvidia, microsoft, intel, amd or sony are interested, that to me is the only solution to aliasing otherwise well keep inventing reconstruction techniques, drs, vrs, upscaling, push for higher resolutions to no end.
Imagine we rasterize a cube. But it ends up very small, actually just a single pixel.
How do you go to get a cube out of single pixel? It's not possible.
So your vector data will contain the errors of the source data, including aliasing.
All we could do is to genrate 2D vector data directly from 3D geometry. And then we render this 2D data at any resolution without error.
This works, but it's orders of magnitudes slower than rasterizing the 3D data directly.
You would need to resolve the visibility problem during the conversation now, and because you use vector data, you can't use a simple and fast method such as the depth buffer.
Instead you have to clip 2D polygons (likely even with curved edges in from of splines) against each other. That's super difficult and slow.
Cherry picked examples.
I've worked as artist for years, and tried this often. To turn 2D logos into vectors, to upscale them.
The results were terrible. I always wasted my time and had to do it manually in the end.
Nowadays there surely are better methods, and ML could improve it as well.
But even if, you're still better off with implementing such methods in image space.
That's especially true if you can't make assumptions about the content.
I mean, you may get a good method to convert cartoons or line drawings (you won't, but let's assume it works). But then this method will fail on real world photos, and vice versa.
you wouldnt see a cube the size of a pixel even in current rendering techniques so thats an acceptable fault, all im saying is research should be done on this, the problem is the way graphics are rendered and maybe were too relaxed and accustomed to how the usual pipelines work but that is why were stuck with aliasing and other problems. its time more alternatives than triangles and raster images are more researched and applied.
The fault is not acceptable if you upscale by large factors. But my point is not that faults may be acceptable, but that your initial data is bad. Using another representation of this data will not fix this.
You can try to fill the missing pieces with other data. E.g., as a human artist i might know this pixel is likely meant to be a box, so i draw a box. That's what ML approaches can try to do, and it may work. Though, no vectors are needed. It would be just much harder to make ML work with vector data than with image data.
Regarding your thought on current gfx being maybe on the wrong track, i totally share this attitude. They make the impression they would care about optimization, and game devs pretend to have even the most expertise on optimization from all programming fields.
But imo, that's not true. They do just low level and close to the metal optimizations. But they forget about coming up with faster algorithms, and they are stuck at brute force solutions for everything related to gfx.
That said, i still insist your vector data is neither practical nor attractive. It's redundant complexity, giving us no new options we would not already have using image data, which is ideal.
Proper arguments for vector data would be compression, for example.
It's very easy to overlook slight inaccuracies in a still image -- in a moving image you would have constant wobbles and pops. Also, of course, this technique would need to run in ~<1ms to fit into the pipeline at 60fps.
I'm not sure why you're fixated on vectors -- you're bringing an interesting intuition (calculating geometry has infinite resolution, frame buffers have limited resolution) but as joej and shifty said earlier, we already have geometry transformed into screen space -- that's what the vertex shader does -- and we already have that converted into an infinitely sharp 2d image at a given resolution -- what's what rasterization does. We also have hardware features to evaluate those edges at a higher resolution for AA -- that's what MSAA does.
What trying to do that after rendering the image would get you is somewhat interesting to think about though:
You'd get vectors representing all of the information which wasn't captured in the initial geometry. Texture detail on surfaces, screen space effects like motion blur, etc. Even if a machine could generate it, that would be a lot of vectors, and it's unlikely the image would look exactly the same scaled up or down. We'd be generating vectors from a given resolution, so even if we did a perfect job, an edge which looked sharp at that res might actually be soft at a higher res in the source content... but we wouldn't have that info to encode into the vectors so we'd lose it during "vector upscaling".
Ultimately that sounds expensive to calculate, and maybe even expensive to just rasterize that image -- a vector drawing capturing all of the photorealistic texture detail of a final render might be almost as complex as the full 3d scene we sent to the gpu initially, with tons of overdraw for transparent gradients all over, so we might be doing just as much rasterization as we would be if we simply rendered the original scene at a higher resolution.
It's hard to imagine doing better in terms of giving vector geoemtry to the screen than we can by culling super well in 3d in the first place, and sending something as close to a "2d slice" of visible 3d triangles as possible, so technologies like nanite (or maybe just regular REYES path tracers) are a much better fit to get your desired data on screen at your desired sharpness.
I don't think that's quite fair, devs are focused on close to the metal stuff because cache coherency and gpu utilization as such huge insurmountable factors for our field that it's hard to look past them. Many state of the art techniques are overly complex algorithms in principle, but are more cache friendly. There's always a need for more boundary pushing and creative algorithms (Look at nanite, which is mostly a cool computer sciency-y tree algo) but game programming is mostly focused on the challenges that exist to get to the next level of complexity with the content we need to ship right now.
Last edited:
everything seems impossible until its tried or done all im saying is reconstruction and higher resolutions have no end. transforming to a vcector image is the final solution
I'll be more direct: It seems possible, it just seems objectively worse in every way than existing methods to generate an arbitrarily high resolution image -- more computation and a loss in visual quality. Maybe it would be cool to get 2d vectors out of the input geometry, so we can apply 2d vector resampling and smoothing operations before rasterization? But I can't imagine how to do that cheaply, and probably 3d subdivsion is better than that too. Doing it on the output render is a nonsense idea.
well raytracing was impossible and other rendering methods like dreams media molecule where, or ue5. Its always impossible until its done. Thats why humans do things.