Non-DLSS Image Reconstruction techniques? *spawn*

Discussion in 'Architecture and Products' started by sonen, Sep 10, 2020.

  1. sonen

    Newcomer

    Joined:
    Jul 13, 2012
    Messages:
    67
    Likes Received:
    69
    https://www.dsogaming.com/news/intel-and-ucsc-showcase-dlss-like-ai-upscaling-technique/

    Intel and UCSC have showcased a DLSS-like AI upscaling technique that will be further detailed at SIGGRAPH ASIA in December 20th. Intel and UCSC have used the Unreal Engine 4 Infiltrator Tech Demo in order to showcase this new AI upscaling image reconstruction tech.

    This new AI upscaling technique is called QW-Net. According to the teams, QW-Net is a neural network for image reconstruction, where close to 95% of the computations can be implemented with 4-bit integers.


    “This is achieved using a combination of two U-shaped networks that are specialized for different tasks, a feature extraction network based on the U-Net architecture, coupled to a filtering network that reconstructs the output image. The feature extraction network has more computational complexity but is more resilient to quantization errors. The filtering network, on the other hand, has significantly fewer computations but requires higher precision. Our network uses renderer-generated motion vectors to recurrently warp and accumulate previous frames. This produces temporally stable results with significantly better quality than TAA; a widely used technique in current games.”

    Unfortunately, though, this image reconstruction technique is for offline usage (at least for now). UCSC claims that an optimized implementation for real-time inference remains future
    work. In other words, we may never see this technique in games. Still, it’s fascinating witnessing alternatives to DLSS. It also shows how ahead of its time NVIDIA was with its image reconstruction
    technique.

    Concurrent to our work, Xiao introduced a reconstruction technique based on U-Net. Using an optimized inference implementation they reconstruct a 1080p image in 18 to 20 ms on a high-end GPU. In comparison, DLSS reconstructs a 4K image in under 2 ms. Both these approaches can reconstruct images at a higher resolution than the input render.”

    https://player.vimeo.com/video/456337582
     
  2. nAo

    nAo Nutella Nutellae
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,345
    Likes Received:
    174
    Location:
    San Francisco
    QW-Net doesn't reconstruct higher resolution images (i.e. supersampling without super resolution).

    See below for more details:

     
  3. TheAlSpark

    TheAlSpark Moderator
    Moderator Legend

    Joined:
    Feb 29, 2004
    Messages:
    21,965
    Likes Received:
    7,983
    Location:
    ಠ_ಠ
    That could be interesting for consoles, which lack tensor cores...
     
    PSman1700 and pharma like this.
  4. nAo

    nAo Nutella Nutellae
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    4,345
    Likes Received:
    174
    Location:
    San Francisco
    Although it does use the tensor cores.
     
  5. pjbliverpool

    pjbliverpool B3D Scallywag
    Legend

    Joined:
    May 8, 2005
    Messages:
    8,063
    Likes Received:
    1,665
    Location:
    Guess...
    So DLSS is ~10x faster on the same hardware? That's pretty crazy.
     
  6. iroboto

    iroboto Daft Funk
    Legend Regular Subscriber

    Joined:
    Mar 6, 2014
    Messages:
    11,992
    Likes Received:
    13,339
    Location:
    The North
    yea. That's what I think many of us were trying to say earlier in response to 'DirectML' as being an option; it's not the hardware or hardware support that is driving the speed. The DLSS model itself that is the IP that needs to be protected. It's not easy to replicate what Nvidia has done. 2ms is insane.
     
    Dictator, DavidGraham, xpea and 7 others like this.
  7. pjbliverpool

    pjbliverpool B3D Scallywag
    Legend

    Joined:
    May 8, 2005
    Messages:
    8,063
    Likes Received:
    1,665
    Location:
    Guess...
    Series S would make things interesting in this regard. The Series X has half the INT4/8 throughput of an RTX 2060 meaning 4k upscale would take -5ms using DLSS or an equivalent model. Still potentially worth it at 60fps if you're upscaling from 1080p to 4k.

    However the Series S with likely no more than 50% of the Series X throughout would be looking at ~10ms. Basically worthless at 60fps. So what happens if XSX does get a DLSS like solution and starts upscaling from 1080p to 4k? Where would that leave the S? I know the S would be targeting a lower resolution (1440p) but does that have a linear impact on the ML processing requirements? I'd guess not.
     
    sonen likes this.
  8. iroboto

    iroboto Daft Funk
    Legend Regular Subscriber

    Joined:
    Mar 6, 2014
    Messages:
    11,992
    Likes Received:
    13,339
    Location:
    The North
    Series S is designed with an advanced hardware scaler to 4K. I don't expect DLSS to ever be used on the S. Even for lower resolution.
    The resolution won't have an impact on model speed unfortunately. It shouldn't at least, but no one knows how Nvidia did it.
     
  9. pjbliverpool

    pjbliverpool B3D Scallywag
    Legend

    Joined:
    May 8, 2005
    Messages:
    8,063
    Likes Received:
    1,665
    Location:
    Guess...
    Indeed but if S is rendering at 1440p native, and a dev wants to use a DLSS type algorithm on the X to target an internal res of 1080p (or 1440p for that matter), that poses a problem. Rendering requirements on the S could potentially be higher than the X. They could always half the frame rate if X is targetting 60fps, but if its already at 30fps then the only route left is massive graphical compromises. Which seem to go against the principle of the S.
     
  10. iroboto

    iroboto Daft Funk
    Legend Regular Subscriber

    Joined:
    Mar 6, 2014
    Messages:
    11,992
    Likes Received:
    13,339
    Location:
    The North
    Yea. I see where you're going with this. I'm not sure what the options are for when they push both PS5 and XSX to the limit. Hopefully MS provides some tools there to help out (aka ML upres or something similar), but it's entirely possible that it's going to miss out on some things.

    I can only assume if they manage to get ML upres working on XSX. They can make a faster model for XSS.
    But using the same model on both, should run in the same time.
     
  11. pharma

    Veteran Regular

    Joined:
    Mar 29, 2004
    Messages:
    3,793
    Likes Received:
    2,682
    I guess GeForce Now, if offered as an option could potentially provide DLSS on the S.
     
    PSman1700 likes this.
  12. PSman1700

    Veteran Newcomer

    Joined:
    Mar 22, 2019
    Messages:
    3,445
    Likes Received:
    1,364
    So theres no need for tensor hardware for dlss?
     
  13. pharma

    Veteran Regular

    Joined:
    Mar 29, 2004
    Messages:
    3,793
    Likes Received:
    2,682
    No, just that the games already have DLSS support. Granted this is just a future possibility for consoles.
    The requirements for Windows are:
     
    PSman1700 likes this.
  14. Malo

    Malo Yak Mechanicum
    Legend Veteran Subscriber

    Joined:
    Feb 9, 2002
    Messages:
    7,902
    Likes Received:
    4,101
    Location:
    Pennsylvania
    But that's not providing DLSS on the S per se, it's just streaming games running on Nvidia hardware through the S.
     
  15. pharma

    Veteran Regular

    Joined:
    Mar 29, 2004
    Messages:
    3,793
    Likes Received:
    2,682
    I don't see any other way other than streaming given the S hardware setup.
     
  16. Malo

    Malo Yak Mechanicum
    Legend Veteran Subscriber

    Joined:
    Feb 9, 2002
    Messages:
    7,902
    Likes Received:
    4,101
    Location:
    Pennsylvania
    It's not going to happen at all. The only alternative is MS providing a shader-based AI reconstruction that's similar. The original comment was a "DLSS type algorithm", not DLSS specifically since that's Nvidia only.
     
  17. pharma

    Veteran Regular

    Joined:
    Mar 29, 2004
    Messages:
    3,793
    Likes Received:
    2,682
    True, if MS can provide a decent AI reconstruction process for the S that would be ideal. However, AFAIK there has been no indication that MS will do anything other than support DirectML.
     
    #17 pharma, Sep 11, 2020
    Last edited: Sep 11, 2020
  18. iroboto

    iroboto Daft Funk
    Legend Regular Subscriber

    Joined:
    Mar 6, 2014
    Messages:
    11,992
    Likes Received:
    13,339
    Location:
    The North
    If they are working on something, they have some time. Maybe 2 more years before the consoles will start needing it and another 3 years after that when it's mature and pushing the envelope with 30fps titles.
     
  19. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    16,260
    Likes Received:
    3,472
    Will console gamers accept 30fps? (especially in 5 years time) for a few years now anything less than a solid 60fps has been unacceptable to many in p.c space
     
    Cuthalu likes this.
  20. BRiT

    BRiT (>• •)>⌐■-■ (⌐■-■)
    Moderator Legend Alpha

    Joined:
    Feb 7, 2002
    Messages:
    17,275
    Likes Received:
    17,678
    They seem to be if they're into the PS5 exclusives.
     
    PSman1700 and Dictator like this.
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...