Nintendo's Switch 2 Upscaling Patent Explained: More Than Just DLSS?

[ocosmosinfinito]

Alright, here’s how it works: the T-239, based on the T-234 (Tegra Orin), brings some interesting tech to the table. It has the L2 cache for the GPU and a secondary cache tied to the Hub Interconnect Memory, known as the Cache System. On the Orin chip, this Cache System is 4MB, but it’s likely Nintendo is using 3MB to streamline cost and design.

Rendering is done in 64-pixel tiles, and these tiles need to fit into the L2 cache. The L2 cache is slower than L1, but it’s still fast enough for this task. If the tile count exceeds the L2 cache capacity, tiles spill over into the Cache System, which benefits from an automatic 2:1 compression, just like the LPDDR5x memory.

A 64-pixel tile requires 128 bytes of uncompressed storage for color data (assuming 4 bytes per pixel).

With 2:1 compression, each tile only needs 64 bytes.
With 3MB of Cache System, you can store 48, 000 tiles at once.


Now let’s look at how that translates into resolutions:

1. 1080p (1920x1080): This resolution has 33, 750 tiles (1920 / 64 * 1080 / 64).
It fits comfortably within the 3MB Cache System.


2. 1440p (2560x1440): This resolution requires 57, 600 tiles.

While it exceeds the L2 cache, most tiles will efficiently flow into the Cache System.
3. 2160p (3840x2160): This resolution needs 135, 000 tiles.

Not all tiles can fit simultaneously in the Cache System, but thanks to the 120GB/s LPDDR5x bandwidth and the 2:1 compression, the system can handle streaming the data for higher resolutions.

Now, here’s where DLSS comes into play. The LPDDR5x memory delivers 120GB/s, which is double the bandwidth of the Xbox One S (68GB/s). With that kind of bandwidth and aggressive compression, DLSS won’t be bottlenecked. The platform will likely render internally at lower resolutions, like 540p to 1080p, then upscale to 1080p, 1440p, or even 2160p using DLSS. Thanks to the efficient cache system and memory bandwidth, DLSS should operate smoothly, maintaining an excellent range of output resolutions between 1080p and 1440p, and even stretching to 2160p in some scenarios.

Even with a focus on efficiency and compact design, the T-239 has enough firepower to deliver solid performance, especially when leveraging DLSS. The combination of automatic compression, L2 and Cache System handling, and 120GB/s LPDDR5x bandwidth ensures that the platform can handle DLSS upscaling without hitting a memory bottleneck.

[tyty]

Alex's constant flip flopping in pronouncing 'patent' is the linguistic equivalent to dynamic resolution switching.

[BrocBois]

Alex's will shattered at the strength of rich's pronunciation of patent. He bent the knee.

[meetinfotech]

Timestamps
00:06 - Nintendo's new upscaling patent explores advanced image processing techniques.
02:13 - Nintendo's patent explores advanced upscaling methods beyond traditional DLSS technology.
04:13 - Nintendo's patent covers advanced upscaling techniques beyond traditional DLSS.
06:21 - Nintendo's upscaling patent focuses on efficiency for low-power devices.
08:32 - Nintendo's Switch 2 may utilize novel real-time upscaling technologies.
10:29 - Nintendo's new patent describes real-time image upscaling to reduce game file sizes.
12:22 - Patent reveals potential upscaling technology for future Nintendo hardware.
14:14 - Testing Switch 2's upscaling capabilities compared to PC.

[MatthewGarbett]

I think a lot of people don't understand is anyone can patent anything but the office only makes sure the paperwork is done correctly not that it's valid. It's the job of the company to defend it in court to determine in the court if it's valid or not.

[mattspeer]

Can see the official announcement now: Switch 2 with DLSS 4, now with the same performance as an RTX 4090... made possible only by AI.

[dahahaka]

Correction 1:55 CNNs do *not* break up the image into tiles, you have a Kernel (a square) that is moving pixel by pixel over the the screen, that's the convolution part, it would be like thousands of "tiles" overlapping with one pixel offset to each other, it's not a grid, as described in this patent.

[soraaoixxthebluesky]

3:47 I kinda get a glimpse of what they trying to achieve here is if they really use z-buffer to upscale the image. This reminds me of SPU based deferred rendering techniques where they use multi stage tiles (2 stages to be precise) to further improve the speed of the light calculation from world space to screen space. They use sub tiles to further refined the targeted pixels and where the pixel intersect in 3D space to activate culling.

As for upscaling with hybrid DLSS & basic algorithm to save CPU time, G-Buffer provide all the necessary information of:

1) What type materials we are dealing with? Diffuse or specular?

2) Their angle to global light source (a.k.a normal)

3) They distance from camera(z buffer) give the pixels level of priorities i.e far object might not get DLSS treatment etc. Multi stage tiles (screen space) and bounding hierarchy box (in 3D space) can also give another step of optimizing to further lay out which pixels deserve the DLSS treatment.

4) Level of transparency. Opaque object might get different treatment than transparent or semi transparent object.

5) Temporal data (motion vectors) like any AA or classic image reconstruction they use this to avoid ghosting artifacts. Drop sample counts or use native res when needed.

All these with eat the system memory but will save tons of compute times. I believe with at least 12GB of on board RAM they can pull this off. This is all hypothetical cause I don’t read the paper.

[StiffAftermath]

Some of this may involve the original Switch patent that talked about the co-processor (eg. The dock) and cloud to process the final output image.

[willx]

Rich is the only one that can say pay-tent.

[TheChivs]

Interesting Nvidia don't have any sort of dynamic DLSS solution as mentioned here. Would love to be able to set DLSS to 'Dynamic' along with a framerate target, and let the GPU scale the DLSS rendering resolution between the settings to keep the framerate locked at the target (if possible).

Hopefully this patent doesn't scupper any chance of that!

[stevoofd]

I like how creative the R&D gets to work with the limitations of a budget and power friendly handheld device while aiming for quality visual fidelity.

[Jeschitown]

The thing I like about Nintendo is they don't make false claims, what I mean is the switch 2 will be like twice or more powerful than the og switch which is natural but significant progression across the board it will be much more capable system. And yet Xbox and Sony always claim their new consoles are dozens of times more powerful than it's predecessor but yet the games don't look or reflect it, at least I didn't see it

[mateo]

The Brits say ‘PAY-tent”? You learn something new everyday

[jamesdavid]

Some of the comments are beyond unhinged, please look after your mental wellbeing in 2025, health is wealth.

[woody]

The feature to drop model quality to up frame rates is a "wholly new" feature because Nintendo is the only one with hardware weak enough to need that.

[RanmaEX]

3rd party support is gonna hit the ground running now that the console has DLSS

[syferz]

15:05 A couple issues with Rich's test, 1 the VRAM was 4GB at 96GB/s. Switch 2 will have 11GB for games at 120GB/s (or better when docked), so higher resolutions than 1080p really eat through that 4GB and end up becoming an issue especially when 4K is concerned, but worse than that is that 750MHz is going to be much too low of a clock for the docked model of the Switch 2, and while the RTX 2050 does have more Tensor cores to do the DLSS, DLSS tests show that most of the tensor cores are idle and use very little of their performance to do DLSS, so a low clock just makes the Tensor cores too slow to do the work for 4K, a higher clock on Switch 2 like 1.1GHz, could drastically improve the speed of the process, especially when combined with more RAM. I'd be very cautious with Rich's test, it was simply too limited to achieve the results and a RTX 3050 6GB would have done a much better job, though down clocking it reliably is an issue.

[erickmarquez]

Based on Nintendo's history, they have consistently had some of the most talented coders in the gaming industry since their inception. The original NES, despite being underpowered compared to other products at the time, released some of the most amazing games. This tradition has continued with its successors. Several games can be mentioned that, despite limitations in CPU, memory, and story, managed to impress the industry. Titles like Super Mario Bros., The Legend of Zelda, and Metroid set the standard for future gaming experiences.

Nintendo also amazed the industry with their game ports. For instance, Doom was successfully ported to the Super Nintendo Entertainment System (SNES), despite the hardware's limitations. Resident Evil 2 was celebrated for its impressive port to the Nintendo 64, managing to retain much of the original game's quality and atmosphere. Another notable example is Resident Evil 4, originally released on the GameCube, which received critical acclaim for its graphics and gameplay. Donkey Kong Country was ported to the Game Boy Color and Game Boy Advance, showcasing impressive graphics and gameplay despite the hardware limitations. Additionally, The Legend of Zelda: Ocarina of Time was masterfully ported to the Nintendo 3DS, enhancing graphics and controls while maintaining the original's charm. These ports demonstrate Nintendo's ability to innovate and deliver high-quality gaming experiences on various platforms.

Even with hardware constraints, Nintendo's innovative approach to game design and development has resulted in legendary titles that have captivated audiences worldwide and influenced game development across the industry. On the Nintendo Switch, exclusives like Super Mario Odyssey, The Legend of Zelda: Breath of the Wild, and Splatoon 3 have continued this legacy, offering unique and engaging experiences. Third-party ports such as The Witcher 3: Wild Hunt, No Man's Sky, Rocket League, Doom 2016, and its sequel, Doom Eternal, were both remarkable ports on the Nintendo Switch, maintaining high-quality graphics and performance.

Looking ahead, there are several potential games that would be amazing to see on the Switch 2. Notable examples include Cyberpunk 2077, known for its expansive open-world and immersive gameplay, and Red Dead Redemption 2, celebrated for its stunning graphics and rich storytelling. Other potential titles could be Elden Ring, a critically acclaimed action RPG, and Star Wars Jedi: Survivor, known for its epic narrative and thrilling combat. These potential ports would continue Nintendo's tradition of delivering exceptional gaming experiences, regardless of hardware constraints.

[splashmaker]

It would be interesting to see if they can have a lighter weight of the new transformer based DLSS 4 upscaling that has higher quality per ms than the older convolution models at that same ms frame time cost.