Linux 7.3 will handle running out of VRAM far more gracefully

Kernel patches that rework how the AMD GPU driver copes with running out of video memory have been merged upstream and queued for Linux 7.3, according to a post published on 17 August on the pixelcluster GPU blog. It continues an earlier effort to improve VRAM management for games.

The physics of running out of VRAM

In theory, VRAM exhaustion should hurt performance, not stability: drivers let an application request more memory than the card holds and move the surplus to CPU RAM. That move is the bottleneck. On a PCIe 4.0 x16 link the bus carries slightly under 32 GiB/s, about 32.2 MiB per millisecond. At 30 FPS a frame has 33.3 ms, capping what the GPU can pull from evicted memory at roughly 1,075.5 MiB — barely over 1 GiB. If one frame needs more, 30 FPS is unreachable.

Caching absorbs part of the cost. Benchmarks on RDNA3 show identical latency for CPU- and VRAM-backed memory while a buffer fits in the 6 MB L2 cache; on a miss, CPU memory latency reaches around 2,400 cycles per access. A PCIe fetch costs about 7.3x an Infinity Cache hit and 4.6x a VRAM fetch.

The stability bug: a deadlock in kernel locking

Exhaustion also breaks stability. RADV reports "Not enough memory for command submission" when the kernel returns -ENOMEM, even though every allocation succeeded. The cause is a textbook ABBA deadlock: one submission tries to evict an allocation another has locked, while that second submission needs memory held by the first. The kernel can detect such deadlocks and retry, but the drm_exec helper was unused in TTM, the shared GPU memory management layer. Patchsets from 2024 never landed, so the developer rebased them and fixed the remaining bugs — a week spent chasing hangs that appeared three minutes into heavy VRAM contention.

Scanout buffers and 4 GiB evictions

Profiling showed most of the lost time sat in SDMA buffer moves rather than rendering. The worst case is the buffer scanned out to the display, which bypasses virtual memory and needs physically contiguous pages, while eviction is a simple LRU loop that ignores physical constraints. In practice up to 4 GiB of VRAM was evicted to make room for scanout images holding about 32 MiB of pixel data each — moving that alone costs at least 130 ms. Hard and soft throttle phases now aim to stop the resulting eviction ping-pong.

What it delivers

With those heuristics, Indiana Jones: The Great Circle requesting 9 GiB on an 8 GiB card averaged 19.6 ms per frame. Doubling the overcommit raised frametime variance, with spikes above 33.3 ms and an average near 29.8 ms. The kernel also honours VK_EXT_pageable_device_local_memory and vkSetDeviceMemoryPriorityEXT, letting applications rank what is safest to evict; vkd3d-proton already maps D3D12 residency APIs onto it. The work ships in SteamOS Stable and Preview.