NVIDIA open kernel module on a Thunderbolt eGPU: Xid 79, driver blocking and power management

Parent: Thunderbolt eGPU on Linux for local LLM inference · Published reference · snapshot 2026-09-08 · skill devops-linux-internals/references/nvidia-open-kernel-modules-blackwell-linux.md

Also known as: 610.57.04, NVreg_DynamicPowerManagement, Xid 143, Xid 79, blackwell linux driver, gsp firmware, install nvidia /bin/false, modprobe --ignore-install, modules-load.d, nvidia open gpu kernel modules, nvidia-driver-610-open, nvidia-persistenced, rtx 50 series linux, sm_120 cuda, snd_hda_intel

↓ Facts as markdown all context files

Running an RTX 50-series (Blackwell) GPU headless for CUDA on Ubuntu 26.04 — why Blackwell needs NVIDIA's open kernel modules and GSP firmware, the 580/595/610/615 driver branches, DKMS versus Canonic

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

NVIDIA open kernel module on a Thunderbolt eGPU: Xid 79, driver blocking and power management

Running an RTX 50-series (Blackwell) GPU headless for CUDA on Ubuntu 26.04 - why Blackwell needs NVIDIA's open kernel modules and GSP firmware, the 580/595/610/615 driver branches, DKMS versus Canonical-signed prebuilt modules, the module parameters that matter over a Thunderbolt tunnel, and the sm_120 status of CUDA, PyTorch, llama.cpp, vLLM and Ollama. [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#nvidia-open-kernel-module-on-a-thunderbolt-egpu-xid-79-driver-blocking-and-power-management]
--- name: nvidia-open-kernel-modules-blackwell-linux title: NVIDIA open kernel modules, GSP firmware and Blackwell (RTX 50) headless compute on Ubuntu 26.04 / kernel 7.x description: >- TRIGGER: RTX 50-series / Blackwell (GB20x) on Linux for compute; why the -open driver is mandatory; driver branches 580/595/610/615 (open vs proprietary vs -server); Ubuntu packaging (nvidia-driver-610-open, nvidia-dkms-610-open, linux-modules-nvidia--open signed vs DKMS); DKMS rebuild on kernel upgrade, pin/rollback; GSP firmware; NVreg_ params for a Thunderbolt eGPU (DynamicPowerManagement, PreserveVideoMemoryAllocations, EnableGpuFirmware, EnablePCIeGen3); nvidia-persistenced; headless bring-up; CUDA 12.8/13.x vs driver 'CUDA UMD 13.3'; sm_120 in PyTorch/llama.cpp/vLLM/Ollama. SKIP: Windows; pre-Turing GPUs; desktop/Wayland/gaming issues; AMD/Intel GPUs; the bolt stack itself. verified_as_of: 2026-09-24 --- [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#nvidia-open-kernel-module-on-a-thunderbolt-egpu-xid-79-driver-blocking-and-power-management]

NVIDIA open kernel modules + GSP for Blackwell headless compute (Ubuntu 26.04, kernel 7.x)

Verified-as-of 2026-09-24. Every substantive claim is tagged [SOURCED <url>], [INFERRED] or [UNVERIFIED]. Parameter names and version numbers are stated only where a primary source shows them; runbook placeholders that are not source-backed are flagged inline. [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#nvidia-open-kernel-modules-gsp-for-blackwell-headless-compute-ubuntu-2604-kernel-7x]
Reference case (the "reference box", 2026-09-24): Ubuntu 26.04.1 · kernel 7.0.0-34-generic · nvidia-driver-610-open 610.57.04-0ubuntu0.26.04.3 via DKMS (builds on 7.0.0-27/31/34) · RTX 5080 (GB203) in a Thunderbolt 4 (TB4) eGPU (external GPU) on an Intel NUC 15 Pro mini PC · Intel Arrow Lake iGPU on xe/i915 · nvidia-smi header reports KMD 610.57.04 / CUDA UMD 13.3 (KMD = kernel-mode driver, i.e. the kernel modules; UMD = user-mode driver, i.e. libcuda.so; see §3) · Ollama's llama-server (its own bundled process, not upstream llama.cpp's, which §14 covers; see also §13 step 8) [INFERRED] already runs on it · boot autoload blocked (install nvidia /bin/false in /etc/modprobe.d) and a systemd unit ordered after bolt.service loads modules with modprobe --ignore-install · /etc/modprobe.d/nvidia-egpu-pm.conf = NVreg_DynamicPowerManagement=0x00 NVreg_PreserveVideoMemoryAllocations=0 · apt-installed CUDA userspace is 12.4 (libcudart12 12.4.127). [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#nvidia-open-kernel-modules-gsp-for-blackwell-headless-compute-ubuntu-2604-kernel-7x]

1. Two kernel-module flavors; Blackwell only works with the open one

"The open flavor of kernel modules supports Turing and later GPUs." … "Blackwell and later are only supported by the open kernel modules." … "The proprietary flavor supports the GPU architectures Maxwell, Pascal, Volta, Turing, and later GPUs until Blackwell." [SOURCED https://download.nvidia.com/XFree86/Linux-x86_64/580.65.06/README/kernel_open.html] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#1-two-kernel-module-flavors-blackwell-only-works-with-the-open-one]
"The open kernel modules cannot support GPUs before Turing, because the open kernel modules depend on the GPU System Processor (GSP) first introduced in Turing." [SOURCED same] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#1-two-kernel-module-flavors-blackwell-only-works-with-the-open-one]
"We recommend the use of open kernel modules on all GPUs that support it." and, in the 610 README, "Installation will default to the open flavor of kernel modules." [SOURCED https://download.nvidia.com/XFree86/Linux-x86_64/610.57.04/README/kernel_open.html] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#1-two-kernel-module-flavors-blackwell-only-works-with-the-open-one]
Features that only the open flavor supports: Confidential Computing, GPUDirect Storage, HMM, CPU affinity, DMABUF. [SOURCED 610 README kernel_open.html] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#1-two-kernel-module-flavors-blackwell-only-works-with-the-open-one]
Practical consequence: nvidia-driver-610 (proprietary) on an RTX 5080 cannot bind the GPU; only -open packages are valid. [INFERRED from the above] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#1-two-kernel-module-flavors-blackwell-only-works-with-the-open-one]

2. GSP firmware is the control plane of the open driver

"Some GPUs include a GPU System Processor (GSP) which can be used to offload GPU initialization and management tasks." … "The GSP firmware will be used by default for all Turing and later GPUs." … "Firmware files gsp_*.bin are installed in /lib/firmware/nvidia/610.57.04/." [SOURCED https://download.nvidia.com/XFree86/Linux-x86_64/610.57.04/README/gsp.html] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#2-gsp-firmware-is-the-control-plane-of-the-open-driver]
The README documents NVreg_EnableGpuFirmware=0 / =1 as the switch. [SOURCED same] Because the open modules "depend on the GSP", =0 is not a usable configuration for the open flavor / Blackwell - leave it at default. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#2-gsp-firmware-is-the-control-plane-of-the-open-driver]
Kernel modules, GSP firmware, and user-space must be version-matched: "The kernel modules built here must be used with GSP firmware and user-space NVIDIA GPU driver components from a corresponding 615.71.09 driver release." [SOURCED https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/README.md] (README main was at 615.71.09 on 2026-09-24; the 610.57.04 tag carries the identical sentence for 610.57.04 [SOURCED https://github.com/NVIDIA/open-gpu-kernel-modules/commit/main via search]). [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#2-gsp-firmware-is-the-control-plane-of-the-open-driver]
On Ubuntu the firmware ships in a versioned package (nvidia-firmware-610-<version> binary from the nvidia-graphics-drivers-610 source). [SOURCED https://launchpad.net/ubuntu/+source/nvidia-graphics-drivers-610] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#2-gsp-firmware-is-the-control-plane-of-the-open-driver]
Failure signature when GSP cannot boot over a flaky eGPU link: kgspBootstrap_GH100: GSP-FMC reported an error while attempting to boot GSP: 0xffffffff alongside Xid 79. [SOURCED https://github.com/NVIDIA/open-gpu-kernel-modules/issues/974] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#2-gsp-firmware-is-the-control-plane-of-the-open-driver]

3. KMD vs CUDA UMD (what the 610 `nvidia-smi` header means)

From the 610 series the nvidia-smi header prints a separate KMD Version (kernel modules) and CUDA UMD Version (user-mode libcuda.so, which ships in the driver package, not in the toolkit); earlier releases printed a single "Driver Version / CUDA Version". [SOURCED https://github.com/MakazhanAlpamys/Soup/issues/827 , https://github.com/unslothai/unsloth/issues/5812 - third-party parsers documenting the format change; NVIDIA's nvidia-smi manual page did not expose the header text via fetch] Confirmed on the reference box (KMD 610.57.04 / CUDA UMD 13.3). [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#3-kmd-vs-cuda-umd-what-the-610-nvidia-smi-header-means]
"CUDA UMD 13.3" = the driver can run applications built against any CUDA 13.x toolkit up to 13.3 with full feature coverage; 13.4-built binaries run via minor-version compatibility except for features that need R615 (see §6). [INFERRED from the compatibility rules in §6] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#3-kmd-vs-cuda-umd-what-the-610-nvidia-smi-header-means]

4. Driver branch taxonomy (NFB / PB / LTSB) and which branches are current in 2026

"New Feature Branch (NFB): Major feature release … targeted towards early adopters who want to evaluate new features." "Production Branch (PB): … qualified for use in production for enterprise/data center GPUs. Bug fixes and security updates are provided for up to 1 year." "Long Term Support Branch (LTSB): A production branch that will be supported and maintained for a much longer time … 3 years." Cadence: NFB at least every 3 months, PB ~every 6 months, LTSB at least once per hardware architecture. [SOURCED https://docs.nvidia.com/datacenter/tesla/drivers/driver-lifecycle.html] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#4-driver-branch-taxonomy-nfb-pb-ltsb-and-which-branches-are-current-in-2026]
580 = the branch CUDA 13.0 launched on (13.0 GA requires ≥580.65.06). [SOURCED https://docs.nvidia.com/cuda/archive/13.0.0/cuda-toolkit-release-notes/index.html] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#4-driver-branch-taxonomy-nfb-pb-ltsb-and-which-branches-are-current-in-2026]
595 = current production/stable branch: "NVIDIA 595.58.03, the first stable version for the new 595 production branch" (March 2026); 595.91.07 released 2026-08-03 as the "latest recommended stable driver". [SOURCED https://9to5linux.com/nvidia-595-linux-graphics-driver-released-as-latest-production-branch-version , https://www.gamingonlinux.com/2026/08/nvidia-stable-driver-595-91-07-and-new-feature-driver-610-57-04-released-for-linux/] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#4-driver-branch-taxonomy-nfb-pb-ltsb-and-which-branches-are-current-in-2026]
610 = new feature branch: 610.57.04 released 2026-08-03 ("driver version … for the new feature branch"); data center notes list it as Linux 610.57.04 / Windows 610.88, "CUDA Toolkit 13: 13.x". [SOURCED gamingonlinux above; https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-610-57-04/index.html] 610.57.04 "fixes a bug that could cause suspend and resume to fail on systems with runtime D3 (RTD3) power management enabled, and a bug causing DKMS kernel module builds to fail after installing with nvidia-installer". [SOURCED https://www.phoronix.com/news/NVIDIA-610.57.04-Linux-Driver] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#4-driver-branch-taxonomy-nfb-pb-ltsb-and-which-branches-are-current-in-2026]
615 = next NFB, already the main of open-gpu-kernel-modules (615.71.09) and the branch CUDA 13.4 "requires". [SOURCED README main; https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#4-driver-branch-taxonomy-nfb-pb-ltsb-and-which-branches-are-current-in-2026]
Whether 580 is formally an LTSB was not verifiable from a fetched page. [UNVERIFIED] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#4-driver-branch-taxonomy-nfb-pb-ltsb-and-which-branches-are-current-in-2026]

5. Ubuntu's three packaging axes: branch × flavor × module-delivery

Flavor suffixes: nvidia-driver-NNN (proprietary), nvidia-driver-NNN-open, nvidia-driver-NNN-server / -server-open (Enterprise Ready Driver). "Unified Driver Architecture (UDA)" drivers are "recommended for the generic desktop use"; ERD -server drivers are "recommended on servers and for computing tasks". [SOURCED https://ubuntu.com/server/docs/how-to/graphics/install-nvidia-drivers/] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#5-ubuntus-three-packaging-axes-branch-flavor-module-delivery]
Module delivery: prebuilt, Canonical-signed linux-modules-nvidia-NNN[-open]-<flavour> from the linux-restricted-modules source, or DKMS via nvidia-dkms-NNN[-open]. "We don't recommend using the DKMS modules unless you are running a custom kernel for which the prebuilt drivers are not supported" because "the DKMS drivers are not signed with Canonical's key and thus do not support secure boot"; ubuntu-drivers "by default, only install the pre-built, signed drivers". [SOURCED same Ubuntu server doc; https://ubuntu.com/desktop/docs/en/latest/how-to/graphics/build-your-own-nvidia-modules-using-the-dkms-package/] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#5-ubuntus-three-packaging-axes-branch-flavor-module-delivery]
610 sits in multiverse and is "NOT maintained by Ubuntu core developers and NOT officially supported by Ubuntu, as it is a non-production branch driver". [SOURCED https://www.ubuntuupdates.org/package/core/resolute/multiverse/updates/nvidia-driver-610-open] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#5-ubuntus-three-packaging-axes-branch-flavor-module-delivery]

7. Ubuntu 26.04 (resolute) package inventory for branch 610 (2026-09-24)

Also available in resolute: 610.43.02-0ubuntu0.26.04.1 (release pocket) and 610.43.03. [SOURCED launchpad/packages.ubuntu.com] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#7-ubuntu-2604-resolute-package-inventory-for-branch-610-2026-09-24]

8. NVIDIA-repo alternative (do not mix with Ubuntu packages)

NVIDIA's own CUDA apt repo installs the open flavor with apt install nvidia-open (compute-only variants use nvidia-dkms-open). [SOURCED https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/latest/ubuntu.html] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#8-nvidia-repo-alternative-do-not-mix-with-ubuntu-packages]
Mixing NVIDIA-repo nvidia-open/cuda-drivers with Ubuntu nvidia-driver-610-open produces file conflicts and version drift between KMD, GSP firmware and UMD. Pick one source. [INFERRED from the version-match requirement] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#8-nvidia-repo-alternative-do-not-mix-with-ubuntu-packages]

9. How a kernel upgrade rebuilds the module (Ubuntu DKMS path)

linux-image-generic-hwe-26.04 (HWE = hardware-enablement kernel series) pulls linux-headers-<new> [INFERRED]; DKMS needs "the linux-headers metapackage for your kernel flavor"; verify with apt-cache policy linux-headers-$(uname -r). [SOURCED https://ubuntu.com/desktop/docs/en/latest/how-to/graphics/build-your-own-nvidia-modules-using-the-dkms-package/] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#9-how-a-kernel-upgrade-rebuilds-the-module-ubuntu-dkms-path]
The kernel package's postinst triggers DKMS, which "will take care of automatically rebuilding registered kernel modules when installing a different Linux kernel". [SOURCED https://wiki.archlinux.org/title/Dynamic_Kernel_Module_Support via search summary] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#9-how-a-kernel-upgrade-rebuilds-the-module-ubuntu-dkms-path]
Verify: dkms status must list nvidia/610.57.04, 7.0.0-34-generic, x86_64: installed; the artefact lands in /lib/modules/<kernel>/updates/dkms/nvidia*.ko(.zst|.xz). [SOURCED https://github.com/sternecker/nvidia-kernel-rebuild ; kernel path INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#9-how-a-kernel-upgrade-rebuilds-the-module-ubuntu-dkms-path]
If missing: sudo apt install linux-headers-$(uname -r) && sudo dkms autoinstall (or sudo dkms install nvidia/610.57.04 -k <kver> --force). [SOURCED same] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#9-how-a-kernel-upgrade-rebuilds-the-module-ubuntu-dkms-path]
Secure Boot: DKMS builds are not Canonical-signed; the DKMS package "may automatically guide you through creating and enrolling a new key" (a Machine Owner Key, MOK). If Secure Boot is on and MOK was never enrolled, the module will refuse to load after the first DKMS rebuild. [SOURCED Ubuntu DKMS how-to] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#9-how-a-kernel-upgrade-rebuilds-the-module-ubuntu-dkms-path]
The reference box's install nvidia /bin/false blocks autoload but does not block DKMS builds; the systemd unit must keep using modprobe --ignore-install. [INFERRED - modprobe(8) semantics] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#9-how-a-kernel-upgrade-rebuilds-the-module-ubuntu-dkms-path]

10. Upgrade / rollback runbook

Run these steps one at a time, not as a pasted script: steps 2, 3, 4 and 6 end in a reboot. Prerequisites: sudo; the DKMS install path (§9); Secure Boot state known (mokutil --sb-state; §9 step 5); console or other-path access in case the host does not return after a reboot. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#10-upgrade-rollback-runbook]
Steps 1–6 combine [SOURCED] package facts above with standard apt/dkms mechanics [INFERRED]. [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#10-upgrade-rollback-runbook]

11. `NVreg_*` parameters: existence, defaults and eGPU verdicts

Existence and defaults checked against kernel-open/nvidia/nv-reg.h on main (615.71.09) and the 580/610 READMEs. [SOURCED https://raw.githubusercontent.com/NVIDIA/open-gpu-kernel-modules/main/kernel-open/nvidia/nv-reg.h] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#11-nvreg_-parameters-existence-defaults-and-egpu-verdicts]
Branch caveat. The reference box runs 610.57.04, but nv-reg.h was read on main (615.71.09) and the two power-management READMEs are 580.65.06, so it is [INFERRED] that the same parameters and defaults hold on 610. Check with modinfo -p nvidia | grep NVreg_ (which exist) and grep -E 'DynamicPowerManagement|PreserveVideoMemoryAllocations' /proc/driver/nvidia/params (in effect). On 2026-09-24 the second showed PreserveVideoMemoryAllocations: 1 on the reference box although nvidia-egpu-pm.conf sets =0: nvidia-graphics-drivers-kms.conf also sets =1 and sorts later. [INFERRED - local check of the loaded values; last-value-wins for a repeated option is standard modprobe.d behaviour] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#11-nvreg_-parameters-existence-defaults-and-egpu-verdicts]

12. Kernel command-line and udev knobs for a Thunderbolt eGPU (not NVreg)

pcie_ports=native pcie_aspm=off pcie_port_pm=off pci=assign-busses,realloc - "Without pcie_ports=native, driver fails: 'Unable to change power state from D3cold to D0'". [SOURCED https://github.com/NVIDIA/open-gpu-kernel-modules/issues/979] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#12-kernel-command-line-and-udev-knobs-for-a-thunderbolt-egpu-not-nvreg]
Write 0 to /sys/bus/pci/devices/<gpu-bdf>/d3cold_allowed (udev rule keyed on ATTR{vendor}=="0x10de"). [SOURCED https://egpu.io/forums/thunderbolt-linux-setup/cant-change-power-state-from-d3cold-to-d0-nvidia-egpu-ubuntu-20-04/] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#12-kernel-command-line-and-udev-knobs-for-a-thunderbolt-egpu-not-nvreg]
Xid 79 = "GPU has fallen off the bus" - logged when the driver "attempts to access the GPU over its PCI Express connection and finds that the GPU is not accessible"; NVIDIA's catalog attributes it to HW error, driver error, system memory corruption, bus error, or thermal. [SOURCED via https://forums.developer.nvidia.com/t/gpu-has-fallen-off-the-bus/217357 quoting docs.nvidia.com/deploy/xid-errors - catalog page itself returned 404 on fetch] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#12-kernel-command-line-and-udev-knobs-for-a-thunderbolt-egpu-not-nvreg]

How to apply these: parameter meanings, the working command line and the case against stacking forum parameters are in pcie-power-management-aer-dpc-egpu-linux.md (Knob Table, Worked case); the d3cold_allowed udev rule is template B under udev Templates there; autoload blocking and load ordering are in linux-egpu-hotplug-boot-orchestration.md. The reference box's working command line differs from the issue #979 set above (for example pci=realloc=off); prefer the working set and treat the #979 list as a field report. [INFERRED from the sibling references] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#12-kernel-command-line-and-udev-knobs-for-a-thunderbolt-egpu-not-nvreg]

13. Headless checklist

Checklist for a host with no X/Wayland on the NVIDIA GPU (iGPU handles any console): [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#13-headless-checklist]
- Packages: nvidia-headless-610-open (or nvidia-driver-610-open if you also want GL/Vulkan libs) + nvidia-utils-610 + nvidia-compute-utils-610. nvidia-headless-no-dkms-610-open pairs with the prebuilt linux-modules-nvidia-610-open-generic-hwe-26.04. [SOURCED launchpad binary list; pairing INFERRED] Ubuntu's tool path: sudo ubuntu-drivers install --gpgpu (installs prebuilt signed by default; --include-dkms otherwise). [SOURCED Ubuntu server doc; Ubuntu DKMS how-to] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#13-headless-checklist]
- Modules: nvidia + nvidia_uvm are sufficient for CUDA; nvidia_modeset/nvidia_drm are display-only. Reference box: unit after bolt.service → modprobe --ignore-install nvidia nvidia_uvm. [INFERRED - module roles are standard; ordering after bolt is what the box does] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#13-headless-checklist]
- Device nodes: /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools. Without udev/X they are created on demand by setuid nvidia-modprobe (first CUDA call) or held open by nvidia-persistenced. [INFERRED - standard driver behaviour; the CUDA install guide's 'Device Node Verification' section was not in the fetched page] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#13-headless-checklist]
- Persistence: enable nvidia-persistenced.service (shipped in nvidia-compute-utils-610). "While the daemon holds the device files open, at least one client, the daemon, has the GPU attached and the driver will not unload the GPU state." NVIDIA "will focus all future development and bug fixes on the daemon" rather than the legacy nvidia-smi -pm. [SOURCED https://docs.nvidia.com/deploy/driver-persistence/persistence-daemon.html ; packages.ubuntu.com filelist] On an eGPU this also pins the GPU in D0 between jobs (belt-and-braces with DynamicPowerManagement=0x00). [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#13-headless-checklist]
- Verify: nvidia-smi (KMD == firmware dir == UMD-owning package version); nvidia-smi -q -d POWER,PERFORMANCE; sudo lspci -vv -s <bdf> | grep -E 'LnkSta|LnkCap' (root needed: unprivileged lspci cannot read the capability registers) expecting 16GT/s x4 over TB4. [SOURCED link figure from issue #979; commands INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#13-headless-checklist]
- Containers: nvidia-container-toolkit from NVIDIA's apt repo, then sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker; prerequisite is only the host driver, no host toolkit. [SOURCED https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#13-headless-checklist]
- Thunderbolt: device must be authorised by bolt (boltctl list / boltctl enroll) before modprobe - hence the unit ordering. [INFERRED - ArchWiki eGPU page was behind Anubis on fetch] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#13-headless-checklist]
- Soak test before trusting: python3 -c "import torch; torch.zeros(1, device='cuda')" is the minimal reproducer that hard-locked an RTX 5080 in a Thunderbolt 5 (TB5) enclosure on a TB4 host (issue #979); a sustained inference run (Ollama's llama-server, or an upstream llama.cpp llama-server build) is the realistic one. Run either test from a session where a hard lock is survivable: nothing unsaved, and a way to power-cycle the host. [SOURCED issue #979] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#13-headless-checklist]

Anti-patterns

Installing nvidia-driver-610 (proprietary) or any pre-Turing-era how-to on Blackwell - "Blackwell and later are only supported by the open kernel modules." [SOURCED] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
NVreg_EnableGpuFirmware=0 on the open flavor - the open modules depend on GSP; the GPU will not initialise. [SOURCED dependency; outcome INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
Carrying NVreg_UsePageAttributeTable=1 or NVreg_EnablePCIeGen3=1 forward - the former no longer exists in nv-reg.h; the latter is a Kepler/SandyBridge-E workaround. [SOURCED nv-reg.h] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
Leaving NVreg_DynamicPowerManagement at default (0x03) on a laptop host with an eGPU - may resolve to fine-grained RTD3 and trigger D3cold transitions the TB tunnel cannot complete. Keep 0x00. [INFERRED; symptom SOURCED #979] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
Installing the cuda metapackage from NVIDIA's repo on top of Ubuntu's 610 packages - it pulls cuda-drivers and conflicts with nvidia-driver-610-open; install cuda-toolkit-13-3 (toolkit only). [INFERRED from NVIDIA install-guide package structure] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
Building llama.cpp / custom CUDA against apt's CUDA 12.4 - cannot emit sm_120; "CUDA 12.8 was the first release to add the sm_100/sm_101/sm_120 Blackwell targets". [SOURCED runaihome/NVIDIA forum via search] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
Unpinned driver upgrades via unattended-upgrades - a branch jump (610→615) silently changes KMD, firmware dir and UMD; Ollama's GPU discovery has broken across such transitions before. [SOURCED Ollama issues; pinning INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
DKMS on a Secure-Boot host without an enrolled MOK - module builds but will not load after the next kernel. [SOURCED Ubuntu docs] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
Assuming a CUDA 13.4-built binary is "just fine" on UMD 13.3 - minor-version compat holds for SASS, but PTX JIT and R615-gated APIs fail (cudaErrorCallRequiresNewerDriver). [SOURCED minor-version-compat page] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
Suspending the host with the eGPU attached and expecting VRAM to survive - needs PreserveVideoMemoryAllocations=1 + systemd suspend hooks + temp space ≥ VRAM + 5 %; the reference box's nvidia-egpu-pm.conf sets =0 (check the effective value, §11). [SOURCED README powermanagement] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
Treating nvidia-smi idle success as proof of eGPU stability - #979: "nvidia-smi works at idle" while any CUDA kernel hard-locks the host. Soak-test under load. [SOURCED #979] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]
**Mixing NVIDIA-repo nvidia-open with Canonical nvidia-driver-*-open** - version-mismatch between KMD, GSP firmware and UMD. [INFERRED from the version-match requirement] [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]

Related references added later: egpu-reproducible-bringup-and-drift-detection-linux.md (capturing this wiring as a restorable manifest, drift verifier and restore order). [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#anti-patterns]

Sources

https://download.nvidia.com/XFree86/Linux-x86_64/580.65.06/README/kernel_open.html - open vs proprietary, Blackwell open-only, GSP dependency [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://download.nvidia.com/XFree86/Linux-x86_64/610.57.04/README/kernel_open.html - installer defaults to open; open-only features [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://download.nvidia.com/XFree86/Linux-x86_64/610.57.04/README/gsp.html - GSP firmware, NVreg_EnableGpuFirmware, firmware path [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://download.nvidia.com/XFree86/Linux-x86_64/580.65.06/README/dynamicpowermanagement.html - NVreg_DynamicPowerManagement semantics [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://download.nvidia.com/XFree86/Linux-x86_64/580.65.06/README/powermanagement.html - NVreg_PreserveVideoMemoryAllocations, suspend services [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://raw.githubusercontent.com/NVIDIA/open-gpu-kernel-modules/main/kernel-open/nvidia/nv-reg.h - parameter existence/defaults [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/README.md - version-match rule, GPU list (615.71.09 on main) [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html - 12.x ≥525 / 13.x ≥580, PTX caveat [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.nvidia.com/cuda/archive/13.0.0/cuda-toolkit-release-notes/index.html - 12.8 ≥570.26, 13.0 ≥580.65.06, dropped archs [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html - CUDA 13.4 U1 requires R615 [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-610-57-04/index.html - 610.57.04 / CUDA 13.x, release date [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.nvidia.com/datacenter/tesla/drivers/driver-lifecycle.html - NFB / PB / LTSB definitions [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/latest/ubuntu.html - apt install nvidia-open [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.nvidia.com/deploy/driver-persistence/persistence-daemon.html - nvidia-persistenced [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html - container toolkit [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://forums.developer.nvidia.com/t/enabling-pcie-3-0-with-nvreg-enablepciegen3-1/28367 - EnablePCIeGen3 background [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://forums.developer.nvidia.com/t/gpu-has-fallen-off-the-bus/217357 - Xid 79 definition (quoting the Xid catalog) [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://forums.developer.nvidia.com/t/software-migration-guide-for-nvidia-blackwell-rtx-gpus-a-guide-to-cuda-12-8-pytorch-tensorrt-and-llama-cpp/321330 - CUDA 12.8 floor for Blackwell [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://launchpad.net/ubuntu/+source/nvidia-graphics-drivers-610 - versions per pocket, binary package list [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://packages.ubuntu.com/resolute-updates/linux-modules-nvidia-610-open-generic-hwe-26.04 - prebuilt signed modules 7.0.0-34.34+1 [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://packages.ubuntu.com/resolute-updates/amd64/nvidia-compute-utils-610/filelist - persistenced unit [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://packages.ubuntu.com/pt-br/resolute-updates/amd64/linux-objects-nvidia-610-open-7.0.0-31-generic [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://ubuntu.com/server/docs/how-to/graphics/install-nvidia-drivers/ - UDA vs ERD, DKMS vs prebuilt, --gpgpu [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://ubuntu.com/desktop/docs/en/latest/how-to/graphics/build-your-own-nvidia-modules-using-the-dkms-package/ - DKMS steps, MOK [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://www.ubuntuupdates.org/package/core/resolute/multiverse/updates/nvidia-driver-610-open - multiverse / unsupported note [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://www.gamingonlinux.com/2026/08/nvidia-stable-driver-595-91-07-and-new-feature-driver-610-57-04-released-for-linux/ [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://9to5linux.com/nvidia-595-linux-graphics-driver-released-as-latest-production-branch-version [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://www.phoronix.com/news/NVIDIA-610.57.04-Linux-Driver [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]

eGPU field reports (open-gpu-kernel-modules) [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]

https://github.com/NVIDIA/open-gpu-kernel-modules/issues/979 - RTX 5080 TB5 eGPU hard lock on CUDA; kernel params [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/NVIDIA/open-gpu-kernel-modules/issues/974 - RTX 5060 Ti TB3 eGPU Xid 79 / GSP-FMC boot error [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/NVIDIA/open-gpu-kernel-modules/issues/900 - RTX 5090 OCuLink Xid 79 under load [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/NVIDIA/open-gpu-kernel-modules/pull/981 - TB4/TB5 detection patch (closed, unmerged) [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://egpu.io/forums/thunderbolt-linux-setup/cant-change-power-state-from-d3cold-to-d0-nvidia-egpu-ubuntu-20-04/ - d3cold_allowed udev [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/sternecker/nvidia-kernel-rebuild - dkms autoinstall recovery pattern [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://pytorch.org/blog/pytorch-2-11-release-blog/ - CUDA 13 default on PyPI [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/pytorch/pytorch/blob/main/RELEASE.md - 2.10–2.13 CUDA matrix [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://dev-discuss.pytorch.org/t/introducing-cuda-13-2-and-deprecating-cuda-12-8-release-2-12/3337 [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/pytorch/pytorch/issues/172663 - 2.11 CUDA support matrix RFC (Blackwell 10.0/12.0) [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://journal.qualiteg.com/pytorch_and_supported_gpu_version/ - PyTorch 2.7.0 first with sm_120 (cu128) [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/ggml-org/llama.cpp/releases - prebuilt cuda-12.8 / cuda-13.4 + cudart tarballs (b11169) [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md - CUDA build flags [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/ggml-org/llama.cpp/issues/19662 - MXFP4 sm_120 compile failure [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.ollama.com/gpu - CC 12.0 / RTX 50xx supported, driver ≥550 [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.ollama.com/development - backend list incl. cuda_v12 / cuda_v13 [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.ollama.com/linux - "Install CUDA drivers (optional)" [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/ollama/ollama/issues/14357 , /issues/14188 , /issues/18581 - driver-transition regressions [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html - CUDA 12.9 default wheels, Blackwell ≥12.8 [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/vllm-project/vllm/pull/50288 - NVFP4 KV cache for SM120/121 [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]
https://github.com/MakazhanAlpamys/Soup/issues/827 , https://github.com/unslothai/unsloth/issues/5812 - "KMD / CUDA UMD" nvidia-smi header format [archived source: https://llms-explorer.com/sources/global-ai-hub/nvidia-open-kernel-modules-blackwell-linux/#sources]

Children

CUDA minor-version compatibility & forward-compat (PTX JIT, cuda-compat) (frontier)
GSP firmware boot diagnostics (GSP-FMC, NVreg_EnableGpuFirmwareLogs, Xid 119/120) (frontier)
NVIDIA driver branch lifecycle (NFB/PB/LTSB, EOL dates) (frontier)
Ollama GPU discovery & bundled CUDA backends (cuda_v12/cuda_v13, NVML) (frontier)
PyTorch CUDA wheel matrix & vendored runtime (cu126/cu128/cu130) (frontier)
Ubuntu Secure Boot MOK / DKMS signing vs prebuilt linux-modules-nvidia (frontier)
llama.cpp CUDA build matrix for sm_120 (CMAKE_CUDA_ARCHITECTURES, MXFP4) (frontier)
nvidia-persistenced & device-node lifecycle headless (nvidia-modprobe, CDI) (frontier)
vLLM consumer-Blackwell sm_120 kernels (NVFP4, FlashInfer, bitsandbytes gaps) (frontier)

Frontier under this node: CUDA minor-version compatibility & forward-compat (PTX JIT, cuda-compat), GSP firmware boot diagnostics (GSP-FMC, NVreg_EnableGpuFirmwareLogs, Xid 119/120), NVIDIA driver branch lifecycle (NFB/PB/LTSB, EOL dates), Ollama GPU discovery & bundled CUDA backends (cuda_v12/cuda_v13, NVML), PyTorch CUDA wheel matrix & vendored runtime (cu126/cu128/cu130), Ubuntu Secure Boot MOK / DKMS signing vs prebuilt linux-modules-nvidia, llama.cpp CUDA build matrix for sm_120 (CMAKE_CUDA_ARCHITECTURES, MXFP4), nvidia-persistenced & device-node lifecycle headless (nvidia-modprobe, CDI), vLLM consumer-Blackwell sm_120 kernels (NVFP4, FlashInfer, bitsandbytes gaps)

← the whole tree · 3D view· how to read this page