Build & SNVMe Testing Guide

This document has two parts:

Build — how to prepare the environment and compile Tutti plus the snvme kernel module.
SNVMe Testing — how to use the smoke-test suite under tutti/device_manager/nvme/kernel_modules/test/ to validate the snvme driver, climbing from the safest test to the most destructive.

Any concrete PCI BDF (e.g. 0000:e3:00.0), disk mount point, etc. shown below is an example / host-specific value. Confirm the right one on your own host with the tooling in Step 4; don’t copy it blindly.

All paths are relative to the project root unless stated otherwise.

Part 1 — Build

1.1 Prepare the environment

Use scripts/prepare_env.sh for one-shot dependency setup. The script:

Installs the compiler toolchain plus protobuf / gRPC / uuid / yaml-cpp / libunwind per distro (Debian/Ubuntu/RHEL/CentOS/TencentOS/Fedora/openSUSE/Arch).
Uses the system gRPC when pkg-config finds grpc++ and grpc_cpp_plugin is on PATH; otherwise it falls back to building grpc + yaml-cpp via vcpkg. Pass --force-vcpkg to bypass this check and always use vcpkg for those C++ dependencies.
Generates CMakePresets.json in the project root (machine-generated, gitignored, do not commit).
Generates build_dependencies.tsv in the project root with the actual provider, version, and location of the selected build tools and libraries. In vcpkg mode this includes installed transitive packages as well as direct dependencies (machine-generated, gitignored, do not commit).

The generated presets require CMake 3.21 or newer. The setup script checks this explicitly and stops with an upgrade instruction when the distro package is too old.

bash scripts/prepare_env.sh            # optional: -j N for parallelism
bash scripts/prepare_env.sh --force-vcpkg -j N  # force vcpkg-backed presets

Common environment variables:

Variable	Effect
`VCPKG_ROOT`	vcpkg install path (default `third_pkgs/vcpkg`)

The dependency manifest is tab-separated and can be inspected directly or consumed by standard TSV tooling. Re-running the setup script replaces it atomically with the current machine state.

1.2 Build Tutti

The CMakePresets.json generated by prepare_env.sh records the dependency provider selected on this machine and exposes one isolated build directory per accelerator profile:

Preset	Build directory	Purpose
`host`	`build/host`	Hardware-free API/SPI and contract tests
`cuda`	`build/cuda`	CUDA userspace stack; snvme module and hardware tests disabled
`cuda-module`	`build/cuda-module`	CUDA stack plus kernel-specific snvme module targets
`musa`	`build/musa`	MUSA porting profile and SDK path configuration
`maca`	`build/maca`	MACA porting profile and SDK root configuration

Configure, build, and test use the same backend name:

cmake --preset host
cmake --build --preset host --parallel 8
ctest --preset host

cmake --preset cuda
cmake --build --preset cuda --parallel 8
ctest --preset cuda

List the available configure/build/test workflows with:

cmake --list-presets=configure
cmake --list-presets=build
cmake --list-presets=test

The generated JSON explicitly records the relevant variables so IDEs can display and edit them. After command-line configuration, inspect all effective cache values without rerunning detection:

cmake -N -LA build/cuda

For a one-off change, append cache overrides to the configure command. The preset’s own build directory keeps the change isolated from other backends:

cmake --preset cuda --fresh \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CUDA_ARCHITECTURES=89 \
  -DTUTTI_FEATURE_LOCAL_NVME=OFF

Persistent machine-specific changes belong in CMakeUserPresets.json, which is gitignored and automatically inherits from CMakePresets.json. Start from CMakeUserPresets.json.example; its examples cover HOST, CUDA, CUDA+module, MUSA, and MACA without modifying the generated presets.

CMake parameters used by the build

The generated presets provide the normal build configuration. The following cache parameters are the ones that may need a host-specific value:

CMake parameter	When it is needed	Effect	Example value
`TUTTI_BUILD_KERNEL_MODULE`	Set to `ON` when the snvme module is part of the build	Enables kernel-specific `modules`, `insmod`, and `rmmod` targets and adds `modules` to the default build; default `OFF` so normal HOST/CUDA builds are independent of the running kernel	`ON`
`TUTTI_P2P_BACKEND`	When selecting the GPU peer-memory implementation compiled into `snvme.ko`	Selects exactly one backend source; defaults to `nvidia` for CUDA and `metax` for MUSA/MACA	`nvidia`
`SNVME_P2P_INCLUDE_DIR`	When the selected backend header is outside the default SDK/driver search paths	Directory containing `nv-p2p.h` or `metax_p2p.h`	`/usr/src/nvidia-570.124.06/nvidia-peermem`
`SNVME_KERNEL_VERSION`	Required when the running kernel name cannot be matched automatically	Selects `tutti/device_manager/nvme/kernel_modules/snvme-<tag>/` as the kernel-module source baseline	`5.15.0-public`
`CMAKE_TOOLCHAIN_FILE`	Supplied automatically by a vcpkg-backed preset; only set it manually when configuring without that preset	Makes CMake resolve C++ dependencies through vcpkg	`/path/to/Tutti/third_pkgs/vcpkg/scripts/buildsystems/vcpkg.cmake`
`CMAKE_BUILD_TYPE`	Supplied automatically by a preset; set it when configuring manually	Selects optimization/debug settings	`RelWithDebInfo`
`CMAKE_CUDA_COMPILER`	Optional, when multiple CUDA installations exist	Selects the `nvcc` compiler during the initial configuration	`/usr/local/cuda-12.8/bin/nvcc`
`CUDAToolkit_ROOT`	Optional, when multiple CUDA installations exist	Directs `find_package(CUDAToolkit)` to the matching headers and libraries	`/usr/local/cuda-12.8`

Pass an override after the preset name. --fresh is recommended when changing dependency or kernel selections so a stale cache cannot retain the old value:

cmake --preset cuda-module --fresh \
  -DTUTTI_P2P_BACKEND=nvidia \
  -DSNVME_KERNEL_VERSION=5.15.0-public \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc \
  -DCUDAToolkit_ROOT=/usr/local/cuda-12.8
cmake --build --preset cuda-module --parallel 8

When configuring without presets, the equivalent vcpkg example is:

cmake -S . -B build --fresh \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DTUTTI_BUILD_KERNEL_MODULE=ON \
  -DTUTTI_P2P_BACKEND=nvidia \
  -DSNVME_KERNEL_VERSION=5.15.0-public \
  -DCMAKE_TOOLCHAIN_FILE="$PWD/third_pkgs/vcpkg/scripts/buildsystems/vcpkg.cmake"
cmake --build build --parallel 8

The values above are examples for a host with an Ubuntu 5.15 generic kernel and CUDA 12.8; inspect the current machine before copying them.

For a Metax userspace profile, enable the same module target with the Metax kernel P2P backend. metax_p2p.h must currently be supplied by the vendor driver SDK; it is not vendored by Tutti:

cmake --preset musa --fresh \
  -DTUTTI_BUILD_KERNEL_MODULE=ON \
  -DTUTTI_P2P_BACKEND=metax \
  -DSNVME_P2P_INCLUDE_DIR=/path/to/metax/p2p/include
cmake --build --preset musa --target modules --parallel 8

snvme baseline auto-selection

The snvme kernel module is maintained per kernel baseline under tutti/device_manager/nvme/kernel_modules/snvme-<tag>/, e.g.:

snvme-5.15.0-public — upstream 5.15.0
snvme-6.8.0-public — Linux 6.8

CMake matches the numeric baseline prefix against uname -r by default and ignores the descriptive -public suffix. For example, both 6.8.0-90-generic and upstream 6.8.0 select 6.8.0-public. Set the value explicitly only when auto-selection is ambiguous or when cross-building:

cmake --preset cuda-module --fresh \
  -DSNVME_KERNEL_VERSION=6.8.0-public

To see the available values, list the baseline directories:

find tutti/device_manager/nvme/kernel_modules -maxdepth 1 \
  -type d -name 'snvme-*' -printf '%f\n'

1.3 Install the snvme kernel module

Build artifacts land in build/cuda-module/module/ (snvme-core.ko + snvme.ko). Install from the build directory:

cmake --build --preset cuda-module --target insmod

Check that it installed cleanly:

$ lsmod | grep snvme
snvme                 217088  0
snvme_core            106496  1 snvme

To unload, use make rmmod. To reload, use make insmod (or your site’s signed-module install flow).

1.4 Build the test code

make -C tutti/device_manager/nvme/kernel_modules/test -j"$(nproc)"

The test binaries depend on the SNVMe UAPI header tutti/include/uapi/tutti_snvme.h (already pulled in via the Makefile). The role of each binary is described in Part 2 Step 3.

Part 2 — SNVMe Testing

Mental model

SNVMe exposes three kinds of /dev objects:

Path	What	Used for
`/dev/snvm_control`	factory entry (1 per module)	bind/unbind a PCI device, create per-ctrl chrdev
`/dev/ssnvme<N>`	per-controller char dev (double s)	BAR0 mmap + all queue ioctls
`/dev/snvme<X>n<Y>`	block device (single s)	normal mount target, appears after bind

Testing climbs a ladder of 6 binaries, safest → most destructive. Golden rule: run snvme_smoke first; only move up a rung after the one below passes.

Step 0 — Pre-flight

uname -r                                                  # 5.15.x → matches the snvme-5.15.0-public baseline
grep CONFIG_MODULE_SIG_FORCE /boot/config-$(uname -r)     # "not set" → unsigned .ko loads fine
lsmod | grep snvme                                        # snvme + snvme_core loaded
ls -l /dev/snvm_control                                   # exists, mode 0666

If all four pass you can skip straight to Step 3. Steps 1–2 are only needed when you’ve changed driver code and have to rebuild.

Step 1 — (Re)build the module (only after editing driver code)

The .kos were built in Part 1 under build/cuda-module/module/. After changing driver code, rebuild:

cmake --build --preset cuda-module --target modules
# or by hand:
cd build/cuda-module/module && make

Step 2 — Rebuild / reload the module (only after editing driver code)

⚠️ Hard rule: after any driver change you must reload the .ko. The smoke binaries embed _IOC_SIZE-derived ioctl numbers; a stale module returns -ENOTTY on valid requests.

Before rmmod, unbind all snvme-owned controllers and stop any process holding /dev/snvm* fds (or mounted /dev/snvme*n* filesystems), otherwise the module stays referenced:

sudo bash scripts/unbind.sh          # unbind all snvme controllers (idempotent)
lsof /dev/snvm_control /dev/ssnvme* /dev/snvme*n* 2>/dev/null   # should print nothing
cmake --build --preset cuda-module --target rmmod     # sudo rmmod snvme snvme_core
cmake --build --preset cuda-module --target insmod    # reload

Queue depth: there is no io_queue_depth module parameter — snvme always takes the controller maximum (q_depth = NVMe CAP.MQES + 1, typically 1024 on datacenter SSDs). It is fixed at probe time; changing it requires rmmod+insmod.

Step 3 — Build the test binaries

# Run from the project root. Builds all CPU binaries and, when nvcc is
# available, the CUDA binary as well.
make -C tutti/device_manager/nvme/kernel_modules/test

# CPU-only build (does not invoke nvcc):
make -C tutti/device_manager/nvme/kernel_modules/test cpu

# CUDA test only; override the architecture when auto-detection is unavailable:
make -C tutti/device_manager/nvme/kernel_modules/test gpu CUDA_ARCH=sm_80

This builds 5 tests + 1 reset helper:

Binary	Binds?	Writes?	What it does
`snvme_smoke`	no	no	libc-only UAPI smoke: chrdev create/remove + `NVM_MAP_HOST_MEMORY` + BAR0 mmap
`snvme_smoke_qgroup`	no	no	queue-group lifecycle (create/destroy + fd-close cascade)
`snvme_smoke_gpu`	no/yes	no	adds `NVM_MAP_DEVICE_MEMORY` / p2p path (built only if `nvcc` is on `$PATH`)
`snvme_smoke_addq`	yes	no	B3 `NVM_ADD_USER_QUEUE` end-to-end (Create I/O CQ+SQ + cascade destroy)
`snvme_smoke_io`	yes	yes	B3 CPU end-to-end, 23 phases, byte-by-byte verification, writes LBA
`snvme_ubind`	—	no	owner-side reset helper: `SNVM_DEVICE_UNBIND` + `SNVM_CHRDEV_REMOVE`

Step 4 — Pick a test device

Inspect local nvme disks and the GPU↔NVMe topology distance, and pick a throwaway disk:

sudo bash scripts/pci_topology_check.sh

The following test will destroy the data on the disk; please ensure that no important data remains on the disk.

The rest of this guide uses a TGT variable for the chosen BDF:

export TGT=0000:e3:00.0     # ← replace with your own target BDF

Step 5 — SAFE tests (no bind, won’t disturb any disk)

These exercise only the UAPI and won’t detach the in-tree nvme driver — safe even on a production host:

cd tutti/device_manager/nvme/kernel_modules/test

# 5a. libc-only UAPI smoke: chrdev create/remove + NVM_MAP_HOST_MEMORY + BAR0 mmap
sudo ./snvme_smoke        $TGT

# 5b. queue-group lifecycle (create/destroy + fd-close cascade), no bind
sudo ./snvme_smoke_qgroup $TGT

Each exits 0 on success. A failure prints [FAIL] step=<N> ... errno=<E> and stops.

Step 6 — DESTRUCTIVE tests (bind required)

🛑 These detach the in-tree nvme driver, and snvme_smoke_io WRITES to LBA. Use a throwaway controller only.

Climb the rungs in order:

# 6a. B3 bind + ADD_USER_QUEUE end-to-end (with host rings; binds, no LBA writes)
sudo ./snvme_smoke_addq    $TGT

# 6b. The big one — B3 CPU end-to-end, 23 phases:
#     PRP1 / PRP1+PRP2 / PRP_List / SGL (auto-skipped on PRP-only devices) + SQ-tail-wrap,
#     byte-by-byte data verification on every IO. WRITES TO DISK.
sudo ./snvme_smoke_io      $TGT

snvme_smoke_io returning 0 is the authoritative CPU-side gate — it’s what catches the vaddr-mask and cap-after-MSI-X regressions.

If 6a or 6c fails with “no room for user queues”

snvme_smoke_addq and snvme_smoke_io cap kernel-side IOQs to 36 by default (NVM_SET_KERNEL_IOQ_CAP), reserving the rest of the controller’s grant for user queues. Some controllers grant few total IOQs; in that case the default cap ≥ the grant, the kernel consumes every queue, and NVM_ADD_USER_QUEUE fails.

Fix: set SNVME_TEST_KERNEL_IOQ_CAP to a value strictly less than the controller’s grant before running:

# Check how many IOQs the controller actually granted (look for max_user_qid):
sudo ./snvme_smoke_addq $TGT

sudo ./snvme_ubind $TGT

# If max_user_qid == 0 (no room), re-run with a smaller cap.
# Example for a 31-queue controller — cap at 16 to leave 15 for userspace:
SNVME_TEST_KERNEL_IOQ_CAP=16 sudo ./snvme_smoke_addq $TGT
SNVME_TEST_KERNEL_IOQ_CAP=16 sudo ./snvme_smoke_io   $TGT

Step 7 — GPU tests (only if the NVIDIA driver is loaded)

Skip this on a host with no GPU (no nvidia-smi). On a GPU host:

# safe (no bind) — adds the NVM_MAP_DEVICE_MEMORY / p2p path
sudo ./snvme_smoke_gpu --gpu 0 $TGT

# destructive, the authoritative B3 GPU gate: 4 full alloc/free rounds
sudo ./snvme_smoke_gpu --gpu 0 --rounds 4 $TGT

Step 8 — Cleanup & recovery

After a clean run the binaries unbind themselves. If a test dies mid-run (e.g. kill -9), the controller can be left half-bound. Reset it with the prebuilt helper:

sudo ./snvme_ubind $TGT          # SNVM_DEVICE_UNBIND + SNVM_CHRDEV_REMOVE

To hand the block device back to in-tree nvme:

sudo ./snvme_ubind $TGT
echo $TGT | sudo tee /sys/bus/pci/drivers/nvme/bind 2>/dev/null

A full reset is scripts/unbind.sh + make rmmod + make insmod (see Step 2).