# Build & SNVMe Testing Guide

This document has two parts:

1. **Build** — how to prepare the environment and compile Tutti plus the snvme kernel module.
2. **SNVMe Testing** — how to use the smoke-test suite under `tutti/device_manager/nvme/kernel_modules/test/` to validate the snvme driver, climbing from the safest test to the most destructive.

> Any concrete PCI BDF (e.g. `0000:e3:00.0`), disk mount point, etc. shown below is an **example / host-specific** value. Confirm the right one on your own host with the tooling in [Step 4](#step-4--pick-a-test-device); don't copy it blindly.

All paths are relative to the project root unless stated otherwise.

---

# Part 1 — Build

## 1.1 Prepare the environment

Use `scripts/prepare_env.sh` for one-shot dependency setup. The script:

- Installs the compiler toolchain plus `protobuf` / `gRPC` / `uuid` / `yaml-cpp` / `libunwind` per distro (Debian/Ubuntu/RHEL/CentOS/TencentOS/Fedora/openSUSE/Arch).
- Uses the system gRPC when `pkg-config` finds `grpc++` and `grpc_cpp_plugin` is on `PATH`; otherwise it falls back to building `grpc` + `yaml-cpp` via vcpkg. Pass `--force-vcpkg` to bypass this check and always use vcpkg for those C++ dependencies.
- Generates `CMakePresets.json` in the project root (**machine-generated, gitignored, do not commit**).
- Generates `build_dependencies.tsv` in the project root with the actual provider, version, and location of the selected build tools and libraries. In vcpkg mode this includes installed transitive packages as well as direct dependencies (**machine-generated, gitignored, do not commit**).

The generated presets require CMake 3.21 or newer. The setup script checks this explicitly and stops with an upgrade instruction when the distro package is too old.

```bash
bash scripts/prepare_env.sh            # optional: -j N for parallelism
bash scripts/prepare_env.sh --force-vcpkg -j N  # force vcpkg-backed presets
```

Common environment variables:

| Variable | Effect |
| --- | --- |
| `VCPKG_ROOT` | vcpkg install path (default `third_pkgs/vcpkg`) |

The dependency manifest is tab-separated and can be inspected directly or consumed by standard TSV tooling. Re-running the setup script replaces it atomically with the current machine state.

## 1.2 Build Tutti

The `CMakePresets.json` generated by `prepare_env.sh` records the dependency
provider selected on this machine and exposes one isolated build directory per
accelerator profile:

| Preset | Build directory | Purpose |
| --- | --- | --- |
| `host` | `build/host` | Hardware-free API/SPI and contract tests |
| `cuda` | `build/cuda` | CUDA userspace stack; snvme module and hardware tests disabled |
| `cuda-module` | `build/cuda-module` | CUDA stack plus kernel-specific snvme module targets |
| `musa` | `build/musa` | MUSA porting profile and SDK path configuration |
| `maca` | `build/maca` | MACA porting profile and SDK root configuration |

Configure, build, and test use the same backend name:

```bash
cmake --preset host
cmake --build --preset host --parallel 8
ctest --preset host

cmake --preset cuda
cmake --build --preset cuda --parallel 8
ctest --preset cuda
```

List the available configure/build/test workflows with:

```bash
cmake --list-presets=configure
cmake --list-presets=build
cmake --list-presets=test
```

The generated JSON explicitly records the relevant variables so IDEs can
display and edit them. After command-line configuration, inspect all effective
cache values without rerunning detection:

```bash
cmake -N -LA build/cuda
```

For a one-off change, append cache overrides to the configure command. The
preset's own build directory keeps the change isolated from other backends:

```bash
cmake --preset cuda --fresh \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CUDA_ARCHITECTURES=89 \
  -DTUTTI_FEATURE_LOCAL_NVME=OFF
```

Persistent machine-specific changes belong in `CMakeUserPresets.json`, which
is gitignored and automatically inherits from `CMakePresets.json`. Start from
`CMakeUserPresets.json.example`; its examples cover HOST, CUDA, CUDA+module,
MUSA, and MACA without modifying the generated presets.

### CMake parameters used by the build

The generated presets provide the normal build configuration. The following cache parameters are the ones that may need a host-specific value:

| CMake parameter | When it is needed | Effect | Example value |
| --- | --- | --- | --- |
| `TUTTI_BUILD_KERNEL_MODULE` | Set to `ON` when the snvme module is part of the build | Enables kernel-specific `modules`, `insmod`, and `rmmod` targets and adds `modules` to the default build; default `OFF` so normal HOST/CUDA builds are independent of the running kernel | `ON` |
| `TUTTI_P2P_BACKEND` | When selecting the GPU peer-memory implementation compiled into `snvme.ko` | Selects exactly one backend source; defaults to `nvidia` for CUDA and `metax` for MUSA/MACA | `nvidia` |
| `SNVME_P2P_INCLUDE_DIR` | When the selected backend header is outside the default SDK/driver search paths | Directory containing `nv-p2p.h` or `metax_p2p.h` | `/usr/src/nvidia-570.124.06/nvidia-peermem` |
| `SNVME_KERNEL_VERSION` | Required when the running kernel name cannot be matched automatically | Selects `tutti/device_manager/nvme/kernel_modules/snvme-<tag>/` as the kernel-module source baseline | `5.15.0-public` |
| `CMAKE_TOOLCHAIN_FILE` | Supplied automatically by a vcpkg-backed preset; only set it manually when configuring without that preset | Makes CMake resolve C++ dependencies through vcpkg | `/path/to/Tutti/third_pkgs/vcpkg/scripts/buildsystems/vcpkg.cmake` |
| `CMAKE_BUILD_TYPE` | Supplied automatically by a preset; set it when configuring manually | Selects optimization/debug settings | `RelWithDebInfo` |
| `CMAKE_CUDA_COMPILER` | Optional, when multiple CUDA installations exist | Selects the `nvcc` compiler during the initial configuration | `/usr/local/cuda-12.8/bin/nvcc` |
| `CUDAToolkit_ROOT` | Optional, when multiple CUDA installations exist | Directs `find_package(CUDAToolkit)` to the matching headers and libraries | `/usr/local/cuda-12.8` |

Pass an override after the preset name. `--fresh` is recommended when changing dependency or kernel selections so a stale cache cannot retain the old value:

```bash
cmake --preset cuda-module --fresh \
  -DTUTTI_P2P_BACKEND=nvidia \
  -DSNVME_KERNEL_VERSION=5.15.0-public \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc \
  -DCUDAToolkit_ROOT=/usr/local/cuda-12.8
cmake --build --preset cuda-module --parallel 8
```

When configuring without presets, the equivalent vcpkg example is:

```bash
cmake -S . -B build --fresh \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DTUTTI_BUILD_KERNEL_MODULE=ON \
  -DTUTTI_P2P_BACKEND=nvidia \
  -DSNVME_KERNEL_VERSION=5.15.0-public \
  -DCMAKE_TOOLCHAIN_FILE="$PWD/third_pkgs/vcpkg/scripts/buildsystems/vcpkg.cmake"
cmake --build build --parallel 8
```

The values above are examples for a host with an Ubuntu 5.15 generic kernel and CUDA 12.8; inspect the current machine before copying them.

For a Metax userspace profile, enable the same module target with the Metax
kernel P2P backend. `metax_p2p.h` must currently be supplied by the vendor
driver SDK; it is not vendored by Tutti:

```bash
cmake --preset musa --fresh \
  -DTUTTI_BUILD_KERNEL_MODULE=ON \
  -DTUTTI_P2P_BACKEND=metax \
  -DSNVME_P2P_INCLUDE_DIR=/path/to/metax/p2p/include
cmake --build --preset musa --target modules --parallel 8
```

### snvme baseline auto-selection

The snvme kernel module is maintained per kernel baseline under `tutti/device_manager/nvme/kernel_modules/snvme-<tag>/`, e.g.:

- `snvme-5.15.0-public` — upstream 5.15.0
- `snvme-6.8.0-public` — Linux 6.8

CMake matches the numeric baseline prefix against `uname -r` by default and
ignores the descriptive `-public` suffix. For example, both
`6.8.0-90-generic` and upstream `6.8.0` select `6.8.0-public`. Set the value
explicitly only when auto-selection is ambiguous or when cross-building:

```bash
cmake --preset cuda-module --fresh \
  -DSNVME_KERNEL_VERSION=6.8.0-public
```

To see the available values, list the baseline directories:

```bash
find tutti/device_manager/nvme/kernel_modules -maxdepth 1 \
  -type d -name 'snvme-*' -printf '%f\n'
```

## 1.3 Install the snvme kernel module

Build artifacts land in `build/cuda-module/module/` (`snvme-core.ko` + `snvme.ko`). Install from the build directory:

```bash
cmake --build --preset cuda-module --target insmod
```

Check that it installed cleanly:

```bash
$ lsmod | grep snvme
snvme                 217088  0
snvme_core            106496  1 snvme
```

To unload, use `make rmmod`. To reload, use `make insmod` (or your site's signed-module install flow).

## 1.4 Build the test code

```bash
make -C tutti/device_manager/nvme/kernel_modules/test -j"$(nproc)"
```

The test binaries depend on the SNVMe UAPI header `tutti/include/uapi/tutti_snvme.h` (already pulled in via the Makefile). The role of each binary is described in [Part 2 Step 3](#step-3--build-the-test-binaries).

---

# Part 2 — SNVMe Testing

## Mental model

SNVMe exposes three kinds of `/dev` objects:

| Path | What | Used for |
| --- | --- | --- |
| `/dev/snvm_control` | factory entry (1 per module) | bind/unbind a PCI device, create per-ctrl chrdev |
| `/dev/ssnvme<N>` | per-controller char dev (**double s**) | BAR0 mmap + all queue ioctls |
| `/dev/snvme<X>n<Y>` | block device (**single s**) | normal mount target, appears after bind |

Testing climbs a ladder of 6 binaries, **safest → most destructive**. Golden rule: **run `snvme_smoke` first; only move up a rung after the one below passes.**

---

## Step 0 — Pre-flight

```bash
uname -r                                                  # 5.15.x → matches the snvme-5.15.0-public baseline
grep CONFIG_MODULE_SIG_FORCE /boot/config-$(uname -r)     # "not set" → unsigned .ko loads fine
lsmod | grep snvme                                        # snvme + snvme_core loaded
ls -l /dev/snvm_control                                   # exists, mode 0666
```

If all four pass you can skip straight to [Step 3](#Step-3--Build-the-test-binaries). Steps 1–2 are only needed when you've changed driver code and have to rebuild.

---

## Step 1 — (Re)build the module (only after editing driver code)

The `.ko`s were built in Part 1 under `build/cuda-module/module/`. After changing driver code, rebuild:

```bash
cmake --build --preset cuda-module --target modules
# or by hand:
cd build/cuda-module/module && make
```

## Step 2 — Rebuild / reload the module (only after editing driver code)

> ⚠️ **Hard rule: after any driver change you must reload the `.ko`.** The smoke binaries embed `_IOC_SIZE`-derived ioctl numbers; a stale module returns `-ENOTTY` on valid requests.

Before rmmod, unbind all snvme-owned controllers and stop any process holding `/dev/snvm*` fds (or mounted `/dev/snvme*n*` filesystems), otherwise the module stays referenced:

```bash
sudo bash scripts/unbind.sh          # unbind all snvme controllers (idempotent)
lsof /dev/snvm_control /dev/ssnvme* /dev/snvme*n* 2>/dev/null   # should print nothing
cmake --build --preset cuda-module --target rmmod     # sudo rmmod snvme snvme_core
cmake --build --preset cuda-module --target insmod    # reload
```

> **Queue depth**: there is no `io_queue_depth` module parameter — snvme always
> takes the controller maximum (`q_depth = NVMe CAP.MQES + 1`, typically 1024 on
> datacenter SSDs). It is fixed at probe time; changing it requires rmmod+insmod.

---

## Step 3 — Build the test binaries

```bash
# Run from the project root. Builds all CPU binaries and, when nvcc is
# available, the CUDA binary as well.
make -C tutti/device_manager/nvme/kernel_modules/test

# CPU-only build (does not invoke nvcc):
make -C tutti/device_manager/nvme/kernel_modules/test cpu

# CUDA test only; override the architecture when auto-detection is unavailable:
make -C tutti/device_manager/nvme/kernel_modules/test gpu CUDA_ARCH=sm_80
```

This builds 5 tests + 1 reset helper:

| Binary | Binds? | Writes? | What it does |
| --- | --- | --- | --- |
| `snvme_smoke` | no | no | libc-only UAPI smoke: chrdev create/remove + `NVM_MAP_HOST_MEMORY` + BAR0 mmap |
| `snvme_smoke_qgroup` | no | no | queue-group lifecycle (create/destroy + fd-close cascade) |
| `snvme_smoke_gpu` | no/yes | no | adds `NVM_MAP_DEVICE_MEMORY` / p2p path (built only if `nvcc` is on `$PATH`) |
| `snvme_smoke_addq` | **yes** | no | B3 `NVM_ADD_USER_QUEUE` end-to-end (Create I/O CQ+SQ + cascade destroy) |
| `snvme_smoke_io` | **yes** | **yes** | B3 CPU end-to-end, 23 phases, byte-by-byte verification, **writes LBA** |
| `snvme_ubind` | — | no | owner-side reset helper: `SNVM_DEVICE_UNBIND` + `SNVM_CHRDEV_REMOVE` |

---

## Step 4 — Pick a test device

Inspect local nvme disks and the GPU↔NVMe topology distance, and pick a **throwaway** disk:

```bash
sudo bash scripts/pci_topology_check.sh
```

**The following test will destroy the data on the disk; please ensure that no important data remains on the disk.**

The rest of this guide uses a `TGT` variable for the chosen BDF:

```bash
export TGT=0000:e3:00.0     # ← replace with your own target BDF
```

---

## Step 5 — SAFE tests (no bind, won't disturb any disk)

These exercise only the UAPI and won't detach the in-tree `nvme` driver — safe even on a production host:

```bash
cd tutti/device_manager/nvme/kernel_modules/test

# 5a. libc-only UAPI smoke: chrdev create/remove + NVM_MAP_HOST_MEMORY + BAR0 mmap
sudo ./snvme_smoke        $TGT

# 5b. queue-group lifecycle (create/destroy + fd-close cascade), no bind
sudo ./snvme_smoke_qgroup $TGT
```

Each exits **0** on success. A failure prints `[FAIL] step=<N> ... errno=<E>` and stops.

---

## Step 6 — DESTRUCTIVE tests (bind required)

> 🛑 **These detach the in-tree `nvme` driver, and `snvme_smoke_io` WRITES to LBA. Use a throwaway controller only.**

Climb the rungs in order:

```bash
# 6a. B3 bind + ADD_USER_QUEUE end-to-end (with host rings; binds, no LBA writes)
sudo ./snvme_smoke_addq    $TGT

# 6b. The big one — B3 CPU end-to-end, 23 phases:
#     PRP1 / PRP1+PRP2 / PRP_List / SGL (auto-skipped on PRP-only devices) + SQ-tail-wrap,
#     byte-by-byte data verification on every IO. WRITES TO DISK.
sudo ./snvme_smoke_io      $TGT
```

`snvme_smoke_io` returning 0 is the **authoritative CPU-side gate** — it's what catches the vaddr-mask and cap-after-MSI-X regressions.

### If 6a or 6c fails with "no room for user queues"

`snvme_smoke_addq` and `snvme_smoke_io` cap kernel-side IOQs to 36 by default (`NVM_SET_KERNEL_IOQ_CAP`), reserving the rest of the controller's grant for user queues. Some controllers grant few total IOQs; in that case the default cap ≥ the grant, the kernel consumes every queue, and `NVM_ADD_USER_QUEUE` fails.

Fix: set `SNVME_TEST_KERNEL_IOQ_CAP` to a value strictly less than the controller's grant before running:

```bash
# Check how many IOQs the controller actually granted (look for max_user_qid):
sudo ./snvme_smoke_addq $TGT

sudo ./snvme_ubind $TGT
```

```bash
# If max_user_qid == 0 (no room), re-run with a smaller cap.
# Example for a 31-queue controller — cap at 16 to leave 15 for userspace:
SNVME_TEST_KERNEL_IOQ_CAP=16 sudo ./snvme_smoke_addq $TGT
SNVME_TEST_KERNEL_IOQ_CAP=16 sudo ./snvme_smoke_io   $TGT
```

---

## Step 7 — GPU tests (only if the NVIDIA driver is loaded)

Skip this on a host with no GPU (no `nvidia-smi`). On a GPU host:

```bash
# safe (no bind) — adds the NVM_MAP_DEVICE_MEMORY / p2p path
sudo ./snvme_smoke_gpu --gpu 0 $TGT

# destructive, the authoritative B3 GPU gate: 4 full alloc/free rounds
sudo ./snvme_smoke_gpu --gpu 0 --rounds 4 $TGT
```

---

## Step 8 — Cleanup & recovery

After a clean run the binaries unbind themselves. If a test **dies mid-run** (e.g. `kill -9`), the controller can be left half-bound. Reset it with the prebuilt helper:

```bash
sudo ./snvme_ubind $TGT          # SNVM_DEVICE_UNBIND + SNVM_CHRDEV_REMOVE
```

To hand the block device back to in-tree nvme:

```bash
sudo ./snvme_ubind $TGT
echo $TGT | sudo tee /sys/bus/pci/drivers/nvme/bind 2>/dev/null
```

A full reset is `scripts/unbind.sh` + `make rmmod` + `make insmod` (see Step 2).

---