Architecture

Phoenix refactors the I/O stack for GPU Direct Storage (GDS) so that data moves from storage straight into accelerator memory via DMA, without being staged through CPU host memory (“phony buffers”). It is structured as a thin middleware layer that AI applications integrate through adapters.

Layers

            AI Application (vLLM, lmcache, ...)
                      │  adapter  (e.g. phxloader, via pybind11)
                      ▼
        ┌─────────────────────────────┐
        │  libphoenix  (user library) │   phxfs_open / regmem / read / write
        │  DevConnector (per-vendor)  │   vendor device lookup + page size
        └──────────────┬──────────────┘
                       │  ioctl / mmap  (char device /dev/phxfs_devN)
                       ▼
        ┌─────────────────────────────┐
        │   phxfs  (kernel module)    │   P2P map GPU BAR, DMA into GPU mem
        │   P2P backend (per-vendor)  │
        └──────────────┬──────────────┘
                       │  vendor P2P API (e.g. nvidia_p2p_get_pages) / PCIe P2P
                       ▼
                 GPU / xPU memory  ◄──── DMA from NVMe / NFS storage

Data path (no phony buffer)

  1. The application allocates a GPU buffer and registers it with Phoenix via phxfs_regmem. Phoenix mmaps a character device and uses ioctl(PHXFS_IOCTL_MAP) so the GPU memory is mapped into a host VMA backed by the GPU’s PCIe BAR (via ZONE_DEVICE).

  2. A phxfs_read / phxfs_write issues the file I/O directly against that VMA. The storage controller DMAs data straight into GPU memory — the CPU never touches the payload.

  3. phxfs_deregmem releases the mapping.

Component responsibilities

Path

Responsibility

module/

phxfs Linux kernel module: GPU BAR remap, per-GPU char device, mmap/ioctl P2P mapping; phxfs-backend.* selects the vendor P2P backend at build time

libphoenix/

User-space C/C++ library wrapping the char device; buffer registration and synchronous/async I/O; connectors/ holds the vendor-specific DevConnector

adapters/

Integration with AI frameworks via pybind11. adapters/vLLM/ ships phxloader for safetensors weight loading

test/

Correctness + performance tests (test_regmem, test_io, test_batch)

Multi-vendor support

Both the kernel module and libphoenix use a single compile-time switch, PHXFS_VENDOR (default NVIDIA), to select the accelerator vendor:

cmake -DPHXFS_VENDOR=NVIDIA ../   # default
cmake -DPHXFS_VENDOR=AMD ../      # requires module/amd-backend.c + libphoenix/connectors/amd_connector.cpp
cmake -DPHXFS_VENDOR=HUAWEI ../   # requires module/huawei-backend.c + libphoenix/connectors/huawei_connector.cpp

Core code (kernel: phxfs.c/phxfs-mem.c; user library: phx_device.cpp/phx_mem.cpp/phx_io.cpp) never references vendor APIs directly — it calls through phxfs_p2p (kernel) / devconn (user library) function-pointer tables. Adding a new vendor only requires implementing one backend file per layer; see doc/kernel-module.md and doc/libphoenix.md.

Supported environment (tested)

  • OS: Ubuntu 22.04 / TencentOS

  • Kernel: Linux 6.1 (see install.md for details)

  • Accelerator: NVIDIA GPU with CUDA 12.4+ and the open nvidia-fs driver

  • Storage: NVMe-of / NFS (local NVMe also supported for the direct path)

  • Other vendors (AMD, Huawei NPU): backend interface is ready; vendor-specific implementations not yet shipped (see roadmap.md)