LibPhoenix

libphoenix is the user-space C/C++ library that simplifies interaction with the phxfs kernel module. It manages device metadata and GPU buffer registration/unregistration.

Multi-vendor DevConnector

Vendor-specific calls (device discovery via CUDA/HIP/CANN) are abstracted behind struct devconn_ops (libphoenix/connectors/devconnector.h). The active connector is selected at compile time via PHXFS_VENDOR (default NVIDIA) and exposed through the global devconn pointer. Core files (phx_device.cpp, phx_mem.cpp, phx_io.cpp) call only through devconn->find_device() / page_size and never include vendor headers (e.g. cuda.h).

libphoenix/connectors/nvidia_connector.cpp implements the NVIDIA connector. Adding a new vendor means writing <vendor>_connector.cpp and pointing devconn at it; no other user-library file needs to change.

Driver management

`phxfs_open`

int phxfs_open(int deviceID);

Opens the character device for deviceID, initializes and stores the metadata required for later buffer registration. Opens are reference-counted: a second phxfs_open on the same device only adds a client reference.

`phxfs_close`

int phxfs_close(int deviceID);

Drops one client reference on deviceID. The last close waits for in-flight operations to drain, then unmaps every registration and closes the device. A concurrent phxfs_open on a draining device fails with -EBUSY (retryable).

Buffer management

`phxfs_regmem`

int phxfs_regmem(int device_id, const void *addr, size_t len, void **target_addr);

Registers a memory region (addr, len) for device_id: mmaps a VMA from the char device, then issues ioctl(PHXFS_IOCTL_MAP) to pin the GPU pages into it. Both addr and len must be non-zero and 64 KiB (device page) aligned. On success, target_addr receives the host-mapped address — an internal handle for reference only; the I/O calls identify a buffer by its original device address addr, never by target_addr.

Registration semantics: an exact-duplicate registration (same addr + len, still live) is reference-counted and reused (deregister once per register); any other overlap with a live registration is rejected with -EINVAL.

`phxfs_deregmem`

int phxfs_deregmem(int device_id, const void *addr, size_t len);

Drops one reference on the registration. The last reference waits for in-flight I/O on the region to drain, then removes the kernel mapping via ioctl(PHXFS_IOCTL_UNMAP) and munmaps the user-space VMA.

Single-request I/O

phxfs_read / phxfs_write transfer data directly between a file descriptor and the registered (GPU-backed) VMA:

ssize_t phxfs_read (int fd, int device_id, void *buf, off_t buf_offset, ssize_t nbyte, off_t f_offset);
ssize_t phxfs_write(int fd, int device_id, void *buf, off_t buf_offset, ssize_t nbyte, off_t f_offset);

device_id selects the buffer the same way as the batch API below: >= 0 means buf must lie inside a registration on that phxfs device; < 0 means buf is a plain CPU (host) address. For a registered buffer, buf may point anywhere inside the region; the host DMA address is resolved as vaddr + (buf - registered_base) + buf_offset, and an internal reference on the mapping is held for the transfer’s duration, so a concurrent phxfs_deregmem cannot unmap it mid-I/O. Large transfers are chunked at PHXFS_IO_CHUNK (1 GiB) to stay under the kernel’s MAX_RW_COUNT.

Batch I/O

For workloads that issue many independent transfers (e.g. KV-cache retrieve/store, weight loading), the batch API submits a whole set of requests in one call. This removes the per-request syscall overhead of looping over phxfs_read/phxfs_write and lets the storage layer service requests concurrently.

Request descriptor

typedef struct phxfs_io_req {
    int      fd;          // open file descriptor (O_DIRECT recommended)
    int      device_id;   // >=0: phxfs device the buf is registered on;
                          //  <0: plain CPU buffer
    void    *buf;         // GPU addr (registered) or CPU addr
    off_t    buf_offset;  // byte offset within buf
    size_t   nbytes;      // transfer length
    off_t    f_offset;    // file offset
    ssize_t  result;      // OUT: bytes transferred, or negative errno
} phxfs_io_req_t;

Each request’s buf is resolved independently to a DMA-able host address:

Registered GPU buffer (device_id >= 0) — buf must lie inside a phxfs_regmem registration on that device, or the request fails with -EFAULT (there is no silent CPU fallback). The mapping is reference-held for the batch’s duration.
Plain CPU buffer (device_id < 0) — buf is used as an ordinary host address (e.g. pinned staging memory).

Synchronous batch

int phxfs_read_batch (phxfs_io_req_t *reqs, int n);
int phxfs_write_batch(phxfs_io_req_t *reqs, int n);

Submits all n requests, blocks until every one completes, and fills each reqs[i].result. Returns 0 if every request transferred exactly nbytes; otherwise the number of failed requests (>0); or a negative errno on a submission-level engine error. Requests whose buffer cannot be resolved are marked result = -EFAULT and never handed to the engine.

Asynchronous batch (compute / I/O overlap)

phxfs_batch_t *phxfs_batch_submit_read (phxfs_io_req_t *reqs, int n);
phxfs_batch_t *phxfs_batch_submit_write(phxfs_io_req_t *reqs, int n);
int            phxfs_batch_wait(phxfs_batch_t *handle);
int            phxfs_batch_destroy(phxfs_batch_t *handle);

submit queues the batch on the internal worker pool and returns an opaque handle immediately; the caller can run GPU compute meanwhile, then phxfs_batch_wait blocks for completion, fills results, and frees the handle. wait returns the same value convention as the synchronous batch. phxfs_batch_destroy abandons a batch whose results are not needed: it waits for in-flight I/O to quiesce, then frees the handle without copying results back.

Submitted batches queue up on a bounded FIFO and run one at a time with the pool’s full worker set, so several submits can be pipelined ahead (e.g. layerwise prefetch); if the queue is full, submit fails with NULL + errno == EBUSY. An empty batch (n <= 0) returns a valid handle whose wait returns 0, matching the synchronous API.

Lifetime & concurrency contract

Everything a request references must stay valid until the call (sync) or phxfs_batch_wait (async) returns:

reqs[i].fd must remain open — fds are not dup()’d internally.
CPU buffers must not be freed.
Registered GPU buffers are protected by an internal reference; a concurrent phxfs_deregmem on them blocks until the batch completes rather than unmapping under an in-flight transfer.

A batch may freely mix requests targeting different devices and CPU buffers; all requests share the same worker pool.

I/O engines & worker pool

The batch path is built on two internal, pluggable layers:

I/O engine (libphoenix/io_engine/io_engine.h) — selected once at library load, in preference order io_uring → sync. The io_uring engine keeps a per-thread ring (QD 1024) with a pre-allocated slice/iovec scratch pool and a sliding-window submit/reap pipeline; it uses the non-fixed OP_READ/OP_WRITE path (same O_DIRECT DIO route as pread, no second long-term pin on device pages). The sync engine is a pread/pwrite loop and is the always-available fallback (also used per-thread if a worker’s ring cannot be created). phxfs_io_engine_name() reports the active engine, for diagnostics/tests.
Worker pool (libphoenix/io_engine/io_pool.cpp) — a single pool of worker threads, each running the engine on its own ring. A batch is striped round-robin across the workers, so a single blocking call can saturate the array while the caller crosses the Python GIL only once. Submits queue up (bounded FIFO) so a caller can pipeline several async batches ahead. There is deliberately no NUMA pinning: the P2P transfer is device-to-device DMA that never touches host RAM, so which CPU/node issues the I/O is irrelevant to the data path. The pool registers a pthread_atfork child handler — a forked child lazily re-creates the pool on first use (in-flight async handles do not survive fork).