Does moving a GPU server's request loop into the kernel save CPU? | eunomia
eunomia.dev
bpf-gpu-server lets eBPF and iouring own the TCP path of a GPU server so that no user thread touches a request. Against an efficient epoll server driving the same GPU executor, its CPU use and latency matched the baseline. This post explains where the per-request cost actually goes.
    • lobsters__valpackett@crustaceanson.theshelf.topM
      1·
      19 days ago

      Can’t tell that either but the entire idea might’ve been vibe-“thought”. It should be generally obvious to anyone with a brain that the throughput of an inference server (even with “smol” models) is gonna be entirely dominated by the, uhh, inference, and the overhead of request processing is negligible.

      score 5 · canonical