Muthaiah Venkatachalam of Intel and Tate Berenbaum of Not Community Labs published a paper August 19 demonstrating that commodity Intel AI PCs can run distributed LLM inference without custom kernels or cloud hardware. A two-node Llama 3.1 8B setup reached 43.97 tokens per second serving two concurrent users—1.79× the throughput of a single machine. A four-node Lunar Lake fleet ran a 70B model at interactive speed; no single machine could fit it.
| Nodes | Hardware | Model | Throughput | Concurrent Users | Key Result |
|---|---|---|---|---|---|
| 1 | Intel AI PC (iGPU) | Llama 3.1 8B | Baseline | 1 | Single-machine ceiling; 70B model does not fit |
| 2 | Intel AI PC (iGPU) | Llama 3.1 8B | 43.97 tok/s (1.79× single) | 2 | 1.79× throughput vs. single node |
| 4 | Lunar Lake (iGPU) | Llama 3.1 70B | Interactive speed | — | Fits a 70B model that no single node can hold |
The architecture partitions models at layer boundaries into per-stage shards, each pre-compiled as INT4 OpenVINO IR. Machines load their shard, decode one step, and pass activations to the next node over TCP. Per-token round trips dominate latency. Speculative decoding—drafting multiple tokens per forward pass—mitigates this, but stateful OpenVINO models hit a wall: KV-cache rewind costs 48 milliseconds per call on Arc B390 at a 72-token cache, erasing gains.
First fix: zero out attention_mask positions instead of physically trimming the cache. This is bit-exact with physical trim and costs nearly nothing, unlocking 1.33× mean speedup on single-node speculation, rising to 1.6× at 2048-token generations.
Second fix: beam_idx Gather injection. Standard export tooling fails on modern shards due to dynamic control flow in rotary embeddings and KV cache ops. The team switched to torch.jit.trace with precomputed rotary embeddings, then injected beam_idx Parameter and Gather nodes post-export. This triggers OpenVINO's IndirectKVCache fusion in the GPU plugin. Without it, shards run 13–23% slower than monolithic models. With it, they match monolithic throughput within 4%.
| Fix | Root Problem | Solution | Measured Gain |
|---|---|---|---|
| 1 — KV-cache rewind | Physical KV-cache trim costs 48 ms per call on Arc B390 at 72-token cache, erasing speculative-decoding gains | Zero out attention_mask positions instead of physically trimming (bit-exact equivalent) | 1.33× mean speedup on single-node speculation; up to 1.6× at 2048-token generations |
| 2 — beam_idx Gather injection | Standard export fails on modern shards due to dynamic control flow; shards run 13–23% slower than monolithic models | torch.jit.trace with precomputed rotary embeddings + post-export beam_idx Parameter & Gather node injection → triggers IndirectKVCache fusion in GPU plugin | Throughput within 4% of monolithic model (from 13–23% deficit) |
| 3 — Micro-batching | Multiple concurrent user streams compete for pipeline stages with no native interleaving support | Leverage OpenVINO InferRequest stateful KV caches to interleave streams across pipeline stages without framework changes | 1.80× system-throughput scaling with 2 streams |
Third: micro-batching. OpenVINO InferRequest objects carry stateful KV caches, letting separate user streams interleave across pipeline stages without framework changes. Two streams yield 1.80× system-throughput scaling. Under 100 millisecond WAN latency, the combined stack—sharding, speculation, and micro-batching—delivers 4.04× the throughput of naive pipeline decode, which drops below interactive threshold in that scenario.
Known gaps remain. KV cache growth still erodes throughput on 70B long-context deployments. NPU support exists but benchmarks run on iGPU. INT8 KV caching ran slower and was dropped.
Code, logs, and reproduction scripts are at github.com/labscommunity/pipeline-sharded-inference-paper under CC BY 4.0. Edge-distributed LLM inference on Intel AI PCs was bottlenecked by three specific graph and scheduling gaps in OpenVINO—each now closed with patches ready for deployment.