Mac Clusters Run 671B Models On-Prem for $38K

A four-node Mac Mini M4 Pro cluster, each with 48 GB RAM and priced at approximately $1,999, along with around $200 in Thunderbolt 5 cables, totals around $8,200. This setup serves Nemotron-70B at 8 tok/s and Qwen2.5Coder-32B at 18 tok/s under EXO Labs. Scaling up to four Mac Studio M3 Ultra workstations with 512 GB each, costing around $38,000, Stabilise.io reports the same stack running DeepSeek V3.1 671B at approximately 25 tok/s and Kimi K2 1T MoE at about 34 tok/s, entirely on-premises without cloud API or data egress.

The key architectural feature is macOS 26.2's RDMA-over-Thunderbolt 5, which Stabilise.io measured at 5–9 µs inter-node latency across an 80 Gb/s link, significantly lower than the ~300 µs typical of pre-RDMA Thunderbolt networking. This latency reduction allows clusters to shard layers with tensor parallelism across unified memory pools, avoiding delays on cross-node transfers. However, this feature is early-stage, requiring manual commands in macOS recovery mode, and any M1 or M2 node on Thunderbolt 4 reverts to TCP/IP. Apple's IOMMU isolates memory per peripheral, containing the blast radius if a node hangs.

Framework selection should be based on whether the critical path is time-to-first-token or sustained throughput. An arXiv benchmark on the Mac Studio M2 Ultra found MLX delivering the highest sustained generation throughput, MLC-LLM the lowest TTFT, llama.cpp the most efficient single-stream serving, and Ollama the most ergonomic deployment at a cost to both throughput and latency. In multi-node configurations, Virge.io reports EXO Labs achieves a 1.8× speedup with two nodes and 3.2× with four via tensor parallelism; an eight-Mac-Mini array pushes DeepSeek V3 671B at 5.37 tok/s.

Inter-node latency, even with RDMA, makes these clusters better suited for batch inference than highly interactive chat, according to Stabilise.io, and the arXiv paper cautions that Apple Silicon inference frameworks still trail NVIDIA GPU-based systems such as vLLM in absolute performance.

Mixture-of-Experts models introduce a specific bottleneck. An arXiv study running DBRX 132B on a Mac Studio M2 Ultra cluster found that communication time approaches computation time during expert routing, requiring custom memory optimization to prevent the interconnect from dominating the critical path. That workload was reported as 1.15× more cost-efficient than an NVIDIA H100 supercomputer, but only after the memory layer was tuned manually.

The compliance argument is architectural, not benchmark-driven. Since weights and prompts never leave the building, Stabilise.io positions the stack as simplifying GDPR, HIPAA, and NIS2 posture: no third-party API, no egress monitoring, no rate-limit negotiation. For teams under strict data-residency rules, this property can outweigh raw tok/s.

Architects should consider the RDMA-over-Thunderbolt 5 tensor-parallel pattern, orchestrated through EXO Labs or MLX Distributed, to deploy a 100B+ model on-prem for under $10,000. Apple's edge is memory capacity per dollar, sub-200W power draw, and data sovereignty—not absolute throughput. If the decision hinges on beating vLLM latency or saturating H100-class bandwidth, this is not the stack. If it hinges on keeping weights inside the firewall while serving a 671B-parameter model, the cluster is viable today, provided engineering hours are budgeted for recovery-mode RDMA setup and MoE memory tuning.