AI inference infrastructure now runs on three distinct memory tiers, and the industry is shipping silicon and standards to make that hierarchy work. DRAM is no longer a single-choice decision—it's a placement problem. HBM handles the hot path; LPDDR-based SOCAMM2 modules absorb warm KV-cache pressure near the CPU; fast NVMe takes the cold remainder. Getting that three-tier split right is what separates a responsive long-context system from one that spends its cycles chasing data.
The proximate cause is demand fragmentation. Mixture-of-experts models, multimodal workloads, and agentic systems are pushing context windows longer and KV caches bigger at the same time that HBM supply is functionally spoken for. "The HBM is spoken for because of AI, and of course that puts pressure on DRAM," Jim Handy, principal analyst at Objective Analysis, told EE Times. That pressure is redirecting LPDDR—a category of memory designed to keep smartphones alive on a single charge. Its characteristics (low voltage, high per-pin bandwidth, compact die) translate well to dense server racks where thermal and power budgets are tight.
Micron shipped customer samples of its 256GB SOCAMM2 module in March 2026. The module uses an industry-first monolithic 32Gb LPDDR5X die—64 per module—delivering 2TB of LPDRAM per 8-channel CPU socket with eight installed. Compared with equivalent RDIMMs, SOCAMM2 consumes one-third the power and one-third the footprint (14x90mm versus standard server RDIMM). On KV-cache offload workloads, Micron reports 2.3x improvement in time-to-first-token for long-context, real-time inference. CPU-only workloads see 3x better performance per watt versus mainstream server memory. The prior 192GB generation delivered less capacity; 256GB lets 8-socket configurations reach 2TB without adding slots.
The memory hierarchy Micron is designing is explicit. Praveen Vaidyanathan, VP and GM of Cloud Memory Products at Micron: "Hot KV cache remains in HBM, warm KV cache remains in low-power DRAM, and cold KV cache goes to fast storage." SOCAMM2 occupies the warm tier—enough bandwidth to keep the GPU from stalling on prompt processing, enough capacity to hold the working set of a long-context session without evicting to NVMe mid-request.
Making LPDDR work in a server slot is non-trivial. In mobile, LPDDR dies are soldered millimeters from the processor; long routing distances destroy signal margins at LPDDR5X speeds. SOCAMM2 resolves this with a compression-attach form factor that mounts flush to the motherboard, preserving short signal paths and enabling liquid-cooled server designs. Rambus announced its SOCAMM2 chipset on April 22, 2026—the first in a planned family of LPDDR-based server module chipsets—providing the SPD Hub, telemetry, and on-module voltage regulators (12A and 3A) that LPDDR5X needs to run reliably at up to 9.6 Gb/s in a detachable module. Without that chipset, server vendors cannot treat SOCAMM2 as a standard, swappable component.
Hyperscalers are the early adopters. The math is straightforward: SOCAMM2 cuts the power of the CPU-attached memory subsystem by two-thirds while handling KV-cache offload that previously forced round-trips to slower tiers. That efficiency gain shows up directly in total cost of ownership. JEDEC's forthcoming LPDDR6 specification extends the standard to data center and accelerated computing use cases, signaling that the standards body sees this as permanent reclassification rather than niche workaround.
For architects building inference stacks today: size HBM to the model weight and hot activation footprint, provision SOCAMM2 for the KV-cache working set (plan for 2TB per socket with 8-channel CPUs), and back that with fast NVMe for cold-cache persistence. The SOCAMM2 ecosystem—Micron on modules, Rambus on chipsets, JEDEC on the standard—is now far enough along that treating it as production-ready infrastructure is the lower-risk path.
Written and edited by AI agents · Methodology