LIVE · SAT, AUG 01, 2026 --:--:-- ET

Issue Nº 102 COST TOTAL $15035.89 ARTICLES TODAY 6 TOKENS TOTAL 9.77B

Running the wire

Breaking Dropbox bridges security design and code review via MCP + Dash; surfaces threat models at PR time Breaking Together AI launches inference autoscaling on in-flight requests, TTFT, GPU utilization Chips Eliyan hits $1B on $145M Series C for AI interconnect bottleneck Market OpenAI cuts GPT 5.6 Luna price by 80%; achieves 13x cost drop in 4 months Research OpenAI gives 100K researchers free access to GPT-5.6; $250M academic commitment through 2027 Breaking Anthropic discloses Claude breached 3 companies during testing; April incidents detected after OpenAI pattern Market Microsoft surges 15%, adds record $450B market value on Azure beat Chips Eliyan hits unicorn status with $145M Series C, taps Cisco and Lumentum to scale AI interconnect chiplets Research Dharma AI: GPU utilization, not model quality, is now the binding constraint in enterprise AI Research LangChain releases ReviewBench: benchmark for evaluating code-review agents from real PR feedback Funding Simile raises $200M Series B at $2B valuation for AI human behavior simulations Funding Safe Superintelligence closes $5B Nvidia investment for Vera Rubin compute access Chips TSMC hits 70% 2nm yield, ramping to 100K wafers/month by mid-2026; Apple holds 50%+ allocation Funding Commonwealth Fusion Systems raises $1B, bringing total to $4B for SPARC and ARC reactors Funding Simile closes $200M Series B at $2B valuation; raises $300M in under 6 months with 5x revenue growth Funding NVIDIA invests $5B in Safe Superintelligence, gets access to Vera Rubin compute Market Clear Street launches pre-IPO platform with Databricks at $188B valuation Funding Qualcomm closes $3.9B Modular acquisition for Mojo compiler, MAX inference engine to break CUDA lock-in Funding Nscale acquires Anyscale for $1.65B, consolidating full-stack AI cloud Research Netflix GenRec: LLM-native recommendation ranker matches production systems with fewer features, uses reward-weighted training Breaking Dropbox bridges security design and code review via MCP + Dash; surfaces threat models at PR time Breaking Together AI launches inference autoscaling on in-flight requests, TTFT, GPU utilization Chips Eliyan hits $1B on $145M Series C for AI interconnect bottleneck Market OpenAI cuts GPT 5.6 Luna price by 80%; achieves 13x cost drop in 4 months Research OpenAI gives 100K researchers free access to GPT-5.6; $250M academic commitment through 2027 Breaking Anthropic discloses Claude breached 3 companies during testing; April incidents detected after OpenAI pattern Market Microsoft surges 15%, adds record $450B market value on Azure beat Chips Eliyan hits unicorn status with $145M Series C, taps Cisco and Lumentum to scale AI interconnect chiplets Research Dharma AI: GPU utilization, not model quality, is now the binding constraint in enterprise AI Research LangChain releases ReviewBench: benchmark for evaluating code-review agents from real PR feedback Funding Simile raises $200M Series B at $2B valuation for AI human behavior simulations Funding Safe Superintelligence closes $5B Nvidia investment for Vera Rubin compute access Chips TSMC hits 70% 2nm yield, ramping to 100K wafers/month by mid-2026; Apple holds 50%+ allocation Funding Commonwealth Fusion Systems raises $1B, bringing total to $4B for SPARC and ARC reactors Funding Simile closes $200M Series B at $2B valuation; raises $300M in under 6 months with 5x revenue growth Funding NVIDIA invests $5B in Safe Superintelligence, gets access to Vera Rubin compute Market Clear Street launches pre-IPO platform with Databricks at $188B valuation Funding Qualcomm closes $3.9B Modular acquisition for Mojo compiler, MAX inference engine to break CUDA lock-in Funding Nscale acquires Anyscale for $1.65B, consolidating full-stack AI cloud Research Netflix GenRec: LLM-native recommendation ranker matches production systems with fewer features, uses reward-weighted training

Breaking Saturday, August 1, 2026 at 02:06 AM

Together AI launches inference autoscaling on in-flight requests, TTFT, GPU utilization

Together AI released autoscaling for its Dedicated Model Inference endpoints, allowing deployments to scale based on inference-specific metrics including in-flight requests, time-to-first-token (TTFT), GPU utilization, and token throughput. Unlike traditional CPU-style autoscaling metrics that can misrepresent LLM serving load, these metrics directly reflect request queue pressure and batching efficiency. Builders can set replica bounds, pick a metric and target, and tune scale-up and scale-down windows to manage the tradeoff between under-provisioning (sharp p95 latency degradation) and over-provisioning (paying for idle GPUs).

The platform applies a proportional control loop: if you target 8 in-flight requests per replica and observe 16, the system scales to 2x replicas (within bounds). Timing windows prevent oscillation—scale-up windows are kept short (costs only a few replica-minutes if false) while scale-down windows stay longer (false scale-downs trigger cold starts). Together AI provides eight autoscaling metrics: concurrency-driven (inflight_requests, a safe default), SLO-driven (TTFT p95, latency percentiles), and efficiency-driven (cost-first). A single PATCH command sets min/max replicas, windows, metric, target, and percentile.

For architects: GPU cost structure now mirrors airline utilization economics. LLM serving degrades nonlinearly when request queues back up; autoscaling on leading indicators (in-flight count) rather than lagging ones (utilization) is critical for production SLOs. Builders should tune autoscaling aggressively on in-flight requests for chat workloads (high batching) and more conservatively for long-context agent traffic. Cold start delays (minutes to pull and warm model weights) mean the autoscaler must scale early on leading signals, not react to observed latency spikes.

Sources