Inference Radar

TL;DR

KV-cache routing became the shared battleground: vLLM, SGLang, Ray Serve, ai-dynamo, and ROCm ATOM all moved work into prefix reuse, cache routing, and disaggregated serving paths.1
On-device inference moved from demo to product plumbing: Google LiteRT-LM, Meta ExecuTorch, Qualcomm GenieX, Apple MLX, and OpenVINO GenAI all shipped work around mobile, browser, NPU, and Apple Silicon inference.
Blackwell and ROCm paths both got sharper: NVIDIA TensorRT-LLM, FlashInfer, SGLang, vLLM, ROCm AITER, and ROCm ATOM all expanded low-precision kernels and model support for newer accelerator targets.
Local runtimes kept closing the UX gap: Ollama, llama.cpp, LocalAI, Open WebUI, and vllm-mlx shipped practical fixes for Gemma, tool calling, multimodal serving, Apple Silicon, and OpenAI-compatible behavior.
Security and model-file safety stayed visible: TensorFlow, Google LiteRT, ONNX Runtime, OpenVINO GenAI, BentoML, GPT4All, and sherpa-onnx all saw parser, deserialization, download, or local-server hardening work.

This Week in Inference

OpenAI previewed GPT-5.6 Sol, Terra, and Luna as a tiered closed-model family, with Sol aimed at premium reasoning, Terra at balanced general use, and Luna at lower-cost serving OpenAI.2 Google pushed the local side of the same trend with Gemma 4 Unified 12B, a local multimodal model aimed at devices in the 16 GB memory class Google Gemma.3 Liquid AI added LFM2.5-230M to the tiny local-model track, which matters because sub-billion-parameter models can run inside phone, keyboard, and sensor workflows where latency and power beat leaderboard scores Bodega One.4

Compression research kept moving below the 4-bit line. CAT-Q proposed ternary PTQ near 1.58-bit weight-only compression with a small calibration set, which gives memory-bound serving teams another path between full retraining and classic W4 PTQ CAT-Q.5 TWLA paired ternary weights with 4-bit activations and reported large memory savings against FP16, which points toward the next production fight: keeping activation cost low without wrecking generation quality TWLA.6 AAAC targeted the practical PTQ sweet spot with activation-aware adaptive codebooks that run in minutes on a single GPU, which fits production teams that need fast conversion loops AAAC.7

Hardware news pushed the same vertical stack message. OpenAI and Broadcom announced Jalapeño, a custom inference ASIC focused on memory bandwidth, compute balance, networking, and performance per watt TechCrunch.8 Google Cloud and NVIDIA highlighted L4-backed inference through Vertex AI, a reminder that many production workloads need cost-efficient midrange GPUs rather than flagship accelerators GPU Trade.9 Rebellions acquired SqueezeBits, tying accelerator hardware to inference optimization software in one company Rebellions.10

Deeper Dive

Everything below is for readers who want the full picture. Feel free to scroll.

Code Changes by Category

Cloud & Datacenter Serving

vLLM had the broadest serving week, with MiniMax-M3, DSpark, DFlash, DeepSeek-V4, MXFP4, FP8 sparse GQA, ROCm tuning, parser fixes, sleep-mode backends, and Gaudi follow-through across long-context and MoE paths vLLM.1 SGLang focused on DeepSeek and GLM long-context serving, Decode-Context Parallelism, DSA fusion, FlashMLA sparse prefill, speculative decoding, HiCache, AMD, NPU, XPU, MLX, and diffusion paths SGLang.24 TensorRT-LLM advanced low-precision MoE, Minimax M3, Kimi, fabric-memory KV allocation, WideEP, Helix, AutoDeploy, and VisualGen while marking the end of support for the legacy TensorRT backend line TensorRT-LLM.15 Ray Serve added vLLM KV-event ingestion, Dynamo SelectionService integration, pre-routing tokenization, and KV-aware request lifecycle tracking Ray.25 ai-dynamo tightened OpenAI and Anthropic compatibility, cancellation-aware routing, LoRA integration, Kubernetes workflows, and GPU power-cap orchestration ai-dynamo.26

Local LLM Runtimes

llama.cpp synced ggml, added OpenCL binary-kernel loading, expanded CUDA, Vulkan, SYCL, WebGPU, HIP, MUSA, and model support, and shipped server fixes for SSE, offline downloads, reasoning, and chat templates llama.cpp.19 Ollama focused on Apple Silicon Gemma performance, MLX tests, Gemma tensor names, CUDA driver docs, Darwin build time, and streamed tool-call parsing Ollama.18 LocalAI added semantic VAD, native voice and face detection backends, distributed-state sync, model import fixes, macOS signing, download resilience, and backend bumps LocalAI.20 llamafile added a transcribefile CPU target, synced llama.cpp, improved Vulkan diagnostics, and fixed CPU fast-path numerical bugs that had looked like GPU regressions llamafile.27 GPT4All saw no code land, but users reported plaintext model metadata/download risks and wildcard CORS exposure on the local API server GPT4All.28

Apple Silicon & MLX Ecosystem

Apple MLX added small-batch quantized matvec for speculative decoding, asymmetric SDPA head dims, CUDA Windows CI, Array API functions, autograd fixes, JACCL deadlock repair, and Swift parity work MLX.29 mlx-swift and mlx-swift-lm shipped releases with optimizer, scheduler, Mamba2, Mixtral, Gemma, Qwen2-VL, tool-call, stop-string, and seeded sampling support mlx-swift-lm.30 Blaizzy pushed MLX audio and vision through ZONOS2, Higgs Audio, Confucius4-TTS, OmniVoice, MiniMax M3, Unlimited-OCR, Qwen3-Omni fixes, and M-RoPE LoRA training support mlx-vlm.31 jundot omlx focused on memory-pressure recovery, VLM cache teardown, long-context head_dim=256 prefill, MiniMax M3, GLM kernels, Gemma QAT, and /v1/responses reasoning output omlx.32 vllm-mlx shipped a substantial release with GPT-OSS harmony rendering, MLLM tool preservation, omni video-audio extraction, SSD cache support, and Metal cache fixes vllm-mlx.33

Mobile & Edge Frameworks

Google LiteRT and LiteRT-LM added runtime memory-safety fixes, StableHLO plumbing, Tensor API work, decoding controls, NPU mask fixes, .litertlm web loading, and MediaPipe text/web/app updates LiteRT.34 Meta ExecuTorch expanded MLX quantized GGUF, top-k sampling, OpenAI-compatible Gemma serving, WebGPU tiled GEMM, Vulkan desktop GPU support, Arm/TOSA, and NXP Neutron coverage ExecuTorch.35 Qualcomm coordinated GenieX, AI Hub Models, and AI Hub Apps around Android and Windows demos, Qwen3-VL, scorecards, Snapdragon NPU docs, HTP fixes, and app publishing GenieX.36 Alibaba MNN improved QNN export reliability for large Qwen graphs, Android app linking, Hugging Face dataset loading, and RISC-V RVV int8 correctness MNN.37 sherpa-onnx improved Windows non-ASCII paths, ASR punctuation spacing, FreeBSD/Ninja builds, Piper install behavior, and QNN Whisper guidance sherpa-onnx.38

Compilers, Runtimes & Graph Engines

Apache TVM continued its PrimType and FFI type-system refactor while improving LLVM scalable-vector lowering, CUDA tile primitives, ONNX/TFLite importers, FlashInfer integration, and CUDA packaging TVM.39 OpenXLA added PDL launch insertion by default, persistent autotune cache layers, ROCm CI support, verifier checks, profiling hooks, and dynamic-slice fusion work that later rolled back OpenXLA.40 Triton added tl.atomic_poll, reshape migration work, sanitizer support, NVIDIA multi-CTA barriers, AMD release fixes, and extension API cleanup Triton.41 ONNX and ONNX Runtime tightened graph safety, reference semantics, plugin EP behavior, CUDA/WebGPU/CPU LLM kernels, external-data handling, and C++ modernization ONNX Runtime.42 OpenVINO added GroupedMatMul, GPU dynamic quantization, ONNX LayerNormalization outputs, GenAI prefix-cache fixes, Qwen3-ASR, Gemma paged attention, and NNCF FP8 compression OpenVINO.43

Models, Quantization & Optimization

Hugging Face Transformers added MiMo-V2, ZAYA1-8B, RADIO, Qwen3 ASR, Nemotron ASR Streaming, XCodec2, export fixes, StaticCache flash attention, PEFT updates, and tokenizer path-traversal hardening Transformers.44 Hugging Face Diffusers added DiffusionGemma, Ideogram4 DreamBooth LoRA, Krea LoRA loading, Kohya conversion fixes, Ovis Image fixes, and multi-GPU VAE handling Diffusers.45 Candle improved CPU quantized inference, CUDA paged flash-attention, GGML quantized embeddings, and prepared a new release cycle Candle.46 FlashInfer expanded Blackwell and SM12x serving coverage, NVFP4 fused MoE, GDN kernels, TRT-LLM FP8 MoE, LoRA payloads, workspace helpers, and AOT wheel-size control FlashInfer.47 ROCm AITER and ATOM pushed DeepSeek-V4, MiniMax-M3, sparse attention, FP8/MXFP4 MoE, GDN, LMCache offload, and SGLang integration ROCm ATOM.48

Other Notable Changes

Open WebUI shipped folder sharing, read-only access fixes, streamed reasoning display, RAG ACL fixes, Pyodide index handling, hidden-model selector cleanup, UI polish, localization, and a large refactor wave Open WebUI.49 LiteLLM shipped a fast provider-compatibility train across Anthropic, Bedrock, Vertex, Databricks, Ollama, MCP, OAuth, guardrails, realtime, and spend telemetry LiteLLM.50 Osaurus moved agent workflows forward with local ML reliability, subagents, Computer Use, Agent Channel safety contracts, diagnostics, image generation, Rampart PII, and MCP fixes osaurus.51 RunanywhereAI stabilized its cross-platform SDK across telemetry, Flutter, React Native, Web/WASM, iOS, VLM streaming, and voice-agent paths Runanywhere SDKs.52 ZETIC expanded Melange demo apps across fire detection, skin classification, multimodal wellness, voice biomarkers, and camera vitals ZETIC Melange Apps.53

Community Pulse

The hottest user pain clustered around Windows and GPU compatibility. Ollama users reported NVIDIA driver and CUDA crashes on older Pascal cards and new Blackwell workstations, while maintainers clarified driver floors and fallback behavior Ollama.54 vLLM users hit Blackwell DeepGEMM startup failures, ROCm GPTQ assertions, tool-parser divergence, and multimodal payload bloat vLLM.55 SGLang users pushed on Qwen, GLM, MLX, diffusion, DSpark, and long-context behavior SGLang.56

Security reports stayed high-signal. BentoML received a CGNAT SSRF bypass report and a sync-timeout concurrency issue, both with production impact BentoML.23 GPT4All saw reports about plaintext model metadata/downloads and wildcard CORS on the local API server GPT4All.57 OpenVINO GenAI fixed a GGUF Q4_K/Q6_K heap overflow, while TensorFlow and LiteRT continued broad parser and bounds hardening OpenVINO GenAI.58

Community energy also moved toward stateful assistants. Open WebUI users debated memory prompts, RAG scoping, SearXNG setup, and agentic search Open WebUI.59 text-generation-webui users revived persistent memory and per-user chat history discussions despite no code landing text-generation-webui.60 Osaurus users asked for in-chat generative UI, native self-improvement loops, CLI-like filesystem menus, and PDF-plugin workflows osaurus.61

Community Debates

Ollama maintainers rejected exposing num_cpu_moe as a user-facing knob because they want hardware-dependent fit behavior to stay automatic, and the thread redirected toward sliding-window KV cache estimation fixes Ollama.62 The same repo kept integrated GPU enablement conservative after maintainers found Windows and Linux instability on a ROCm iGPU path Ollama.63

vLLM debates centered on where saturation control belongs. A max-waiting-queue proposal raised the boundary between API-server rejection and scheduler backpressure, while dtype-auto work exposed how many kernels still assume explicit BF16 or FP16 KV semantics vLLM.64 That is the right fight for vLLM to have, because admission control and dtype defaults now affect reliability as much as peak tokens per second.

SGLang maintainers pushed design work away from isolated memory pools toward shared-pool and segment-tracking approaches, and they closed async diffusion finalization after adaptive gating failed to beat always-async behavior SGLang.65 The project also used several unmerged PRs as CI probes for release and hardware validation, which shows a mature but noisy path for testing large serving changes SGLang.66

TensorFlow and TFLite Micro showed the classic safety versus fast-path split. TensorFlow maintainers rejected overflow checks in RuntimeShape::FlatSize because they want FlatSize to remain the optimized path and CheckedFlatSize to carry validation TensorFlow.67 TFLite Micro closed several hardening PRs in favor of a canonical defensive-programming guide and narrower fixes TFLite Micro.68

Triton maintainers redirected a per-lane loop-retirement pass out of tree because reviewers viewed it as a niche optimization better suited to an extension, even though the author reported strong speedups on witness kernels Triton.69 That debate matters because compiler projects need extension surfaces that let specialized inference kernels move faster than core maintainers can safely accept.

Worth Watching

KV-cache ownership will keep spreading upward. Ray Serve, ai-dynamo, vLLM, SGLang, ROCm ATOM, and OpenVINO GenAI all added cache-aware behavior or fixed prefix-cache correctness this week Ray.70 The next step is likely cross-engine interoperability around cache events, token lifecycle metadata, and admission control.

Speculative decoding is shifting from a feature into infrastructure. vLLM moved DSpark and DFlash support forward, SGLang expanded draft-token and KV budgeting safety, llama.cpp added DFlash, Ollama accelerated Gemma on Apple Silicon with draft-token tuning, and MLX added small-batch quantized matvec for speculation vLLM.71

Edge inference will keep fragmenting by hardware, then reconverging through app SDKs. Google LiteRT-LM, ExecuTorch, Qualcomm GenieX, RunanywhereAI, FluidAudio, sherpa-onnx, MNN, and OpenVINO GenAI all moved platform-specific fixes this week Qualcomm GenieX.36 The useful abstraction will not be “mobile inference,” it will be model download, token streaming, telemetry, memory fallback, and per-device accelerator choice.

Sub-4-bit work needs production proof. FlashInfer, TensorRT-LLM, ROCm AITER, vLLM, OpenVINO/NNCF, Intel Neural Compressor, and Hugging Face Candle all pushed low-bit kernels or quantization tooling, while CAT-Q, TWLA, and AAAC kept the research pressure high FlashInfer.47

Major Releases

Version numbers and release-note links live here as the canonical reference.

vLLM shipped v0.24.0, a large release with 571 commits from 256 contributors and headline support for MiniMax-M3, BF16/FP8 indexers, MXFP4, FP8 sparse GQA, and AMD/ROCm tuning. The release also tracks the project’s move into speculative decoding, multimodal serving, and backend-specific reliability..1

SGLang shipped v0.5.14, adding release-note coverage for GLM-5.2, LiquidAI LFM2.5, Kimi-K2.7-Code, Poolside Laguna-M.1, DiffusionGemma, Zyphra ZAYA1, and MiMo-V2-ASR. The broader week focused on DeepSeek/GLM long-context serving, speculative decoding, cache correctness, and AMD/NPU/XPU coverage..11

NVIDIA shipped TensorRT-LLM v1.3.0rc20, the last release candidate with TensorRT backend support, alongside notes on DeepSeek warmup instability, Qwen autotuning crashes, TeaCache APIs, and chat_template behavior. NVIDIA’s week centered on low-precision serving, Blackwell/Thor readiness, AutoDeploy, and VisualGen..15

Triton Inference Server shipped v2.70.0, corresponding to the NGC 26.06 release train. The repo also moved main toward the 26.07 container line, hardened Java SDK Maven dependency routing, and restored Torch-TensorRT QA coverage on newer NVIDIA hardware..72

Ollama shipped v0.30.11, v0.30.12-rc0, and v0.31.1, with the latest focused on faster Gemma 4 generation on Apple Silicon through multi-token prediction and draft-token tuning. The release train also carried Windows hybrid GPU fixes, MLX updates, llama.cpp refreshes, and streamed tool-parser fixes. Latest release.18

ggml shipped a rapid llama.cpp daily build train ending at b9859, with the latest build focused on OpenCL binary/precompiled-kernel loading and MoE GEMM paths. The weekly train also covered DeepSeek V4, DFlash, CUDA, Vulkan, SYCL, WebGPU, OpenVINO, offline downloads, and chat/server fixes. Latest release.19

Apple MLX ecosystem shipped mlx-swift 0.31.5 and mlx-swift-lm 3.31.4. The Swift releases focused on SwiftPM compatibility warnings, Linux/CUDA SwiftPM work, optimizer parity, and a stable Swift LLM tag before larger main-branch changes. mlx-swift-lm release.30

BerriAI shipped a dense LiteLLM release train spanning v1.84.2 through v1.92.0-dev.1, with stable, prerelease, and backport cuts. The dominant theme was provider compatibility, Docker cosign verification, MCP/OAuth hardening, spend-path performance, guardrails, realtime, and dashboard polish. Stable release.50

Ray shipped Ray 2.56.0, focused on Ray Data stability, safer multi-dataset execution, automatic CPU sizing for map_batches, default logical memory settings, tighter iter_batches, and early executor shutdown. Serve LLM KV-cache routing work continued on main after the release..73

FlashAttention shipped fa4-v4.0.0.beta20, a prerelease covering CuTe DSL compatibility, SM100 pointer cleanup, caller sync changes, and a temporary split-alignment build flag. The week’s main work centered on FA4 readiness, SM100 correctness, and CI hardening..74

FlashInfer shipped three nightly-v0.6.13 builds during the week. The release cadence reflects active Blackwell/SM12x serving work, NVFP4 MoE expansion, GDN kernels, MoE/LoRA payloads, and AOT wheel-size control. Latest nightly.75

LocalAI shipped v4.5.1 through v4.5.6, a fast patch train around realtime voice, native ggml detection backends, distributed-mode fixes, macOS packaging, model import, and backend bumps. The latest release fixed distributed NATS permissions and cache/docs updates. Latest release.20

Open WebUI shipped v0.10.0, v0.10.1, and v0.10.2. The train introduced shared folders, fixed read-only shared-folder access errors, and added streamed reasoning display, with related RAG, tools, UI, and localization fixes on main. Latest release.49

Qualcomm shipped ai-hub-models v0.57.0 and GenieX v0.3.10 through v0.3.13. The releases focused on a lighter discovery CLI, GenieX docs and packaging, signed HTP catalog support, Android libomp.so, quant fixes, and the broader GenieX app/model/SDK split. AI Hub Models release.14

ROCm shipped AITER v0.1.16.post2 and v0.1.16.post3. The releases backported MiniMax M3 MoE work and fixed unified-attention fp16 behavior for vLLM upgrades, while the main branches pushed DeepSeek-V4, MiniMax, MI400/gfx1250, sparse attention, and MoE kernels. Latest release.16

osaurus-ai shipped 0.21.0 through 0.21.3, focused on agent collaboration, Agent Channel safety, Computer Use, diagnostics, image-generation UX, Rampart PII, Mistral provider support, localization, and vmlx/plugin-auth fixes. The cadence shows a fast product loop around safe autonomous local agents. Latest release.51

jundot shipped omlx v0.4.5.dev1, a prerelease focused on GLM and MiniMax prefill speedups, custom kernels, model presets, and VLM/cache hardening. The week also addressed memory pressure recovery, teardown cleanup, Gemma/Qwen support, and reasoning output..32

vllm-mlx shipped v0.4.0, with GPT-OSS harmony rendering, omni video-audio fusion, SSD KV cache coverage for multimodal models, configurable Metal memory behavior, and tool-calling/reasoning/MLLM fixes. It was one of the week’s most practical Apple Silicon serving releases..33

KV Cache Eats The Scheduler