TL;DR
- Decision models enter the serving path: Ollama, LocalAI, llama.cpp, Modular MAX, RunanywhereAI, and LiteLLM expanded decision and SystemOne surfaces, turning routing and scoring into first-class inference APIs.1
- Datacenter serving gets a new baseline: vLLM and SGLang pushed DeepSeek, NVFP4, KV-cache, AMD, XPU, and multimodal serving work across the high-throughput stack.2
- On-device embeddings became product features: Google AI Edge coordinated Gallery, LiteRT-LM, MediaPipe, and samples around EmbeddingGemma, while Apple MLX and MLX-VLM added local multimodal momentum.3
- Local runtimes widened hardware reach: llama.cpp, exllamav3, Ollama, Rapid-MLX, and omlx pushed portable performance across CUDA, Metal, Vulkan, ROCm, SYCL, and Apple Silicon.4
- Kernel work set the pace: FlashInfer, FlashAttention, Triton, OpenXLA, ROCm AITER, and TensorRT-LLM all focused on FP4, FP8, MoE, attention, and next-gen NVIDIA or AMD paths.5
This Week in Inference
Model momentum showed up as runtime support rather than a single launch: Google AI Edge centered its app, runtime, and samples on EmbeddingGemma, while vLLM and SGLang concentrated on DeepSeek Flash, GLM, Kimi, MiniMax, and Qwen-family serving paths Google AI Edge.3 Multimodal support also moved down the stack, with vLLM-Omni adding image, speech, video, music, and realtime serving paths while Apple MLX, MLX-VLM, Core AI, and Core ML conversion work improved local model execution on Apple hardware vLLM-Omni.6
The optimization story was memory and routing: KV-cache transfer, prefix-cache correctness, Mamba or GDN state reuse, disaggregated prefill/decode, and speculative decoding showed up in vLLM, SGLang, Dynamo, TensorRT-LLM, Ray Serve, and Mooncake vLLM.2 At the kernel layer, FlashInfer, FlashAttention, Triton, ROCm AITER, AMDMIGraphX, TensorRT-LLM, and OpenXLA all spent the week on FP8, NVFP4, MXFP4, sparse MLA, MoE routing, and Blackwell or MI-series execution paths FlashInfer.5
Hardware and infrastructure news came from code, not vendor launches: Google pushed LiteRT, MediaPipe, and XNNPACK for mobile and WebGPU; Qualcomm hardened HTP and QAIRT workflows; AMD’s Ryzen AI and ROCm stacks expanded NPU, MI-series, and local Windows paths; and NVIDIA’s TensorRT-LLM kept tuning OpenEngine, KV transfer, and speculative decoding Qualcomm GenieX.7 Industry moves were absent from the supplied market source set, so the signal this week came from repository behavior: more projects now treat cloud, local, browser, and edge as one inference continuum RunanywhereAI.8
Top Stories
vLLM turns the release train into a hardware-breadth test
vLLM shipped a large serving update around DeepSeek Flash, NVFP4, FlashMLA, decoder-boundary fusion, speculative decoding, and accelerator coverage across NVIDIA, AMD, Intel XPU, and Gaudi-adjacent work vLLM.2 vLLM-Omni followed with a matching prerelease that aligned multimodal and realtime serving to the same core runtime, including speech, diffusion, music, and Ray multi-node paths vLLM-Omni.6 This release cycle makes vLLM less a single OpenAI-compatible server and more the center of a multi-backend serving contract.
SGLang hardens the cache and omni layers
SGLang shipped a release while merging broad work across checkpoint loading, KV-cache tracking, HiCache, disaggregated serving, DeepSeek and GLM paths, AMD support, diffusion, and omni-modal lifecycle control SGLang.9 The core repo focused on cache correctness and performance, while sglang-omni tightened admin keys, media policy, CUDA IPC handling, cancellation, realtime speech, and TTS/video serving sglang-omni.10 SGLang’s week reads like production hardening after a fast model-enablement phase.
Google AI Edge makes embeddings visible to users
Google shipped a coordinated AI Edge release wave across Gallery, LiteRT, LiteRT-LM, and MediaPipe, with EmbeddingGemma powering on-device semantic media search, video moment finding, and embeddings APIs LiteRT-LM.3 Gallery turned that runtime work into user-facing Instant Media Search and Video Moment Finder, while samples and MediaPipe added decision, retriever, and universal embedder examples Gallery.11 The result is a clean example of model support becoming product behavior on phones and local devices.
Decision APIs spread across local and server runtimes
Ollama added Clef and SystemOne paths, LocalAI shipped native decision routing, RunanywhereAI added a local decision primitive and hosted Eve, Modular MAX added decision endpoints, and llama.cpp exposed SystemOne support LocalAI.12 LiteLLM also discussed Decisions API support and provider routing for decision-style models LiteLLM.13 Decision models now sit beside chat, embeddings, rerank, and tool use as a serving surface that many stacks need to route, meter, and secure.
Local runtimes chase portability without giving up speed
llama.cpp cut a major release with API changes, new model support, MoE caching, MTP work, and broad backend updates across CUDA, Metal, Vulkan, WebGPU, SYCL, OpenVINO, HIP, MUSA, and Hexagon llama.cpp.4 exllamav3 added preliminary ROCm support while improving CPU offload, memory lifecycle, vision model compatibility, and quantization correctness exllamav3.14 Apple-side projects such as MLX, Rapid-MLX, omlx, and Osaurus pushed the same race on M-series hardware Rapid-MLX.15
Deeper Dive
Everything below is for readers who want the full picture. Feel free to scroll.
Code Changes by Category
Cloud & Datacenter Serving
vLLM, SGLang, TensorRT-LLM, Dynamo, Ray Serve, Triton Inference Server, and LiteLLM all worked on production serving semantics rather than surface syntax vLLM.2 vLLM focused on NVFP4, FlashMLA, DeepSeek Flash, KV connectors, prefix-cache correctness, speculative decoding, and OpenAI-compatible behavior vLLM.16 SGLang worked across HiCache, disaggregated prefill/decode, DeepSeek and GLM kernels, diffusion serving, and sglang-omni lifecycle hardening SGLang.9
NVIDIA TensorRT-LLM advanced OpenEngine lifecycle behavior, KV transfer, C++ KVCacheManager work, speculative decoding metrics, MiniMax and Kimi paths, and telemetry or build modernization TensorRT-LLM.17 Dynamo tightened OpenAI-compatible parsing, structured outputs, multimodal routing, sidecar behavior for vLLM, SGLang, and TensorRT-LLM, and planner simulations Dynamo.18 Ray Serve improved FIFO routing recovery, LLM replica callbacks, health plumbing, gRPC benchmarks, and released a stable LLM serving line Ray.19
Triton Inference Server hardened HTTP generate request-size handling and moved build/runtime image baselines forward Triton Inference Server.20 LiteLLM strengthened enterprise proxy controls, MCP catalog handling, client-facing error behavior, usage aggregation, guardrails, and release signing notes LiteLLM.21
Local LLM Runtimes
llama.cpp and whisper.cpp drove a large ggml release cycle, with llama.cpp adding API updates, MTP and MoE work, model support, and backend modernization, while whisper.cpp synced ggml and improved model loading safety llama.cpp.4 Ollama shipped decision/SystemOne support, MLX fixes, OpenAI tool-message conversion, multimodal embeddings, cloud usage proxying, and a Windows Clef fix Ollama.1 LocalAI shipped audio scenes, ordered failover, native decision routing, signed OCI galleries, and operations UI updates LocalAI.12
exllamav3 made ROCm the headline platform addition and added HIP plumbing, ROCm kernels, CPU-offload improvements, memory cleanup, and vision model fixes exllamav3.14 Open WebUI focused on RAG/vector database reliability, Qdrant and Milvus correctness, pgvector search, tool-call serialization, paused tool flows, and citation behavior Open WebUI.22 text-generation-webui had no merged code, but users and contributors tracked a tool-call parser collision and AMD ROCm setup gaps text-generation-webui.23
Apple Silicon & MLX Ecosystem
Apple’s MLX stack worked across core kernels, quantized matmul, Metal residency, distributed communication, MLX-LM tool parsing, NVFP4 MoE scale support, Swift LM cache/session behavior, and Core AI Gemma export/runtime paths MLX.24 Core AI models added Gemma export/runtime support, GPT OSS prefill export, FP4/FP8 presets, and asset-validation fixes coreai-models.25 MLX-LM added MiniCPM tool parsing, Qwen parser fixes, server validation, and batched padding fixes MLX-LM.26
Blaizzy’s MLX projects shipped audio streaming fixes, Voxtral realtime handling, Qwen TTS timestamp and streaming fixes, EmbeddingGemma support in MLX-VLM, Responses API compaction, Kolibri support, and RF-DETR/VLM correctness work mlx-vlm.27 Rapid-MLX shipped several releases around accelerated model profiles, EmbeddingGemma embeddings, LTX video extension, agent setup, desktop reliability, and Apple-local serving Rapid-MLX.15 omlx pushed Qwen, GLM, MoE, MTP, SSD expert streaming, long-context cache fixes, vLLM-compatible tokenization endpoints, and MLX dependency updates omlx.28
Mobile & Edge Frameworks
Google AI Edge shipped a full on-device cycle across LiteRT, LiteRT-LM, Gallery, MediaPipe, samples, and XNNPACK LiteRT.29 LiteRT emphasized FlashAttention, Flash-Decode, fused GPU kernels, RAM reduction, Metal, MediaTek, vendor integration, and API updates LiteRT.29 MediaPipe updated build support, Python builds, LiteRT integration, Windows fixes, and iOS Tasks packaging MediaPipe.30
Meta ExecuTorch worked on Python bindings, CUDA allocation, Vulkan operators, Arm TOSA and VGF paths, Samsung and Qualcomm delegates, LLM batching, and MLX examples ExecuTorch.31 Qualcomm’s AI Hub and GenieX focused on Snapdragon deployment, QAIRT/QNN compatibility, HTP/FastRPC stability, export CLI changes, model catalog work, and benchmark tooling GenieX.7 Tencent ncnn expanded x86 SIMD, RISC-V Vector, Vulkan Tile, PNNX STFT export, and GPU test coverage ncnn.32
Compilers, Runtimes & Graph Engines
OpenXLA merged a large GPU and CPU compiler week, with PTX capability detection, collective runtime fixes, oneAPI/SYCL sort support, ROCm FP8 state, CPU fusion cleanup, TopK fallback, and dynamic-shape correctness fixes OpenXLA.33 TensorFlow carried related XLA fixes while TensorFlow Lite Micro removed legacy dependencies and internalized headers, namespaces, C APIs, and build logic TensorFlow Lite Micro.34 Triton language added AMD gfx and FP8 work, NVIDIA TMEM and mbarrier synchronization, low-precision Blackwell conversions, reduction correctness, and interpreter/runtime fixes Triton.35
Apache TVM continued TIRx and S-TIR migration, renamed core TIR concepts, added region/device semantics, reworked CUDA lowering, and improved Relax and WebGPU correctness TVM.36 TVM-FFI improved Rust object registration, module metadata, docs builds, and crates.io release tooling tvm-ffi.37 OpenVINO shipped a runtime and GenAI release train with ONNX/frontend LLM fixes, GPU/NPU/CPU work, GGUF multimodal projector support, and GenAI Qwen-family features OpenVINO.38
Models, Quantization & Optimization
Hugging Face shipped Diffusers and Transformers releases, adding Qwen-Image, EmbeddingGemma, Kandinsky, Echo, OpenVINO model support, distributed MoE work, hardware correctness, and Candle CPU quantized prefill optimization Transformers.39 FlashInfer spent the week consolidating generated CAKE kernels, reducing generated-code bloat, expanding Kimi, DeepSeek, MegaMoE, sparse MLA, SM100, SM103, and SM107 paths, and cutting release candidates FlashInfer.5 FlashAttention continued Blackwell hd256 forward/decode/backward unification, SM100 and SM103 scheduler work, and FA2 security-release prep FlashAttention.40
ROCm’s AITER, ATOM, and AMDMIGraphX formed one of the strongest optimization clusters, with MI-series decode, MLA, sparse MLA, MoE routing, MXFP4/MXFP8, DeepSeek, MiniMax, Kimi, Qwen, and AMDMIGraphX fusion work AITER.41 Microsoft Olive added NPU, ONNX IR, INT3, GPTQ, QKV precision, Gemma graph surgery, and GenAI runtime profile work Olive.42 Intel neural-compressor added JAX quantization-axis metadata propagation to quantize/dequantize composites neural-compressor.43
Other Notable Changes
RunanywhereAI made decision models a product surface across SDK, Wally CLI, local llama.cpp/MLX decision checkpoints, and local HTTP serving RunanywhereAI SDK.8 Osaurus shipped four macOS app releases around local JSON schema output, K2-Horizon, MTP controls, Qwen-Image, MCP tooling, paired-phone management, privacy presets, and UI stability Osaurus.44 FluidAudio shipped Phonon-2 Core ML ASR support while Mobius added Core ML conversion infrastructure for a Clef vision model FluidAudio.45
Community Pulse
The highest-energy communities were vLLM, SGLang, FlashInfer, LiteLLM, Open WebUI, Rapid-MLX, Google AI Edge, ggml, and ROCm, where review traffic mixed model enablement with production bug reports vLLM.2 Open WebUI’s process bots closed several useful proposals before technical review, which shows the project’s scale now creates governance overhead alongside product demand Open WebUI.46 FlashInfer and Triton showed a stricter bar for upstream compiler and kernel changes, with maintainers rejecting ad hoc optimizations unless they came with broad evidence and maintainable abstractions FlashInfer.47
Users kept reporting real deployment failures: Ollama users reproduced Windows Clef non-finite logits, vLLM users tracked NVFP4 and disaggregated corruption, SGLang users chased DeepSeek and GLM cache issues, and Google AI Edge users reported device accelerator crashes and mismatches Ollama.48 The pattern matters because the open-source inference stack now absorbs hardware, model, serving, and UX bugs in one public issue tracker vLLM.49
Community Debates
vLLM pushed back on compression without real packed KV storage. TurboQuant dynamic KV compression promised memory savings, but maintainers wanted packed storage plus Triton attention before accepting the design vLLM TurboQuant.50 That rejection sets a useful bar: serving projects want end-to-end memory behavior, not format claims that still store expanded state.
FlashInfer rejected CUDA graph autotuning as a default. The proposal to turn CUDA graph autotuning on by default ran into capture-safety, OOM, and timeout concerns for large groupwise FP8 GEMM autotuning FlashInfer CUDA graph debate.47 FlashInfer maintainers favored safety because kernel autotuning now runs close to production serving paths.
Triton maintainers resisted ad hoc compiler optimizations. Several Triton PRs around contiguous-load recovery, redundant-load narrowing, and layout propagation closed after maintainers argued that the passes were too narrow or belonged in user code or extensions Triton optimization debate.51 That stance fits a broader compiler trend: upstream projects want fewer clever rewrites unless they prove broad value.
Open WebUI closed subagent model selection on process and age. A proposal to let built-in subagents use a separate model attracted long-context planner and small execution model use cases, but the PR aged out with conflicts and checklist gaps Open WebUI subagent debate.46 The technical demand remains clear even though the contribution did not land.
TVM debated whether TIRx should split out. A formal vote proposed separating the TIRx foundation into its own repo, with captured responses in favor TVM TIRx vote.52 That governance move matters because compiler foundations now need faster iteration than mature runtime APIs sometimes allow.
Worth Watching
Decision endpoints may become the next compatibility layer. Ollama, LocalAI, llama.cpp, Modular MAX, RunanywhereAI, LiteLLM, and Dynamo all touched decision, SystemOne, or Decisions API concepts, so clients may soon need the same kind of compatibility matrix they already maintain for chat and embeddings RunanywhereAI.53
KV-cache correctness is now the main production fault line. vLLM, SGLang, Dynamo, TensorRT-LLM, Mooncake, Ray Serve, and Gaudi all worked on cache transfer, prefix reuse, resumed requests, or disaggregated serving bugs Mooncake.54
Embedding workloads are becoming first-class local workloads. Google AI Edge, Ollama, Rapid-MLX, MLX-VLM, and Hugging Face all moved embedding or multimodal embedding support forward LiteRT-LM.3
ROCm and AMD NPU paths deserve close testing. exllamav3, ROCm AITER, ATOM, AMDMIGraphX, AMD Ryzen AI Lemonade, FastFlowLM, and Qualcomm-like NPU tooling all show that non-CUDA inference stacks now move fast but still hit correctness and packaging edges exllamav3.14
Major Releases
Version numbers and release-note links live here as the canonical reference list.
vLLM Project shipped core vLLM v0.31.0 and vLLM-Omni v0.31.0rc1, with the main release centered on DeepSeek-V4.1-Flash, NVFP4 compressed KV cache defaults on SM100, FlashMLA, DeepGEMM sparse MQA logits, Mega-Gate, and decoder-boundary fusion. vLLM-Omni aligned to the same core runtime and focused on Ray multi-node inference, realtime and music generation, speech, multimodal performance, CUDA and ROCm image paths, and auxiliary outputs. vLLM release notes.2
SGLang shipped v0.5.21, citing 779 PRs from 227 contributors and adding model support for DeepSeek-V4.1 Flash, GigaChat 3.5, IQuest-Q1, and MiMo-V2.6 families. The release lands amid a broader push on HiCache, DeepSeek, GLM, AMD, diffusion, and omni-modal serving..9
ggml shipped llama.cpp v0.6.0 and whisper.cpp v1.9.5 while also cutting a large prerelease train. llama.cpp focused on API changes, GLM, Clef, SystemOne, Qwen MTP, model support, MoE caching, and backend updates; whisper.cpp synced ggml v0.26.0 and refreshed talk-llama integration. llama.cpp release notes.4
Google AI Edge shipped Gallery 1.0.20, LiteRT v2.3.0, LiteRT-LM v0.18.0, and MediaPipe v1.1.0. The dominant theme was on-device multimodal embeddings and runtime performance, with EmbeddingGemma 2, FlashAttention, Flash-Decode, GPU/NPU work, and MediaPipe build modernization. LiteRT-LM release notes.3
Ollama shipped v0.40.1, focused on cloud usage and balance proxying, Windows Clef stability, CLI onboarding cleanup, Windows manifest symlink avoidance, and README fixes. The most impactful change fixed a Windows Clef failure caused by large head reads..1
LocalAI shipped v4.11.0, centered on audio scenes, transcription, diarization, sound detection, ordered failover, SystemOne decision models, signed OCI galleries, Kimodo text-to-animation, and operations UX. The release reflects LocalAI’s move from local model server toward full local inference control plane..12
Hugging Face shipped Diffusers v0.41.0 and Transformers v5.19.0. Diffusers highlighted Qwen-Image 2.1 support, while Transformers highlighted EmbeddingGemma2 and the broader multimodal embedding path. Transformers release notes.39
Microsoft shipped foundry-local cli-preview-0.11.0, ONNX v1.23.2, and ONNX Runtime v1.28.3. The week focused on Foundry Local runtime safety, ONNX opset fixups, and ONNX Runtime memory-safety, model-validation, plugin execution, and packaging fixes. ONNX Runtime release notes.55
OpenVINO shipped OpenVINO 2026.4.1 and OpenVINO GenAI 2026.4.1.0, with NNCF moving its pins to match. The release train focused on LLM-friendly ONNX/frontend fixes, KV-cache correctness, GenAI model support, GPU/NPU backend work, and quantization edge cases. OpenVINO release notes.38
ROCm shipped AITER v0.1.24.post1 and AMDMIGraphX rocm-10.1. AITER released ROCm 7.0 wheels for gfx942 and gfx950, while AMDMIGraphX highlighted MoE fusion, RNN builder refactoring, code-review tooling, and dynamic slicing support. AMDMIGraphX release notes.56
FlashInfer shipped v0.7.1rc2, v0.7.1rc4, and v0.7.1rc5. The release candidates packaged the high-volume CAKE kernel work across MoE, Kimi-K3, DeepSeek sparse MLA, attention/decode kernels, and next-gen NVIDIA targets. Latest prerelease.5
FlashAttention shipped fa4-v4.0.0.beta34, with Windows CUDA 13 and PyTorch build fixes, apache-tvm-ffi support, SM100 S/P ping-pong decode, and SM100 hd256 forward unification. The team also prepared a FA2 security patch for scanner-blocking issues..40
BerriAI LiteLLM shipped a large release train from v1.100.5 through v1.106.0-dev.1, with available notes emphasizing Docker image signature verification and signing-key pinning. The code changes around those releases focused on enterprise proxy access control, MCP consistency, usage analytics, guardrails, and provider error behavior. Latest release.57
AMD Ryzen AI shipped Lemonade v2026.41.1 plus release candidates and merged a GAIA v0.25.0 release train through PRs. Lemonade focused on ROCm backend movement, streaming disconnect fixes, HRX catalog updates, and local backend robustness; GAIA focused on unified agent/chat UI and security controls. Lemonade release notes.58
Qualcomm shipped ai-hub-models v0.64.0 and GenieX v0.8.0. ai-hub-models made a breaking export CLI cutover and updated catalog support, while GenieX shipped a Microsoft-signed Hexagon HTP catalog and QAIRT-related updates. GenieX release notes.7
Ray shipped Ray 2.59.0, marking Ray Data LLM and Ray Serve LLM as GA or stable while upgrading vLLM integration and adding disk-backed Ray Data shuffle work. The surrounding code improved Serve routing, LLM callbacks, Data resumability, and core reliability..19
RunanywhereAI shipped Wally v0.8.0, v0.8.1, v0.8.2, SDK v0.20.38, and several C++ desktop kit releases. The dominant theme was decision support, including ABI v13, local decision primitives, hosted Eve, on-device checkpoints, and local HTTP serving. SDK release notes.8
Osaurus shipped 0.25.16 through 0.25.19. The releases added local JSON schema output, K2-Horizon, native MTP controls, Qwen-Image 2.1 generation and editing, MCP improvements, paired-phone management, privacy presets, and UI reliability fixes. Latest release.44
Rapid-MLX shipped v0.15.4 through v0.15.7 plus matching macOS app releases. The release train focused on accelerated Apple Silicon profiles, EmbeddingGemma embeddings, LTX video extension, agent setup, Desktop Share Compute, and server recovery. Latest release.15
Blaizzy shipped mlx-audio v0.5.8, mlx-audio-swift v0.1.4, mlx-audio-swift v0.1.5, mlx-vlm v0.7.5, and mlx-vlm v0.7.6. The releases focused on Voxtral/Qwen audio streaming, MLX Swift compatibility, Responses API compaction, oversized-request recovery, and VLM support. mlx-vlm release notes.27
FluidInference shipped FluidAudio v0.17.5, adding Phonon-2 as a Core ML ASR model option. The release also fits a broader week of ASR CLI, diarization, cache, and Core ML deployment work..45
try-mirai shipped uzu 0.6.0 and uzu 0.6.1. The release line landed alongside trellis/QTIP runtime work, a desktop GUI, Metal 4 compilation, Mac Catalyst support, and cleanup of older MoE/runtime paths. Latest release.59
turboderp-org shipped exllamav3 v1.5.4 and v1.6.0. The earlier release focused on token-embedding quantization, Transformers integration, faster loading, and lower memory overhead, while the later release added preliminary ROCm support, AVX2 CPU offload improvements, vision fixes, and ROCm wheels. Latest release.14
NVIDIA-adjacent and edge packaging projects had no TensorRT-LLM release, but TensorRT-Edge-LLM and TensorRT saw CI, Jetson, and toolchain work. The major NVIDIA release-like movement remained in TensorRT-LLM commits rather than tags. TensorRT-LLM commits.60
Zetic AI shipped ZeticMLangeiOS 1.11.0, updating Swift Package metadata to a new binary XCFramework artifact with checksum verification. The release was narrow but important for iOS SDK consumers..61