Ray Serve Ships KV-Aware LLM Routing

TL;DR

• KV routing moved up the stack: Ray Serve LLM, ai-dynamo, Mooncake, vLLM, and SGLang all pushed cache-aware routing, KV transfer, or offload work toward production serving.1
• Apple local inference hardened fast: Ollama, Apple MLX, Rapid-MLX, oMLX, Osaurus, and Uzu focused on MLX cache correctness, structured outputs, ANE execution, speculative decoding, and desktop app reliability.
• New model support stayed broad: Hugging Face Transformers, OpenVINO, LMDeploy, SGLang, Google LiteRT, and Qualcomm AI Hub moved support for Qwen4-Exp, GLM-5.3-Flash, Gemma 4, Kimi K2.6, Muse-Glimmer-30B, and Qwen3-ASR into inference paths.
• Kernel work chased new hardware: FlashInfer, TensorRT-LLM, CUTLASS, Triton, ROCm AITER, and OpenXLA all advanced Blackwell, ROCm, NVFP4, MXFP, sparse attention, and MoE execution.2
• Tool calls gained guardrails: Open WebUI added human approval for model tool use, while Ollama, LiteLLM, LocalAI, and Rapid-MLX tightened provider, budget, and structured-output behavior.3

This Week in Inference

Model support remained the public face of the week. Hugging Face Transformers added Qwen4-Exp support, then followed with GLM-5.3-Flash support.4 OpenVINO enabled Muse-Glimmer-30B and published a matching INT4 OpenVINO artifact.5 LMDeploy added Kimi K2.6 for text and multimodal serving.6 Argmax brought Qwen3-ASR 1.7B into its Apple-platform Playground.7 FluidInference pushed NemotronLabs VoiceChat-11B through Core ML conversion and Apple Silicon latency tests.8

The technique story was state management. vLLM pushed speculative decoding, fused FlashKDA, Kimi and DeepSeek performance, and decode-context parallel work into its main release.9 SGLang advanced speculative decoding, HiCache, prefill-decode paths, multimodal serving, and hardware portability.10 Ray framed KV-cache routing as a Serve LLM feature, with token-aware ingress, KV lifecycle broadcasts, and CPU offload awareness.1 ai-dynamo kept router architecture, KV-aware routing, and distributed engine integrations moving in the same direction.11

Hardware work split across datacenter and device, with little gap between them. NVIDIA CUTLASS, TensorRT-LLM, FlashInfer, and Triton all worked on Blackwell-era attention, MoE, FP4, FP8, and CuTe DSL paths.12 ROCm AITER, ATOM, AMDMIGraphX, and vLLM ROCm work pushed MI-series serving, gfx enablement, FP8 blockscale, sparse attention, and MoE kernels.13 Qualcomm GenieX shipped Microsoft-signed Hexagon HTP catalog support for Windows on Snapdragon, while Google LiteRT kept Gemma 4 export, NPU deployment, and mobile model tooling active.14

Product integrations mattered more than capital news in the tracked repos. Ollama added Claude Desktop gateway configuration and model recommendation flows.16 Open WebUI made human approval for tool calls a first-class user and admin control.3 LiteLLM focused on spend accounting, provider pricing, proxy reliability, and signed Docker images, which shows how inference gateways now compete on governance as much as model reach.17

Top Stories

vLLM and SGLang set the pace for stateful datacenter serving

vLLM shipped a large release centered on Kimi, DeepSeek, speculative decoding, fused FlashKDA kernels, decode-context parallelism, and kernel-level speedups.9 SGLang matched that tempo with speculative decoding, HiCache, prefill-decode work, hardware portability, and broad model support.10 The two projects now define the reference loop for open serving engines: add frontier model support, expose cache state, tune kernels, then repair the edge cases that appear across NVIDIA, AMD, Ascend, XPU, and MUSA.

KV cache work escaped the engine boundary

Ray Serve LLM made KV-cache and token-aware routing part of its Serve story, including in-process tokenization and out-of-band token transmission.1 ai-dynamo added more router logic, engine integration, and OpenAI-compatible classify and pooling endpoints.11 Mooncake kept transport, storage, shutdown, and RDMA paths moving.18 These projects are converging on the same serving shape: route requests by state, move KV cheaply, and avoid paying prefill twice.

Apple local inference became a serious serving target

Ollama added MLX structured outputs, Qwen3.8 Flash Next support, cache fixes, and Claude Desktop integration.19 Apple MLX kept quantized kernels, CUDA parity, array semantics, and attention controls moving.20 Rapid-MLX, oMLX, Osaurus, and Uzu pushed model lifecycle, ANE prefill, prefix caches, DFlash, MTP, and desktop orchestration.21 The Apple stack now looks less like hobbyist local inference and more like a parallel serving ecosystem with its own cache, tool, audio, and vision semantics.

Edge stacks turned model catalogs into deployable products

Google AI Edge shipped quantizer, PyTorch conversion, and visualization updates while LiteRT, LiteRT-LM, MediaPipe, XNNPACK, and samples moved in sync.22 Qualcomm AI Hub Models expanded its Snapdragon model catalog, and Qualcomm GenieX tied that catalog to signed HTP runtime support.14 ExecuTorch pushed wheel-consumable C++ SDK artifacts and mobile delegates.24 Mobile inference now depends on release engineering, manifests, model cards, signed runtimes, and conversion paths as much as raw kernels.

Kernel projects raced Blackwell, ROCm, and low-bit formats

FlashInfer worked across Blackwell CAKE and KDA attention, sparse attention, Mamba, MoE, routing, NVFP4, FP8, and W4A8.2 NVIDIA CUTLASS shipped CuTe DSL bugfixes while TensorRT-LLM advanced Blackwell top-K, MoE, VisualGen, and disaggregated serving.12 ROCm AITER fixed FP8 blockscale MoE correctness and expanded gfx support, while Triton pushed AMD profiling, GLUON, FP4, and memory-barrier correctness.13 The low-level race is now about correctness under new formats as much as peak throughput.

Deeper Dive

Everything below is for readers who want the full picture. Feel free to scroll.

Code Changes by Category

Cloud & Datacenter Serving

vLLM led with Kimi-K3, DeepSeek V4, Blackwell decode, ROCm FP8 and MLA work, DFlash2 speculative decoding, KV cache layout ownership, and RL/offload/disaggregated serving plumbing.9 SGLang advanced beam search, speculative decoding, HiCache, P-D infrastructure, hardware backends, Cosmos3, LongCat-Image, GLM-5.3-Flash, Qwen3.8-Flash-Next, and large CI coverage.10 TensorRT-LLM worked on disaggregated serving, KV-cache sizing, DeepSeek paths, Kimi behavior, Blackwell top-K, MiniMax-M3 decode, NVFP4 MoE, VisualGen, and parser reliability.26

Ray made Serve LLM more cache-aware with KV routing, token-aware ingress, Dynamo integration, TPU docs, and NIXL tensor transport.1 ai-dynamo concentrated on router architecture, approximate LRU routing, vLLM and SGLang integration, multimodal serving, FFmpeg video paths, OpenTelemetry, and supply-chain hardening.11 LightLLM pushed LightSpec dynamic speculative decoding, prefill-decode capacity limits, PD cancellation, NIXL recovery, process supervision, Anthropic token counting, and visual proxy semantics.27

Local LLM Runtimes

llama.cpp made the largest local-runtime move with DeepSeek 4 tensor split work, RPC and RDMA scaling, CUDA, Metal, Vulkan, WebGPU, SYCL, ROCm, OpenVINO, model conversion fixes, and semver release coordination.28 Ollama focused on Claude Desktop integration, MLX structured outputs, cache cancellation, Qwen3.8 Flash Next, llama.cpp updates, and slow-storage Metal timeout behavior.19 LocalAI shipped deny-by-default auth, chat context compression, distributed serving fixes, OCI resume downloads, Python backend boolean handling, and a large gallery refresh.29

exllamav3 added validated vision-tower quantization, vision offload from system memory, MoE CPU offload, tensor-parallel CPU cache support, a new quant-optimizer pipeline, and TP shutdown fixes.30 llamafile synced to upstream llama.cpp and validated Mac, Linux, Windows, CUDA, Vulkan, and Metal paths.31 vllm-mlx added DeepSeek-V4-Flash prompt, tool, reasoning, cache, shutdown, MTP, and structured-output fixes for Apple local serving.32

Apple Silicon & MLX Ecosystem

Apple MLX focused on quantized kernels, NAX, CUDA parity, MXFP8 graph reliability, convolution and normalization fixes, array API behavior, and attention controls.20 mlx-lm hardened generation, threading, prefill controls, Kimi and Llama caches, tool parsing, adapter validation, and model templates.33 mlx-swift-lm added variance-normalized KV cache, parallel safetensors loading, rerankers, shard discovery, LoRA and QLoRA fixes, model ports, and Xcode mitigations.34

Rapid-MLX made model replacement a server-owned transaction, tightened MLX residency, cache correctness, audio paths, vision routing, Desktop packaging, R2 mirrors, and release automation.21 oMLX pushed Qwen ANE prefill, fused offload, ANE compile caching, prefix-cache correctness, SSD write-through, MTP re-entry, and hardware-specific validation.35 Osaurus tied DFlash, MTP, SSD cache limits, prefix proof tests, local agents, and provider compatibility into a native macOS agent harness.36

Mobile & Edge Frameworks

Google LiteRT added SwiGLU, XNNPACK transformer exports, Gemma4 examples, weight externalization, IrModel compatibility, Objective-C and Metal fixes, LiteRT.js cleanup, and NPU-related deployment work.37 LiteRT-LM advanced constrained decoding, Gemma4 and Qwen3 templates, TTS and NPU componentization, embedding fixes, and platform polish.38 MediaPipe worked on Windows LiteRT-LM embedders, Java task lifecycle, iOS utilities, Apple and Metal Bazel cleanup, tokenizers, OpenCV, and task API sweeps.39

ExecuTorch split runtime libraries from the Python extension, improved wheel-consumable C++ artifacts, advanced MLX delegate work, removed a deprecated Apple MPS delegate, and kept Arm, Ethos-U, Qualcomm QNN, CUDA, ROCm, Vulkan, and XNNPACK paths active.24 Qualcomm AI Hub Models expanded its optimized Snapdragon catalog, while GenieX added signed HTP support, multi-HTP llama.cpp controls, Gemma4 integration, and QNN runtime selection work.14 Cactus fixed shared mmap duplication for multi-component models, which targets iOS-style memory mapping failures.40

Compilers, Runtimes & Graph Engines

OpenXLA improved GPU in-place output aliasing, Triton fusion rewriting, latency-hiding scheduling, autotuner cache insertion, cuDNN convolution fusions, CPU xtile compilation, PjRT sharding reads, and Bzlmod builds.41 JAX added mutual TLS for distributed initialization, PJRT output scheduling, ROCm and OneAPI backend updates, Mosaic and Pallas kernel work, FFT and eigengradient fixes, and documentation reorganization.42 Triton advanced AMD ROCProfiler PC sampling, GLUON scale handling, FP4 conversions, Rubin tuning, Membar correctness, ConSan scratch tracking, and frontend interpreter fixes.25

TVM kept pace with LLVM toolchains, fixed arena destructor leaks, improved WebGPU tensor-cache uploads, added PagedAttention KV-cache checkpoint primitives, and closed ONNX and Torch Relax frontend gaps.43 ONNX added local and sliding-window attention semantics, fixed TensorScatter circular behavior for KV-cache safety, and hardened low-precision tensor plumbing.44 ONNX Runtime expanded CUDA paged XQA, continuous-batching state ops, WebGPU PagedAttention, MLAS and KleidiAI work, and plugin EP allocator support.45

Models, Quantization & Optimization

Hugging Face Transformers added Qwen4-Exp and GLM-5.3-Flash support, DTensor-native tensor parallelism, pipeline-parallel inference, continuous-batching fixes, FP8 TP loading fixes, MXFP4 XPU behavior, and VLM conversion fixes.46 Diffusers shipped LTX2.5, MiniMax-H3, Wan Animate 2, stable Modular Diffusers, tensor-parallel support, ring context-parallel fixes, and Bria Fibo Edit batching.47 Optimum Intel added OpenVINO paths for Z-Image Turbo, Mistral VLMs, Muse-Glimmer, and active Gemma4 compression and video work.48

Google AI Edge Quantizer kept low-bit quantization and scale-recovery work moving.49 Intel Neural Compressor added agent benchmark workloads with vLLM setup scripts and continued MXFP W4A8 evaluation and AutoRound examples.50 OpenVINO NNCF opened grouped-MoE compression support for OpenVINO backends.51

Other Notable Changes

LiteLLM focused on provider pricing, spend enforcement, per-key callbacks, prompt management, Rust proxy paths, OS keychain login storage, MCP OAuth, and signed Docker images.17 Open WebUI added tool-call approval, large-install SQLite speedups, Anthropic passthrough fixes, streaming reliability, web-fetch hardening, RAG controls, and localization.3 BentoML added Claude Code deployment skills for Kubernetes and EC2 as BentoCloud winds down.52

Community Pulse

SGLang and vLLM had the loudest issue queues, with reports clustered around speculative decoding, cache behavior, Blackwell and ROCm paths, model bring-up, and scheduler behavior.53 Open WebUI drew heavy user traffic around reasoning streams, tool specs, OAuth, sidebar state, and OpenRouter behavior.55 Ollama saw community reports on context-limit behavior, cloud model thinking output, Ornith tool calls, Qwen3.8 ROCm output, and Windows desktop UI regressions.56

Apple local inference had the strongest grassroots validation loop. oMLX users tested Qwen ANE prefill, MTP, compile caching, and prefix caches across several Mac generations.57 Rapid-MLX users drove Desktop release blockers, dictation coexistence, disk streaming, cache behavior, and model replacement semantics.58 WhisperKit received a detailed large-v2 transcription-quality report, while sherpa-onnx showcased offline Android and Tauri dictation apps.59

Community Debates

The vLLM FlashInfer default flip showed how fragile accelerator defaults remain.61 Maintainers reverted the move to FlashInfer one-sided all-to-all after CI and Instinct-path breakage, even though the direction remains attractive for faster distributed communication. The next step points toward better gating and backend-specific validation before default changes.

Apple MLX maintainers codified AI contribution rules while closing several broad correctness and refactor PRs.62 The pushback focused on review load, automatic PR quality, correctness uncertainty, and lack of tight tests. The same pattern appeared downstream in mlx-lm, mlx-c, and Swift bindings, where maintainers favored narrow changes with clear evidence.

XNNPACK rejected hardware-dependent dot-product tiling despite benchmark gains on Apple M4.63 Maintainers objected that K tiling tied to cache behavior can change numerical results. That debate captures a recurring inference tradeoff: faster kernels need portability and stable numerics before they can become defaults.

CUTLASS maintainers pushed dynamic-region fixes toward FrontendNext and closed pycute fixes against the vendored copy.64 The team told users to target newer CuTe surfaces instead of patching older bundled paths. That sets a clear maintenance boundary as CuTe DSL grows into more inference stacks.

Open WebUI rejected a native Android app proposal procedurally because it targeted the wrong branch, lacked the expected issue or discussion, and had too much scope.65 The same repo also closed enterprise auth and protected-model proposals because maintainers wanted explicit schemas before merging access-control behavior. The project is growing fast enough that process now shapes product velocity.

Worth Watching

Cache-aware serving will keep spreading. Mooncake, Ray, ai-dynamo, LMDeploy, and SGLang all have open work around routing, cache visibility, offload, distributed KV stores, or observability.69

Realtime voice will stress serving APIs. vllm-omni, sglang-omni, FluidInference mobius, and Argmax are all working through streaming, duplex, transcription, TTS, latency, and commitment-policy questions.7

Blackwell and ROCm low-bit paths still need correctness time. FlashInfer, TensorRT, ROCm AITER, and Triton all surfaced accuracy, replay, sparse-attention, or performance regressions around new kernels.74

Model catalogs are becoming runtime contracts. Qualcomm AI Hub Models, RunanywhereAI, Google LiteRT, and LocalAI now treat model metadata, manifests, precision labels, provider mappings, and download behavior as part of inference infrastructure.23

Major Releases

ai-dynamo shipped v1.4.1 as a patch focused on production serving compatibility. The release added OpenAI-compatible classify and pooling endpoints to Dynamo Frontend with vLLM worker support, plus fixes for router overload recovery, logprob token ID passthrough, and vLLM-Omni NIXL object transfer..11

AMD Ryzen AI shipped Lemonade v11.8.0, centered on AMD local-AI runtime support, model catalog updates, cloud provider passthrough, and backend hardening. The most visible change was experimental ds4 support for DeepSeek V4 Flash on AMD Strix Halo and ROCm..80

Apple shipped coreai-torch v0.4.2 and MLX v0.32.2. coreai-torch focused on PyTorch converter coverage, while MLX focused on attention controls, array semantics, bool casts, divmod behavior, and general runtime fixes. MLX release notes.20

BerriAI shipped a fast LiteLLM release train from v1.98.0 through v1.100.0-dev.1, with the public notes centered on Docker image signature verification. The code work around the releases focused on pricing, budget enforcement, prompt management, proxy reliability, and provider coverage. Stable release notes.17

Blaizzy shipped mlx-vlm v0.6.16 and v0.6.17. The releases focused on Qwen3.8, GLM-5.3-Flash, Qwen3.8-Flash-Next, GLiNER2.5, MiniMax-M3 leak fixes, KV-cache quantization, and MLX compatibility. Latest release notes.81

Dao-AILab shipped fa4-v4.0.0.beta28 for FlashAttention as a prerelease. The release relaxed a CuTe/CUTLASS DSL requirement, while open work continued around SM100 decode, ROCm build controls, and paged-varlen correctness..82

ggml shipped llama.cpp v0.2.0, llama.cpp v0.3.0, many nightly builds, and whisper.cpp v1.9.3 plus nightly artifacts. The dominant theme was semver coordination, ggml 0.22.0 sync, backend stabilization, DeepSeek 4 tensor split, and whisper.cpp Ryzen AI NPU encoder offload. llama.cpp release notes.28

Google AI Edge shipped AI Edge Quantizer v0.9.0, litert-torch v0.9.4, and Model Explorer v0.1.33. The releases focused on low-bit quantization workflows, PyTorch-to-LiteRT export, Gemma 4 architecture support, ASR export, dynamic context length, and LiteRT-LM visualization. Model Explorer release notes.22

Hugging Face shipped diffusers v0.40.0, transformers v5.16.0, transformers v5.16.1, and optimum-intel v2.1.0-muse-glimmer. The releases focused on LTX2.5, MiniMax-H3, Wan Animate 2, Modular Diffusers, Qwen4-Exp, GLM-5.3-Flash, and Muse-Glimmer OpenVINO support. Diffusers release notes.47

k2-fsa shipped the sherpa-onnx tauri release. The release provided a minimal Tauri app that displays sherpa-onnx version information across Linux, macOS, Windows, Android, and iOS while Rust and Windows ARM64 packaging work continued..83

kvcache-ai shipped Mooncake v0.3.13. The release focused on TENT io_uring stale-status work, manual CI cancellation workflows, an Ascend transport request-map leak fix, and a broad set of assets for transport and storage paths..18

Microsoft shipped ONNX Runtime WebGPU Plugin EP v0.3.0. The release expanded model and data-type coverage, improved generative-model performance, and added initial PagedAttention plus several operator implementations for browser and device inference..45

LocalAI shipped v4.9.0. The release emphasized deny-by-default HTTP authentication, chat context compression, consolidated model and backend pages, MiniMax-H3 video generation, Qwen3-TTS on llama.cpp, and KNN routing..29

NVIDIA shipped CUTLASS v4.6.3 and v4.7.1. Both releases focused on CuTe DSL bugfixes, setmaxnreg, JIT and decorator leaks, JAX cutlass_call behavior, export-to-C i64 shapes, TVM-FFI fixes, and validation against major kernel consumers. CUTLASS v4.7.1 release notes.12

Ollama shipped v0.33.0 and v0.33.1. The releases focused on Claude Desktop gateway configuration, cancelled-prefill cache correctness, MLX structured outputs, MLX Qwen3.8 Flash Next support, llama.cpp updates, and Metal timeout fixes. Latest release notes.19

Open WebUI shipped v0.11.1. The release added human-in-the-loop approval for saved-conversation tool calls, giving users and admins approve or deny controls before model tools execute..3

OpenVINO shipped OpenVINO 2026.3.1. The release highlighted Muse-Glimmer-30B functional enablement, an INT4 OpenVINO artifact, and related Muse-Glimmer and VLM chatbot notebooks..5

Qualcomm shipped ai-hub-models v0.61.0 and GenieX v0.5.0. AI Hub Models expanded the optimized Snapdragon catalog with WeDetect, SigLIP2, RepViT, FunASR-Conformer-EN, OWL-V2, MobileFaceNet, and OSNet, while GenieX added Microsoft-signed Hexagon HTP catalog support for Windows on Snapdragon. GenieX release notes.14

Rapid-MLX shipped v0.12.18, rapid-mac-v0.12.18, v0.13.0-rc1, rapid-mac-v0.13.0-rc2, v0.13.0, and rapid-mac-v0.13.0. The release train focused on Mac setup, same-process chat and speech, long-prompt responsiveness, Qwen3-Next disk streaming, Nemotron Diffusion 3B serving, Desktop reliability, and release automation. Stable release notes.21

Ray shipped ray-2.58.0. The release centered on Ray Serve LLM KV-cache and token-aware routing, in-process tokenization on the LLMRouter ingress replica, out-of-band token transmission, KV lifecycle broadcasts, and CPU KV cache/offload awareness..1

ROCm shipped AITER v0.1.20.dev0 as a development wheel for gfx1250 on ROCm 7.14. The release pinned to an earlier commit because later changes hit a FlyDSL lowering crash on DeepSeek-V4 gfx1250 runs, while the week’s code work focused on FP8 blockscale MoE correctness, OPUS, attention, and gfx enablement..13

SGLang shipped sglang v0.5.18 and sglang-omni v0.1.3. The core release highlighted a large contributor cycle and new model support, while omni focused on Qwen3-Omni outbox-drain performance, Whisper service paths, Fun-ASR CUDA Graph behavior, and async decode defaults. SGLang release notes.10

RunanywhereAI shipped runanywhere-sdks v0.20.25 through v0.20.29 and RCLI v0.4.0 through v0.5.1. The release train focused on Electron packaging, pinned NeuRT and QHexRT artifacts, external C++ desktop kits, checksum and provenance validation, Windows ARM64 delivery, and a kit-backed standalone CLI. Latest SDK release notes.78

turboderp-org shipped exllamav3 v1.4.3 and v1.4.4. The releases focused on GLM support, CPU expert offload, dynamic draft sizing, quant-optimizer tooling, MTP in expert-parallel mode, validated vision quantization, and vision offload from system memory. Latest release notes.30

try-mirai shipped Uzu 0.5.19, 0.5.20, 0.5.21, and 0.5.22. The tags carried a fast Metal runtime train around attention hotfixes, quantized matvec, OpenAI-compatible tool behavior, thinking support, non-causal KV cache, and runtime reorganization. Latest release notes.84

Osaurus shipped 0.23.1. The release tied together cache, prefix, and reasoning proof work, including injected-clock coarsening, timestamp-related cross-chat reuse loss, disk-cap reporting, reasoning flip costs, Step-3.7 correction, and restart-speed claim validation..36