← All issues
2026-W37·Sep 10 — Sep 16, 2026·17 min read

vLLM Kicks Off The KV Cache Wars

Cloud servers, Mac runtimes, and edge frameworks all spent the week chasing the same bottleneck: memory movement. KV cache routing, low-bit kernels, and model-specific serving paths linked the whole stack from Blackwell GPUs to Apple Silicon and Android NPUs.

Cover for 2026-W37
5,936 commits
1,970 issues
4,662 PRs
144 releases
Weekly activity chart

TL;DR

  • KV cache work crossed the stack: vLLM pushed sparse attention, NIXL, Mooncake, and offload fixes as cache movement became the main serving constraint.1
  • Apple Silicon moved from experiment to product path: Ollama made MLX safetensors creation a first-class route and hardened memory handling on Macs.2
  • Edge inference got a real SDK week: ExecuTorch shipped C++ packaging, delegate libraries, LLM export paths, and ROCm-backed AOT work for mobile and embedded inference.3
  • Kernel libraries raced toward new accelerators: FlashInfer expanded Blackwell, Rubin, MoE, cuDNN, and distributed serving kernels.4
  • Local runtimes kept broadening model and backend coverage: llama.cpp synced ggml, added new model paths, and pushed CUDA, HIP, Vulkan, Metal, SYCL, OpenCL, OpenVINO, and WebGPU fixes.5

This Week in Inference

External market searches did not confirm new model, hardware, funding, acquisition, or partnership announcements for the window, so the week’s signal came from code rather than press releases. GitHub activity showed a clear pattern: projects spent less time chasing fresh model drops and more time making recent model families serve well under real memory, cache, and hardware constraints, with vLLM working on DeepSeek Flash paths, sparse attention, speculative decoding, and NIXL transport.1 The same pressure showed up in SGLang, where router, prefill/decode, KV transfer, DeepSeek, and omni-modal audio work all moved in the same week.6

Model enablement still mattered, but it looked like runtime plumbing rather than a launch cycle. Hugging Face expanded Transformers and Optimum Intel coverage for Qwen, Gemma, DeepSeek, HyperCLOVAX, LTX, and OpenVINO export paths through Transformers and related repos.7 Local and Apple stacks followed the same curve: Ollama promoted MLX safetensors creation, MLX-LM added and fixed model loaders and tool-call parsing, and oMLX pushed large-model serving, SSD offload, MTP, and Qwen acceleration on Macs.2

Hardware news came through code paths. NVIDIA-side projects pushed Blackwell, Rubin, MoE, FP8, cuDNN attention, and disaggregated serving through TensorRT-LLM and FlashInfer.10 AMD-side work moved through ROCm AITER, ATOM, MIGraphX, and vLLM ROCm fixes, while edge stacks advanced through LiteRT, ExecuTorch, ncnn, and Qualcomm AI Hub tooling.3 The industry move to track is still the same one from the briefing: inference advantage now comes from the full path, model format, quantization, cache policy, scheduler, kernel library, interconnect, and deployment target.

Top Stories

vLLM turns cache movement into the core serving problem

vLLM pushed DeepSeek Flash serving, sparse MLA, speculative decoding, NVFP4 compressed KV cache, and MegaMoE work in the same week, which makes vLLM look less like an API server and more like a distributed memory system.1 The project also tightened NIXL, Mooncake, offload accounting, KV-transfer metrics, and prefix-hit behavior, which shows where large-model serving breaks first under load. vLLM-Omni extended the same pattern into audio, video, diffusion, and realtime serving, so the cache problem now spans text and multimodal inference.

Ollama makes MLX a first-class local runtime path

Ollama moved safetensors import into its MLX create flow, reorganized MLX packages out of experimental space, and removed older conversion paths that treated MLX as a side route in Ollama.2 That matters because Apple Silicon users now get a clearer path from model weights to resident local serving without detouring through GGUF conversion for every workflow. The team also tightened MLX memory bounds, runner replacement, cache eviction, and quantized matmul dependencies, which points to Macs as serious local serving hosts rather than demo machines.

ExecuTorch ships the mobile and embedded release Meta needed

Meta shipped ExecuTorch with C++ SDK packaging, delegate libraries, LLM serving and export support, GGUF packed constants, prefix-cache session cloning, and ROCm support in AOTI CUDA through ExecuTorch.3 The release gives mobile and embedded teams a cleaner package boundary across CUDA, Core ML, MLX, OpenVINO, Qualcomm, XNNPACK, Vulkan, and Arm paths. PyTorch core also kept feeding that release path with AOT, guard, dynamic-shape, packaging, and backend fixes.

FlashInfer and FlashAttention push the kernel layer toward Blackwell and Rubin

FlashInfer added Blackwell, Rubin, SM10x, cuDNN decode/prefill, MoE, FP8, distributed communication, and paged-attention work across FlashInfer.4 Dao-AILab’s FlashAttention work focused on FA4 sparse MLA, attention sinks, HD256 inference readiness, and Hopper backward regressions in FlashAttention.14 Together they show a kernel ecosystem that now has to track serving engines, model variants, and accelerator SKUs at the same time.

ONNX Runtime and OpenVINO keep pulling GenAI toward edge deployment

Microsoft shipped ONNX Runtime releases with generative AI, CUDA/GQA, INT4 paged KV cache, WebGPU, Go bindings, and speculative decoding work through ONNX Runtime.15 OpenVINO shipped a GenAI-heavy release with Gemma, Kokoro, Qwen, DeepSeek OCR, Granite, FLUX, and image-generation model coverage across CPU, GPU, and NPU in OpenVINO.16 The shared theme is clear: edge inference now needs server-style features, including paged KV, quantized attention, graph surgery, model conversion, and browser or mobile bindings.

Deeper Dive

Everything below is for readers who want the full picture. Feel free to scroll.

Code Changes by Category

Cloud & Datacenter Serving

vLLM had the densest serving week, with DeepSeek Flash support, sparse attention, speculative decoding, KV offload, NIXL, Mooncake, Responses API, reasoning parser, CPU, ROCm, XPU, Gaudi, and omni-modal work spread across core vLLM, vllm-gaudi, and vllm-omni.17 SGLang matched that with DeepSeek kernel work, router hardening, PD/KV reliability, heterogeneous hardware support, Qwen3-TTS, Fun-CosyVoice, and omni-modal scheduling in sglang and sglang-omni.20

NVIDIA pushed TensorRT-LLM through runtime profiling, KV-cache event streaming, NIXL bounce buffers, KVCacheManager defaults, cuDNN attention, DeepGEMM FP8 block scales, DeepEP, CuTeDSL MoE, MiniMax, and speculative decoding cleanup in TensorRT-LLM.22 ai-dynamo hardened distributed inference control planes with GMS startup fixes, KV relay source modes, SGLang multimodal paths, vLLM worker behavior, KServe/Triton tensor support, and Kubernetes operator fixes in Dynamo.23 LMDeploy focused on throughput, TurboMind GEMM routing, BF16 LM-head all-gather, DFlash speculative decoding, structured-output parsers, DistServe fixes, and checkpoint weight updates in LMDeploy.24

Ray Serve and Ray Data improved model multiplexing, routing, shutdown behavior, object-location publishing, metrics, autoscaling, and data execution paths that matter for inference pipelines in Ray.25 LiteLLM had a security and platform week with password reset enforcement, provider registry updates, billing fixes, Azure passthrough, guardrails, MCP OAuth, and router UX in LiteLLM.26

Local LLM Runtimes

llama.cpp synced ggml, added Maple, Nemotron, Mimo, HrmText, Qwen, Gemma, speculative decoding, MTP CUDA Graph reuse, unified-KV sizing, OpenCL expert matmul tuning, and broad backend fixes in llama.cpp.27 Ollama promoted MLX safetensors creation, reorganized MLX packages, tightened Apple Silicon memory handling, bumped llama.cpp, removed the built-in CLI agent, and refined capability reporting in Ollama.28 LocalAI refreshed a large set of backend engines, added private registry and download credentials, improved Responses WebSocket behavior, exposed model context size, and updated gallery entries in LocalAI.29

exllamav3 shipped MoE, CPU offload, quantization, deterministic execution, Spark, GLM, Gemma, DFlash, and Nemotron support in exllamav3.30 llamafile added CUDA/HIP graphs, release-style ggml builds, Windows GPU timer fixes, API docs, and Vulkan toolchain diagnostics in llamafile.31 WebLLM added OPFS-backed crash-resumable generation and fixed streaming cancellation model-lock leaks in web-llm.32

Apple Silicon & MLX Ecosystem

Apple’s Core AI and MLX work moved across model export, Core ML conversion, quantization, guided generation, queueing, Qwen presets, SAM video export, PyTorch SDPA GQA conversion, and activation quantization in coreai-models, coremltools, and coreai-optimization.33 MLX added Gated Delta Net kernels, quantized matmul fixes, distributed all-gather correctness, global-scale qmm paths, CUDA and Windows memory fixes, and API cleanup in MLX.36 MLX-LM added OLMo Hybrid, Spark, Ministral, Laguna, DeepSeek, Qwen, parser, serialization, quantization, cache, and tokenizer fixes in mlx-lm.37

The wider MLX app layer was busy. oMLX pushed MTP, Qwen acceleration, SSD expert offload, K2 Horizon, GLM, Spark, cluster recovery, and structured-output fixes in oMLX.38 Rapid-MLX shipped Desktop Agent Mode, DeepSeek and K2 Horizon qualification, Qwen and GLM native MTP, serving fixes, and release automation in Rapid-MLX.39 Osaurus shipped repeated macOS app updates around repair, model integrity, RAM admission, MTP policy, image chat, n8n, orchestration, and prompt-cache telemetry in Osaurus.40

Mobile & Edge Frameworks

ExecuTorch shipped C++ SDK packaging, delegate libraries, LLM workflows, ROCm-backed AOTI CUDA, GGUF constants, prefix-cache cloning, Arm, Cortex-M, Vulkan, XNNPACK, QNN, and portable-runtime fixes in ExecuTorch.41 Google’s edge stack advanced LiteRT, LiteRT-LM, litert-torch, Gallery, MediaPipe, samples, XNNPACK, DDP benchmarking, Embedding Gemma, MiniCPM, Kokoro, WebGPU, and Apple packaging in LiteRT-LM.42 Qualcomm added AI Hub model recipes, QAIRT defaults, GenieX runtime controls, VLM overrides, raw base64 audio, benchmark fixes, and sample app releases in GenieX and ai-hub-models.44

ncnn hardened parameter parsing, graph loading, layer validation, Vulkan shader behavior, descriptor-pool construction, SPIR-V diagnostics, and Windows Arm coverage in ncnn.45 sherpa-onnx added streaming NeMo contextual biasing, speaker embedding readback, diarization confidence, Kotlin diacritization, Termux Rust CI, and packaging fixes in sherpa-onnx.46 FluidAudio and mobius moved Chatterbox CoreML export into Swift runtime support, Kokoro Japanese frontends, ASR seam fixes, deterministic scoring, and model-download repairs in FluidAudio.47

Compilers, Runtimes & Graph Engines

TVM and tvm-ffi aligned around structural traversal, mutation semantics, Rust parity, BigInt, TIRx cleanup, Relax/TIR fixes, CUDA scheduling, ONNX and PyTorch importer fixes, TensorRT engine embedding, and cuDNN attention partition safety in TVM and tvm-ffi.49 JAX and OpenXLA pushed distributed host transfer, Pallas and Mosaic GPU semantics, numerics, command-buffer readiness, SelectK CUDA Graph support, IFRT, Shardy, CPU and GPU autotuning, ROCm, oneAPI, and HLO fixes in JAX and XLA.50

Triton modernized AMD LLVM packaging, NVIDIA lowering, 64-bit reductions, CGA layouts, FP8, Gluon, and CI in Triton.52 ONNX Runtime shipped CUDA/GQA, INT4 paged KV cache, WebGPU kernels, Qwen contrib ops, Core ML updates, Swift bool tensors, Go bindings, and graph fixes in ONNX Runtime.53 OpenVINO shipped GenAI model coverage, GPU/NPU/CPU hardening, NNCF alignment, GenAI benchmarks, GGUF runtime extension support, and Node updates in OpenVINO.54

Models, Quantization & Optimization

Hugging Face Transformers added distributed FSDP and expert parallel work, GGUF support, generation/cache fixes, HyperCLOVAX Vision, DINO modernization, MUSA Flash Attention, XPU fixes, and many multimodal model corrections in Transformers.55 Diffusers added Cosmos ModelOpt FP8 mixed denoising, TorchAO fixes, LTX modular pipeline fixes, security workflow hardening, and pipeline cleanup in Diffusers.56 Optimum Intel added Qwen TTS OpenVINO export/runtime, LTX export, MiniCPM INT4 defaults, and quantization fallback cleanup in Optimum Intel.57

Intel Neural Compressor fixed unsafe Keras deserialization, added vLLM QDQ NVFP4 dense and MoE support, improved trace logging, expanded AutoRound examples, and added MXFP8 diffusion workflows in Neural Compressor.58 ktransformers shipped Kimi and Qwen LoRA releases, native RAWINT4 routed-expert LoRA SFT, immutable wheel gates, model acceptance CI, MXFP4 fallback fixes, and pre-permuted activation speedups in ktransformers.59 Dao-AILab FlashAttention advanced FA4 sparse MLA, attention sinks, HD256 paged-KV support, Hopper backward fixes, and PyTorch/CUTLASS compatibility in FlashAttention.60

Other Notable Changes

Open WebUI fixed safe link rendering, chat fork folder permissions, async sign-in rate limiting, tool-result image extraction, editor spellcheck behavior, and localization throughput in Open WebUI.61 Wally shipped a fast release train for coding-agent routing, Claude Code and OpenCode integration, Qwen message ordering, console rate-limit behavior, installer fixes, and a Go rewrite in Wally.62 BentoML saw no merged code, but community PRs targeted save_config crashes, bare iterator annotations, and human-presence middleware in BentoML.63

Community Pulse

Project activity stayed high in the core serving layer, with vLLM, SGLang, LiteLLM, TensorRT-LLM, and ai-dynamo all showing heavy PR and issue flow around production correctness rather than headline features. vLLM’s issue traffic focused on tracing, batch invariance, ModernBERT LoRA, GLM tool calls, DeepSeek ROCm, XPU offload, and DSpark warmup in vLLM issues.64 SGLang users raised HiCache, hybrid-cache, OpenAI API gaps, dLLM serving, reasoning flags, and GLM disaggregation failures in SGLang issues.65

Security reports cut across the stack. LightLLM users flagged unauthenticated pickle deserialization and unauthenticated node registration in LightLLM.66 Predibase LoRAX received a private-disclosure request around path traversal and SSRF risks in adapter isolation in LoRAX.67 Open WebUI, LocalAI, Intel Neural Compressor, ncnn, and TFLite Micro all saw safety or validation work tied to user-controlled inputs, unsafe deserialization, malformed models, or untrusted links.

Apple Silicon community pressure was visible in release requests, benchmarks, and dependency concerns. MLX-LM users asked for a release after heavy commits in mlx-lm discussions.68 MLX-Swift-LM users reported security-policy problems caused by unreleased dependency pins in mlx-swift-lm.69 oMLX, Rapid-MLX, Osaurus, Ollama, and exo all had users testing large models, distributed Macs, SSD offload, and model residency under real workloads.

Community Debates

vLLM maintainers rejected deterministic prefix caching because the project does not promise batching-invariant determinism and the scheduler cost looked too high for the expected gain in vLLM #34046.70 That debate matters because more users now expect reproducible outputs from systems that use continuous batching, async scheduling, prefix reuse, and speculative decoding. The project pointed work toward safer cache productization and validation rather than a determinism guarantee.

Ollama merged removal of the built-in CLI agent, then users asked for an opt-in return path for locked-down enterprise environments in Ollama #18490.71 The disagreement was product-surface, not kernel-level engineering. It shows the tension between a clean local model runner and the agent workflows users now expect from that runner.

TensorFlow maintainers closed a proposed Apple Silicon Metal PluggableDevice after pushing back on vendoring a large out-of-tree plugin into TensorFlow core in TensorFlow #126384.72 The discussion pointed toward third-party discoverability instead of core inclusion. That is the same boundary many inference projects now face as accelerator-specific code grows.

Triton Inference Server opened a monorepo proposal to combine server, backend, common, core, and python backend repos in Triton #8977.73 Reviewers cautioned that migration tables and history guarantees need to be exact before approval. The proposal shows how serving infrastructure projects are hitting coordination costs across too many repos.

OpenVINO GenAI maintainers rejected a benchmark fallback change because benchmarks should fail when a backend path is unavailable rather than fall through to a different path in OpenVINO GenAI #4460.74 That stance favors measurement integrity over smoother user experience. Expect more of this as PA, SDPA, NPU, GPU, and CPU paths share a benchmark surface.

Worth Watching

  • NIXL and Mooncake are becoming shared infrastructure: vLLM, SGLang, ai-dynamo, TensorRT-LLM, and kvcache-ai all worked around KV transport, disaggregated serving, relay, offload, or cache recovery in Mooncake.75
  • Apple Silicon serving is moving past hobby use: Ollama, MLX, oMLX, Rapid-MLX, Osaurus, ds4, Uzu, and exo all pushed Mac-side serving, offload, MTP, or distributed MLX work in MLX.76
  • Blackwell and Rubin support will stay noisy: FlashInfer, FlashAttention, TensorRT-LLM, CUTLASS, Triton, ONNX Runtime, and XLA all changed code for new NVIDIA paths in FlashInfer.4
  • ROCm inference is now a full-stack effort: AITER, ATOM, MIGraphX, vLLM, SGLang, llama.cpp, and Triton all moved AMD paths that span kernels, compilers, and serving logic in AITER.11
  • Structured output has become an inference feature, not an app feature: LMDeploy, LiteLLM, Ollama, Rapid-MLX, Open WebUI, vLLM, SGLang, Foundry Local, and Uzu all touched tool-call, reasoning, grammar, JSON, or Responses behavior in LMDeploy.77

Major Releases

ai-dynamo shipped three experimental Dynamo snapshots for Solar Open2, A.X-K2, and DeepSeek-V4.1-Flash, with the main theme being B200 model snapshots, vLLM/SGLang backend validation, and non-QA-gated early support. The most visible release targeted Solar Open2 250B on B200 through Dynamo vLLM..78

AMD Ryzen AI shipped GAIA v0.24.1 and Lemonade candidate-v2026.39.1 while FastFlowLM advanced in-tree build metadata. The releases focused on agent UX cleanup, /images/edit, allowed-origin policy, VRAM auto-eviction, and release-chain repair. GAIA release notes.79

Apache shipped tvm-ffi v0.1.14 and v0.1.14-post0 while TVM consumed the structural API work. The release theme was structural map and mutation APIs, Rust parity, stronger typing, and traversal performance for the larger TVM IR refactor..49

BerriAI shipped a dense LiteLLM release train from v1.100.1 through v1.103.0-dev.1. The dominant theme was signed Docker image verification with cosign, while code work also focused on auth hardening, routing UX, provider registry correctness, Azure passthrough, guardrails, and billing. Stable release notes.80

Blaizzy shipped mlx-audio v0.5.4 and mlx-vlm v0.7.1. The releases focused on MLX media models, with rumik-oss-1 TTS, Nemotron live input, VibeVoice-ASR dtype work, DINOv2, SAM video tracking, Qwen MXFP4 acceleration, and Spark-X2.5 support. mlx-vlm release notes.81

Dao-AILab shipped fa4-v4.0.0.beta31 for FlashAttention. The prerelease focused on SM100 Flash MLA attention sink, HD256 decoder paged-KV support, non-causal hdim128 spill fixes, and output-alignment work..60

DeepSpeedAI shipped DeepSpeed v0.19.7. The release focused on ZeRO checkpoint export dtype control, norm-combination fixes, Ulysses all-to-all layout repair, and flops-profiler aggregation fixes..82

FlashInfer shipped v0.7.0rc3. The prerelease centered on release-trace cleanup and the broader branch carried Blackwell, Rubin, cuDNN, MoE, FP8, distributed, and paged-attention work..83

Fluid Inference shipped FluidAudio v0.15.7. The release focused on Nemotron decode-time custom vocabulary biasing, multilingual blank rescue, direct NeMo ITN linking, and Kokoro-ANE fixes, while the week’s code pushed Chatterbox CoreML into Swift runtime paths..47

ggml shipped llama.cpp v0.4.1, whisper.cpp v1.9.4, and a large nightly/prerelease train. The releases focused on ggml v0.24.0 sync, model coverage, backend fixes, Windows-on-Arm artifacts, and API or server changes. llama.cpp release notes.27

Google AI Edge shipped LiteRT-CLI v0.2.0 and LiteRT-LM v0.17.1. The releases focused on benchmark tooling, ADC login, Android file push, QNN extraction, and a LiteRT-LM tool-call integer argument fix. LiteRT-LM release notes.42

k2-fsa shipped sherpa-onnx v1.13.8. The release focused on publishing fixes, FunASR-Nano and Qwen3-ASR recognizer data-race fixes, Termux support, diarization confidence, Windows ARM64 Go APIs, OpenVINO Intel NPU acceleration, Kokoro work, ONNX Runtime 1.28.2, and a large asset set..46

kvcache-ai shipped ktransformers v0.7.1 and v0.7.0.post4. The releases focused on Qwen VLM LoRA, Kimi K2.5 and K2.6 LoRA fine-tuning, RAWINT4 routed experts, SGLang adapter loading, checkpoint resume, and PyPI packaging gates. Latest release notes.59

Meta shipped ExecuTorch v1.5.0. The release focused on C++ SDK packaging, delegate libraries for CUDA, Core ML, MLX, OpenVINO, and Qualcomm, expanded LLM serving and export workflows, ROCm support in AOTI CUDA, and packaged TorchAO kernels..41

Microsoft shipped ONNX Runtime v1.30.0 and v1.29.1. The releases focused on generative AI inference, CPU and GPU performance, Go bindings, variable-length causal convolution, speculative decoding in paged XQA, INT4 paged KV caches, and GQA layout fixes. ONNX Runtime release notes.53

Mozilla AI shipped llamafile v0.10.6. The release focused on a llama.cpp bump, zip safety fixes, build-link recipes, -DNDEBUG ggml compilation, CUDA/HIP graphs, MSVC timer initialization, and Vulkan diagnostics..31

Ollama shipped v0.34.1 and v0.34.2-rc1. The release line focused on non-experimental MLX safetensors ollama create, improved MLX memory handling, faster tags, repeat-token limits, capability reporting, typical_p deprecation, and llama.cpp updates..28

OpenVINO Toolkit shipped OpenVINO 2026.4.0. The release focused on GenAI and model support across CPU, GPU, and NPU, including Gemma, Kokoro, Qwen, DeepSeek OCR, Granite, FLUX, and early image-generation model coverage..54

Osaurus AI shipped five releases from 0.25.1 to 0.25.5. The release train focused on model repair and digest verification, RAM admission, MTP policy, prompt-cache efficiency, n8n integration, orchestration, image chat, multi-device identity, and macOS app polish. Latest release notes.40

Qualcomm shipped ai-hub-apps v0.37.1, ai-hub-apps v0.37.2, ai-hub-models v0.62.1, and ai-hub-models v0.62.2. The releases focused on Face Recognition samples, Ubuntu Python app launch flow, LLM/VLM quantization fixes, YOLOv5-Face, Bevformer, Intern3.5-VL-2B, and demo tooling. ai-hub-apps release notes.84

Rapid-MLX shipped v0.14.0, v0.14.1, v0.14.2, and matching rapid-mac releases. The release train focused on faster Apple Silicon inference, Desktop Agent Mode, K2 Horizon, DeepSeek V4.1 qualification, Qwen and GLM MTP, long-document reliability, multimodal chat, and serving fixes. Latest release notes.39

ROCm shipped AITER v0.1.22. The release focused on manylinux wheels for gfx942 and gfx950 across ROCm and Python combinations, while the week’s code pushed attention, MoE, FP8, DSV4, autotuning, and serving integration..85

RunanywhereAI shipped Wally v0.5.5, v0.5.6, and v0.5.7. The release train focused on Claude Code and OpenCode harnessing, Qwen system-message folding, installer fixes, GLM model routing, rate-limit resilience, release fixes, and model-labeling corrections. Latest release notes.62

SGLang shipped sglang-omni v0.1.5 while core sglang did not cut a release. The release focused on README news ordering, quieter Hugging Face downloads, MOSS-TTS speech input limits, and Qwen3-TTS breakable Prefill CUDA Graph support..86

try-mirai shipped Uzu 0.5.27. The release notes were sparse, but the release bump followed DFlash2 runtime support, OpenAI-compatible consecutive assistant messages, request logging, streaming use-after-free fixes, test harness changes, and dependency updates..87

turboderp-org shipped exllamav3 v1.4.9 and v1.5.0. The releases focused on Spark2.5, GLM4.7-Flash, DeepSeek vision, autosplit accounting, AVX512BW CPU offload, zero-copy pinned arena CPU offload, MoE prefill and decode, quantization speed, and nondeterminism fixes. Latest release notes.30

vLLM Project shipped vllm-omni v0.29.0rc1 while core vLLM and vllm-gaudi did not publish GitHub releases. The prerelease focused on diffusion batching, full-duplex runtime graduation, realtime speech serving, new image and audiovisual integrations, and rebasing onto vLLM 0.29.0..88

← All issuesPowered by RunAnywhere