TL;DR
- Cloud serving engines reset the bar: vLLM and SGLang shipped major serving updates around new model families, disaggregated execution, cache transfer, and hardware backend coverage.1
- Kernel work moved to the front page: FlashInfer, TensorRT-LLM, CUTLASS, Triton, AITER, and TileLang all pushed low-precision attention, MoE, and GPU codegen work that now shapes serving behavior.2
- Local runtimes became agent infrastructure: Ollama, llama.cpp, LocalAI, exllamav3, Rapid-MLX, Wally, and vllm-mlx all worked on reasoning, tool calls, structured output, cache reuse, and desktop integration.3
- Edge stacks converged on deployability: LiteRT, ExecuTorch, OpenVINO, ncnn, sherpa-onnx, Qualcomm GenieX, and TensorFlow Lite Micro all focused on packaging, accelerator coverage, model import, and runtime correctness.4
- Security and provenance moved up the queue: LocalAI, LiteLLM, llamafile, LightLLM, Open WebUI, and TensorRT all saw work or reports around signed artifacts, credential safety, malformed inputs, and exposed serving surfaces.5
This Week in Inference
The supplied market scan did not confirm a clean external model-launch cycle, so the week’s strongest signal came from repository activity rather than press releases. Engines and runtimes raced to make new or fast-moving model families usable across serving stacks: vLLM highlighted DeepSeek, GLM, K2, and vision paths in its release notes, SGLang added GLM and Hy-family support, and Hugging Face Diffusers plus OpenVINO GenAI moved Qwen image pipelines into deployable code paths (vLLM release, SGLang release, Diffusers Qwen-Image work, OpenVINO GenAI Qwen-Image work).6
The core optimization pattern was memory and movement control. FlashInfer formalized a MoE and sparse attention release, LightLLM added multi-token KV-cache pages and FP8 KV paths, SGLang expanded HiCache and disaggregated serving, vLLM pushed KV transfer and offload work, and Mooncake added transport and store hardening for large-model serving data movement (FlashInfer release, LightLLM KV pages, SGLang HiCache work, vLLM KV work, Mooncake transport work).10 This is the same fight across cloud, desktop, and edge: less wasted prefill, better cache reuse, smaller KV, and fewer copies between stages.
Hardware work spread across the whole map. NVIDIA cut CUTLASS and TensorRT releases while TensorRT-LLM kept moving toward Blackwell and Rubin serving paths, ROCm pushed AITER and ATOM through attention and MoE work, Google moved LiteRT and XNNPACK pieces for device benchmarking and low-bit execution, Meta shipped ExecuTorch packaging for delegate libraries, and Qualcomm GenieX added HTP power-mode controls for Snapdragon deployment (CUTLASS release, TensorRT release, AITER release, LiteRT-CLI release, ExecuTorch release, GenieX power-mode work).15
Top Stories
vLLM and SGLang turn releases into platform resets
vLLM shipped a major serving release with DeepSeek, GLM, MXFP8 KV, async prefetch, disaggregated serving, and multimodal work across core and Omni paths (vLLM release).6 SGLang answered with a large core release plus an Omni release, adding GLM and Hy support while advancing HiCache, disaggregation, speculative decoding, and speech serving (SGLang release).7 The competitive line between the two projects now sits less on OpenAI-compatible APIs and more on cache systems, scheduler policy, hardware specialization, and multimodal latency.
FlashInfer makes kernels a serving product
FlashInfer shipped a release that unified MoE APIs, expanded sparse and linear attention, added Autotuner work, and pushed Blackwell-era serving kernels into a more explicit product surface (FlashInfer release).10 The project’s issue and PR flow also showed direct pressure from SGLang, vLLM, Blackwell, SM12x, NVFP4, and GDN users, which makes FlashInfer a live dependency for the serving engines above it (FlashInfer issue flow).21
Google, Meta, OpenVINO, and Qualcomm tighten the edge path
Google advanced LiteRT, LiteRT-LM, MediaPipe, LiteRT-CLI, Gallery, and XNNPACK around device benchmarking, model packaging, retrieval APIs, and low-level kernels (LiteRT-CLI release).18 Meta’s ExecuTorch release packaged SDK-style wheels and linkable delegate libraries across CUDA, Core ML, MLX, OpenVINO, Qualcomm, and TorchAO paths (ExecuTorch release).19 OpenVINO GenAI and Qualcomm GenieX added model and runtime coverage for Qwen, audio, vision, and Snapdragon power control, showing that edge inference is now a full deployment stack rather than a converter plus a demo app (OpenVINO GenAI release, GenieX release).22
Local runtimes move from chat apps to agent runtimes
Ollama simplified thinking plus structured-output behavior and pushed MLX acceleration on Apple Silicon (Ollama release).24 LocalAI added signed OCI model galleries, zero-shot NER, skeletal animation, and broader backend reliability, while Wally narrowed its CLI around offline local LLM harnesses for coding tools (LocalAI release, Wally release).25 exllamav3 added fractional-trellis quantization and broader tensor-parallel model support, keeping NVIDIA desktop inference competitive for users who need high-compression local serving (exllamav3 release).27
Supply-chain and safety issues became operational work
LocalAI added signed OCI gallery support and tightened backend signature verification (LocalAI signed gallery work).28 LiteLLM focused its release train on Docker image signature verification, while llamafile users asked for stronger build provenance and release attestations (LiteLLM release, llamafile provenance issue).29 LightLLM also received a critical report about unauthenticated remote code execution risk in PD disaggregation over NCCL transport, which is the kind of bug that turns inference architecture into security architecture (LightLLM security report).31
Deeper Dive
Everything below is for readers who want the full picture. Feel free to scroll.
Code Changes by Category
Cloud & Datacenter Serving
vLLM, SGLang, TensorRT-LLM, Dynamo, Ray Serve, LightLLM, LMDeploy, Triton, and ATOM all worked on the same production problems: model admission, cache transfer, disaggregated prefill/decode, hardware-specific kernels, and OpenAI-compatible behavior (vLLM release, SGLang release, TensorRT-LLM release, Dynamo release, Ray Serve metrics work, LightLLM KV work, LMDeploy tool-call fix, Triton cancellation work, ATOM release).6 Dynamo’s router and selection changes, SGLang’s HiCache work, vLLM’s KV-transfer work, and Mooncake’s transport hardening all point to the same serving design: cache locality and transfer semantics now drive latency and capacity planning (Dynamo router work, SGLang HiCache work, vLLM KV offload work, Mooncake transport work).12
Local LLM Runtimes
llama.cpp shipped a stable release while syncing ggml backend work into both llama.cpp and whisper.cpp, and Ollama layered reasoning, structured output, registry, MLX, and desktop fixes on top of its local model UX (llama.cpp release, whisper.cpp ggml sync, Ollama release).24 LocalAI, exllamav3, vllm-mlx, Wally, Open WebUI, GPT4All, and text-generation-webui showed the local stack shifting toward agents, tool calls, signed model distribution, structured output, and defensive API validation (LocalAI release, exllamav3 release, vllm-mlx release, Wally release, Open WebUI release, GPT4All memory proposal, text-generation-webui validation proposal).25
Apple Silicon & MLX Ecosystem
Apple’s Core AI and MLX work touched export reliability, diffusion quantization, Swift distributed support, model coverage, and runtime correctness across Core ML and MLX libraries (coreai-models SAM work, coreai-optimization work, mlx-swift distributed work, mlx-lm MiniMax work).45 Rapid-MLX, oMLX, Osaurus, FluidAudio, Blaizzy’s MLX audio and VLM projects, Ollama MLX, and vllm-mlx all treated Apple Silicon as a production platform for multimodal and agent workloads rather than a local curiosity (Rapid-MLX release, oMLX release, Osaurus release, FluidAudio release, mlx-audio release, vllm-mlx release).41
Mobile & Edge Frameworks
ExecuTorch, LiteRT, OpenVINO, Qualcomm GenieX, ncnn, sherpa-onnx, TensorFlow Lite Micro, MediaPipe, XNNPACK, and MNN all advanced packaging, operator coverage, accelerator routing, and runtime reliability for on-device workloads (ExecuTorch release, LiteRT-CLI release, OpenVINO GenAI release, GenieX release, ncnn Vulkan work, sherpa-onnx Moonshine fix, TFLM dependency work, MediaPipe retrieval work, XNNPACK low-bit work, MNN backend work).18 The week’s edge signal was practical rather than flashy: benchmark on real devices, ship linkable delegates, fix package slices, support more model families, and expose errors users can debug (LiteRT-CLI benchmarking, ExecuTorch delegate packaging, FluidAudio Catalyst fix, Qualcomm model catalog release).19
Compilers, Runtimes & Graph Engines
Triton, OpenXLA, TVM, JAX, TensorFlow, XNNPACK, AMDMIGraphX, TileLang, and Luminal all moved compiler and runtime internals that inference users feel as lower latency, fewer miscompiles, wider hardware coverage, and safer exports (Triton FP8 fix, OpenXLA async work, TVM release candidate, JAX release, TensorFlow XLA work, XNNPACK work, AMDMIGraphX HIP Graph work, TileLang block-scaled GEMM, Luminal SSA merge).65 The compiler stack also had a clear correctness mood, with rejected or redirected PRs in Triton, JAX, OpenXLA, TVM, and TensorFlow showing maintainers pushing back on fixes that changed semantics, added backend debt, or lacked benchmark proof (Triton debate, JAX debate, OpenXLA debate, TVM debate, TensorFlow debate).72
Models, Quantization & Optimization
Hugging Face Diffusers and Optimum Intel added Qwen image support across PyTorch and OpenVINO paths, while Transformers worked through cache, model, MoE, Mamba, MPS, XPU, OpenVINO, vLLM, and GGUF compatibility fixes (Diffusers Qwen work, Optimum Intel Qwen work, Transformers cache work).8 Quantization and low-precision work spread across FlashInfer, CUTLASS, TensorRT-LLM, LightLLM, exllamav3, AITER, NNCF, Olive, Intel Neural Compressor, Quark discussions, and TileLang (FlashInfer release, CUTLASS release, TensorRT-LLM release, LightLLM FP8 KV work, exllamav3 release, AITER release, NNCF release, Olive INT2 work, Neural Compressor request, TileLang FP4 work).10
Other Notable Changes
LiteLLM had a high-volume security and proxy-governance week around MCP controls, passthrough speech providers, auto routing, spend accounting, and signed Docker images (LiteLLM release).29 Open WebUI focused on packaging, RAG, pgvector recall, chat performance, voice, image handling, and localization while shipping a much smaller slim image (Open WebUI release).42 Osaurus pushed native macOS automation, MCP/tool-call hardening, MLX/Bonsai runtime work, and rapid patch releases for an agent-style desktop product (Osaurus release).51
Community Pulse
The biggest visible momentum came from repos that sit directly in user workflows: ggml gained more than a thousand stars across llama.cpp and whisper.cpp activity, vLLM gained hundreds around its release, Open WebUI saw heavy issue volume around the release, and BerriAI LiteLLM continued high PR throughput around proxy operations (ggml llama.cpp, vLLM, Open WebUI, LiteLLM).84 Apple-local projects also showed strong community pressure, with Rapid-MLX, oMLX, Osaurus, Ollama, MLX libraries, FluidAudio, Blaizzy projects, and vllm-mlx all receiving feature and bug traffic around multimodal serving, Apple runtime behavior, and local agents (Rapid-MLX issues, oMLX discussion, Osaurus issues, Ollama issues, MLX issues, FluidAudio issues, Blaizzy mlx-vlm issues, vllm-mlx issues).88 Security and provenance were unusually visible, with llamafile provenance, LightLLM RCE risk, LocalAI signed artifacts, LiteLLM image signing, and Open WebUI URL/security discussions all appearing in one week (llamafile issue, LightLLM issue, LocalAI OCI work, LiteLLM release, Open WebUI security thread).29
Community Debates
Triton resisted one-off scheduling fixes in favor of a cleaner compiler model. A proposal for FA3-style overlap of CUDA-core and tensor-core work closed because maintainers viewed it as too ad hoc for the current pipeliner and preferred a more programmable scheduling path (Triton PR).72 That debate mirrors similar pushback across compilers this week, where maintainers accepted targeted correctness fixes but resisted backend debt without a broader design.
SGLang maintainers pushed external-cache design back toward HiCache. An external-cache linker proof of concept closed after reviewers argued it duplicated HiCache and should expose backend L3-to-L2 and L3-to-L1 capabilities instead (SGLang PR).97 The discussion matters because external cache APIs decide how KV stores, disk tiers, and remote memory systems plug into real serving engines.
vLLM rejected several performance ideas that lacked evidence or safe fallbacks. FlashAttention FP8 output quantization closed after questions about H100 benefit and fallback behavior, while KV-offload batching closed after prefill regressions from queue and GIL contention (vLLM FP8 PR, vLLM KV-offload PR).98 The pattern shows vLLM’s maintainers treating release-scale serving stability as more important than isolated microbenchmark wins.
OpenVINO and TensorFlow both drew firm boundaries around deprecated or wrong-layer fixes. OpenVINO rejected disabling optimized SDPA by default for Whisper GPU and chose to document the workaround, while TensorFlow rejected TFLite overflow checks that bypassed model verification because they violated the LiteRT security model (OpenVINO GenAI PR, TensorFlow PR).76 These decisions show mature inference projects pushing security and support boundaries into the correct layer.
Worth Watching
KV-transfer APIs are converging into a standards fight. vLLM, SGLang, Dynamo, Mooncake, LightLLM, LMDeploy, ATOM, and Ray all touched cache movement, prefill/decode separation, or scheduling fairness this week (vLLM KV work, SGLang cache work, Dynamo routing work, Mooncake work, LightLLM KV work, LMDeploy Mooncake work, ATOM KV work, Ray Serve issue).11 Apple Silicon is becoming its own inference platform tier. MLX, Core AI, Rapid-MLX, oMLX, Osaurus, Ollama, vllm-mlx, FluidAudio, and Blaizzy’s libraries are converging on local multimodal agents, not just local chat (MLX Swift work, Core AI work, Rapid-MLX release, oMLX release, Osaurus release, Ollama release, vllm-mlx release, FluidAudio release, mlx-vlm release).24 Supply-chain provenance will become table stakes for local inference. LocalAI, LiteLLM, llamafile, Open WebUI, TensorRT, and LightLLM all exposed signing, artifact, malformed-input, or remote-execution concerns in one week (LocalAI signing, LiteLLM release, llamafile issue, Open WebUI issue, TensorRT issue, LightLLM issue).29
Major Releases
Version numbers and release links live here as the canonical reference list.
ai-dynamo shipped v1.5.0 as a feature release centered on router policy, KV indexing, guided tool-call streaming, and over-context rejection, while also publishing an experimental K-EXAONE prerelease for evaluation. The release reflected a large router, multimodal, SGLang, vLLM, and operator-hardening push across the repo..33
AMD Ryzen AI shipped FastFlowLM v1.0.6 and Lemonade v2026.39.1 plus a candidate v2026.40.0, with work centered on AMD local serving, NPU/GPU packaging, token accounting, tray/server UX, and model catalog updates. The most user-visible change was Lemonade’s broader AMD GPU support and configurable VRAM eviction. Lemonade release.106
Apache published TVM v0.27.0 release candidates and tvm-ffi v0.1.14-post1, with work centered on TIRx, BigInt IR, CUDA host codegen, WebGPU correctness, ORC JIT, and Rust traversal. The release cycle shows TVM hardening compiler infrastructure rather than chasing model-specific headlines. TVM release candidate.65
BerriAI shipped a rapid LiteLLM release train from v1.99.3 through v1.104.0-dev.1, with release notes focused on Docker image signature verification using cosign. The code work around those releases also touched MCP access control, passthrough speech providers, routing, savings accounting, and admin security..29
Blaizzy shipped mlx-audio v0.5.5 and mlx-vlm v0.7.2, centered on Apple Silicon audio, ASR, TTS, VLM, image, and multimodal support. The highest-impact changes were Nemotron diarization, Irodori-TTS, MiMo, Qwen image, APC memory fixes, and LFM corrections. mlx-vlm release.104
Dao-AILab shipped FlashAttention fa4-v4.0.0.beta32, focused on sparse MLA recompute-P and token-chunked backward, C++20 wheel builds, PyTorch wheel-matrix work, and registry-free GPU CI. The week’s code work also tightened SM100 sparse MLA training numerics and Q-head coverage..107
FlashInfer shipped v0.7.0 after a release candidate, with unified MoE APIs, PrimTS TRT-LLM Gen MoE kernels, Autotuner v2, experimental API policy, sparse and linear attention, expert parallel serving, diffusion, and Blackwell support. This was one of the week’s most important kernel-stack releases..10
FluidInference shipped FluidAudio v0.17.1, v0.17.0, v0.16.1, and v0.15.8 plus text-processing-rs v0.3.1, with a focus on Core ML audio and Apple-platform packaging. Nemotron diarization, CUA-S1-FORMS scoring, Kokoro language support, and Catalyst slices were the dominant changes. FluidAudio release.52
ggml shipped llama.cpp v0.5.0 plus many prerelease builds, centered on ggml backend sync, model coverage, server/router robustness, multimodal/API fixes, and chat parser work. whisper.cpp did not cut a named release in the supplied summary, but it absorbed the same ggml backend updates. llama.cpp release.39
Google AI Edge shipped LiteRT-CLI v0.2.1, adding real-device benchmarking for LiteRT and LiteRT-LM models through Google Cloud and Developer Device Platform. The wider org also moved LiteRT, LiteRT-LM, MediaPipe, Gallery, samples, XNNPACK, and quantizer work in the same direction..18
Hugging Face shipped optimum-intel v2.2.0 with new OpenVINO model support spanning Mistral, DeepSeek-OCR, Muse, Ouro, Fun-ASR, LTX, Qwen-Image, Z-Image, and DFlash draft models. The broader Hugging Face week was also strong in Diffusers, Transformers, Candle, and CI/security hardening..108
JAX shipped jax-v0.11.2 with jnp.minmax, first-class lax.log2, lax.one_minus_square, and symbolic export improvements. The surrounding work advanced XLA integration, Pallas/Mosaic GPU, numerical AD, and platform baselines..66
jundot shipped oMLX v0.7.0.dev4 as a prerelease warning users about large internal changes from core-library upgrades. The release sits on a week of MiMo multimodal support, MoE offload, GLM/DeepSeek memory fixes, Qwen/VLM speed recovery, and cluster work..50
Meta shipped ExecuTorch v1.5.1 with CMSIS-NN v8.0.0, expanded Arm dynamic-shape support, shared EXIR Q/DQ folding, and SDK-style wheels with linkable delegate libraries. PyTorch core had no release, but it landed compiler, export, precompile, MPS, CUDA, ROCm, XPU, and symbolic-shape work. ExecuTorch release.19
Microsoft shipped ONNX v1.23.0 and ONNX Runtime plugin-ep-webgpu/v0.4.0, with ONNX adding IR updates, SwiGLU, attention updates, FLOAT6 types, and bfloat16 support, while ORT WebGPU expanded kernel support and reliability. Foundry Local and Olive had no release, but both moved low-bit, catalog, and model-loading work. ONNX release.109
Modular shipped MAX 26.6 / Mojo 1.1.0, with MiniMax-Music3, Inkling multimodal NVFP4 MoE, GLM-5.3, broader Kimi/Qwen/Gemma coverage, and speculative decoding with DFlash/DSpark draft models. The repo also advanced MAX/Mojo nightlies and serving telemetry..110
mudler shipped LocalAI v4.10.0 after a broad cycle focused on fleet operations, private OCI and gallery credentials, benchmarking, ds4 CUDA reliability, distributed mode, CVE patches, thinking controls, and Apple startup fixes. The week’s code added signed OCI gallery artifacts and several new modalities..25
Open WebUI shipped v0.11.4, focused on a much smaller slim image, RAG/vector fixes, chat performance, terminal/automation fixes, media handling, and localization. The pgvector recall fix and slim-image reset were the standout operational changes..42
OpenVINO Toolkit shipped NNCF v3.4.0 and OpenVINO GenAI 2026.4.0.0, with compression work around GroupedMatMul, AWQ, Scale Estimation, and RoPE ignored patterns, plus GenAI release alignment. Core OpenVINO also landed frontend, NPUW, GPU, CPU, and CI improvements. NNCF release.80
NVIDIA shipped CUTLASS v4.8.0, TensorRT v11.3, and TensorRT-LLM v1.3.0 release candidates. CUTLASS focused on Rubin, FP8, FP4, Tensor Core, TMEM, shared memory, and extension compiler work, while TensorRT moved to CUDA 13.4 defaults and TensorRT-LLM pushed Blackwell/Rubin serving paths. CUTLASS release.15
Ollama shipped v0.34.3 and v0.34.4, focused on thinking controls, single-pass structured output for thinking models, MLX and llama.cpp updates, Qwen prompt-processing speed, dynamic Gemma image resolution, registry fixes, and macOS responsiveness. The releases tightened both model behavior and desktop UX. Latest release.24
Osaurus shipped seven patch releases from 0.25.6 through 0.25.12, centered on native macOS app tools, MCP/tool-call robustness, Bonsai and MLX runtime support, provider integrations, and UI hang fixes. The cadence shows a product in rapid stabilization around local agent workflows. Latest release.51
Qualcomm shipped ai-hub-models v0.63.0 and GenieX v0.7.0, expanding the model catalog and adding a Microsoft-signed Hexagon HTP catalog for Windows on Snapdragon. GenieX also advanced power-mode controls, server metrics, VLM behavior, and llama.cpp integration. GenieX release.23
Rapid-MLX shipped v0.15.0 and rapid-mac-v0.15.0, plus v0.14.3 desktop and CLI releases earlier in the week. The release line focused on telemetry v2, Qwen-Image generation/editing, clearer failures, lower desktop idle CPU, multimodal speedups, and benchmark UX..49
RunanywhereAI shipped Wally v0.5.8, v0.5.9, v0.5.10, v0.6.0, and runanywhere-sdks v0.20.37, focused on offline LLM workflows, coding harnesses, catalog behavior, Windows packaging, and SDK ABI/server support. Wally v0.6.0 was the key shift toward a narrower local LLM CLI. Wally release.26
SGLang shipped sglang v0.5.20 and sglang-omni v0.1.6, with the core release adding model support and the Omni release adding stage factory, Qwen3-ASR, scheduler, and Nemotron Voicechat work. The two releases show tighter alignment between high-throughput LLM serving and speech/multimodal serving. Core release.7
ROCm shipped AITER v0.1.22.post1 and ATOM v0.1.6, with AITER focused on manylinux wheels and cherry-picks for MXFP4 MoE, routing, FP8 MQA, and DSv4 tuning, while ATOM published ROCm wheels and dev images. The surrounding work connected AITER kernels to ATOM serving paths. ATOM release.37
try-mirai shipped Uzu 0.5.28 and 0.5.30 with empty release bodies, while the code work focused on tool-aware chat, benchmark replay, continuation handling, bindings, Apple packaging, and release automation. Lalamo did not ship a release but improved quantization-parameter layout semantics. Uzu release.111
turboderp-org shipped exllamav3 v1.5.1, focused on KimiLinear, DFlash2, tensor-parallel support for Qwen, GLM, and DeepSeek families, MoE optimizations, fractional-trellis quantization, YAQA and self-calibration work, memory fixes, and Windows/Conda fixes. This was a major local NVIDIA inference release..27
vllm-project shipped vLLM v0.30.0 and vLLM Omni v0.30.0rc1, with core vLLM focused on DeepSeek, GLM, MXFP8 KV, DeepGEMM, async prefetch, and vision work, and Omni rebased onto core while expanding full-duplex, speech, image, video, VLA, and cross-stage transfer. vllm-gaudi did not publish a release but kept HPU support synchronized. vLLM release.6
waybarrios shipped vllm-mlx v0.5.0, adding reusable conversation caching, broader model and parser support, more reliable streaming and resource management, prefix-trie caching, and DeepSeek-V4-Flash prompt, reasoning, and DSML tool-call support. The release positions vllm-mlx as an Apple Silicon local serving bridge for agent clients..41