TL;DR
- Cache economics took over: vLLM and SGLang pushed cache, offload, scheduler, and hardware paths that reduce wasted prefill and keep long-context serving viable.1
- Local runtimes chased frontier model pressure: llama.cpp added Qwen Flash Next support, DFlash-style speculation, lazy tensor reads, and backend work that Ollama and exllamav3 now have to absorb.2
- Apple Silicon moved up-market: MLX, Core AI, oMLX, Rapid-MLX, and Osaurus all worked on large-model serving, cache reuse, multimodal inputs, and Mac memory limits.3
- Kernel work set the hardware agenda: FlashInfer, TensorRT-LLM, CUTLASS, ROCm AITER, Triton, and tilelang all focused on MoE, FP8, NVFP4, Blackwell, Rubin, and ROCm serving paths.4
- Gateways became control planes: LiteLLM, LocalAI, Foundry Local, Open WebUI, and Triton hardened OpenAI-compatible APIs, routing, pricing, auth, streaming cancellation, and distributed worker behavior.5
This Week in Inference
Repository activity, not outside launch news, carried the week: vLLM added new frontier-model paths and hardware coverage while its omni stack pushed diffusion, speech, video, and multimodal serving into the same release lane.1 SGLang spent the week on allocator, radix cache, CUDA Graph, disaggregation, and platform bring-up work, which shows how serving engines now compete on the memory and scheduler layer before they compete on API shape.6
Local and edge runtimes followed the same pattern at smaller scale: llama.cpp added large-model loader changes, speculative decoding, lazy reads, and backend acceleration, while Ollama pulled in llama.cpp updates and hardened MLX multimodal support.2 Google LiteRT, ExecuTorch, Qualcomm GenieX, sherpa-onnx, ncnn, and MNN all moved on mobile, NPU, CPU, and app-facing deployment paths.8
The kernel and compiler layer also stayed hot: FlashInfer expanded MoE, attention, MLA, and Rubin support, while TensorRT-LLM added KV compression, DeepSeek and Kimi paths, and Rubin work.4 ROCm AITER, ATOM, Triton, OpenXLA, TVM, and OpenVINO all point to the same direction: model-serving speed now depends on compiler IR, cache layout, kernel packaging, and hardware-specific fallbacks.15
Top Stories
vLLM turns frontier support into platform plumbing vLLM added Qwen Flash Next and DeepSeek Flash Vision support while pushing multimodal cache hashing, parser plugins, admission-control errors, Model Runner defaults, and hardware fixes across CUDA, ROCm, XPU, and Ascend in vLLM.1 The same org shipped an omni stack with diffusion paged KV, duplex speech, MiniMax-H3 serving, and large multimodal CI updates in vLLM-Omni.21 The pattern matters because vLLM now has to serve text, vision, speech, diffusion, and hardware-plugin users without splitting the platform into separate products.
SGLang keeps pushing the serving hot path SGLang worked across unified cache state, allocator cleanup, radix behavior, CUDA Graph correctness, disaggregation, speculative decoding, FP4/NVFP4 paths, and CPU/NPU/XPU/ROCm support in SGLang.6 sglang-omni added TTS, ASR, graph capture, Apple Silicon, Intel CPU/XPU, Ascend, and profiling work in sglang-omni.22 The project is turning “fast serving” into a full runtime policy layer, with cache admission, graph capture, hardware routing, and multimodal schedulers all in scope.
llama.cpp makes local serving absorb frontier-model complexity llama.cpp added Qwen Flash Next support, DFlash speculative decoding, lazy mmap reads for large tensors, server media inputs, and backend work across CUDA, ROCm, Metal, Vulkan, OpenCL, SYCL, CPU, Hexagon, and WebGPU in llama.cpp.2 Whisper.cpp also fixed Parakeet TDT decoding and zero-sample input handling in whisper.cpp.23 Local runtimes now inherit the same hard problems as datacenter stacks: multimodal prompts, sparse attention, giant embeddings, offload, and tight cache accounting.
FlashInfer, TensorRT-LLM, CUTLASS, and ROCm fight the kernel battle FlashInfer expanded unified MoE, MXFP8, NVFP4, MLA, sparse attention, Kimi Linear, Rubin, and packaging paths in FlashInfer.4 NVIDIA paired CUTLASS Rubin work with TensorRT-LLM changes for KV compression, DeepSeek sparse decode, SafeTensors loading, Kimi paths, MoE, and visual-generation serving in TensorRT-LLM.14 ROCm matched that pressure through AITER and ATOM work on DeepSeek, Kimi, GLM, DCP, FP8, sparse attention, MoE, and MI-class serving paths in AITER.15
Gateways and UIs become inference infrastructure LiteLLM added routing, auto-routing, time-window pricing, guardrails, MCP Gateway work, Bedrock fixes, Redis behavior, secure-cookie handling, Rust/Python route splitting, and observability in LiteLLM.5 LocalAI rebuilt distributed worker connectivity around outbound tunnels, improved DS4 reasoning behavior, added OpenAI Realtime WebRTC compatibility, and refreshed its backend gallery in LocalAI.24 Open WebUI, Foundry Local, Triton, and GPT4All all show the same shift: the API gateway, desktop app, and admin UI now make routing, auth, cost, and model behavior decisions.
Deeper Dive
Everything below is for readers who want the full picture. Feel free to scroll.
Code Changes by Category
Cloud & Datacenter Serving
vLLM had one of the heaviest weeks in serving, with new model paths, Model Runner defaults, parser plugins, admission-control responses, kernel work, and hardware fixes across CUDA, ROCm, XPU, Ascend, and NIXL.1 SGLang matched that scale through scheduler, cache, allocator, CUDA Graph, speculative decode, MoE, FP4, NVFP4, CPU, NPU, XPU, ROCm, and GB-class platform work.6
NVIDIA TensorRT-LLM focused on Rubin, Blackwell, DeepSeek, Kimi, MoE, KV compression, SafeTensors loading, speculative decode, and visual-generation infrastructure.14 ai-dynamo added large-model recipes, SGLang and vLLM backend fixes, sidecar routing, Kubernetes hardening, NIXL work, and GB-class deployment docs.25
LMDeploy added Mooncake-backed distributed KV storage, DeepEP integration, W4A16 AWQ backend work, MLA and DSA fixes, OpenAI-style logprobs, guided decoding, and long-context workarounds.26 LightLLM moved on PD serving limits, multi-level KV placement, short-prefill scheduling, reasoning-token accounting, and MTP communication reductions.27
DeepSpeed improved AutoEP, AutoTP, ZeRO offload, OPSD Hybrid Engine rollout, distributed barriers, Ulysses layout, CPU Adam, DeepCompile, and profiler correctness.28 Ray worked on Ray Data Parquet reads, Serve scale-to-zero behavior, RuntimeEnv cleanup, KubeRay auth, dashboard fixes, and security guards for unpickling.29
Triton Inference Server shipped release-train work, OpenAI-compatible stream cancellation, authenticated build secrets, TensorRT QA cleanup, and Torch AOTI smoke coverage.30 LiteLLM turned provider routing, pricing, guardrails, MCP Gateway, Bedrock, realtime behavior, proxy hardening, Redis, database, and observability into one gateway control plane.5
Local LLM Runtimes
llama.cpp carried the local runtime week with Qwen Flash Next, DFlash speculative decoding, lazy reads for large embeddings, model-loader expansion, server media inputs, and backend work across most major device targets.2 Ollama added Gemma multimodal support on MLX, prompt-cache metrics, model-defined generation defaults, MLX CI, llama.cpp bumps, desktop fixes, and Claude Desktop polish.7
LocalAI rebuilt distributed mode so workers connect outbound through tunnels, then layered DS4 reasoning fixes, Realtime WebRTC support, backend bumps, gallery entries, and UI/docs improvements.24 exllamav3 added GLM Flash and Qwen Flash Next support, PLE streaming, QSA and KDA pieces, VRAM accounting, async generation failure isolation, and recurrent-state leak fixes.31
web-llm removed unsafe HTML rendering from examples, added lint guards against XSS-prone sinks, improved worker recovery, and fixed chat initialization hangs.32 GPT4All had no code land, but users documented OpenAI-compatible custom-provider setup and production-hardening concerns.33
text-generation-webui had no merges, but community PRs targeted tool-message prompt truncation, preset filename validation, and LaTeX rendering.34 CTranslate2 tightened malformed model metadata checks, allocation overflow handling, and runtime dependency cleanup.35
Apple Silicon & MLX Ecosystem
Apple coreai-models added session-aware KV prefix reuse, stats and readiness endpoints, tool calling, macOS prefill/decode split, quantized graph-mode prefill export, INT8 KV-cache export, and new export presets.36 MLX improved CUDA, Metal, numerics, safetensors, GGUF loading, attention routing, and Python API behavior.3
mlx-lm added Kimi, Gear, Nanbeige, GPT-J, Qwen MoE parallelism, MoE training fixes, a packed gated-delta kernel, and generation fixes.37 mlx-swift-lm added Qwen compiled decode, GDN, MoE work, checkpoint loading fixes, async loading, and FoundationModels integration.38
mlx-vlm rebuilt APC cache behavior, added MoE expert paging, Qwen Flash Next work, GLM support, DeepSeek vision, LongCat, LLaVA-OneVision, Video Depth Anything, and many video/server fixes.39 mlx-audio added Phonon, Granite Speech, Audio8, ArkTTS, VibeVoice streaming, Qwen ASR/TTS fixes, Whisper fixes, and audio decoding improvements.40
oMLX focused on Qwen Flash Next, GLM Flash, QSA, Lightning MTP, TurboQuant, PLE mmap, ANE offload, continuous batching, prefix cache, and Apple memory limits.41 Rapid-MLX added local benchmark contracts, video generation foundations, GLM serving, Qwen MTP work, catalog migration, memory auto-tuning, and Desktop observability.42
Osaurus shipped a rapid macOS release train around vMLX, native MTP, cache persistence, allocator ceilings, agent-loop cache reuse, eval gates, custom endpoints, sandboxing, and Claude Code integration.43 FluidAudio saw Apple Neural Engine, CoreML, diarization, BNNS, and XCFramework linkage issues drive the week.44
Mobile & Edge Frameworks
ExecuTorch advanced on-device LLM batching, MLX attention lowering, Core ML export, Apple iOS builds, Arm demos, QNN fixes, Vulkan issues, WebGPU cleanup, Android preprocessing, and runtime IR work.9 Google LiteRT worked on GPU kernels, MLDrift paths, TensorRT, OpenVINO, Qualcomm backends, StableHLO conversion, TFLite converter safety, Apple packaging, and OSS GPU tests.8
LiteRT-LM standardized assistant roles, expanded Qwen, Gemma, and MiniCPM templates, added speculative-decoding config, exposed multimodal controls, and improved TTS/ASR runner coverage.45 XNNPACK added Intel AMX GEMM, KleidiAI SME2 kernels, Wasm FP16 work, AVX and BF16 paths, RVV kernels, constant propagation fixes, and overflow hardening.46
Qualcomm GenieX improved cache continuation detection, tool-call streaming, QAIRT metadata dispatch, llama.cpp and Hexagon VLM routing, model downloads, disk checks, benchmarking, and RB3 behavior.10 sherpa-onnx expanded Rust, Tauri, iOS, Android, JavaScript, Flutter, Swift, Java, Python, and HarmonyOS packaging while fixing Qwen ASR silent-audio behavior.11
ncnn added x86 SDPA flash attention, GQA benchmark coverage, FMA/FMA4 paths, allocator fixes, Vulkan shader fixes, and CI repairs.12 MNN fixed AWQ multimodal calibration, Metal attention masks, RISC-V prefill behavior, shared-context release handling, and converter robustness.13
cactus needle added on-device tool-calling environments, telemetry controls, slimmer install extras, schema fixes, and playground temp-directory hardening.47 mllm added ARM CPU support for Liquid LFM and promoted Qwen Gated Delta Rule into a first-class runtime op.48
Compilers, Runtimes & Graph Engines
PyTorch shipped a major release while continuing CUDA BF16x9 matmul, Inductor NVGEMM and CuTeDSL work, Dynamo migration, AOTInductor fixes, MPS work, ROCm, XPU, MTIA, profiler, build, and releng changes.49 Triton shipped a compiler release with layout-analysis fixes, rematerialization guards, loop propagation, interpreter math fixes, NVIDIA TMEM work, and AMD ROCm enablement.17
JAX focused on Mosaic GPU, Pallas GPU and TPU lowering, thread-safe array internals, linear algebra numerics, NumPy compatibility, oneAPI, ROCm, and XLA pins.50 OpenXLA pushed IFRT proxy remapping, PJRT cleanup, cuDNN fusion, Triton tile guards, oneAPI F64 matmul, ROCm runfiles, IR correctness, StableHLO import behavior, TPU work, and Shardy fixes.18
TVM added CUDA and PTX support for newer NVIDIA targets, TIRx and Core IR refactors, Relax and Torch frontend fixes, ONNX correctness fixes, scheduling fixes, and web BF16 cache fixes.19 OpenVINO landed LLM attention and KV-cache fixes, SSM state work, GPU JIT and caching, NPUW routing, CPU snippets, GGUF handling, and malformed-IR safeguards.20
ONNX Runtime advanced WebGPU, CUDA LLM kernels, PagedAttention, speculative decode, fpA-intB and NVFP4 GEMV, CUDA graph replay, CPU/MLAS, graph validation, and build maintenance.51 AMDMIGraphX improved reduce JIT vectorization, fusion, attention matching, GPU backend passes, rocMLIR GEMM fusion, problem-cache behavior, symbolic ONNX parsing, packaging, and CI.52
tilelang shipped reducer, layout inference, warp scheduling, Hopper and Blackwell, TMA, WGMMA, UMMA, ROCm, validation, CPU GEMM, docs, and testing work.53 Luminal reorganized runtime search, CUDA-lite, CUDA graph cache residency, PyTorch importer coverage, serving scheduler cache caps, and API design for caller-owned NN layers.54
Models, Quantization & Optimization
Transformers added VibeVoice, Canary, NeuCodec, NeoMME, batch rebalance sampling, hidden-state capture, MTP memory reductions, attention and RoPE fixes, and generation guards.55 Diffusers added LTX DFR, Hub revision resolution, attention mask fixes, callback and offload fixes, and a large test migration.56
Candle added opt-in Rust CUDA/cuTile fused MoE support, better unsupported-GPU errors, alias-aware in-place ops, backend fixes, BERT safetensors compatibility, and converter dependency cleanup.57 Optimum Intel moved Gemma, Qwen, DFlash, OpenVINO export, Gemma video input, and INT4 config work forward.58
Intel Neural Compressor added a CuTe/CUTLASS-DSL backend for MXFP4 and MXFP8 QDQ in its vLLM QDQ plugin, with runtime selection, fallback checks, CUDA Graph support, benchmarks, and tests.59 NNCF added OpenVINO grouped MatMul and grouped-MM MoE support for AWQ, scale estimation, and compression flows.60
Liquid4All leap-finetune added KTO training, simplified single-GPU trainer behavior, fixed SFT eval-loss accounting, aligned causal-LM token counts, and rejected all-zero assistant masks.61 FastFlowLM added Gemma multimodal support, new NPU model coverage, concurrent language/vision/audio paths, and weight refreshes.62
Other Notable Changes
Open WebUI hardened migrations, streaming performance, registry caching, rejected-tool behavior, schedule editing, frontend layout, file serving, MIME handling, and localization.63 BentoML added an agent skill for scaffolding BentoML projects and documented the agent-skills workflow.64
Foundry Local added C++ SDK docs, native BYOM registration for local ONNX paths, manylinux packaging fixes, Linux ARM64 wheel tag fixes, release-pipeline signing safeguards, and fork-friendly CI test model hosting.65 Model Explorer improved Linux ARC CI, OpDef sync, StableHLO tags, C-API conversion callbacks, diagnostics, and dependency maintenance.66
Community Pulse
SGLang had one of the week’s largest public review surfaces, with heavy churn around cache correctness, GLM tracking, hardware bring-up, constrained decoding, and production deployment profiles.67 vLLM saw intense issue flow around Qwen determinism, XPU all-reduce, DFlash prefix-cache misses, GLM support, FlashInfer failures, optional imports, slot mapping, and GDS KV offload.68
Open WebUI had high user pressure across tool calling, CUDA images, search normalization, scalar JSON tool arguments, native MCP image content, RBAC, and vector metadata propagation.69 Ollama drew hardware and runtime reports around Jetson, Windows/NVIDIA TDR, context OOMs, RTX errors, macOS memory growth, video inputs, and Claude Desktop.70
AMD FastFlowLM surfaced a governance thread around XDNA kernel openness and rebuildability.71 exo saw a direct community question about maintainer cadence, release plans, branch guidance, and pending PR review.72
Community Debates
FastFlowLM issue 690 became the week’s clearest openness debate, with users asking AMD to publish XDNA kernel code and conversion/build tooling.71 The thread matters because FastFlowLM is becoming an NPU-facing local runtime, and users want reproducible kernels before they trust it as infrastructure.
Ollama PRs 18174 and 18196 tried to force configured MLX context length behavior.73 Maintainers closed the attempts after clarifying that mlxrunner behavior is intentional and that Ollama plans to move away from configured context lengths.
Google LiteRT PR 9682 proposed allowing fp16 GATHER input to feed fp32 output for Chromium WebNN fallback avoidance.74 Reviewers pushed the fix back toward Chromium because GATHER should preserve value type and use an explicit CAST.
NVIDIA TensorRT-Edge-LLM PR 46 proposed multi-stage, multi-arch export and runtime containers.75 NVIDIA deferred the contribution because container maintenance and infrastructure cost would become a standing burden.
Open WebUI PR 25076 proposed exact PDF citation IDs that open a source panel and jump to cited pages.76 Maintainers closed it in favor of the newer display-file flow, while commenters argued that KB and chat-file citation workflows still need native handling.
vLLM PR 54113 proposed tighter DoS bounds, but the broader vLLM closure set shows maintainers steering risky API, socket ownership, and MTP work toward narrower designs.77 The theme is clear across vLLM, SGLang, TensorRT-LLM, and LMDeploy: serving engines are rejecting wide changes unless they fit scheduler, cache, and observability boundaries.
Worth Watching
- Mooncake is becoming a shared KV substrate for distributed serving, with LMDeploy already wiring Mooncake-backed prefix reuse into its serving path.78
- FlashInfer needs to solve import-time CUDA initialization and JIT packaging pain because downstream servers like vLLM and SGLang depend on clean worker spawning.79
- OpenVINO GenAI has active EAGLE and MoE speculative decoding failures, which will test how well OpenVINO handles fast-moving LLM serving features.80
- RunanywhereAI is turning ANE, NeuRT, QHexRT, OCR, TTS, embeddings, and app SDKs into one on-device API surface.81
- Triton, OpenXLA, and TVM all have active compiler-runtime work that will decide which hardware paths become easy enough for serving teams to adopt.82
Major Releases
ai-dynamo shipped v1.4.2 plus a Qwen development prerelease, focused on NIXL loader-path fixes, SGLang runtime images, and experimental large-model backend support. The most important stable change fixed silent use of non-functional Rust NIXL stubs in frontend and SGLang runtime images..83
AMD Ryzen AI shipped FastFlowLM v1.0.3 and v1.0.4, GAIA v0.23.1, and Lemonade v11.8.1 and v11.9.0, with work spread across Gemma4 multimodal NPU support, Qwen weight refreshes, Lemonade reliability, and GAIA app lifecycle fixes. Lemonade v11.9.0 was the most user-facing repair because it fixed LAN allowed-origins breakage and streaming prefill timeouts..84
Apache shipped tvm-ffi v0.1.14-rc2, a release candidate focused on Rust reflected methods, structural map and mutate APIs, stronger typing, opaque pointer handlers, and packaging correctness. The release candidate followed cancelled earlier votes and shows the FFI layer maturing under TVM’s Core IR refactor..85
BerriAI shipped a dense LiteLLM train from v1.99.0 candidates through v1.101.0-dev.1, centered on signed Docker image verification, traceable Docker tags, and proxy release hygiene. The most meaningful production signal was the emphasis on cosign-verified containers rather than API surface expansion..86
Blaizzy shipped mlx-audio v0.5.1 and mlx-vlm v0.7.0rc0, pairing audio model support and correctness fixes with Qwen MTP, quantized KV, Qwen4 optimization, and checkpoint support in VLM serving. mlx-vlm carried the higher serving impact because it staged large-model VLM and speculative paths for Apple Silicon..87
Dao-AILab shipped fa4-v4.0.0.beta29 for FlashAttention, focused on CuTe scalar SM100 mask-compilation speedups. The release targets first-call compile latency on Blackwell-era hardware while leaving steady-state CUDA Graph latency unchanged in the reported benchmark..88
DeepSpeed shipped v0.19.6, a patch release covering Ulysses sequence-parallel checkpoint rank selection, OnebitLamb copy semantics, versioning, and OneCycle per-param-group support. The larger week still revolved around AutoEP, AutoTP, ZeRO, Hybrid Engine, and DeepCompile hardening..89
ggml shipped a long llama.cpp prerelease train from b10643 to b10769, covering Qwen4exp, DFlash2, lazy loading, Hexagon, ROCm, SYCL, Vulkan, Metal, CUDA, OpenCL, OpenVINO, WebGPU, RPC, and packaging. The key theme was frontier local-runtime support under memory pressure. Highlighted release.90
Google shipped Gallery 1.0.19, focused on Android permission cleanup, benchmark-button config refactoring, and model allowlist support for extra data files. The release sat on top of broader LiteRT, LiteRT-LM, MediaPipe, XNNPACK, and sample-app work..91
InternLM shipped LMDeploy v0.17.0, centered on DeepEPv2, Kimi K2.6 support, Mooncake KV-store support, and chat-completions fan-out. The most important change was Mooncake-backed distributed KV reuse for reducing repeated prefill across instances and cold starts..92
jundot shipped oMLX v0.6.3 and v0.6.4, focused on Qwen3.8-Flash-Next, GLM-5.3-Flash, QSA, Lightning MTP, prefix-cache reconstruction, continuous batching, and Apple Silicon large-model correctness. v0.6.4 mattered most because it hardened the newly added model paths under real user workloads..93
k2-fsa shipped sherpa-onnx v1.13.7, with updates across Rust, Tauri, JavaScript, Flutter, Dart, Android, iOS, Python, Swift, Java, and HarmonyOS. The strongest inference impact came from richer streaming ASR APIs and Qwen3-ASR silent-audio fixes..94
kvcache-ai shipped Mooncake v0.3.13.post1, a packaging release that kept non-CUDA wheels CUDA-free and backported a TestPyPI prerelease gate. The larger week focused on high-performance TCP transfer, Store HA, recovery, and KV transport reliability..95
Meta shipped PyTorch v2.14.0, with visible release highlights starting around NVGEMM and CuTeDSL-generated CUTLASS kernels for Inductor. The release landed while PyTorch and ExecuTorch continued backend and on-device inference work across CUDA, MPS, MLX, Arm, Qualcomm, Vulkan, and WebGPU..96
Microsoft shipped foundry-local v2.0.1, focused on a unified cross-language inference API, shared native runtime, Linux and ARM64 packaging, and Session APIs for chat, embeddings, and audio. Foundry Local’s BYOM and SDK work made local ONNX deployment more user-facing..97
tile-ai shipped tilelang v0.1.14, focused on Reducer v2, warp-specialization schedules, layout-inference cost models, unified backend resolution, lazy Z3 materialization, and TMA lowering. The release matters because more inference kernels now depend on DSL-level compiler features rather than hand-written CUDA alone..98
NVIDIA shipped CUTLASS v4.8.0dev and TensorRT-LLM v1.3.0rc25, pairing Rubin dense GEMM and FP8/FP4 Tensor Core work with broader KV Cache Manager defaults in TensorRT-LLM. TensorRT-LLM carried the serving impact because it made the new KV manager default for many major model families..99
Ollama shipped v0.33.2 and v0.33.3-rc2, focused on desktop fixes, Claude Desktop stability, cached prompt-token reporting, GGUF model defaults, MLX updates, and llama.cpp updates. The MLX and prompt-cache work were the core inference changes..100
Open WebUI shipped v0.11.2 and v0.11.3, focused on terminal-file previews and accessibility-mode contrast improvements. The release train also carried migration recovery, streaming hot-path work, registry caching, and i18n fixes..101
OpenNMT shipped CTranslate2 v4.8.2, focused on lazy Python converter imports, out-of-bounds read fixes, GCC compatibility, and model variable-size validation before allocation. The release was mainly a security and robustness hardening update..102
Qualcomm shipped ai-hub-apps v0.36.0, adding Ubuntu sample apps for semantic segmentation and 3D object detection plus app and docs updates. The release connected to broader GenieX and AI Hub model work around Snapdragon deployment, QAIRT, VLMs, and sample workflows..103
ROCm shipped AITER v0.1.21, AITER v0.1.21.dev0, and AMDMIGraphX rocm-10.0, with emphasis on manylinux wheels, gfx targets, compiler options, dynamic slicing, pointwise fusion, and GPU backend work. AITER v0.1.21 was the main serving release because it packaged kernels for ROCm/Python combinations used by LLM serving stacks..104
RunanywhereAI shipped RCLI v0.5.2 plus runanywhere-sdks v0.20.30 through v0.20.36, focused on QHexRT, NeuRT, ANE, image embeddings, TTS, OCR, packaging, and release metadata. The SDK train drove downstream Electron, iOS, and CLI fixes..105
SGLang shipped sglang-omni v0.1.4, focused on Qwen3-Omni encoder performance, Qwen3-ASR streaming transcription output, encoder/backlog overlap, and omni-modal serving performance. Core SGLang did not publish a release in the supplied data, but it had heavy merge activity..106
Triton Inference Server shipped v2.72.0 for NGC container 26.08, with release-train, QA, build, and documentation updates. The week’s most useful runtime fix hardened OpenAI-compatible stream cancellation for abandoned or truncated streams..107
Triton language shipped v3.8.0, covering frontend and dialect changes, compiler and backend work, AMD/HIP, NVIDIA, Gluon/layout, kernels, Proton profiling, testing, build infrastructure, and docs. The release paired with active fixes in layout analysis, NVIDIA TMEM scheduling, and ROCm CI..108
try-mirai shipped lalamo v0.16.0, focused on reasoning-effort handling, chat-codec tests, model import specs, attention layer fixes, fused gated attention projections, GPT-OSS MXFP4 layout preservation, and JAX import warnings. Uzu did not release, but it advanced Metal, MoE, energy reporting, build reproducibility, and KV reuse design..109
turboderp-org shipped exllamav3 v1.4.5 and v1.4.6, adding GLM5.3-Flash, Qwen3.8-Flash-Next, MoE MTP performance work, MoE slab allocation, Triton autotuner race mitigation, Unigram tokenizer fixes, and experimental n-gram streaming for Windows. v1.4.5 carried the main model-support payload..110
Osaurus shipped 0.24.0 through 0.24.4, focused on vMLX repins, Qwen Flash Next native runtime work, allocator and cache behavior, agent-loop telemetry, Claude Code CLI integration, model-switch warnings, custom endpoints, and eval scoring. 0.24.4 was the latest release and continued the cache/MTP stabilization theme..111
Rapid-MLX shipped v0.13.1 through v0.13.3 plus matching rapid-mac releases, focused on Qwen Flash Next, GLM-5.3-Flash production serving, Desktop recovery, hybrid multimodal prefix reuse, offline speech runtime pulls, and Mac memory safety. v0.13.3 was the most complete release because it promoted GLM serving and Desktop reliability together..112
vLLM-project shipped vLLM-Omni v0.28.0 and v0.28.0rc1, with hundreds of merged changes across MiniMax-H3 GPU/NPU serving, scheduler-managed paged KV for diffusion, full-duplex speech, realtime speech, multimodal CI, and rebasing onto vLLM 0.28. Core vLLM and vLLM-Gaudi had no release in the supplied data, but both had heavy compatibility and serving work..113