TL;DR
- Decision endpoints reached local runtimes: Ollama and LocalAI added System One style decision APIs, turning local servers into structured classifiers alongside chat endpoints.1
- Omni-modal serving moved into the main path: vLLM, SGLang, MLX projects, and WebLLM pushed speech, TTS, video, and multimodal cache handling into normal serving loops.2
- Kernel work followed new hardware: FlashInfer, FlashAttention, TensorRT-LLM, ROCm, TileLang, and Triton focused on Blackwell, Rubin, ROCm, Ascend, and Apple Silicon paths.3
- KV-cache became infrastructure: Mooncake, ai-dynamo, vLLM, Ray Serve, and exo all worked on routing, cache transfer, cancellation, and failure semantics.4
- Security moved closer to serving code: LightLLM, vLLM, TensorRT, PowerInfer, and sherpa-onnx all surfaced loader, parser, template, or remote-service hardening work.5
This Week in Inference
The supplied market search did not surface a verified external model launch, but repo activity showed model enablement racing through inference stacks. AMD FastFlowLM added Qwen-family NPU speculative decoding support for Ryzen AI systems, while TensorRT-LLM added GLM, MiniMax, and Nemotron serving paths in its main runtime workstream FastFlowLM.6 Hugging Face Transformers added GTE, Snowflake GTE, and Nemotron Omni support, and Diffusers added Qwen-Image training and attention/cache fixes Transformers.7
The clearest technique trend was memory and cache control. FlashAttention added a dense and sparse MLA path for Blackwell SM100 that targets decode and prefill speed, while FlashInfer expanded CAKE-generated DeepGEMM and Kimi-K3 kernels for Blackwell and Rubin-class serving FlashAttention.8 vLLM, SGLang, ai-dynamo, Mooncake, Ray Serve, and exo all worked on the same layer from different angles: prefix cache safety, KV movement, cancellation, scheduler accounting, and disaggregated serving ai-dynamo.9
Hardware work split across cloud and edge. Google LiteRT pushed Gemma tensor-core paths, WebGPU coverage, and XNNPACK hardening; Apple MLX shipped training and attention fixes; Qualcomm GenieX improved Windows-on-Snapdragon and Hexagon runtime paths; ROCm advanced AITER, ATOM, and MIGraphX for DeepSeek, Kimi, Qwen, GLM, and MiniMax workloads LiteRT.10 No funding, acquisition, or cloud-price signal appeared in the supplied weekly market data, so the best industry read came from code: vendors and open runtimes both chased portability across NVIDIA, AMD, Gaudi, NPUs, Apple Silicon, and browser GPUs.
Top Stories
Ollama turns local inference into decision serving
Ollama added POST /v1/systemone for local decision models that return choices, probabilities, yes/no likelihoods, and rubric scores instead of free-text answers Ollama.1 The team then added model capability declarations and docs for choice, yes/no, and scoring workflows Ollama.11 This puts Ollama in the same product lane as LocalAI and Qualcomm GenieX, where a local model server can act as a structured decision layer for apps, agents, and eval harnesses LocalAI.12
vLLM and SGLang push omni-modal serving into production
vLLM-omni shipped a full-duplex serving framework with native cross-stage KV and multimodal payload transfer, plus realtime audio, video, and diffusion serving paths vLLM-omni.2 SGLang-omni moved in the same direction with Qwen3-Omni talker work, OmniTyper streaming ASR, XPU install paths, and speech serving fixes SGLang-omni.13 The shared signal is that text-only OpenAI-compatible serving no longer covers the main workload shape for fast-moving inference teams.
FlashInfer, FlashAttention, and TensorRT-LLM chase the next GPU curve
FlashInfer added a CAKE-generated DeepGEMM family for Blackwell, with bindings, catalogs, benchmarks, and tests for sparse, dense, MoE, FP8, FP4, and mixed GEMM paths FlashInfer.3 FlashAttention merged SM100 one-CTA MLA kernels for dense decode and sparse top-k acceleration, with the authors reporting large decode and sparse-prefill gains over the older path FlashAttention.8 TensorRT-LLM added Rubin FMHA cubins, VisualGen sparse attention, GLM support, MiniMax NVFP4 KV storage, and model-specific serving extensions TensorRT-LLM.14
Edge runtimes harden while adding model reach
Google LiteRT worked on Gemma tensor-core paths, WebGPU Tensor API coverage, quantizer pieces, and LiteRT-LM C API hardening LiteRT.10 Meta ExecuTorch hardened schema rejection, metadata bounds, tensor transfer validation, memory planning, and MLX backend behavior for on-device inference ExecuTorch.15 Apple MLX added SDPA VJP, Metal kernel fixes, MoE gather improvements, and Swift hotfixes, while Microsoft ONNX Runtime advanced CUDA, WebGPU, CPU, CoreML, and QMoE execution-provider work MLX.16
KV-cache is now the serving control plane
Mooncake added embedded Master support, segment status APIs, tenant-first metadata, managed weight revisions, Transfer Engine adapters, and Conductor APIs for KV and weight movement Mooncake.4 ai-dynamo tightened TensorRT-LLM, vLLM, SGLang, KV-router, IPv6, cancellation, and disaggregated serving paths in one dense reliability cycle ai-dynamo.17 exo had no merged code, but its open PR queue targeted event-log growth, master failover, MLX transport aborts, request hangs, and prefix-cache memory pressure in distributed local inference exo.18
Deeper Dive
Everything below is for readers who want the full picture. Feel free to scroll.
Code Changes by Category
Cloud & Datacenter Serving
vLLM focused on backend breadth and safety: chat-template DoS mitigation, shared-memory multimodal cache authentication, ROCm/AITER paths, Gaudi integration, CPU and XPU kernels, and long-context cache correctness all moved in core and satellite repos vLLM.19 SGLang pushed scheduler correctness, Rust frontend extraction, speculative decoding, MoE placement, router validation, Qwen-family support, and omni-modal graph capture SGLang.20 TensorRT-LLM removed AutoDeploy from the tree, added Rubin FMHA artifacts, VisualGen sparse attention, GLM and MiniMax paths, and a per-model serving extension registry TensorRT-LLM.21
Ray Serve moved its LLM integration toward vLLM, added LoRA coverage for direct-streaming and prefill/decode-disaggregated deployments, and fixed ingress, routing, placement-group cleanup, and pending-request handling Ray.22 Triton published its NGC container release and merged compatibility docs, while open fixes targeted gRPC trace propagation, HTTP request-size enforcement, and S3-compatible storage addressing Triton Server.23 LiteLLM hardened Redis and S3 batching, service-tier billing, spend retention, Langfuse tracing, OTel filtering, Agent 365 guardrails, and MCP tool scanning LiteLLM.24
Local LLM Runtimes
llama.cpp added GLM-5.3-Flash and GLM5-Next support, CPU tiled k-quant mul_mat, Vulkan RDNA cooperative matrix work, CUDA and HIP fixes, GGUF safety checks, server UI model-download flows, and continued migration toward llama_batch_ext llama.cpp.25 Ollama added System One decision serving, explicit model capabilities, MLX and llama.cpp bumps, higher web-search limits, and MLX pull reliability fixes Ollama.1 LocalAI added decision APIs, vLLM-CPP gallery entries, distributed failover, audio backend refreshes, SGLang reasoning controls, and safer model deletion paths LocalAI.26
ExLlamaV3 added experimental Turing support, MiMo-V2.6 paths, DFlash and MTP fixes, autosplit improvements, CPU MoE offload, and smaller Windows wheels ExLlamaV3.27 llamafile synced to llama.cpp b11100 and refreshed downstream patch handling, docs, build rules, and ggml backend patches llamafile.28 Open WebUI merged a large RAG and API correctness wave around Milvus hybrid search, Qdrant strict mode, provider 429 propagation, Anthropic SSE failures, OAuth error handling, and upgrade cache behavior Open WebUI.29
Apple Silicon & MLX Ecosystem
Apple MLX added SDPA VJP, gated-delta-net backward kernels, SDPA/Metal coverage, MoE gather fixes, fused cross entropy work, Array API parity, Python and SDK compatibility, and Swift hotfixes MLX.16 MLX-LM added Ling, DeepSeek, Falcon, Mistral, and MiniMax fixes, while open work targeted quantized attention and fused low-bit decode mlx-lm.30 Blaizzy’s mlx-vlm added DeepSeek V4.1, Qwen3.5 MoE, InternVL, batching, tool-calling, image generation, and multimodal cache fixes mlx-vlm.31
oMLX shipped a major Apple serving release with Qwen-family Flash-Next prefill, Lightning MTP reliability, prefix-cache repair, vision-feature caches, and cluster/admin improvements oMLX.32 Rapid-MLX added supervised Computer Use, QuickSilver GLM support, Laya and CLM System One, serving recovery, endpoint setup, strict structured output, and Desktop integration Rapid-MLX.33 Osaurus tightened vMLX cache correctness, local runtime pins, agent orchestration, phone continuity, and document-editing tools for a native macOS agent harness Osaurus.34
Mobile & Edge Frameworks
Google LiteRT added Gemma tensor-core work, WebGPU Tensor API coverage, INT4 quantizer pieces, LiteRT-LM API hardening, and sample coverage across iOS, Android, Web, Swift, and Kotlin LiteRT.10 ExecuTorch hardened native/PTN runtime metadata, serialization, graph validation, memory planning, MLX backend behavior, Qualcomm QNN flows, Arm/VGF packaging, and mobile CI ExecuTorch.35 Qualcomm GenieX improved MiniCPM tool-call parsing, benchmark timing, VLM mmproj selection, QCS defaults, OpenCL workarounds, Windows ARM CI, and signed Hexagon catalog support GenieX.36
Cactus added relative-position attention Metal kernels and Parakeet audio support, while Needle hardened engine publishing and temporal grounding Cactus.37 FluidAudio shipped Parakeet Redux and Ultra model variants, ANE/Nemotron fixes, LocalVQE beta runtime, TTS text normalization, and privacy controls FluidAudio.38 ncnn optimized Vulkan cooperative-matrix GEMM and SDPA paths, then opened hardening work for Concat shape validation and binary parameter loading ncnn.39
Compilers, Runtimes & Graph Engines
Apache TVM modernized TVMScript and TIRx parsing, delegated LLVMModule execution to an external ORCJIT addon, and fixed CUDA, Vulkan, NVPTX, Metal, WebGPU, Relax, and ONNX paths TVM.40 OpenXLA advanced GPU dynamic-slice handling, CUDA graph APIs, fusion autotuning, collectives, ROCm cost models, CPU YNNPACK fusion, PJRT/IFRT reliability, StableHLO import/export, and CI OpenXLA.41 Triton added Gluon/TMA cache policies, chained comparisons, integer clamp lowering, NVIDIA backend fixes, AMD gfx1250 and CDNA work, and toolchain updates for Python and LLVM Triton.42
OpenVINO fixed GPU NaN semantics, FlashAttention/PagedAttention paths, SDPA, KV-cache quantization, NPUW MoE behavior, GGUF import, Q1_0 quantization, PyTorch/ONNX conversion, and GenAI LTX2 and DeepSpec DFlash support OpenVINO.43 ONNX Runtime advanced CUDA/QMoE/decode paths, WebGPU kernels, CoreML factory support, CPU kernels, graph correctness, packaging, and ONNX plumbing ONNX Runtime.44 MIGraphX added MoE/QMoE parsing, GQA handling, int4/QDQ support, GPU JIT gridsample, rocMLIR integration, and fusion correctness MIGraphX.45
Models, Quantization & Optimization
FlashInfer expanded Blackwell CAKE-generated kernels, Kimi-K3 NVFP4 and MXFP4 paths, Rubin support, sparse attention, MoE, sampling, quantization, and release hardening FlashInfer.3 FlashAttention added SM100 one-CTA MLA kernels, SplitKV work, sparse/top-k acceleration, ROCm/CK fixes, and Windows portability FlashAttention.8 ROCm AITER added gfx950 FP8 prefill skip-mask gains, Qwen/Kimi gated-delta work, SonicMoE, Kimi merged-MoE, MXFP8/MXFP4 configs, and MHA/MLA paths AITER.46
Hugging Face Transformers added GTE, Snowflake GTE, Nemotron Omni, multimodal generation fixes, MoE router-logit handling, AMD, TPU, MPS, and backend work Transformers.7 Diffusers added Qwen-Image LoRA training, SAGE attention updates, Cosmos SeaCache, MiniMax fixes, and video/VAE cleanup Diffusers.47 Intel Neural Compressor shipped NVFP4, Keras/JAX selective quantization, PyTorch model-free quantization, Gaudi fused MoE bias, and a vLLM QDQ plugin Neural Compressor.48
Other Notable Changes
Microsoft Foundry Local improved BYO ONNX Runtime GenAI model use, Java SDK ASR, Responses behavior, token budgets, disconnect cancellation, Qwen tool-call enums, and SDK/security polish Foundry Local.49 TensorFlow cut a core release candidate, shipped TensorFlow Serving, hardened XLA/GPU/StableHLO/PJRT paths, and moved TFLite Micro toward standalone ownership TensorFlow.50 RunanywhereAI landed its Rust Wally CLI port, Linux ARM64 kit support, local coding harness restoration, and desktop SDK rebuilds Wally.51
Community Pulse
Security reports clustered near serving surfaces. LightLLM users filed unauthenticated RCE, SSRF, NCCL transfer, and RL control-plane exposure reports LightLLM.5 PowerInfer received GGUF loader memory-safety reports and an open tensor-bounds fix PowerInfer.52 sherpa-onnx users reported heap-buffer-overflow risks from malformed model metadata and CUDA VRAM growth in TTS services sherpa-onnx.53
Context-window and cache transparency came up across projects. OpenLLM users asked for effective served context length in /v1/models, and LocalAI users raised silent context truncation concerns OpenLLM.54 Ollama users reported CUDA hangs, GPU overhead behavior, Vulkan detection crashes, and proxy pull failures Ollama.55 SGLang users filed precise GLM, GDN, Mamba, detokenizer, and launcher issues that often pointed directly to suspected regressions SGLang.56
Community Debates
Apache TVM closed a proposal to restore a standalone C++ Relax-VM inference runner because maintainers want TVM core to stay lightweight and avoid carrying application-level runners in the main repo TVM proposal.57 That debate captures a broader compiler trend: runtime hooks and FFI move forward, while product wrappers move out of core.
FlashInfer’s generated-kernel scale triggered maintainability pressure after users opened a tracker criticizing millions of generated CAKE lines, near-duplicates, unreachable instances, and per-architecture copies FlashInfer issue.58 The project kept landing generated Blackwell/Rubin kernels, but the conversation shows the cost of chasing many model and hardware shapes at once.
Ollama closed MLX !cgo stubs after the author concluded the patch risked a permanent shadow API and probably reflected a build-environment mismatch Ollama PR.59 The maintainers kept the build surface tighter instead of adding compatibility stubs for a narrow path.
Google XNNPACK rejected zeroing intentionally unused bias lanes because maintainers wanted memory sanitizers to catch real reads rather than hide them XNNPACK PR.60 That thread shows a hard line on correctness tooling even when defensive zeroing would make a report disappear.
Worth Watching
Decision APIs are spreading. Ollama, LocalAI, Qualcomm GenieX, Rapid-MLX, and vLLM community work all point toward local models as choice, scoring, and routing engines, not only chat completions Ollama.1
Full-duplex and speech serving will keep pulling server stacks away from text-only abstractions. vLLM-omni and SGLang-omni now carry enough TTS, ASR, realtime, and audio-model work to force new scheduling and cache paths vLLM-omni.2
Blackwell and ROCm kernel work will keep fragmenting the serving backend layer. FlashInfer, FlashAttention, TensorRT-LLM, ROCm AITER, Triton, TileLang, and Modular all landed architecture-specific work, and each stack now needs its own validation loop Triton.61
Browser and WebGPU inference moved beyond demos. WebLLM added manifest-backed audio input, ONNX Runtime WebGPU work expanded, Google WebGPU LiteRT/XNNPACK work continued, and MediaPipe web samples defaulted more demos to GPU WebLLM.62
Major Releases
AMD Ryzen AI shipped FastFlowLM v1.0.7, Lemonade v2026.40.0 plus a v2026.41.0 candidate, and Quark v0.13, with the week focused on Qwen3.8-MTP, NPU speculative decoding, Lemonade runtime polish, and quantization toolkit compatibility. The most visible release was FastFlowLM v1.0.7, which added qwen3.8-mtp:27b and packaged the stack for NPU speculative decoding.63
Apache shipped TVM v0.27.0 and v0.27.0.post1, with the main line focused on TVMScript/TIRx parser modernization, CUDA low-precision cluster kernels, and runtime cleanup. The post release targeted TIRx/CUDA low-precision, tensor-map, and cluster-kernel fixes.64
Apple shipped coreai-optimization v0.3.0, coreai-torch v0.4.3, MLX v0.32.3, mlx-swift 0.32.2 and 0.32.3, and mlx-swift-lm 3.32.3. The dominant theme was runtime alignment across Core AI and MLX, with MLX v0.32.3 carrying scan, sort, variance, CI, extension, and gather_qmm fixes.65
BerriAI shipped a rapid LiteLLM release train from v1.98.1 through v1.105.0-dev.1, with release excerpts centered on Docker signature verification while merged code focused on gateway reliability, spend governance, tracing, and guardrails. The newest development tag documents the signed-image verification path.66
Blaizzy shipped mlx-audio v0.5.6 and v0.5.7 plus mlx-vlm v0.7.3 and v0.7.4, with multimodal MLX support, audio loading fixes, model expansion, and image-generation controls as the common thread. mlx-vlm v0.7.4 added multi-image num_images generation and MiMo audio server-thread fixes.67
Cactus Compute shipped Cactus v2.2.1 and v2.2.2, first expanding Parakeet capture/model support and then adding relative-position attention kernels and bindings. The v2.2.2 release carried the relative-position attention path.68
Dao-AILab shipped FlashAttention fa4-v4.0.0.beta33, with FA4 sparse MLA training/backward accuracy and stability as the release focus. The prerelease highlights fp32 normalization, exact running softmax max, dpsum, and recompute-P changes.69
FlashInfer shipped v0.7.0.post1 and v0.7.1rc1, with the newer prerelease focused on copy-engine PCIe IPC all-reduce, SM100 NVFP4 decode, SVDQuant GEMM, and MoE routing fixes. The v0.7.1rc1 release also ties into the project’s Kimi K3 and NVFP4 serving push.70
FluidInference shipped FluidAudio v0.17.2, v0.17.3, and v0.17.4, with Nemotron ANE stability and Parakeet Redux/Ultra model variants as the through line. The v0.17.4 release fixed Nemotron 3 diarization on M3-class Macs through re-exported monolithic bundles.71
ggml published many llama.cpp build tags while whisper.cpp had no release, with the build train covering CUDA Conv3D, Vulkan RDNA cooperative-matrix matmul, NVFP4 precision policy, CPU tiled k-quant matmul, GLM-5.3-Flash, and batch_ext migration. The later build tag reflects the continued batch_ext and graph-shape checks.72
Hugging Face shipped Transformers v5.18.0, with the release notes focused on Nemotron 3 Diarization for streaming and offline speaker diarization. The broader org week also included Diffusers, Candle, and Optimum Intel work, but the release anchor was Transformers.73
Intel shipped Neural Compressor v3.10, focused on experimental PyTorch NVFP4 with E5M3 scaling, selective and mixed quantization for Keras/JAX, model-free PyTorch quantization, GPT-OSS dynamic fused MoE bias on Gaudi, and a vLLM QDQ plugin. This was Intel’s only tracked release this week.48
InternLM shipped lmdeploy v0.18.0, with release notes listing output input logprobs, SM90 quantized GEMM expansion, unified TurboMind linear execution, Ascend GLM 5.2 support, and Qwen3.5 dflash support. The visible weekly code change was the version bump, but the release notes point to a larger serving cycle.74
Microsoft shipped foundry-local v2.1.0 and ONNX v1.23.1, with local model ownership, tool calling, SDK security, and ONNX build fixes as the release themes. The most product-facing release was Foundry Local v2.1.0, which improved bring-your-own ONNX Runtime GenAI model usage.49
Miscellaneous compiler and runtime projects shipped OpenArc v3.0.0 and TileLang v0.1.15, with YAML-first configuration on one side and Ascend 950 plus CUDA warp specialization on the other. TileLang v0.1.15 was the more infrastructure-heavy release, adding Ascend 950, unified block-scaled GEMM, and frontend improvements.75
NVIDIA shipped TensorRT Edge-LLM v0.11.0 and TensorRT-LLM v1.3.0rc29, with the week focused on Edge-LLM wheel installs and IGX Thor support alongside TensorRT-LLM AutoDeploy removal, Qwen3.5 FP8 checkpoints, Gemma4 vision optimization, CuTe DSL MoE, and MiniMax CUDA Graph work. TensorRT-LLM v1.3.0rc29 was the most relevant datacenter serving release.76
Ollama shipped v0.35.0, v0.35.1-rc0, and v0.40.0-rc0, with decision models and MLX-by-default on Apple Silicon as the main release themes. The v0.35.0 release introduced /v1/systemone, Nimble, and Tev1 for local decisions, probabilities, and scores.77
osaurus-ai shipped 0.25.13, 0.25.14, and 0.25.15, with agent descriptions, iOS integrations, phone run resilience, runtime pins, orchestrator audit fixes, and local inference cache safety as the theme. The 0.25.15 release carried orchestrator and phone continuity fixes.78
Qualcomm shipped ai-hub-apps v0.38.0 and GenieX v0.7.1, with sample-app delivery and Windows-on-Snapdragon runtime polish as the focus. GenieX v0.7.1 added a Microsoft-signed Hexagon HTP catalog for llama.cpp 942561, removing certificate import needs on Windows on Snapdragon.79
Rapid-MLX shipped v0.15.1, v0.15.2, v0.15.3, and matching rapid-mac Desktop tags, with first-start diagnostics, Agent connection setup, QuickSilver GLM support, Laya/CLM System One, and supervised Computer Use as the product arc. The v0.15.3 CLI/server release added opt-in supervised Computer Use and local API docs.80
ROCm shipped AITER v0.1.23 and v0.1.24, both bi-weekly releases with manylinux wheels for gfx942;gfx950 across ROCm and Python variants. The newer v0.1.24 tag anchors a week of gfx950, FP8, MLA, MoE, and serving-kernel work.81
RunanywhereAI shipped Wally v0.6.1, v0.7.1, v0.7.2, plus cpp-desktop-v0.20.38 and cpp-desktop-v0.20.38-1, with the Rust CLI port and Linux ARM64 desktop kit support as the release spine. Wally v0.7.2 fixed Debian 12 and Ubuntu 22.04 ARM64 release bottles.82
SGLang shipped sglang-omni v0.1.7 while core sglang had no release in the supplied data. The release focused on Qwen3-Omni talker MoE finalize changes, OmniTyper streaming ASR, OmniTyper Simplified Chinese and no-text-API dictation fixes, and XPU install work.13
TensorFlow shipped TensorFlow Serving 2.21.0 and TensorFlow v2.22.0-rc0, with Serving release maintenance and core framework packaging/API changes as the release themes. TensorFlow v2.22.0-rc0 removed default TensorBoard installation and experimental C filesystem transaction-token APIs.50
Triton Inference Server shipped v2.73.0 for NGC container 26.09, with the visible release body limited to boilerplate in the supplied data. Compatibility docs around vLLM Python and ONNX Runtime moved in the same release window.83
Turboderp shipped ExLlamaV3 v1.5.2 and v1.5.3, focused on MiMoV2, experimental Turing support, transient VRAM/autosplit improvements, smaller wheels, n-gram corpus support, and dynamic draft sizing. The v1.5.3 release carried the smaller-wheel and n-gram work.84
try-mirai shipped Lalamo v0.17.0, with fixes for eps, group behavior, pulling, DFlash2 support, quantization-parameter storage, and model-format alignment with Uzu. The release anchors a week of Apple Metal and runtime-format coordination across Lalamo and Uzu.85
vLLM shipped vllm-omni v0.30.0, with 504 merged changes from 183 contributors and release notes highlighting full-duplex serving, native cross-stage KV and multimodal transfer, streaming video generation, realtime MiniMax-H3 inference on Blackwell and Ascend, and a rebase onto vLLM 0.30.0. This was the week’s largest serving release by scope.2