Tracked Repositories

HuggingFace Transformers — state-of-the-art NLP/ML model library (~140K stars)

HuggingFace Diffusers — diffusion model inference & training (Stable Diffusion, Flux, etc.)

Minimalist Rust ML framework for inference — targets browser WASM and GPU, zero Python dependency

HuggingFace TGI — LLM serving (archived March 2026, read-only)

One-command HuggingFace Hub → OpenVINO IR export; INT4/INT8 quantization via NNCF for Intel NPU deployment

Industry-standard deep learning framework with XLA compilation backend

TensorFlow Serving — high-performance gRPC/REST serving for TF models (multi-version, canary, batching)

TensorFlow Lite for microcontrollers and embedded devices

High-performance LLM inference in C/C++ (CPU + GPU)

High-performance Whisper speech recognition in C/C++

User-friendly local LLM runner built on llama.cpp (~167K stars)

Self-hosted ChatGPT alternative with built-in RAG, offline-capable (~104K stars)

Primary ML framework; torch.compile + AOTInductor for production inference optimization

PyTorch's portable execution framework for on-device inference

TorchServe — production PyTorch model serving (archived August 2025)

Reference inference code for DeepSeek-V3 (671B MoE); includes FP8 training framework

Most widely adopted open-source LLM serving engine; PagedAttention, continuous batching

vLLM omni-modality inference (audio, vision, speech)

vLLM community plugin for Intel Gaudi accelerators

Cross-platform ML pipeline framework (vision, audio, NLP)

AI Edge model gallery

LiteRT for language model inference

Official Gemma model cookbook — recipes, fine-tuning, deployment guides

Google's Lite Runtime (successor to TensorFlow Lite)

Sample apps using MediaPipe

Highly optimized neural network operators library (ARM, x86, WASM)

Model visualization and exploration tool

LiteRT integration with PyTorch

Sample code for LiteRT

Python API for Coral Edge TPU inference (archived)

Quantization tooling for AI Edge models

C++ API for Coral Edge TPU inference (archived)

Web samples for MediaPipe

Command-line tooling for LiteRT

Sample models for AI Edge

Evaluation tooling for AI Edge models

Google AI Edge documentation site

Desktop AI app + SDK for running LLMs locally (~73K stars)

DwarfStar native DeepSeek V4 inference engine for Metal, CUDA, and ROCm

MLC's universal LLM deployment engine (multi-backend)

Smartphone/PC LLM inference exploiting activation sparsity (PowerInfer-2: Qualcomm NPU, Mixtral MoE up to 47B); org transferred from SJTU-IPADS

Tile-based ML language and compiler

Tile-based runtime for ultra-low-latency LLM inference

Community on-device LLM project

vLLM-style inference on Apple silicon via MLX

OpenVINO-based OpenAI-compatible inference server for Intel CPU/GPU/NPU

Array framework for ML on Apple silicon (Python)

Example models and applications using MLX

Reverse-engineered Apple Neural Engine (ANE) — hardware ops, memory layout, firmware interactions

LLM inference and fine-tuning with MLX

Tools for converting & running models with Core ML

Example apps using MLX Swift

Model export recipes, Python primitives, and Swift runtime utilities for on-device AI (Core AI)

Swift bindings for MLX

LLM inference in Swift via MLX

Efficient data loading for MLX

C bindings for MLX

Bridges PyTorch and Core AI — convert models to Core AI IR, composite ops, custom lowerings, inline Metal kernels

PyTorch model compression and optimizations for deployment via Core AI on Apple silicon

Unified OpenAI-compatible proxy for 100+ LLM providers (vLLM, Ollama, Bedrock, Azure, etc.)

Free, open-source OpenAI drop-in replacement — runs locally, no GPU required (~36K stars)

Microsoft's cross-platform, high-performance ONNX inference engine

Open Neural Network Exchange format specification

Microsoft on-device / local model runtime

Model optimization tool (HF → ONNX → quantize → NPU deployment); used under the hood by Microsoft Foundry Local

Run LLMs distributed across heterogeneous devices (Mac, iPhone, etc.)

Gradio web UI for LLMs — multi-backend (llama.cpp, ExLlamaV2, transformers) (~43K stars)

Distributed AI compute engine; Ray Serve handles online and async batch inference (~39K stars)

Microsoft DeepSpeed — distributed training and inference (ZeRO, MII, FastGen)

LLM serving framework and home of Chatbot Arena (~37K stars)

NVIDIA's optimized LLM inference library (GPU)

NVIDIA's high-performance deep learning inference SDK (GPU)

CUDA C++ templates for high-performance matrix-multiply (GEMM) and convolution kernels

C++ LLM/VLM inference runtime for Jetson and NVIDIA edge devices

High-throughput LLM/VLM serving with RadixAttention and structured generation

SGLang serving for TTS, ASR, speech, and omni-modal models

Composable NumPy transformations (JIT, grad, vmap) compiled via XLA to GPUs and TPUs — primary DeepMind research/production runtime

Modular Platform monorepo — MAX inference server/framework + Mojo programming language for portable, high-performance AI on CPUs and GPUs

High-performance neural network inference for mobile (Android/iOS)

Tencent Neural Network — mobile and edge inference

CPU-GPU hybrid inference; runs DeepSeek 671B on 14GB VRAM + 382GB DRAM with massive speedup over llama.cpp

Kimi serving platform — KV-cache and disaggregated LLM serving

Single-file LLM executables via Cosmopolitan Libc — zero install, all platforms (~21K stars)

Official FlashAttention — fast, memory-efficient exact attention (FA-2/FA-3) kernels for GPUs

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

Run open-source LLMs as OpenAI-compatible API endpoints

Unified serving framework: real-time APIs, task queues, batching, multi-model chains

Python-like GPU kernel language used by vLLM FlashAttention and PyTorch inductor

Tiny on-device foundation model for phones, wearables, and robots

Cactus core edge inference framework

React Native bindings for Cactus

Kotlin/Android bindings for Cactus

Flutter bindings for Cactus

Demo chat app using Cactus

High-performance LLM inference in web browsers via WebGPU

Alibaba's neural network inference framework for mobile & edge

Alibaba high-performance LLM inference engine

ONNX-based runtime for ASR, TTS, VAD, and keyword spotting

Audio models (TTS, ASR) with MLX

Vision-language models on Apple silicon via MLX

Swift audio inference using MLX

Text embedding models with MLX

Video model inference with MLX

Apache TVM ML compiler — auto-tunes models for any hardware target

Apache TVM Foreign Function Interface for deep learning compilation

On-device speech AI for Apple Silicon (WhisperKit sibling)

On-device Whisper inference for Apple platforms (Swift)

Python tooling for WhisperKit model optimization

On-device AI benchmarking framework

Swift playground for ArgMax SDK

Intel's toolkit for optimizing & deploying deep learning on Intel hardware

Neural Network Compression Framework — quantization, pruning, sparsity for OpenVINO

OpenVINO GenAI — generative AI layer with speculative decoding & KV-cache opt

Lemonade SDK — high-level, multi-vendor LLM inference SDK (OGA + llama.cpp), OpenAI-compatible server mode

AMD-backed Linux NPU runtime, used alongside Lemonade Server on Ryzen AI XDNA 2 (moved to ROCm)

Xilinx/AMD AI Engine dev stack — source of the Vitis AI Execution Provider for XDNA

Open-source RAG/chat/agent app for Ryzen AI NPU, built on Lemonade SDK

Official Ryzen AI Software stack — Vitis AI EP, quantization, LLM deployment; contains an experimental XDNA-NPU llama.cpp fork

Ingests any PyTorch model, optimizes, benchmarks/deploys across hardware targets incl. XDNA NPU

AMD's quantization toolkit (INT4/INT8 PTQ + QAT) for Ryzen AI NPU deployment

RunAnywhere SDKs for on-device inference deployment

RunAnywhere CLI tool

Swift iOS/macOS starter example for the RunAnywhere SDK

Android starter example for the RunAnywhere SDK

Browser WASM reference app for the RunAnywhere Web SDK

React Native starter app using the RunAnywhere SDK

Flutter starter app for the RunAnywhere on-device SDK

Electron desktop app for on-device LLM, VLM, STT, TTS, VAD, and RAG

Intel IPEX-LLM — local LLM acceleration on Intel hardware (archived Jan 2026, read-only)

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity

NVIDIA Triton — production multi-model inference server (HTTP/gRPC, multi-backend)

Official minimal inference library for all Mistral models (7B, Mixtral, Pixtral)

On-device LLM/VLM SDK for Snapdragon NPU, GPU, and CPU (formerly Nexa AI / nexa-sdk; acquired by Qualcomm)

State-of-the-art ML models optimized for Qualcomm Snapdragon NPU/DSP/QNN deployment

Sample apps and tutorials for deploying models on Qualcomm hardware (TFLite, ONNX, QNN)

DNN inference library & tutorials for NVIDIA Jetson

Datacenter-scale distributed inference serving framework (Rust + Python, disaggregated prefill/decode, engine-agnostic)

High-throughput LLM serving with TurboMind engine (C++/CUDA)

Native macOS AI agent harness in Swift — any model, persistent memory, autonomous execution, MCP server, MLX + Apple Neural Engine, fully offline

Lightweight inference engine for mobile & embedded from PaddlePaddle

High-performance GPU kernel library for LLM serving — attention, sampling, and KV-cache primitives

High-performance EXL2-quantized inference for consumer NVIDIA GPUs

Successor to ExLlamaV2 — quantized local LLM inference on consumer NVIDIA GPUs

Fast C++ inference for Transformer models; INT8/INT16 CPU quantization, multi-platform

Compiler for JAX, TF, PyTorch targeting GPU, TPU, and CPU from a unified IR

Lightweight, high-throughput Python-based LLM inference and serving framework

The fastest local AI engine for Apple Silicon — 4.2x faster than Ollama, 0.08s cached TTFT, 100% tool calling, drop-in OpenAI-compatible replacement

Multi-LoRA inference server — serve thousands of fine-tuned adapters on a single GPU

Examples, tutorials and apps for Liquid AI LFM + LEAP SDK

Speech-to-Speech audio models by Liquid AI

Minimal fine-tuning repo for LFM2, fully open-source

Example apps for LeapSDK

Liquid AI documentation

On-device audio inference framework

Fluid Inference core runtime

Rust text processing library for inference

Rust-based deep learning compiler with a small static graph IR for fast, portable inference (CUDA, Metal, CPU)

Mirai's on-device inference runtime

Mirai's LLaMA-based on-device model

Multimodal LLM inference framework for mobile & edge

AI Tensor Engine for ROCm — centralized repo for high-perf AI operators on AMD Instinct GPUs

AMD's graph inference engine for MI-series GPUs

ROCm fork of FlashAttention with Composable Kernel (CK) and Triton backends

AiTer Optimized Model — lightweight vLLM-like server built on AITER kernels for ROCm

ARM Neural Network SDK for ARM & Mali devices

iOS/macOS workbench for Apple's Foundation Models framework — recipes, guided labs, prompt playground, AFM CLI, FMFBench eval suite, and reusable Swift packages for on-device Apple Intelligence

NimbleEdge's deliteAI on-device inference framework

NimbleEdge fork of ExecuTorch with edge optimizations

A simple, fast and robust program-aware agentic inference system

Picovoice's on-device LLM inference engine

MLange sample applications

MLange extension library

iOS framework for MLange

iOS extension framework for MLange

MLange SDK documentation