Open-source tools for running AI models locally in 2026
30 open-source projects, ranked by GitHub stars and how recently each was worked on. Ollama leads with 186k stars, ahead of Llama.cpp. 25 had code pushed in the last three months. Most common licenses: Apache-2.0 (18), MIT (11) and AGPL-3.0 (1). Commercial tools for running AI models locally are listed separately.
New this season
-
1
Kaggle-tpu-lab ARahim3
Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.
Template or starter Experimental You want to experiment with big open models in a coding agent without paying for GPUs.
β 758 MIT Python Active, pushed 27 days ago -
2
Resin ekzhang
𧱠Experimental library for building inference engines
Building block Experimental You are building or researching a custom inference engine for specific hardware.
β 272 Apache-2.0 Python Active, pushed 5 days ago
All open-source tools for running AI models locally, ranked
Complete systems you can adopt 10
-
1
AnythingLLM Mintplex-Labs
Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience
Complete system Usable You want private document chat for a small team without building a RAG stack yourself.
β 68k MIT JavaScript Active, pushed today -
2
LocalAI mudler
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.
Complete system Usable You want a drop-in local replacement for OpenAI's API covering more than just text.
β 50k MIT Go Active, pushed today -
3
RikkaHub rikkahub
RikkaHub is an Android APP that supports for multiple LLM providers.
Complete system Usable You want one Android app for chatting with several AI providers using your own API keys.
β 8.6k AGPL-3.0 Kotlin Active, pushed today -
4
GPUStack gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Complete system Usable An organisation pools its own GPUs and wants a control panel for serving models to internal teams.
β 6k Apache-2.0 Python Active, pushed today -
5
GPT4All nomic-ai
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
Complete system Usable Only if you already use it; newer local-LLM apps are better maintained.
β 77k MIT C++ Quiet, pushed 17 months ago -
6
Dive OpenAgentPlatform
Dive is an open-source MCP Host Desktop Application that seamlessly integrates with any LLMs supporting function calling capabilities. β¨
Complete system Usable You want to use MCP tools with models other than Claude in a desktop chat app.
β 1.8k MIT TypeScript Slowing, pushed 3 months ago -
7
LLM X mrdjohnson
LLMX; Easiest 3rd party Local LLM UI for the web!
Complete system Usable You run Ollama locally and want a no-install chat interface.
β 296 MIT TypeScript Active, pushed 35 days ago -
8
Llama.ui PromptPasture
A minimal interface for AI Companion that runs entirely in your browser.
Complete system Usable You run llama.cpp or vLLM and want a lightweight chat front end.
β 221 MIT TypeScript Active, pushed 8 days ago -
9
Nextjs-vllm-ui yoziru
Fully-featured, beautiful web interface for vLLM - built with NextJS.
Complete system Usable You serve a model with vLLM and want a simple chat UI next to it.
β 185 MIT TypeScript Active, pushed 3 months ago -
10
Faster Chat 1337hero
A blazingly fast, privacy first & OPEN AI Chat Interface
Complete system Experimental You want a fast, self-hosted multi-provider chat UI without Open WebUI's weight.
β 175 MIT JavaScript Active, pushed 9 days ago
Frameworks and platforms to build on 4
-
1
BentoML bentoml
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
Framework or platform Production-ready An ML team needs to serve custom models as production APIs without writing serving infrastructure.
β 8.9k Apache-2.0 Python Active, pushed today -
2
Mosec mosecorg
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
Framework or platform Usable You need to put custom PyTorch or TensorFlow models behind an efficient HTTP API with batching.
β 903 Apache-2.0 Python Active, pushed today -
3
Flama vortico
The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.
Framework or platform Usable You want one framework to expose scikit-learn, PyTorch or LLM models as production APIs with little code.
β 304 Apache-2.0 Python Active, pushed today -
4
Superduper superduper-io
Superduper: End-to-end framework for building custom AI applications and agents.
Framework or platform Experimental Rarely now; consider it only if you already depend on it.
β 5.3k Apache-2.0 Python Quiet, pushed 14 months ago
Building blocks for your own system 10
-
1
Llama.cpp ggml-org
LLM inference in C/C++
Building block Production-ready You need efficient local or edge LLM inference and want the engine many other tools are built on.
β 133k MIT C++ Active, pushed today -
2
vLLM vllm-project
A high-throughput and memory-efficient inference and serving engine for LLMs
Building block Production-ready You serve open-weight models to many users on your own GPUs and need throughput.
β 96k Apache-2.0 Python Active, pushed today -
3
FastDeploy PaddlePaddle
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
Building block Usable You deploy ERNIE or other PaddlePaddle models, especially on Chinese hardware.
β 3.7k Apache-2.0 Python Active, pushed 39 days ago -
4
Chitu thu-pacman
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
Building block Usable You serve large models on a mix of Chinese and NVIDIA hardware.
β 3k Apache-2.0 Python Active, pushed yesterday -
5
Kvcached ovg-project
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
Building block Experimental You run multiple models on shared GPUs with bursty traffic and want better utilisation.
β 2k Apache-2.0 Python Active, pushed today -
6
RTP-LLM alibaba
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
Building block Usable You serve Qwen-family models at scale and want an engine proven inside Alibaba.
β 1.4k Apache-2.0 Python Active, pushed today -
7
PegaInfer pegainfer-project
Pure Rust + CUDA LLM inference engine β no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
Building block Experimental You want a lean, single-binary inference server for supported models on NVIDIA GPUs.
β 730 Apache-2.0 Rust Active, pushed today -
8
Parallax GradientHQ
Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere
Building block Experimental You want to experiment with running large models across several consumer machines.
β 1.4k Apache-2.0 Python Slowing, pushed 4 months ago -
9
ZhiLight zhihu
A highly optimized LLM inference acceleration engine for Llama and its variants.
Building block Experimental You run Llama or DeepSeek models on PCIe consumer or data-centre GPUs and want to benchmark against vLLM.
β 906 Apache-2.0 C++ Slowing, pushed 8 months ago -
10
Resin ekzhang
𧱠Experimental library for building inference engines
Building block Experimental You are building or researching a custom inference engine for specific hardware.
β 272 Apache-2.0 Python Active, pushed 5 days ago
Integrations 1
-
1
vLLM Ascend vllm-project
Community maintained hardware plugin for vLLM on Huawei Ascend
Integration Usable You have Huawei Ascend accelerators and want to use vLLM on them.
β 3k Apache-2.0 Python Active, pushed today
Developer tools 4
-
1
Ollama ollama
Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
Developer tool Production-ready You want to run local LLMs for development, privacy or offline use with minimal setup.
β 186k MIT Go Active, pushed today -
2
OpenLLM bentoml
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
Developer tool Usable You want a quick way to stand up an open model behind an OpenAI-style API.
β 13k Apache-2.0 Python Active, pushed today -
3
InferenceX SemiAnalysisAI
Open Source AI Accelerator Research Platform Standard / εΌζΊζ¨ηη η©ΆεΉ³ε°
Developer tool Usable You compare inference hardware or serving stacks and want reproducible benchmarks.
β 2k Apache-2.0 Python Active, pushed today -
4
ServeGen alibaba
A framework for generating realistic LLM serving workloads
Developer tool Experimental You load-test an LLM serving stack and want traffic patterns closer to production than uniform benchmarks.
β 191 Apache-2.0 Python Active, pushed 2 months ago
Templates and starters 1
-
1
Kaggle-tpu-lab ARahim3
Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.
Template or starter Experimental You want to experiment with big open models in a coding agent without paying for GPUs.
β 758 MIT Python Active, pushed 27 days ago
Compared head to head
Looking for an alternative to
Open-source licenses
- Permissive: 29. Use it anywhere, including in closed products. You keep the copyright notice; Apache-2.0 also grants patent rights.
- Network copyleft: 1. Like GPL, and running a changed version as a web service counts as distributing it, so its users get the source too.
Languages
Python 17, TypeScript 4, C++ 3, Go 2, JavaScript 2, Kotlin 1 and Rust 1.
Made software that belongs here? Add it. Listing is free.