vllm-mlx: an inference server for Apple Silicon

Core facts

Read by a script from the official API, not rewritten by a model.

Repository: https://github.com/waybarrios/vllm-mlx
License: Apache-2.0
Languages: Python
Maintainer: waybarrios
Homepage: https://vllm-mlx.is-a.dev/
Verified: 2026-10-31

What it is

vllm-mlx is a vLLM-style inference server for Apple Silicon Macs. It runs LLMs, vision models, audio, and embeddings on Metal with unified memory, with no conversion step. The project is licensed under Apache 2.0. vllm-mlx is inspired by vLLM and adopts its continuous-batching and paged KV-cache design for Apple Silicon via MLX.

Main features

The SSD-tiered KV cache spills prefix cache to disk for long-context agents, warm prompts preload popular prefixes at startup, and the prefix cache is trie-based and shared across requests.
The OpenAI-compatible API offers chat completions, completions, embeddings, rerank, and responses endpoints, the Anthropic-compatible API supports streaming, tool use, and system prompts, MCP tool calling has 19 parsers (OpenAI, Anthropic, Gemini, Qwen, DeepSeek, Gemma, and more), and structured output uses JSON Schema with lm-format-enforcer.
Reasoning extraction uses Qwen3 and DeepSeek-R1 parsers, speculative decoding is for supported models, and sparse prefill is attention-based for supported model/draft combinations.
Prometheus metrics are exposed through a metrics endpoint with the enable-metrics option.
In its multimodal feature list, the README names Gemma 3, Gemma 4, Qwen3-VL, Pixtral, and Llama vision as the vision models it supports.
In the multimodal feature list, the README gives native TTS as 11 voices and 15+ languages, with Kokoro, Chatterbox, VibeVoice, and VoxCPM named in parentheses.

Getting started

Install openai in the Python client environment first.
After the first response, restart with the continuous batching option to try continuous batching.
Initial model download and loading time depends on the machine and connection.

From the README · bash

python -m pip install 'vllm-mlx==0.4.1'
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --host 127.0.0.1 --port 8000

From the README · python

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
r = client.chat.completions.create(model="mlx-community/Llama-3.2-3B-Instruct-4bit", messages=[{"role": "user", "content": "Hi!"}])
print(r.choices[0].message.content)

From the README · bash

export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claude

From the README · bash

uv tool install 'vllm-mlx==0.4.1'

Who it is for

Native GPU acceleration supports Apple Silicon only (M1, M2, M3, M4, M5), with Metal kernels via MLX. It uses unified memory with no model conversion. The built-in benchmarker runs prompt sweeps with CSV/JSON output.

The README says it was validated with OpenCode, pi, Codex, Claude Code, GitHub Copilot CLI, Cline CLI, and OpenClaw's embedded agent. Rapid-MLX is a community fork of vllm-mlx. The README also states that the coding client validation results apply to the tested client versions and settings, so they describe those versions and settings.

Related tools listed on this site.