py-xiaozhi: multi-modal AI interaction framework

Core facts

Read by a script from the official API, not rewritten by a model.

Repository: https://github.com/huangjunsen0406/py-xiaozhi
License: MIT
Languages: Python
Maintainer: huangjunsen0406
Homepage: https://huangjunsen0406.github.io/py-xiaozhi/
Verified: 2026-11-01

What it is

py-xiaozhi is a multi-modal AI interaction framework built on Python's async architecture. py-xiaozhi supports real-time voice streaming, vision-language tasks, and IoT device control. py-xiaozhi evolved from the xiaozhi-esp32 firmware project. py-xiaozhi is released under the MIT License.

Main features

Multi-modal Vision provides camera capture and vision-language model integration for image understanding and scene perception.
The MCP Tool Ecosystem is a modular JSON-RPC 2.0 tool server covering music player, camera, screenshot, app management, weather, and volume control.
py-xiaozhi offers PySide6 + QML GUI, CLI, and GPIO UI modes, adapting to desktop, headless server, and embedded environments.
Offline wake word uses Sherpa-ONNX based on-device keyword spotting with custom wake word support.
py-xiaozhi provides a GPIO interface for robotics control, hardware actuation, and sensor integration.
WebSocket / MQTT dual protocol communication provides WSS/TLS encrypted transmission and auto-reconnection.

Getting started

Among the basic requirements, the Python version is 3.10 - 3.12.
The operating systems listed in the basic requirements are Windows 10+, macOS 10.15+, and Linux.
The basic requirements call for microphone and speaker devices.
Voice wake-up is an optional feature that requires downloading Sherpa-ONNX speech recognition models.
Camera features are an optional feature that requires a camera device and OpenCV support.
The main branch has the latest code; after each update, manually reinstall pip dependencies to ensure you have new dependencies.

From the README · bash

# Clone project
git clone https://github.com/huangjunsen0406/py-xiaozhi.git
cd py-xiaozhi

# Base install (CLI / GPIO mode)
uv sync                                    # Recommended (uv users)
# or: pip install -e .                    # pip users

# GUI mode (extra: PySide6 + qasync)
uv sync --extra gui                        # Recommended (uv users)
# or: pip install -e '.[gui]'             # pip users

# Full development environment (GUI + test / packaging tools)
uv sync --extra gui --group dev

# Code formatting
./format_code.sh

# Run program - GUI mode (default; requires gui extra)
python main.py

# Run program - CLI mode (base install is enough)
python main.py --mode cli

# Specify communication protocol
python main.py --protocol websocket  # WebSocket (default)
python main.py --protocol mqtt       # MQTT protocol

From the README · bash

# Installed command (GUI still needs the gui extra)
xiaozhi --mode cli
# Equivalent module entry point
python -m src --mode cli

# Full regression suite, including Qt model tests without a display
uv run --extra gui --extra tui python -m pytest -q -W error::RuntimeWarning

# Build sdist, then build the wheel from that sdist
uvx --from build pyproject-build
# Install the wheel into a temporary venv outside the checkout and verify it
python scripts/check_wheel.py dist/*.whl

Who it is for

py-xiaozhi is deployable across Windows, macOS, Linux desktops, and ARM embedded platforms (Raspberry Pi, Horizon Robotics RDK, Jetson Nano). Cross-platform deployment covers Windows 10+, macOS 10.15+, and Linux (x86_64 and ARM). The recommended configuration lists memory of at least 4GB RAM, with 8GB or more recommended.

The recommended configuration lists storage of at least 2GB available disk space for model files and cache. The basic requirements list a stable internet connection for AI services and online features. The event-driven architecture is based on the asyncio asynchronous event loop.

The layered design separates the application layer, protocol layer, and UI layer. Dependency injection manages component lifecycle via a bootstrap container. The plugin system loads audio, UI, MCP tools, and other components.

The configuration system provides hierarchical configuration, dot notation access, and dynamic updates. Source checkouts, installed wheels, and frozen builds share the same main entry function. Importing the py-xiaozhi launcher does not parse arguments or initialize configuration and logging.

For wheel installs, FFmpeg is resolved from the system PATH. PRs and branch pushes run Python tests on 3.10/3.12 and installed-wheel checks.

Related tools listed on this site.