Nunchaku: inference engine for 4-bit neural networks
Nunchaku is an inference engine optimized for 4-bit neural networks.
Verified
Core facts
Read by a script from the official API, not rewritten by a model.
- Repository
- https://github.com/nunchux-ai/nunchaku
- License
- Apache-2.0
- Languages
- Python
- Maintainer
- nunchux-ai
- Homepage
- nunchux.ai
- Verified
What it is
Nunchaku is an inference engine optimized for 4-bit neural networks. It implements SVDQuant, a post-training quantization technique for 4-bit weights and activations. The project points to DeepCompressor as the quantization library. The SVDQuant paper is titled SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models.
Main features
The project states that naively running the low-rank branch with rank 32 introduces 57% latency overhead, due to extra reads of 16-bit inputs in Down Projection and extra writes of 16-bit outputs in Up Projection.
Nunchaku optimizes this overhead with kernel fusion.
In the first stage of SVDQuant, both the activation and the weights contain outliers, which makes 4-bit quantization challenging.
In the second stage of SVDQuant, the outliers are migrated from activations to weights.
In SVDQuant, the low-rank branch, which runs at 16-bit precision, alleviates the quantization difficulty.
Getting started
The project's getting-started list includes an installation guide.
The same list also includes a usage tutorial.
The same list names the ComfyUI plugin ComfyUI-nunchaku.
The same list names DeepCompressor for custom model quantization.
The same list also has Gradio demo apps.
The same list likewise includes an API reference.
Who it is for
The project says the official Nunchaku documentation is now live. The project also lists Frequently Asked Questions. A news entry of the project says it supports ComfyUI.
A news entry says the Python backend is now available. The project reports that Qwen-Image now supports asynchronous offloading, reducing VRAM usage to as little as 3 GiB with no performance loss. A news entry lists support for FLUX.1-Kontext.
Nunchaku FLUX models can now be loaded as a single file. The project reports that a 4-bit text encoder and per-layer CPU offloading reduce FLUX's minimum memory requirement to just 4 GiB while maintaining a 2–3× speedup. A news entry says NVFP4 precision is supported on NVIDIA RTX 5090.
A news entry says 4-bit SANA is supported. SVDQuant has been accepted to ICLR 2025. The project reports that on 12B FLUX.1-dev, Nunchaku achieves 3.6× memory reduction compared to the BF16 model.
The project reports that with Nunchaku, its INT4 model runs 3.0× faster than the NF4 W4A16 baseline on both desktop and laptop NVIDIA RTX 4090 GPUs. The project reports that its NVFP4 model is 3.1× faster than both BF16 and NF4 on the RTX 5090 GPU. The project reports that SVDQuant reduces the 12B FLUX.1 model size by 3.6× and cuts the 16-bit model's memory usage by 3.5×.
The project's safety checker is borrowed from hart. Nunchaku is inspired by many open-source libraries.
Related tools
Related tools listed on this site.
updream
Described on its site as an AI video creation product developed in-house by Bilibili for its uploaders, stressing a lightweight and intelligent creation experience.
抠抠图
Koukoutu is an image-processing tool platform. Its official site describes batch and interactive background removal, image generation/editing, product images, upscaling and restoration, plus image translation, layer separation and vectorization.
Recraft AI
Recraft is an image-model and design platform for generating and editing images, vectors, custom styles and mockup assets.
QuiverAI
QuiverAI generates editable SVG vector graphics, describing text/image-to-SVG, editing and animation with platform and developer entry points.
腾讯混元大模型
An official Tencent Hunyuan project entry. This source covers Hunyuan-Large MoE models, pretrained and instruction variants, and inference and fine-tuning guidance.
讯飞星辰MaaS
The public primary description positions iFLYTEK Xingchen MaaS as a developer-facing model fine-tuning service, covering data processing, model optimization, evaluation and deployment/hosting. This review did not run those console workflows.
LiblibAI·哩布哩布AI
A creative community centered on AI image models and workflows, with model sharing, online generation, and training entry points.
RunningHub
An online creative platform that brings together ComfyUI workflows, AI apps, and a canvas for generating and processing images and videos.
LiblibAI
AI image creation platform and model-sharing community whose site lists online drawing tools, downloadable models and online model training.
Genmo
The site says Genmo is developing open video world models; Mochi 1 is its open-source text-to-video model, usable online in the Playground or run locally via the open-source repository or ComfyUI.
Sources
This page is written from the official sources below. Open them to check the original text.
- GitHub READMEhttps://github.com/nunchux-ai/nunchakuFetched
Verified ·
This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.