In-Depth AI Infra: Quantitative Analysis and System Design

A repository titled "In-Depth AI Infra: Quantitative Analysis and System Design"; the manuscript is still a first draft.

Verified

Core facts

Read by a script from the official API, not rewritten by a model.

License
Apache-2.0
Languages
Python
Maintainer
bojieli
Verified

What it is

The title of this repository is "In-Depth AI Infra: Quantitative Analysis and System Design". The author writes that the AI Infra field still lacks a book that starts from hardware constraints and model architecture and derives system design quantitatively, and this is the motivation for writing the book. The author of the book is Bojie Li (李博杰). The manuscript is still a first draft and is under continuous revision.

Main features

  1. The method running through the whole book is deriving design from constraints: first clarify the task and quality requirements, list computation, storage, communication and dependencies, then make order-of-magnitude estimates against hardware capacity, bandwidth and compute.

  2. The book repeatedly asks five questions: what to move, how much to move, how many times, through where, and who must wait for it.

  3. The calculation tool comes with model configurations and a results index, and static calculation needs only the Python 3.10 or newer standard library, without a GPU or model weights.

  4. Experiments that need a GPU state their hardware and dependency requirements; without the matching device, the existing records can still be read.

  5. The reading website provides chapter navigation, full-text search and dark mode, and the language can be switched at the top right.

  6. The EPUB has an adjustable font size, and its layout reflows automatically with the screen.

Getting started

  1. Building the reading website locally requires Python 3.10 or newer.

  2. Building the full-book PDF and EPUB locally additionally requires Pandoc, XeLaTeX and fonts.

  3. The images of the translated editions are stored in Git LFS and must be downloaded before building.

  4. The repository uses Git LFS to store papers and larger inputs and measurement records, about 20 GB in total.

  5. The calculations directory contains the resource calculation tool, fixed inputs and recomputed results.

  6. GitHub Actions checks the website and the per-language PDF and EPUB builds of Pull Requests; after a push to main it publishes a Release, with PDF and EPUB for each of four languages, and deploys the online reading websites in four languages.

  7. The PDF is typeset by XeLaTeX, with complete formulas, tables, footnotes and cross-references.

From the README · bash
git clone https://github.com/bojieli/ai-infra-book.git
cd ai-infra-book

# 查看模型支持情况
python3 calculations/calc.py models

# 估算 Qwen3-8B 在 8192-token prefill 下的逐算子资源需求
python3 calculations/calc.py forward --model qwen3-8b --tokens 8192 --format md
From the README · bash
git lfs install
git lfs pull --include="experiments/ch05/05-01/**" --exclude=""
From the README · bash
python3 -m venv .venv-site
source .venv-site/bin/activate
python -m pip install -r website/requirements.txt
python scripts/build_site.py
python scripts/check_site.py
python scripts/build_site.py --serve
From the README · bash
bash book/build_pdf.sh                        # 简体中文 → book/AI-Infra-Book.pdf
bash book-en/build_pdf.sh                     # English → book-en/AI-Infra-Book-EN.pdf
bash book-zh-tw/build_pdf.sh                  # 繁體中文 → book-zh-tw/AI-Infra-Book-ZH-TW.pdf
python3 book/build_epub.py --edition zh       # EPUB 只需 Pandoc 与 Poppler；--edition 可选 zh、en、zh-tw、ru

Who it is for

The author writes that readers who have called a model API or run a model on their own machine, and want to understand further why models are slow and how to lower cost, can use this book as a starting point. Reading requires some background in Python, linear algebra and computer systems. The author suggests reading chapters 1 to 3 first, to understand models and workloads.

Readers working on model applications and inference services can focus on chapters 8 and 9. Readers working on systems or networks can focus on chapters 5 to 7. Readers working on chips and architecture can focus on chapters 4 to 7.

Readers who want to study systematically can read in table-of-contents order. Chapter 8, on inference optimization, asks when batching, KV management, offloading and speculative decoding are effective. Chapter 7, on data center networks, asks how network bandwidth, communication methods and congestion affect computing efficiency.

Chapter 10, on training systems, asks how to arrange memory, communication and recomputation to make training more efficient. The manuscripts directory contains the preface, twelve chapters of text, figures and drawing scripts. The experiments directory contains experiments and run records organized by chapter.

The case-studies directory contains case analyses of models, hardware and systems, indexed by chapter. The book's original text, figures and companion code are licensed under Apache License 2.0. The English edition was translated and contributed by @tg1482.

The Traditional Chinese edition was translated and contributed by @edward821220. The Russian community edition was organized, translated and checked by @ilkruglov.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.