KubeAI · AI Nav City

Core facts

Read by a script from the official API, not rewritten by a model.

Repository: https://github.com/kubeai-project/kubeai
License: Apache-2.0
Languages: Go
Maintainer: kubeai-project
Homepage: https://www.kubeai.org
Verified: 2026-11-01

What it is

KubeAI deploys and scales machine learning models on Kubernetes. KubeAI is built for LLMs, embeddings, reranking and speech-to-text. KubeAI consists of two primary sub-components. The KubeAI model operator manages backend server Pods directly.

Main features

KubeAI operates vLLM and Ollama servers.
KubeAI generates embeddings with Infinity.
KubeAI reorders search results with cross-encoder models.
KubeAI scales from zero to meet demand.
The KubeAI model proxy provides an OpenAI-compatible API.
The KubeAI proxy implements a prefix-aware load balancing strategy that optimizes for the KV cache utilization of the backend serving engines (i.e. vLLM).

Getting started

Install KubeAI and wait for components to be ready, which may take a minute.
Some predefined models can be installed.
Starting a watch on Pods in a standalone terminal shows how KubeAI deploys models.
Start a local port-forward to the bundled chat UI.

From the README · bash

kind create cluster # OR: minikube start

From the README · bash

helm repo add kubeai https://www.kubeai.org
helm repo update

From the README · bash

helm install kubeai kubeai/kubeai --wait --timeout 10m

From the README · bash

cat <<EOF > kubeai-models.yaml
catalog:
  deepseek-r1-1.5b-cpu:
    enabled: true
    features: [TextGeneration]
    url: 'ollama://deepseek-r1:1.5b'
    engine: OLlama
    minReplicas: 1
    resourceProfile: 'cpu:1'
  qwen2-500m-cpu:
    enabled: true
  nomic-embed-text-cpu:
    enabled: true
EOF

helm install kubeai-models kubeai/models \
    -f ./kubeai-models.yaml

From the README · bash

kubectl get pods --watch

From the README · bash

kubectl port-forward svc/open-webui 8000:80

Who it is for

KubeAI orchestrates LoRA adapters across replicas. KubeAI integrates with Kafka, PubSub, and more. The KubeAI proxy also implements request queueing (while the system scales from zero replicas) and request retries (to handle bad backends).

The KubeAI model operator automates common operations such as downloading models, mounting volumes, and loading dynamic LoRA adapters via the KubeAI Model CRD. The two components, the model proxy and the model operator, are co-located in the same deployment. KubeAI works with OpenAI client libraries.

KubeAI does not require Istio, Knative, etc. KubeAI does not depend on other systems like Istio and Knative (for scale-from-zero), or the Prometheus metrics adapter (for autoscaling). KubeAI runs on CPU, GPU, or TPU.

The project ships with a catalog of popular models, pre-configured for common GPU types. Telescope uses KubeAI for multi-region large scale batch LLM inference. KubeAI can be deployed on Vultr Managed Kubernetes using the application marketplace.

Arcee uses KubeAI for multi-region, multi-tenant SLM inference. In the local quickstart, Qwen2 responds slowly at first because minReplicas is 0 and KubeAI needs to spin up a new Pod. KubeAI documentation covers concepts, guides, and API documentation.

Related tools listed on this site.