OptiLLM: an OpenAI API-compatible optimizing inference proxy

Core facts

Read by a script from the official API, not rewritten by a model.

Repository: https://github.com/algorithmicsuperintelligence/optillm
License: Apache-2.0
Languages: Python
Maintainer: algorithmicsuperintelligence
Verified: 2026-11-02

What it is

OptiLLM is an OpenAI API-compatible optimizing inference proxy. optillm is a transparent proxy. Once the proxy is running, it can be used as a drop-in replacement for an OpenAI client by setting base_url.

Main features

The readurls plugin fetches URLs found in prompts, blocks requests to non-public destinations by default, re-validates every redirect hop and rejects unresolvable hosts.
If a deployment intentionally needs to read internal documentation pages, the check can be explicitly opted out of; the documentation notes this should only be enabled on trusted networks where internal URL fetching is actually required.
The documentation lists a known limitation (DNS rebinding): the underlying HTTP client resolves the host a second time when it opens the connection, so a domain under attacker control could rebind to an internal address between the two lookups; high-security deployments should therefore not rely on this check alone and should additionally restrict the network egress of the optillm process (firewall or service-mesh policy).
The Privacy plugin anonymizes PII data in requests and deanonymizes it back to the original value in responses.
The Web Search plugin performs Google searches using Chrome automation (Selenium) to gather search results and URLs.
The Proxy plugin provides load balancing and failover across multiple LLM providers with health monitoring and round-robin routing.
For the Memory plugin, setting OPTILLM_MEMORY_FILE opts in to file-backed persistence so memories survive across requests.

Getting started

optillm supports various command-line arguments for configuration.
When using Docker, these arguments can also be set as environment variables prefixed with OPTILLM_.

From the README · bash

python optillm.py
2024-09-06 07:57:14,191 - INFO - Starting server with approach: auto
2024-09-06 07:57:14,191 - INFO - Server configuration: {'approach': 'auto', 'mcts_simulations': 2, 'mcts_exploration': 0.2, 'mcts_depth': 1, 'best_of_n': 3, 'model': 'gpt-4o-mini', 'rstar_max_depth': 3, 'rstar_num_rollouts': 5, 'rstar_c': 1.4, 'base_url': '', 'host': '127.0.0.1'}
 * Serving Flask app 'optillm'
 * Debug mode: off
2024-09-06 07:57:14,212 - INFO - WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:8000
2024-09-06 07:57:14,212 - INFO - Press CTRL+C to quit

From the README · python

import os
from openai import OpenAI

OPENAI_KEY = os.environ.get("OPENAI_API_KEY")
OPENAI_BASE_URL = "http://localhost:8000/v1"
client = OpenAI(api_key=OPENAI_KEY, base_url=OPENAI_BASE_URL)

response = client.chat.completions.create(
  model="moa-gpt-4o",
  messages=[
    {
      "role": "user",
      "content": "Write a Python program to build an RL model to recite text from any position that the user provides, using only numpy."
    }
  ],
  temperature=0.2
)

print(response)

From the README · bash

2024-09-06 08:35:32,597 - INFO - Using approach moa, with gpt-4o-mini
2024-09-06 08:35:35,358 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-09-06 08:35:39,553 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-09-06 08:35:44,795 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-09-06 08:35:44,797 - INFO - 127.0.0.1 - - [06/Sep/2024 08:35:44] "POST /v1/chat/completions HTTP/1.1" 200 -

From the README · bash

response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[{ "role": "user","content": "" }],
  temperature=0.2,
  extra_body={"optillm_approach": "bon|moa|mcts"}
)

From the README · bash

response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[{ "role": "user","content": "<optillm_approach>re2</optillm_approach> How many r's are there in strawberry?" }],
  temperature=0.2
)

From the README · bash

OPTILLM_API_KEY=your_secret_api_key

Who it is for

By default, optillm binds to 127.0.0.1 (localhost only) for security. The inbuilt inference server also exposes the legacy text completions endpoint, and OpenAI-legacy clients work against it. Streaming is not supported yet on the legacy text completions endpoint.

The Anthropic API, llama.cpp-server, and ollama currently do not support sampling multiple responses from a model. For models on HuggingFace, the built-in local inference server can be used as it supports multiple responses. OptILLM supports SSL certificate verification configuration for working with self-signed certificates or corporate proxies.

The offline Docker image variant is a self-contained image with pre-downloaded models (spaCy) for fully offline operation. All tests are automatically run on pull requests via GitHub Actions. The workflow tests multiple Python versions: 3.10, 3.11, and 3.12.

The Model Context Protocol (MCP) plugin enables OptiLLM to connect with MCP servers. OptiLLM supports both local and remote MCP servers through multiple transport methods. The websocket transport was removed in MCP Python SDK 2.x and is no longer supported.

The Majority Voting plugin generates k candidate solutions and selects the most frequent answer through majority voting (default k=6). The Z3 Solver approach utilizes the Z3 theorem prover for logical reasoning. PlanSearch implements a search algorithm over candidate plans for solving a problem in natural language.

The Monte Carlo Tree Search approach uses MCTS for decision-making in chat responses. The Router plugin uses the optillm-modernbert-large model to route requests to different approaches based on the user prompt.

Related tools listed on this site.