Conversation State (OpenAI)

Definition

OpenAI provides a few ways to manage conversation state. Each text generation request is independent and stateless, but multi-turn conversations can still be implemented by providing additional messages as parameters to the text generation request. Alternating user and assistant messages capture the previous state of a conversation in one request to the model.

The Conversations API works with the Responses API to persist conversation state as a long-running object with its own durable identifier. Conversations store items, which can be messages, tool calls, tool outputs, and other data.

At a glance

When making an API request to Chat Completions with the o1 model, token counts apply toward the context window total. Model inputs and outputs are metered in tokens, which are parsed from inputs to analyze their content and intent and assembled to render logical outputs.

Input tokens are the inputs included in the messages array with Chat Completions.
Output tokens are the tokens generated by a model in response to a prompt.
Reasoning tokens are used by the model to plan a response.
When making an API request to the Responses API with a reasoning enabled model, like the o1 model, token counts apply toward the context window total.
Input tokens are the inputs included in the input array for the Responses API.
Each model has different limits for output tokens.
The model gpt-4o-2024-08-06 can generate a maximum of 16,384 output tokens.
The model gpt-4o-2024-08-06 has a total context window of 128k tokens.

The source does not state the order of these points, so they are listed without arrows.

How it works

To manually share context across generated responses, the model's previous response output is included as input and appended to the next request. The Responses API returns encrypted reasoning items by default. Replaying the complete output keeps reasoning items and assistant phase values intact.

After a conversation object is created, it can be kept in use across sessions, devices, or jobs. In a multi-turn interaction, the conversation can be passed into subsequent responses to persist state and share context across subsequent responses, rather than having to chain multiple response items together. The previous_response_id parameter shares context across generated responses and lets you chain responses and create a threaded conversation.

In Responses API WebSocket mode, continuation uses the same previous_response_id semantics as HTTP mode, but over a persistent socket with repeated response.create events. Response objects are saved for 30 days by default, and setting store to false disables this behavior. Conversation objects and items in them are not subject to the 30 day TTL.

Example

Another example asks the model to tell a joke, followed by a request for another joke. The context window is the maximum number of tokens that can be used in a single request. A context window describes the total tokens that can be used for both input and output tokens, and for some models, reasoning tokens.

Tokens generated in excess of the context window limit may be truncated in API responses. The tokenizer tool, built with the tiktoken library, shows how many tokens are in a particular string of text.

Related tools listed on this site.