Evals (evaluations) · AI Nav City

Definition

Evaluations (often called evals) test model outputs to ensure they meet style and content criteria that you specify. The OpenAI evals guide focuses on configuring evals programmatically using the Evals API. If you prefer, you can also configure evals in the OpenAI dashboard.

Describing the task as an eval, running the eval with test inputs, and analyzing the results then iterating on the prompt is a process somewhat similar to behavior-driven development (BDD), where you begin by specifying how the system should behave before implementing and testing it. OpenAI is deprecating the Evals platform.

At a glance

Creating an eval begins by describing a task to be done by a model.

Describe the task to be done as an eval.
Run the eval with test inputs (a prompt and input data).
Analyze the results, then iterate and improve on the prompt.
data_source_config is a schema for the test data used along with the eval.
testing_criteria are the graders that determine if the model output is correct.
Uploading a JSONL file may be convenient, with data in the schema specified when the eval was created.
With test data in place, a prompt is evaluated against the test criteria via the API by creating an eval run.
The eval run is queued and executes asynchronously, processing every row in the data set and generating responses with the specified prompt and model.

The source does not state the order of these points, so they are listed without arrows.

How it works

When a run is created, a prompt is set up using either a Chat Completions messages array or a Responses input, and that prompt is used to generate a model response for every line of test data in the data set. After the eval is created, it is assigned a UUID that is needed to address it later when kicking off a run. Depending on the size of the dataset, the eval run may take some time to complete.

To receive updates when a run succeeds, fails, or is canceled, create a webhook endpoint and subscribe to the eval.run.succeeded, eval.run.failed, and eval.run.canceled events. You can view the current status in the dashboard or fetch it via API, and fetching the status of an eval run needs the UUID of both the eval and the eval run. The API response contains information about test criteria results, API usage, and a report_url property that takes you to a dashboard page where you can explore the results visually.

Example

The guide's example uses a model to classify the contents of IT support tickets into one of 3 categories: Hardware, Software, or Other. To implement this ticket classification use case, you can use either the Chat Completions API or the Responses API. In this ticket example, ticket_text is a string of text with the contents of a support ticket.

In this ticket example, correct_label is a ground truth output, provided by a human, that the model should match. Because the example's test criteria reference a sample, the output generated by a model given the prompt, the example also sets include_sample_schema to true. In this case, a string_check grader is used to evaluate the output.

In this example, the string the model outputs should exactly match the human-labeled correct_label field in the test data. The example JSONL data set contains both test inputs and ground truth labels to compare model outputs against, and files can be uploaded in the dashboard or via API.

Related tools listed on this site.