OpenAI moderation models for harmful content in text and images

OpenAI moderation models are used to detect harmful content in text and images.

Verified

Definition

OpenAI moderation models are used to detect harmful content in text and images. The moderation endpoint can classify standalone inputs, and moderation scores can be requested alongside a generated response. The moderation results can be used to enforce an application's policy, such as filtering content, routing a request for review, or intervening with accounts that submit flagged content.

The omni-moderation-latest model accepts text and image inputs. The omni-moderation-latest model does not classify audio.

At a glance

The moderation of generated content applies when an application generates text with the Responses API or Chat Completions API and needs moderation signals. In a generation request that carries a moderation request, the model still generates normally.

  • When the application needs generated text and moderation scores together, it passes a top-level moderation object in the generation request.
  • The API returns moderation scores for the model input and generated output without a separate moderation request.
  • If the application needs to handle moderation failures, check the moderation result type before reading scores.
  • If a moderation step cannot complete, the corresponding input or output moderation field can contain an error instead of moderation scores.
  • Start with flagged for a first-pass decision, then inspect categories and category_scores for logging, routing, audit trails, or human-review queues.
  • A refusal or other safety-aware response can still trigger a flag if it discusses harmful content.
  • Treat moderation scores as signals for the application's policy, not as an automatic blocking decision.
  • Review the moderation results before showing the output to a user or taking downstream actions.

The source does not state the order of these points, so they are listed without arrows.

How it works

The moderation endpoint can be used to classify text or image inputs without generating a model response. For tool-calling requests, moderation covers tool-call arguments and tool outputs when they appear in conversation content, and it does not cover tool names, tool descriptions, tool schemas, or response-format schemas. If a generated response is streamed, moderation scores arrive after the full generated output is available, and they are not included with partial output deltas.

The JSON response includes fields that describe which categories are present in the input and the model's confidence in each category. The category_scores field contains a dictionary of per-category scores. A category_scores value is between 0 and 1, where higher values denote higher confidence.

The categories field contains a dictionary of per-category violation flags. The category_applied_input_types field contains the input types that the category score applies to. Do not send known or suspected child sexual abuse material (CSAM) to the Moderation API.

The Moderation API is not designed for CSAM detection or handling and is not a substitute for dedicated child-safety safeguards.

Example

The illicit category covers content that gives advice or instruction on how to commit illicit acts. A phrase like “how to shoplift” would fit the illicit category. The violence category covers content that depicts death, violence, or physical injury.

The sexual/minors category covers sexual content that includes an individual who is under 18 years old. The self-harm category covers content that promotes, encourages, or depicts acts of self-harm. Categories marked as “Text only” do not support image inputs.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.