Codex: a GPT model fine-tuned on publicly available GitHub code

The paper introduces Codex, a GPT language model fine-tuned on publicly available code from GitHub, and studies its Python code-writing capabilities.

Verified

Definition

The paper introduces Codex, a GPT language model fine-tuned on publicly available code from GitHub, and studies its Python code-writing capabilities. According to the paper, a distinct production version of Codex powers GitHub Copilot. The paper describes several early Codex models, whose descendants power GitHub Copilot and the Codex models in the OpenAI API.

The paper says its early investigation of GPT-3 revealed that GPT-3 could generate simple programs from Python docstrings. In the paper, the focus is the task of generating standalone Python functions from docstrings, with the correctness of code samples evaluated automatically through unit tests.

At a glance

The paper calls the supervised fine-tuned models Codex-S and says they produce consistent gains across model size. The paper reports that Codex-S outperforms the corresponding Codex by an average margin of 6.5 percentage points on pass@1 and by a larger average margin of 15.1 percentage points on pass@100 across model size.

Diagram: The paper fine-tunes GPT models containing up to 12B parameters on code to produce Codex. > To improve function synthesis from docstrings, the paper fine-tunes Codex on standalone, correctly implemented functions. > The paper trains the docstring generating models Codex-D by minimizing negative log-likelihood of the docstring. > The paper grades sample docstrings by hand, considering a docstring correct if it uniquely and accurately specifies the code body.1The paper fine-tunes GPT models containing upto 12B parameters on code to produce Codex.2To improve function synthesis from docstrings,the paper fine-tunes Codex on standalone,correctly implemented functions.3The paper trains the docstring generatingmodels Codex-D by minimizing negativelog-likelihood of the docstring.4The paper grades sample docstrings by hand,considering a docstring correct if it uniquelyand accurately specifies the code body.

How it works

After filtering, the paper's final dataset totaled 159 GB. The paper reports that models fine-tuned from GPT converge more quickly, so it applies this strategy for subsequent experiments. The paper states that generative models for code are predominantly benchmarked by matching samples against a reference solution, where the match can be exact or fuzzy, as in BLEU score.

The paper describes developing a sandbox environment to safely run untrusted programs against unit tests. The paper selected the gVisor container runtime as the main host protection component. The paper collected problem statements, function signatures, and solutions from several programming contest and interview preparation websites.

In total, the paper curated 10,000 problems in this way.

Example

The paper evaluates functional correctness on a set of 164 hand-written programming problems, which it calls the HumanEval dataset. Each of these problems includes a function signature, docstring, body, and several unit tests, with an average of 7.7 tests per problem. The paper says these problems assess language comprehension, algorithms, and simple mathematics, with some comparable to simple software interview questions.

On HumanEval, which the paper describes as a new evaluation set it releases to measure functional correctness for synthesizing programs from docstrings, its model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. The paper reports that it solves 70.2% of its problems with 100 samples per problem. The paper reports that within 100 samples, Codex-S generates at least one correct function for 77.5% of the problems.

The paper says its investigation of its model reveals limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables, and that Codex struggles to parse increasingly long and higher-level or system-level specifications.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.