SWE-agent: a system for LM agents solving software engineering tasks

The paper introduces SWE-agent, a system that facilitates LM agents to autonomously use computers to solve software engineering tasks.

Verified

Definition

The paper introduces SWE-agent, a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. The paper investigates how interface design affects the performance of language model agents. The paper calls the interface LM agents use to interact with computers the agent-computer interface (ACI).

The paper introduces SWE-agent as an agent composed of an LM and ACI that can interact with a computer to solve real-world software engineering problems, such as those proposed in SWE-bench. The paper argues that LM agents represent a new category of end user, with their own needs and abilities. The paper states that SWE-agent's custom ACI significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs.

At a glance

The paper states that the ACI comprises several principal components, including search/navigation, file viewer, file editor, and context management. The paper describes SWE-agent as an LM interacting with a computer through the ACI, which includes the commands the agent uses and the format of the feedback from the computer.

  • Search and navigation: the find_file command searches for filenames in the repository, while search_file and search_dir locate strings in files of a subdirectory.
  • File viewer: after edits are applied, the file viewer automatically displays the updated content.
  • File editor: the edit command works with the file viewer, allowing agents to replace a specific range of lines in the open file.
  • Code checking: the paper integrates a code linter into the edit function to alert the agent of mistakes it may have introduced.
  • Context management: the SWE-agent system uses informative prompts, error messages, and history processors to keep agent context concise and informative.

The source does not state the order of these points, so they are listed without arrows.

How it works

The paper states that at each step, SWE-agent generates a thought and a command, then incorporates the feedback from the command's execution in the environment. Built atop the Linux shell, SWE-agent also allows access to common Linux commands and utilities when needed. The ACI specifies both the commands available to the LM and how the environment state is communicated back to the LM.

The paper states that SWE-agent's ACI offers a small set of simple actions for viewing, searching through and editing files. The paper states that the ACI uses guardrails to prevent common mistakes, and that an agent receives specific, concise feedback about a command's effects at every turn. The paper reports that LM agents can struggle to reliably take actions in a setting where an agent interacts directly with a Linux shell.

The paper says that in this setting the Linux shell fails to provide simple commands to edit a small file segment, and does not provide feedback if the user makes an invalid edit. The paper shows that ACIs tailored specifically for LMs outperform existing user interfaces designed for human users, such as the Linux shell. One design principle listed in the paper is that actions should be simple and easy to understand for agents.

Another design principle listed in the paper is that actions should be compact and efficient. The design principles in the paper also include that environment feedback should be informative but concise. The design principles in the paper also include that guardrails mitigate error propagation and hasten recovery.

The paper primarily evaluates on the SWE-bench dataset. The paper also tests SWE-agent's basic code editing abilities with HumanEvalFix, a short-form code debugging benchmark. The paper shows that its ACI, although developed for GPT-4 Turbo, is portable to a different LM.

The paper describes a baseline in which a BM25 retrieval system retrieves the most relevant codebase files using the issue as the query. The paper describes another baseline system that, following the InterCode environment, asks the LM to resolve the issue by interacting with a shell process on Linux.

Example

In one case study in the paper, the agent was tasked with resolving a problem in the requests library. In that case study, the agent began by finding the sessions.py file in the requests library directory. The agent opened sessions.py and used the search_file command to locate the builtin_str function call.

The agent edited the sessions.py file to check if the method was a bytes object and, if so, decoded it to ASCII before applying the builtin_str function. The paper states that, upon evaluation, the agent's proposed changes successfully pass the instance's unit tests.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.