AGENTS.md Files: Higher Costs, but Hardly Better Results for AI Programmers

Imagine giving a new classmate detailed directions through the school building so they can find the classroom faster. Sounds logical, right? This is exactly the idea behind so-called context files like AGENTS.md for AI programmers. But a new study shows: The directions don't really help the AI student arrive faster or better – they just cost them more time reading.

What are AGENTS.md files?

AI coding agents are programs that can independently write code and fix bugs. To help them find their way around an unfamiliar software project, developers often place special text files like AGENTS.md or CLAUDE.md in the project folder. These files contain a kind of profile for the AI: an overview of the project, notes on tools used, or special style rules for the code. The idea behind it is that the AI has to guess less and finds better solutions. Over 60,000 public projects on GitHub already use such files.

The Big Practical Test

Researchers from ETH Zurich and LogicStar.ai took a closer look at this practice. They wanted to know: Do these profiles actually do anything in the real world? To find out, they set up two test scenarios:

1. Known projects with AI-generated profiles: Here, they used the established SWE-bench test, where AI agents have to fix real bugs in popular open-source projects. The context files were automatically created by an AI, as many manufacturers recommend.
2. New projects with real developer profiles: Since no such files existed for many older test projects, the researchers created their own benchmark called CTX BENCH. This contains 138 programming tasks from lesser-known projects where the developers themselves had created a context file.

In both scenarios, the AI agents competed under three conditions: without any context file, with a file created by an AI, and with a file written by a human.

The Surprising Result

The results were clear and sobering: Context files do not improve the success rate of AI agents. Regardless of whether the file was written by a human or an AI, the agents did not solve the programming tasks more often on average. The only noticeable change was an average cost increase of over 20 percent. This is because the AI reads and processes the additional information, but cannot translate it into better results.

A more detailed analysis of the agents' working methods, however, revealed an interesting difference: Human-written context files performed on average 7% better than AI-generated ones. The researchers found that the agents followed the instructions in the files very precisely, but especially the typical project overviews – recommended by many providers – were no help. The AI invested more time in exploring and testing without this benefiting the solution.

What does this mean for developers?

The study advises a more cautious approach to the hype surrounding context files. According to the current state of knowledge, AI-generated profiles should be omitted entirely, as they only cause costs without helping. Human-written files should be limited to the bare essentials, for example, unusual programming conventions not already found in the normal project description (README). Above all, every new aid for an AI should be thoroughly tested to see if it really provides a benefit before being deployed in the project.