Vibe Coding: AI evaluation for greenfield software engineering

Callum Barbour published a paper on arXiv titled "Vibe Coding Ate My Homework: An evaluation of AI approaches to greenfield software engineering and programming" on 15 Jun 2026, submitting a 10 page manuscript that includes 2 figures and an accompanying evaluation suite. The paper, arXiv:2606.18293, develops tests that measure an LLM's ability to carry out simple, isolated greenfield programming tasks in Python.

What is vibe coding?

Vibe coding is the use of natural language prompts to build applications and coding infrastructures without underlying knowledge of programming, a practice the paper explicitly labels "vibe coding." The author frames it as an approach that removes traditional code syntax in favor of expressing requirements in a user’s mother tongue, positioning it as an extreme endpoint of higher-level programming input.

How did the paper evaluate AI on greenfield programming?

Barbour created an evaluation suite designed to analyse an LLM's proficiency on simple, isolated greenfield programming tasks in Python. The suite scopes experiments to greenfield tasks so the results focus on a model’s ability to produce working code from natural language prompts rather than on integration or maintenance work. The submission notes the paper is 10 pages long with 2 figures and is available on arXiv under identifier arXiv:2606.18293 and DOI https://doi.org/10.48550/arXiv.2606.18293.

The abstract explains the motivation: recent generative AI advances have driven growth in using natural language to construct software, and the paper aims to evaluate the viability of that practice. To do this, the author assembled a testbed that targets isolated Python tasks so the analysis remains scoped and reproducible. The paper emphasizes benchmark analysis too, discussing the benchmarks that have been used to measure software engineering performance for such approaches.

Why it matters

Vibe coding reframes the question of who can build software by transferring the primary interface from code to natural language. That shift would change expectations for educational requirements, tooling, and evaluation. By supplying a focused evaluation suite and dissecting existing benchmarks, the paper supplies a concrete tool for comparing models on greenfield programming tasks rather than on downstream software engineering outcomes. The work therefore helps separate raw code-generation ability from integration, testing, and long-term maintainability concerns.

What to watch

Look for extensions of the evaluation suite beyond Python and into multi-file or integration scenarios, and for other researchers reusing arXiv:2606.18293’s testbed to compare LLMs. A clear signal that vibe coding is maturing will be independent benchmark comparisons using the suite and follow-up papers that expand the tasks from isolated scripts to full project scaffolds.

References and file details are available on the arXiv record submitted 15 Jun 2026; the submission package size listed is 1,268 KB and the paper contains 2 figures across its 10 pages.