AI in Software Development: 26% Productivity Increase

Software development is undergoing a remarkable transformation. Artificial intelligence (AI) plays a central role in this — not just as a technical buzzword, but as a driving force that is fundamentally changing the way software is written, tested, and deployed. But what actually makes AI so relevant for the industry?

The answer lies in its ability to address core challenges: increasing competitive pressure, the growing complexity of modern software projects, and the ongoing shortage of skilled professionals. AI-powered tools like GitHub Copilot and other generative AI systems provide relief by making the development process more efficient, faster, and more precise.

From Routine Work to Innovation

Every software developer knows it: a significant portion of the work consists of repetitive tasks, such as writing boilerplate code, checking syntax, or debugging. These activities are necessary, but they often hold up creative work. AI tools take over these routines and give developers more room for complex, innovative challenges. And it’s not just high-level tasks — AI tools can now increasingly take on more complex assignments as well, or at least relieve a substantial amount of the workload.

Productivity and Quality at a New Level

Studies show that the use of AI in software development leads to significant productivity gains. According to a recent study by MIT and Princeton, which we will examine more closely in the following chapters, using AI-powered tools like GitHub Copilot enables an average productivity increase of 26%. Even more impressive: less experienced developers were able to increase their performance by up to 39%.

But it’s not only speed that benefits. The quality of the code improves as well. AI-generated suggestions are often more readable, more maintainable, and meet high functional standards — a characteristic that is especially crucial for long-term projects.

AI as a Strategic Tool

The significance of AI, however, goes beyond pure productivity. Companies that integrate AI into their development processes early and strategically secure a head start. They can realize complex projects with fewer resources, scale their teams more effectively, and respond more quickly to market demands. For small and medium-sized software firms in the DACH region, this could be the key to staying competitive on the international stage.

A Look Back at Existing Studies: GitHub Copilot and Code Quality

The introduction of GitHub Copilot two years ago fundamentally changed software development. Previous investigations already showed promising results, but one crucial question remained open: does GitHub Copilot actually improve code quality? A comprehensive study by GitHub itself now provides convincing answers.

Methodology of the Study

GitHub conducted a randomized controlled trial with 202 experienced developers. Half of the participants used GitHub Copilot, while the other half worked without AI support. All developers were given the task of programming API endpoints for a web server. The evaluation was carried out through unit tests and expert reviews.

Key Findings

1. Increased Functionality: Developers using GitHub Copilot were 56% more likely to pass all 10 unit tests. This points to significantly improved code functionality.
2. Improved Readability: Code written with GitHub Copilot showed considerably fewer readability errors. On average, developers were able to write 13.6% more lines of code without readability issues.
3. Generally Better Code Quality:

• 56% higher probability that unit tests pass.
• 13.6% more readable lines of code
• 3.6% higher reliability
• 2.5% higher maintainability

4. Higher Approval Rates: Code created with GitHub Copilot was 5% more likely to be approved. This means the code can be merged and put into production faster.

Deeper Insights

The study showed that developers using GitHub Copilot wrote 18.2 lines of code per error, compared to 16.0 lines without the tool. This corresponds to an improvement of 13.6% and can lead to significant time savings, since fewer corrections and revisions are necessary.

Interestingly, the commits from the GitHub Copilot group were more numerous but, on average, smaller. This suggests that developers were able to spend more time refining and improving the quality of the code, rather than focusing primarily on basic functionality.

Conclusion

This study provides the first controlled evidence of GitHub Copilot’s positive impact on code quality. It confirms earlier research findings, which showed that 85% of developers had more confidence in their code and 88% felt more productive when using GitHub Copilot.

The results underscore that AI tools like GitHub Copilot not only increase development speed but also contribute substantially to improving code quality. This has far-reaching implications for software development, from faster development cycles to more robust and easier-to-maintain codebases.

In the following sections, we will look at how these findings align with the latest studies from MIT and Princeton and what practical consequences arise from them for developers and companies.

New Findings from the MIT/Princeton Study

The recently published study by researchers from MIT and Princeton University delivers groundbreaking insights into the impact of generative AI on the productivity of software developers. This large-scale investigation analyzed data from three randomized field experiments at Microsoft, Accenture, and an anonymous Fortune 100 electronics manufacturer. In total, 4,867 software developers were observed. The results are of great importance both for companies and for individual developers.

Productivity Increase of 26% Through AI Tools

The central finding of the study is an average productivity increase of an impressive 26.08% (standard deviation: 10.3%) among developers who used the AI tool GitHub Copilot. This productivity increase was measured by the number of completed tasks.

Specifically, it showed:

• The number of weekly pull requests (a measure of completed tasks) rose by 26.08%.
• The number of commits increased by 13.55%.
• The frequency of code compilations (builds) even rose by 38.38%.

These figures are especially noteworthy when you consider that they come from real working environments and not from controlled laboratory experiments. They show that AI-powered coding assistants like GitHub Copilot can indeed have a significant impact on the daily work of software developers.

Stronger Effects Among Less Experienced Developers (Up to 39%)

A particularly interesting aspect of the study is the analysis of productivity gains in relation to developers’ experience and seniority. Here, clear differences emerged:

• Developers with shorter tenure (below the median) increased their productivity by 27% to 39%.
• Among developers with longer tenure, the increase was only 8% to 13%.
• Junior developers recorded productivity gains of 21% to 40%.
• Senior developers reached increases of 7% to 16%.

These results suggest that AI tools can be a powerful instrument for boosting productivity, especially for entry-level and less experienced developers. They enable newcomers to work at a higher level more quickly and to handle complex tasks more efficiently.

Interestingly, it also turned out that younger and less experienced developers were more willing to adopt the AI tool and use it continuously:

• Developers with shorter tenure were 9.5 percentage points more likely to use Copilot (84.3% vs. 74.8%).
• Junior developers were 5.3 percentage points more likely to adopt the tool (82.1% vs. 76.8%).

These figures underscore how important it can be for companies to give younger team members in particular access to AI-powered development tools. It could be a way to shorten onboarding time and increase the productivity of new employees more quickly.

The results of this study are a strong argument for the use of AI in software development. They show that tools like GitHub Copilot have the potential to significantly boost productivity — especially among less experienced developers. For companies, this means that integrating such tools into the development process can represent a real competitive advantage. At the same time, new opportunities open up for developers to expand their skills and work more efficiently.

It remains exciting to observe how these technologies continue to evolve and what long-term effects they will have on the software industry. One thing, however, is clear: AI-powered development tools are not a passing fad, but will lastingly change the way we develop software.

Acceptance and Usage: Reasons for the 60–70% Adoption Rate of AI Suggestions

The MIT/Princeton study provides interesting insights into developers’ usage behavior when dealing with AI-generated code suggestions. Despite the obvious productivity benefits, it turns out that not all of the code snippets suggested by GitHub Copilot are adopted. Let’s take a closer look at the reasons for the 60–70% acceptance rate.

Acceptance Rate of AI Suggestions

The study shows that, on average, about 60–70% of the code suggestions generated by GitHub Copilot were accepted by developers and integrated into their code.

These figures raise important questions:

1. Why aren’t all AI suggestions accepted?
2. Which factors influence a developer’s decision to accept or reject a suggestion?

The Role of the Developer: Why Training Remains Indispensable

The acceptance rate of 60–70% underscores that the role of the human developer remains indispensable even in the age of AI assistants. Here are a few reasons:

1. Critical Thinking: The fact that not all AI suggestions are accepted shows that developers critically review the suggestions. They use their expertise to decide which suggestions make sense and which don’t.
2. Contextual Understanding: Developers understand the broader context of a project, the specific requirements, and the company guidelines. An AI tool cannot fully grasp these aspects, which explains why some suggestions are rejected.
3. Code Quality and Style: Development teams often have specific and varying notions of code style and quality that do not always match the AI suggestions.
4. Project-Specific Requirements: AI-generated suggestions can sometimes provide generic solutions that need to be adapted for specific project requirements.

These factors underscore the necessity of training, so that developers learn to evaluate and use AI suggestions effectively.

Limitations of GitHub Copilot: Scenarios in Which Copilot Is Less Effective

The study also points to situations in which GitHub Copilot suggestions might be less effective or accepted less frequently:

1. Complex, Project-Specific Tasks: For tasks that require a deep understanding of the project architecture or specific business logic, Copilot suggestions might be less fitting.
2. Highly Specialized Domains: In areas with very specific expertise or proprietary technologies, the accuracy of AI suggestions might be limited if this contextual information is not available to the AI.
3. Security-Critical Areas: In areas where security is the top priority, developers might hesitate to accept AI-generated suggestions without thorough review.
4. Innovative Design: When developing entirely new approaches or architectures, AI suggestions based on existing code might be less useful, since the data underlying the LLM is usually over a year old.

Possible Alternatives: Examples of Other Tools and Their Strengths.

While the study focuses on GitHub Copilot, there are other AI-powered development tools that could achieve higher acceptance rates in certain scenarios:

CursorAI

Cursor AI outperforms GitHub Copilot in several key areas:

1. Code generation and accuracy: Cursor excels at generating and refining entire code blocks, adapting to your coding style and project context over time. It’s particularly effective for larger, more complex tasks and projects.
2. Customization and personalization: Cursor AI offers more flexibility in customizing the coding experience, with its learning algorithm adapting to your specific coding style.
3. Multi-file edits and project-wide understanding: Cursor has a better grasp of the entire project context, allowing it to make changes across multiple files and enhance code consistency.
4. Standalone editor with deep AI integration: Unlike Copilot, which is an extension, Cursor offers a dedicated editor with more comprehensive AI features.
5. API integration: Cursor provides more flexibility with API integration compared to Copilot’s limited integration outside of OpenAI models.
6. Advanced features: Cursor offers superior capabilities in error detection, refactoring support, and debugging for larger projects.
7. Custom documentation context: Cursor can index documentations to improve its understanding and suggestions. For example up-to-date API or framework docs or internal project files.

While GitHub Copilot excels in multi-language support and integration with the GitHub ecosystem, Cursor AI’s comprehensive approach to project understanding and code generation makes it a more powerful tool for many developers, especially those working on large, complex projects

Codeium Windsurf

1. Windsurf provides (at this time) unlimited access to larger Codeium models, GPT-4, and Claude Sonnet.
2. Windsurf features full repo context awareness, which is more comprehensive than GitHub Copilot’s capabilities.
3. Windsurf emphasizes user control and data security, with no training on non-permissive data. It offers encryption for data in transit and optional zero-day data retention.
4. Windsurf supports multiple deployment methods, including SaaS, on-premises, and in-VPC, while GitHub Copilot is SaaS-only.
5. Windsurf supports over 70 programming languages, compared to GitHub Copilot’s approximately 40.

JetBrains AI

JetBrains AI offers several advantages over GitHub Copilot, particularly for developers who are already integrated into the JetBrains ecosystem:

1. Advanced refactoring tools specific to JetBrains environments
2. It focuses more on structuring projects rather than just generating code snippets
3. The AI analyzes your project’s structure and provides tailored recommendations
4. It adapts to various coding styles, offering a more personalized experience
5. Built-in documentation generation feature integrated into the editor
6. The tool is powered by JetBrains’ own models, which aim to provide higher quality suggestions for Java-related tasks

Conclusion: AI as a Strategic Advantage in Software Development

The MIT/Princeton study provides compelling evidence that AI-powered tools like GitHub Copilot can offer a significant strategic advantage in software development. Let’s summarize the most important findings and consider their implications for the future of the industry.

Summary of the Main Findings

1. Significant Productivity Increase: The study shows an average productivity increase of 26.08% among developers who use GitHub Copilot. This is a considerable gain that can directly affect the efficiency and speed of software projects.
2. Particular Benefit for Less Experienced Developers: Younger and less experienced developers benefited the most from the AI tools, with productivity increases of up to 39%. This shows that AI tools can act as an equalizing factor within development teams.
3. Selective Acceptance of AI Suggestions: The fact that developers accept only 60–70% of AI suggestions underscores the continued important role of human expertise and critical thinking in the development process.
4. Different Adoption Rates: The study shows that younger and less experienced developers are more willing to adopt AI tools and use them continuously.

Strategic Implications for Companies

1. Competitive Advantage Through Early Adoption: Companies that integrate AI tools into their development processes early can gain a significant lead in terms of development speed and efficiency.
2. Optimization of Team Composition: The differing effects on experienced and less experienced developers suggest that a balanced team composition could be optimal for fully exploiting the benefits of AI tools.
3. Focus on Continued Education: Companies should invest in training that not only instructs developers in the use of AI tools but also sharpens their ability to critically assess AI suggestions.
4. Adaptation of Development Processes: Integrating AI tools could require a revision of existing development processes and best practices in order to fully realize their potential.
5. Cost Efficiency: The productivity gains could lead to significant cost savings, especially for large development teams or complex projects.

Outlook on the Future of Software Development

1. AI as a Standard Tool: It is to be expected that AI-powered development tools will become part of the standard repertoire of every software developer in the near future.
2. Evolving Role of the Developer: The role of the software developer will continue to evolve toward that of an “AI conductor” who skillfully deploys AI tools to solve complex problems.
3. Accelerated Innovation: The increased productivity could lead to an acceleration of innovation cycles in software development, which in turn brings new challenges and opportunities.
4. Ethical and Legal Considerations: With the increasing integration of AI into the development process, questions of copyright, liability, and the ethical use of AI-generated code will also gain in importance.
5. Continuous Improvement of AI Tools: It is to be expected that the quality and precision of AI suggestions will continue to increase, which could potentially lead to even higher acceptance and productivity rates.

Final Thoughts

The new MIT and Princeton study shows an important breakthrough in software development. It makes clear that AI tools like GitHub Copilot not only have theoretical potential but already offer significant and measurable benefits in practice. For companies and developers alike, it will be crucial to understand these tools, adopt them, and integrate them effectively into their workflows.

The future of software development lies not in replacing human developers with AI, but in a synergistic collaboration between humans and machines. Companies that can best leverage this synergy will have a decisive advantage in the rapidly evolving technology landscape.

Ultimately, this study underscores that AI in software development is no longer merely an option but increasingly a necessity to stay competitive and drive innovation. The challenge and opportunity for the industry lies in using these tools responsibly and effectively in order to unleash the full potential of human creativity and artificial intelligence.

Further Information: Reference to the Full Study

For all those who want to dive deeper into the subject and explore the detailed results, methods, and analyses of the study, the full study is available at the following link:

The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers

This comprehensive research paper, authored by Kevin Zheyuan Cui (Princeton University - Bendheim Center for Finance), Mert Demirer (Massachusetts Institute of Technology (MIT)), Sonia Jaffe (Microsoft Research), Leon Musolff (University of Pennsylvania - Business & Public Policy Department), Sida Peng (Microsoft Corporation), and Tobias Salz (Massachusetts Institute of Technology (MIT); National Bureau of Economic Research (NBER)), offers an in-depth look at the impact of generative AI on highly skilled work, particularly in the field of software development.

What You’ll Find in the Full Study:

1. Detailed Methodology: Learn more about the design and execution of the three field experiments at Microsoft, Accenture, and the anonymous Fortune 100 electronics manufacturer.
2. Statistical Analyses: Dive into the extensive statistical data and analyses that underpin the results presented in this blog post.
3. Additional Insights: Discover further interesting observations and conclusions that go beyond the scope of this blog post.
4. Discussion of Implications: Read the researchers’ detailed discussion of the possible long-term effects of their findings on the software development industry and beyond.
5. Limitations and Future Research Directions: Learn more about the limits of the current study and which questions remain open for future research.

The AI Software Company supports B2B software firms in the DACH region in making their development processes more efficient, faster, and future-proof with AI.

Curious how to make meaningful use of AI in your software team? Sign up for our newsletter and receive valuable tips, insights, and updates!