In late June 2021, Microsoft, owner of the collaborative web platform GitHub, and Open AI launched a technical preview of GitHub Copilot, an AI-based completion tool designed to help developers code faster. It’s actually been available to them since June 21 as a subscription, but is free for students and maintainers of popular open source projects. GitHub surveyed more than 2,000 developers who participated in the technical preview, the recently published study found that Copilot increases their productivity.

Completion tools are designed to make developers more productive in an increasingly complex business. GitHub Copilot, an extension of Visual Studio Code, is powered by OpenAI Codex, an AI system created by OpenAI that performs better than GPT-3 for code generation, in part because it was trained on a dataset that includes a much higher concentration of public source code. GitHub Copilot works with a wide range of frameworks and languages, but works particularly well for Python, JavaScript, TypeScript, Ruby and Go. It offers suggestions that the developer is free to accept or reject.

The launch of Copilot’s technical preview had sparked a lot of reactions in the developer community and, in particular, from the Free Software Foundation regarding copyright, ownership issues of AI-generated code and legal impacts for GitHub authors. It had then launched a call for white papers on co-piloting, copyright, machine learning and free software, selected and published 5 of them on February 24.

GitHub’s study on the productivity of developers using Copilot

Last April, GitHub had conducted a 1st study with 24 users to understand how programmers use and perceive Copilot, it had revealed that, although it did not reduce task time or increase success rate, most participants preferred to use Copilot in daily programming tasks, as it often provided a useful starting point and saved the effort of searching online. However, participants had encountered difficulties in understanding, editing, and debugging the code snippets generated by Copilot, which significantly hindered their task-solving efficiency.

For this new study, the research and engineering teams partnered to combine qualitative survey data from more than 2,000 U.S.-based developers with anonymized data to determine whether developers believe they are more productive using GitHub Copilot and whether the data proves they actually are.

GitHub Copilot usage correlates with improved developer productivity

Developers who report the highest productivity gains with GitHub Copilot are those who accepted the most code suggestions posted :

• Those who accepted 29.73% of the suggestions reported “huge” productivity gains;
• Those reporting “high” productivity gains had accepted 27.77% of the suggestions;
• Users who reported “medium” productivity gains had accepted 26.94% of the suggestions;
• 23.16% of suggestions were accepted by those reporting “modest” gains.

The study also found that acceptance rates differed by language, which was no surprise to GitHub authors, with Javascript and Python enjoying the highest acceptance rates. The study also found that developers were primarily looking for a suitable starting point.

Translated from Etude de GitHub : l’utilisation de GitHub Copilot est en corrélation avec l’amélioration de la productivité des développeurs