Coding is being positioned as place where generative AI is going to have a huge impact. But a new study of 800 developers found GitHub Copilot did little to improve productivity, while introducing 41% more bugs into the code.
Code analysis firm sees no major benefits from AI dev tool when measuring key programming metrics, though others report incremental gains from coding copilots with emphasis on code review.
@caiocesarmsouza.bsky.social 💁🏻♀️✨ here comes the so expected revolution! hahaha
I have been using it as an assistant when learning new sub-specialties and tools (a constant need for most devs) and as a “computer aided custom documentation summaries”. For these applications it is regrettably unbeatable. Simply generating code to deploy is the most naïve and destructive use.
Yes yes yes I knew it I feel vindicated AI is not worth it
Can confirm! The only developers I've seen speak the praises of copilot are newbies for whom is still takes a while to recall / Google things, and need sump'n to "get them started". It's also got no situational awareness; of phrasing the code in a particular way to make it more readable in context.
It’s interesting that this research was produced by another AI firm this AI firm seems to primarily market AI for use by management to track and optimize worker productivity, so it seems pretty clear the org is neither anti-AI nor a direct competitor to CoPilot pretty gross usecase for Zipwhip tho
Respectfully, everyone's outcome-shopping these studies which all hinge on some quantitative measure of developer output. If such a thing existed, we'd be using it industry wide already. (IMHO only reasonable measure of effectiveness is voluntary adoption over time. And maybe some qual research.)
I’m glad no one is judging my productivity based on “pull request cycle time”. May as well just use lines of code if we’re not going to use any sort of modern metric for dev productivity.
The number of times I’ve seen copilot or ChatGPT generate a sql Injection vulnerability is obscene 😂
I use it to give me regexp samples but the proposals often need a lot of iterations to do what I really need. But regular expressions confound me so it is helpful at speeding up text matching