You have 1 article left to read this month before you need to register a free LeadDev.com account.
Estimated reading time: 3 minutes
Key takeaways:
- AI-assisted students scored higher on tasks but retained far less knowledge than those using traditional methods.
- Students using AI felt only 45% of their code was really theirs.
- The fix involves pairing juniors with seniors and training engineers, not programmers.
AI-coding tools can make inexperienced programmers look much better at their jobs. The problem is they may also be stopping them learning how to do those jobs – creating issues for the long run.
That’s the concerning finding of a new study by researchers at the University of New South Wales (UNSW), who gave undergraduate computer science students three introductory C programming tasks and either ChatGPT-4.5 or conventional web search to help them.
The students using AI scored 89% on the coding tasks, compared with 69% among those using Google.
However, that 20 percentage point gap belied a bigger issue – that the supposedly smart-sounding students actually weren’t.
Your inbox, upgraded.
Receive weekly engineering insights to level up your leadership approach.
When the researchers tested what knowledge the students had actually retained, the result flipped. The group using ChatGPT scored 41% when tested immediately afterwards, compared with 53% for the non-AI group.
When the same groups were quizzed two days later, the AI-enhanced group scored just 39% compared to 52%. The reason why that could be was given by the students themselves: those using ChatGPT estimated just 45% of the code they submitted was really theirs, compared with 81% among the students using conventional search.
Big issue or a mountain out of a molehill?
The study was small, with just 55 students included in the final analysis. All were from one university and the study looked at students rather than developers working inside real engineering teams.
Nonetheless, the findings raise an awkward question for engineering leaders: if junior developers can use AI to produce better code before they have built the knowledge needed to understand it, what is the longer term impact on the talent pipeline, and what does it mean for the people who should eventually become senior engineers?
Adrian Harwood, head of research software engineering at the University of Manchester, thinks trying to stop juniors using the tools is futile. His department takes undergraduate students on year-in-industry placements, and says the current cohort are already enthusiastically using generative AI.
That enthusiasm is unlikely to dim, but to try and counteract AI hollowing out knowledge, juniors in his department are deliberately kept close to more experienced engineers.
“Every person who’s considered junior would never be asked to do a project solo,” he says. Senior engineers review their pull requests, including code produced with AI, and point out problems the junior developer may not yet have the experience to see.
More like this
Changing what’s taught
That changes what some of the learning looks like. Rather than just teaching junior engineers to produce code from scratch, Harwood says part of the job now is teaching them “how to police the tool” by reviewing what it produces and understanding why some of the choices it makes might cause problems.
Getting juniors to think, rather than race to push out code, also helps to avoid a constant peril for developer teams: the risk of confusing quantity with quality.
New York • September 8 & 9, 2027
Loved LDX3 New York? Pre-sale tickets for 2027 are now available.
The UNSW study shows why output can be a misleading metric, with junior developers shipping out code they don’t necessarily understand or can debug.
Harwood thinks engineering organizations need to be clearer about what they are actually trying to develop in their talent.
“I’m trying to train engineers in my department. I’m not trying to train programmers,” he says. Writing code is only one part of that: engineers also need to understand requirements, turn them into sensible specifications, verify what has been built, and work out what has gone wrong when it fails.
That may mean changing junior development programmes rather than banning AI from them. Harwood argues that new engineers should spend more time learning how systems can fail, critically assessing outputs and, as he puts it, “distrusting everything that you produce,” regardless of whether it came from a human or an AI.
“The skills that will survive the AI revolution will be the engineering skills,” Harwood says. “It won’t be the software skills.”
