Learning a language with AI: what studies show – and what they don't

What do the studies actually measure?

“AI tutor” sounds like the future, but research on it goes back several years. There are now enough studies for meta-analyses, which combine the results of many individual studies.

One of them appeared in 2024 in the Review of Educational Research: 70 effect sizes from 28 studies. Learners who practised with a chatbot did better on average than comparison groups without one. The effect size was g = 0.48.⁠^[1] A second meta-analysis of 31 comparative studies arrives at g = 0.61 and calls it a medium effect.⁠^[2]

What does “g = 0.5” mean? Roughly: the difference between the groups is about half a standard deviation. In education research that is a noticeable but not overwhelming difference – and it is an average across very different studies.

When does AI help more, and when less?

Both meta-analyses stress that the effect is not the same everywhere. The first found four characteristics that made a difference: the learners' educational level and language level, the interface design, and the interaction capability of the chatbot.⁠^[1] The second names, among other things, whether the bot could be used on mobile, whether it writes or speaks, whether generative AI was involved – and what it was compared with.⁠^[2]

That last point matters. “Better than without a chatbot” can mean: better than no extra practice at all. It does not automatically mean better than a good teacher or a conversation with a native speaker. The authors of the second study say themselves that a more refined perspective on the benefits is needed.⁠^[2]

Why speaking with feedback works

At its core, a good AI tutor does two things that language-learning research has studied for a long time: it gets you speaking, and it gives you feedback on mistakes.

On corrective feedback there is a much-cited meta-analysis of 33 studies. It found a medium overall effect that held up on later tests. Interestingly, the effect of implicit corrections – for example when the other person simply repeats your sentence correctly – was better maintained than that of explicit ones.⁠^[3] For pronunciation training, too, a meta-analysis of 86 study reports found larger effects when learners received feedback.⁠^[5]

That suggests a chatbot that just chats helps less than one that corrects you specifically and has you say the correct sentence again. For AI tutors, though, this conclusion is not proven: the meta-analysis on corrective feedback dates from 2010 and does not look at AI tutors of the kind available today.

What about the fear of speaking?

A meta-analysis of 14,128 learners found a moderate negative association between foreign language classroom anxiety and achievement (r = −0.39 for general achievement).⁠^[4] That is an association, not proof that anxiety is the cause – poorer results can just as well make people anxious.

An AI has no audience, doesn't get impatient and doesn't laugh. For many people that makes it a comfortable first practice partner. But we did not find a reduction in speaking anxiety as an established result in the meta-analyses we read. So we don't claim it.

Where AI reaches its limits

• AI can be wrong. Language models phrase wrong explanations just as fluently and convincingly. For grammar rules or meanings that matter to you, check a second source.
• It doesn't replace people. Conversations with real people have surprises, dialects, humour and relationships – an AI can only imitate that to a limited degree.
• The research is young. The meta-analyses combine studies with very different bots, languages and learners; the technology has changed quickly since.