What researchers studied

Two-year cluster randomized trial in 18 Tennessee middle schools during existing daily remedial mathematics sessions. Students in 18 Tennessee middle schools who were assigned to use Khan Academy with Khanmigo configured to coach rather than simply provide answers

What they found

  • Assignment to the Khan Academy plus Khanmigo condition raised math achievement by about 1.3 national percentile ranks per term, approximately 0.06 to 0.08 standard deviations over a school year.
  • The authors estimate that a full year of active participation corresponds to an effect of about 0.14 standard deviations.
  • The measured gains resembled those seen from Khan Academy practice without AI assistance, so the experiment does not support treating the gains as a clean estimate of Khanmigo's added value by itself.
  • Although 96% of students tried Khanmigo at least once, the median student messaged it on only about one third of practice days and used it in only 17% of exercise sessions in which they made a mistake.

What the study does not prove

  • This is an NBER working paper and should not be described as a peer-reviewed journal publication.
  • Students engaged with Khanmigo infrequently, which makes it difficult to isolate the incremental effect of the AI tutor from Khan Academy practice more broadly.
  • The study took place in remedial mathematics sessions in Tennessee middle schools, so the findings should not automatically be generalized to every age group, subject, or tutoring setting.
  • The 0.14 standard-deviation figure is an implied estimate for a full year of active participation, not the primary intent-to-treat effect for every student assigned to the program.

Evidence strength: Strong experimental evidence; NBER working paper.

Why this matters for families

The takeaway is not that AI tutoring works or does not work in isolation. It is that implementation and engagement matter. A sophisticated tool can still underperform when students rarely use it at the moments when help would be most valuable.

Noor interpretation

How Noor translates the evidence into practice

The study reinforces a distinction Noor considers important: giving a learner access to a strong tool is not the same as creating productive learning behavior. Students still need to recognize when they are stuck, ask for help, stay with the reasoning process, and use feedback after mistakes.

For Noor Lyra, AI is most useful when it sits inside a structured learning process rather than replacing one. That means guided mistake review, active prompting, educator follow-through, and attention to whether the student is actually engaging with support instead of merely having it available.

Read the original source

Noor links to the original or authoritative source so families can distinguish the evidence itself from our interpretation.

Open original source →

DOI: 10.3386/w35620

Research notes

KhanmigoKhan AcademyAI tutoringmiddle school mathrandomized trialstudent engagement

No single study determines a student's plan. Noor uses research as one input alongside the learner's goals, observed performance, academic context, and response to instruction.