What researchers studied

Observational concurrent and predictive validity study comparing supervisor ratings, Graded Response Model rescaling, and GPT-4o-mini scoring of the same leader-authored school-improvement narratives. 1,385 principal-year observations from 103 Missouri school districts between 2016 and 2022, excluding 2020

What they found

  • Human supervisor and GRM scores were strongly correlated at 0.924, while LLM scores correlated only 0.116 with human scores and 0.074 with GRM scores.
  • The scoring approaches related to different outcomes: GRM scores showed some relationship with achievement and teacher surveys, while LLM scores uniquely tracked staffing stability, especially teacher retention.
  • Surface text features explained 72 percent of the variation in LLM scores, although the staffing-stability relationship remained after those features were controlled.
  • Higher human-based scores predicted worse staffing stability in the highest-minority and highest-FRPL schools, while the LLM scores did not show the same inversion; this does not establish that AI scoring was unbiased or superior.

What the study does not prove

  • The study used one Missouri evaluation system and leader-authored retrospective narratives, not direct observations of leadership behavior.
  • Each approach had a single rater or implementation, including one proprietary GPT-4o-mini version that may change over time.
  • The relationships with school outcomes were correlational and generally weaker in predictive models, so the scores should not be interpreted as causing later outcomes.
  • Teacher and student survey coverage was incomplete, and the text-feature analysis could capture only coarse properties of the narratives.

Evidence strength: Large multi-year observational validity study applying three scoring approaches to the same artifacts and several outcomes; single-state context, proprietary model dependence, self-authored narratives, and correlational outcome links limit generalization; working paper.

Why this matters for families

When a rating changes with the scoring method, ask what the number captures, what it misses, and what evidence sits closest to the learner.

Noor interpretation

How Noor translates the evidence into practice

A score is partly a product of what the scoring method is designed to notice. Families should treat a single rating as one signal and look for agreement across student work, growth, experience, continuity, and direct observation.

Noor Lyra can help families build a fuller learning picture from multiple forms of evidence rather than allowing one dashboard number, automated score, or evaluator judgment to define the student or the support plan.

Read the original source

Noor links to the original or authoritative source so families can distinguish the evidence itself from our interpretation.

Open original source →

DOI: 10.26300/bv8c-s810

Research notes

AI assessmentschool leadershipmeasurementteacher retentioneducation data

No single study determines a student's plan. Noor uses research as one input alongside the learner's goals, observed performance, academic context, and response to instruction.