Beyond Gameplay: Item-Level Diagnostics and Psychometric Rigor of Resonance Odyssey Stealth Assessment

Abstract

Stealth assessment embeds educational measurement directly into gameplay to evaluate student competencies without disrupting the learning flow. However, ensuring the psychometric rigor of game-based instruments at the item level remains a crucial challenge. This study evaluated the psychometric properties and item-level diagnostics of a 30-item stealth assessment integrated into the educational board game Resonance Odyssey. Data were collected from 129 participants (N = 129) and analyzed using the Rasch measurement model via Winsteps 5.2.5.1. The analysis confirmed strong construct validity and unidimensionality, with 36.8% of the raw variance explained by the measure and a first residual contrast eigenvalue of 3.03 (6.4%). High internal consistency and item hierarchy were demonstrated by excellent item reliability (= 0.95, separation = 4.60) and good person reliability ( = 0.87, separation = 2.61). Wright map targeting showed an optimal alignment between student ability and task difficulty, indicated by a mean person measure of +0.03 logit. Local independence was fully satisfied, with maximum standardized residual correlation reaching only r = 0.47. Detailed item-level diagnostics identified four misfitting items (Q22, Q25, Q26, and Q30) characterized by high outfit mean square values (MNSQ > 3.0) and unexpected option endorsements by high-ability students. Overall, the findings demonstrate that while Resonance Odyssey provides a psychometrically rigorous framework for non-intrusive skill measurement, targeted revision of problematic distractors is essential to optimize its diagnostic precision.