Date of Completion

10-18-2016

Embargo Period

10-13-2016

Keywords

Conceptual knowledge, latent class analysis, LCA, LTA, Assessment, Diagnostic, Force, Physics

Major Advisor

H. Jane Rogers

Associate Advisor

D. Betsy McCoach

Associate Advisor

Hariharan Swaminathan

Field of Study

Educational Psychology

Degree

Doctor of Philosophy

Open Access

Open Access

Abstract

To uphold the core premise of cognitive diagnostic assessment, it is necessary to align different aspects of assessment design. The structure of the test, the measurement model, and the score reporting must align with each other and with the construct being targeted. Conceptual knowledge, as targeted by the Force and Motion Conceptual Evaluation (FMCE), can be modeled as a set of overlapping categorical states. As such, latent class analysis (LCA) is the most appropriate measurement model—where students have a probabilistic membership within classes that tend to match with observed mental models.

This dissertation focuses on one particular application of conceptual knowledge instruments: evaluating the effectiveness of instructional interventions within a controlled trials design. In these studies, students in different sections are taught using methods and assessed using the same instrument at pretest and posttest. Typically, the statistic of interest is the difference between the average changes in scores across the two time points. Given randomization and proper controls, researchers can use the results to make claims about which methods are more effective.

In the latent class framework, changes across time are captured with latent transition analysis (LTA) models, where transition parameters describe student posttest classes given membership in pretest classes. A multi-group LTA model can allow the transition parameters to vary across treatment groups while constraining measurement parameters. The difference in transitions from pretest to posttest across groups answers similar questions about the effectiveness of the instructional treatments, while providing more diagnostically relevant information.

The study described in this dissertation applies a multi-group LTA model to FMCE data from two large scale studies. The model was applied individually to each of the FMCE testlets, which focus on different concepts. This first application of latent class modeling to conceptual change was successful because many of the models converged, were fully identified, and provided interpretable results. Not every model converged, providing some clues about the limits of this method. The transition results agreed with conventional results that the instructional treatments were more effective than the more conventional instruction. However, it was difficult to find useful diagnostic information within the transition parameters.

Download

Share

COinS