Expert-novice distinctions among educational evaluators: Dynamic conceptions of knowledge organization and problem representation

Date of Completion

January 2001

Keywords

Education, Tests and Measurements|Education, Educational Psychology|Psychology, Cognitive

Degree

Ph.D.

Abstract

The present study, operating from a dynamic, self-organizing systems perspective, sought to address whether or not expert and novice educational evaluators differ with respect to the organizational characteristics of their knowledge bases, and with respect to the problem representations they generate en route to providing a diagnostic analysis of the results of a student's educational evaluation. ^ Two expert and two novice participants were asked to think out loud as they read through the results of a student's educational evaluation and generated an analysis of the student's reading, writing, and mathematics achievement profiles. Think-aloud sessions were audio-taped, later segmented according to naturally occurring pauses in speech, coded according to an 11 category scheme, and entered into an SPSS database. ^ Frequency counts were run for each coding category and resulting protocols were subject to qualitative analysis in an effort to identify and explore emergent themes. Representational maps were generated using AutoCad 14, a drafting software, in order to provide a visual illustration of participants' problem representations, including those instances in which participants interpreted data, generated hypotheses based upon the data, and generated recursive-reasoning loops. ^ The results of this inquiry support that expert educational evaluators' knowledge bases are more extensive and organized than the knowledge bases of novices, thus allowing for more dynamic problem solving. Also, experts' problem representations are bound to deep structure features of the domain of educational evaluation, while novices' are bound to surface structure features of the domain, and are heavily data dependent. Experts' problem representation maps illustrated the recursive nature of their reasoning processes and the greater breadth and depth of their recursive searches. Because novices lacked the ability to reason abstractly about presented data, they were unable to make use of recursive reasoning to foster either problem comprehension or solution. ^ Results are consistent with recent, dynamic theories of cognition, and provide support for viewing problem representations as dynamic constructs. In addition, results support the need to further explore whether or not training and competency standards are sufficiently comprehensive for those individuals entrusted with the task of evaluating students for the presence of learning disabilities. ^

Link to Full Text

Share

COinS