Using Differential Item Functioning Procedures to Explore Sources of Item Difficulty and Group Performance Characteristics

Statistics used to detect differential item functioning can also reflect differential strengths and weaknesses in the performance characteristics of population subgroups. In turn, item features associated with the differential performance patterns are likely to reflect some facet of the item task and hence its difficulty, that might previously have been overlooked. In this study, several item features were identified and coded for a large number of reading comprehension items from the two admissions testing programs. Item features included subject matter content, various properties of item structure, cognitive demand indicators, and semantic content (propositional analysis). Differential item functioning was evaluated for males and females and for White and Black examinees. Results showed a number of significant relationships between item features and indicators of differential item functioning—many of which were consistent across testing programs. Implications of the results for related areas of research are discussed.     

歧义格式及其分类

歧义格式是能生成歧义实例的抽象句法格式，歧义实例即通常所说的歧义结构，指含有多个认知意义的具体句法结构。歧义格式按其结构项可分为含常项歧义格式和不含常项歧义格式，按变项的类别可以分为词类变项歧义格式和句法成分类变项歧义格式，按歧义源的类别可分为成分源歧义格式和关系源歧义格式，按照单义式之间句法语义关系的异同可分为内部歧义格式和外部歧义格式，按照生产歧义实例的能力可分为强势歧义格式、弱势歧义格式和临界歧义格式。     

Sources of difficulty in assessment: example of PISA science items

ABSTRACT

The understanding of what makes a question difficult is a crucial concern in assessment. To study the difficulty of test questions, we focus on the case of PISA, which assesses to what degree 15-year-old students have acquired knowledge and skills essential for full participation in society. Our research question is to identify PISA science item characteristics that could influence the item’s proficiency level. It is based on an a-priori item analysis and a statistical analysis. Results show that only the cognitive complexity and the format out of the different characteristics of PISA science items determined in our a-priori analysis have an explanatory power on an item’s proficiency levels. The proficiency level cannot be explained by the dependence/independence of the information provided in the unit and/or item introduction and the competence. We conclude that in PISA, it appears possible to anticipate a high proficiency level, that is, students’ low scores for items displaying a high cognitive complexity. In the case of a middle or low cognitive complexity level item, the cognitive complexity level is not sufficient to predict item difficulty. Other characteristics play a crucial role in item difficulty. We discuss anticipating the difficulties in assessment in a broader perspective.     

Analysis of the Latin Square Task with linear logistic test models

The Latin Square Task (LST) was developed by Birney, Halford, and Andrews [Birney, D. P., Halford, G. S., & Andrews, G. (2006). Measuring the influence of cognitive complexity on relational reasoning: The development of the Latin Square Task. Educational and Psychological Measurement, 66, 146–171.] and represents a non-domain specific, language-free operationalization of Relational Complexity (RC-)Theory. The current study investigates the basic cognitive parameters and structure of LST as defined by RC-Theory, using IRT-based linear logistic test models (LLTM). 850 German school students completed 26 systematically designed LST items. Results support the notion of Rasch-scalability. LLTM analyses reveal that both operation complexity and number of operations affect item difficulty. It is shown how LLTM and its variants can provide substantial insights into cognitive solution processes and composition of item difficulty in relational reasoning in order to make item construction more efficient.     

Explicit and implicit confidence judgments and developmental differences in metamemory: an eye-tracking approach

In the present study, primary school children’s ability to give accurate confidence judgments (CJ) was addressed, with a special focus on uncertainty monitoring. In order to investigate the effects of memory retrieval processes on monitoring judgments, item difficulty in a vocabulary learning task (Japanese symbols) was manipulated. Moreover, as a first exploratory step to uncover fast and retrieval bound (implicit) monitoring processes that take place before explicit CJ are openly reported, fixation time allocation during recognition and monitoring was recorded with an eye-tracking device. Results revealed developmental progression in uncertainty (but not in certainty) monitoring between the age of 7 and 9 years. Differences in CJ across levels of item difficulty point to a substantial impact of retrieval processes on 9-yr-olds’ but not on 7-yr-olds’ monitoring. Eye-tracking data revealed an overall bias towards medium and high CJ, and confirmed evidence on developmental progression in monitoring skills.     

Effects of Item Modifications on Test Accessibility for Persistently Low-Performing Students with Disabilities

ABSTRACT

Construct-irrelevant cognitive complexity of some items in the statewide grade-level assessments may impose performance barriers for students with disabilities who are ineligible for alternate assessments based on alternate achievement standards. This has spurred research into whether items can be modified to reduce complexity without affecting item construct. This study uses a generalized linear mixed modeling analysis to investigate the effects of item modifications on improving test accessibility by reducing construct-irrelevant cognitive barriers for persistently low-performing fifth-grade students with cognitive disabilities. The results showed item scaffolding was an effective modification for both mathematics and reading. Other modifications, such as bolding/underlining of key words, hindered test performance for low-performing students. We discuss the findings’ potential impact on test development with universal design.     

A Mindfulness Experiential Small Group to Help Students Tolerate Ambiguity

This study explored the lived experiences of 20 counselors‐in‐training (CITs) in a mindfulness experiential small group. Using grounded theory, the authors described a 5‐dimensional model for navigating ambiguity. Findings suggest mindfulness training provides CITs self‐reflection skills and a greater ability to manage cognitive complexity.     

Test item linguistic complexity and assessments for deaf students

Linguistic complexity of test items is one test format element that has been studied in the context of struggling readers and their participation in paper-and-pencil tests. The present article presents findings from an exploratory study on the potential relationship between linguistic complexity and test performance for deaf readers. A total of 64 students completed 52 multiple-choice items, 32 in mathematics and 20 in reading. These items were coded for linguistic complexity components of vocabulary, syntax, and discourse. Mathematics items had higher linguistic complexity ratings than reading items, but there were no significant relationships between item linguistic complexity scores and student performance on the test items. The discussion addresses issues related to the subject area, student proficiency levels in the test content, factors to look for in determining a "linguistic complexity effect," and areas for further research in test item development and deaf students.     

Relationships between psychometric dimensions of item quality and student ratings of item relevancy and ambiguity

Students rated the quality of the items on a classroom test that had been taken previously. On the same test, psychometric item indices were calculated. The results showed that the student ratings were related to the item difficulty, but not to the item-test correlation. In addition, the better-achieving students tended to rate the items as less ambiguous. Finally, the ambiguity ratings were more highly related to the item-test correlations for the better achieving students. These findings support opinions held by many instructors of students' judgments of item quality.     

刻意曲解的语用分析

刻意曲解是一种特殊的语用策略,是语言使用者有意利用对方话语中的歧义性和语用模糊歪曲对方的话语意图,以此来实现某种交际目的。文章主要从间接言语行为、面子理论、合作原则和关联理论等方面对刻意曲解进行语用分析。     

Taming a beast of burden – On some issues with the conceptualisation and operationalisation of cognitive load

Research on cognitive load theory (CLT) has not yet provided facet-specific measures of cognitive load. The lack of valid methods to measure intrinsic, extraneous and germane cognitive load makes it difficult to empirically test theoretical explanations of effects caused by manipulations of instructional designs. This situation also imposes challenges to testing CLT as a theory. This paper critically reflects the conceptualisation of CLT's core concept and the implications for its operationalisation. In order to address some of the challenges we propose a complexity framework that allows the derivation of a priori estimates of mental load that go beyond CLT's notion of element interactivity. In a study we test hypotheses with regard to effects of the variation of sources for intrinsic cognitive load (increase of complexity within tasks) and the variation of sources for extraneous cognitive load (reduction of extraneous cognitive load between tasks) in three ability groups. Complexity-based estimates prove superior to element interactivity-based estimates of mental load in the prediction of performance outcomes. Results also indicate that individual differences in information-processing capacity determine to what extent complexity is reflected as cognitive load. In this respect the proposed framework extends the focus of CLT beyond the discussion of the role of prior knowledge and acquired levels of expertise.     

会话交际中的刻意曲解策略及其关联解释

刻意曲解与语用误解不同。刻意曲解是语言使用者的一种语用策略,是其为了达到某种交际目的,有意利用对方话语中的歧义性和语用模糊,歪曲对方的话语意图。本文在区分这两种语言现象的基础上,尝试分析曲解者如何实施其刻意曲解策略,以及用关联理论如何来解释交际双方曲解与被曲解的过程。     

汉语句法歧义消解的认知机制

句法结构体的意义由词语的意义和句法结构形成的意义共同构成,其意义较为灵活,它随句法结构的变化而变化。对有歧义的句法结构所存在的歧义情况的认知,反映在激活速度和抑制程度上,它们由认知难度决定,其基本规律是,歧义的认知激活速度与认知难度成反比,抑制度与认知难度成正比。认知消解歧义的能力除与认知者本身的语言能力有关外,还与句法结构的特点有关。     

Dimensionality in Compensatory MIRT When Complex Structure Exists: Evaluation of DETECT and NOHARM

This study investigated the effect of complex structure on dimensionality assessment in compensatory multidimensional item response models using DETECT- and NOHARM-based methods. The performance was evaluated via the accuracy of identifying the correct number of dimensions and the ability to accurately recover item groupings using a simple matching similarity (SM) coefficient. The DETECT-based methods yielded higher proportion correct than the NOHARM-based methods in two- and three-dimensional conditions, especially when correlations were ≤.60, data exhibited ≤30% complexity, and sample size was 1,000. As the complexity increased and the sample size decreased, the performance of the methods typically diminished. The NOHARM-based methods were either equally successful or better in recovering item groupings than the DETECT-based methods and were mostly affected by complexity levels. The DETECT-based methods were affected largely by the test length, such that with the increase of the number of items, SM coefficients would decrease substantially.     

Motivation and metacognition when learning a complex system

Our cognitive-motivational process model (Vollmeyer & Rheinberg, 1998) assumes that motivational factors (i.e., mastery confidence, incompetence fear, interest, and challenge) affect performance via mediators. Previous studies (Vollmeyer, Rollett, & Rheinberg, 1997) found that strategy systematicity and motivational state during learning mediate the impact of initial motivation on the learning of a complex system. Potential mediators could be other cognitive (e.g., hypothesis testing) and metacognitive aspects, in that more motivated learners (high mastery confidence, low incompetence fear, high interest) analyse more deeply. Verbal protocols from 44 students who learnt to control a complex dynamic system were collected. We measured their initial motivation (on the four factors specified), then during learning we assessed their strategy systematicity and motivational state. Additionally, we analysed the verbal protocols to obtain indicators of learners’ cognitive and metacognitive processes. Performance measures were levels of knowledge acquisition and application. The cognitive-motivational process model was replicated. Qualitative cognitive aspects were added as mediators, however, the results for metacognition were problematic, partly because participants gave relatively few clearly expressed metacognitive statements.     

Student perceptions of communication skills in undergraduate science at an Australian research-intensive university

Higher education institutions globally are acknowledging the need to teach communication skills. This study used the Science Student Skills Inventory to gain insight into how science students perceive the development of communication skills across the degree programme. Responses were obtained from 635 undergraduate students enrolled in a Bachelor of Science at an Australian research-intensive university. Students rated their perceptions of two communication skills, scientific writing and oral scientific communication, across the following indicators: importance of, and improvement in, developing communication skills; the extent to which communication skills were included and assessed in the degree; confidence in using communication skills; and belief of future use of communication skills. While the majority of students perceived both communication skills to be important and of use in the future, their perceptions of the extent to which those skills were included and assessed were less, with oral communication being included and assessed less than scientific writing skills. Significant differences among year levels were discerned for most indicators, signifying a lack of coherent opportunities for students to learn and develop these skills across year levels. Results are discussed through the lens of progressive development of complex learning outcomes, with suggested areas for curriculum development and future research.     

Automatic Item Generation: A More Efficient Process for Developing Mathematics Achievement Items?

The continual supply of new items is crucial to maintaining quality for many tests. Automatic item generation (AIG) has the potential to rapidly increase the number of items that are available. However, the efficiency of AIG will be mitigated if the generated items must be submitted to traditional, time‐consuming review processes. In two studies, generated mathematics achievement items were subjected to multiple stages of qualitative review for measuring the intended skills, followed by empirical tryout in operational testing. High rates of success were found. Further, items generated from the same item structure had predictable psychometric properties. Thus, the feasibility of a more limited and expedient review processes was supported. Additionally, positive results were obtained on measuring the same skills from item structures with reduced cognitive complexity.     

Cognitive Diagnostic Multistage Testing by Partitioning Hierarchically Structured Attributes

In cognitive diagnostic models (CDMs), a set of fine-grained attributes is required to characterize complex problem solving and provide detailed diagnostic information about an examinee. However, it is challenging to ensure reliable estimation and control computational complexity when The test aims to identify the examinee's attribute profile in a large-scale map of attributes. To address this problem, this study proposes a cognitive diagnostic multistage testing by partitioning hierarchically structured attributes (CD-MST-PH) as a multistage testing for CDM. In CD-MST-PH, multiple testlets can be constructed based on separate attribute groups before testing occurs, which retains the advantages of multistage testing over fully adaptive testing or the on-the-fly approach. Moreover, testlets are offered sequentially and adaptively, thus improving test accuracy and efficiency. An item information measure is proposed to compute the discrimination power of an item for each attribute, and a module assembly method is presented to construct modules anchored at each separate attribute group. Several module selection indices for CD-MST-PH are also proposed by modifying the item selection indices used in cognitive diagnostic computerized adaptive testing. The results of simulation study show that CD-MST-PH can improve test accuracy and efficiency relative to the conventional test without adaptive stages.     

The use of keywords for delivering immediate performance feedback on teacher competence development

Literature shows that feedback that is specific, immediate and goal-oriented is effective on (pre-service) teachers’ performance. Synchronous coaching gives this kind of feedback. Due to immediateness of feedback, pre-service teachers can suffer from cognitive load. We propose a set of standardised keywords through which this performance feedback can be delivered – each keyword acts as a summary for the feedback message. The construction and the selection of the keywords is aimed at the reduction of message ambiguity, while at the same time a low level of cognitive load on the pre-service teacher must be maintained. An in vivo pilot-study with 40 respondents (pre-service teachers and their coaches) supported our hypothesis that usage of such sets of standardised keywords will mitigate the levels of ambiguity and cognitive load. These findings and other considerations for additional research using immediate performance are addressed.     

外语学习策略的归类及其存在的问题

外语学习策略归类问题是策略研究领域五个有争议的基本问题之一。策略归类系统归纳为七个子系统:1)从外语学习成功者的视角;2)从认知心理学的视角;3)从认知心理和语言的双视角;4)从策略运用目的的视角;5)从策略对学习影响程度的视角;6)从宏观和微观的视角;7)从英语教学的视角。目前策略归类还存在五个方面的问题:1)策略归类重叠;2)策略界定模糊;3)归类具有任意性;4)忽视情感策略;5)缺乏语言学科的独特性。这些问题严重阻碍了外语学习策略的研究。