Exploring the Influence of Judge Proficiency on Standard‐Setting Judgments

Setting performance standards is a judgmental process involving human opinions and values as well as technical and empirical considerations. Although all cut score decisions are by nature somewhat arbitrary, they should not be capricious. Judges selected for standard‐setting panels should have the proper qualifications to make the judgments asked of them; however, even qualified judges vary in expertise and in some cases, such as highly specialized areas or when members of the public are involved, it may be difficult to ensure that each member of a standard‐setting panel has the requisite expertise to make qualified judgments. Given the subjective nature of these types of judgments, and that a large part of the validity argument for an exam lies in the robustness of its passing standard, an examination of the influence of judge proficiency on the judgments is warranted. This study explores the use of the many‐facet Rasch model as a method for adjusting modified Angoff standard‐setting ratings based on judges’ proficiency levels. The results suggest differences in the severity and quality of standard‐setting judgments across levels of judge proficiency, such that judges who answered easy items incorrectly tended to perceive them as easier, but those who answered correctly tended to provide ratings within normal stochastic limits.     

On the relationships between bibliographic characteristics of scientific documents and citation and Mendeley readership counts: A large-scale analysis of Web of Science publications

In this paper we present a first large-scale analysis of the relationship between Mendeley readership and citation counts with particular documents’ bibliographic characteristics. A data set of 1.3 million publications from different fields published in journals covered by the Web of Science (WoS) has been analyzed. This work reveals that document types that are often excluded from citation analysis due to their lower citation values, like editorial materials, letters, news items, or meeting abstracts, are strongly covered and saved in Mendeley, suggesting that Mendeley readership can reliably inform the analysis of these document types. Findings show that collaborative papers are frequently saved in Mendeley, which is similar to what is observed for citations. The relationship between readership and the length of titles and number of pages, however, is weaker than for the same relationship observed for citations. The analysis of different disciplines also points to different patterns in the relationship between several document characteristics, readership, and citation counts. Overall, results highlight that although disciplinary differences exist, readership counts are related to similar bibliographic characteristics as those related to citation counts, reinforcing the idea that Mendeley readership and citations capture a similar concept of impact, although they cannot be considered as equivalent indicators.     

Will you be my friend? A multilevel network analysis of friendships of students with and without special educational needs backgrounds in inclusive classrooms

The objective of this study was to determine the relationship between having a special educational needs background (SEN) and the likelihood of having friends in inclusive classes. We assumed that a combination of individual, dyadic and contextual variables can sufficiently explain the relation between a SEN diagnosis and the likelihood of friendship. Data analysis was based on a cross-sectional sample of students (N = 1241) in second and third grade primary-school classes. To address the different levels adequately, the present study improves upon previous research in two ways: First, the sociometric data were analyzed with the p2 model, a specialized multilevel network model. Second, the study focused solely on friendships and emphasized the concept’s unique features with respect to inclusive education. Data analysis indicated that students with SEN had a decreased probability of becoming friends with their classmates compared to students without SEN. Even when individual, dyadic, and contextual variables were included into the model, the association between a SEN diagnosis and the likelihood of friendship persisted. The implications of the results are discussed with respect to their implications for inclusive teaching practice.     

Preschoolers’ Motivation to Over-Imitate Humans and Robots

From preschool age, humans tend to imitate causally irrelevant actions—they over-imitate. This study investigated whether children over-imitate even when they know a more efficient task solution and whether they imitate irrelevant actions equally from a human compared to a robot model. Five-to-six-year-olds (N = 107) watched either a robot or human retrieve a reward from a puzzle box. First a model demonstrated an inefficient (Trial 1), then an efficient (Trial 2), then again the inefficient strategy (Trial 3). Subsequent to each demonstration, children copied whichever strategy had been demonstrated regardless of whether the model was a human or a robot. Results indicate that over-imitation can be socially motivated, and that humanoid robots and humans are equally likely to elicit this behavior.     

Nonparametric Evidence of Validity,Reliability, and Fairness for Rater‐Mediated Assessments: An Illustration Using Mokken Scale Analysis

Numerous researchers have proposed methods for evaluating the quality of rater‐mediated assessments using nonparametric methods (e.g., kappa coefficients) and parametric methods (e.g., the many‐facet Rasch model). Generally speaking, popular nonparametric methods for evaluating rating quality are not based on a particular measurement theory. On the other hand, popular parametric methods for evaluating rating quality are often based on measurement theories such as invariant measurement. However, these methods are based on assumptions and transformations that may not be appropriate for ordinal ratings. In this study, I show how researchers can use Mokken scale analysis (MSA), which is a nonparametric approach to item response theory, to evaluate rating quality within the framework of invariant measurement without the use of potentially inappropriate parametric techniques. I use an illustrative analysis of data from a rater‐mediated writing assessment to demonstrate how one can use numeric and graphical indicators from MSA to gather evidence of validity, reliability, and fairness. The results from the analyses suggest that MSA provides a useful framework within which to evaluate rater‐mediated assessments for evidence of validity, reliability, and fairness that can supplement existing popular methods for evaluating ratings.     

Exploring the Effects of Rater Linking Designs and Rater Fit on Achievement Estimates Within the Context of Music Performance Assessments

When good model-data fit is observed, the Many-Facet Rasch (MFR) model acts as a linking and equating model that can be used to estimate student achievement, item difficulties, and rater severity on the same linear continuum. Given sufficient connectivity among the facets, the MFR model provides estimates of student achievement that are equated to control for differences in rater severity. Although several different linking designs are used in practice to establish connectivity, the implications of design differences have not been fully explored. Research is also limited related to the impact of model-data fit on the quality of MFR model-based adjustments for rater severity. This study explores the effects of linking designs and model-data fit for raters on the interpretation of student achievement estimates within the context of performance assessments in music. Results indicate that performances cannot be effectively adjusted for rater effects when inadequate linking or model-data fit is present.     

Looking at Eye Gaze Processing and Its Neural Correlates in Infancy—Implications for Social Development and Autism Spectrum Disorder

The importance of eye gaze as a means of communication is indisputable. However, there is debate about whether there is a dedicated neural module, which functions as an eye gaze detector and when infants are able to use eye gaze cues in a referential way. The application of neuroscience methodologies to developmental psychology has provided new insights into early social cognitive development. This review integrates findings on the development of eye gaze processing with research on the neural mechanisms underlying infant and adult social cognition. This research shows how a cognitive neuroscience approach can improve our understanding of social development and autism spectrum disorder.     

What is more important for fourth-grade primary school students for transforming their potential into achievement: the individual or the environmental box in multidimensional conceptions of giftedness?

Multidimensional models of giftedness specify individual and environmental moderators or catalysts that help transform potential into achievement. However, these models do not state whether the importance of the ‘individual boxes’ and the ‘environmental boxes’ changes during this process. The present study examines whether, during the early stages of talent development, the ‘environmental boxes’ play a more important role than the ‘individual boxes.’ To answer this question, we analyzed individual moderators and environmental moderators of achievement for fourth-grade primary-school students (N?=?976). A cluster analysis that included intelligence, achievement, and two individual moderators (motivation and learning behavior) revealed three groups of students, two of which are of particular interest, as they both displayed high intelligence and achievement but differed in their motivation and learning behavior. Questionnaire data on family environment (filled out by parents, N?=?682) and school environment (filled out by teachers, N?=?47) supported the assumption that among young students an inauspicious set of individual moderators, in this case maladaptive motivation and learning behavior, can be compensated by a sufficiently propitious set of environmental moderators, in this case parents’ and teachers’ learning support, cultural capital within the family, and teachers’ commitment to support their students’ learning.     

Clear Self,Better Relationships: Adolescents’ Self‐Concept Clarity and Relationship Quality With Parents and Peers Across 5 Years

This study examined reciprocal associations between adolescents’ self‐concept clarity (SCC) and their relationship quality with parents and best friends in a five‐wave longitudinal study from age 13 to 18 years. In all, 497 adolescents (57% boys) reported on their SCC and all informants (i.e., adolescents, both parents, and adolescents’ best friends) reported on support and negative interaction. Within‐person cross‐lagged analyses provided systematic evidence for both parent effects and child effects, with the direction of effects being strongly dependent on the relational context. For example, higher maternal support predicted higher adolescent SCC, supporting a parent effects perspective, whereas higher SCC predicted lower paternal negative interaction, supporting a child effects perspective. Peer effects on adolescent SCC were not consistently found across adolescent and best friend reports.     

Detecting Measurement Disturbances in Rater‐Mediated Assessments

The term measurement disturbance has been used to describe systematic conditions that affect a measurement process, resulting in a compromised interpretation of person or item estimates. Measurement disturbances have been discussed in relation to systematic response patterns associated with items and persons, such as start‐up, plodding, boredom, or fatigue. An understanding of the different types of measurement disturbances can lead to a more complete understanding of persons or items in terms of the construct being measured. Although measurement disturbances have been explored in several contexts, they have not been explicitly considered in the context of performance assessments. The purpose of this study is to illustrate the use of graphical methods to explore measurement disturbances related to raters within the context of a writing assessment. Graphical displays that illustrate the alignment between expected and empirical rater response functions are considered as they relate to indicators of rating quality based on the Rasch model. Results suggest that graphical displays can be used to identify measurement disturbances for raters related to specific ranges of student achievement that suggest potential rater bias. Further, results highlight the added diagnostic value of graphical displays for detecting measurement disturbances that are not captured using Rasch model–data fit statistics.