The Impact of Vertical Scaling Decisions on Growth Interpretations

Most growth models implicitly assume that test scores have been vertically scaled. What may not be widely appreciated are the different choices that must be made when creating a vertical score scale. In this paper empirical patterns of growth in student achievement are compared as a function of different approaches to creating a vertical scale. Longitudinal item‐level data from a standardized reading test are analyzed for two cohorts of students between Grades 3 and 6 and Grades 4 and 7 for the entire state of Colorado from 2003 to 2006. Eight different vertical scales were established on the basis of choices made for three key variables: Item Response Theory modeling approach, linking approach, and ability estimation approach. It is shown that interpretations of empirical growth patterns appear to depend upon the extent to which a vertical scale has been effectively “stretched” or “compressed” by the psychometric decisions made to establish it. While all of the vertical scales considered show patterns of decelerating growth across grade levels, there is little evidence of scale shrinkage.     

Relationships between cognitive diagnosis, CTT, and IRT indices: an empirical investigation

Cognitive diagnosis models (CDMs) continue to generate interest among researchers and practitioners because they can provide diagnostic information relevant to classroom instruction and student learning. However, its modeling component has outpaced its complementary component??test construction. Thus, most applications of cognitive diagnosis modeling involve retrofitting of CDMs to assessments constructed using classical test theory (CTT) or item response theory (IRT). This study explores the relationship between item statistics used in the CTT, IRT, and CDM frameworks using such an assessment, specifically a large-scale mathematics assessment. Furthermore, by highlighting differences between tests with varying levels of diagnosticity using a measure of item discrimination from a CDM approach, this study empirically uncovers some important CTT and IRT item characteristics. These results can be used to formulate practical guidelines in using IRT- or CTT-constructed assessments for cognitive diagnosis purposes.     

ITEM BANKS: WHAT, WHY, HOW

In this paper, we discuss curricular implications of item banking and its practical value to teachers and students, and list a variety of working banks with their sources. We also review the psychometric basis of item banking, outline a family of computer programs for accomplishing banking, and give the equations necessary to build a bank. We conclude with a discussion of item quality control and examples of items that misfit because of miskeying, guessing, or carelessness.     

Using the Attribute Hierarchy Method to Make Diagnostic Inferences about Examinees’ Cognitive Skills in Critical Reading

The purpose of this study is to apply the attribute hierarchy method (AHM) to a subset of SAT critical reading items and illustrate how the method can be used to promote cognitive diagnostic inferences. The AHM is a psychometric procedure for classifying examinees’ test item responses into a set of attribute mastery patterns associated with different components from a cognitive model. The study was conducted in two steps. In step 1, three cognitive models were developed by reviewing selected literature in reading comprehension as well as research related to SAT Critical Reading. Then, the cognitive models were validated by having a sample of students think aloud as they solved each item. In step 2, psychometric analyses were conducted on the SAT critical reading cognitive models by evaluating the model‐data fit between the expected and observed response patterns produced from two random samples of 2,000 examinees who wrote the items. The model that provided best data‐model fit was then used to calculate attribute probabilities for 15 examinees to illustrate our diagnostic testing procedure.     

Synthesising Research in Education: a case‐study of getting it wrong,Part 2

Syntheses of research are influential in regard to subsequent research, policy and practice. They provide the empirical bases for applications for research grants, for higher‐degree dissertations and theses, and for individual and institutional research. They are used by policy makers in designing strategies for development, and they are used to guide practitioners in the enhancement of professional activity. They provide the contents of highly regarded publications in handbooks, encyclopaedias, and textbooks, and become the best‐known statements of the state of knowledge on the topics to which they are addressed. The processes by which syntheses of research are disseminated are equally important, because they determine which syntheses are available publicly for the above purposes. If the authors of the original sources are wrong in announcing their findings, if the synthesisers of that research do not identify the errors, if the referees of the syntheses submitted for publication are lax, if the editors of the reports of those referees are not on guard, then a potentially damaging synthesis can be released. As a result, whole programmes of misguided research, policy and practice can eventuate. The likelihood that a poor synthesis would survive the rigorous refereeing process employed by prestigious scholarly journals must be very small. Nevertheless, the consequences of such a mistake warrant contemplation. It could happen! Indeed, it appears to have happened! In what follows I will give further demonstration that it has happened. In the first part of this case study (Dunkin, 1995), I reported an evaluation of that part of Kagan's (1992) synthesis of research on teachers’ professional growth that concerned preservice teachers. In this paper, I will focus on the synthesis of research on first‐year and beginning teachers.     

Restrictive Stochastic Item Selection Methods in Cognitive Diagnostic Computerized Adaptive Testing

This paper proposes two new item selection methods for cognitive diagnostic computerized adaptive testing: the restrictive progressive method and the restrictive threshold method. They are built upon the posterior weighted Kullback‐Leibler (KL) information index but include additional stochastic components either in the item selection index or in the item selection procedure. Simulation studies show that both methods are successful at simultaneously suppressing overexposed items and increasing the usage of underexposed items. Compared to item selection based upon (1) pure KL information and (2) the Sympson‐Hetter method, the two new methods strike a better balance between item exposure control and measurement accuracy. The two new methods are also compared with Barrada et al.'s (2008) progressive method and proportional method.     

Allowing the Voices of Parents to Help Shape Teaching and Learning

The authors observe that the nature of knowledge is multidimensional, and that current efforts to shape curriculum are guided by patterns of cognitive interests. Building from an example related to teacher education, they argue that higher education experiences represent a clash between freedom and conformance that relates to these curricular interpretations. They discuss the impact of these ideas on teacher education curricula and posit a curriculum model that emphasizes depth and connectedness at all levels, calling for a reconceptualization of curriculum to bring about a more fulfilling learning process.     

How Assigned Faculty Mentors View their Mentoring Relationships: An Interview Study of Mentors in Medical Education

This qualitative study of 29 physician mentors reports their perspectives on mentoring medical students in a well‐respected medical school’s formal, assigned, longitudinal mentoring program that has a curricular component in the second year. Using a phenomenologic inductive approach, common themes identified centered on mentors’ relationships with their students and the characteristics of the relationships. Most mentors said their relationships solidified in the second year, where curricular responsibilities included teaching and evaluation. Mentors saw their roles change, yet relationships continued to develop as mentees advanced through medical school. They were surprised when students sought contact, support, and advice in years three and four of their programs. Mentors worried whether they “did enough” for their students, yet described their experiences and relationships as rewarding. Mentors subsequently found satisfaction in the continuing relationship during the clinical training years. The researchers note the importance of the curricular component for providing a purpose and structure for relationship development and in facilitating relationship development in this assigned mentoring program.     

Effects of Curricular Differences on Achievement Test Data at Item and Objective Levels

The objectives of this study were to examine the impact of different curricula on standardized achievement test scores at item and objective levels and to determine if different curricula generate different patterns of item factor loadings. School buildings from a middle-sized district were rated regarding the degree to which their curricula matched the content of the standardized test, and the actual textbook series used within each building (classroom) was determined. Covariate analyses of objective scores and plots and correlations of item p values indicated very small, nonsignificant differential effects across ratings and textbook series. Factor patterns indicated no curricular effects on large first factors. These findings parallel the results of a previous study conducted at the subtest level. We conclude that educators need not be unduly concerned about the impact of specific and generally small differences in curricular offerings within a district on standardized test scores or inferences to a broad content domain.     

Making on the Move: Mobility,Makerspaces, and Art Education

Combining the excitement from the maker movement and the novel creation of deployable makerspaces, we review the development of the Mobile Atelier for Kinaesthetic Education (MAKE) 3D. MAKE 3D is a mobile makerspace platform that can be deployed anywhere there is electricity to create a curricular spectacle of digital fabrication in particular additive manufacturing or what is more commonly referred to as 3D printing. Our project combines this notion of curricular spectacle and a mobile makerspace platform, to develop strategies in how to meet the novice user almost anywhere and to entice them into a series of hands‐on activities that would give them a range of knowledge and aptitude for additive techniques in digital fabrication. We review the component parts of our Material to Form curriculum and explore thematic connections between the maker movement and art education including STEAM and interdisciplinarity; design thinking and kinaesthetic learning; and place‐based education and the mobile platform. Informal practices in art education and the mobile makerspace advances forms of place and kinaesthetic learning. Similar curricular setups are therefore encouraged to reinforce and expand prior knowledge, broaden participation and provide an adaptable learning space for STEAM initiatives.     

A Measure of Counselor Competency

Counselor educators need to be able to demonstrate their effectiveness in training new counselors; however, currently few valid or reliable measures exist for assessing educators' impact. The authors describe the development of such an instrument, the Counseling Skills Scale. They began by revising an existing scale and then they solicited feedback from experts and a focus group. They used the instrument to compare beginning counselors‐in‐training with those who had completed a counseling skills course. Finally, they conducted an item analysis. A paired t test showed significant improvements in counseling skills (t = 4.51, p < .000) from pretest to posttest. Cronbach's alpha showed internal consistency to be .90.     

Digital Module 06: Bayesian Psychometrics—Posterior Predictive Model Checking https://ncme.elevate.commpartners.com

Drawing valid inferences from modern measurement models is contingent upon a good fit of the data to the model. Violations of model‐data fit have numerous consequences, limiting the usefulness and applicability of the model. As Bayesian estimation is becoming more common, understanding the Bayesian approaches for evaluating model‐data fit models is critical. In this instructional module, Allison Ames and Aaron Myers provide an overview of Posterior Predictive Model Checking (PPMC), the most common Bayesian model‐data fit approach. Specifically, they review the conceptual foundation of Bayesian inference as well as PPMC and walk through the computational steps of PPMC using real‐life data examples from simple linear regression and item response theory analysis. They provide guidance for how to interpret PPMC results and discuss how to implement PPMC for other model(s) and data. The digital module contains sample data, SAS code, diagnostic quiz questions, data‐based activities, curated resources, and a glossary.     

An Item‐Level Expected Classification Accuracy and Its Applications in Cognitive Diagnostic Assessment

Most of the existing classification accuracy indices of attribute patterns lose effectiveness when the response data is absent in diagnostic testing. To handle this issue, this article proposes new indices to predict the correct classification rate of a diagnostic test before administering the test under the deterministic noise input “and” gate (DINA) model. The new indices include an item‐level expected classification accuracy (ECA) for attributes and a test‐level ECA for attributes and attribute patterns, and both of them are calculated based solely on the known item parameters and Q ‐matrix. Theoretical analysis showed that the item‐level ECA could be regarded as a measure of correct classification rates of attributes contributed by an item. This article also illustrates how to apply the item‐level ECA for attributes to estimate the correct classification rate of attributes patterns at the test level. Simulation results showed that two test‐level ECA indices, ECA_I_W (an index based on the independence assumption and the weighted sum of the item‐level ECAs) and ECA_C_M (an index based on Gaussian Copula function that incorporates the dependence structure of the events of attribute classification and the simple average of the item‐level ECAs), could make an accurate prediction for correct classification rates of attribute patterns.     

The Philosophical Aspects of IRT Equating: Modeling Drift to Evaluate Cohort Growth in Large‐Scale Assessments

Calibration and equating is the quintessential necessity for most large‐scale educational assessments. However, there are instances when no consideration is given to the equating process in terms of context and substantive realization, and the methods used in its execution. In the view of the authors, equating is not merely an exhibit of the statistical methodology, but it is also a reflection of the thought process undertaken in its execution. For example, there is hardly any discussion in literature of the ideological differences in the selection of an equating method. Furthermore, there is little evidence of modeling cohort growth through an identification and use of construct‐relevant linking items’ drift, using the common item nonequivalent group equating design. In this article, the authors philosophically justify the use of Huynh's statistical method for the identification of construct‐relevant outliers in the linking pool. The article also dispels the perception of scale instability associated with the inclusion of construct‐relevant outliers in the linking item pool and concludes that an appreciation of the rationale used in the selection of the equating method, together with the use of linking items in modeling cohort growth, can be beneficial to the practitioners.     

School‐Industry twinning

The Non‐statutory Guidance for Information Technology Capability identifies the information technology (IT) co‐ordinator as key in implementing IT across the whole school, and defines the role as a member of the management team with policy and personnel rather than technical skills. This paper presents evidence from a survey of secondary school coordinators across an English local education authority and shows that they deal principally with technical management of the resource. They are allocated a low managerial status, despite being expected to produce and implement a whole school strategic policy. The discussion highlights that IT management requires complex curricular managerial skills, and recognition within school management structures.     

Feed-forward assessment,exemplars and peer marking: evidence of efficacy

The paper reports the findings of evaluative research that attempted to rigorously assess the efficacy of a feed-forward, formative assessment intervention. The aim was to improve participants’ conceptions of quality, and hence improve the quality of a complex piece of summative assessment, by asking them to mark exemplars produced by former students. Feed-forward assessment has theoretical support in the literature, but empirical confirmation has been slight. Research findings were encouraging. A statistical model incorporating feed-forward was developed which accounted for a large effect in the improvement of results for the summative item. Importantly, there was improvement across student ability levels. Students, in the main, made accurate judgements about different levels of exemplar quality, although they had some difficulty discerning different pathways to high-quality products. Qualitative analysis indicated improved student conceptions of coherence and integration in the summative piece.     

Ethnocentricity,Post Modernity and European Curricular Systems

This paper concerns educational curricula and the notions of European knowledge systems which they espouse or reflect. Its focus is derived from social and cultural theories of modernity and post‐modernity. The paper examines the emergence of European supernational‐ity as a political and subsequent educational and curricular construct. The wider ethnocentrism emerging from the European supernationality is examined. The aspects of wider international knowledge and culture which are excluded from this phenomenon are exemplified through a consideration of cultural relativism and its relevance to knowledge and curricular systems. A taxonomy is then offered of traditional, modern and post‐modern knowledge. The Eurocentric nature of traditional and modernistic knowledge and school knowledge is emphasised. An alternative post‐modern and internationalist perspective on knowledge systems and curriculum planning is offered. International aspects of European knowledge as reflected in curricular practice are considered with regard to their place in this taxonomy. The importance of a full recognition of the worth of all the knowledges and cultural systems represented in and influential on the European continent in the erosion of ethnocentrismo is stressed by way of conclusion.     

Trumping Shame by Blasts of Noise: Narcissism,Self‐Esteem,Shame, and Aggression in Young Adolescents

This experiment tested how self‐views influence shame‐induced aggression. One hundred and sixty‐three young adolescents (M = 12.2 years) completed measures of narcissism and self‐esteem. They lost to an ostensible opponent on a competitive task. In the shame condition, they were told that their opponent was bad, and they saw their own name at the bottom of a ranking list. In the control condition, they were told nothing about their opponent and did not see a ranking list. Next, participants could blast their opponent with noise (aggression measure). As expected, narcissistic children were more aggressive than others, but only after they had been shamed. Low self‐esteem did not lead to aggression. In fact, narcissism in combination with high self‐esteem led to exceptionally high aggression.     

Early career challenges in secondary school music teaching

The article reports an Economic and Social Research Council‐funded study of the early career experiences of secondary school music teachers in England, set within a wider national picture of decreasing age‐related pupil engagement with school music, career perceptions of music teaching, variable patterns of teacher recruitment and possible mismatches between the musical biographies of young people and intending music teachers. Qualitative and quantitative data were collected from a short‐term longitudinal survey (first questionnaire: n = 74, second questionnaire: n = 29), supplemented by case studies (n = 6) and open‐ended, written questions (n = 20). Analyses suggest that only a half of the newly qualified participants chose to teach full‐time in a mainstream, state‐funded school music classroom. Of these, the majority were faced with a range of early career challenges stemming from curricular, extra‐curricular and non‐curricular school expectations. These included the need to balance their existing musical performer identity with that of being a new teacher.     

Teacher adaptation of document-based history curricula: results of the Reading Like a Historian curriculum-use survey

Although most teachers adapt curriculum, we know little about teachers’ rationales for modifying materials, how these rationales align with actual modifications, nor whether any patterns exist in the modifications that teachers make. This is especially the case in history/social studies, where research on curriculum is scant and research on teacher adaptation of curriculum is virtually non-existent. This paper addresses that gap. We report the results of a large-scale survey on curriculum use with over 1900 history teachers. The online survey focused on how and why teachers use and adapt lesson materials from a free online history curriculum and prompted teachers to upload examples of curriculum materials they had modified. We found that individual differences among teachers correlated with particular types of modifications. Moreover, we found that teachers were motivated to modify materials to address their students’ needs, and that their modifications rarely affected the core structure – or theory of content – of the lessons. We argue that such alignment between teachers and curricular materials represents an example of curricular fit. We discuss what curricular design features may have contributed to the high level of curricular fit among users as well as the implications of this construct for curriculum implementation efforts across subject areas.