Digital Module 14: Planning and Conducting Standard Setting

In this digital ITEMS module, Dr. Michael Bunch provides an in-depth, step-by-step look at how standard setting is done. It does not focus on any specific procedure or methodology (e.g., modified Angoff, bookmark, and body of work) but on the practical tasks that must be completed for any standard setting activity. Dr. Bunch carries the participant through every stage of the standard setting process, from developing a plan, through preparations for standard setting, conducting standard setting, and all the follow-up activities that must occur after standard setting in order to obtain the approval of cut scores and translate those cut scores into score reports. The digital module includes a 120-page manual, various ancillary files (e.g., PowerPoint slides, Excel workbooks, sample documents, and forms), links to datasets from the book Standard Setting (Cizek & Bunch, 2007), links to final reports from four recent large-scale standard setting events, quiz questions with formative feedback, and a glossary.     

Digital Module 18: Automated Scoring https://ncme.elevate.commpartners.com

In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open-ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows scores to be returned faster at lower cost. In the module, they discuss automated scoring from a number of perspectives. First, they discuss benefits and weaknesses of automated scoring, and what psychometricians should know about automated scoring. Next, they describe the overall process of automated scoring, moving from data collection to engine training to operational scoring. Then, they describe how automated scoring systems work, including the basic functions around score prediction as well as other flagging methods. Finally, they conclude with a discussion of the specific validity demands around automated scoring and how they align with the larger validity demands around test scores. Two data activities are provided. The first is an interactive activity that allows the user to train and evaluate a simple automated scoring engine. The second is a worked example that examines the impact of rater error on test scores. The digital module contains a link to an interactive web application as well as its R-Shiny code, diagnostic quiz questions, activities, curated resources, and a glossary.     

Digital Module 13: Monte Carlo Simulation Studies in Item Response Theory

In this digital ITEMS module, Dr. Brian Leventhal and Dr. Allison Ames provide an overview of Monte Carlo simulation studies (MCSS) in item response theory (IRT). MCSS are utilized for a variety of reasons, one of the most compelling being that they can be used when analytic solutions are impractical or nonexistent because they allow researchers to specify and manipulate an array of parameter values and experimental conditions (e.g., sample size, test length, and test characteristics). Dr. Leventhal and Dr. Ames review the conceptual foundation of MCSS in IRT and walk through the processes of simulating total scores as well as item responses using the two-parameter logistic, graded response, and bifactor models. They provide guidance for how to implement MCSS using other item response models and best practices for efficient syntax and executing an MCSS. The digital module contains sample SAS code, diagnostic quiz questions, activities, curated resources, and a glossary.     

数字式多功能车刀角度测量仪的研究

为了提高车刀几何角度的测量精度,在原有车刀角度测量仪的基础上,改进设计了一种数字式多功能车刀角度测量仪.该角度测量仪以单片机为控制核心,利用角度传感器为信息采集模块,通过模数转换模块,将所测车刀角度值以数字形式实时显示出来,可以方便的测量车刀几何角度.     

Digital Module 11: Bayesian Psychometric Modeling https://ncme.elevate.commpartners.com

In this digital ITEMS module, Dr. Roy Levy describes Bayesian approaches to psychometric modeling. He discusses how Bayesian inference is a mechanism for reasoning in a probability-modeling framework and is well-suited to core problems in educational measurement: reasoning from student performances on an assessment to make inferences about their capabilities more broadly conceived, as well as fitting models to characterize the psychometric properties of tasks. The approach is first developed in the context of estimating a mean and variance of a normal distribution before turning to the context of unidimensional item response theory (IRT) models for dichotomously scored data. Dr. Levy illustrates the process of fitting Bayesian models using the JAGS software facilitated through the R statistical environment. The module is designed to be relevant for students, researchers, and data scientists in various disciplines such as education, psychology, sociology, political science, business, health, and other social sciences. It contains audio-narrated slides, diagnostic quiz questions, and data-based activities with video solutions as well as curated resources and a glossary.     

A usability study of users' perceptions toward a multimedia computer‐assisted learning tool for neuroanatomy

This usability study evaluated users' perceptions of a multimedia prototype for a new e‐learning tool: Anatomy of the Central Nervous System: A Multimedia Course. Usability testing is a collection of formative evaluation methods that inform the developmental design of e‐learning tools to maximize user acceptance, satisfaction, and adoption. Sixty‐two study participants piloted the prototype and completed a usability questionnaire designed to measure two usability properties: program need and program applicability. Statistical analyses were used to test the hypothesis that the multimedia prototype was well designed and highly usable, it was perceived as: (1) highly needed across a spectrum of educational contexts, (2) highly applicable in supporting the pedagogical processes of teaching and learning neuroanatomy, and (3) was highly usable by all types of users. Three independent variables represented user differences: level of expertise (faculty vs. student), age, and gender. Analysis of the results supports the research hypotheses that the prototype was designed well for different types of users in various educational contexts and for supporting the pedagogy of neuroanatomy. In addition, the results suggest that the multimedia program will be most useful as a neuroanatomy review tool for health‐professions students preparing for licensing or board exams. This study demonstrates the importance of integrating quality properties of usability with principles of human learning during the instructional design process for multimedia products. Anat Sci Ed 2008. © 2008 American Association of Anatomists.     

Cultural practices in Italian family conversations: Verbal conflict between parents and preadolescents

This work studies verbal conflict between parents and preadolescents through analysis of family dinnertime conversations. We move from the assumption that verbal conflict is an educational instrument of interaction and a tool of socialization. The aim of the research is to analyze the participants' strategies in situations of conflict and to highlight the relations between parents and preadolescents in these specific situations. The data collection methodology is mainly ethnographical: videotape recordings of Italian middle class family dinners (the families were selected as being dual career ones with two children) were fully transcribed and codified following the approach of Conversation Analysis. The results of the quantitative (by the use of the log-linear models) and qualitative analyses confirm the importance of verbal conflict in family context as a particular cultural strategy of relationships between parents and preadolescents. The analytical data underline the relations between different aspects of conflict and the pragmatic implications for all family members at dinnertime conversations.     

Measurement properties as a possible cause of digital device effects on a standardized assessment of learning

Recent research found a substantial effect of digital device type on a computerized, standardized assessment of learning. Findings were consistent across grade and content domain. To investigate the source of these differences, the current research probed whether measurement differences or, equivalently, test bias due to device type could explain achievement or proficiency differences. We used 2018–2019 results on the Indiana Learning Evaluation Assessment Readiness Network (ILEARN). The data are census-based and cover grades 3 and 8 in the content domains of mathematics and English language arts (ELA). For our analysis, we use the root mean-squared deviation (RMSD) for detecting measurement differences across students that took ILEARN on different digital devices. Our findings suggest that few, if any, achievement differences that depend on the type of digital device can be explained by test bias. We discuss our findings in the context of multiple-group versus multiple indicator, multiple cause measurement models. Our findings suggest that digital device operates directly on the construct (math or ELA) rather than on the indicators that measure the construct. Finally, we conclude that a competing hypothesis – that differences are due to a digital device familiarity effect – remains plausible.     

Construction of life cycle assessment software system platform

There has not been life cycle assessment (LCA) software applicable to China's products. To faciliate the development of LCA software system in China, we built an object-oriented LCA software system plantform based on LCA international standards ISO 14040 to ISO 14043 and the commonness of diverse products. The system includes six modules: disassembly module, data collection module, arithmetic module, drawing module, database module and control module. It evaluates all economic and environmental impacts during the whole life cycle of a product. The integrated life cycle inventory database system allows dynamic expansion, which makes the effort required for data collection reduce with the increase of the system's application. With the improvement of human environmental-awareness, the demand of environment-friendly products is increasing, and ecological design of products has become a critical part in products development. Consummation of the LCA software system will provide a powerful tool for designing and developing Chinese ecological products.     

Digital Module 10: Rasch Measurement Theory https://ncme.elevate.commpartners.com

In this digital ITEMS module, Dr. Jue Wang and Dr. George Engelhard Jr. describe the Rasch measurement framework for the construction and evaluation of new measures and scales. From a theoretical perspective, they discuss the historical and philosophical perspectives on measurement with a focus on Rasch's concept of specific objectivity and invariant measurement. Specifically, they introduce the origins of Rasch measurement theory, the development of model‐data fit indices, as well as commonly used Rasch measurement models. From an applied perspective, they discuss best practices in constructing, estimating, evaluating, and interpreting a Rasch scale using empirical examples. They provide an overview of a specialized Rasch software program (Winsteps) and an R program embedded within Shiny (Shiny_ERMA) for conducting the Rasch model analyses. The module is designed to be relevant for students, researchers, and data scientists in various disciplines such as psychology, sociology, education, business, health, and other social sciences. It contains audio‐narrated slides, sample data, syntax files, access to Shiny_ERMA program, diagnostic quiz questions, data‐based activities, curated resources, and a glossary.     

Disabled readers' processing of prose: Do any processes change because of intervention?

A multiple baseline design was implemented to address the questions, “Do qualitative changes in reported strategies occur during treatment and does strategy instruction concurrently influence a conceptually related task?” The dependent measure for the prose recall task was the number of idea units recalled, while the number of facts comprehended during silent reading sessions assessed transfer effects. “Thinking aloud protocols” were used to identify cognitive processes during prose recall. The results suggested that a qualitative shift in verbal strategy reports occurred across training sessions. The introduction of cognitive training (visual and verbal mapping of idea units) increased the recall of prose compared to the baseline conditions, although concurrent effects on reading comprehension tasks were minimal. It was assumed that unstable cognitive processes during prose recall, as well as nonequivalent mental processes between the two tasks, accounted for the poor transfer of training effects.     

Eye-movement modeling of integrative reading of an illustrated text: Effects on processing and learning

Integrative processing of verbal and graphical information is crucial when students read an illustrated text to learn from it. This study examines the potential of a novel approach to support the processing of text and graphics. We used eye movement modeling example (EMME) in the school context to model students' integrative processes of verbal and pictorial information by replaying a model's gazes while reading an illustrated text on a topic different from that of the learning episode. Forty-two 7th graders were randomly assigned to an experimental (EMME) or a control condition (No-EMME) and were asked to read an illustrated science text about the food chain. Online measures of text processing and offline measures of reading outcomes were used. Eye-movement indices indicated that students in the EMME condition showed more integrative processing than students in the No-EMME condition. They also performed better than the latter in the verbal and graphical recall, and in the transfer task. Finally, the relationship between the duration of reprocessing the graphical segments while rereading the correspondent verbal segments and transfer performance was stronger in the EMME condition, after controlling for the individual differences of prior knowledge, reading comprehension, and achievement in science. Overall, the findings suggest the potential of eye-tracking methodology as an instruction tool.     

Associations between Secondary School Pupils' Definitions of Bullying,Attitudes towards Bullying,and Tendencies to Engage in Bullying: Age and sex differences

A self-report questionnaire about involvement in different types of bullying, what behaviours were regarded as bullying, and attitudes towards bullying, bullies and victims was completed by pupils in Year 7 (aged 11/12) through to Year 10 (aged 14/15) ( n = 170). Overall, direct verbal assault was the most commonly reported, and stealing the least frequently reported, type of bullying. For six specific types of bullying investigated, and for a composite measure of all types of bullying, significantly fewer Year 9 pupils than pupils in the other three years reported that they had behaved in these ways in the previous week. No significant sex differences emerged on these measures. These findings suggest that general patterns in bullying activities as a function of age and sex obtained in previous studies do not always hold. Although most pupils indicated that they thought that six out of eight types of behaviour viewed as bullying by researchers should be regarded as bullying, a substantial minority did not. The present study also extended bullying research by examining associations between pupils' definitions and attitudes towards bullying and their reports of bullying others. For one specific type of bullying, 'Forcing people to do things that they don't want to do', significantly fewer pupils who reported that they had behaved in this way than who reported that they had not done so included it in their definition of bullying. A consistent pattern of significant negative correlations of moderate size between attitudes and self-reported involvement in specific types of bullying were obtained. The implications of these findings for those concerned with tackling bullying in schools were discussed.     

Feelings towards Older vs. Younger Adults: Results from the European Social Survey

The study evaluated the association of modernization (at the macro/societal-level) and modernity (at the micro/individual-level) with feelings towards older vs. younger adults. Analysis was based on the fourth wave of the European Social Survey, which includes a rotated module on ageism. The sample consisted of 28 countries and a total of 54,988 respondents. The outcome was based on two affective items measuring explicit feelings towards older vs. younger adults. Hierarchical linear modeling was used in the analysis. A small portion of the variance was attributed to cross-country differences. There was a trend towards more positive feelings towards older adults compared to younger adults. The findings provide support to the claim that modernity is associated with reports of affective ageism, but only minimal support to a relationship between modernization and affective ageism.     

AIDA: An integrated authoring environment for educational software

This paper describes an integrated authoring environment, AIDA (Ambiente Integrado de Desenvolvimento de Aplicações educacionais). It is an open tool, giving the author a high degree of freedom both in package characteristics and design methodologies used. This environment integrates several modules. The core of the design module is a prototyping tool, allowing the creation of evolutionary prototypes. These prototypes can include several types of multimedia components and can be used as part of the final package. The AIDA environment also includes other options that support evaluation, learning materials re-use, communication between authoring team members and translation and cultural adaptation of packages to work in different environments. An experimental version of this authoring environment was implemented and the results of its evaluation are encouraging. In this paper we also present an example of a package created with the AIDA authoring environment.     

Digital Module 17: Data Visualizations: Effective Evidence-Based Practices https://ncme.elevate.commpartners.com

In this digital ITEMS module, Nikole Gregg and Dr. Brian Leventhal discuss strategies to ensure data visualizations achieve graphical excellence. Data visualizations are commonly used by measurement professionals to communicate results to examinees, the public, educators, and other stakeholders. To do so effectively, it is important that these visualizations communicate data efficiently and accurately. These visualizations can achieve graphical excellence when they simultaneously display data effectively, efficiently, and accurately. Unfortunately, measurement and statistical software default graphics typically fail to uphold these standards and are therefore not suitable for publication or presentation to the public. To illustrate best practices, the instructors provide an introduction to the graphical template language in SAS and show how elementary components can be used to make efficient, effective, and accurate graphics for a variety of audiences. The module contains audio-narrated slides, embedded illustrative videos, quiz questions with diagnostic feedback, a glossary, sample SAS code, and other learning resources.