Sampling Variability of Performance Assessments

In this article, performance assessments are cast within a sampling framework. More specifically, a performance assessment is viewed as a sample of student performance drawn from a complex universe defined by a combination of all possible tasks, occasions, raters, and measurement methods. Using generalizability theory, we present evidence bearing on the generalizability and convergent validity of performance assessments sampled from a range of measurement facets and measurement methods. Results at both the individual and school level indicate that task-sampling variability is the major source ofmeasurment error. Large numbers of tasks are needed to get a reliable measure of mathematics and science achievement at the elementary level. With respect to convergent validity, results suggest that methods do not converge. Students' performance scores, then, are dependent on both the task and method sampled.     

A context-specific detrimental effect of UCS preexposure on place conditioning with morphine

The effect of morphine preexposure on place conditioning with morphine was investigated. In the first experiment, five injections of 10 mg/kg morphine were administered to rats prior to place conditioning or taste-aversion training with morphine. Although this number of preexposures retarded taste-aversion learning, there was no effect on place conditioning. In subsequent experiments we investigated the role of context blocking in UCS preexposure in place conditioning. In one experiment, preexposure to five morphine injections prior to place conditioning resulted in a reduced place preference, compared with preexposure and place conditioning in different contexts. However, the overall detrimental effect of morphine preexposure was questionable, because the rats that were preexposed were only marginally different from those that were not preexposed. In a final experiment we examined the effect of a context change from preexposure to place conditioning with 15 preexposures and demonstrated a detrimental effect of preexposure on place conditioning that was context specific. These results support a role of classical conditioning in place-preference conditioning with morphine.     

Evaluation of Procedure-Based Scoring for Hands-On Science Assessment

This article evaluates a procedure-based scoring system for a performance assessment (an observed paper towels investigation) and a notebook surrogate completed by fifth-grade students varying in hands-on science experience. Results suggested interrater reliability of scores for observed performance and notebooks was adequate (>.80) with the reliability of the former higher. In contrast, interrater agreement on procedures was higher for observed hands-on performance (.92) than for notebooks (.66). Moreover, for the notebooks, the reliability of scores and agreement on procedures varied by student experience, but this was not so for observed performance. Both the observed-performance and notebook measures correlated less with traditional ability than did a multiple-choice science achievement test. The correlation between the two performance assessments and the multiple-choice test was only moderate (mean = .46), suggesting that different aspects of science achievement have been measured. Finally, the correlation between the observed-performance scores and the notebook scores was .83, suggesting that notebooks may provide a reasonable, albeit less reliable, surrogate for the observed hands-on performance of students.     

A comparison of pregnant women presenting for alcohol and other drug treatment by CPS status

OBJECTIVE: The purpose of this study was to describe the demographic, substance use, and treatment variables of alcohol and other drug (AOD) treatment seeking pregnant women. This study also compared women on these variables based on Child Protective Service (CPS) status. METHOD: Secondary data of all pregnant women with at least one child entering all publicly funded AOD treatment programs (N=678), in a southwest county, over a 3-year period were analyzed to examine the differences between pregnant women also involved with CPS and those not involved with CPS. RESULTS: Pregnant women reporting CPS involvement were similar to non-CPS women on demographic variables but differed on drug use and treatment variables. CPS women were more likely to report marijuana use as their primary problem drug, be mandated to treatment, attend day treatment and be released from treatment unsatisfactorily compared to the non-CPS pregnant women. Those without CPS involvement were more likely to report cocaine or crack as their primary drug, attend outpatient treatment and be found to have a satisfactory release from treatment compared to those with CPS involvement. Results from the multiple logistic regression analysis found that the significant predictors of CPS involvement were mandated status and unsatisfactory treatment release. CONCLUSIONS: Differences in treatment outcomes may be based on AOD agencies' difficulty in working with mandated clients and/or with the multiple demands placed by CPS on clients as part of their treatment plans. Further research is needed to determine reasons for CPS clients' high failure rates in AOD treatment.     

Using a Multilevel Assessment Scheme in Reforming Science Methods Courses

The development of new courses is strengthened by assessment and a response to the assessment. Two new science methods courses for elementary and secondary preservice teachers were developed, fostered by the Great Salt Lake Project. The preservice teachers designed and performed research projects that they then converted into inquiry-based teaching units. The professors applied a 3-layer assessment scheme to evaluate the courses and the preservice teachers based on changes in student attitudes, student-generated curricula, and internal course evaluations. Each type of assessment effectively informed the reform process, and strengths and weaknesses of the courses were revealed. Strengths of the courses included students learning to see science as a process and comprehending inquiry as a pedagogical approach. Weaknesses observed included a lack of understanding in three areas: the role of assessment, the need to address cultural issues, and the significance of scientific literacy.     

On the Stability of Performance Assessments

This study examined the stability of scores on two types of performance assessments, an observed hands-on investigation and a notebook surrogate. Twenty-nine sixth-grade students in a hands-on inquiry-based science curriculum completed three investigations on two occasions separated by 5 months. Results indicated that: (a) the generalizability across occasions for relative decisions was, on average, moderate for the observed investigations (.52) and the notebooks (.50); (b) the generalizability for absolute decisions was only slightly lower; (c) the major source of measurement error was the person by occasion (residual) interaction; and (d) the procedures students used to carry out the investigations tended to change from one occasion to the other.     

Measuring the Expertise of Paediatric Rehabilitation Therapists

This article describes the development of a classification system to measure the expertise levels of practicing paediatric rehabilitation therapists. Seventy‐five therapists from five disciplines (physical, occupational, speech‐language, behaviour, and recreational therapy) were involved, along with 170 peers, and 188 parents of children with disabilities. A cluster analysis of 10 indicators of expertise (derived from a battery of self‐completed, peer‐completed, and parent‐completed measures) indicated three levels of therapist expertise—novice, intermediate, and expert. Expertise level was related to various aspects of experience but not to therapy discipline. The classification system will be useful for the education and professional development of paediatric therapists, and for research on expertise in paediatric rehabilitation.     

Developmentally Effective Experiences for Promoting Self-Authorship

ABSTRACT—Many national studies have identified types of experiences that are associated with enhancing college students' learning. This study contributes to the small but growing body of research on transformative educational experiences that assist and enable college students to develop ways of understanding and being in the world that help them adapt and respond to life's complexities and prepare for future civic, occupational, and family roles. The focus of this exploratory study is on those experiences that had a positive impact on college students' development toward self-authorship ( Baxter Magolda, 2001 ; Kegan, 1994 ). Using interview data from the Wabash National Study of Liberal Arts Education, we analyzed 300 selected experiences from 174 students. We found that students' approaches to interpreting and understanding their experiences was the major student characteristic that affected their learning (the effect of the experience). We then mapped these findings onto a sequenced developmental curriculum designed to promote self-authorship.     

“How are running shoes assessed? A systematic review of characteristics and measurement tools used to describe running footwear”

Background: Many footwear characteristics are argued as risk factors for running related injuries (RRI). Several footwear assessment tools are available; however, their use in studies of RRI is unknown. Objective: This systematic review evaluated the characteristics and methods of assessing footwear in studies of RRI. Design: Five online databases were searched for studies on adult runners, in running style footwear, who experienced running-related pain or injury. The methodological quality of included articles was independently assessed by two raters using a modified Downs and Black checklist. Study and participant characteristics, footwear assessment tools used, and footwear characteristics reported were extracted for qualitative synthesis. Results: Twenty-four articles were included in the review. Low risk of bias was determined for 11 (44%) of the included studies. Twenty-eight different footwear characteristics were grouped into four categories: nomenclature, measurements, qualitative features, and subjective features. Fifteen different methods for assessing the 28 footwear characteristics were reported among the included studies. Only three methods were described previously, as valid and reliable. Conclusion: Differences in assessing footwear may mask the link between footwear characteristics and injury risk. Systematic footwear assessments and nomenclature are needed to evaluate the effects of footwear characteristics on RRI.