DETECTING PARENTAL DECEPTION USING A BEHAVIOR RATING SCALE DURING ASSESSMENT OF ATTENTION‐DEFICIT/HYPERACTIVITY DISORDER: AN EXPERIMENTAL STUDY

It is often assumed that parents completing behavior rating scales during the assessment of attention‐deficit/hyperactivity disorder (ADHD) can deliberately manipulate the outcomes of the assessment. To detect these actions, items designed to detect over‐reporting or under‐reporting of results are sometimes embedded in such rating scales. This study presents the results of an experimental study in which parents (a) read a scenario telling them that their hypothetical son's teacher has suggested their son may have ADHD and (b) considered assigned goals for the assessment. Parents then completed the accompanying Conners 3Parent Short form (Conners 3‐P[S]) in a manner that they believed would achieve their assigned goals. Findings showed that parents are able to engage in deception when completing behavior rating scales. The validity scales embedded in the Conners 3‐P(S), however, demonstrated mixed results for detecting parental deception with the Negative Impression validity scale, accurately detecting attempts to malinger in the majority of cases, whereas the Positive Impression validity scale appears to have little to no diagnostic utility for the detection of defensive responding. Clinicians utilizing behavior rating scales should carefully consider results, and nonresults, obtained from embedded validity scales when interpreting parent responses to behavior rating scales as part of an ADHD assessment.     

Attention‐Deficit/Hyperactivity Disorder (ADHD): A national survey of training and current assessment practices in the schools

The primary purpose of this research was to survey school psychologists to investigate their training and current assessment practices for attention‐deficit/hyperactivity disorder (ADHD) in the schools. The survey consisted of 38 questions regarding participant characteristics and three main areas relevant to ADHD: (a) training, (b) caseloads/referral patterns, and (c) assessment. The respondents reported receiving adequate training in the assessment of ADHD, with doctorate‐level psychologists self‐reporting being better trained than nondoctorate psychologists. Results confirmed a substantial caseload of ADHD referrals. In the assessment of ADHD, the results indicated school psychologists are using multiple informants, methods, and settings for the assessment of ADHD with rating scales, observations, and interviews the most common methods identified. Limitations of current practices will be discussed. © 2003 Wiley Periodicals, Inc. Psychol Schs 40: 583–597, 2003.     

Nonparametric Evidence of Validity,Reliability, and Fairness for Rater‐Mediated Assessments: An Illustration Using Mokken Scale Analysis

Numerous researchers have proposed methods for evaluating the quality of rater‐mediated assessments using nonparametric methods (e.g., kappa coefficients) and parametric methods (e.g., the many‐facet Rasch model). Generally speaking, popular nonparametric methods for evaluating rating quality are not based on a particular measurement theory. On the other hand, popular parametric methods for evaluating rating quality are often based on measurement theories such as invariant measurement. However, these methods are based on assumptions and transformations that may not be appropriate for ordinal ratings. In this study, I show how researchers can use Mokken scale analysis (MSA), which is a nonparametric approach to item response theory, to evaluate rating quality within the framework of invariant measurement without the use of potentially inappropriate parametric techniques. I use an illustrative analysis of data from a rater‐mediated writing assessment to demonstrate how one can use numeric and graphical indicators from MSA to gather evidence of validity, reliability, and fairness. The results from the analyses suggest that MSA provides a useful framework within which to evaluate rater‐mediated assessments for evidence of validity, reliability, and fairness that can supplement existing popular methods for evaluating ratings.     

Diagnosis and treatment children and adolescents with autism and ADHD

Attention-deficit/hyperactivity disorder (ADHD) and autism are neurodevelopmental disorders that emerge in childhood. There is increasing recognition that ADHD and autism frequently co-occur. Yet, questions remain among clinicians regarding the best ways to evaluate and treat co-occurring autism and ADHD. This review outlines issues relevant to providing evidence-based practice to individuals and families who may be experiencing difficulties associated with co-occurring autism and ADHD. After describing the complexities of the co-occurrence of autism and ADHD, we present practical considerations for best practice assessment and treatment of co-occurring autism and ADHD. Regarding assessment, this includes considerations for interviewing parents/caregivers and youth, using validated parent and teacher rating scales, conducting cognitive assessments, and conducting behavior observations. Regarding treatment, consideration is given to behavioral management, school-based interventions, social skills development, and the use of medications. Throughout, we note the quality of evidence that supports a particular component of assessment or treatment, highlighting when evidence is most relevant to those with co-occurring autism and ADHD across stages of development. In light of the current evidence for assessment and treatment of co-occurring autism and ADHD, we conclude by outlining practical implications for clinical and educational practice.     

Quantitative analysis of the rubric as an assessment tool: an empirical study of student peer‐group rating

Although the rubric has emerged as one of the most popular assessment tools in progressive educational programs, there is an unfortunate dearth of information in the literature quantifying the actual effectiveness of the rubric as an assessment tool in the hands of the students. This study focuses on the validity and reliability of the rubric as an assessment tool for student peer‐group evaluation in an effort to further explore the use and effectiveness of the rubric. A total of 1577 peer‐group ratings using a rubric for an oral presentation was used in this 3‐year study involving 107 college biology students. A quantitative analysis of the rubric used in this study shows that it is used consistently by both students and the instructor across the study years. Moreover, the rubric appears to be ‘gender neutral’ and the students' academic strength has no significant bearing on the way that they employ the rubric. A significant, one‐to‐one relationship (slope = 1.0) between the instructor's assessment and the students' rating is seen across all years using the rubric. A generalizability study yields estimates of inter‐rater reliability of moderate values across all years and allows for the estimation of variance components. Taken together, these data indicate that the general form and evaluative criteria of the rubric are clear and that the rubric is a useful assessment tool for peer‐group (and self‐) assessment by students. To our knowledge, these data provide the first statistical documentation of the validity and reliability of the rubric for student peer‐group assessment.     

School-based assessment of attention deficit-hyperactivity disorder

Schools are uniquely situated to provide information relevant to the assessment and treatment of Attention Deficit-Hyperactivity Disorder (ADHD) across a variety of tasks, settings, and observers. The importance of the school setting to the assessment and treatment of ADHD has resulted in the development of numerous measures appropriate for such assessment, but few guidelines for their use. The purpose of this article is to describe school-based assessment procedures and their relationship to ADHD. These procedures include teacher rating measures, direct observations in classrooms and on playgrounds, peer rating and sociometric measures, and permanent product measures such as academic performance. It is recommended that school-based assessment of ADHD involve the concurrent use of multiple measures to assess adequately the wide variety of symptoms associated with this disorder. It is further recommended that assessment serve the purpose of determining treatment efficacy to avoid unwarranted preoccupation with diagnostic issues that are as yet unresolved. The components of a comprehensive school-based assessment of ADHD are described, and their relationship to school-based treatment is discussed.     

Multi-Method Assessment of ADHD Characteristics in Preschool Children: Relations between Measures

Several forms of assessment tools, including behavioral rating scales and objective tests such as the Continuous Performance Test (CPT), can be used to measure inattentive and hyperactive/impulsive behaviors associated with Attention-Deficit/Hyperactivity Disorder (ADHD). However, research with school-age children has shown that the correlations between parent ratings, teacher ratings, and scores on objective measures of ADHD-characteristic behaviors are modest at best. In this study, we examined the relations between parent and teacher ratings of ADHD and CPT scores in a sample of 65 preschoolers ranging from 50 to 72 months of age. No significant associations between teacher and parent ratings of ADHD were found. Parent-ratings of both inattention and hyperactivity/impulsivity accounted for variance in CPT omission errors but not CPT commission errors. Teacher ratings showed evidence of convergent and discriminant validity when entered simultaneously in a hierarchical regression. These tools may be measuring different aspects of inattention and hyperactivity/impulsivity.     

Psychometric evaluation and comparison of three retrospective, multi-item measures of childhood sexual abuse

OBJECTIVE: To evaluate and compare the psychometric qualities of three retrospective, multi-item measures of childhood sexual abuse (CSA): the CSA Frequency Scale, CSA Count Index, and CSA Multiple Characteristics Index. METHODS: Two samples of women 20-50 years old who experienced CSA were recruited from a family practice clinic (N=132) and the community (N=19). The measures were designed to represent the three most commonly used approaches for multi-item measurement of past CSA, as determined by a review of literature. Items were derived from a self-administered instrument from a larger study. The CSA Frequency Scale was evaluated for test-retest reliability, internal consistency, and criterion-related validity. The CSA Count and CSA Multiple Characteristics Indexes were evaluated for content specification, indicator specification, indicator collinearity, external validity, criterion-related validity, and test-retest reliability. Criterion-related validity variables were (a) physical and psychosocial symptomatology and (b) depression. RESULTS: Upon evaluation, all three measures met criteria for satisfactory quality, with one exception: external validity for both indexes. When compared across test-retest reliability and criterion-related validity results, none of the measures proved superior. Intraclass correlation coefficients for test-retest reliability ranged from .92 to .94; correlations with the criterion-related validity variables ranged from r=.36 to .41. CONCLUSIONS: The results support the continued use of established scales similar to the CSA Frequency scale. In addition, the results suggest further development and standardization of indexes similar to the CSA Count and Multiple Characteristics Indexes is warranted.     

Critical review of the TEMAS: A step within the development of thematic apperception instruments

The TEMAS (acronym for Tell‐Me‐a‐Story)—an objectively scored, projective thematic personality instrument for children and adolescents—is analyzed, reviewed, and critiqued with regard to theoretical underpinnings and rationale for development, administration, scoring, psychometric properties, and research to date. The TEMAS appears to be an improvement over existing projective personality measures used by school psychologists. Although it requires more training than other projective techniques, competency in administration, scoring, and interpretation can be achieved within a one semester course in personality assessment. The test has evidence of reliability and validity, and it is a multicultural alternative to the TAT and other thematic apperception instruments. The use of the TEMAS by psychologists may achieve more accurate assessment of Black and Hispanic children. Limitations include geographically limited standardization samples and little research conducted by individuals other than the authors. © 1999 John Wiley & Sons, Inc.     

Building Validity Evidence for Scores on a State-Wide Alternate Assessment: A Contrasting Groups, Multimethod Approach

The relationships between ratings on the Idaho Alternate Assessment (IAA) for 116 students with significant disabilities and corresponding ratings for the same students on two norm-referenced teacher rating scales were examined to gain evidence about the validity of resulting IAA scores. To contextualize these findings, another group of 54 students who had disabilities, but were not officially eligible for the alternate assessment also was assessed. Evidence to support the validity of the inferences about IAA scores was mixed, yet promising. Specifically, the relationship among the reading, language arts, and mathematics achievement level ratings on the IAA and the concurrent scores on the ACES-Academic Skills scales for the eligible students varied across grade clusters, but in general were moderate. These findings provided evidence that IAA scales measure skills indicative of the state's content standards. This point was further reinforced by moderate to high correlations between the IAA and Idaho State Achievement Test (ISAT) for the not eligible students. Additional evidence concerning the valid use of the IAA was provided by logistic regression results that the scores do an excellent job of differentiating students who were eligible from those not eligible to participate in an alternate assessment. The collective evidence for the validity of the IAA scores suggests it is a promising assessment for NCLB accountability of students with significant disabilities. The methods of establishing this evidence have the potential to advance validation efforts of other states' alternate assessments.     

多侧面模式在教育测试中的应用研究

主观测试实施过程中,由于存在多种因素导致最终测试结果的信度和效度降低,因此,对影响测试信度和效度各种因素的发现和分析就显得格外重要.本文主要介绍基于试题反应理论的多侧面模式产生背景、基本框架、在国内外教育测评上的典型应用以及此模式的局限性,从而说明多侧面模式作为一种新的测评模式,可以较全面地找出影响测试信度和效度的因素,特别是评分员主观效应因素,并能够对其进行客观分析.近年来,该模式在国内外教育测评上的应用也越来越广泛.     

Alternate Assessments of Students with Significant Disabilities: Alternative Approaches,Common Technical Challenges

This article examines three typical approaches to alternate assessment for students with significant cognitive disabilities—portfolios, performance assessments, and rating scales. A detailed analysis of common and unique design features of these approaches is provided, including features of each approach that influence the psychometric quality of their results. Validity imperatives for alternate assessments are reviewed, and approaches for addressing the need for validity evidence are outlined. The article concludes with an examination of three technical challenges—alignment, scores and scoring, and standard setting—common to all alternate assessments. In light of these challenges, existing methods and professional testing standards are endorsed as necessary guidance for understanding and advancing alternate assessment practices.     

The assessment of lesson plans in teacher education: a case study in assessment validity and reliability

This paper forms part of an exploration of assessment on one part‐time higher education (HE) course: an in‐service, professional qualification for teachers and trainers in the learning and skills sector which is delivered on a franchise basis across a network of further education colleges in the north of England. This paper proposes that the validity and reliability of portfolio‐based assessment, a key component of many HE programmes in addition to the course being researched here, is contestable. Analysis of the processes of compiling portfolios for assessment, through the conceptual framework of the New Literacy Studies, suggests that the ways in which portfolios are assessed and the ways in which the crucial requisites of validity and reliability are assigned to them, mask complexities and contradictions in their creation by the student. This paper argues for a new, critical analysis of portfolio production and raises a number of questions about the validity, reliability and authenticity of the assessment process that the portfolios reify.     

Using Rasch Analysis to Inform Rating Scale Development

The use of surveys, questionnaires, and rating scales to measure important outcomes in higher education is pervasive, but reliability and validity information is often based on problematic Classical Test Theory approaches. Rasch Analysis, based on Item Response Theory, provides a better alternative for examining the psychometric quality of rating scales and informing scale improvements. This paper outlines a six-step process for using Rasch Analysis to review the psychometric properties of a rating scale. The Partial Credit Model and Andrich Rating Scale Model will be described in terms of the pyschometric information (i.e., reliability, validity, and item difficulty) and diagnostic indices generated. Further, this approach will be illustrated through the example of authentic data from a university-wide student evaluation of teaching.     

Discriminative validity of the behavior assessment system for children‐parent rating scales in children with recurrent abdominal pain and matched controls

Examined discriminative validity of the Parent Rating Scale (PRS) of the Behavior Assessment System for Children (BASC; Reynolds & Kamphaus, 1992, Circle Pines, MN: American Guidance Services). Two groups were compared: a cohort with recurrent abdominal pain (RAP) (n= 49) and children from the BASC‐PRS standardization sample (n = 49) matched on the background characteristics of age, race/ethnicity, and gender. A multivariate, two‐group discriminant function analysis was used to compare groups across standard scores from the nine clinical scales of the PRS. Results demonstrated that children with RAP could be differentiated (Wilks λ = .642, F = 6.45, df (9, 88), p < .001), and demonstrated higher scores on the Somatization, Depression, Anxiety, Attention Problems, and Withdrawal scales. Subsequent jackknifed classification analysis, diagnostic efficiency statistics, and an odds ratio for the classification analysis added to the overall validity of results. The practical utility of the BASC‐PRS is further supported in light of expanding roles for school psychologists in the assessment and treatment of children with health problems. © 2003 Wiley Periodicals, Inc. Psychol Schs 40: 145–154, 2003.     

The measurement of classroom management self‐efficacy: a review of measurement instrument development and influences

Teachers' self‐efficacy (SE) in their classroom management capabilities is thought to be an important factor in teachers' overall judgements of their teaching SE. Low SE in classroom management has been linked to teacher attrition and burnout, and reduced student learning outcomes. This article provides the first comprehensive review of classroom management as a factor in the construct of teacher SE. Twenty‐five peer‐reviewed articles published from 1984 to 2009 that reported on the use of SE scales containing at least one novel classroom management self‐efficacy (CMSE) item were reviewed. The validity and reliability of CMSE scales and items were found to be very good, with classroom management items pertaining to maintaining order and control the most frequent category included. Approximately one in four items in the SE scales reviewed was CMSE item, and, in general, CMSE items were not linked explicitly to classroom management research or contemporary psychological or philosophical approaches.     

Bicultural Work Motivation Scale for Asian American college students

The bicultural work motivations of Asian Americans have not yet been comprehensively captured by contemporary vocational constructs and scales. For this study, we conducted two studies on the initial reliability and validity of the Bicultural Work Motivation Scale (BWMS) by combining qualitative and quantitative methods. First, a pilot study was conducted with 14 Asian American college students to assess their motivation to work, and items were generated from the themes developed in the qualitative analysis. Results from the exploratory and confirmatory factor analyses supported a stable four-factor structure of the BWMS: Honoring Parents, Family Financial Obligation, Fulfilling Personal Interest, and Being Independent. Adequate reliability and evidence for concurrent and discriminant validity are presented. Implications of the scale for further vocational assessment and research are discussed.     

Factor analysis of the pervasive developmental disorders rating scale with teacher ratings of students with autistic disorder

The Pervasive Developmental Disorders Rating Scale (PDDRS; Eaves, 2003) is a rating scale that is used in the screening process for pervasive developmental disorders (PDD). The PDDRS contains three scales: Arousal, Affect, and Cognition. In this study, the construct validity of the PDDRS was examined with teacher ratings from a sample of 168 children with autistic disorder through exploratory and confirmatory factor analysis. In the exploratory factor analysis, a three‐factor structure was examined. Confirmatory factor analysis was employed to evaluate the model‐fit of the data for the hypothesized second‐order factor structure. The results supported the averred factor structure of the PDDRS with teacher ratings for students with autistic disorder. Limitations of the data and areas for future research are discussed. © 2005 Wiley Periodicals, Inc. Psychol Schs 42: 207–216, 2005.     

Validity of the home and community social behavior scales: Comparisons with five behavior‐rating scales

Three separate studies focusing on convergent and discriminant validity evidence for the Home and Community Social Behavior Scales are presented. The HCSBS is a 65‐item social behavior‐rating scale for use by parents and caretakers of children and youth ages 5–18. It is a parent‐rating version of the School Social Behavior Scales. Within these studies, relationships with five behavior‐rating scales were examined: the Social Skills Rating System, Conners Parent Rating Scale–Revised‐Short Form, Child Behavior Checklist, and the child and adolescent versions of the Behavior Assessment System for Children. HCSBS Scale A, Social Competence, evidenced strong positive correlations with measures of social skills and adaptability, strong negative correlations with measures of externalizing behavior problems, and modest negative correlations with measures of internalizing and atypical behavior problems. HCSBS Scale B, Antisocial Behavior, evidenced strong positive correlations with measures of externalizing behavior problems, modest positive correlations with measures of internalizing and atypical behavior problems, and strong negative correlations with measures of social skills and adaptability. These results support the HCSBS as a measure of social competence and antisocial behavior of children and youth. © 2001 John Wiley & Sons, Inc.