Technical adequacy of the most popular tests selected by responding school psychologists in Illinois

Eleven of the most popular tests used by school psychologists in the State of Illinois were evaluated as to their technical adequacy along three dimensions. Results suggest that the most popular tests used by school psychologists are not typically technically adequate devices.     

Psychoeducational decision making as a function of the amount of information reviewed

Psychoeducational assessment is commonplace in America's public school systems. Important decisions for and about school children are made on the basis of assessment results; test scores and information derived from the administration of tests clearly contribute to these decisions. Current “best practice” is guided by the concept of multidisciplinary team decision making; parents, teachers, psychologists, and other professionals review assessment data when participating as team members. In this research, decision makers were allowed to review information from a variety of sources prior to making decisions about a case study child. Individuals who reviewed large amounts of information made decisions similar to those of individuals who reviewed little information. Approximately 49% of the participants made accurate decisions when judged against an a priori criterion. The results are discussed with regard to psychoeducational assessment practice.     

Assessment practices in model early childhood special education programs

Examination of assessment practices in 54 model programs for early education of handicapped children revealed extreme variability. Only 19 tests were used by at least five programs; only one device was used by over half of the responding programs. Most tests were listed as being used for more than one purpose, but some tests appeared to be used more exclusively than others for a particular purpose. Although the technical adequacy, in terms of norms, validit, and reliability, was reportedly an important factor influencing selection and continued use of tests, analysis of the 19 most commonly used devices revealed that only three were technically adequate. Other methods of assessment also were examined. Implications for model practice are discussed.     

Heuristic principles and cognitive bias in decision making: Implications for assessment in school psychology

Research has indicated that the diagnoses and classification decisions made by professionals are often unreliable. People employ several heuristic strategies when making decisions under conditions of uncertainty. Use of these heuristics can have a detrimental effect on the decision-making process. This paper describes factors that may bias psychoeducational decision making and discusses three heuristic principles that affect decision making. The means by which school psychologists can be made aware of these heuristic principles and encouraged to consider them when making psychoeducational decisions are discussed. Other methods by which bias in the psychoeducational process can be reduced, including the use of statistical and actuarial-based assessment systems, the multitrait multimethod approach to multifactored assessment advanced by Gresham (1983), and direct instruction in reasoning and decision making are also discussed.     

Measuring self-efficacy,aptitude to participate and functioning in students with and without impairments

Including vulnerable groups of students such as students with learning disabilities in mainstream school research, require ethical considerations and questionnaire adaptation. These students are often excluded, due to low understanding or methodologies generating inadequate data. Students with disability need be studied as a separate group and provided accessible questionnaires. This pilot study aims at developing and evaluating student self-reported measures, rating aspects of student experiences of school-based Physical Education (PE). Instrument design, reliability and validity were examined in Swedish secondary school students (n = 47) including students, aged 13, with intellectual disability (n = 5) and without impairment and test–retested on 28 of these students. Psychometric results from the small pilot-study sample were confirmed in analyses based on replies from the first wave of data collection in the main study (n = 450). Results show adequate internal consistency, factor structure and relations between measures. In conclusion, reliability and validity were satisfactory in scales to measure self-efficacy in general, in PE, and aptitude to participate. Adapting proxy ratings for functioning into self-reports indicated problems. Adequacy of adjustments made were confirmed and a dichotomous scale for typical/atypical function is suggested for further analyses.     

A measure of cultural competence as an ethical responsibility: Quick‐Racial and Ethical Sensitivity Test

This article presents the psychometric qualifications of a new video‐based measure of school professionals' ethical sensitivity toward issues of racial intolerance in schools. The new scale, titled the Quick‐Racial and Ethical Sensitivity Test (Quick‐REST) is based on the ethical principles commonly shared by school‐based professional organisations and James Rest's model of moral decision making. The validation of the measure is established through two separate studies: one conducted with 238 school professionals, including teachers, administrators, psychologists, coaches and others who work in diverse schools; and the other conducted with 57 student teachers. The results, with regard to the internal reliability of the items and convergent and overall construct validity, showed that the Quick‐REST is a psychometrically sound measure of school professionals' ability to recognise violations of ethical principles as depicted in two five‐minute videotapes illustrating instances of racial intolerance in schools. There was no evidence of bias due to social desirability. Areas of use for the Quick‐REST in teacher education and ways to improve the scale are discussed.     

Introduction to the Special Issue: Effective Practices for Children and Youth with Autism Spectrum Disorders

This article focuses on the development and use of alternative reading measures that are performance based (authentic and direct), technically adequate, and help teachers make instructional decisions. Although performance assessments are promulgated as better than many traditional, published measures of achievement, few studies on technical adequacy have been conducted. We present 2 studies that focus on concurrent criterion validity and instructional decision making; the studies are framed as examples of reading measurement within the debate on whole language programs. The first study focuses on the relation of the performance measure of oral reading with several other reading measures; the second study focuses on teacher decision making, using quantitative and qualitative outcomes reflecting individual student change in performance (in reading fluency and prosody) over time. In this article, both studies are described briefly, and the emphasis of discussion is placed on appropriately evaluating programs so that the measures match the interventions (are systemically valid) and support their effec- tiveness for use with individual students.     

An analysis of difference score reliabilities on three measures with a sample of low-achieving youngsters

A common practice in the field of learning disabilities is analysis of ability-achievement discrepancies. The reliability of discrepancy scores is an important statistic in such decision making. In this study, selected ability and achievement devices were administered to a sample of low achievers (N = 99), and the reliability of various difference scores was analyzed. In all cases, the reliabilities of difference scores were moderately high. Reliabilities of differences for devices normed on the same population and differences for devices normed on different populations were comparable. These results are discussed in light of current psychometric practices.     

Assisting children and families who use augmentative and alternative communication (AAC) devices: Best practices for school psychologists

In developing individual education plans for children who need assistive technology devices, school psychologists will increasingly be called upon to assist in the decision making process regarding electronic augmentative and alternative communication (AAC) devices. A thorough assessment is required to select a device that is most appropriate for the child and their family. Assessment factors which must be considered within five domains (child, AAC device, service system, family, culture) are reviewed. A self-checklist is provided to help school psychologists develop and conduct thorough “best practice” assessments for the target population. © 1997 John Wiley & Sons, Inc.     

Evaluating risk for targeted violence in schools: Comparing risk assessment,threat assessment,and other approaches

In the wake of recent school shootings, fear over violence in schools has prompted increased requests for psychologists, educators, and law enforcement professionals to assist in preventing future school violence incidents. We attempt to lay a foundation for developing effective assessment and prevention approaches by first distinguishing planned school‐based attacks from other forms of school and youth violence. We then review the three assessment approaches that have been advocated and used in some jurisdictions (profiling, guided professional judgment, automated decision‐making) and demonstrate why they are inappropriate—and potentially harmful—in preventing planned school‐based attacks. We then describe the contours of the threat assessment approach, developed by the U.S. Secret Service to prevent assassinations, and examine its utility for responding to communications or behaviors of concern that students may present in school settings. © 2001 John Wiley & Sons, Inc.     

The Development and Initial Validation of an Assessment of Algebraic Thinking for Students in the Elementary Grades

Elementary school students often exhibit a variety of conceptions associated with algebraic thinking that their teachers fail to recognize or understand. It is crucial that elementary school teachers possess knowledge of the variety of student conceptions and also have abilities to address varying states of conceptions. Otherwise, students who are not provided with opportunities to think algebraically may continue to struggle with algebra throughout school. The purpose of this study, therefore, was to develop and provide initial validity evidence for an assessment tool of algebraic thinking for the elementary grades. In the development of the tool, validation efforts were pursued, including multiple rounds of pilot-testing and think aloud interviews. Evidence for reliability of scores and validity of score interpretations of the overall score were collected. Results suggest that the assessment is a potentially valuable tool for teachers to elicit and learn about students’ algebraic thinking knowledge.     

National monitoring in social studies: the reliability of assessment interviews

The National Education Monitoring Project (NEMP) is responsible in New Zealand for the national assessment of primary school children's achievement in the essential learning areas, one of which is social studies. Individual interviews are one of the approaches used to assess students' understanding. The assessors are registered teachers, selected by NEMP, who attend a training week where they learn how to conduct the standardised assessment tasks and associated interviews. This study examines the reliability of the assessment interviews used in the 2005 round of social studies monitoring, in particular the variations between teacher administrators (TAs) in their use of prompts and probes. The extent of variation observed in the use of three kinds of prompt was sufficient to raise questions about the reliability of the assessment process. A surprising outcome was the consistent failure of TAs to clarify and elucidate students' social studies understandings through the judicious use of probes. The prompt-related variations between TAs and their failure fully to ‘tap into’ understandings assessed by interview-based tasks are serious threats to the validity of claims regarding students' achievement in social studies. This is particularly concerning as NEMP data are used as the basis for identifying and reporting national patterns and trends in educational performance and making recommendations to policy-makers, curriculum planners and educators.     

Peer assessment using comparative and absolute judgement

Peer assessment exercises yield varied reliability and validity. To maximise reliability and validity, the literature recommends adopting various design principles including the use of explicit assessment criteria. Counter to this literature, we report a peer assessment exercise in which criteria were deliberately avoided yet acceptable reliability and validity were achieved. Based on this finding, we make two arguments. First, the comparative judgement approach adopted can be applied successfully in different contexts, including higher education and secondary school. Second, the success was due to this approach; an alternative technique based on absolute judgement yielded poor reliability and validity. We conclude that sound outcomes are achievable without assessment criteria, but success depends on how the peer assessment activity is designed.     

Intersecting Domains of Assessment Knowledge: School Typologies Based on Interviews with Secondary Teachers

This study used qualitative interviewing with teachers at three high schools to answer research questions about teachers' assessment knowledge, school-specific assessment cultures, and teachers' perceptions of the assessment literacy of other key stakeholders. Data analysis revealed shared knowledge and practices across schools—use of formative assessment to guide instructional decision making; reliance on collaborative processes as the basis for cultivating school-wide assessment practices; and the belief among teachers that students, parents, and school administrators often hold naive views about the relative primacy of summative over formative assessment. Data analysis also identified distinctive features of the assessment culture at each of the schools.     

TEM4平行模拟测试信度及差异检验

对于全国性测试，经常性的评估是必不可少的。语言测试评估、有效性研究的关键是信度或一致性研究。本研究使用TEM4平行试卷，分别进行信度统计、差异分析。它不仅检验了平行测试之间的一致性问题，还在有差异的情况下，对有差异的测试或题项进行定位。这种定位对以后的测试编制、预测及拼卷将起到积极的作用。     

SCHOOL PSYCHOLOGIST DIAGNOSTIC DECISION‐MAKING: A PILOT STUDY

The current study examined the diagnostic decision‐making of school psychologists as a function of a student's disability and academic performance with three research questions using a randomly‐selected sample of school psychologists from the state of Texas. Results from the first research question indicated that school psychologists significantly differentiated between what diagnoses were considered most correct according to the DSM‐IV and what diagnoses would be in a child's best interest at school. Results from the second research question indicated that this differentiation in diagnostic decision‐making was present as significantly associated with academic performance. Results from the third and final research question indicated that school psychologists differed in their diagnostic decision‐making according to their perceptions about the particular disability.     

Improving assessment of learning and for learning

The focus of this article is recent work by the Assessment Reform Group (ARG) on the role of teachers' judgements in the summative use of assessment. A brief overview of the early work of the ARG is followed by discussion of the desirable properties of assessment for summative uses. The work of the ARG's Assessment Systems for the Future project provided evidence and arguments concerning the validity, reliability, impact and cost of tests and of summative assessment by teachers. Whilst there is ample evidence that the teachers' judgements are more valid than, and equally reliable as, tests, there is a danger of unwanted impact on teaching as long as results are used for ‘high stakes’ evaluation of teachers and schools. Implications for policy include an end to the practice of using the results of pupils' summative assessment, however they are derived, as the sole basis for target setting and school accountability.     

The reliability of results from national tests,public examinations,and vocational qualifications in England

National tests, public examinations, and vocational qualifications in England are used for a variety of purposes, including the certification of individual learners in different subject areas and the accountability of individual professionals and institutions. However, there has been ongoing debate about the reliability and validity of their results. This debate prompted the Office of Qualifications and Examinations Regulation (Ofqual), the regulator of qualifications and examinations in England, to initiate its Reliability of Results Programme in 2008 to investigate the reliability of results from regulated assessments. The purpose of this programme was to gather evidence to develop regulatory policy on assessment reliability in order to improve the national assessment systems in England. This paper provides a brief introduction to the Reliability Programme, discusses its main findings to date and their implications, and explores the impact of the development of regulatory policy on reliability for regulated assessments in England.