Administrative Effectiveness in Higher Education: Improving Assessment Procedures

Holding higher education personnel accountable for measurable outcomes is a relatively new phenomenon. Assessing the performance of administrators may well provide needed and appropriate information about the functioning of the university, but it may represent high stakes assessment for individuals; that is, measures of administrative effectiveness may be tied to promotion, salary augmentation, contract renewal, or dismissal. Little previous research exists, however, on the assessment of administrators' performance in higher education (Gmelch et al., in press; Seldin, 1988). We develop an evaluation model for assessing and monitoring the effectiveness of academic deans and directors, using generalizability theory (i.e., an approach that focuses on identifying multiple sources of error in performance assessment) as a basis for developing more accurate assessment procedures. We illustrate this approach using faculty and staff assessments of their deans' leadership effectiveness. We also provide guidelines for improving the quality of assessments by adjusting various aspects of the evaluation model.     

Perceived School and Neighborhood Safety, Neighborhood Violence and Academic Achievement in Urban School Children

Community and school violence continue to be a major public health problem, especially among urban children and adolescents. Little research has focused on the effect of school safety and neighborhood violence on academic performance. This study examines the effect of the school and neighborhood climate on academic achievement among a population of 3rd–5th grade students in an urban public school system. Community and school safety were assessed using the School Climate Survey, an annual City-wide assessment of student’s perception of school and community safety. Community violence was measured using the Neighborhood Inventory for Environmental Typology, an objective observational assessment of neighborhood characteristics. Academic achievement was measured using the Maryland State Assessment (MSA), a standardized exam given to all Maryland 3rd–8th graders. School Climate Data and MSA data were aggregated by school and grade. Objective assessments of neighborhood environment and students’ self-reported school and neighborhood safety were both strongly associated with academic performance. Increasing neighborhood violence was associated with statistically significant decreases from 4.2 to 8.7% in math and reading achievement; increasing perceived safety was associated with significant increases in achievement from 16 to 22%. These preliminary findings highlight the adverse impact of perceived safety and community violence exposure on primary school children’s academic performance.     

The visible hand of research performance assessment

Far from allowing a governance of universities by the invisible hand of market forces, research performance assessments do not just measure differences in research quality, but yield themselves visible symptoms in terms of a stratification and standardization of disciplines. The article illustrates this with a case study of UK history departments and their assessment by the Research Assessment Exercise (RAE) and the Research Excellence Framework (REF), drawing on data from the three most recent assessments (RAE 2001, 2008, REF 2014). Symptoms of stratification are documented by the distribution of memberships in assessment panels, of research active staff, and of external research grants. Symptoms of a standardization are documented by the publications submitted to the assessments. The main finding is that the RAEs/REF and the selective allocation of funds they inform consecrate and reproduce a disciplinary center that, in contrast to the periphery, is well-endowed with grants and research staff, decides in panels over the quality standards of the field, and publishes a high number of articles in high-impact journals. This selectivity is oriented toward previous distributions of resources and a standardized notion of “excellence” rather than research performance.     

Formative assessment of procedural skills: students’ responses to the Objective Structured Clinical Examination and the Integrated Performance Procedural Instrument

Assessment of clinical skills is a critical element of undergraduate medical education. We compare a traditional approach to procedural skills assessment – the Objective Structured Clinical Examination (OSCE) with the Integrated Performance Procedural Instrument (IPPI). In both approaches, students work through ‘stations’ or ‘scenarios’ undertaking defined tasks. In the IPPI, all tasks are contextualised, requiring students to integrate technical, communication and other professional skills. The aim of this study was to explore students’ responses to these two assessments. Third‐year medical students participated in formative OSCE and IPPI sessions on consecutive days. Although performance data were collected in both assessments, quantitative data are not presented here. Group interviews with students were conducted by independent researchers. Data were analysed thematically. The OSCE and the IPPI were both valued, but for different reasons. Preference for the OSCE reflected the format of the summative assessment. The IPPI was valued for the opportunity to practise patient‐centred care in a simulated setting which integrated technical, communication and other professional skills. We posit that scenario‐based assessments such as the IPPI reflect real‐world issues of patient‐centred care. Although the limitations of this study prevent wide extrapolation, we encourage curriculum developers to consider the influence of assessments on what and how their students learn.     

Measuring the Impact of Peer Interaction in Group Oral Assessments with an Extended Many-Facet Rasch Model

Many language proficiency tests include group oral assessments involving peer interaction. In such an assessment, examinees discuss a common topic with others. Human raters score each examinee's spoken performance on specially designed criteria. However, measurement models for analyzing group assessment data usually assume local person independence and thus fail to consider the impact of peer interaction on the assessment outcomes. This research advances an extended many-facet Rasch model for group assessments (MFRM-GA), accounting for local person dependence. In a series of simulations, we examined the MFRM-GA's parameter recovery and the consequences of ignoring peer interactions under the traditional modeling approach. We also used a real dataset from the English-speaking test of the Language Proficiency Assessment for Teachers (LPAT) routinely administered in Hong Kong to illustrate the efficiency of the new model. The discussion focuses on the model's usefulness for measuring oral language proficiency, practical implications, and future research perspectives.     

State Perspectives on Implementing,or Choosing Not to Implement,an Alternate Assessment Based on Modified Academic Achievement Standards

Since Federal regulations have given states the option to implement alternate assessments based on modified academic achievement standards (AA-MAS) as part of their accountability systems for a small group of students with disabilities, a number of states have made decisions about whether or not to develop and implement such an assessment. State-level directors of assessment and directors of special education were surveyed about their state's decisions on implementing AA-MAS. Improvements in accessibility and appropriateness were reasons given for choosing to implement an AA-MAS, whereas lack of resources and guidance were identified as barriers. This article presents the findings from a survey on 22 states' decisions concerning implementation of AA-MAS.     

Alternate Assessments of Students with Significant Disabilities: Alternative Approaches,Common Technical Challenges

This article examines three typical approaches to alternate assessment for students with significant cognitive disabilities—portfolios, performance assessments, and rating scales. A detailed analysis of common and unique design features of these approaches is provided, including features of each approach that influence the psychometric quality of their results. Validity imperatives for alternate assessments are reviewed, and approaches for addressing the need for validity evidence are outlined. The article concludes with an examination of three technical challenges—alignment, scores and scoring, and standard setting—common to all alternate assessments. In light of these challenges, existing methods and professional testing standards are endorsed as necessary guidance for understanding and advancing alternate assessment practices.     

Special educational needs,disability and school accountability: an international perspective

Standards-based accountability approaches in education are increasingly being adopted in many industrialised countries. This tends to involve the scrutiny of student performance in national academic tests and then holding schools accountable for subsequent progress. International assessments such as Programme for International Student Assessment (PISA) are not accountability measures in themselves, but are often utilised in a similar way and national policies are judged against performance in these tests. Drawing upon the examples of PISA and national assessments in the USA, this paper explores how students with special educational needs and disability (SEND) are included in, and served by, these assessment policies. The analysis suggests that while these approaches are inclusive in their intent (most obviously through efforts to include students with SEND in testing) they may poorly represent the progress of these students. It is argued that more equitable inclusion of students with SEND must find a balance between accountability measures which measure performance in traditional attainment-focused tests and other relevant assessments which are meaningful to the given student and/or SEND sub-group.     

Calculator-Available Assessments: The Why,What, and How

In the Curriculum and Evaluation Standards for School Mathematics, the National Council of Teachers of Mathematics (1989) advocated that "appropriate calculators should be available to all students at all times" (p. 8) and that calculators (or any instructional tool) should be allowed in the assessment of those students. In response, large-scale assessments such as the National Assessment of Educational Progress, Scholastic Achievement Test, American College Testing, Advanced Placement, and a variety of state-mandated tests moved to incorporate the allowance of calculator technologies in the assessment scheme. Although rhetoric and large-scale assessments have embraced the allowance of technology, classroom teachers may have difficulty developing their own assessments that permit calculator use. This article discusses the factors involved in h e design and implementation of calculator-available assessments. In particular, four issues associated with calculator-available assessments are examined. (a) the benefits and problems associated with calculator use in assessment, (b) the components of various types of calculator-available assessments, (c) the construction of calculator-available assessments, and (d) the need for additional examination of items included on calculator-available assessments. From this examination, teachers and test developers are provided with a set of recommendations that focus on developing calculator-available assessments and minimizing difficulties arising at the item 1evel.     

Complicating the World of Reading Assessment: Toward Better Assessments for Better Teaching

In this article, we share with readers our hopes, fears, and predictions for reading assessment in American schools at a critical policy juncture—the production of new assessments to measure achievement of the Common Core State Standards. It isn't just Partnership for Assessment of Readiness for College and Careers and Smarter Balanced Assessment Consortium in play. There is a new stream of progressive-minded pedagogical reform rekindling the fire for higher-order thinking, project-based learning, and learning that transfers to new and different situations and problems. This is a perfect storm—a genuine opportunity to change the course of assessment for accountability and classroom decision-making. Our hope is that readers will find our perspectives useful as we collectively avail ourselves of this unique opportunity.     

Summative and Formative Assessments in Mathematics Supporting the Goals of the Common Core Standards

Being proficient in mathematics involves having rich and connected mathematical knowledge, being a strategic and reflective thinker and problem solver, and having productive mathematical beliefs and dispositions. This broad set of mathematics goals is central to the Common Core State Standards for Mathematics.

High-stakes testing often drives instructional practice. In this article, I discuss test specifications and sample assessment items from the two major national testing consortia and the prospects that their assessments will be positive levers for change.

For more than 20 years, the Mathematics Assessment Project has focused on the development of assessments that emphasize productive mathematical practices, most recently creating formative assessment lessons (FALs) designed to help teachers build up student understandings through focusing on student thinking while engaging in rich mathematical tasks. This article describes our recent work.     

英国开放大学课程考核制度探析——基于对《学生考核手册》的分析

英国开放大学课程考核制度是检验学生学习效果的重要机制之一。通过分析《学生考核规则》的基本框架与内容,发现英国开放大学的课程考核制度具有如下特点:一是内容详尽,指导性强;二是明确规定学生考核的权利与义务;三是考核体现严肃性与服务性;四是提供救济途径,尊重学生权利;五是考核形式灵活多样;六是重视培养学生诚信的道德品质,禁止剽窃。英国开放大学课程考核制度的特点对完善我国开放大学课程考核制度建设具有重要的借鉴意义。     

Technical Quality Criteria for Evaluating District Assessment Portfolios Used in the Nebraska STARS

Nebraska districts use different strategies for measuring student performance on the state's content standards. District assessments differ in type and technical quality. Six quality criteria were endorsed by the state. These criteria cover content and curricular validity, fairness, and appropriateness of score interpretations. District assessment portfolios document how well assessments meet these criteria. Districts receive ratings on how well their assessments meet each of the quality criteria and are given a rating from Unacceptable to Exemplary. This article presents these technical quality criteria and explains how they are (a) individually rated and (b) combined for the district's overall quality rating.     

美国“评估三角形”研究：框架、设计及应用

“评估三角形”是美国评价基金委员会于2001年在《知道学生所知道的:教育评估的科学和设计》一书所提出的一种教育评估框架,是由认知、观察、解释三个要素构成的一个协调、联系的整体,能够为不同形式的教育评估提供普遍性的设计原则。文章具体阐释了该框架的结构和设计原则,并举例介绍了该框架在教育实践中的具体应用及其衍生框架。     

Navigating the Measurement and Monitoring Maze

Assessment of students with disabilities is a critical component of special education. In addition to the complexity of assessment (e.g., high-stakes assessments, progress monitoring, diagnosis), the issue is complicated further by the use of acronyms. This can make both explaining and understanding important information related to students very difficult. Acronyms related to assessment create a virtual alphabet soup that can be confusing for parents, teachers, and school personnel. Assessment acronyms include terms related to both state testing and progress monitoring such as AYP, AA-AAS, AA-MAS, AA-GLAS, RTI, CBA, and CBM. Educators must understand not only the nature and purpose of the various assessments, but they must also be familiar with the assessment acronyms in order to select appropriate assessments to measure the academic progress and performance of students, and to communicate effectively with families and related school professionals. The purpose of this article is to describe common assessments administered in the field of education today as well as to demystify the use of the acronyms related to these assessments and implications for teachers and other educators. Additionally, we point to some resources that can be of value to individuals wishing to establish or expand their repertoires of assessment practices as the current emphasis on progress monitoring continues to unfold.     

Teacher Performance Assessment (edTPA): An Instructor’s Development and Evaluation of an Embedded Signature Assessment in an Early Childhood Literacy Course

As of May 2014, all candidates for teacher certification in New York State must successfully complete the teacher performance assessment (edTPA). This article describes an assignment piloted in an early childhood literacy class as an embedded signature assessment and has three goals: to explain the context that led to the development of the assignment, to describe how the assignment was created and piloted, and to describe the methodology and results of an action research project intended to collect data on candidates’ perceptions of the challenges and value of the assignment. Implications for candidates’ understanding and utilization of developmentally appropriate practice, academic language, and self-reflection, as well as for the design of embedded signature assessments are discussed.     

Feedback,feedforward: evaluating the effectiveness of an oral peer review exercise amongst postgraduate students

Assessment for learning approaches, such as peer review exercises may improve student performance in summative assessments and increase their satisfaction with assessment practices. We conducted a mixed methods study to evaluate the effectiveness of an oral peer review exercise among post-graduate students. We examined: (1) final assessment grades among students who did and did not take part in the peer review exercise; (2) student perceptions of the impact of the peer review exercise; and (3) student understanding of, and satisfaction with, this new assessment practice. We found that students who took part in the exercise had a significantly higher mean grade in a subsequent summative oral presentation assessment than students who did not take part in the exercise. Students gained a better understanding of assessment and marking criteria and expressed increased confidence and decreased anxiety about completing the subsequent summative assessment. Assessment for learning improves academic attainment and the learning experience in postgraduate students.     

Aligning formative and summative assessments: A collaborative action research challenging teacher conceptions

Assessment innovations require explicitly challenging teachers’ assessment conceptions. In changing towards outcome-based curricula, aligning formative to new summative assessments is a challenge. This study, conducted in Dutch Agricultural Vocational Education and its new outcome-based summative assessments, aims to provide concrete insights into what teacher conceptions hinder the development of aligned formative assessment practices. In a structured collaborative action research, practitioners, researchers and consultants analyse teachers’ formative practices, identify underlying current conceptions and their misalignment to the new summative assessment system. An iterative process resulted in an overview of current conceptions versus required conceptions, showing required conceptual changes that are prerequisite for teachers to change towards formative assessment practices that are aligned to the new outcome-based summative assessment system.     

History and Background of Nebraska's School-based Teacher-led Assessment and Reporting System (STARS)

Nebraska's approach to standards, assessment, and accountability, the School-based Teacher-led Assessment and Reporting System (STARS) is based upon local control and the belief that classrooms and teachers must be at the heart of student learning and accountability. STARS relies on locally-developed assessment systems to accurately measure and report student performance on state content standards. Each local system in Nebraska's 500+ school districts is reviewed for technical quality, and districts are publicly rated for assessment quality and student performance. The purpose of this article is to establish the historical background.     

Using Interviews to Predict the Success of Teacher‐education Students

In November 1987, the School of Education at the University of Tasmania at Launceston (then the Tasmanian State Institute of Technology) commenced a programme of pre‐entry assessment of BEd applicants with a view to conducting an ongoing research programme aimed at improving the quality of trainee teachers. Pre‐entry data gathering included English and mathematics skills, learning style, Higher School Certificate (HSC) aggregate scores and interview assessments of each applicant by a panel of at least two academic staff of the School. This paper reports findings from the interview data gathered on students who entered the BEd programme in 1989. In particular, the evidence from this cohort of students suggests that interview ratings by academic staff are successful predictors of future practice‐teaching performance, whereas HSC aggregate scores are not at all predictive. In addition to this finding, the data indicate that staff interview assessments are only marginally weaker than HSC aggregate scores as predictors of academic performance on the BEd course.