Measuring characteristics of teacher professional development

The primary purpose of the study was to create and psychometrically test an instrument which measured teachers’ perceptions of characteristics of professional development. The sample consisted of elementary teachers from five school districts in Washington State participating in a district improvement initiative. Results of exploratory factor analysis resulted in a five-component solution which provided evidence of construct validity. All components/subscales had moderate to strong internal consistency as measured by Cronbach’s alpha. The secondary purpose was to determine if characteristics of professional development predicted teachers’ use of new knowledge and skills, and student learning outcomes, thus contributing to the predictive validity of the instrument. There was a slight, but significant, correlation between Active Learning in Classroom and teachers’ use of new knowledge and skills, as measured by classroom observation scores. A small negative correlation emerged between Collective Participation and student learning, as measured by student scores on the state’s mathematics assessment. Potential uses of the instrument, called Characteristics of Teacher Professional Development (CTPD), are discussed.     

Assessing student exposure to and use of computer technologies through an examination of course syllabi

A syllabus analysis instrument was developed to assist program evaluators, administrators and faculty in the identification of skills that students use as they complete their college coursework. While this instrument can be tailored for use with a variety of learning domains, we used it to assess students' use of and exposure to computer technology skills. The reliability and validity of the instrument was examined through an analysis of 88 syllabi from courses within the teacher education program and the core curriculum at a private Midwest US university. Results indicate that the instrument has good inter‐rater reliability and ratings by and interviews with faculty and students provide evidence of construct validity. The use and limitations of the instrument in educational program evaluation are discussed.     

Validity evidence for the Experiences of Teaching and Learning Questionnaire (ETLQ) in evaluations of quality learning: A systematic critical literature review

The use of the Experiences of Teaching & Learning Questionnaire (ETLQ) for the evaluation of learning quality in higher education has been expanding during the last decade, thus a review of the instrument’s validity evidence is warranted. The design of the study was a systematic critical literature review. We evaluated the strength of the validity evidence of 17 included studies with a quality appraisal framework reflecting current standards for educational testing. The evidence supporting the central validity assumptions of the ETLQ scales is currently weak to moderate and incomplete. Thus, caution against the uncritical use of ETQL scores for high-stakes educational decisions is warranted. The appraisal framework used was useful for creating an overview of the evidence. However, attention to more general aspects of study quality, and consensus deliberations with three to four raters was also important for sufficiently reliable appraisal of the evidence.     

Development of the Efficacy Beliefs for Conceptual Change Learning Questionnaire

The purpose of this study was to develop an instrument to assess college students’ efficacy beliefs for conceptual change and to examine the psychometric properties of the instrument. Participants were 692 students. Results of the confirmatory factor analysis supported the hypothesized single factor structure of Efficacy Beliefs for Conceptual Change Learning Questionnaire providing evidence for the construct validity. Evidence for concurrent validity also is provided. On the basis of the evidence provided in this study, the questionnaire appears to produce valid and reliable scores for college students. With the use of the questionnaire, conceptual change researchers might be able to better assess the relationship between students’ efficacy beliefs and the change in their conceptual understandings of various science concepts.     

一种新的评定临床技能成绩的方法

英国北部设菲尔德市儿童医院开发了一种用门诊病人医案来评定儿科医生临床技能的量表,并估计了该量表的信度、可行性、效度和教育的影响。实践表明该量表是可行的,在对培训中的儿科注册医师进行年度评定中它提供的结果是可靠的,量表的反馈有利于被评医生提高自身的临床书面交流能力。这一考评方法具有操作简便、信度和效应较好的优点,它可以作为评定综合性临床能力的一部分,经过试验,这一方法也可用于其他医疗专业的考评。     

Measuring the Quality of Teaching in Elementary School Classrooms

This paper reviews the means by which teacher quality has been measured. It considers data sources such as students, peers, experts, and examines the psychometrics and scale properties of teacher quality assessment instruments with respect to reliability and validity. A list of items for possible inclusion in an elementary student focussed instrument is considered, together with the potential use of such an instrument in measuring teacher quality.     

Assessing cohesion in children’s writing: Development of a checklist

Cohesion in writing is achieved through the use of linguistic devices that tie ideas together across a text, and is an important element in the development of coherent writing. Research shows that inter- and intra-developmental differences may appear in how children learn to use these devices, but cohesion is commonly overlooked in the evaluation and instruction of writing. In this study, we developed a checklist to assess cohesion in the writing of children in Grades 4–7, with the purpose of informing instructional practices. Following the procedure outlined by Crocker and Algina (1986), we developed and evaluated a checklist designed to assess the types of cohesive devices present in the writing of children. The checklist items showed fair to good discrimination between high and low scoring writers as demonstrated by a classical item analysis. We also found good interrater reliability, and evidence for discriminative validity. As internal consistency was weak, however, further research is needed to refine the instrument. Implications for the assessment of cohesion and future research are discussed.     

Approaches to measuring use of research evidence in K-12 settings: A systematic review

An increased focus on the use of research evidence (URE) in K-12 education has led to a proliferation of instruments measuring URE in K-12 education settings. However, to date, there has been no review of these measures to inform education researchers’ assessment of URE. Here, we systematically review published quantitative measurement instruments in K-12 education. Findings suggest that instruments broadly assess user characteristics, environmental characteristics, and implementation and practices. In reviewing instrument quality, we found that studies infrequently report reliability, validity, and demographics about the instruments they develop or use. Future work evaluating and developing instruments should explore environmental characteristics that affect URE, generate items that match up with URE theory, and follow standards for establishing instrument reliability and validity.     

Critical review of the TEMAS: A step within the development of thematic apperception instruments

The TEMAS (acronym for Tell‐Me‐a‐Story)—an objectively scored, projective thematic personality instrument for children and adolescents—is analyzed, reviewed, and critiqued with regard to theoretical underpinnings and rationale for development, administration, scoring, psychometric properties, and research to date. The TEMAS appears to be an improvement over existing projective personality measures used by school psychologists. Although it requires more training than other projective techniques, competency in administration, scoring, and interpretation can be achieved within a one semester course in personality assessment. The test has evidence of reliability and validity, and it is a multicultural alternative to the TAT and other thematic apperception instruments. The use of the TEMAS by psychologists may achieve more accurate assessment of Black and Hispanic children. Limitations include geographically limited standardization samples and little research conducted by individuals other than the authors. © 1999 John Wiley & Sons, Inc.     

When Assessment Validation Neglects Any Strand of Validity Evidence: An Instructive Example from PISA

The Standards for Educational and Psychological Testing identify several strands of validity evidence that may be needed as support for particular interpretations and uses of assessments. Yet assessment validation often does not seem guided by these Standards, with validations lacking a particular strand even when it appears relevant to an assessment. Consequently, the degree to which validity evidence supports the proposed interpretation and use of the assessment may be compromised. Guided by the Standards, this article presents an independent validation of OECD's PISA assessment of mathematical self-efficacy (MSE) as an instructive example of this issue. OECD identifies MSE as one of a number of “factors” explaining student performance in mathematics, thereby serving the “policy orientation” of PISA. However, this independent validation identifies significant shortcomings in the strands of validity evidence available to support this interpretation and use of the assessment. The article therefore demonstrates how the Standards can guide the planning of a validation to ensure it generates the validity evidence relevant to an interpretive argument, particularly for an international large-scale assessment such as PISA. The implication is that assessment validation could yet benefit from the Standards as what Zumbo calls “a global force for testing”.     

A General Framework for the Validation of Embedded Formative Assessment

In educational practice, test results are used for several purposes. However, validity research is especially focused on the validity of summative assessment. This article aimed to provide a general framework for validating formative assessment. The authors applied the argument‐based approach to validation to the context of formative assessment. This resulted in a proposed interpretation and use argument consisting of a score interpretation and a score use. The former involves inferences linking specific task performance to an interpretation of a student's general performance. The latter involves inferences regarding decisions about actions and educational consequences. The validity argument should focus on critical claims regarding score interpretation and score use, since both are critical to the effectiveness of formative assessment. The proposed framework is illustrated by an operational example including a presentation of evidence that can be collected on the basis of the framework.     

A diagnostic assessment for introductory molecular and cell biology

We have developed and validated a tool for assessing understanding of a selection of fundamental concepts and basic knowledge in undergraduate introductory molecular and cell biology, focusing on areas in which students often have misconceptions. This multiple-choice Introductory Molecular and Cell Biology Assessment (IMCA) instrument is designed for use as a pre- and posttest to measure student learning gains. To develop the assessment, we first worked with faculty to create a set of learning goals that targeted important concepts in the field and seemed likely to be emphasized by most instructors teaching these subjects. We interviewed students using open-ended questions to identify commonly held misconceptions, formulated multiple-choice questions that included these ideas as distracters, and reinterviewed students to establish validity of the instrument. The assessment was then evaluated by 25 biology experts and modified based on their suggestions. The complete revised assessment was administered to more than 1300 students at three institutions. Analysis of statistical parameters including item difficulty, item discrimination, and reliability provides evidence that the IMCA is a valid and reliable instrument with several potential uses in gauging student learning of key concepts in molecular and cell biology.     

Assessors’ perceptions of their judgement processes: Successful strategies and threats underlying valid assessment of student teachers

As current procedures for teacher assessment are often based on non-standardized, qualitative information derived from multiple sources, the overall validity of the assessment depends heavily on the judgement processes of the assessors. Because it is of great importance for assessors to be aware of their own judgement processes and of the possible threats to validity in these processes, investigating assessors’ perceptions is of vital significance. In the present study, the perceptions of 22 assessors who judged a student teacher pair-wise using a specific assessment procedure were explored using semi-structured interviews. A qualitative analysis of the individual assessors’ perceptions with regard to the essential judgement processes of consideration of evidence and combination of evidence to attain an overall judgement resulted in an overview of successful strategies and threats underlying a valid assessment process. General implications for ensuring the validity of the assessment process and the preparation of assessors are discussed.     

Standards Performance Continuum: Development and Validation of a Measure of Effective Pedagogy

Abstract

The authors present the development and validation of the Standards Performance Continuum (SPC), a measure for assessing teacher performance of the Standards for Effective Pedagogy. The authors created the SPC to serve as an instrument for research on the relationship between teachers' use of these standards and instructional effectiveness, as a guide for teacher professional development, and as a tool for education reform. The goal of this research was to construct a quantitative instrument that is practical, easily scored, and readily interpretable. The standards are discussed, and 3 studies provide evidence of interrater reliability, concurrent validity, and criterion-related validity supporting the validity of interpretations of data gathered with the SPC.     

Articulating the validity evidence for a science alternate assessment

Students with the most significant cognitive disabilities (SCD) are the 1% of the total student population who have a disability or multiple disabilities that significantly impact intellectual functioning and adaptive behaviors and who require individualized instruction and substantial supports. Historically, these students have received little instruction in science and the science assessments they have participated in have not included age‐appropriate science content. Guided by a theory of action for a new assessment system, an eight‐state consortium developed multidimensional alternate content standards and alternate assessments in science for students in three grade bands (3–5, 6–8, 9–12) that are linked to the Next Generation Science Standards (NGSS Lead States, 2013 ) and A Framework for K‐12 Science Education (Framework; National Research Council, 2012 ). The great variability within the population of students with SCD necessitates variability in the assessment content, which creates inherent challenges in establishing technical quality. To address this issue, a primary feature of this assessment system is the use of hypothetical cognitive models to provide a structure for variability in assessed content. System features and subsequent validity studies were guided by a theory of action that explains how the proposed claims about score interpretation and use depend on specific assumptions about the assessment, as well as precursors to the assessment. This paper describes evidence for the main claim that test scores represent what students know and can do. We present validity evidence for the assumptions about the assessment and its precursors, related to this main claim. The assessment was administered to over 21,000 students in eight states in 2015–2016. We present selected evidence from system components, procedural evidence, and validity studies. We evaluate the validity argument and demonstrate how it supports the claim about score interpretation and use.     

NSSE Benchmarks and Institutional Outcomes: A Note on the Importance of Considering the Intended Uses of a Measure in Validity Studies

Surveys play a prominent role in assessment and institutional research, and the NSSE College Student Report is one of the most popular surveys of enrolled undergraduates. Recent studies have raised questions about the validity of the NSSE survey. Although these studies have themselves been criticized, documenting the validity of an instrument requires an affirmative finding regarding the adequacy and appropriateness of score interpretation and use. Using national data from NSSE 2008, the present study found that the NSSE benchmarks provided dependable means for 50 or more students and were significantly related to important institutional outcomes such as retention and graduation rates.     

Establishing Construct Validity and Reliability for the NAESP Professional Development Inventory: Simplifying Assessment Center Techniques

The Professional Development Inventory, an instrument used in assessment centers sponsored by the National Association of Elementary School Principals, has recently been revised from a two- to a one-day process to reduce the time for those being assessed and for those assessing. This article reports a preliminary study to determine the validity and reliability of the instrument using field-based data from the first 113 candidates assessed. The assessment consists of thirteen skills assessed in twelve different simulated activities that principals might face in their daily duties. When finished, the artifacts produced during the simulations are then analyzed by trained assessors, scores entered into a computer data file, and results returned to the participants for direction in professional development. Preliminary results as reported indicate user satisfaction with the simplified process, construct validity for the thirteen skills, and a high degree of validity and reliability for the instrument.     

Using Assessment to Induct Students and Staff into the PBL Tutorial Process

The introduction of a new problem-based Graduate Medical Course with over 230 students and more than 200 staff required a systematic process for induction into the PBL philosophy and tutorial process. The 4SAT (4 Step Assessment Task) was designed for use as an instrument for formative and summative assessment of reasoning ability, elements of the PBL tutorial (group) process and individual learning outcomes. The instrument was developed through evaluative cycles based upon staff and student feedback, reflection and peer review. The 4SAT has face validity with the learning process and emphasises reasoning and integration of knowledge. An unexpected out come of its development was its use as a powerful tool to induct staff and students into the PBL group processes, scaffold development of hypothetico-deductive reasoning and provide guidance on the depth and breadth of learning expected in the course.     

Gathering Content-Related Validity Evidence for the School Leaders Licensure Assessment

A licensure assessment's purpose is to measure the relevant knowledge and skills required for safe and effective professional practice. Given the important role of licensure assessments, great care must be paid to the issue of validity: Is the assessment measuring what it claims to measure? In particular, evidence of a licensure assessment's content-related validity must be collected and evaluated prior to incorporating the assessment into a licensure process. The School Leaders Licensure Assessment was developed to be part of state licensure processes for entry-level school principals. To evaluate the use of the assessment for this purpose, a multistate panel of professionals examined the assessment and rendered judgments concerning its appropriateness. The results of this content-related validity study support the use of the School Leaders Licensure Assessment by affirming the relevance and importance of the content being assessed.     

Assessing Readiness for E‐Learning

ABSTRACT Today, e‐learning is a common delivery media for education and training within many organizations. Yet, while both the supply and demand for e‐learning opportunities has risen in recent years, many professionals are beginning to question whether e‐learn‐ers are prepared to be successful in an online learning environment (e.g., Gug‐lielmino & Guglielmino, 2003; Watkins & Corry, 2005). After all, a learner's demonstrated success in a conventional education and training classroom may not be an adequate predictor of success in an e‐learning classroom. One way of gauging a potential online learner's readiness is through self‐assessment. As a first step in defining an instrument that measures an e‐learner's readiness, with the cooperation of volunteer participants from the U.S. Coast Guard, this study looked into the validity and internal consistency of items within a self‐assessment of e‐learning readiness that is under development, and provided data for the continuing development and revision of the instrument. Having demonstrated evidence of internal consistency and construct validity, the self‐assessment now provides a tool for continuing research into the prediction of e‐learning performance. Funding for this study was provided by the International Society for Performance Improvement.