Teachers’ and students’ perceptions of assessments: A review and a study into the ability and accuracy of estimating the difficulty levels of assessment items

In today's higher education, high quality assessments play an important role. Little is known, however, about the degree to which assessments are correctly aimed at the students’ levels of competence in relation to the defined learning goals. This article reviews previous research into teachers’ and students’ perceptions of item difficulty. It focuses on the item difficulty of assessments and students’ and teachers’ abilities to estimate item difficulty correctly. The review indicates that teachers tend to overestimate the difficulty of easy items and underestimate the difficulty of difficult items. Students seem to be better estimators of item difficulty. The accuracy of the estimates can be improved by: the information the estimators or teachers have about the target group and their earlier assessment results; defining the target group before the estimation process; the possibility of having discussions about the defined target group of students and their corresponding standards during the estimation process; and by the amount of training in item construction and estimating. In the subsequent study, the ability and accuracy of teachers and students to estimate the difficulty levels of assessment items was examined. In higher education, results show that teachers are able to estimate the difficulty levels correctly for only a small proportion of the assessment items. They overestimate the difficulty level of most of the assessment items. Students, on the other hand, underestimate their own performances. In addition, the relationships between the students’ perceptions of the difficulty levels of the assessment items and their performances on the assessments were investigated. Results provide evidence that the students who performed best on the assessments underestimated their performances the most. Several explanations are discussed and suggestions for additional research are offered.     

A cross-sectional study of mathematics achievement, estimation skills, and academic self-perception in students of varying ability

This study investigated students' mathematics achievement, estimation ability, use of estimation strategies, and academic self-perception. Students with learning disabilities (LD), average achievers, and intellectually gifted students (N = 135) in fourth, sixth, and eighth grade participated in the study. They were assessed to determine their mathematics achievement, ability to estimate discrete quantities, knowledge and use of estimation strategies, and perception of academic competence. The results indicated that the students with LD performed significantly lower than their peers on the math achievement measures, as expected, but viewed themselves to be as academically competent as the average achievers did. Students with LD and average achievers scored significantly lower than gifted students on all estimation measures, but they differed significantly from one another only on the estimation strategy use measure. Interestingly, even gifted students did not seem to have a well-developed understanding of estimation and, like the other students, did poorly on the first estimation measure. The accuracy of their estimates seemed to improve, however, when students were asked open-ended questions about the strategies they used to arrive at their estimates. Although students with LD did not differ from average achievers in their estimation accuracy, they used significantly fewer effective estimation strategies. Implications for instruction are discussed.     

The Cognitive Correlates of Computational Estimation Skill Among Third‐Grade Students

The purpose of this study was to examine the relations of various cognitive abilities and aspects of math performance with computational estimation skill among third graders. Students (n= 315) were assessed on language, nonverbal reasoning, concept formation, processing speed, long‐term memory, working memory, inattentive behavior, basic reading skill, arithmetic number combination skill, double‐digit computation skill, and computational estimation ability. One‐way analysis of variance indicated significant differences in estimation skill among students of low, average, and high math computation performance. The unique predictors of estimation skill were arithmetic number combination skill, nonverbal reasoning, concept formation, working memory, and inattentive behavior.     

Early identification of young struggling readers: Preliminary benchmarks for intervention for students in years one and two in schools in New South Wales

It is now widely acknowledged that early intervention programs should be provided for young students who are experiencing difficulty in learning to read. It is important, therefore, to have efficient ways of identifying students who are most in need of intervention. The aim of this study was to establish preliminary performance benchmarks to identify the bottom 25% of year 1 and year 2 students in NSW schools at two key points in the school year. Students from grades 1 and 2 (n = 335) at two schools nominated as performing at state average levels were assessed on a battery of tests assessing various aspects of reading and related skills at the beginning and in the middle of the school year. Preliminary estimates of benchmarks for determining the bottom quartile of students were thereby obtained together with estimates of average performance at these two points in the school year to serve as performance goals for intervention.     

Estimating Linear Size and Scale: Body rulers

The National Science Education Standards emphasise the use of concepts and skills that cut across the science domains. One of these cross‐cutting areas is measurement. Students should know measurement systems, units of measurement, tools and error in measurement as well as the importance of measurement to scientific endeavours. Even though measurement is an essential skill, little is known about how students estimate and use measurement in different contexts. This study examines the impact of teaching students to use their bodies as rough measurement tools (body rulers) on their ability to estimate linear measurements. Nineteen middle school students participated in metric intervention tasks and completed a pre‐instruction and a post‐instruction Linear Measurement Assessment. Results showed that teaching students to use rough body measures as tools (a body ruler) for estimation had a significant influence on their estimation accuracy. After instruction, students were better able to estimate the sizes of objects, use their body in making estimations of size, and estimate while touching an object or pacing a distance. Furthermore, proportional reasoning was significantly correlated with students’ post‐instruction scores on the Linear Measurement Assessment.     

Teaching collaborative skills with a group leader computer tutor

This paper discusses how a group leader computer tutor may aid students in learning collaborative skills in a co-operative learning environment. However, students need to learn collaborative skills and practice using them. The group leader computer tutor discussed in this paper is designed on the principles of co-operative learning, intelligent tutoring systems and computer-supported collaborative work within an intelligent collaborative learning system (ICLS). The group leader aims to facilitate group work on the task level and to teach students how to use collaborative skills in the discussion level as students work on networked computers in the Jigsaw method of co-operative learning. The ICLS and its group leader were used by two classes at a liberal arts university. Qualitative research shows that the students' co-operative attitudes changed and academic achievement improved from pre- to post-treatment. Students, especially, used the communication skill of openness in comment type discussions. The students enjoyed working collaboratively through the ICLS and their teachers thought that the experience was valuable for them.     

Assessment of mathematics and reading performance: An examination of the correspondence between direct assessment of student performance and teacher report

Teacher judgments have been identified as a primary source of information regarding student academic achievement. Research examining the accuracy of teachers' judgments in assessing students' academic abilities has shown relatively high accuracy. However, previous studies have relied primarily on norm‐referenced measures to obtain estimates of students' achievement in reading and mathematics. Recent developments in the assessment of students' academic skills, such as Curriculum‐Based Measurement (CBM), provide a direct estimate of students' skill levels in basic areas such as reading and mathematics. The purpose of the present study was to examine the extent to which teachers' perceptions of students' reading and mathematics skills corresponded to direct estimates of students' reading and mathematics skills. Two second‐grade teachers estimated the reading and mathematics skills of 33 second‐grade students. Results of this study indicated that teachers were not accurate in assessing their students' mathematics functioning. Teachers were more accurate in assessing the occurrence of Mastery mathematics levels in basic addition, but were very inaccurate in assessing the occurrence of Mastery, Instructional, or Frustrational mathematics levels in all other skills assessed. In reading, teachers' judgment accuracy varied as a function of grade‐level material and instructional level. Specifically, teachers experienced considerable difficulty accurately identifying students who were reading at a Mastery level in grade‐level or above‐grade‐level material. © 2006 Wiley Periodicals, Inc. Psychol Schs 43: 247–265, 2006.     

Remediating Computational Deficits at Third Grade: A Randomized Field Trial

The major purposes of this study were to assess the efficacy of tutoring to remediate 3rd-grade computational deficits and to explore whether remediation is differentially efficacious depending on whether students experience mathematics difficulty alone or concomitantly with reading difficulty. At 2 sites, 127 students were stratified on mathematics difficulty status and randomly assigned to 4 conditions: word recognition (control) tutoring or 1 of 3 computation tutoring conditions: fact retrieval, procedural computation and computational estimation, and combined (fact retrieval + procedural computation and computational estimation). Results revealed that fact retrieval tutoring enhanced fact retrieval skill, and procedural computation and computational estimation tutoring (whether in isolation or combined with fact retrieval tutoring) enhanced computational estimation skill. Remediation was not differentially efficacious as a function of students' mathematics difficulty status.     

Remediating Computational Deficits at Third Grade: A Randomized Field Trial

Abstract

The major purposes of this study were to assess the efficacy of tutoring to remediate 3rd-grade computational deficits and to explore whether remediation is differentially efficacious depending on whether students experience mathematics difficulty alone or concomitantly with reading difficulty. At 2 sites, 127 students were stratified on mathematics difficulty status and randomly assigned to 4 conditions: word recognition (control) tutoring or 1 of 3 computation tutoring conditions: fact retrieval, procedural computation and computational estimation, and combined (fact retrieval + procedural computation and computational estimation). Results revealed that fact retrieval tutoring enhanced fact retrieval skill, and procedural computation and computational estimation tutoring (whether in isolation or combined with fact retrieval tutoring) enhanced computational estimation skill. Remediation was not differentially efficacious as a function of students' mathematics difficulty status.     

预防医学专业学生实践技能培养模式探讨

目的：探索预防医学专业学生实践技能培养模式,提高学生实践技能水平。方法：采取综合性设计性实验训练、学生参与教师科研、学生申报课题、创新性实验等实践。结果：学生实践能力得到较好的锻炼;科研能力得到较大提高,2012年学生实践技能活动大幅度提升,创新性实验申报7项,大学生科研申报60多项,学生科技论文发表10篇以上;实践技能全面强化,得到实习单位和用人单位的好评。结论：学生的实践技能有一定的提高,但在进入实习、社会实际中还有一定的差距,实践技能培养模式还待进一步的探讨。     

Assessing high school chemistry students’ modeling sub-skills in a computerized molecular modeling learning environment

Much knowledge in chemistry exists at a molecular level, inaccessible to direct perception. Chemistry instruction should therefore include multiple visual representations, such as molecular models and symbols. This study describes the implementation and assessment of a learning unit designed for 12th grade chemistry honors students. The organic chemistry part of the unit was taught in a Computerized Molecular Modeling (CMM) learning environment, where students explored daily life organic molecules through assignments and two CMM software packages. The research objective was to investigate the effect of the CMM learning unit on students’ modeling skill and sub-skills, including (a) drawing and transferring between a molecular formula, a structural formula, and a model, and (b) transferring between symbols/models and microscopic, macroscopic, and process chemistry understanding levels. About 600 12th grade chemistry students who studied the CMM unit responded to a reflection questionnaire, and were assessed for their modeling skill and sub-skills via pre- and post-case-based questionnaires. Students indicated that the CMM environment contributed to their understanding of the four chemistry understanding levels and the links among them. Students significantly improved their scores in the five modeling sub-skills. As the complexity of the modeling assignments increased, the number of students who responded correctly and fully decreased. We present a hierarchy of modeling sub-skills, starting with understanding symbols and molecular structures, and ending with mastering the four chemistry understanding levels. We recommend that chemical educators use case-based tools to assess their students’ modeling skill and validate the initial hierarchy with a different set of questions.     

Effects of curriculum within curriculum-based measurement.

This study assessed the effects of curriculum on technical features within curriculum-based measurement in reading. Curriculum was defined as the difficulty of material and the basal series from which students read. Technical features were the criterion validity and developmental growth rates associated with the measurement. Ninety-one students took a commercial, widely used test of reading comprehension and read orally for 1 minute from each of 19 passages, one from each grade level within two reading series. Correlations between the oral reading samples and the test of reading comprehension were similar across difficulty levels and across series. Developmental growth rates also remained strong regardless of difficulty level and series.     

Reading and spelling difficulties in high school students: Causes and consequences

Basic skills in reading and spelling and supporting metalinguistic abilities were assessed in ninth and tenth grade students in two school settings. Students attending a private high school for the learning disabled comprised one group and the other comprised low to middle range students from a public high school. Both the LD students and the regular high school students displayed deficiencies in spelling and in decoding, a factor in reading difficulty that is commonly supposed to dwindle in importance after the elementary school years. Treating the overlapping groups as a single sample, multiple regression analysis was used to investigate the contribution of nonword decoding skill and phonological and morphological awareness to spelling ability. The analysis revealed that decoding was the major component, predicting about half of the variance in spelling. The effect of phonological awareness was largely hidden by its high correlation with decoding, but was a significant predictor of spelling in its own right. Morphological awareness predicted spelling skill when the words to be spelled were morphologically complex. An additional study showed that differences in decoding and spelling ability were associated with differences in comprehension after controlling for reading experience and vocabulary. Even among experienced readers individual differences in comprehension of text reflect efficiency of phonological processing at the word level.     

The University Experiences of Students with Learning Disabilities

To explore the university experiences of students with learning disabilities (LD), 63,802 responses to the 2014 Student Experience in the Research University Survey were analyzed. Compared to other students, those with self‐reported LD (5.96 percent) had difficulty with assignments and had more obstacles caused by nonacademic responsibilities and imposed by their skill levels. Students with self‐reported LD sensed more bias toward people with disabilities on campus, and they were less satisfied with their overall experience. Interactions between disability status and age suggested even more challenges for older students who self‐reported LD. Approximately one‐third of students who self‐reported LD received accommodations. The rate of accommodations was higher among individuals who were wealthy, who lived alone, and who were out‐of‐state students. Compared to students who self‐reported LD but reported no accommodations, those with accommodations had more contact with faculty and less difficulty with assignments.     

Deutsches Museum,München‐‐Science,Technology and History as an Educational Challenge

Difficulties in reading comprehension of physics texts by college students were investigated. A strategy was developed to improve students’ reading comprehension. Students were taught the skill of formulating clear questions on textual material. The strategy stimulated students’ awareness of their difficulties in reading comprehension and could be used by students as a self‐monitoring technique to improve their reading comprehension.     

Standard Errors of Proportions Used in Reporting Changes in School Performance With Achievement Levels

The primary purpose of this study was to estimate the amount of variability in the proportions of students in a school district, scoring within each of three achievement levels that could be attributed to factors other than random sampling error. The approach taken is based on a general conceptual framework that collectively incorporates five sources of variability: instructional intervention, random sampling error, measurement error, equating error, and systematic error. Statewide school-level assessment data for reading and mathematics in grades four and eight from four consecutive years were used to examine annual grade-group change. The intent was to assess the impact of random sampling error in grade-group change estimates when either single-year proportions or 2-year average proportions are used to report school improvement with achievement levels. Observed variability in change was compared with theoretically-derived estimates of change due to random sampling error to determine the relative influence of sampling error and the aggregate of the other four sources of variability. Results indicate that the error variance of estimates of change at the school level is large enough to interfere with interpretations of annual change estimates. Recommendations are offered for establishing annual improvement goals and for reporting results with achievement levels-all in the context of adequate yearly progress (AYP)-while taking error estimates into account.     

Climbing Bloom's taxonomy pyramid: Lessons from a graduate histology course

Bloom's taxonomy was adopted to create a subject‐specific scoring tool for histology multiple‐choice questions (MCQs). This Bloom's Taxonomy Histology Tool (BTHT) was used to analyze teacher‐ and student‐generated quiz and examination questions from a graduate level histology course. Multiple‐choice questions using histological images were generally assigned a higher BTHT level than simple text questions. The type of microscopy technique (light or electron microscopy) used for these image‐based questions did not result in any significant differences in their Bloom's taxonomy scores. The BTHT levels for teacher‐generated MCQs correlated positively with higher discrimination indices and inversely with the percent of students answering these questions correctly (difficulty index), suggesting that higher‐level Bloom's taxonomy questions differentiate well between higher‐ and lower‐performing students. When examining BTHT scores for MCQs that were written by students in a Multiple‐Choice Item Development Assignment (MCIDA) there was no significant correlation between these scores and the students' ability to answer teacher‐generated MCQs. This suggests that the ability to answer histology MCQs relies on a different skill set than the aptitude to construct higher‐level Bloom's taxonomy questions. However, students significantly improved their average BTHT scores from the midterm to the final MCIDA task, which indicates that practice, experience and feedback increased their MCQ writing proficiency. Anat Sci Educ 10: 456–464. © 2017 American Association of Anatomists.     

A word identification strategy for adolescents with learning disabilities

Students with learning disabilities frequently experience difficulty on reading tasks. This difficulty is heightened for adolescents with learning disabilities who are responsible for reading and understanding materials written at several grade levels above their reading ability. Word identification becomes an increasingly important skill for these students, especially when confronted with unfamiliar, polysyllabic words. The present study investigated the effects of training 12 adolescents with learning disabilities in a word identification strategy, DISSECT. The results indicated that the strategy was effective in reducing reading errors for all subjects. However, it was found that increases in word identification differentially affected reading comprehension and indicate the need for separate and/or simultaneous attention to comprehension processes.     

A Cohort-Driven Assessment Task for Scientific Report Writing

A formative assessment task was developed to improve the scientific report writing skills of university students. Students undertaking this task typically possessed varying levels of scientific literacy and were drawn from a cohort of mixed abilities. The assessment task involved the construction of a scientific report that included feedback from instructor to students before final submission of the assessment piece. After initial submission of a scientific report, the instructor developed a cohort-specific marking scheme based on the deficiencies that were evident within the class group. Using a mixture of peer and self-review against specific criteria, the students were required to resubmit an amended report. This resulted in elevated marks compared with those that would have been obtained after first submission, thus rewarding the student for the application of feedback. This technique proved to be efficient for both parties and also resulted in improvement of skills of the entire student population, regardless of the ability of the student prior to the assessment task. Using this methodology, students of varying aptitudes were able to measure their own skill improvement against tangible criteria, and enjoy a degree of learning success independent of the ranking within their group.     

Computational Estimation Procedures of School Children

Abstract

The necessity for developing computational estimation procedures has been well established. However, the teaching of this skill has been neglected in schools. There is little information available on the cognitive processes used by most school-aged children when doing estimation problems. In this study, 26 middle school students, representing a variety of backgrounds and achievement levels, were individually asked to estimate answers to 12 computational problems and explain how they obtained their answers. They were not allowed to use writing materials. Results indicate that estimation skills are highly dependent upon a student’s "number sense."