Peer assessment accounting for student agreement

Peer assessment has been gaining significant ground as a means of fostering an environment of accountability and responsibility for group projects in higher education. A popular peer assessment method assigns individual grades via a linear relationship to peer assessment scores. Using this method, a group member who receives his or her group’s average peer assessment score is assigned the instructor‐given group grade; the group member who receives a total peer assessment score of zero is assigned an individual grade of zero. Unfortunately, applying this method often requires time‐consuming exploratory data analysis because of the potential for the method to yield a high spread of grades, including grades over 100%. The present study proposes a modification, which accounts for the degree of student agreement. When the modified procedure results in a change from the group grade, the direction of change is always the same, but with smaller magnitude, requiring less subsequent data analysis and instructor time demands. Easy‐to‐modify SAS code for applying the method is presented in the Appendix.     

The development of a questionnaire on metacognition for students in higher education

Interest in the role of metacognition has been steadily rising in most forms of education. This study focuses on the construction of a questionnaire for measuring metacognitive knowledge, metacognitive regulation and metacognitive responsiveness among students in higher education and the subsequent process of testing to determine its validity.

Purpose

The aim of the study was to construct an original instrument for measuring features of metacognition, henceforth referred to as the Awareness of Independent Learning Inventory (AILI), and further to establish the similarities and differences between this model and existing instruments for measuring metacognition.

Sample

The AILI questionnaire was distributed to 1058 students in various types of Teacher Training Institutes in the Netherlands and Belgium. The abridged English version of the questionnaire was administered to another sample of 729 students reading Economics and Business Administration at the University of Maastricht in the south of the Netherlands.

Design and methods

The AILI instrument was constructed on the basis of a facet design along two dimensions: components of metacognition and topics of concern to students in higher education. The data gathered with the instrument was analyzed by means of a generalisability study and a decision study, respectively. The validity of the instrument was investigated by using confirmatory factor analysis.

Results

The generalisability study showed that the reliability of the instrument was satisfactory. The decision study revealed that the number of items included in the questionnaire could be reduced substantially by leaving out two components of one of the dimensions in the facet design, without losing too much generalisability. The validity study showed that there was a considerable level of congruity between parts of the AILI questionnaire and the relevant parts of the Motivated Strategies for Learning Questionnaire (MSLQ).

Conclusions

The AILI questionnaire is a reliable and valid instrument for measuring metacognitive knowledge, regulation and responsiveness. It is suitable for use in the evaluation of the effects of interventions that purport to increase metacognitive knowledge, regulation and responsiveness of students in higher education.     

The impact of the scoring system of a large-scale standardized EFL writing assessment on its score variability and reliability: Implications for assessment policy makers

Using generalizability (G-) theory and rater interviews as research methods, this study examined the impact of the current scoring system of the CET-4 (College English Test Band 4, a high-stakes national standardized EFL assessment in China) writing on its score variability and reliability. One hundred and twenty CET-4 essays written by 60 non-English major undergraduate students at one Chinese university were scored holistically by 35 experienced CET-4 raters using the authentic CET-4 scoring rubric. Ten purposively selected raters were further interviewed for their views on how the current scoring system could impact its score variability and reliability. The G-theory results indicated that the current single-task and single-rater holistic scoring system would not be able to yield acceptable generalizability and dependability coefficients. The rater interview results supported the quantitative findings. Important implications for the CET-4 writing assessment policy in China are discussed.     

The assessment of lesson plans in teacher education: a case study in assessment validity and reliability

This paper forms part of an exploration of assessment on one part‐time higher education (HE) course: an in‐service, professional qualification for teachers and trainers in the learning and skills sector which is delivered on a franchise basis across a network of further education colleges in the north of England. This paper proposes that the validity and reliability of portfolio‐based assessment, a key component of many HE programmes in addition to the course being researched here, is contestable. Analysis of the processes of compiling portfolios for assessment, through the conceptual framework of the New Literacy Studies, suggests that the ways in which portfolios are assessed and the ways in which the crucial requisites of validity and reliability are assigned to them, mask complexities and contradictions in their creation by the student. This paper argues for a new, critical analysis of portfolio production and raises a number of questions about the validity, reliability and authenticity of the assessment process that the portfolios reify.     

Peer assessment in group projects accounting for assessor reliability by an iterative method

This study proposes an advanced method to factor in the contributions of individual group members engaged in an integrated group project using peer assessment procedures. Conway et al. proposed the Individual Weight Factor (IWF) method for peer assessment which has been extensively developed over the years. However, most methods associated with IWF use a simple average algorithm which still has limitations in effectively reflecting abnormal situations. Therefore, this study proposes an iterative method for measuring a fair IWF by considering each assessor's reliability. The proposed algorithm considers each assessor's reliability which is evaluated by an analysis of variance of the scores given by the assessor. This relative reliability factor is used to improve the fairness of the IWF and reduce the effect of unfair or unreliable assessors' scores. A comparison with previous methods demonstrates that the proposed method resulted in substantially more fair grades being given to individual students for group projects.     

The impact of a rubric and friendship on peer assessment: Effects on construct validity,performance, and perceptions of fairness and comfort

Construct validity of peer assessment (PA) is important for PA application, yet difficult to achieve. The present study investigated the impact of an assessment rubric and friendship between the assessor and assessee on construct validity of PA. Two-hundred nine bachelor students participated: half of them assessed a peer's concept map with a rubric whereas the other half did not use a rubric. The results revealed a substantial reliability and construct validity for PA. All students over-score their peers’ performance, but students using a rubric were more valid. Moreover, when using a rubric a high level of friendship between assessor and assessee resulted in more over-scoring. Use of a rubric resulted in higher quality concept maps for peer and expert ratings.     

基于Wiener过程和竞争失效分析的系统可靠性评估

考虑突发失效和性能退化之间的相关性和竞争关系,研究了基于竞争失效分析的产品可靠性评估问题.将突发失效的发生率视为性能退化量的函数,并采用Weibull分布加以描述;利用Wiener过程描述性能退化过程,采用极大似然估计法估计模型参数;提出了一种基于竞争失效分析的可靠性评估模型.以一种GaAs激光器性能退化和失效数据为例,验证模型及其求解方法的有效性.结果表明：若仅考虑退化失效将使得评估结果偏向乐观,突发失效与退化失效的相关性对可靠性评估的准确性有很大影响.     

Leading assessment for enhanced student outcomes

This paper reports the leadership findings that emerged from the Alberta Student Assessment Study, a large-scale study employing a mixed-method design. Findings included the importance of assessment leadership, credibility of leaders, capacity to make hard decisions and ability to lead professional learning. An optimal assessment leadership profile emerged which includes values and assessment knowledge, mediated by personal qualities and accompanied by an interplay between procedural knowledge and professional skills. This informs leadership vision and actions leading to increased assessment literacy, more effective monitoring and reporting practice, enhanced instructional practice, enriched partnerships with parents and community members and positive student outcomes.     

The influence of student gender on the assessment of undergraduate student work

The aim of this study was to investigate the influence of perceived student gender on the feedback given to undergraduate student work. Participants (n = 12) were lecturers in higher education and were required to mark two undergraduate student essays. The first student essay that all participants marked was the control essay. Participants were informed that the control essay was written by Samuel Jones (a male student). Participants then marked the target essay. Although participants marked the same essay, half of the participants (n = 6) were informed that the student essay was written by Natasha Brown (a female student), while the remaining participants were informed that it was written by James Smith (a male student). In-text and end-of-text feedback were qualitatively analysed on six dimensions: academic style of writing; criticality; structure, fluency and cohesion; sources used; understanding/knowledge of the subject; and other. Analysis of feedback for both the control and target essay revealed no discernible differences in the number of comments (strengths of the essay, areas for improvement) made and the content and presentation of these comments between the two groups. Pedagogical implications pertaining to the potential impact of anonymous marking on feedback processes are discussed.     

基于比例风险退化模型的产品可靠性评估

为了利用退化数据评估长寿命产品的可靠性,提出一种新的比例风险退化模型.利用时间退化数据与应力加速寿命之间的相似关系,并假定退化数据的失效概率函数是时间协变量的比例函数,从而实现基于比例风险退化模型的可靠性评估.采用最小二乘法估计模型参数.由已知时刻的退化量失效率及比例函数外推产品在特定时间、预定失效阈值下的失效率及可靠度函数.以一种长寿命GaAs激光器的退化数据为例,完成该产品的可靠性评估.结果表明：所提方法能准确描述退化过程,是长寿命产品可靠性评估的一种有效方法.     

含公布电源配电网可靠性的评估方案研究

在原有配电网引入分布式电源的情况下,提出一种基于应用改进最小路法结合可靠性指标求解评估含分布式电源distributed generation（DG）的配电网可靠性评估方案。以IEEE-RBTS Bus6 系统为例进行不同方案的仿真。仿真结果表明,所提出的可靠性评估模型可以明显地改善配电网系统对用户提供电能的可靠性。     

Evaluation of curriculum and student learning needs using 360 degree assessment

This research used a 360 degree assessment tool modelled from the competing values framework to assess the curriculum. A total of 100 Master’s of Business Administration students and 746 of their work colleagues completed the 360 degree assessment tool. The students were enrolled in a course on leadership and management. The results of the assessment demonstrated similar patterns of strengths and development needs between the mean self-assessment of the students and those of their work colleagues, although work colleagues tended to assess the students slightly higher. The patterns revealed development needs in soft skills such as conflict management and mentoring, as well as in strategic management and project planning. Gender effects were evident with women needing particular development in competencies related to developing and communicating a vision, setting goals and objectives and designing and organising. The results provide evidence-based approach to assessing curriculum. As many programmes use a variety of assessment tools in their programmes, collecting data across cohorts can be a valuable evidence-based strategy to assess and evaluate curriculum content against real learner needs.     

The quality assurance of a national English writing assessment: Policy implications for quality improvement

The purpose of this study was to examine the quality assurance issues of a national English writing assessment in Chinese higher education. Specifically, using generalizability theory and rater interviews, this study examined how the current scoring policy of the TEM-4 (Test for English Majors – Band 4, a high-stakes national standardized EFL assessment in China) writing could impact its score variability and reliability. Eighteen argumentative essays written by nine English major undergraduate students were selected as the writing samples. Ten TEM-4 raters were first invited to use the authentic TEM-4 writing scoring rubric to score these essays holistically and analytically (with time intervals in between). They were then interviewed for their views on how the current scoring policy of the TEM-4 writing assessment could affect its overall quality. The quantitative generalizability theory results of this study suggested that the current scoring policy would not yield acceptable reliability coefficients. The qualitative results supported the generalizability theory findings. Policy implications for quality improvement of the TEM-4 writing assessment in China are discussed.     

Test‐based accountability: Potential benefits and pitfalls of science assessment with student diversity

Recent test‐based accountability policy in the U.S. has involved annually assessing all students in core subjects and holding schools accountable for adequate progress of all students by implementing sanctions when adequate progress is not met. Despite its potential benefits, basing educational policy on assessments developed for a student population of White, middle‐ and upper‐class, and native speakers of English opens the door for numerous pitfalls when the assessments are applied to minority populations including students of color, low SES, and learning English as a new language. There exists a paradox; while minority students are a primary intended beneficiary of the test‐based accountability policy, the assessments used in the policy have been shown to have many shortcomings when applied to these students. This article weighs the benefits and pitfalls that test‐based accountability brings for minority students. Resolutions to the pitfalls are discussed, and areas for future research are recommended. © 2009 Wiley Periodicals, Inc. J Res Sci Teach 47: 6–24, 2010     

Research Abstracts and Bibliographies

ABSTRACT

The authors address the reliability of scores obtained on the summative performance assessments during the pilot year of our research. Contrary to classical test theory, we discussed the advantages of using generalizability theory for estimating reliability of scores for summative performance assessments. Generalizability theory was used as the framework because of the flexibility this approach provides for examining sources of inconsistency within a complex assessment. Two major sources of inconsistency on scores considered in this study were raters and agencies (teachers' rating vs. researchers' rating). Overall, results showed that the inconsistency in scores attributable to raters and agencies was relatively small. Suggestions regarding improvement of consistency in the subsequent years of our research were provided.     

The content aspect of validity in a rubric-based assessment system for course syllabuses

The growing trend among universities to promote systems of programme and course evaluation entails more responsibility for faculties and departments. These systems require resources to ensure that they are not only valid and reliable but also effective and sustainable. The design of rubric-based assessment systems may provide a solution, but there is a gap in the research on curriculum evaluation concerning their use and validation. We examine the content aspect of validity in a rubric-based assessment system for course syllabuses using a mixed method that combines an analysis of the agreement among 23 experts with a phenomenographic study. With data gathered through a questionnaire linked to the Delphi technique, content validity indexes were calculated and the experts' different perspectives were identified. The content validity indexes (greater than .80) met the standards set out in literature, and the qualitative study of the experts' feedback showed three different perspectives on the system's use. Beyond providing evidence of the system's content validity, the study highlights the extent to which it is important to give appropriate consideration to experts' – and by extension final users' – experience in order to ensure the successful implementation of rubric-based assessment systems.     

The Five-Factor Model personality assessment for improved student design team performance

Researchers have long noted the correlation of various personality traits and team performance. Studies relating aggregate team personality traits to team performance are scattered in the literature and may not always be relevant to engineering design teams. This paper synthesizes the results from applicable Five-Factor Model (FFM)-based personality studies related to engineering design team performance, into a form that can be readily used by non-experts—engineering faculty and students. In addition, an approach is presented where aggregate data is visually presented to recognize patterns that correspond to strength and existence of personality traits within the team as measured by the FFM model while maintaining student confidentiality. With this approach, identification of team strengths and weaknesses stemming from the personality trait distribution is simplified. An assessment of the usability of the approach—completed in two first-year engineering courses—is presented to demonstrate its potential.