Examining the examiners: How inexperienced examiners approach the assessment of research theses

Earlier research by Mullins and Kiley (2002) [Studies in higher education, 27(4), 369–386] reported on the processes that experienced examiners go through when they assess research theses. Since that study two further studies have been undertaken, interviews with novice Australian examiners, reported here, and the analysis of approximately 100 examiners’ reports (Kiley, 2004). These studies now allow us to address the comment made by many of the experienced examiners who were interviewed that it was not wise to send a research thesis, particularly one which is not strong, to inexperienced examiners. We also wanted to determine the extent to which less experienced examiners follow the same processes and use the same criteria as their more experienced colleagues, and whether there was any basis for the experienced examiners’ comments. This paper draws particularly on the results of the interviews with 26 examiners and supervisors which indicated that by comparison with experienced examiners, less experienced examiners paid more attention to institutional criteria and the summative dimension of assessment; were unsure of the boundary conditions relating to very good or poor theses; and were more likely to refer back to their own postgraduate experience when reflecting on their approach to both supervision and examination.     

Examining PhD and research masters theses

The examination of research theses has only relatively recently attracted research interest that has focused on what examiners do and how consistent they are. The research questions in this study address firstly whether PhD and research masters theses were treated by examiners as qualitatively different on 12 indicators of importance across the areas: contribution of the thesis, the literature review, approach and methodology, analyses and results and presentation. Secondly what was the examiner assessment of quality of a recently examined thesis on the same indicators and, finally, how well the indicators were reflected in a holistic assessment of thesis quality. The work reported here draws on the responses of 353 PhD and 74 research masters thesis examiners. Findings showed the examiners generally rated the relative importance of the indicators very similarly at both degree levels. Further the order of importance across indicators was essentially the same for the two levels of thesis. Anticipated differences did emerge with the examiners giving higher quality gradings for all contribution indicators for PhD as compared with research masters theses. The 12 specific quality indicators, individually and collectively were strongly related to the holistic assessment of thesis quality, particularly at the PhD level.     

Investigating PhD thesis examination reports

There has been a slow but steady accretion of findings on doctoral assessment and examination processes over the past decade and a half. The study of Australian PhD examination reported here draws on the written reports on 301 theses across all discipline areas. Text categories identified in the reports are linked to other data including the recommendation examiners give the thesis and the final report of the committee. The assessment discourse of examiners is explored in depth with reference to high- and low-quality theses. The questions addressed are: (1) What qualities, attributes and characteristics of the thesis are examiners emphasising in their reports? (2) What form does their evaluative comment take? (3) What can the reports indicate about the relative strengths and weaknesses of a thesis and the standards examiners apply in determining a recommendation? and (4) what areas of report content and types of comment most clearly differentiate between the high-and low-quality theses? While it was evident in the reports that examiners weigh and balance many features of the thesis against their expectations to arrive at a recommendation, there were three general criteria that distinguished each of high-quality low-quality theses.     

Examiner practices and culturally inflected doctoral theses

Increase in numbers of postgraduate students worldwide represent an opportunity and necessity for nurturing and recognising the diversity of culturally inflected research topics, methodologies and expression. However, there are tensions in the definitions, encouragement and recognition of diversity in theses, and in balances of power in supervisor, student and examiner relationships involved in doctoral achievement. We consider such tensions and opportunities using small-scale qualitative research with experienced examiners. We explore ways in which examiners perceive they take culturally aware approaches to responding to, enabling and rewarding the quality of culturally inflected doctoral theses.     

Assessing the doctoral thesis when it includes published work

In this article we explore how examiners assess a thesis that includes published work. An online survey was used to gather data on approaches to assessing publication-based theses (PBTs). The respondents were 62 supervisors who had experience examining PBTs across a range of disciplines at a research-intensive university in New Zealand. Nearly half of the respondents had examined ‘hybrid’ theses with papers inserted as chapters, 41% had examined theses with publications appended and 14% had examined PhDs by publication (i.e. papers alone). Twenty-nine per cent of examiners used their own extended set of criteria to assess PBTs, 48% found them easier to assess but 26% wanted more guidance. Our analysis also indicated that 86% of the examiners were highly influenced by publications in top-ranked journals and international peer-reviewed journals. Among the concerns of the examiners was the intellectual input of the candidate in any multi-authored publication, as well as the coherence of the thesis. We recommend the need for clearer guidelines for doctoral candidates, supervisors and examiners managing PBTs.     

Examining the Creative Arts Doctorate in Australia: Implications for supervisors

Abstract

One of the significant roles performed by the higher degree research (HDR) supervisor is to assist students to prepare their dissertations for examination. At a time when there is increasing interest in how the academy manages the transition of creative arts HDR candidates from apprentice to peer, there is also concern about the processes, practices, and policies associated with this largely under-researched area of research training. In a recent national Office of Learning and Teaching funded project, we investigated the policy expectations, expert and peer beliefs and expectations, and examiners’ practice around HDR examination, and canvassed the creative arts academic community for their recommendations on best practice in the examination of creative arts doctorates. An unexpected finding was the role of the HDR supervisor in relation to these key areas, and the impact of supervisors upon the examination of students’ theses. This article presents our findings with special reference to the role, understandings, and aspirations of HDR supervisors in the context, and process, of preparing their students for creative arts HDR examination.     

Churches and the Australian community

Abstract

It is rare for a PhD candidate who submits a thesis for examination to fail outright. If a thesis exhibits significant flaws the candidate may be required to make major revisions and re‐submit the work for re‐examination. The written comments of examiners before and after resubmission can provide important insights into the process of examination and the qualities examiners identify in a marginal thesis. Drawing on 101 of the most recent, completed theses across fields in one Australian university, this article investigates the differences in examiner comment on the qualities of theses by the same candidates before and after major revision and re‐submission (N = 6), and between these theses and those that were ‘passed’ at the first examination (N=95). Critical comments about the literature review and the degree to which the examiner moved into a supervisory role were found to be strong indicators of theses at the margin’.     

What examiners do: what thesis students should know

Although many articles have been written about thesis assessment, none provide a comprehensive, general picture of what examiners do as they assess a thesis. To synthesise this diverse literature, we reviewed 30 articles, triangulated their conclusions and identified 11 examiner practices. Thesis examiners tend to be broadly consistent in their practices and recommendations; they expect and want a thesis to pass, but first impressions are also very important. They read with academic expectations and the expectations of a normal reader. Like any reader, thesis examiners get annoyed and distracted by presentation errors, and they want to read a work that is a coherent whole. As academic readers, examiners favour a thesis with a convincing approach that engages with the literature and the findings, but they require a thesis to be publishable research. Finally, examiners give not only a final evaluation of a thesis, but also instruction and advice to improve the thesis and further publications and research. We hope that these generalisations will demystify the often secret process of assessing a thesis, and reassure, guide and encourage students as they write their theses.     

The processes and parameters of Fine Art PhD examination

The merger of art schools into academic institutions and the consequent proliferation of higher degree courses in the visual arts has created pressure for these courses to justify that their research content ‘measures up’ to more traditional research practices.This study aimed to identify the parameters for PhD examination and research practices in the field of Fine Art through interviews with Fine Art examiners as well as an analysis of written PhD examination reports. A comparison of PhD examination reports across disciplines revealed that there were similarities as well as some significant differences between the type of feedback provided by Fine Art examiners and feedback by examiners from other disciplines. Interviews with 15 Fine Art examiners from 11 Australian institutions provided information about the expectations, standards and models of evaluation in the Fine Art field as well as how examiners perceive their role and what constraints are evident in current examination processes.     

The concept of ‘originality’ in the Ph.D.: how is it interpreted by examiners?

This paper explores ways in which examiners, supervisors and others interpret the concept of ‘originality’ when evaluating candidates’ achievements in the final Ph.D. examination. It is based on institutional responses to a question in a 2006 discussion paper on doctoral assessment about how universities define originality for the purpose of Ph.D. assessment and what other criteria are used to guide examiners in making judgements in the final examination. The paper also includes emerging findings from the first stages of data generation for a Ph.D. study about how examiners assess the Ph.D. This study involves viva observations and interviews with candidates, examiners, supervisors and independent chairs of vivas. Most participants, irrespective of role, discipline or field, confirmed that ‘originality’ or ‘a contribution to knowledge’ is required for a candidate to pass the Ph.D. examination, and that this criterion is interpreted in a subject-specific context by examiners. The studies suggest that the discipline or field in which the candidate’s topic is situated influences the examiners’ interpretation of ‘originality’ and their expectations of the candidate. However, ‘originality’ is not the only criterion for the Ph.D., and students are also required to demonstrate other achievements to pass the doctorate. The studies show that there is some degree of consistency among examiners in the general criteria used to evaluate the candidate’s thesis and performance in the viva. These findings are discussed in the light of the literature.     

An analysis of Ph.D. examiners’ reports in engineering

In recent years, there have been increasing calls for an overall transformation of the nature of engineering Ph.D. programs and the way theses are assessed. There exists a need to understand the examination process to ensure the best quality outcome for candidates in engineering. The work we present in this paper uses data collected between 2003 and 2010 for a total of 1220 Australian Ph.D. theses by analysing examiner reports. Our analysis indicates that Ph.D. theses in engineering, N?=?106, differ considerably from those in other fields in areas such as gender of candidates and examiners and the examiners’ geographical location. We also found that assessment areas such as significance and contribution of the thesis, publications arising from the thesis, breadth, depth and recency of the literature review and communication and editorial correctness are areas in which the proportion of text of engineering examiners' comments differs significantly from other fields.     

Moths,candles and fires: examining dance as creative practice research in a Master's degree

This article reflects on the purposes and functions of examination criteria for Masters' research degrees with creative practice components. With a particular focus on dance projects that incorporate both a performed and a written expression of the research, we consider how a rubric for creative practice research degrees might address formative and summative assessment purposes and clarify goals for learners, supervisors, examiners and academic institutions. Our discussion is situated within a South Pacific postgraduate learning context and is informed by post-colonial and post-Cartesian concerns over how dance knowledge is recognized in global academia. Much current literature has argued how particular research methods and diverse approaches to the presentation of research outputs can destabilize dominant, logocentric methods of valuing knowledge. Our queries extend this argument into the actual quantifiable measuring and evaluation of such knowledge, which Masters' grading necessitates in order to maintain its value as an institutional currency. This leads to the reflections on the rubric for Masters' degrees with Creative Practice components that is currently used within the Dance Studies programme of the University of Auckland, and how this rubric might be seen as contributing to an evolving ‘cultivated’ community of practice within postgraduate examination. The rubric continues to grow through ongoing consultation with postgraduate students, supervisory staff, internal and external examiners and international experts in creative arts academia. Our reflections extend the argument for how creative practice might be further rationalized within academia.     

把好学位论文关 提高研究生培养质量

本文对工科院研究生学位论文存在的问题进行分析，提出了要打破学科壁垒，加强研究生导师队伍的建设，建立和完善学位和研究生教育质量的保证体系，培养高层次，创新型建设人才，为我国科技论文的产出和科技实力的再发展培养后备力量。     

对我院体育专业学生毕业论文质量分析与对策研究

本文以绵阳师范学院体育教育专业学生毕业论文为资料,采用文献资料、数理统计等方法,对学生在撰写毕业论文中存在的问题进行分析研究,明确了我院学生在撰写毕业论文时整体质量水平逐渐下滑趋势,部分毕业论文存在拼凑、抄袭、造假现象,涉及到的问题将直接影响毕业论文质量高低,也对毕业论文规范性、严肃性、公平性提出了挑战.因此;毕业论文不仅是检验学生在校所掌握知识结构与解决实际问题能力的考核,同时关系到我院办学质量水平高低,为今后更好指导毕业论文工作顺利完成提供参考.     

Examination involving students as peer examiners

The main interest in this article is students’ involvement in assessment as a part of growth towards self‐directedness in learning. In order to enhance students’ development of autonomy in learning, a project involving ‘older’ students as peer examiners for ‘younger’ students was designed and carried out. Students in the sixth semester in a PBL‐based Master’s program of Medical Biology participated, together with faculty, as examiners of fifth‐semester students. The examination and the assessment situation was carefully designed based on learning theories, empirical evidence and experiences underpinning student‐centred learning, especially in the form of PBL used at the faculty. The project was evaluated and analysed in order to understand students’ learning processes related to the responsibility for assessing peers. The situation of the peer examiners was interpreted based on their own experiences with statements from the students assessed and faculty involved in the assessment. Evaluations from six occasions, spring and fall, 2003–2005, were included in the study. The findings suggest that involving students in assessment as equal partners with faculty makes it is possible for students to apprehend the metacognitive competences needed to be responsible and autonomous in learning. The peer examiners experience motivation to learn about learning, they acquire tacit knowledge about assessment and they learn through being involved and trusted. The student‐centred educational context, which requires responsibility throughout the programme, is recognized as very important.     

A Masters Degree Program: Structural Components and Examiners' Comments

This study examined data relating to 125 part‐time masters candidates in education. In particular, the study considered various structural aspects of candidature and examiners’ concerns regarding the theses submitted by these candidates. Some of the structural components considered were the choice of thesis weighting in the degree, completion time, suspension and extension of candidature, course work grades, agreement between examiners and the level of honours awarded. An analysis of thesis examiners’ reports revealed a diversity in the nature of criticism made. Criticism related to such things as the review of literature, writing style, typographical and spelling errors, conclusions reached and the theoretical framework used. In many instances the nature of examiners’ criticism raised questions as to the role that thesis supervisors are expected to perform.     

基于学术型硕士拔尖创新人才科研能力的培养

高校学术型硕士的培养侧重于理论和学术研究,其“拔尖创新人才”是学术型人才,更要突出科研能力的培养。本文针对学术型硕士培养中存在的“有研究无能力”、“理论不结合实际”、“学位论文没有应用价值”等问题,提出了相应对策：学位论文开题实行课题导向制;学位论文撰写实行“小论文”构成制;学位论文评审实行“双向”盲审制,同时,设立学术研究分,推动研究生全方位参与科技创新。     

Learn by doing: How to include new requirements of research in engineering education

Co-operation between university and industry for technology transfer is not only useful for industry, but also for students. In the Fraunhofer Institute of the author, about 40 projects annually for individual companies as well as co-operative research projects with many partners for international programmes such as EUREKA, ESPRIT and LEONARDO are carried out. Students are involved in many ways, as part-time assistants or during their intermediate and diploma thesis projects as well as PhD theses, and contribute significantly to the research and development. By being members of a team of professionals they learn about their later work in industry to the extent that they get ideas for planning their careers and even consider becoming entrepreneurs themselves.     

英语毕业论文摘要的体裁分析对教学的启示

英文摘要是英语专业毕业论文写作的一个薄弱环节。该文通过运用体裁分析法,对本科生毕业论文英文摘要的图式结构和语言体现两个方面进行研究。研究发现,英语专业本科论文的摘要写作存在结构过于简单、语言表达不符合英语论文写作规范等问题。笔者呼吁改革论文写作课,把体裁教学法引入到写作课中;在基础阶段的教学中,要以语篇为中心来组织教学,全面提高学生的综合语言运用能力。