Modeling the student in intelligent tutoring sytems: The promise of a new psychometrics

This paper reviews a number of relatively new and promising psychometric approaches to the problem of modeling student achievement (the student model) within intelligent tutoring systems (ITS). A shared characteristic of most ITSs is their need to estimate a model of the student's understanding of the domain, and use this model to modify and adapt subsequent instructional content and sequence. Sound cognitive diagnosis and the need to advance ITS technology require the development of student models that are integrated with cognitive theory and instructional science. A number of cognitively oriented psychometric approaches — including latent-trait models, statistical pattern recognition methods, and causal probabilistic networks — are described and discussed within the current ITS framework. As measurement-based student models are refined, we anticipate their compatibility with future generations of intelligent tutoring systems.     

Collaboration in the Assessment Process: An initial evaluation of collaboration on an undergraduate business and management course

It is often claimed that collaborative approaches to assessment improve student performance and help students to develop key transferable skills. There are, however, considerable difficulties associated with the introduction and management of such approaches. This article utilises the author's experience of introducing collaborative assessment on an undergraduate business and management course. The article focuses on the role of values, and how both student and lecturer values can have a significant influence on their attitude to such forms of assessment. The paper concludes by suggesting that we should attempt to obtain a more in-depth understanding of student and staff values, and try to address these when introducing collaborative assessment. It also suggests that an incremental approach might be adopted in order to foster commitment, help create a culture of co-operation, and facilitate an action research approach that enables those involved to learn from their experience.     

Understanding how a case-based assessment instrument influences student teachers’ learning approaches

In order to stimulate student teachers to thoroughly comprehend the main variables influencing their work, teaching and assessment strategies in teacher education have changed significantly. One of the changes in the assessment of student teachers in teacher education programs is the use of case-based assessment instruments. Such instruments evaluate the extent to which student teachers are able to handle authentic problems. It is expected that implementing this mode of assessment will stimulate student teachers to adopt a deep approach to learning. In this study we examine student teachers’ learning approaches in the context of case-based assessment. Hereto, we investigated the direct effects of the student teachers’ general beliefs on the cognitive demands of assessment on their learning approaches. Also the student teachers’ perceptions of the cognitive demands of the case-based assessment instrument were considered as a mediating variable. The present study is conducted in the Netherlands, in the third year of a 4-year initial teacher training program for primary education. A case-based assessment instrument, called the OverAll Test, was implemented as the major determinate of the third-year final examination grade of the student teachers.     

A comparison of three major instruments used for the assessment of university student experience: Toward a comprehensive and distributed approach

Higher education's rapid expansion is paired with growing social expectations of its benefits and concern on its teaching quality. In response to these, institutional/national surveys based on an array of theories are widely used in universities for quality assurance, enhancement, and benchmarking. This paper reviews three major types of instruments used for such purposes, including two distinct schools of theory that have guided the development of such assessment in the United States, Australia and the United Kingdom and then spread to the other parts of the world. The theories shaping the development of the two instruments, the dimensions assessed, and the challenges and criticisms involved when using such instruments for quality assurance are each discussed. This review concludes with a call for comparisons of different lines of research in this area, discussions on student learning experience that include more diverse characterizations of student experience across different educational contexts, development of tools to enable distributed leadership among teachers, and encouragement of students as partners for quality enhancement in higher education.     

Measuring the Quality of Teaching in Elementary School Classrooms

This paper reviews the means by which teacher quality has been measured. It considers data sources such as students, peers, experts, and examines the psychometrics and scale properties of teacher quality assessment instruments with respect to reliability and validity. A list of items for possible inclusion in an elementary student focussed instrument is considered, together with the potential use of such an instrument in measuring teacher quality.     

Context-based assessment: creating opportunities for resonance between classroom fields and societal fields

There is on-going international interest in the relationships between assessment instruments, students’ understanding of science concepts and context-based curriculum approaches. This study extends earlier research showing that students can develop connections between contexts and concepts – called fluid transitions – when studying context-based courses. We provide an in-depth investigation of one student’s experiences with multiple contextual assessment instruments that were associated with a context-based course. We analyzed the student’s responses to context-based assessment instruments to determine the extent to which contextual tests, reports of field investigations, and extended experimental investigations afforded her opportunities to make connections between contexts and concepts. A system of categorizing student responses was developed that can inform other educators when analyzing student responses to contextual assessment. We also refine the theoretical construct of fluid transitions that informed the study initially. Implications for curriculum and assessment design are provided in light of the findings.     

Reflective Classroom Practice: case studies of student teachers at work

Abstract

This paper addresses the nature of reflective classroom practice in a setting where action research has been undertaken by both the student teachers and the teaching practice supervisor. It is based on a cross‐case study of the processes through which student teachers learn to teach. Specifically, the analysis focuses on how student teachers reflect on their experiences in learning to teach. The data are based on student teachers’ reported thoughts about their learning over a period of one year. The results contribute to the understanding of reflective classroom practice by highlighting first student teachers’ perceptions about learning to teach and second their reviews on classroom practice. The discussion also adds to the literature on teacher development taken from the novice‐expert research tradition. Accordingly, implications for curriculum development in teacher education are drawn.     

Assessment innovation and student experience: a new assessment challenge and call for a multi-perspective approach to assessment research

The impact of innovative assessment on student experience in higher education is a neglected research topic. This represents an important gap in the literature-given debate around the marketisation of higher education, international focus on student satisfaction measurement tools and political calls to put students at the heart of higher education in the UK. This paper reports on qualitative findings from a research project examining the impact of assessment preferences and familiarity on student attainment and experience. It argues that innovation is defined by the student, shaped by diverse assessment experiences and preferences, and therefore its impact is difficult to predict. It proposes that future innovations must explore assessment choice mechanisms which allow students to shape their own assessments. Cultural change and staff development will be required to achieve this. To be accepted, assessment for student experience must be viewed as a complementary layer within a complex multi-perspective model of assessment, which also embraces assessment of learning, assessment for learning and assessment for lifelong learning. Further research is required to build a meta-theory of assessment to enhance the synergies between these alternative approaches and minimise the tensions between them.     

The impact of sub-skills and item content on students’ skills with regard to the control-of-variables strategy

The so-called control-of-variables strategy (CVS) incorporates the important scientific reasoning skills of designing controlled experiments and interpreting experimental outcomes. As CVS is a prominent component of science standards appropriate assessment instruments are required to measure these scientific reasoning skills and to evaluate the impact of instruction on CVS development. A detailed review of existing CVS instruments suggests that they utilize different, and only a few of the four, critical CVS sub-skills in the item development. This study presents a new CVS assessment instrument (CVS Inventory, CVSI) and investigates the validity of student measures derived from this instrument utilizing Rasch analyses. The results indicate that the CVSI produces reliable and valid student measures with regard to CVS. Furthermore, the results show that the item difficulty depends on the CVS sub-skills utilized in item development, but not on the item content. Accordingly, previous instruments that are restricted to a few CVS sub-skills tend to over- or underestimate students’ CVS skills. In addition, these results indicate that students are able to use CVS as a domain general strategy in multiple content areas. Consequences for science instruction and assessment are discussed.     

本土化学生学习调查工具的开发初探

当前我国高校内部评价体系中普遍缺少直接针对学生学习的评估。为此,国内院校有必要构建本土化的学生学习调查。学生学习调查作为院校研究范畴内学生学习评估最常用的范式,无论在理论上还是实践上都已经十分成熟。本研究通过借鉴美国的相关经验,从院校研究的角度出发,利用社会科学的研究方法,构建出符合自身发展需要的学生学习调查问卷,为调查后续的构建工作打下良好的基础。     

Identifying pathways of teachers’ PCK development

This paper describes a method of analysing teacher growth in the context of science education. It focuses on the identification of pathways in the development of secondary school teachers’ pedagogical content knowledge (PCK) by the use of the interconnected model of teachers’ professional growth (IMTPG).The teachers (n = 12) participated in a one-year action research project focused on their individual concerns related to teaching science. The use of the IMTPG revealed that teachers use different pathways of learning to develop different aspects of their PCK. For each PCK component, three distinct pathways could be identified, two of which clearly were associated with professional growth. When examining these two pathways in detail, it was found that (1) teachers learned about new instructional strategies and assessment methods mostly through literature reviews and discussions with peers and (2) teachers who analyzed and reflected on student learning as it happened in their classrooms developed understandings that helped them to select and apply instructional strategies to further promote student learning. Both the analytical method as well as the identification of the different developmental pathways help to better understand teacher development in the context of classroom practices.     

On the alignment of teachers’ mathematical content knowledge assessments with the common core state standards

Instruments designed to measure teachers’ knowledge for teaching mathematics have been widely used to evaluate the impact of professional development and to investigate the role of teachers’ knowledge in teaching and student learning. These instruments assess a mixture of content knowledge and pedagogical content knowledge. However, little attention has been given to the content alignment between such instruments and curricular standards, particularly in regard to how content knowledge and pedagogical content knowledge items are distributed across mathematical topics. This article provides content maps for two widely used teacher assessment instruments in the USA relative to the widely adopted Common Core State Standards. This common reference enables comparisons of content alignment both between the instruments and between parallel forms within each instrument. The findings indicate that only a small number of items on both instruments are designed to capture teachers’ pedagogical content knowledge and that the majority of these items are focused on curricular topics in the later grades rather than in the early grades. Furthermore, some forms designed for use as pre- and post-assessment of professional development or teacher education are not parallel in terms of curricular topics, so estimates of teachers’ knowledge growth based on these forms may not mean what users assume. The implications of these findings for teacher educators and researchers who use teacher knowledge instruments are discussed.

     

Challenges and Opportunities in the Applied Assessment of Student Social and Emotional Learning

Interest in school-based strategies to support student social and emotional learning (SEL) is strong. Although SEL policies and programs designed to support the development of student competencies have advanced significantly, less work has been done to develop methods of assessing student social and emotional competence. This article briefly reviews developments in the field of social and emotional competence assessment and examines challenges and opportunities in their applied use, including (a) balancing the priorities of assessment developers and educators; (b) ensuring that the inferences and decisions made from SEL assessment scores are supported by evidence of the assessment’s psychometric merit; (c) establishing conditions for SEL assessment and data use that maximize benefit while mitigating risks; (d) coordinating standards, assessment, programs, and professional learning; and (e) balancing highly focused assessments that by design do not vary in content or format, and the varied cultural contexts in which they may be used.     

Assessment of Integration of Technology in Education: Countering the “No Significant Differences” Argument

Literature in the field of information technology in educations in full of references to “no significant differences” findings from studies on the efficacy of technology integration. These findings may be due factors such as flawed assessment procedures, widely varying definitions of technology integration, and research design problems. The current special issue focuses on the development and use of reliable and valid instruments and procedures to access technology integration. This editorial reviews the concepts of reliability and validity, and summarizes assessment methods presented in the articles in this special issue.     

Time on task in intensive modes of delivery

This paper reports on an investigation into how staff teaching in compressed courses can encourage student engagement and enhance student use of learning time, despite significant restraints of time as well as distance. Typically these courses (described here as units) are expected to have comparable learning outcomes to their full-semester counterparts and provide an opportunity for students to either retake failed units or to acquire credit for their chosen degrees in accelerated time. Organising teaching and learning through intensive modes of delivery may require different approaches to curriculum development and pedagogy compared to traditional unit planning and delivery, especially when the intensive delivery utilises online technologies. This paper explores strategies employed by successful intensive mode teachers in the development and delivery units for maximised student engagement. It concludes that many of these strategies are equally applicable in online and distance education regardless of the unit being intensive or otherwise.     

Is the enhancement of student experience a strategic priority in Australian universities?

Universities in many countries are developing strategies to enhance the student experience. This focus has never been so important since the development of rankings and the use of student experience measures in institutional performance assessment. Australian government policies to link student experience measures to performance funding were a key driver to increase the prominence of the student voice between 2004 and 2008. This paper analyses the strategic plans of 33 Australian universities with the aim of outlining the extent to which the enhancement of student experience is embedded as one of their key priorities. This review comes at a time when universities in Australia and elsewhere are realigning their future strategies and directions to achieve government aspirations for tertiary education with the introduction of various policy instruments including the possibility of rewarding universities on the basis of student experience measures. The paper also provides a comparative analysis of government policies in Australia and the UK on the measurement and enhancement of the student experience.     

Integrating data-based decision making,Assessment for Learning and diagnostic testing in formative assessment

Recent research has highlighted the lack of a uniform definition of formative assessment, although its effectiveness is widely acknowledged. This paper addresses the theoretical differences and similarities amongst three approaches to formative assessment that are currently most frequently discussed in educational research literature: data-based decision making (DBDM), Assessment for Learning (AfL) and diagnostic testing (DT). Furthermore, the differences and similarities in the implementation of each approach were explored. This analysis shows that although differences exist amongst the theoretical underpinnings of DBDM, AfL and DT, the combination of these approaches can create more informed learning environments. The thoughtful integration of the three assessment approaches should lead to more valid formative decisions, if a range of evidence about student learning is used to continuously optimise student learning.     

The Quality of Teaching and Learning in Higher Education: Evaluating the Evidence

The issue of the quality of teaching in higher education is in the process of becoming a matter of greater public interest than hitherto. This paper concerns the evidence on which quality can be assessed. Drawing on the relevant literature from both the secondary and tertiary sectors of education, it reviews the types of evidence which are available. A critical analysis is offered of the extent to which each isolates teaching quality from distinct variables such as student characteristics and assessment standards. It is argued that quantitative approaches do not currently offer potential to assess teaching quality and are unlikely to do so given the current structure of higher education. The greatest promise lies in the external scrutiny recently instituted by the Higher Education Funding Councils and the Higher Education Quality Council. It is recommended that this scrutiny would be further strengthened by integrating the functions of these two authorities and by phasing out the use of numerical assessment scales in favour of wholly qualitative reporting.     

Towards a sociocognitive model of progression in spoken English

Current approaches to oral assessment of English in English secondary schools tend to concentrate more on ‘confidence’ and ‘participation’ than on the quality of children's thinking. This undermines the rich possibilities in classroom talk for cognitive development. Behavioural assessment approaches deny the essentially cognitive character of spoken English. This paper compares two brief representative extracts from a larger data‐set of students engaged in small group debate on the subject of abortion. One group of students are hindered in their discussion by an inability to conceptualize abstractly. It is suggested that a formative assessment approach based on a model of cognitive progression such as Vygotsky's could enable all students to develop as speakers by encouraging their teachers to focus more explicitly on the development of the quality of thinking. A sociocognitive framework for conceptual progression could guide teachers in their interpretation of peer debate in order to develop the quality of students' understanding and argument. Building on the work of Newman, Griffin and Cole, and Torrance and Pryor, further research should be conducted into the ways in which formative teacher–student and student–student assessment dialogue might enhance students' ability to think through talk.