Observing response processes with eye tracking in international large-scale assessments: evidence from the OECD PIAAC assessment

This paper reports on a pilot study that used eye tracking techniques to make detailed observations of item response processes in the OECD Programme for the International Assessment of Adult Competencies (PIAAC). The lab-based study also recorded physiological responses using measures of pupil diameter and electrodermal activity. The study tested 14 adult respondents as they individually completed the PIAAC computer-based assessment. The eye tracking observations help to fill an ‘explanatory gap’ by providing data on variation in item response processes that are not captured by other sources of process data such as think aloud protocols or computer-generated log files. The data on fixations and saccades provided detailed information on test item response strategies, enabling profiling of respondent engagement and response processes associated with successful performance. Much of that activity does not include the use of the keyboard and mouse, and involves ‘off-screen’ use of pen and paper (and calculator) that are not captured by assessment log-files. In conclusion, this paper points toward an important application of eye tracking in large-scale assessments. This includes insights into response processes in new domains such as adaptive problem-solving that aim to identify individuals’ ability to select and combine resources from the digital and physical environment.     

Conditions affecting the usefulness of pre- and post-tests for assessment purposes

Interest in measuring and evaluating student learning in higher education is growing. There are many tools available to assess student learning. However, the use of such tools may be more or less appropriate under various conditions. This study provides some evidence related to the appropriate use of pre/post‐tests. The question of whether graded tests elicit a higher level of performance (better representation of actual learning gains) than ungraded post‐tests is examined. We examine whether the difficulty level of the questions asked (knowledge/comprehension vs. analysis/application) affects this difference. We test whether the student’s level in the degree programme affects this difference. Results indicate that post‐tests may not demonstrate the full level of student mastery of learning objectives and that both the difficulty level of the questions asked and the level of students in their degree programme affect the difference between graded and ungraded assessments. Some of these differences may be due to causes other than grades on the assessments. Students may have benefited from the post‐test, as a review of the material, or from additional studying between the post‐test and the final examination. Results also indicate that pre‐tests can be useful in identifying appropriate changes in course materials over time.     

The Controversy Over Controversies: A Plea for Flexibility and for “Soft‐Directive” Teaching

A controversy rages over the question of how should controversial topics be taught. Recent work has advanced the “epistemic criterion” as the resolution to this controversy. According to the epistemic criterion, a matter should be taught as controversial when contrary views can be entertained on the matter without the views being contrary to reason. When an issue is noncontroversial, according to the epistemic criterion, the correct position can be taught “directively,” with the teacher endorsing that position. When there is a legitimate controversy, the view should be taught “nondirectively,” with the teacher remaining neutral. In response, Bryan Warnick and Spencer Smith argue that the proponents of the epistemic criterion fail to recognize the multidimensional nature of what it means to learn to be rational. Recognizing this complexity undermines the link between the epistemic status of the controversy and directiveness of one's teaching, suggesting more flexibility in how teachers approach controversial issues. It also implies the need for a new category of teaching, which Warnick and Smith call “soft‐directive” teaching.     

Addressing the Challenges and Needs of English-Speaking Caribbean Immigrant Students: Guidelines for School Counselors

Caribbean students are among the distinct immigrant groups in U.S. public schools with particular needs to be addressed by school counselors. This article discusses the challenges Caribbean immigrant students face that create obstacles to their academic and personal/social success. Guidelines for school counselors are outlined, which can be used to meet the needs of Caribbean immigrant students and promote their healthy adjustment.     

Social moderation,assessment and assuring standards for accounting graduates

Evidencing student achievement of standards is a growing imperative worldwide. Key stakeholders (including current and prospective students, government, regulators and employers) want confidence that threshold learning standards in an accounting degree have been assured. Australia’s new higher education regulatory environment requires that student achievements are benchmarked against intended programme learning outcomes, guided by published disciplinary standards and a national qualifications framework, and against other higher education providers. Here, we report on a process involving academics from 10 universities, aided by professional practitioners, to establish and equip assessors to reliably assure threshold learning standards in accounting that are nationally comparable. Importantly, we are learning more about how standards are interpreted. Based on the premise that meaning is constructed from tacit experiences, social interactions and intentional reflection on explicit information, we report outcomes of three multi-part calibration interventions, situated around judgements of the quality of the written communication skills exhibited in student work and their related assessment tasks. Qualitative data from 30 participants in the calibration process suggest that they perceive that the process both assists them both in developing a shared understanding of the accounting threshold learning standards and in the redesign of assessment tasks to more validly assess the threshold learning standards.     

Turning Silence into Speech and Action: Prison Activism and the Pedagogy of Empowered Citizenship

Based on almost 50 years of combined experience as prison activists and prison teachers, the authors offer three case studies of prison activism and pedagogy in action. The first case study, by Hartnett, details the “artistry of agency” as enacted in poetry workshops in prison and in public poetry events, thus illustrating artistic communication. The second, by Wood, examines how friendship becomes political in the epistolary communication between free and imprisoned correspondents, thus addressing interpersonal communication. The third, by McCann, addresses web-based communication as a tool for advocacy for condemned prisoner/activists on Texas’s death row, and hence political communication. Taken as a whole, the three case studies celebrate different communication strategies as avenues of enlightenment and empowerment while offering powerful arguments for abolishing the prison–industrial complex.