Assessing metacognitive activities: the in-depth comparison of a task-specific questionnaire with think-aloud protocols

Teaching and assessing metacognitive activities are important educational objectives, and teachers are calling for efficient instruments. The advantages of questionnaires in measuring metacognitive activities are obvious, but serious validity issues appear. For example, correlations of questionnaire data with think-aloud measures are generally moderate to low. An explanation may be that these questionnaires are not constructed in line with the metacognitive activities measured by the think-aloud method. In the present study, a questionnaire is constructed based directly on a taxonomy for coding think-aloud protocols. Twenty ninth-graders studied a text while thinking aloud, after which they immediately received the questionnaire. The overall correlation between the questionnaire and the think-aloud protocols (r?=?0.63) was promising. However, scale and item analyses clearly demonstrate some new validity issues. Comparing the questionnaire and the think-aloud results, the students seem to report overt metacognitive activities corresponding more with their behavior reported in the protocols than covert ones. In-depth explorations are presented.     

Exploring the potential impact of reciprocal peer tutoring on higher education students’ metacognitive knowledge and regulation

It is widely recognized that metacognition is an important mediator for successful and high-level learning, especially in higher education. Nevertheless, a majority of higher education students possess insufficient metacognitive knowledge and regulation skills to self-regulate their learning adequately. This study explores the potential of reciprocal peer tutoring to promote both university students’ metacognitive knowledge and their metacognitive regulation skills. The study was conducted in a naturalistic higher education setting, involving 67 students tutoring each other during a complete semester. A multi-method pretest–posttest design was used combining a self-report questionnaire, assessing students’ metacognitive knowledge and their perceived metacognitive skilfulness, with the analysis of think-aloud protocols, revealing students’ actual use of metacognitive strategies. Results indicate no significant pretest to posttest differences in students’ metacognitive knowledge, nor in their perception of metacognitive skill use. In contrast, significant changes are observed in students’ actual metacognitive regulation. At posttest, students demonstrate significantly more frequent and more varied use of metacognitive regulation, especially during the orientation, monitoring, and evaluation phases. Furthermore, our findings point to an increase in more profound and higher-quality strategy use at posttest.     

Consistent results with the consistency hypothesis? The effects of epistemic beliefs on metacognitive processing

We explored relations between students’ epistemic beliefs, metacognitive monitoring and recall performance in the context of learning physics through metaphor. Eighty-three university undergraduate students completed questionnaires designed to measure their epistemic beliefs and prior knowledge about Newtonian physics. Students were epistemically profiled as rational, empirical, or metaphorical in their approaches to knowing. Using a think-aloud protocol, students read a text on Newton’s First and Third Laws. The text included metaphors as examples of the various laws described. Results revealed that students profiled as metaphorical engaged in more metacognitive processing compared to students profiled as rational or empirical. Moreover, path analyses revealed that metacognitive monitoring positively predicted recall performance. Results challenge Muis’ (2008) consistency hypothesis; the ways in which knowledge is represented in text may be the linking factor for relations between metacognitive monitoring and epistemic beliefs rather than the underlying epistemology of the domain.     

Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia?

We examined how self-regulated learning (SRL) and externally-facilitated self-regulated learning (ERL) differentially affected adolescents’ learning about the circulatory system while using hypermedia. A total of 128 middle-school and high school students with little prior knowledge of the topic were randomly assigned to either the SRL or ERL condition. Learners in the SRL condition regulated their own learning, while learners in the ERL condition had access to a human tutor who facilitated their self-regulated learning. We converged product (pretest-posttest shifts in students’ mental models and declarative knowledge measures) with process (think-aloud protocols) data to examine the effectiveness of self- versus externally-facilitated regulated learning. Findings revealed that learners in the ERL condition gained statistically significantly more declarative knowledge and that a greater number of participants in this condition displayed a more advanced mental model on the posttest. Verbal protocol data indicated that learners in the ERL condition regulated their learning by activating prior knowledge, engaging in several monitoring activities, deploying several effective strategies, and engaging in adaptive help-seeking. By contrast, learners in the SRL condition used ineffective strategies and engaged in fewer monitoring activities. Based on these findings, we present design principles for adaptive hypermedia learning environments, engineered to foster students’ self-regulated learning about complex and challenging science topics.

Signaling task awareness in think-aloud protocols from students selecting relevant information from text

Self-regulated learning has been described as an adaptive process: students adapt their learning strategies for attaining different learning goals. In order to be adaptive, students must have a clear notion of what the task requirements consist of. Both trace data and questionnaire data indicate that students adapt study strategies in limited ways and that their awareness of task demands seems to be low. In the present think-aloud study we examined task awareness of tenth-graders who selected text fragments in three different selection tasks. Students’ task awareness was analysed at a global and a local task level. Awareness at the global task level refers to processing the instruction and reflecting about general selection goals; awareness at the local task level includes spontaneous reasoning about selecting specific text fragments. On either level, students showed difficulties in verbalizing task demands. In line with previous work, tenth-graders apparently experience a limited awareness of task demands.     

Adaptive Human Scaffolding Facilitates Adolescents’ Self-regulated Learning with Hypermedia

This study examines the effectiveness of three scaffolding conditions on adolescents’ learning about the circulatory system with a hypermedia learning environment. One hundred and eleven adolescents (n = 111) were randomly assigned to one of three scaffolding conditions (adaptive scaffolding (AS), fixed scaffolding (FS), or no scaffolding (NS)) and were trained to use a hypermedia environment to learn about the circulatory system. Pretest and posttest data were collected to measure the qualitative changes in students’ mental models of the topic and quantitative changes in their declarative knowledge. Verbal protocols were collected during the 40-minute learning task to examine how each condition affected the way in which students regulated their learning. Findings revealed that learners in both the AS and NS conditions gained significantly more declarative knowledge than did those in the FS condition. Also, the AS condition was associated with shift in learners’ mental models significantly more than the other conditions. Associated with these significant shifts in their mental models, learners in the AS condition regulated their learning by planning and activating prior knowledge, monitoring their cognitive activities and their progress toward learning goals, using several effective strategies, and engaging in adaptive help-seeking. By contrast, those in the NS condition used fewer effective strategies, while those in the FS regulated their learning by using several regulatory processes which seemed to impede their learning. Implications for the design of scaffolds for fostering students’ self-regulated learning with hypermedia are presented.     

Reading characteristics of Chinese-English adolescents: knowledge and application of strategic reading

Integrating both quantitative and qualitative approaches, this study examined the knowledge and application of strategic reading among Taiwanese adolescents learning to read English as a foreign language. A total of 144 Grade 11 students participated in this study. Correlational analysis and t-tests were applied to analyzing quantitative data, including an English reading comprehension test and two sets of reading strategy questionnaires; the constant comparative method (Glaser and Strauss 1967) was used for analyzing qualitative data, comprising semi-structured interviews and think-aloud protocols. Results from quantitative analysis show a relatively strong link between the perceived use of first language (L1, Chinese) and second language (L2, English) strategies. Nonetheless, metacognitive and cognitive strategies were reportedly used more frequently for L1 reading, whereas support strategies were more often used for L2 reading. One salient theme that emerged from the qualitative analysis is that the reading of the English texts more closely resembled the reading of the texts written in the classic literary style of Chinese than the reading of the texts written in contemporary Chinese. These findings to some extent support the reading universals hypothesis (Goodman in Journal of Typographic Reseach, 103–110, 1970); yet, skills transfer is not uniformly automatic (Bell in TESOL Quarterly, 687–704, 1995; Urquhart and Weir 1998).     

Towards efficient measurement of metacognition in mathematical problem solving

Metacognitive monitoring and regulation play an essential role in mathematical problem solving. Therefore, it is important for researchers and practitioners to assess students?? metacognition. One proven valid, but time consuming, method to assess metacognition is by using think-aloud protocols. Although valuable, practical drawbacks of this method necessitate a search for more convenient measurement instruments. Less valid methods that are easy to use are self-report questionnaires on metacognitive activities. In an empirical study in grade five (n?=?39), the accuracy of students?? performance judgments and problem visualizations are combined into a new instrument for the assessment of metacognition in word problem solving. The instrument was administered to groups of students. The predictive validity of this instrument in problem solving is compared to a well-known think-aloud measure and a self-report questionnaire. The results first indicate that the questionnaire has no relationship with word problem solving performance, nor the other two instruments. Further analyses show that the new instrument does overlap with the think-aloud measure and both predict problem solving. But, both instruments also have their own unique contribution to predicting word problem solving. The results are discussed and recommendations are made to further complete the practical measurement instrument.     

Characteristics of Workplace Learning Among Finnish Vocational Students

In Finnish VET, students’ work experience is explicitly defined as workplace learning, instead of the practice of already learnt skills. Therefore, vocational students’ learning periods in the workplace are goal-oriented, guided and assessed. This paper examines the characteristics of students’ workplace learning and compares them with the characteristics of employees’ workplace learning. The data were collected with an Internet questionnaire from final-year vocational students (N = 3106). In total, 1603 students (52 %) answered the questionnaire. The data were analysed using quantitative methods. The results indicate that features typical of employees’ workplace learning can also be found in student learning as well. However, VET-related workplace learning has a number of characteristics that have not been brought to light in research on employees’ workplace learning thus far. We suggest that in developing educational practices it would be useful to draw on some of the features of workplace learning such as the use of collaboration and shared practices; conversely for workplace practices it would be useful to draw on some of the features of educational practices. For example, by utilising the structures of students’ workplace learning system presented in this study, learning at work could be transformed towards more goal-directed, guided and assessed activity.     

Using a Think-Aloud with Diverse Students: Three Primary Grade Students Experience Chrysanthemum

Many struggling readers, students with English as a second language, and children with disabilities do not engage in the strategies that good readers use when reading for understanding. Reading comprehension depends upon the students’ ability to successfully use strategies to monitor and control their own comprehension. Teachers need to help students develop skills that will aid in reading comprehension. The think-aloud is one strategy that can help struggling readers improve fluency and comprehension. One of the most important components to determine the success of the think-aloud is the teacher’s ability to model and facilitate the think-aloud procedure. The teacher needs to have a basic understanding of what is meant to be accomplished using this method. This article profiles how three teachers model and facilitate a think-aloud with three struggling readers.     

Exploring the uses of mobile phones to support informal learning

This paper explores how a group of undergraduate students in the United Arab Emirates (UAE) use their mobile phones to perform informal learning activities related to the content of their courses outside the classroom. The paper also addresses the usefulness of informal learning activities to support students’ learning. The study adopts an exploratory case study design and uses multiple methods of data collection including questionnaires, interviews and diary entries. Main findings suggest that students performed informal learning activities mostly from home, interacting mainly with classmates. It also shows that students were in control of their informal learning activities without teachers’ input. However, it was found that students used only a limited number of applications but these were considered useful to their learning. The paper contributes to a discussion of the implications of training and instructor support to help students to take more advantage of mobile phone applications to support informal learning. Recommendations for further research are discussed.     

Cognitive strategies for learning from static and dynamic visuals

An experimental study with 60 students investigated the effects of including static or dynamic visuals in an expository text on a learning outcome and the use of learning strategies while working with those visuals. For the study, two illustrated and one text-only version of a computer-based learning text on an astrophysical subject were developed and served as the learning material. Considering the cognitive task demand in a learning test, we found significant differences between the illustrated versions and the text-only version, but not between the two illustrated ones. We used think-aloud protocols to examine the learning processes initiated by both types of visuals. The coding of the recorded learning activities was based on recent theories of learning strategies. The results for both types of illustrations indicate different frequencies in the use of learning strategies relevant for the learning outcome, and therefore indicate the contribution of the cognitive process quality for the supportive function of visuals.     

THE EFFECT OF COOPERATIVE LEARNING STRATEGIES ON ELEMENTARY STUDENTS’ SCIENCE ACHIEVEMENT AND SOCIAL SKILLS IN KUWAIT

This study compares the effects of two methods of teaching—teacher-centered and cooperative learning—on students’ science achievement and use of social skills. The sample consists of 163 female elementary science students in 8 intact grade 5 classes who were assigned to 2 instructional methods and were taught an identical science unit by 4 classroom teachers. The students’ science achievement was measured by a researcher-designed achievement test given to students as a pretest and a posttest. Students’ social skills were determined by a researcher-designed survey administered as a pretest and posttest. Analysis of the achievement test scores and the social skills survey responses revealed that cooperative learning strategies have significantly (p > 0.05) more positive effects on both students’ achievement and social skills than teacher-centered strategies. These results provide an evidential base to inform policy decisions and encourage and persuade teachers to implement cooperative learning methods in Kuwaiti classrooms.     

Metacognitive strategies that enhance critical thinking

The need to cultivate students’ use of metacognitive strategies in critical thinking has been emphasized in the related literature. The present study aimed at examining the role of metacognitive strategies in critical thinking. Ten university students with comparable cognitive ability, thinking disposition and academic achievement but with different levels of critical thinking performance participated in the study (five in the high-performing group and five in the low-performing group). They were tested on six thinking tasks using think-aloud procedures. Results showed that good critical thinkers engaged in more metacognitive activities, especially high-level planning and high-level evaluating strategies. The importance of metacognitive knowledge as a supporting factor for effective metacognitive regulation was also revealed. The contribution of metacognitive strategies to critical thinking and implications for instructional practice are discussed.     

Open plan

This study reports on an investigation into the impact of strategy-based English language instruction (SBELI) on the language proficiency of gifted learners. The participants were students who were identified as gifted and thus received additional out-of-school training for a few days a week in Adana Science and Arts Centre. Their ages ranged from 10 to 13. One experimental group (n = 17) and one control group (n = 17) were designed to explore the possible English performance differences between the learners who were exposed to SBELI, which included 11 language-learning strategies for eight weeks, and those who were not. Data were collected through proficiency exams, written accounts of participants in the experimental group, think-aloud protocols and an open-ended questionnaire. The findings obtained revealed that SBELI produced a positive impact on the language proficiency of the gifted students and enabled them to enhance their already established attitude to make their language learning quick, easy, effective and enjoyable.     

听力高水平者与低水平者策略使用差异研究

本文通过问卷调查和有声思维的方法,调查研究大学非英语专业高水平者和低水平者听力策略使用差异。问卷调查显示高水平者在元认知策略、认知策略和社交/情感策略三方面使用频率都高于低水平者,在元认知策略和认知策略使用方面具有显著性差异,而在社交/情感策略使用方面不具有显著性差异。通过有声思维的方法研究高低水平学习者的听力过程,进一步发现高水平者比较善于使用策略,而低水平者忙于应付语音、生词等的基本困难,策略使用受到一定的限制。采用有声思维的质化方法研究听力策略在国内并不多见,本文抛砖引玉,研究结果对听力教学具有一定的参考价值。     

Teachers’ beliefs and intentions concerning teaching in higher education

A questionnaire measuring nine different aspects of teachers’ beliefs and intentions concerning teaching in higher education was distributed to teachers at four institutions in the United Kingdom, yielding 638 complete sets of responses. There was a high degree of overlap between the participants’ scores on the subscales measuring beliefs and intentions, and analyses of both sets of scores yielded two factors reflecting an orientation towards learning facilitation and an orientation towards knowledge transmission. However, teachers’ intentions were more orientated towards knowledge transmission than were their beliefs, and problem solving was associated with beliefs based on learning facilitation but with intentions based on knowledge transmission. Differences in teachers’ intentions across different disciplines and between men and women seemed to result from different conceptions of teaching, whereas differences in teachers’ intentions across different institutions and between teachers with different levels of teaching experience seemed to result from contextual factors. Teaching intentions thus reflect a compromise between teachers’ conceptions of teaching and their academic and social contexts     

Korean and foreign students’ perceptions of the teacher’s role in a multicultural online learning environment in Korea

This study investigated Korean and foreign students’ perception of the teacher’s role in multicultural online learning environments in Korea. On the basis of a literature review, a survey questionnaire was developed and piloted with 248 university students in Korea. The questionnaire was subsequently modified on the basis of the results of the pilot study, and a final 20-item questionnaire was constructed. Three hundred and eighty-four university students participated in the main study. Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were used for statistical analyses. The results indicated that students have a particular perception of the teacher’s role in a multicultural online learning environment. Five different types of teacher’s roles—pedagogical, managerial, technical, affective, and differentiating—were identified. The teacher’s affective and differentiating roles, which were not identified as crucial in previous research, were found to be significant roles for both Korean and foreign students.