Memory characteristics and modality in multimedia learning: An aptitude–treatment–interaction study

According to the modality effect in multimedia, a text accompanying a picture should be auditorily presented instead of visually in order to avoid split of attention. In two experimental studies (34 and 78 participants, respectively), the impact and possible compensatory effects of two aptitude variables, that is, memory strategy skills and working memory capacity, on multimedia learning were tested. Aptitude–treatment–interaction effects were found with respect to comprehension (Study 1) and transfer (Study 2). The modality effect was confirmed for less-skilled learners in memory strategy use but not for highly skilled ones. Memory strategy skills and working memory capacity differentially affected multimedia learning, depending on task features and demands.     

A closer look at split visual attention in system- and self-paced instruction in multimedia learning

Two experiments examined visual attention distribution in learning from text and pictures. Participants watched a 16-step multimedia instruction on the formation of lightning. In Experiment 1 (N = 90) the instruction was system-paced (fast, medium, slow pace), while it was self-paced in Experiment 2 (N = 31). In both experiments the text modality was varied (written, spoken). During learning, the participants’ eye movements were recorded. Results from both experiments revealed that learners spent more time studying the visualizations with spoken text than those with written text. In written text conditions learners consistently started reading before alternating between text and visualization; moreover, they spent more time reading the text than inspecting the visualizations. While in Experiment 1 additional time that was made available in conditions with a slow or medium instruction pace was spent inspecting visualizations, in Experiment 2 longer learning times resulted from reading the text more intensively. With respect to learning outcomes (retention, transfer, and visual memory) Experiment 1 revealed an effect of text modality for visual memory only. In Experiment 2 no modality effects were found. Instruction pace was hardly related to learning outcomes. Overall, the results confirm prior findings suggesting that the distribution of visual attention in multimedia learning is largely guided by the text.     

Explaining the modality effect in multimedia learning: Is it due to a lack of temporal contiguity with written text and pictures?

The study examined whether the modality effect is caused by either high visuo-spatial load or a lack of temporal contiguity when processing written text and pictures. Students (N = 147) viewed pictures on the development of tornados, which were accompanied by either spoken or written explanations presented simultaneously with, before, or after the pictures. For verbal recall no modality effect was observed, whereas for transfer the influence of modality varied as a function of phonological working memory capacity. For pictorial recall the results showed a modality effect that was limited to simultaneous presentation, apparently in line with the temporal contiguity explanation. However, spoken simultaneous presentations were not superior to spoken sequential presentations, which contradicts the temporal contiguity explanation for the modality effect. Rather, it seems as if learners with simultaneous presentation of written text and picture concentrated more on the text and ignored the pictures, resulting in worse pictorial recall.     

Statement-Pie: a strategy to improve the paragraph writing skills of adolescents with learning disabilities

This study investigated the effects of teaching a metacognitive text structure strategy upon the paragraph writing skills of eighth-grade students with learning disabilities. The technique, called Statement-Pie (Hanau, 1974), teaches students to understand the relationship of supporting details to a main idea. Four students were taught to use an outline as a paragraph planning guide, which they then used to convert information into written expository paragraphs. All subjects reached instructional outcome criterion on the writing of comparison/contrast and sequence paragraphs. One week after reaching mastery in the special education classroom, students generalized their paragraph writing skills to other teachers and to different classrooms. The results of this investigation indicate that when provided with direct, intensive instruction in a text structure strategy, adolescents with learning disabilities can improve their skills in paragraph writing.     

Task Experience as a Boundary Condition for the Negative Effects of Irrelevant Information on Learning

Research on multimedia learning has shown that learning is hampered when a multimedia message includes extraneous information that is not relevant for the task, because processing the extraneous information uses up scarce attention and working memory resources. However, eye-tracking research suggests that task experience might be a boundary condition for this negative effect of extraneous information on learning, because people seem to learn to ignore task-irrelevant information over time. We therefore hypothesised that extraneous information might no longer hamper learning when it is present over a series of tasks, giving learners the chance to adapt their study strategy. This hypothesis was tested in three experiments. In experiments 1a/1b, participants learned the definitions of new words (from an artificial language) that denoted actions, with matching pictures (same action), mismatching pictures (another action), or without pictures. Mismatching pictures hampered learning compared with matching pictures. Experiment 2 showed that task experience may indeed be a boundary condition to this negative effect on learning: the initial negative effect was no longer present when learners gained experience with the task. This suggests that learners adapted their study strategy, ignoring the mismatching pictures. That hypothesis was tested in experiment 3, using eye tracking. Results showed that attention to the pictures waned with task experience, and that this decrease was stronger for mismatching than for matching pictures. Our findings demonstrate the importance of investigating multimedia effects over time and in relation to study strategies.     

A generative theory of textbook design: Using annotated illustrations to foster meaningful learning of science text

In three experiments, college students read a text explaining how lightning works and then took problem-solving transfer tests. Some students (integrated group) also viewed illustrations depicting the major stages in the formation of lightning that (a) were placed adjacent to corresponding text paragraphs and (b) contained annotations repeating the verbal cause-and-effect information from the text. Other students (separated group) viewed the same illustrations (a) on a separate page and (b) without annotations, after they had finished reading the text. The integrated group generated approximately 50% more creative solutions on transfer problems than the separated group, and this pattern was stronger for students who lacked experience in meteorology than for high-experience students. The positive effects of integrated illustrations depended on incorporating annotations (i.e., captions and labels) into the illustrations rather than placing illustrations close to corresponding paragraphs. Results were interpreted in light of a generative theory of multimedia learning which posits that meaningful learning requires constructing connections between visual and verbal representations of a system.     

Enabling, facilitating, and inhibiting effects of animations in multimedia learning: Why reduction of cognitive load can have negative results on learning

New technologies allow the display of text, static visuals, and animations. Although animations are inherently attractive, they are not always beneficial for learning. Problems may arise especially when animations modify the learner's cognitive load in an unintended way. In two learning experiments with 40 and 26 university students, the effects of animated pictures on knowledge acquisition were investigated. Some pictures displayed visual simulations of changes over time, whereas other pictures could be manipulated by learners to represent different states in time. Results showed that manipulation pictures had an enabling function for individuals with high learning prerequisites, whereas simulation pictures had a facilitating function for individuals with low learning prerequisites. However, the facilitating function was not beneficial for learning, because learners were prevented from performing relevant cognitive processes on their own. A careful analysis of the interrelation between different kinds of cognitive load and the process of learning is therefore required.     

Cognitive load theory and the effects of transient information on the modality effect

Based on cognitive load theory and the transient information effect, this paper investigated the modality effect while interpreting a contour map. The length and complexity of auditory and visual text instructions were manipulated. Experiment 1 indicated that longer audio text information within a presentation was inferior to the equivalent longer visual text information demonstrating a reversal of the modality effect due to transient information imposing a heavy working memory load. However, the expected modality effect was not obtained from the equivalent shorter auditory text presentation compared to shorter visual text information. It was hypothesised that the shorter text still contained too much auditory information for working memory to readily process. Experiment 2 further decreased the shorter auditory text information which then resulted in a traditional modality effect including a modality by text length interaction in which shorter, audio-visual information was better than visual only information but longer, audio-visual information was worse than visual only information.     

Learning from split-attention materials: Effects of teaching physical and mental learning strategies

Learners learn more from spatially separated text and pictures after they have been instructed to physically integrate these sources than without such an instruction. We investigated whether instructing learners to mentally integrate textual and pictorial information would yield similar results. Eighty-seven participants studied a picture with accompanying text about an electrical circuit. Text and picture were presented as spatially separated sources or in an integrated format. In the separated format, participants were instructed to use (1) a mental learning strategy, (2) a physical learning strategy, or (3) no learning strategy. Participants in the separated condition using a mental learning strategy and the integrated condition obtained higher recall and comprehension (but not causal inference) performance than participants in the separate conditions with the physical- or no learning strategy. This indicates that instructing learners to mentally integrate spatially separated text and pictures when studying split-attention material can be an effective learning strategy.     

Do drawing tasks improve monitoring and control during learning from text?

In two experiments it was investigated how drawing as a monitoring task affects self-regulated learning and cognitive load. To this end, participants (Exp. 1: N = 73, Exp. 2: N = 69) were randomly assigned to one of two conditions. In the experimental condition, students were asked to read an expository text on the formation of polar lights consisting of five paragraphs, whereby, after each paragraph, they had to create a drawing of the text's content. In the control condition, students read the same text, but performed no drawing task. In both conditions, students had to give judgments of learning (JoLs) after each paragraph and after reading the whole text as well as rate their cognitive load. Then, they were asked to select paragraphs for restudy. In Experiment 1, participants continued with an assessment of their learning outcomes immediately after their restudy selection, whereas in Experiment 2 they were first given the opportunity to actually restudy the selected paragraphs before working on the posttest. Results of both experiments indicate that JoLs rather than cognitive load predicted posttest performance. Moreover, students in the drawing condition compared with the control condition exhibited more accurate (relative) monitoring in Experiment 1 in that their JoLs were more strongly related to performance. Moreover, JoLs predicted students' restudy decisions in both experiments; however, this effect was by-and-large independent of whether they had to draw. Overall, results hint towards the potential of drawing to support metacognitive monitoring.     

What contributes to the split-attention effect? The role of text segmentation,picture labelling,and spatial proximity

In the split-attention effect spatial proximity is frequently considered to be pivotal. The transition from a spatially separated to a spatially integrated format not only involves changes in spatial proximity, but commonly necessitates text segmentation and picture labelling as well. In an experimental study, we investigated the influence of spatial proximity, text segmentation, and picture labelling on learning performance. A total of 165 students, divided into five groups, participated in the study. Four of the groups learned from spatially separated texts and pictures in a 2 × 2 design with the factors text segmentation (continuous vs. segmented text) and picture labelling (unlabelled vs. labelled picture). The fifth group learned from a spatially integrated text and picture. Retention and comprehension of the learning material were assessed. Students' working memory capacity and spatial ability were also assessed. The results replicated the split-attention effect with respect to retention only. This effect is attributed mainly to text segmentation and only partially to picture labelling. Spatial integration, however, did not enhance learning.     

Memory for Expository Text: the effects of verbal and pictorial enhancements

Previous research has shown that pictures can be effective in enhancing text memory. For example, Waddill and McDaniel (1992) found that pictures depicting detail information in an expository text effectively enhanced recall of those details and pictures depicting relational information effectively enhanced recall of those relations. However, the research on picture enhancement effects raises the question of whether the mnemonic value of pictures is simply a by‐product of having drawn attention to parts of the text through selective repetition of text content. The present study addressed the question by comparing the relative effectiveness of pictures versus simple verbal captions in promoting text memory. The results replicated the Waddill and McDaniel (1992) pictorial enhancement effect. However, the results showed that repeating the targeted information in a verbal caption was as effective as providing a pictorial illustration, thus suggesting that repetition of text content rather than nonverbal pictorial illustrations produced the effect. The findings generalized across three texts that differed in the amount of spatial content they conveyed.     

The modality effect in a mobile learning environment: Learning from spoken text and real objects

The finding that under split-attention conditions students learn more from a picture and spoken text than from a picture and written text (ie, the modality effect) has consistently been found in many types of computer-assisted multimedia learning environments. Using 58 fifth-grade and sixth-grade elementary school children as participants, we investigated whether the modality effect can also be found in a mobile learning environment (MLE) on plants' leaf morphology, in which students had to learn by integrating information from text and real plants in the physical environment. A single factor experimental design was used to examine the hypothesis that students in a mixed-mode condition with real plants and spoken text (STP condition) would pay more attention to the real plants, and achieve higher performance on retention, comprehension, and transfer tests than the single-mode condition with real plants and written text (WTP condition). Whereas we found that participants in the STP condition paid more attention to observing the plants, and achieved a higher score on the transfer test than participants in the WTP condition, no differences were found between the conditions for retention and comprehension test performance.     

FROM COVERT PROCESSES TO OVERT OUTCOMES OF REFUTATION TEXT READING: THE INTERPLAY OF SCIENCE TEXT STRUCTURE AND WORKING MEMORY CAPACITY THROUGH EYE FIXATIONS

This study extends current research on the refutation text effect by investigating it in learners with different levels of working memory capacity. The purpose is to outline the link between online processes (revealed by eye fixation indices) and off-line outcomes in these learners. In science education, unlike a standard text, a refutation text acknowledges readers’ alternative conceptions about a topic, refutes them, and presents scientific conceptions as viable alternatives. Lower and higher memory span university students with alternative conceptions about the topic read either a refutation or a non-refutation text about tides. Off-line measures of learning revealed that both groups of refutation text readers attained higher knowledge gains. During the reading process, refutation text readers fixated for longer on the refutation segments while reading the parts presenting the scientific information (look-froms). Non-refutation text readers looked back to the informational parts for longer. Look-froms (positively) and reading time (negatively) predicted learning from refutation text, indicating that the quality, not quantity, of reading was related to it. In contrast, learning from non-refutation text was predicted only by working memory capacity. The refutation effect is discussed and educational implications are drawn.     

The impact of stereoscopic imagery and motion on anatomical structure recognition and visual attention performance

Gross anatomy is located in a three‐dimensional space. Visualizing aspects of structures in gross anatomy education should aim to provide information that best resembles their original spatial proportions. Stereoscopic three‐dimensional imagery might offer possibilities to implement this aim, though some research has revealed potential impairments that may result from observing stereoscopic visualizations, such as discomfort. However, possible impairments of working memory such as decreased visual attention performance due to applying this technology in gross anatomy education have not yet been investigated. Similarly, in gross anatomy education the impact of stereoscopic imagery on learners’ recognition of anatomical‐spatial relationships and the impact of different presentation formats have only been investigated in a small number of studies. In this study, the performance of 171 teacher trainees working on the anatomy of hearing was examined, either with non‐stereoscopic or stereoscopic imagery. Static and dynamic picture presentations were applied. Overall, benefits for stereoscopic imagery on estimating anatomical‐spatial relations were found. The performance on a visual attention test indicates that the impact of stereoscopic visualizations on the human cognitive system varies more from person to person compared to non‐stereoscopic visualizations. In addition, combinations of temporarily moving pictures and stereoscopic imagery lead to decreased visual attention performance compared to combinations of moving pictures and non‐stereoscopic imagery. Anat Sci Educ 11: 15–24. © 2017 American Association of Anatomists.     

解析PKS背诵模式的认知优势——基于思维导图和词块的应用

PKS背诵模式从语言学和认知心理学的角度对背诵重新解析.该模式以词块为关键词作为背诵的基本单位,以图片表达词块帮助学习者形成视觉映像,利用思维导图管理词块,再辅助以视听读结合的输入方式,寓背诵于娱乐,激发学习者的学习动机,提高背诵效率.PKS背诵有着强大的认知优势,词块可以清除理解障碍,思维导图可以促进背诵图式的建构,二者结合拥有强大的助记功能.该模式的应用有利于培养学习者的想象力和自主学习能力.     

What you see is what you remember? Depictions of historical figures influence memory for historical facts

We investigated whether the characteristics of pictures affect memory for verbal learning materials. In the learning phase, participants watched a slideshow about historical figures that were depicted as more or as less powerful. The slideshow included spoken information that was more prototypical for either powerful or powerless people and thus semantically congruent (e.g., depiction and verbal information rated as powerful) or incongruent with the pictures. Congruency resulted in more errors when participants had to reject incorrect items in a memory test. There was no effect of congruency on verifying correct items. This pattern was observed for a slideshow that included one picture for each historical figure (Experiment 1) and for a slideshow including additional pictures (Experiment 2). Hence, both experiments provide corroborating evidence that pictures affect learners’ memory for verbal learning materials. These findings are relevant for instructional practice with regard to the design of learning materials and test items.     

The role of decorative pictures in learning

Three experiments with students from 7th and 8th grade were performed to investigate the effects of decorative pictures in learning as compared to instructional pictures. Pictures were considered as instructional, when they were primarily informative, and as decorative, when they were primarily aesthetically appealing. The experiments investigated, whether and to what extent decorative pictures affect the learner’s distribution of attention, whether they have an effect on the affective and motivational state and whether they affect the learning outcomes. The first experiment indicated with eye-tracking methodology that decorative pictures receive only a bit initial attention as part of the learner’s initial orientation and are largely ignored afterwards, which suggests that they have only a minor distracting effect if any. The second experiment showed that despite the small amount of attention they receive, decorative pictures seem to induce better mood, alertness and calmness with learners. The third experiment indicated that decorative pictures did not intensify students’ situational interest, but reduced perceived difficulty of the learning material. Regarding outcomes of learning, decorative pictures were altogether neither harmful nor beneficial for learning. However, they moderated the beneficial effect of instructional pictures––in essence: the multimedia effect. The moderating effect was especially pronounced when learners had lower prior knowledge. The findings are discussed from the perspective of cognitive, affective and motivational psychology. Perspectives of further research are pointed out.     

The Role of Visual Indicators in Dual Sensory Mode Instruction

Advances in our knowledge of the structure of working memory suggest that under some circumstances, effectively more processing capacity is available to learners if instructional materials use multiple information modes (e.g. auditory and visual) instead of equivalent single mode formats. This paper examined this modality effect from a cognitive load perspective in three experiments using geometry instruction. In accordance with cognitive load theory, it was predicted that the additional processing capacity provided in an audio/visual format would only enhance learning if mental resources were not devoted to extensive visual based search in order to coordinate auditory and visual information. Using two different areas of geometry, Experiments 1 and 2 found that if visual search was clearly high, then audio‐visual instruction was only beneficial if visual indicators in the form of electronic flashing were incorporated into the instructional format. Under high search conditions, a standard audio/visual format without the aid of flashing proved no better than visual only instruction. Experiment 3 attempted to clarify earlier results by using instructional materials trivially low in visual search. Data indicated that a standard, audio‐visual format resulted in superior learning to a visual only format. There was no beneficial effect of electronic flashing in Experiment 3, suggesting that visual indicators were not necessary in areas of low visual search. It was concluded that cognitive load as determined by visual search established the effectiveness of visual indicators.