The Role of Working Memory in Multimedia Instruction: Is Working Memory Working During Learning from Text and Pictures?

A lot of research has focused on the beneficial effects of using multimedia, that is, text and pictures, for learning. Theories of multimedia learning are based on Baddeley’s working memory model (Baddeley 1999). Despite this theoretical foundation, there is only little research that aims at empirically testing whether and more importantly how working memory contributes to learning from text and pictures; however, a more thorough understanding of how working memory limitations affect learning may help instructional designers to optimize multimedia instruction. Therefore, the goal of this review is to stimulate such empirical research by (1) providing an overview of the methodologies that can be applied to gain insights in working memory involvement during multimedia learning, (2) reviewing studies that have used these methodologies in multimedia research already, and (3) discussing methodological and theoretical challenges of such an approach as well as the usefulness of working memory to explain learning with multimedia.     

Focus of attention and choice of text modality in multimedia learning

The term ‘modality effect’ in multimedia learning means that students learn better from pictures combined with spoken rather than written text. The most prominent explanations refer to the split attention between visual text reading and picture observation which could affect transfer of information into working memory, maintenance of information in working memory or the effective size of working memory. The assumption of a continuous need for split attention is questionable, however. Learners can keep pictorial information in working memory, when they have seen the picture before, especially if they have higher prior knowledge. Instead of suffering from a permanent split attention, learners frequently show tendencies to simply ignore pictures. This suggests guiding learners towards picture analysis by picture-related text paragraphs. We assume that these paragraphs are associated with stronger modality effects than content-related paragraphs, especially if the pictures are new to learners. These assumptions were tested in an experiment with 120 students learning about volcanism from illustrated text consisting of segments each including a content-related paragraph followed by a picture-related paragraph describing the accompanying visualization. Content-related and picture-related paragraphs were presented as visual or auditory texts leading to 2x2 conditions of text presentation. Picture novelty was manipulated by presenting a picture throughout the whole segment or only when the picture-related paragraph was read. As expected, picture-related paragraphs were associated with stronger modality effects than content-related paragraphs if picture novelty is high. The distinction between different kinds of paragraphs seems to be important for the prediction of modality effects.     

Identifying processes underlying the multimedia effect in testing: An eye-movement analysis

Test items become easier when a representational picture visualizes the text item stem; this is referred to as the multimedia effect in testing. To uncover the processes underlying this effect and to understand how pictures affect students' item-solving behavior, we recorded the eye movements of sixty-two schoolchildren solving multiple-choice (MC) science items either with or without a representational picture. Results show that the time students spent fixating the picture was compensated for by less time spent reading the corresponding text. In text-picture items, students also spent less time fixating incorrect answer options; a behavior that was associated with better test scores in general. Detailed gaze likelihood analyses revealed that the picture received particular attention right after item onset and in the later phase of item solving. Hence, comparable to learning, pictures in tests seemingly boost students' performance because they may serve as mental scaffolds, supporting comprehension and decision making.     

Learning from split-attention materials: Effects of teaching physical and mental learning strategies

Learners learn more from spatially separated text and pictures after they have been instructed to physically integrate these sources than without such an instruction. We investigated whether instructing learners to mentally integrate textual and pictorial information would yield similar results. Eighty-seven participants studied a picture with accompanying text about an electrical circuit. Text and picture were presented as spatially separated sources or in an integrated format. In the separated format, participants were instructed to use (1) a mental learning strategy, (2) a physical learning strategy, or (3) no learning strategy. Participants in the separated condition using a mental learning strategy and the integrated condition obtained higher recall and comprehension (but not causal inference) performance than participants in the separate conditions with the physical- or no learning strategy. This indicates that instructing learners to mentally integrate spatially separated text and pictures when studying split-attention material can be an effective learning strategy.     

Processing multimedia material: Does integration of text and pictures result in a single or two interconnected mental representations?

We investigated whether individuals construct either two interconnected mental representations or a single mental representation that merges information from two representations when processing multimedia materials. Individuals memorized text-picture stimuli in four different versions that differed in the specificity of information contained in either text or pictures: general pictures/general sentences, general pictures/specific sentences, specific pictures/general sentences, and specific pictures/specific sentences. Afterwards, individuals decided whether they had previously seen the specific or the general version of the sentences and the pictures. Across two experiments, individuals more frequently falsely recognized the specific sentences after having seen general sentences/specific pictures. This indicates that individuals had integrated the specific picture information with the general sentence information into a single mental representation. No such effects were observed for picture recognition. The implications of these results for multimedia learning are discussed.     

Killing the Buddha: Towards a heretical philosophy of learning

This article explores how different philosophical models and pictures of learning can become dogmatic and disguise other conceptions of learning. With reference to a passage from St. Paul, I give a sense of the dogmatic teleology that underpins philosophical assumptions about learning. The Pauline assumption is exemplified through a variety of models of learning as conceptualised by Israel Scheffler. In order to show how the Paulinian dogmatism can give rise to radically different pictures of learning, the article turns to St. Augustine’s and Robert Brandom’s examples of language learning, and to general strands in scholarship on moral education. Dewey’s view of childhood immaturity and the problem of adult maturity are used as first attempt at a counter picture to the idea that learning must have an end. The article takes Dewey’s idea further by suggesting how the Zen-Buddhist idea of killing the Buddha and Wittgenstein’s method of destroying pictures work on the dogmatic focus on uses of ‘learning’ that assume ends. In conclusion, the article suggests three possible uses of ‘learning’—learning from wonder, intransitive learning and passionate learning—that do not assume that learning has or must have a teleological end.     

Individual and co-operative learning with interactive animated pictures

Computer-based multimedia learning environments make it possible to present interactive animated pictures, which can be manipulated for active exploratory learning and which allow the dynamic behavior of complex subject matter to be displayed. Due to the large range of possibilities for exploratory interaction, such learning environments seem well suited for co-operative learning in which different learners analyse the presented subject matter from different perspectives. This paper first describes a theoretical framework for learning from texts and pictures together with an analysis of possible effects of animation and interactivity on knowledge acquisition. It then presents two empirical studies in which knowledge acquisition from interactive animated pictures was compared with knowledge acquisition from static pictures under the conditions of individual learning (Study I) and of cooperative learning (Study II). In Study I, learning with interactive animated pictures resulted in a superior encoding of detail information, but did not facilitate performance of mental simulation tasks. In Study II, learning with interactive animated pictures resulted both in inferior encoding of detail information and poorer results in mental simulations. These findings and the analysis of discourse protocols of subjects’ co-operation suggest that exploratory learning with interactive animated pictures is associated with extraneous cognitive load and that this load can be further increased by the co-ordination demands of co-operative learning. Although animated pictures may provide external support for mental simulations, they do not appear to be generally beneficial for learning, because they can prevent individuals from performing relevant cognitive processes.     

Influence of meaningfulness,picture detail,and presentation mode on visual retention

Summary The present study investigated the relationships of meaningfulness, picture detail, and presentation mode on visual learning. Subjects, 123 broadcasting students at Brooklyn College, viewed slides of animal pictures previously classified by judges into high-and low-meaningful groups. All Ss saw both high-and low-meaningful stimulus items. Half saw the stimulus items as full-color pictures while the other half saw them as line drawings. In addition, one-third saw the stimulus items as pictures presented alone, one-third saw them as pictures with each animal name printed underneath its picture, and one-third saw the pictures (without printed names) accompanied by the spoken animal name as each slide was shown. Following presentation of stimulus slides, Ss were shown a series of full-color test slides containing dummy items randomly intermixed with stimulus items. During two showings of test slides, Ss first indicated those animals recognized from the learning trial and second, wrote down those animal names they knew. Results included significant main effects on recognition accuracy for meaningfulness and presentation mode in addition to significant interactions for meaningfulness-by-mode and mode-by-picture-detail. A significant main effect of presentation mode resulted for correctly named stimulus items. Three significant main effects (meaningfulness, picture detail, and mode) and one significant interaction (mean-ingfulness-by-picture-detail) were obtained on error scores.     

Interactive and non-interactive pictures in multimedia learning environments: Effects on learning outcomes and learning efficiency

New technologies enable flexible combinations of text and interactive or non-interactive pictures. The aim of the present study was to investigate (a) whether adding pictures to texts is generally beneficial for learning or whether it can also have detrimental effects, (b) how interactivity of pictures affects learning, (c) whether the visualization format of pictures affects the structure of the learner's mental model, and (d) whether the visualization format modifies the effects of interactivity. One hundred university students were randomly assigned to five groups. In four groups, a text about the different daytimes and days on the earth was combined with interactive or non-interactive pictures of different visualization formats. In the fifth group, the text was presented without pictures. According to the results, adding pictures to text was neither beneficial nor harmful for learning. In terms of learning efficiency, however, learning from text only was more successful than learning from text and pictures. Interactivity was beneficial for one learning task, but not for the other task. The visualization format affected participants’ interaction with pictures, but not the learning outcomes; however this effect was not influenced by interactivity. Implications for multimedia design and for further research are pointed out.     

How repeated studying and testing affects multimedia learning: Evidence for adaptation to task demands

Two biases can occur in multimedia learning: overconfidence and over-reliance on text processing. The present research sought to identify these biases and to investigate whether they can be reduced, and hence learning fostered, when studying and testing are repeated. In 2 experiments (Exp.1: N = 79, Exp.2: N = 52), students learned either with text only or with text and pictures (multimedia) about how the toilet flush works, gave judgments-of-learning (JOLs), were tested on the learning contents; afterwards this study-test cycle was repeated. Results from both experiments revealed stronger overconfidence due to multimedia in both study-test cycles (JOLs higher than learning outcomes). Eye movement data showed a relative increase in attention on the picture versus text from cycle 1 to cycle 2; this relative increase in attention was related to better learning outcomes. Repeated studying and testing thus helped to reduce over-reliance on text processing in multimedia learning, fostering performance.     

Mobile and Online Learning Journal: Effects on Apprentices’ Reflection in Vocational Education and Training

While learning journals (LJs) have been shown to support self-regulated learning strategies, reflection and learning outcomes in academic contexts, few studies have investigated their relevance in vocational education. A mobile and online learning journal (MOLJ) was developed to support reflection on workplace experiences. However, acceptance of the mobile and online learning journal by apprentices and supervisors is not trivial. This study investigated how apprentices use the mobile and online learning journal and why they use it that way. The mobile and online learning journal was developed for apprentices in the field of bakery and contains a recipe book, a smartphone app to take pictures of workplace experiences, and a learning journal for reflection. Apprentices’ workplace supervisors have access to the mobile and online learning journal to provide support and feedback. A mixed method study was conducted with 16 apprentices. Indicators of their use of the mobile and online learning journal were recorded, while reasons for use were studied by questionnaires and semi-structured interviews. Four kinds of users of the mobile and online learning journal were identified: Multifunction users, picture takers, learning journal users, and overall low users. The most determining factors for the acceptance and use of the mobile and online learning journal by apprentices were their interest in the use of a smartphone at the workplace, their acceptance of reflection on workplace experiences through the mobile and online learning journal, and supervisors’ participation and provision of feedback. Promoting the use of learning journals by apprentices requires strong guidance in the design of the learning journal as well as support and feedback from supervisors. Taking pictures from the workplace proves useful for learning and sharing experience in vocational training.     

Preschoolers’ learning when pictures do not tell the whole story

Preschool children heard two ten-sentence stories. Each sentence was accompanied by two pictures (one of the subject and one of the object of the sentence), by a picture of the sentence’s subject, by a picture of the sentence’s object, or by no pictures. After presentation of the stories, the children answered questions requiring recall of the sentence objects. Only the two-picture illustration variation produced significantly beffer learning relative to no-picture control performance. This finding held both when children were instructed to make mental images representing the prose and when they simply listened to the text. The results are relevant both to theories of children’s imagery and to practical issues about how to illustrate children’s texts. This research was supported by a grant to the second author from the Natural Sciences and Engineering Research Council of Canada.     

One Cow Does Not an Animal Make: Young Children Can Extend Novel Words at the Superordinate Level

Six match-to-sample picture/object selection experiments were designed to explore children's knowledge about superordinate words (e.g., "food") and how they acquire this knowledge. Three factors were found to influence the learning and extension of superordinate words in 3- to 5-year-old children (N = 230): The number of standards (one versus two), the type of standards presented (from different basic-level categories versus from the same basic-level category), and the nature of the object representations used (pictures versus objects). A different pattern of superordinate word acquisition was found between 3-year-olds and 4- and 5-year-olds. Although 4- and 5-year-olds could learn and extend novel words to superordinate categories in the presence of two picture exemplars from different categories or a single three-dimensional (3-D) exemplar, 3-year-olds could do so only in the presence of two 3-D exemplars. These findings indicate that young children's acquisition of superordinate words is influenced by multiple factors and that there is a developmental progression from multiple exemplars to single exemplars in superordinate word learning.     

The role of decorative pictures in learning

Three experiments with students from 7th and 8th grade were performed to investigate the effects of decorative pictures in learning as compared to instructional pictures. Pictures were considered as instructional, when they were primarily informative, and as decorative, when they were primarily aesthetically appealing. The experiments investigated, whether and to what extent decorative pictures affect the learner’s distribution of attention, whether they have an effect on the affective and motivational state and whether they affect the learning outcomes. The first experiment indicated with eye-tracking methodology that decorative pictures receive only a bit initial attention as part of the learner’s initial orientation and are largely ignored afterwards, which suggests that they have only a minor distracting effect if any. The second experiment showed that despite the small amount of attention they receive, decorative pictures seem to induce better mood, alertness and calmness with learners. The third experiment indicated that decorative pictures did not intensify students’ situational interest, but reduced perceived difficulty of the learning material. Regarding outcomes of learning, decorative pictures were altogether neither harmful nor beneficial for learning. However, they moderated the beneficial effect of instructional pictures––in essence: the multimedia effect. The moderating effect was especially pronounced when learners had lower prior knowledge. The findings are discussed from the perspective of cognitive, affective and motivational psychology. Perspectives of further research are pointed out.     

What you see is what you remember? Depictions of historical figures influence memory for historical facts

We investigated whether the characteristics of pictures affect memory for verbal learning materials. In the learning phase, participants watched a slideshow about historical figures that were depicted as more or as less powerful. The slideshow included spoken information that was more prototypical for either powerful or powerless people and thus semantically congruent (e.g., depiction and verbal information rated as powerful) or incongruent with the pictures. Congruency resulted in more errors when participants had to reject incorrect items in a memory test. There was no effect of congruency on verifying correct items. This pattern was observed for a slideshow that included one picture for each historical figure (Experiment 1) and for a slideshow including additional pictures (Experiment 2). Hence, both experiments provide corroborating evidence that pictures affect learners’ memory for verbal learning materials. These findings are relevant for instructional practice with regard to the design of learning materials and test items.     

A Gorilla with ‘Grandpa's Eyes’: How Children Interpret Visual Texts—A Case Study of Anthony Browne's Zoo

This article explores the multilayered nature of a single picture book by Anthony Browne and the sophisticated responses (including their own pictures) children aged 4–11 bring to interpreting such a text. Emphasis is laid on the high-level cognitive skills involved in reading visual images and links are made between seeing and thinking. Some features of the children's understanding are examined in detail; for example, how they interpret visual imagery and deal with a variety of challenging artistic features, and how their drawings show knowledge and feelings they are not yet able to articulate. The findings also suggest that some children who are not yet confident at reading print have developed impressive capacities for analysing image.     

Young primary students making sense of text and illustrations about how refuse can become soil

Explanatory pictures and models are frequently used in teaching and learning situations. However, it seems to be simply assumed that they are always beneficial. In this article results from an investigation with 16 Swedish pupils aged 7–9 year are presented based on an analysis that has examined how well this assumption holds up. Concepts from multi-modal theory have been used to investigate how young learners deal with illustrations and text from an early reader booklet about composting domestic refuse. The analysis suggests that expectations that illustrations facilitate the meaning-making of young pupils may be exaggerated. Although the booklet claimed to provide interactive support between image and text most of the examples show pupils ignoring pictures or misinterpreting vital information about composting in both the verbal and non-verbal material. The illustrations did not compensate for the most crucial deficiencies in the written text.     

Solving complex problems: A convergent approach to cognitive load measurement

The study challenged the current practices in cognitive load measurement involving complex problem solving by manipulating the presence of pictures in multiple rule‐based problem‐solving situations and examining the cognitive load resulting from both off‐line and online measures associated with complex problem solving. Forty‐eight participants were recruited from a subject pool in the Educational Psychology department in a large research university in western USA. Results showed that the presence of pictures had no effect on learners' complex problem solving as measured by the response time and accuracy. However, the online measure (eg, pupillometric measures) revealed a change in cognitive load associated with the presence of pictures in complex problem solving. The authors concluded that different measures of cognitive load may actually be measuring separate aspects of cognitive load. Discussions were made on how research on multimedia learning and cognitive load could be advanced by carefully considering multiple aspects of cognitive load and by including the use of convergent measurement techniques to capture the variations of cognitive load involved in learning.     

Pictures as Feedback in Single Word Learning

A number of recent studies have indicated that familiarity with pictures can interfere with single word learning, if the pictures are presented with their written equivalents. Research has also suggested that focusing children's attention on written words while responding to the paired stimuli is not sufficient to overcome this interference. Two experiments were carried out to investigate the possibility of eliminating the adverse effects of pictures by encouraging children to elicit naming‐responses in the absence of the pictures and then using pictures as response feedback. The results indicated that this feedback cueing technique can avoid the adverse effect of pictures and achieve a learning rate equivalent to that obtained when words were presented alone. It was concluded that pictures could be used in a way which did not hinder learning, but there is no evidence to suggest that they can be used in a way to enhance learning.     

In the physics class: university physics students’ enactment of class and gender in the context of laboratory work

This article explores how the doing of social class and gender can intersect with the learning of science, through case studies of two male, working-class university students’ constitutions of identities as physics students. In doing so, I challenge the taken-for-granted notion that male physics students have an unproblematic relation to their chosen discipline, and nuance the picture of how working-class students relate to higher education by the explicit focus on one disciplinary culture. Working from the perspective of situated learning theory, the interviews with the two male students were analysed for how they negotiated the practice of the physics student laboratory and their own classed and gendered participation in this practice. By drawing on the heterogeneity of the practice of physics the two students were able to use the practical and technological aspects of physics as a gateway into the discipline. However, this is not to say that their participation in physics was completely frictionless. The students were both engaged in a continuous negotiation of how skills they had learned to value in the background may or may not be compatible with the ones they perceived to be valued in the university physicist community.