Propensity Score Matching for Education Data: Worked Examples

Abstract

Randomized controlled trials are not always feasible in educational research, so researchers must use alternative methods to study treatment effects. Propensity score matching is one such method for observational studies that has shown considerable growth in popularity since it was first introduced in the early 1980s. This paper outlines the concept of propensity scores by explaining their theoretical principles and providing two examples of their usefulness within the realm of educational research. Through worked examples, we highlight the effectiveness of propensity scores as a method for reducing bias and increasing the balance between treatment and comparison groups. To aid in the understanding and future use of propensity scores, we provide R syntax for all our analyses.     

Improving Conceptual Understanding and Representation Skills Through Excel-Based Modeling

The National Research Council framework for science education and the Next Generation Science Standards have developed a need for additional research and development of curricula that is both technologically model-based and includes engineering practices. This is especially the case for biology education. This paper describes a quasi-experimental design study to test the effectiveness of a model-based curriculum focused on the concepts of natural selection and population ecology that makes use of Excel modeling tools (Modeling Instruction in Biology with Excel, MBI-E). The curriculum revolves around the bio-engineering practice of controlling an invasive species. The study takes place in the Midwest within ten high schools teaching a regular-level introductory biology class. A post-test was designed that targeted a number of common misconceptions in both concept areas as well as representational usage. The results of a post-test demonstrate that the MBI-E students significantly outperformed the traditional classes in both natural selection and population ecology concepts, thus overcoming a number of misconceptions. In addition, implementing students made use of more multiple representations as well as demonstrating greater fascination for science.     

Improving Student Performance Through Computer-based Assessment: Insights from recent research

One of the benefits claimed for computer-based assessment is that it can improve student performance in summative assessments. During the introduction of computer-based assessment in a first-year module on numeracy and statistics in Biology, online assessment was used to replace OMR-marked multiple-choice tests. Analysis of student results after the first year (Ricketts & Wilks, 2001) showed that students using online assessment did not perform as well as those using OMR-marked multiple-choice questions. The difference in performance could not be attributed to a weaker student cohort. In the second year student performance improved dramatically when they were not required to scroll through the question paper. Our results suggest that students may be disadvantaged by the introduction of online assessment, unless care is taken with the student-assessment interface.     

Interactive Example-Based Learning Environments: Using Interactive Elements to Encourage Effective Processing of Worked Examples

This review describes parts of our research program on example-based learning that relates to recent efforts to incorporate interactive elements into learning environments designed to support learning from worked-out examples. Since most learners spontaneously study or process examples in a very passive or superficial manner, this review focuses on how a variety of specific interactive elements in example-based leaning environments are capable of encouraging learners to actively process the examples. The review begins with an overview of the literature on worked examples and the associated self-explanation, which is important given that the quality of self-explanation is a major factor in determining whether learners benefit from studying examples. The review notes that example-based learning environments tend to be effective but often promote passive processing. It then highlights the strengths and limitations of three types interactivity introduced to example-based learning environments. The review concludes with a discussion of the role that these interactive elements play in these learning environments.     

Improving Silent Reading Performance Through Feedback on Eye Movements: A Feasibility Study

This feasibility study investigated if feedback about individual eye movements, reflecting varying word processing stages, can improve reading performance. Twenty-five university students read 90 newspaper articles during 9 eye-tracking sessions. Training group participants (n = 12) were individually briefed before each session, which eye movement parameter(s) (fixation count, first fixation duration, regression, and/or skipping) to address, and informed about changes achieved in preceding sessions. Control group participants (n = 13) were told that self-instruction to read faster would produce training gains. Total fixation times decreased significantly more for training than for control group participants. Important to note, faster reading did not impair comprehension. Results are interpreted as first indications for a possible applicability of the feedback approach to silent reading. In addition to implications for future studies, alternative result interpretations (e.g., motivational effects, reduction of mindless reading) are discussed.     

Worked Examples Versus Non‐goal‐specific Problems: a test of schema development in geometry

A multiple baseline across subjects experimental design was used to compare the effects of a teaching programme designed to enhance schemata acquisition with that of worked examples, traditionally used to teach geometry to high school students. In the first treatment, students were guided to form their own personal, independent schema through the use of non‐goal‐specific questions, that is, questions that did not ask to find a single, specific unknown. The second treatment introduced students to solved examples of problems asking for specific values. Following exposure to one or other of the treatments, measures were obtained of students’ success in problem solving, the time taken and the processes used. Results show that students in both groups had gains in the number of problems solved following intervention, with those exposed to the non‐goal‐specific procedure showing greater rates of improvement and greater efficacy in their problem‐solving strategies. These findings are discussed in terms of their implications for a schemata acquisition and problem‐solving hypothesis.     

Cognitive Load Theory: Advances in Research on Worked Examples, Animations, and Cognitive Load Measurement

The contributions to this special issue document some recent advances of cognitive load theory, and are based on contributions to the Third International Cognitive Load Theory Conference (2009), Heerlen, The Netherlands. The contributions focus on developments in example-based learning, amongst others on the effects of integrating worked examples in cognitive tutoring systems; specify the effects of transience on cognitive load and why segmentation may help counteract these effects in terms of the role of time in working memory load; and discuss the possibilities offered by electroencephalography (EEG) to provide a continuous and objective measure of cognitive load. This article provides a short introduction to the contributions in this issue.     

Performance Indicator Maps: A Visual Tool for Understanding,Managing, and Continuously Improving Your Business Metrics

Organizations are complex systems that require managers and employees alike to understand the interdependencies between their performance accomplishments and those of others, as well as the specific means and actions that affect them. Performance indicator maps (PIMs) are graphical representations that illustrate the relationships among organizational performance indicators. When properly designed, performance indicator maps can increase the effectiveness of feedback generated by dashboards, trigger tangible improvement actions today, and allow us to get the results we want tomorrow.     

Improving Academic Advising Through Student Seminars: A Case Study

Effective advising can be a challenge for most criminal justice and criminology programs due to the high enrollment and the variety of careers available in the field. One technique designed to address these challenges is the group advising method. The purpose of this study was to examine the details, processes, and execution of a particular group advising method, called student advising seminars, implemented over a two-year period in a criminal justice department at a four-year state-owned university. Preliminary data measuring student accountability and satisfaction are also examined. The findings indicate student advising seminars, as it is described in this case study, have a potential of providing effective and efficient advising.     

Total Quality Management: Improving Pupil Performance

It is frequently argued that work relating to key areas of pupils’ personal and social education starts too late, that it takes place after crucial attitudes, perceptions and understandings have already been developed by pupils. Generally the problem is that work does not start until the secondary stage. This paper describes an evaluation of careers work undertaken with Year 4 pupils in a primary school. In this school, a yearly careers week was held when structured activities were organized around talks about their work given by parents. The researcher found that 5 months after the careers work, the Year 4 pupils remembered much of significance. These findings are then related to a number of theories of career development which support such work early in pupils’ careers. The paper concludes by outlining the benefits that careers work in the primary school offers.     

Improving the CBT Development Process Through Knowledge Sharing: An Empirical Study

In this paper, techniques from the research field of knowledge management are used to improve the maturity of the Computer Based Training (CBT) development process at an IT-oriented organization involved in producing courseware. The project team is facilitated with a knowledge-sharing solution that enables them to unambiguously define their project process, and link templates and best practices to the activities in that process. In addition, a virtual discussion room is provided for them to share new insights. Results from measurement experiments indicate more positive project experiences with projects that used the knowledge-sharing solution.     

Class-wide Positive Behavior Support in Preschool: Improving Teacher Implementation Through Consultation

As the number of young children displaying challenging behavior in early childhood grows, so too does the need to implement evidence-based practices that prevent challenging behavior. Positive Behavior Support (PBS) provides a framework of tiered interventions focused on promoting social-emotional development and preventing challenging behavior. This study investigated the effects of consultation on teachers’ implementation of universal PBS practices and children’s academic engagement. A multiple-baseline design was applied across four preschool classrooms serving children from 33 to 63 months of age. A strong relationship was documented between consultation and teachers’ implementation of PBS skills. High levels of academic engagement were maintained following consultation. Implications of the results are provided for applications of universal PBS practices in early childhood settings.     

Education as Service: The Understanding of University Experience Through the Service Logic

With the marketization of UK higher education, this paper develops a framework from services marketing that can assist universities in understanding what market orientation means and how students would value their offerings. Our study shows that the core service in a university experience is a learning experience that is cocreated and that the value is emergent, unstructured, interactive, uncertain, with a hedonic dimension. Our paper modifies the gap model of service quality to show that an ideological gap exists that may also impede the quality of the university experience. We propose that a one-sided expectation by students leads to student consumerism and disengagement. Paradoxically, we show that a true student-orientated marketing puts the university ideology at the center of marketing efforts and that marketing may well be an effective tool to communicate such ideologies.     

Learning 'How' Versus Learning 'When': Improving Transfer of Problem-Solving Principles

How can problem solving be improved in domains where similar principles are learned? A series of three experiments based on Ross (1987) examined how instructing learners about when to apply problem-solving principles may later improve performance. In Experiment 1, subjects studied a similar pair (combinations and permutations) or a distinct pair (combinations and conditional probability) of probability principles. Haft of the subjects received information on when to apply the principles (applicability-instructions condition), whereas other subjects received instructions that reviewed how to solve problems using the principles (procedural-review condition). Subjects who received applicability instructions made fewer confusion errors than subjects who received procedural-review instructions when learning the similar problem pair. However, this instructional manipulation had no effect when subjects were learning the distinct problem pair. Because applicability instructions affect confusion errors but not overall performance, they may improve the ability to identify when to apply a procedure but may not improve memory for the formula or the ability to instantiate it. Experiment 2 supported this notion by showing that subjects given applicability instructions did better at selecting when to use each formula. Experiment 3 tested whether giving learners the pairs of examples to review together afforded them the opportunity to learn the applicability conditions on their own. The results showed that the important differences between the principles (i.e., why the formulas apply in certain situations) must be identified in the instructional material because subjects were not likely to induce the differences on their own. The results showed that when differences between the related principles are identified in instructional material (i.e., why the formula applies in certain problems), these applicability instructions may serve to reduce confusion in noticing and selecting...     

Improving Design Understandings and Skills through Enhanced Metacognition: Reflective Design Journals

The main aim of this study was to investigate and discover whether going through the process of reflection by keeping reflective design journals (RDJ) enhances architecture students’ metacognition and whether this enhanced metacognition improves their design understandings and skills. The study was a mixed‐methods design and utilised content analysis method to identify the metacognitive actions of the participants. The study also investigated participants’ attitudes towards RDJs and their views regarding the effect of enhanced metacognition on their design understandings and skills. Twenty college students registered to an undergraduate course offered by the department of Architecture participated in the research. The findings of the study revealed that by writing in their RDJs, participants were able to progressively enhance their metacognitive skills and performed several metacognitive actions by using the four main metacognitive strategies: awareness, organisation and planning, monitoring, and evaluation. The results also disclosed that participants found RDJ keeping exceptionally effective and stated that their enhanced metacognition improved their design understanding and abilities.