What you design is not what you get (WYDINWYG): Cognitive,affective, and social impacts of learning with ils — An integration of findings from six-years of qualitative and quantitative studies

Despite the wide application of ILSs in schools today, little research is available to inform us of their actual effects on teaching and learning and to guide us how to design an optimal ILS environment. This chapter integrates results of six years of qualitative/naturalistic and quantitative studies of arithmetic practice in four ILSs (two in the U.S. and two in Israel), in order to provide us with this important information. These studies examined a variety of cognitive, behavioral, sociological, affective, and instructional issues identified in the ILS work of students and their teachers, and the effects of a variety of ILS design issues related to hardware, software, and method of operation on students' and teachers' behavior in the ILS environment.     

Integrated learning systems: where does the management take place?

For the last four years of the UK national investigation into the effectiveness of integrated learning systems (ILSs) we have been monitoring the educational contexts within which ILSs are used. Management issues related to ILS use are a significant factor in that educational context. In this paper some of the key management issues related to the effective use of ILSs are outlined. These will include: styles and levels of management of the learning by the ILS; management of the ILS as a learning tool (models of ILS usage); partnership or subservience (‘who’ manages ‘who’ and does it matter?). For some schools, however, the introduction of an ILS has had more profound impacts than this, in that we have recorded changes in teachers' pedagogy. Where such profound changes have been reported then those factors and processes now clearly modelled in school effectiveness and improvement debates have been shown to be operating     

Embedding cooperative learning into the design of integrated learning systems: Rationale and guidelines

An Integrated Learning System (ILS) is an advanced computer-based instructional system, generally consisting of a set of computerized courseware covering several grade levels and content areas, and complex classroom management and reporting features. Although ILSs have become increasingly popular in schools over the past five to ten years, they introduce several potential factors that could have negative effects on students' academic and social growth. These factors include: (a) de-emphasis of affective outcomes and increased student isolation, (b) lack of teacher involvement in curriculum planning and delivery, and (c) disparate effects on student achievement based on students' academic level. One possible strategy for addressing these concerns is integrating cooperative learning with ILS instruction. This paper will examine the research dealing with integrating cooperative learning strategies and computer-based instruction and provide guidelines and strategies for designing ILS instruction that enhances opportunities for cooperative learning.Thomas A. Brush is Assistant Professor of Educational Technology at Arizona State University where he can be reached at Educational Media and Computers, Arizona State U., Box 870111, Tempe, AZ 85287-0111     

A comparative study of the impact of integrated learning systems on students' time-on-task

Time-on-task has long been thought to be an important predictor of student learning and achievement. Thus, it is important to understand the extent to which computer-based instruction (CBI), in general, and more specifically, computerized integrated learning systems (ILSs) affect students' time-on-task behavior. This study reports data from a year-long, 14-school national study in which rigorous time-on-task measures were used to compare students' engaged time in three primary situations: (1) an ILS used in a computer laboratory configuration; (2) an ILS used in a distributed configuration (work stations distributed throughout a school's classrooms); and (3) regular classroom instruction in which no ILS was used. The results indicate that students using the ILS spend more time actively engaged in the learning tasks than students in the non-ILS classroom. This appears to be a very robust finding, occurring regardless of whether or not there is an adult present. It appears that the ILS provides a highly motivating learning environment that engages and maintains student attention. The level of on-task behavior is significantly higher than that found in more typical classroom settings. Further, the ILS showed a significant advantage in increasing time-on-task with at-risk students. Implications of this study for further research are also discussed.     

Mindless or mindful use of integrated learning systems

This chapter presents the results of a study of computer-based integrated learning systems at two elementary schools using different vendors' ILS software. The research design incorporated same-classroom pretest-matched controls in which one-half of each class used the ILS for mathematics while the other half used it for reading. Overall effects at both schools were not strongly different from zero. However, a curvilinear pattern of effects occurred at one school in math and at the other in reading. In these situations, students at the top and the bottom of their school's prior achievement distribution did better when using ILSs, and students in the middle did better with only traditional instruction provided to them. The results are interpreted as challenging the practicality of individualized instructional approaches. Such approaches require teachers to be attentive to many different learning activities simultaneously. To accomplish a greater integration of ILS and teacher-directed activity, ILS use should incorporate several features: some teacher-directed lessons followed by whole-class (not individualized) computer use, teacher-led remediation of small groups based on system-supplied information making the formation of such groups easily done, and the establishment of heterogeneous student teams that provide motivation for accomplishment and for peer assistance.     

More or better student teaching?

Drawing from data on over 1000 prospective teachers in a large urban district including pre and post-student teaching survey data, this study investigates whether lengthening student teaching improves teachers’ perceptions of instructional preparedness, efficacy, and career plans. The findings suggest that the duration of student teaching has little effect on teacher outcomes; however, the quality of student teaching has significant and positive effects. Moreover, the magnitude of the effects of student teaching quality are greater when student teaching is shorter and in schools with more historically underserved racial groups. The authors discuss policy implications and directions for further research.     

Two decades of use in K-12 education

Conclusions ILSs are here to stay. School boards, administrators and other decision makers interested in what they perceive as a way of preparing their students to do well on standardized tests will continue to drive the purchases of ILSs upward each year. If ILSs are to become more effective for the rapidly growing number of students from diverse backgrounds (including many students of color) who will spend months and years using them perhaps it is time for ILS companies to consider changes in their approach to design as opposed to “just tweaking” the current design. A good starting point for changes in the approach to the design of ILSs would be to consider the importance of the awareness of cultural diversity in the design and to attempt to integrate cooperative learning into the design as well. If ILS companies seriously consider these suggestions they can create a system for the next two decades that has the potential to be more effective for the large numbers of students from diverse backgrounds who are the main users of these systems. His research interests include cooperative learning, instructional technology, and collaborative learning environments.     

Agents control in intelligent learning systems: The case of reactive characteristics

Intelligent learning systems (ILSs) have evolved in the last few years basically because of influences received from multi-agent architectures (MAs). Conflict resolution among agents has been a very important problem for multi-agent systems, with specific features in the case of ILSs. The literature shows that ILSs with cognitive or pedagogical agents are prone to arbitration methods, where as ILSs with reactive agents are much in favor of control mechanisms. For these kind of systems, different control types are proposed based on the different stimuli that these agents will receive. These stimuli are aspects to be evaluated during the teaching/learning process such as: (1) error analysis, (2) learning styles, (3) analogies, (4) social aspects, etc.

The paper reviews several ILSs, related to our work; different control mechanisms are proposed to solve the agents' intervention conflicts. Finally, the use of several mechanisms is exemplified by the results of a specific ILS.     

High-stakes teacher evaluation policy: US principals’ perspectives and variations in practice

Principals’ implementation of new teacher evaluation policies in a suburban and rural southeastern area of the United States was examined over a five-year period. This study reports findings on two of eleven interview questions examining changes in principals’ perceptions over time regarding policy concerns and benefits. Findings indicate while initially overwhelmed, principals eventually managed implementation time challenges and later focused on the benefits of evaluation. Secondly, principals quickly integrated the instructional rubric criteria into classroom observations and professional development work. Third, increasing doubts emerged regarding the inconsistent application of the rubric criteria, the inclusion of student test scores in teacher evaluation, and the calculation of teacher effectiveness ratings. The authors conclude that mandating rigorous evaluation policy will not sufficiently address teacher effectiveness and may complicate principals’ instructional leadership. They assert that policy-makers must consider the long-term effects of implementation before substantial teacher evaluation change results.     

Understanding teacher design practices for digital inquiry–based science learning: the case of Go-Lab

Designing and implementing online or digital learning material is a demanding task for teachers. This is even more the case when this material is used for more engaged forms of learning, such as inquiry learning. In this article, we give an informed account of Go-Lab, an ecosystem that supports teachers in creating Inquiry Learning Spaces (ILSs). These ILSs are built around STEM–related online laboratories. Within the Go-Lab ecosystem, teachers can combine these online laboratories with multimedia material and learning apps, which are small applications that support learners in their inquiry learning process. The Go-Lab ecosystem offers teachers ready–made structures, such as a standard inquiry cycle, alternative scenarios or complete ILSs that can be used as they are, but it also allows teachers to configure these structures to create personalized ILSs. For this article, we analyzed data on the design process and structure of 2414 ILSs that were (co)created by teachers and that our usage data suggest have been used in classrooms. Our data show that teachers prefer to start their design from empty templates instead of more domain–related elements, that the makeup of the design team (a single teacher, a group of collaborating teachers, or a mix of teachers and project members) influences key design process characteristics such as time spent designing the ILS and number of actions involved, that the characteristics of the resulting ILSs also depend on the type of design team and that ILSs that are openly shared (i.e., published in a public repository) have different characteristics than those that are kept private.

     

“Don’t Be a Sucker!”-the Mathematics of Gambling

Although the Internet has been touted as a boon to social studies education, little research exists that documents the impact of using it as a contextualizing tool for analyzing primary sources and developing Web sites on high school students' achievement. In this article, the authors address both issues by using a quasi-experimental design to measure impact on student achievement. However, there was no appreciable difference in student achievement, and the provided assignment and amount of content that students were expected to learn in the unit of study may have hindered higher-order thinking. The authors discuss the implications of these findings.     

School based management effects: Resources or governance change? Evidence from Mexico

In their struggle to improve student learning, many developing countries are introducing school-based management (SBM) reforms that provide cash-grants to school councils. School councils are expected to work collaboratively and decide on the best use of the funds. In this paper, we study the effects of one such program in Mexico on student outcomes. We complement the differences-in-differences analysis by qualitatively exploring program implementation. Results suggest the program had substantial positive effects on third grade Spanish test scores, with most benefits accruing to schools receiving SBM cash grants for the first time. These results are robust to alternative model specifications. The implementation analysis suggests school councils did monitor grant use, but parental participation did not significantly improve in other areas. Our findings suggest that the observed positive program effects are likely to be the result of providing schools with financial resources to meet pressing equipment, material, and infrastructure needs.     

The efficacy of training cooperating teachers

This study investigated outcomes of a clinical faculty training program designed to prepare cooperating teachers for supervising pre-service teachers. Drawing on multiple data sources from more than a decade of implementation, researchers investigated initial outcomes of the program for cooperating teachers, student teachers, and new teachers. Findings suggest that the training resulted in a greater sense of efficacy for aspects of the role and may lead to more effective evaluation practices by clinical faculty and to stronger performances by student teachers. The lack of other significant results may have implications for policies related to the evaluation of teacher preparation programs.     

Can you increase teacher engagement with evaluation simply by improving the evaluation system?

We know various factors can influence how teaching staff engage with student evaluation, such as institutional policies or staff beliefs. However, little research has investigated the influence of the technical processes of an evaluation system. In this article, we present a case study of the effects of changing the technical system for administering student evaluations at one New Zealand university. We develop a socio-technical model of the institutional evaluation system, and use this model to examine whether introducing an online system for ordering student feedback questionnaires and reducing processing time influenced academic staff engagement with evaluation. Survey responses, interview comments and data about ordering trends suggest the change did increase staff engagement by: (1) improving staff perceptions of evaluation and (2) increasing engaged behaviour, such as voluntarily ordering more evaluations. The outcomes of this study imply that the ‘practical implementation’ of an evaluation system is an important factor in influencing engagement with evaluation. We conclude that we can increase teacher engagement with evaluation simply by improving the ‘practical implementation’ of the evaluation system.     

The Impact of Management Education

This article summarizes research findings of a study that examined the effects of a sales management training program conducted at Digital Equipment Corporation. The authors used a typology of training outcomes developed by Kirkpatrick (1959a, 1959b, 1960a, and 1960b) to assess students' satisfaction, learning, and field implementation of skills taught in the class. In addition, they estimated the costs of the program and the actual increase in sales that can be attributed to the implementation of the skills in the workplace. The results indicate that properly conceived and carefully implemented programs have a positive net effect on the corporation's financial position.     

Student perceptions of assessment and student self-efficacy in competence-based education

The purpose of this study was to provide insight into the interplay between student perceptions of competence-based assessment and student self-efficacy, and how this influences student learning outcomes. Results reveal that student perceptions of the form authenticity aspect and the quality feedback aspect of assessment do predict student self-efficacy, confirming the role of mastery experiences and social persuasions in enhancing student self-efficacy as stated by social cognitive theory. Findings do not confirm mastery experiences as being a stronger source of self-efficacy information than social persuasions. Study results confirm the predictive role of students’ self-efficacy on their competence outcomes. Mediation analysis results indicate that student’s perceptions of assessment have an indirect effect on student’s competence evaluation outcomes through student’s self-efficacy. Study findings highlight which assessment characteristics, positively influencing students’ learning, contribute to the effectiveness of competence-based education. Limitations of the study and directions for future research are indicated.     

Within the teacher evaluation policies black box: two case studies

This paper analyzes the relationship between teacher evaluation policies in Chile and Uruguay, teacher professional development, and teacher performance effectiveness. The analysis tries to understand the extent to which the existing processes for teacher evaluation stimulate an improvement in the quality of teaching. The comparison of two national cases with similar educational results but radically different educational policies allows the authors to take a close look within the ‘black box’ of educational policy-making. The paper is based on in-depth analysis of the PISA survey to heads of schools and in-depth interviews with teachers, heads, and key informants. The paper finds that policies aiming to strengthen teacher development and teacher performance through teacher evaluation cannot be understood in dichotomist ways; lack of attention to teachers’ perspectives and unintended effects of policy implementation can lead to marginal effects. Uruguay, where teacher evaluation policies have weak practical effects, has managed to obtain similar educational results to Chile. However, the experience of Chile shows that teacher evaluation can have an effect on teacher professional development and, in the long run, this may have an effect on the quality of teaching and student results.     

Improving the academic outcomes of undergraduate ESL students: the case for discipline‐based academic skills programs

This article examines the impact of an adjunct English language tutorial program on the academic performance of first‐year Architecture students. With increasing numbers of international students enrolling in Australian universities, language and academic skills programs have shifted from generic to more discipline‐specific models; however, there has been little research on the effects of these programs on ESL students' academic performance. This study uses a mixed‐method approach to evaluate the effectiveness of an English as a second language (ESL) intervention program. Findings indicate that students value a highly discipline‐specific approach to language and academic skills support and that regular participation in a content‐based ESL program can lead to positive learning outcomes for students. Preliminary findings also suggest that there are positive longer‐term benefits on students' academic outcomes. The authors suggest the need for longitudinal studies on the impact of discipline‐based programs for ESL students.     

New Directions in the Evaluation of the Effectiveness of Educational Technology

Summary

Drawing from work by Shadish, Cook, and Leviton (1991) on social program evaluation, the authors discuss recent changes in evaluation theory and practices, and they connect these changes to technology and student learning. Concluding with a list of recommendations for evaluating the effectiveness of technology in teaching and learning, the authors challenge the purposes of education and prevalent goals for evaluation. After questioning how technology can impact student learning, they call for new and expanded definitions of student learning outcomes. Recommendations include redefining technology as a process rather than as a product, conducting implementation evaluations prior to outcome evaluations; reducing the reliance on standardized test scores as the primary outcome measure; and adopting multifaceted evaluation approaches (including case studies).     

Contingent Instructors and Student Outcomes: An Artifact or a Fact?

This study addresses methodological problems surrounding existing research on exposure to contingent instructors and student outcomes. By applying non-aggregated and aggregated measures of exposure to contingent instructors to the same data, this analysis demonstrates that effects of commonly used measures of exposure to contingent instructors have little to do with actual contingent instructor effects on student outcomes. Two multi-level approaches—cross-classified and multiple membership models—are applied in the single-institution analysis of faculty status effect on student outcomes—grades and first year retention. The analysis showed no variability in student retention and a significant variability in grades by faculty characteristics. Compared to their tenured and tenure-track peers, contingent instructors are more likely to assign higher grades, which may lead to lowered levels of academic challenge and student motivation to do their best work.