On an Approach to Testing and Modeling Competence

E. L. Thorndike contributed significantly to the field of educational and psychological testing as well as more broadly to psychological studies in education. This article follows in his testing legacy. I address the escalating demand, across societal sectors, to measure individual and group competencies. In formulating an approach to measuring competence, I draw on measurement research I have done over my career; the Thorndike lecture is to be as much autobiographical as substantive and/or methodological. I present an approach to defining, measuring, and statistically modeling competency measurements. The article unpacks Hartig et al.'s (2008) definition of competence as a complex ability construct closely related to real-life-situated performance. The intent is to make the construct, competence, amenable to measurement. Once unpacked, criteria for building competence measurements are set forth and exemplified by research from business, military, and education sectors. Generalizability theory, a statistical theory for modeling and evaluating the dependability of competence scores, is applied to several of these examples. The article then pulls together the threads into a general competency measurement model and concludes by noting its limitations.     

Intelligence Recreated

Many of us are frustrated with the overuse of intelligence tests. But intelligence tests have become so entrenched in our society that it is hard to imagine how they realistically could be replaced. Schools would be without a well-established screening device, and intelligence research would be without an external measure of validity. But what if we started over and imagined thinking about intelligence without the benefit (some would say hindrance) of Binet? What would theories and tests of intelligence look like? The articles in this special issue address this topic; here, I discuss the articles. The discussion is divided into three sections. The first section deals with definitional issues: How can intelligence be operationally defined, and can a single definition capture cognitive abilities of individuals at all ages? The second section briefly summarizes and evaluates each of the seven theories: How intelligent are these theories of intelligence? The final section focuses on the implications of the theories and theory-based tests reported in this issue: How can future research and educational practices benefit from the views presented here?     

批判与超越:警惕标准化测验对好教育的误读——基于比斯塔教育测量思想的省思

基于证据的测量文化在教育理论和实践领域均备受追捧,测量数据被看作“客观”与“科学”的代名词。但通过专注于结果“有效”的标准化测验,教育质量真的能获得提升吗?通过比斯塔的教育测量理论发现,标准化测验在推动教育改进和绩效问责的同时,遗失了教师专业判断的价值、无法被测量数据所衡量的事物、教育本身弱点所保存的开放性与可能性、关涉主体存在的教育目的等方面,而后者正是好教育区别于普通测验训练的根基所在。由此得到,标准化测验无法确保带来好教育;好教育也应包含着主体以成熟的方式来到与他者共存之世界的理想;在所谓的测量时代里,警惕一味追求安全稳定的技术性测量对好教育的扼杀是非常紧迫的事。我们应追求工具性与人文性融合的好教育衡量方式。     

Color television and educational TV

Conclusions Color television is so new that there is little concrete information on the subject. However, it seems logical to doubt that color will provide any general increased effectiveness to educational TV. In some areas, such as art and medicine, the impact may be great; but for the greater proportion of educational subjects, it will not contribute a great deal. In view of this, together with the present high cost of color TV, it would seem foolish for educational TV to delay development because color television is coming. The transition to color can always be made after the piomeering is done, when costs have gone down, when color receiver circulation is large enough to make color worth while, and when the numerous problems of color programming and production have been solved. What effect will color TV have upon educational TV programs? How expensive will color TV be for educational use? How will educational TV stations use color TV? What competition will commercial color TV offer to educational black and white TV stations? These questions are answered and an overview of the implications of color for educational TV is presented. E. G. Sherburne, Jr., was formerly Television Coordinator at the U. S. Navy Special Devices Center, Port Washington, Long Island, New York.     

Self-efficacy and behavioral intention: A mediational analysis of the effects of commitment on career counseling

This article examines two questions of research. Can we make educational and vocational information sessions effective by applying commitment theory (Kiesler, 1971) to this particular aspect of career counseling? Does commitment as part of an initial low-cost behavior (taking part in an information session) have a direct effect or is the effect a mediate one? Data gathered during an experiment involving six sixth-year secondary school classes would appear to suggest that there is a mediate process. To be more precise, the accepted impact of commitment upon the effectiveness of information sessions (postsession information and orientation is more actively sought) would seem to be linked to its mediation by perceived self-efficacy (Bandura, 1997/2003) in orientation. On the other hand, it is not mediated by the behavioral intention (Fishbein, 1980) to take such steps, this being an intention upon which commitment has no effect. The discussion links these results to some of Kiesler's initial questions (commitment as a process of internal self-attribution), and also places them within the framework (proper to counseling) of factors common to all intervention theories and techniques designed to explain the effectiveness of said interventions.     

Tests,Test Scores,and Constructs

In the service of educational accountability, student achievement tests are being used to measure constructs quite unlike those envisioned by test developers. Scores are compared to cut points to create classifications like “proficient”; scores are combined over time to measure growth; student scores are aggregated to measure the effectiveness of teachers, schools, and school districts; indices are created to measure college and career readiness. These and other new uses rely on derived scores created to measure new constructs. The field of educational and psychological measurement has largely ignored these significant, consequential measurement applications. The conceptual frameworks and analytical tools of educational and psychological measurement should be used to study such derived scores and the validity of their uses and interpretations.     

IQ GAINS AND THE BINET DECREMENTS

Thorndike compiled Stanford-Binet data which made it appear that children aged 6 and under have made greater IQ gains than older children and that this pattern dominated the whole period from 1932 to 1971–72. Therefore, he sought causal factors likely to affect preschoolers more than others, for example, TV in general and educational TV in particular. A wide array of data show that the atypical gains of young children are either an artifact of sampling error or totally antedate 1947, ruling out TV as an age-specific factor. This data also suggest that Americans have gained about 12 IQ points from 1932 to 1972 with verbal gains being a point lower and performance gains a point higher.     

BOOK REVIEW

Critical reviews from “outside”, notably educational sociologists arguing mainly from a British context, have caused some ripples, and maybe even waves, among school effectiveness researchers. To a large extent these external criticisms and the overall nature of the response from school effectiveness researchers are neatly summarized in the following quote from Townsend: “be like us”, say the critics, and the answer is “no thanks”. In this article the arguments form the “external” critics and the response from school effectiveness researchers will not be repeated. Some of the topics in the debate will be revisited, however. The first one is the discussion with respect to the impact of “contextual” or composition effects concerning the average socioeconomic background of students in schools and classrooms. This is one area, that bears upon the foundations of the school effectiveness concept, although it is not, as the critics would have it, a neglected area. The second one concerns conceptualization and theoretical explanation of school effectiveness, as the debate may not have been sufficiently explicit on this issue. The rest of the paper deals with “foundational” issues in school effectiveness research that have not been settled decisively and with changes in perspectives on learning and instruction and educational technology that provide serious challenges. On these issues there is reason for self-criticism and realism in the way school effectiveness research can make progress.     

A historical overview of educational psychology

A historical review of educational psychology might improve understanding of current status and problems. Before American development of the field, the Greeks, Juan Vives in the 16th century, and Pestalozzi and Herbart in the 19th century applied psychological views to educational problems. Despite the absence of scientific psychology in 19th century America, normal schools offered courses purporting to be psychological in content; these courses were supported by textbooks on the subject. James began, and Thorndike developed, the discipline that by the mid-1920s had assumed much of its current form. Journals and academic departments appeared in the first quarter of the century also. The continuing search through the middle of the century for a satisfying professional organization reflected educational psychology's difficulties in establishing an identity. It is not clear even today that the field has “crystallized,” as one writer described it in the 1920s.     

Some Thoughts on Educational Testing: Measurement Policy Issues Into the Next Millennium

What are key issues that educational measurement professionals must confront in the next decade? What resources can we draw on as we face these issues? How can educational measurement contribute most effectively to educational reform?     

“测验连接”概念框架演变述评

测验连接是心理与教育测量研究中一个重要的领域,是通过统计方法将一个测验的分数以另一个测验的分数单位表示,或者将两个测验的分数表示在共同的分数量尺上。虽然测验连接有较长的研究历史,但是不同学者对其有不同分类。其中有些分类术语别无二致,但其定义却大相径庭,这使研究者和实践者产生了极大混乱。鉴于此,有必要从历史的角度梳理连接的概念框架及其变化,以便更好地理解和应用测验连接。     

The Future of Nation-Wide Testing in the Visual Arts

Final examinations and compulsory testing are powerful means to support educational reforms. In order to be really supportive, these tests have to do justice to both the intentions of the reform and the characteristics of the discipline. In Dutch educational reform titled ‘Basic education’ the classroom teaching should give more attention to skills and abilities and less to knowledge in itself. Although the visual arts have always been skill-oriented, the introduction of compulsory testing of practical work has raised new issues. Is it possible to compare students’ work when the assessment is made by the students’ teachers only? Here the idea is presented to have collections published as part of the test that exemplify the visual possibilities and the levels of accomplishment.     

浅谈高等教育教学改革与社会发展

高校作为为社会提供教育服务的机构，它服务于社会并带动社会的进步与发展。本文提出高校提供的教育服务应包含通才教育和发展教育才能更好地推动社会的发展，体现教育的主动性，并让服务职能归位，教师再培训以提高服务质量；给被服务者提供多种学习方式和学习环境；给学生提供施展才华的时空，重新设定教学效果检验标准等教学改革。     

Educational Testing in Poland

What has been the history of educational assessment in Poland? How are the social system, school system, examination forms, measurement theory, and attitudes toward tests related to each other? How are recent efforts at democratization of political systems likely to influence educational testing in Poland?     

The public understanding of measurement inaccuracy

Assessment agencies are increasingly facing pressure on two fronts; first, to increase transparency and openness and second, to improve public confidence. Yet, in relation to one of the central concepts of educational measurement—inherent error—many believe that increased public understanding is incompatible with public confidence: a general recognition of the true nature and extent of measurement inaccuracy would fatally undermine trust in the system. The present article is premised on a contrary proposal: not understanding measurement inaccuracy is a far greater threat than understanding it, since it will result in the system repeatedly being held to account for more than it can possibly deliver. As unrealistic expectations are unmet, so the system will appear to have failed; and this recurrent process will gradually erode public confidence. The article develops ethical and practical arguments in favour of educating the public about the inherent limitations of educational measurement. Primary amongst the ethical arguments is the proposal, from contemporary validity theory, that users who fail to understand measurement inaccuracy will be ill equipped to draw valid inferences from results.     

Educational Research and the Science of Education

This paper argues that educational research has not created a science of education in the sense of amassing the body of knowledge, skills, and values that, once acquired, would certify a candidate as a professional teacher. Educational research has not developed a science of education due to: the rigid boundaries between disciplines, the masculinization of research and the feminization of teaching, and the legacy of the grandmasters, Thorndike, James, and Dewey. The contemporary acceptance of interdisciplinary work, the occupational liberation of women and the advent of computers, and the availability of senior investigators in many different disciplines may make it a propitious time to create a true science of education.     

Comparing Futures or Comparing Pasts?

This article, firstly and briefly, suggests that there is no single or unified 'comparative education' but that there are multiple comparative educations. How may such a variety of comparative educations be distinguished? Rather more importantly and secondly, what might an 'interesting' comparative education constructed in universities look like, and on what criteria would it be interesting? The specific suggestion offered here is that at least one kind of comparative education, for a decade or so, should concentrate on exploring moments of educational metamorphosis, rather than assuming that the equilibrium conditions and the dynamic linearities of development of educational systems can be predicted. Thus for the moment the correct answer to the question, how far can we learn anything of practical value from the study of foreign educational systems is: 'not a lot'. The correct question is, why have we as scholars taken that question so seriously for so long?     

The State of NCME: Remembering the Past, Looking to the Future

What are the roots of NCME? What were our initial goals and how well have we attained them? Are we drifting away from our practitioner orientation? What should be our policy on important matters in educational practice and our role in sociopolitical issues in educational measurement?     

Partisanship in Educational Research

In his empirical study of educational research, James Tooley claims to have uncovered 'partisanship' in the 'focus content and argument of educational research'. The purpose of this paper is to suggest that Tooley's study is simply one more manifestation of the failure of many educational researchers to take account of the extensive theoretical and methodological developments that have occurred over the last four decades in the political and social sciences. It is suggested that a debate about partisanship and educational research which took these developments seriously would not be a narrow debate in which researchers with right wing affinities try to score political points against researchers of the left. Instead, it would be a more intellectually rigorous and theoretically informed debate about the complex relationship between educational values and political beliefs on the one hand and research methodologies and practices on the other.     

The failure of the public schools and the free market remedy

A Summary The replacement of the existing system of publicly operated schools by a market of private ones-supported by government vouchers—would probably yield mixed results. On the one hand, some parents would have greater choices among schools and some schools would have to be productive in order to survive in the competitive framework. The increase in consumer choice and the resultant competition among schools would be likely to lead to greater educational benefits for many students and their families (private benefits) than those which they receive under the present monopolistic system.On the other hand, the schools are also expected to fulfill certain social functions. It is in these that a market approach to schooling is likely to yield poor results. For example, basic schooling represents the primary device for equalizing opportunities among racial and social groups. Yet, advantaged children would probably receive far better schooling under the market proposal than would disadvantaged ones, and it is likely that this disparity would lead to larger future inequalities in opportunity between the children of the middle class and those of the poor. Further, it is not clear that a set of largely autonomous schools could provide the common set of values and knowledge necessary for the functioning of a democratic society. Finally, it is likely that the market proposal would increase racial and social stratification of students among schools. Whatever the success of the market in meeting consumer preferences, it would be offset by the market's failure to satisfy the social goals of basic schooling.Fortunately, we are not limited to choosing between the traditional educational bureaucracy on the one hand or an unmitigated free market for educational services on the other. There are several ways to create competition within a public school system. Jencks, Sizer, and Coleman have suggested particular plans based upon the competitive framework, and the proposal for community schools represents a more general framework in which the competition of the market place might be used to advantage. The time is ripe to experiment with at least one of these plans for the children of the ghetto. Do we have any buyers? Henry M. Levin is a research associate with the Economic Studies Division of the Brookings Institute. He has written on economic and educational matters in the Saturday Review, the Journal of Human Resources,and other publications. He is currently organizing a Brookings conference on the community school.