Guglielmino's self-directed learning readiness scale: A validation study

The attributes of self-direction in learning are becoming increasingly important as the need for lifelong learning grows in strength. Educators are challenged to assist in the development of self-directed learning skills and to encourage learners to more freely use self-direction in their learning activities. Unfortunately, there are few validated procedures for identifying the self-directed learners. Guglielmino's Self-Directed Learning Readiness Scale is one of the few instruments identified in the literature for the purpose of measuring self-direction in learning. Even though the scale has been widely used, additional validation is needed. This study was designed to use a multitrait-multimethod procedure for determining the validity of the SDLRS. The sample included 136 college students from two different colleges: 63 black students, 70 white students and 3 students of other nationalities (other than USA). Thirty-seven specific hypotheses were tested. Findings concerning selected hypotheses are discussed. Three general conclusions concerning the validity of the SDLRS are as follows: (1) The findings are supportive of the validity of the SDLRS; (2) Significant differences were noted in faculty ratings according to racial composition and student scores on the SDLRS; (3) Significant associations exist between the SDLRS scores and variables such as age, educational level and ARS (agreement response set).     

An Evaluation of Factors Regarding Students’ Assessment of Faculty in a Business School*

Student faculty ratings are used at most institutions of higher learning for three important reasons. First, the ratings provide direct feedback to the faculty, and this enables faculty to adjust their teaching styles. Second, the ratings provide the administration with information intended to assist in guiding and mentoring faculty toward more effective pedagogical performance in the classroom. Third, the ratings also provide the administration with information to be used in the reappointment, tenure, and promotion processes, as well as for assignment of salary range adjustments and teaching awards. To be of real value, however, all of this is predicated on the use of a valid and reliable faculty‐rating instrument along with a system designed to provide both the faculty and the administration with norming reports that allow for appropriate comparisons of ratings. This article reports such a study conducted within a large department of a business school and recommends that the process used be adapted by other business school departments and other academic units across the university and at other universities to ensure a more universally appropriate usage of students’ ratings.     

Assessing student exposure to and use of computer technologies through an examination of course syllabi

A syllabus analysis instrument was developed to assist program evaluators, administrators and faculty in the identification of skills that students use as they complete their college coursework. While this instrument can be tailored for use with a variety of learning domains, we used it to assess students' use of and exposure to computer technology skills. The reliability and validity of the instrument was examined through an analysis of 88 syllabi from courses within the teacher education program and the core curriculum at a private Midwest US university. Results indicate that the instrument has good inter‐rater reliability and ratings by and interviews with faculty and students provide evidence of construct validity. The use and limitations of the instrument in educational program evaluation are discussed.     

The Gifted Rating Scales-School Form: A Validation Study Based on Age,Gender, and Race

This study examined the internal consistency and validity of a new rating scale to identify gifted students, the Gifted Rating Scales-School Form (GRS-S). The study explored the effect of gender, race/ethnicity, age, and rater familiarity on GRS-S ratings. One hundred twenty-two students in first to eighth grade from elementary and middle schools in the southeastern United States participated in the investigation. Results indicated high internal consistency for the six GRS-S scales: Intellectual Ability, Academic Ability, Creativity, Artistic Talent, Leadership, and Motivation. Results revealed no effect of race/ethnicity, age, or rater familiarity with the student. There was no significant effect for gender, although a trend was noted for girls rated slightly higher than boys across all scales. This trend was consistent with analyses of the standardization data and with cross-cultural findings using translated versions of the GRS-S. The present findings provided support for the GRS-S as a valid gifted screening instrument.     

Assessment of attention in school children: teachers' ratings related to tests of attention

The paper provides (1) a teacher-administered rating instrument for inattention without confounding the rating with hyperactivity and conduct disorder, and (2) evidence that the ratings correlate with the scores obtained from cognitive tests of attention. In Study I, the first objective was to investigate the construct validity and the inter-rater reliability of the Attention Checklist (ACL) by factor analysing the teacher ratings of 110 Grade 4 children, obtained by using the ACL. The second objective was to investigate the predictive validity of the ACL by examining the relationship between the scores obtained for the participants from teachers' ratings using the ACL and the scores obtained by participants in the lab-type attention tests. The results of factor analysis showed that a single factor labelled ‘inattention’ underlies the 12 items in the ACL. Examining the differences in performance on attention tests, the ‘low attention’ children as rated by the teachers on the ACL scored lower than the ‘high attention’ children on the objective tests of attention. These findings were replicated in Study II, which was conducted to test further the construct validity and predictive validity of the ACL. This time, only those two tests (Auditory Attention and Visual Attention) that had shown relatively poor discrimination between the high and low attention groups in Study I were, again, administered to another cohort of 97 Grade 4 children, as it was our intention to further challenge the reliability of the ACL. Overall, the results of both studies suggest that comprehensive assessment of attention skills should include both ACL and objective measures of selective attention.     

Depressive symptomatology in learning-disabled and nonlearning-disabled students

The present study examined the level of depressive symptomatology in a sample of 100 learning-disabled and nonlearning-disabled fourth-, fifth-, and sixth-grade students. Depressive symptomatology was assessed using the Children's Depression Inventory (CDI). In addition, students were rated by their classroom teachers on behaviors related to depression based on the DSM-III criteria. It was found that students identified as learning disabled obtained significantly higher scores on the CDI and the behavior rating scale than did children in the regular education classroom. There were no significant differences among grade levels for CDI scores, but a trend was noted. A significant difference was found among grade levels for the behavior rating scale scores. There was a significant positive correlation between CDI scores and teacher ratings of depressive characteristics.     

Different administrative directions and student ratings of instruction: Cognitive versus affective effects

A manipulation of the instructions students received prior to completing the 7-item Endeavor Instructional Rating card differentially affected their ratings on two types of items. Specifically, when students were led to believe their ratings would have a strong impact on the instructor's career, they tended to be more lenient on items measuring rapport (i.e., the affective domain); this same effect was not observed for items measuring pedagogical skill (i.e., the cognitive domain). The different items on our instructional rating instrument appear to be measuring different things. One implication of this observation is that the inconsistent findings reported in past research on student ratings of instruction may be due to the differential mix of items from one instrument to another. When instructors are compared on ratings given them by students, unbiased interpretation requires that the multidimensional nature of teaching (and of the rating instrument) be considered.     

Assessing Enhancement of Learning, Personal Learning Environment, and Student Efficacy: Alternatives to Traditional Faculty Evaluation in Higher Education

This study was informed by three bodies of important literature: (1) research on faculty teaching performance and course evaluation in higher education, (2) research on students' personal (constructivist-based) views of characteristics of teaching and learning environments, and (3) human efficacy. The article describes the development and validation of a new measure designed to assess students' perceptions of the extent to which higher education learning environment characteristics enhance students' personal learning. Results of factor analyses of the Student Assessment of Teaching and Learning (SATL) and criterion-related validity analyses are reported for 2,190 students in 145 separate classes. Criteria used for analysis are (1) measures of students' perceptions of constructivist-based, personal learning environments and students' self-efficacy beliefs, (2) course emphasis on higher-order thinking skills and personal and applied knowledge, and (3) two summative judgments of overall course quality. Implications of the findings are discussed in light of using student rating information for formative evaluation purposes and using students' reflections on personal learning as an element of assessing the quality of teaching and learning in higher education settings. This latter view is contrasted to procedures that are more traditionally used to evaluate faculty, teaching, and course characteristics.     

Building Validity Evidence for Scores on a State-Wide Alternate Assessment: A Contrasting Groups, Multimethod Approach

The relationships between ratings on the Idaho Alternate Assessment (IAA) for 116 students with significant disabilities and corresponding ratings for the same students on two norm-referenced teacher rating scales were examined to gain evidence about the validity of resulting IAA scores. To contextualize these findings, another group of 54 students who had disabilities, but were not officially eligible for the alternate assessment also was assessed. Evidence to support the validity of the inferences about IAA scores was mixed, yet promising. Specifically, the relationship among the reading, language arts, and mathematics achievement level ratings on the IAA and the concurrent scores on the ACES-Academic Skills scales for the eligible students varied across grade clusters, but in general were moderate. These findings provided evidence that IAA scales measure skills indicative of the state's content standards. This point was further reinforced by moderate to high correlations between the IAA and Idaho State Achievement Test (ISAT) for the not eligible students. Additional evidence concerning the valid use of the IAA was provided by logistic regression results that the scores do an excellent job of differentiating students who were eligible from those not eligible to participate in an alternate assessment. The collective evidence for the validity of the IAA scores suggests it is a promising assessment for NCLB accountability of students with significant disabilities. The methods of establishing this evidence have the potential to advance validation efforts of other states' alternate assessments.     

Student evaluation of instruction in higher education: exploring issues of validity and reliability

Many personnel committees at colleges and universities in the USA use student evaluation of faculty instruction to make decisions regarding tenure, promotion, merit pay or faculty professional development. This study examines the construct validity and internal consistency reliability of the student evaluation of instruction (SEI) used at a large mid‐western university in the USA for both administrative and instructional purposes. The sample consisted of 73,500 completed SEIs for undergraduate students who self‐reported as freshman, sophomore, junior or senior. Confirmatory factor analysis via structural equation modelling was used to explore the construct validity of the SEI instrument. The internal consistency of students' ratings was reported to provide reliability evidence. The results of this study showed that the model fits the data for the sample. The significance of this study as well as areas for further research are discussed.     

Using behavioral and academic indicators in the classroom to screen for at‐risk status

The present study validated a brief at‐risk screening instrument designed for easy use by teachers in the elementary school. School performance measures were collected for students in first to sixth grade one year following initial teacher ratings using the Screening For At‐Risk Status screening instrument. Findings indicated that the instrument is best seen as measuring a single at‐risk construct with items drawn from three domains: academic skills, social confidence, and social cooperation. Correlations between at‐risk scores and school performance measures taken one year later demonstrated predictive validity. The screening instrument correctly identified at‐risk students with 88% accuracy and not‐at‐risk students with 74% accuracy. There were 12% false negatives. Use of the instrument provides teachers with a quick, easy screening of students who may develop difficulties in the future. For schools, the screening can be used as the first step in a supportive response system to assist at‐risk students from developing serious school difficulties and possibly failure in the longer term. © 2005 Wiley Periodicals, Inc. Psychol Schs 42: 151–158, 2005.     

Faculty ratings of course evaluation items

Instructors whose teaching was evaluated by students were given the opportunity to rate how applicable the evaluation items were to their classes. This study examined the kinds of items which instructors felt to be applicable or inapplicable, the relationships between the student ratings and the instructor applicability ratings, and the effect on an overall evaluation score of using the instructor applicability judgments as weights.Results generally support the consensus procedure of establishing rating forms; they suggest that the common criticism that faculty judgments of item applicability are influenced by anticipation of student ratings may be true for specific items and that while weighting composite evaluation scores by means of faculty applicability judgments does not affect those overall scores, the distributions of certain items may be altered.     

Self-directed learning readiness and performance in the workplace

Deep and rapid changes in a high-tech society exceed the ability of formal educational systems to respond to new learning needs. Therefore, the future of post secondary education will increasingly include emphases on self-directed learning skills and activities. Despite recognition of such a trend, knowledge of self-directed learning and self-directed learners is in its infancy.This study of self-directed learning and performance was designed to determine the relationship of self-directed learning readiness and performance in an occupational setting. Business, government and industry are expected to depend heavily upon self-directed learning in the future. Because of the increasingly close relationship between education and these sectors of the economy, educators should have a keen interest in this kind of research.The sample included 753 individuals selected from the population of a large American utility company: 421 were managers and 318 were non-managers; 14 subjects did not define their occupational status. Subjects provided information on their management level, sex, age, education, race, job performance rating, creativity required on their jobs, problem-solving ability needed in their jobs and degree of change in their work. Nine hypotheses were tested at the 0.05 level of significance.Selected conclusions are as follows: (a) outstanding performers in jobs requiring a very high level of creativity had significantly higher SDLRS scores than others; (b) outstanding performers in jobs requiring a very high degree of problem-solving skill had significantly higher SDLRS scores than others; and (c) individuals who have completed higher levels of education tend to have higher SDLRS scores.     

A diagnostic assessment for introductory molecular and cell biology

We have developed and validated a tool for assessing understanding of a selection of fundamental concepts and basic knowledge in undergraduate introductory molecular and cell biology, focusing on areas in which students often have misconceptions. This multiple-choice Introductory Molecular and Cell Biology Assessment (IMCA) instrument is designed for use as a pre- and posttest to measure student learning gains. To develop the assessment, we first worked with faculty to create a set of learning goals that targeted important concepts in the field and seemed likely to be emphasized by most instructors teaching these subjects. We interviewed students using open-ended questions to identify commonly held misconceptions, formulated multiple-choice questions that included these ideas as distracters, and reinterviewed students to establish validity of the instrument. The assessment was then evaluated by 25 biology experts and modified based on their suggestions. The complete revised assessment was administered to more than 1300 students at three institutions. Analysis of statistical parameters including item difficulty, item discrimination, and reliability provides evidence that the IMCA is a valid and reliable instrument with several potential uses in gauging student learning of key concepts in molecular and cell biology.     

The instructor self-evaluation form: Development and validation of an ipsative forced-choice measure of self-perceived faculty performance

An ipsative forced-choice college faculty self-evaluation questionnaire was developed. The instrument was validated by comparing self-evaluation scores to student rating scores and by comparing superior and nonsuperior instructors on the basis of student ratings. Both analyses yielded positive, but modest, indications of validity. Implications are discussed relative to the questionnaire's utility in faculty development activities.     

Effects of grading practices and time of rating on student ratings of faculty performance and student learning

The effects of grading practices (strict, lenient) and time of rating (after a lecture, after taking an examination, after receiving feedback on performance on the examination) on student ratings of faculty performance and student learning (i.e., performance on an examination) were assessed in two experimental studies. Results indicated that (1) student ratings were directly affected by grading practices and this effect was a function of the time at which ratings were cornpleted, and (2) student learning and students' ratings of faculty were positively correlated, and grading practices had no effect on student learning. Implications of these findings for the administrative use of student ratings are considered.     

Student learning in higher education: a longitudinal analysis and faculty discussion

Answering a call put forth decades ago by the higher education community and the federal government, we investigated the impact of US college coursework on student learning gains. Students gained, on average, 3.72 points on a 66-item test of quantitative and scientific reasoning after experiencing 1.5 years of college. Gain scores were unrelated to the number of quantitative and scientific reasoning courses completed when controlling and not controlling for students’ personal characteristics. Unexpectedly, yet fortunately, gain scores showed no discernable difference when corrected for low test-taking effort, which indicated test-taking effort did not compromise the validity of the test scores. When gain scores were disaggregated by amount of completed coursework, the estimated gain scores of students with quantitative and scientific reasoning coursework were smaller than what quantitative and scientific reasoning faculty expected or desired. In sum, although students appear on average to be making gains in quantitative and scientific reasoning, there is not a strong relationship between learning gains and students’ quantitative and scientific reasoning coursework, and the gains are less than desired by faculty. We discuss implications of these findings for student learning assessment and learning improvement processes.     

Observing justice in the primary school classroom

Justice-related situations are a part of students? everyday life. In order to test the antecedents, correlates, and consequences of (in)justice in school, valid measures of justice are needed. To our knowledge, this is the first study to develop an observer low inference rating instrument that can be applied to measure justice in the primary classroom. In two pre-studies, justice-relevant situations in the classroom were extracted and observable indicators for these situations were developed. In the main study, this instrument was used to observe 208 primary school students with regard to their experiences of justice or injustice. In addition to this, other measures of justice were developed to examine the convergence between observer low inference ratings of classroom justice and high inference rating instruments for teachers, students, and external observers.Factor analyses and correlations between the different indices of the observer low inference rating and the high inference rating items suggested that incidents of justice and injustice in the classroom do not tend to co-occur frequently. Teachers do not appear to have a general tendency to treat a child more or less justly across a large number of situations.The findings suggest that a comprehensive assessment of classroom justice requires a multi-method approach where the justice ratings of students, teachers and external observers are all taken into consideration.     

Faculty evaluation: Reliability of peer assessments of research,teaching, and service

In this paper, assessments of faculty performance for the determination of salary increases are analyzed to estimate interrater reliability. Using the independent ratings by six elected members of the faculty, correlations between the ratings are calculated and estimates of the reliability of the composite (group) ratings are generated. Average intercorrelations are found to range from 0.603 for teaching, to 0.850 for research. The average intercorrelation for the overall faculty ratings is 0.794. Using these correlations, the reliability of the six-person group (the composite reliability) is estimated to be over 0.900 for each of the three areas and 0.959 for the overall faculty rating. Furthermore, little correlation is found between the ratings of performance levels of individual faculty members in the three areas of research, teaching, and service. The high intercorrelations and, consequently, the high composite reliabilities suggest that a reduction in the number of raters would have relatively small effects on reliability. The findings are discussed in terms of their relationship to issues of validity as well as to other questions of faculty assessment.     

Measuring graduate students’ global competence: Instrument development and an empirical study with a Chinese sample

In globalization, global competence (GC) is a crucial competence for graduate students to possess; thus, graduate education should prepare students with GC to compete globally. However, no instrument has been designed to measure graduate students' GC, and the theoretical structure of GC has not been empirically examined. To fill these gaps, first, we developed the Global Competence Scale for graduate students (GCSG) based on a three-dimensional theoretical framework (knowledge, skills, and attitudes). Second, we administered the GCSG to Chinese graduate students sampled from five universities in Beijing. Third, we examined the theoretical framework, and examined the reliability and validity of the scale. Finally, we described the Chinese graduate student sample’s GC by using the instrument. The results supported the theoretical model and provided evidence for the reliability and validity of the instrument. We also found that the sample showed higher ratings in knowledge and attitudes but lower ratings in communication skills.