Establishing Weights for Test Plans for Licensure and Certification Examinations

This study evaluated two methods for establishing weights for test plans for certification examinations. One method required a panel of experts to provide holistic judgments indicating the percentage of test questions to allocate to each content category. Weights were first obtained from individual panel members, discussed by the entire panel, and then finalized by group consensus. The other method derived weights using a statistical model. The model included ratings of task frequency and task criticality provided by a large sample of practitioners as well as information from a panel of experts concerning the linkages between specific tasks and the knowledge and skills required to perform those tasks. The study was replicated for four medical imaging specialties in the field of radiologic technology. The weights for the two methods exhibited moderate to high levels of agreement for sections of the test plans comprised of specific imaging procedures. However, there was much less agreement for those sections of the test plans that addressed more general topics. Possible reasons for the observed pattern of agreement and disagreement are considered.     

Vocabulary learning from reading: examining interactions between task and learner related variables

In the field of second and foreign language learning, how various task characteristics affect language learning has been the focus of many recent studies. Much of this research examined the relationship between task characteristics and task performance without fully taking into account learner related variables. The present study aimed to assess task complexity and sequence in relation to the learner related variables drawn from the social cognitive perspective of self-regulated learning, i.e. self-efficacy beliefs and frequency of learning strategy use, as they were applied to two versions of vocabulary learning from reading tasks. The tasks designed for the present study were based on the componential framework for second language task design. With tasks and task sequence counterbalanced, 146 first-year university students (mean age?=?18.59 years) were randomly assigned to one of four groups. Results reveal a significant effect of task sequence on vocabulary learning self-efficacy beliefs, frequency of learning strategy use and task performance, and a significant interaction effect of sequence with task complexity. Findings are discussed in terms of complex interactions between task and learner factors.     

How is knowledge generated about memory encoding strategy effectiveness?

This study evaluated how people learn about encoding strategy effectiveness in an associative memory task. Individuals studied two lists of paired associates under instructions to use either a normatively effective strategy (interactive imagery) or a normatively ineffective strategy (rote repetition) for each pair. Questionnaire ratings of imagery effectiveness increased and ratings of repetition effectiveness decreased after task experience, demonstrating new knowledge about strategy effectiveness. Cued recall confidence judgments, measuring confidence in recall accuracy, were almost perfectly correlated with actual recall and strongly correlated with postdictions—estimates of recall for each strategy. A structural regression model revealed that postdictions mediated both changes in second-list predictions and changes in strategy effectiveness ratings, implicating accurate performance estimates based on item-level monitoring as the key to updating strategy knowledge.     

Generalizing screen inferiority - does the medium,screen versus paper,affect performance even with brief tasks?

Screen inferiority in performance and metacognitive processes has been repeatedly found with text learning. Common explanations for screen inferiority relate to technological and physiological disadvantages associated with extensive reading on screen. However, recent studies point to lesser recruitment of mental effort on screen than on paper. Learning tasks involving a heavy reading burden confound technological and physiological media differences with potential media effects on recruitment of mental effort. The present study focused on media effects on effort recruitment. We examined whether screen inferiority remains even with a brief task that nevertheless requires effort recruitment. In two experiments, participants faced three short math problems that require systematic processing to solve correctly. We examined media effect on solving these problems, and the potential of disturbed perceptual fluency (i.e., disfluent versus fluent fonts) to induce effort investment. Overall, there were no performance differences between the media. However, when collecting confidence ratings, disfluency improved performance on screen and hindered it on paper. Only on paper confidence ratings were sensitive to performance differences associated with fluency, and resolution was better with the disfluent font than with the fluent font. Correspondingly, another sample reported on their preference of media for solving the problems. They expressed a clear reluctance to working on screen despite the task being brief. This preference is suggestive of reliable meta-metacognitive judgments reflecting the general lower quality of metacognitive processes on screen. The findings call for considering medium and presentation format effects on metacognitive processing when designing computerized environments, even for brief tasks.     

Increasing the Validity of Angoff Standards Through Analysis of Judge-Level Internal Consistency

The purpose of the present study was to extend past work with the Angoff method for setting standards by examining judgments at the judge level rather than the panel level. The focus was on investigating the relationship between observed Angoff standard setting judgments and empirical conditional probabilities. This relationship has been used as a measure of internal consistency by previous researchers. Results indicated that judges varied in the degree to which they were able to produce internally consistent ratings; some judges produced ratings that were highly correlated with empirical conditional probabilities and other judges’ ratings had essentially no correlation with the conditional probabilities. The results also showed that weighting procedures applied to individual judgments both increased panel-level internal consistency and produced convergence across panels.     

Urban Middle School Students’ Perceptions of the Value and Difficulty of Inquiry

Following their participation in a guided-inquiry unit, 129 seventh-graders from five diverse urban middle schools were asked about their perceptions of specific inquiry tasks, from an expectancy-value framework. Students were asked to rate the interest value, utility value, and task difficulty of (a) data collection design; (b) explanation; (c) data analysis; and (d) citing evidence for claims. The utility of all tasks was rated highly, while interest ratings were moderate. Students perceived these tasks as moderately different from their usual work, and not especially difficult. No gender differences were found in students’ ratings. Investigation tasks were rated as more interesting and useful than argumentation tasks. Students from lower SES schools found all tasks more useful and interesting than their peers in higher SES schools. Students’ justifications for their ratings suggest they valued the utility of knowing how to back up their ideas with evidence.     

Academic persistence and attributional style in fifth graders

Students' attributional styles regarding academic successes and failures were hypothesized to be moderators of persistence in academic tasks. Attributional style was assessed in 72 fifth graders using the Sydney Attribution Scale (SAS). Persistence was assessed using two behavioral measures and teacher ratings. The behavioral persistence measures involved the number of tasks attempted and time spent working on a difficult reading task and a problem-solving task. The behavioral measures were highly correlated (r = .74) but were unrelated to teacher-rated persistence. Attributional style predicted teacher-rated persistence, R² = .42, F(12, 59) = 3.6, p<.001, but did not predict any of the behavioral persistence measures. Results suggest that students' self-reported attributional styles are related to teacher judgments of persistence. The lack of agreement between teacher ratings and behavioral measures of persistence may have implications for the generalization of research findings relying on either behavioral or teacher-rated persistence measures.     

Factorial Validity of Student Ratings of Instruction

This study tested the Systematic Distortion Hypothesis by examining the factorial validity of student ratings of university teaching. Factorial validity is defined as the degree to which covariance among judged traits resembles the actual or true covariation of observable behaviors underlying these traits. Although many studies have examined the factorial validity of ratings, results are inconsistent. The present study used a more complete methodology to address some of the limitations of previous studies. Student ratings of teaching and measurements of actual teaching behaviors were obtained for 32 instructors. Student ratings were compared to frequency counts of actual teaching behaviors obtained from videotape and to students’ similarity judgments of teacher characteristics. It was found, first, that the structure of student ratings showed a moderately strong relation to the structure of actual behaviors, and a somewhat stronger relation to the structure of conceptual associations; and second, that the effects of systematic distortion were more pronounced for low-inference student ratings than for high-inference ratings.     

Reading and reading-related skills in children using cochlear implants: prospects for the influence of cued speech

We assessed the reading and reading-related skills (phonemic awareness and phonological short-term memory) of deaf children fitted with cochlear implants (CI), either exposed to cued speech early (before 2 years old) (CS+) or never (CS-). Their performance was compared to that of 2 hearing control groups, 1 matched for reading level (RL), and 1 matched for chronological age (CA). Phonemic awareness and phonological short-term memory were assessed respectively through a phonemic similarity judgment task and through a word span task measuring phonological similarity effects. To assess the use of sublexical and lexical reading procedures, children read pseudowords and irregular words aloud. Results showed that cued speech improved performance on both the phonemic awareness and the reading tasks but not on the phonological short-term memory task. In phonemic awareness and reading, CS+ children obtained accuracy and rapidity scores similar to CA controls, whereas CS- children obtained lower scores than hearing controls. Nevertheless, in phonological short-term memory task, the phonological similarity effect of both CI groups was similar. Overall, these results support the use of cued speech to improve phonemic awareness and reading skills in CI children.     

Innovations in Measuring Rater Accuracy in Standard Setting: Assessing “Fit” to Item Characteristic Curves

Standard setting methods such as the Angoff method rely on judgments of item characteristics; item response theory empirically estimates item characteristics and displays them in item characteristic curves (ICCs). This study evaluated several indexes of rater fit to ICCs as a method for judging rater accuracy in their estimates of expected item performance for target groups of test-takers. Simulated data were used to compare adequately fitting ratings to poorly fitting ratings at various target competence levels in a simulated two stage standard setting study. The indexes were then applied to a set of real ratings on 66 items evaluated at 4 competence thresholds to demonstrate their relative usefulness for gaining insight into rater “fit.” Based on analysis of both the simulated and real data, it is recommended that fit indexes based on the absolute deviations of ratings from the ICCs be used, and those based on the standard errors of ratings should be avoided. Suggestions are provided for using these indexes in future research and practice.     

基于任务型教学设计原则的高职综合英语教学设计

本文以任务型教学理论为基础,阐述任务型教学法的定义,然后根据Jane Willis的任务实施框架来设计高职综合英语任务型教学的各个步骤,即任务的导入;任务前阶段;任务中阶段;任务后阶段,同时阐述了任务型教学设计的原则及教学设计,并指出在实施过程中应注意的问题。     

Cognitive and linguistic constraints on phoneme isolation in Dutch kindergartners

This study investigated whether task instructions affect sound-isolation performance. The effects of phoneme class and phoneme position were also assessed. Two hundred Dutch kindergartners were presented with a free-sound-isolation task and its constrained counterparts: an initial-, a middle-, and a final-sound-isolation task. All tasks contained 17 CVC words. Children's performance on the free-sound-isolation task was better than on the constrained tasks. On all four tasks, children made fewer errors in isolating the initial phoneme than the final phoneme. Isolating the middle phoneme proved to be the most demanding. The effect of phoneme class depended on the type of task and on phoneme position. Findings were placed against the background of sonority and word-final phoneme vocalization in Dutch.     

Modeling Judgments in the Angoff and Contrasting-Groups Method of Standard Setting

Essential for the validity of the judgments in a standard-setting study is that they follow the implicit task assumptions. In the Angoff method, judgments are assumed to be inversely related to the difficulty of the items; contrasting-groups judgments are assumed to be positively related to the ability of the students. In the present study, judgments from both procedures were modeled with a random-effects probit regression model. The Angoff judgments showed a weaker link with the position of the items on the latent scale than the contrasting-groups judgments with the position of the students. Hence, in the specific context of the study, the contrasting-groups judgments were more aligned with the underlying assumptions of the method than the Angoff judgments .     

Transfer of judgments of control to a target stimulus and to novel stimuli through derived relations

Three experiments examined the effect of response?Coutcome contingencies on human ratings of causal efficacy and demonstrated that such ratings transfer to novel situations through derived stimulus relations. Efficacy ratings generally followed the delta probability rule when positive response-outcome contingencies were employed (Experiment 1) and when some outcomes were not contingent on participants?? responses (Experiment 2). Experiment 3 employed a negative response?Coutcome contingency and manipulated performance expectancies in the task. All three groups overestimated their causal efficacy ratings. A learned helplessness effect was observed when the response?Coutcomes were uncontrollable and in the high-expectancy group when participants?? performance in the task was worse than they had expected. In all experiments, ratings transferred to a stimulus presented during the task and often generalized to novel stimuli through derived relations. These results corroborate the view that outcome probability is a determinant of causal efficacy ratings and that schedules can be employed as UCS in procedures that share characteristics of evaluative conditioning procedures.     

Effects of performance feedback valence on perceptions of invested mental effort

We investigated whether the valence of performance feedback provided after a task, would affect participants’ perceptions of how much mental effort they invested in that same task. In three experiments, we presented participants with problem-solving tasks and manipulated the presence and valence of feedback between conditions (no, positive, or negative feedback valence), prior to asking them to rate how much mental effort they invested in solving that problem. Across the three experiments–with different problem-solving tasks and participant populations–we found that subjective ratings of effort investment were significantly higher after negative than after positive feedback; ratings given without feedback fell in between. These findings show that feedback valence alters perceived effort investment (possibly via task perceptions or affect), which can be problematic when effort is measured as an indicator of cognitive load. Therefore, it seems advisable to measure mental effort directly after each task, before giving feedback on performance.     

KNOWLEDGE ON ACCELERATED MOTION AS MEASURED BY IMPLICIT AND EXPLICIT TASKS IN 5 TO 16 YEAR OLDS

The present study aimed at investigating children's and adolescents' understanding of constant and accelerated motions. The main objectives were (1) to investigate whether different task formats would affect the performance and (2) to track developmental changes in this domain. Five to 16 year olds (N = 157) predicted the distances of a moving vehicle on the basis of its movement durations on both a horizontal and an inclined plane. The task formats involved: (1) nonverbal action tasks, (2) number-based missing-value word problems, and (3) verbal judgments. The majority of participants of all age groups based their reactions in the first two task types on the assumption of a linear relationship between time and distance—which is correct for motions with constant speed but incorrect for accelerated motions. However, in the verbal judgments that tapped conceptual understanding, children from the age of 8 years onwards correctly assumed that an object rolling down an inclined plane would accelerate. The role of the task format in evoking erroneous beliefs and strategies is discussed.     

How to sequence video modeling examples and inquiry tasks to foster scientific reasoning

Scientific reasoning skills can be acquired through technology-enhanced inquiry tasks or video modeling examples showing how to conduct virtual experiments. However, inquiry tasks can be cognitively demanding for novice learners, whereas video modeling examples can induce overconfidence. The present study investigated the effectiveness of both approaches in isolation and combination. We compared the effects of four groups (example-example, example-task, task-example and task-task) on learning outcomes, perceived difficulty and mental effort, judgments of learning, and monitoring accuracy among 107 seventh graders. In line with our hypotheses, watching a video modeling example first led to lower mental effort, better learning outcomes, and higher judgments of learning than solving an inquiry task first. Contrary to our hypotheses, all groups underestimated their performance. Results for mental effort and learning outcomes corroborate research on worked examples, whereas results for judgments of learning and monitoring accuracy indicate an underconfidence-with-practice effect.     

Test Anxiety and Working Memory

In the present study, information processing of test anxiety is explained within the framework of the ACT* model. The author used the speed-accuracy tradeoff method to investigate the effect of test anxiety on each subsystem of working memory. The sample was made up of 119 college students enrolled in an educational psychology course. Test anxiety affected performance on the verbal-analogies task but not on the rhyming-judgment and visual-spatial tasks. The participants' subvocalization of the rhyming words may have drawn attention to the task itself and preempted the effect of test anxiety on task performance. Also, the activation processes for the visual-spatial tasks may have occurred in a different dimension or separate from the verbal processes of test anxiety.     

Functional Measurement Assessment of Young Children''s Ability to Predict Future Behavior

The reported experiments demonstrate that young children's ability to use previous behavioral information to predict future behavior emerges on quantitative, but not dichotomous, judgment tasks. In a first experiment, kindergartners, second graders, and fourth graders made quantitative liking judgments and predictions for peers after being presented 2 pieces of behavioral information. Children of all 3 age groups used both pieces of information in their liking judgments and predictions. In a second experiment, kindergartners were presented with 2 types of tasks; one was a quantitative prediction, comparable to the task in Experiment 1, and the second were dichotomous predictions, comparable to judgment tasks typically used in other experiments. Children's predictions were significantly more consistent with the behavioral information on the quantitative task than on either of the dichotomous tasks. These results suggest that young children believe in the ability of interpersonal behavior, but have difficulty dealing with the complexity of some prediction tasks.