Methodological Choices in the Content Analysis of Textbooks for Measuring Alignment With Standards

With the recent adoption of the Common Core standards in many states, there is a need for quality information about textbook alignment to standards. While there are many existing content analysis procedures, these generally have little, if any, validity or reliability evidence. One exception is the Surveys of Enacted Curriculum (SEC), which has been widely used to analyze the alignment among standards, assessments, and teachers’ instruction. However, the SEC can be time‐consuming and expensive when used for this purpose. This study extends the SEC to the analysis of entire mathematics textbooks and investigates whether the results of SEC alignment analyses are affected if the content analysis procedure is simplified. The results indicate that analyzing only every fifth item produces nearly identical alignment results with no effect on the reliability of content analyses.     

Imagining Recovery,Resilience, and Vulnerability at 75: Perceptions of Social Work Students

BSW and MSW students (N = 183) read one of three vignettes and were asked to respond to 21 items investigating their perceptions about depression, anxiety, and resilience of the vignette's character. While the incidents in each vignette were identical, the main character was (a) a 75-year-old Mr. Jones, (b) a 75-year-old Ms. Jones, or (c) “Imagine you are 75 years old.” The character of each vignette had lost a spouse, was a victim of a mugging, and was exhibiting symptoms of depression and anxiety. The 21 items were analyzed by vignette type using a General Linear Model (GLM) and controlling for student type (BSW, MSW) and ethnocultural identity (member of the dominant or minority group). The GLM was significant (F = 3.020, df = 42, p = .000). There were significant differences among the vignette types in 11 of the 21 items. In general, respondents were more likely to perceive that they would be more resilient and less vulnerable at 75 than other 75-year-old men and women.     

How Item Writers Understand Depth of Knowledge

An important part of test development is ensuring alignment between test forms and content standards. One common way of measuring alignment is the Webb (1997, 2007) alignment procedure. This article investigates (a) how well item writers understand components of the definition of Depth of Knowledge (DOK) from the Webb alignment procedure and (b) how consistent their DOK ratings are with ratings provided by other committees of educators across grade levels, content areas, and alternate assessment levels in a Midwestern state alternate assessment system. Results indicate that many item writers understand key features of DOK. However, some item writers struggled to articulate what DOK means and had some misconceptions. Additional analyses suggested some lack of consistency between the item writer DOK ratings and the committee DOK ratings. Some notable differences were found across alternate assessment levels and content areas. Implications for future item writing training and alignment studies are provided.     

Evaluating the alignment among curriculum,instruction, and assessments: Implications and applications for research and practice

Alignment has been defined as the extent to which curricular expectations and assessments are in agreement and work together to provide guidance for educators' efforts to facilitate students' progress toward desire academic outcomes. The Council of Chief State School Officers has identified three preferred models as frameworks for evaluating alignment: Webb's alignment model, the Surveys of Enacted Curriculum model, and the Achieve model. Each model consists of a series of indices that summarize or describe the general match or coherence between state standards, large‐scale assessments, and, in some cases, classroom instruction. This article provides an overview of these frameworks for evaluating alignment and their applications in educational practice and the research literature. After providing an introduction to the use of alignment to evaluate large‐scale accountability systems, the article presents potential extensions of alignment for use with vulnerable populations (e.g., students with disabilities, preschoolers), individual students, and classroom teachers. These proposed applications can provide information for facilitating efforts to improve teachers' classroom instruction and students' educational achievement. © 2008 Wiley Periodicals, Inc.     

基于广义估计方程的时间相关性事故频次建模

研究目的：创新要点：研究方法：重要结论：采用广义估计方程模型对存在时间相关件的事故频次数据进行建模,并与传统广义线性模型的估计效果进行对比。 通过广义估计方程来考虑事故频次建模中数据的时间相关性,从而提高参数估计准确度以及模型预测精度。基于4年高速公路交通事故频次数据,建立考虑时间相关性的广义估计方程以及传统的广义线性模型,并采用统计指标对模型效果进行对比。1．事故频次数据样本最对预测精度影响很大;2．广义估引方程能够有效考虑事故频次数据中存住的时间相关性;3．广义估计方程的参数估计比传统广义线性模型史准确,且精度更高。     

Alignment of Mathematics State-Level Standards and Assessments: The Role of Reviewer Agreement

This article examines the role of reviewer agreement in judgments about alignment between tests and standards. We used case data from three state alignment studies to explore how different approaches to incorporating reviewer agreement changes alignment conclusions. The three case studies showed varying degrees of reviewer agreement about correspondences between objectives and test items. Moreover, taking into account reviewer agreement in the analyses sometimes had a marked effect on alignment conclusions. We discuss reasons for differences across case studies and alignment approaches, as well as implications for future alignment efforts.     

Testing measurement invariance of mathematics self-concept and self-efficacy in PISA using MGCFA and the alignment method

The aim of the study is to investigate the measurement invariance of mathematics self-concept and self-efficacy across 40 countries that participated in the Programme for International Student Assessment (PISA) 2003 and 2012 cycles. The sample of the study consists of 271,760 students in PISA 2003 and 333,804 students in PISA 2012. Firstly, the traditional measurement invariance testing was applied in the multiple-group confirmatory factor analysis (MGCFA). Then, the alignment analyses were performed, allowing non-invariance to a minimum to estimate all of the parameters. Results from MGCFA indicate that mathematics self-concept and self-efficacy hold metric invariance across the 80 groups (cycle by country). The alignment method results suggest that a large proportion of non-invariance exists in both mathematics self-concept and self-efficacy factors, and the factor means cannot be compared across all participating countries. Results of the Monte Carlo simulation show that the alignment results are trustworthy. Implications and limitations are discussed, and some recommendations for future research are proposed.

     

The Rating and Matching Item-Objective Alignment Methods

Central to the standards-based assessment validation process is an examination of the alignment between state standards and test items. Several alignment analysis systems have emerged recently, but most rely on either traditional rating or matching techniques. Little, if any, analyses have been reported on the degree of consistency between the two methods and on the item and objective characteristics that influence judges' decisions. We randomly assigned judges to either rate item-objective links or match items to objectives while reviewing the 2004 Arizona high school mathematics standards and assessment. Across items we found moderate convergence between methods, and we detected apparent reasons for divergently scored items. We also found that judges relied on item and objective content and intellectual skill features to render decisions. Based on our evidence, we contend that a thorough alignment analysis would involve judges using both rating and matching, while focusing on both content and intellectual skill. The findings have important implications for states when examining the alignment between their standards and assessments.     

独立与相依情形下加权广义线性模型选择的重对数律和强一致性

在温和条件下,考虑相依响应情形(ρ-混合或m-相依),针对一般广义线性模型(GLM)任意改变其参数权重构造出加权GLM,给出了参数的最大似然估计(MLE),并推导了加权后模型的重对数律(LIL)。应用独立情形下强极限理论证明了任意修正模型与全模型的对数似然之差的渐近结果。借助惩罚加权对数似然函数技术,基于相依LIL证明了模型选择准则的强一致性,并推导出若惩罚项的阶数介于O(log log n)与O(n)之间时,则该准则选取最简单修正模型几乎是必然的。     

猪Klf4、Klf5和Egr2基因内含子的克隆及进化分析

从猪肝脏提取基因组作为模板,分别扩增了Klf4、Klf5和Egr2的第3、第2和第1内含子,长度分别为916、1027和1342bp,并通过其两端连接的部分外显子序列与Genbank序列比对加以确认,并和人相应基因内含子作长度和序列同源性比较。结果表明,由内含子比对得出的这些基因在人和猪间的保守程度与这些基因在氨基酸水平上比对得出的保守程度相一致。     

Aligning an Early Childhood Assessment to State Kindergarten Content Standards: Application of a Nationally Recognized Alignment Framework

This article describes an alignment study conducted to evaluate the alignment between Indiana's Kindergarten content standards and items on the Indiana Standards Tool for Alternate Reporting. Alignment is the extent to which standards and assessments are in agreement, working together to guide educators' efforts to support children's learning and development. The alignment process in this study represented a modification of Webb's nationally recognized method of alignment analysis to early childhood assessments and standards. The alignment panel (N = 13) in this study consisted of early childhood educators and educational leaders from all geographic regions of the state. Panel members were asked to rate the depth of knowledge (DOK) stage of each objective in Kindergarten standards; rate the DOK stage for each item on the ISTAR rating scale; and identify the one or two objectives from the standards to which each ISTAR item corresponded. Analysis of the panel's responses suggested the ISTAR inconsistently conformed to Webb's DOK consistency and ROK correspondence criteria for alignment. A promising finding was the strong alignment of the ISTAR Level F1 and F2 scales to the Kindergarten standards. This result provided evidence of the developmental continuum of skills and knowledge that are assessed by the ISTAR items .     

Navigating Community College Transfer in Science,Technical, Engineering,and Mathematics Fields

Given financial barriers facing community college students today, and workforce projections in science, technical, engineering, and math (STEM) fields, the costs of unnecessary delays while navigating transfer pathways are high. In this phenomenological study, we analyzed the delay experiences of 172 students (65% female) navigating community college transfer pathways in STEM fields in Massachusetts. When focusing on institutional delays, three central elements emerged: (a) informational setbacks from dissatisfactory advising, (b) imperfect program alignment with four-year institutions, and (c) college resource limitations. Students took unnecessary courses or could not get into courses in a timely manner, resulting in lost time, money, and credit. An accumulation of delays is particularly detrimental to STEM women and men, given the sequential nature of their programming. Implications for policy and practice are discussed.     

Gauging Item Alignment Through Online Systems While Controlling for Rater Effects

The alignment of test items to content standards is critical to the validity of decisions made from standards‐based tests. Generally, alignment is determined based on judgments made by a panel of content experts with either ratings averaged or via a consensus reached through discussion. When the pool of items to be reviewed is large, or the content‐matter experts are broadly distributed geographically, panel methods present significant challenges. This article illustrates the use of an online methodology for gauging item alignment that does not require that raters convene in person, reduces the overall cost of the study, increases time flexibility, and offers an efficient means for reviewing large item banks. Latent trait methods are applied to the data to control for between‐rater severity, evaluate intrarater consistency, and provide item‐level diagnostic statistics. Use of this methodology is illustrated with a large pool (1,345) of interim‐formative mathematics test items. Implications for the field and limitations of this approach are discussed.     

Effects of social and economic factors on four-year higher-education enrollments in Virginia

Based on human capital theory, the demand for higher education is investigated within a statewide system. A multiple regression approach is applied to data that measure the following by geographic municipality (i.e., county or city): the ratio of high school graduates enrolled in a given institution to that institution's entering freshman enrollment (dependent variable); eligible population, academic ability, educational attainment, income level, wage rate, unemployment rate, cost of attendance, and local environment (independent variables). Using the SAS GLM procedure, various breakdowns of the demand function by institutional grouping are presented. Squared multiple correlations for the various models range from .3 to .5.Presented at the 22nd Annual Forum of the Association for Institutional Research, Denver, May 1982.     

European standards for quality assurance and institutional practices of student assessment in the UK,the Netherlands and the Czech Republic

This article analyses the student assessment procedures of 12 universities in the UK, the Netherlands and the Czech Republic with respect to their alignment with the European standards and guidelines on the quality of assessing higher education students (European Standards and Guidelines for Quality Assurance [ESG] 1.3). Based on qualitative methodology combining document review with semi-structured interviews, the analysis has yielded three major results. First, the assessment procedures of the UK universities studied indicate the highest alignment with the ESG 1.3 when compared to the Czech institutions, with the Dutch universities occupying the middle ground. Second, a preference for summative assessment to account for pressures of massification can be observed at institutions in all three countries. Third, it is argued that contrary to some (ministerial) expectations, the influence of the ESG on institutions seems to be insignificant, due to unawareness of the ESG 1.3 at all of the UK and Dutch universities analysed and only moderate knowledge of ESG 1.3 in the Czech case. For this reason, there seems to be a need for communicating the ESG to higher education institutions rather than revising the scope of the ESG as envisaged in the 2012 Bologna policy document (Bucharest Communiqué).     

Does Test Item Performance Increase with Test-to-Standards Alignment?

Variation in test performance among examinees from different regions or national jurisdictions is often partially attributed to differences in the degree of content correspondence between local school or training program curricula, and the test of interest. This posited relationship between test-curriculum correspondence, or “alignment,” and test performance is usually inferred from highly distal evidence, rather than directly examined. Utilizing mathematics standards content analysis data and achievement test item data from ten U.S. states, we examine the relationship between topic-specific alignment and test item performance. When a particular item’s content type is emphasized by the standards, we find evidence of a positive relationship between the alignment measure and proportion-correct test item difficulty, although this effect is not consistent across samples. Implications of the results for curricular achievement test development and score interpretation are discussed.     

教育工作者及其报酬： 政策、实施与系统协调问题（下）

本文从管理学和政策学的视角,探讨了公立学校教师薪酬改革的问题.文章在回溯了教师薪酬制度历史沿革的基础上,分析了当前教师薪酬体制的结构及其对个人和组织层面的消极后果.作者认为,绩效工资改革是增强学校效能、促进学生成绩提高的有效途径;改革的未来取决于一系列多变因素之间的复杂互动,而系统的整合与协调是确保绩效工资计划有效并持久推进的关键.     

Implementing constructive alignment in higher education – cross-institutional perspectives from Australia

ABSTRACT

Constructive alignment (CA) has become internationally established as an educational approach linking strategic planning and corporate policy to discipline and course teaching and learning practice. The literature to date has focused predominantly on either single institutional or specific discipline settings and curriculum level initiatives. This research study seeks to broaden the conceptual and methodological perspective on constructive alignment with a cross-institutional study from two Australian universities. The case study learnings from a top-down institutional implementation of CA at one university and the bottom-up teaching and learning approach within the other have been jointly investigated. By mapping the implementation process of constructive alignment, the key strengths and constraints for both approaches have been identified. These findings offer new insights for institutional managers as well as academics and teaching and learning professionals. In addition, a potential ‘gap in the middle’ at faculty level is highlighted. It is within this nexus that the transitions from institutional policy and targets to successful discipline-specific teaching and learning practices and outcomes are shaped. This research seeks to raise the awareness that CA design, planning and implementation should be conceptualised from the outset as a dynamic, multi-directional and iterative process, irrespective of whether a predominantly top-down or bottom-up approach for implementation is chosen.     

Using comparative genomics for inquiry-based learning to dissect virulence of Escherichia coli O157:H7 and Yersinia pestis

Genomics and bioinformatics are topics of increasing interest in undergraduate biological science curricula. Many existing exercises focus on gene annotation and analysis of a single genome. In this paper, we present two educational modules designed to enable students to learn and apply fundamental concepts in comparative genomics using examples related to bacterial pathogenesis. Students first examine alignments of genomes of Escherichia coli O157:H7 strains isolated from three food-poisoning outbreaks using the multiple-genome alignment tool Mauve. Students investigate conservation of virulence factors using the Mauve viewer and by browsing annotations available at the A Systematic Annotation Package for Community Analysis of Genomes database. In the second module, students use an alignment of five Yersinia pestis genomes to analyze single-nucleotide polymorphisms of three genes to classify strains into biovar groups. Students are then given sequences of bacterial DNA amplified from the teeth of corpses from the first and second pandemics of the bubonic plague and asked to classify these new samples. Learning-assessment results reveal student improvement in self-efficacy and content knowledge, as well as students' ability to use BLAST to identify genomic islands and conduct analyses of virulence factors from E. coli O157:H7 or Y. pestis. Each of these educational modules offers educators new ready-to-implement resources for integrating comparative genomic topics into their curricula.