The validity and utility of the California Family Risk Assessment under practice conditions in the field: A prospective study

Objective

Analysis of the validity and implementation of a child maltreatment actuarial risk assessment model, the California Family Risk Assessment (CFRA).

Questions addressed

(1) Is there evidence of the validity of the CFRA under field operating conditions? (2) Do actuarial risk assessment results influence child welfare workers’ service delivery decisions? (3) How frequently are CFRA risk scores overridden by child welfare workers? (4) Is there any difference in the predictive validity of CFRA risk assessments and clinical risk assessments by child welfare workers?

Method

The study analyzes 7,685 child abuse/neglect reports originating in 5 California counties followed prospectively for 2 years to identify further substantiated child abuse/neglect. Measures of model calibration and discrimination were used to assess CFRA validity and compare its accuracy with the accuracy of clinical predictions made by child welfare workers. The extent of use of an override feature of the CFRA and child welfare worker reliance on CFRA risk scores for making service decisions were analyzed.

Results

Imperfect but better-than-chance predictive validity was found for the CFRA on a range of measures in a large temporal validation sample (n = 6,543). For 114 cases where both CFRA risk assessments and child welfare worker clinical risk assessments were available, the CFRA exhibited evidence of imperfect but better-than-chance predictive validity, while child welfare worker risk assessments were found to be invalid. Child welfare workers overrode CFRA risk assessments in only 114 (1.5%) of 7,685 cases and provided in-home services in statistically significantly larger proportions of higher- versus lower-risk cases, consistent with heavy reliance on the CFRA.

Conclusions/practice implications

Until research identifies actuarial models exhibiting superior predictive validity when applied in every-day practice, the CFRA is, and will be a valuable tool for assessing risk in order to make in-home service-provision decisions.     

Factors associated with mental health services referrals for children investigated by child welfare

Although child welfare caseworkers are responsible for facilitating mental health services access for maltreated children, little is known about caseworkers’ decisions to refer children to services. We aimed to identify factors associated with caseworker referral of children to mental health services after a maltreatment investigation. We analyzed data from 1956 children 2–17 years old from the Second National Survey of Child and Adolescent Well-being. We examined associations of children’s predisposing, enabling, and need-related factors and caseworkers’ work environment characteristics with referral to mental health services. Caseworkers referred 21.0% of children to mental health services. In multivariable analyses controlling for potential covariates, factors associated with increased odds of caseworker referral included: older child age; child sexual abuse (versus neglect); child out-of-home placement; caregiver mental health problems; prior maltreatment reports; clinically significant child behavioral problems; and child welfare agency collaborative ties with mental health providers (all p < .05). Factors associated with decreased odds of caseworker referral included child Black race (versus White race) and lack of insurance (versus Private insurance) (all p < .05). In summary, children’s need for mental health services was positively associated with caseworker referral to services but certain predisposing and enabling factors and caseworker work environment characteristics also correlated with services referral. Interventions to reduce disparities in services referral by race and insurance type are critically needed. These may include child welfare agency implementation of policies for mental health screening, assessment, and services referral based on clinical need and establishment of child welfare-mental health agency collaborative ties.     

A conceptual framework for trust in standardised assessments: Commercial,quasi-market and national systems

This article presents a conceptual framework for trust in standardised assessments. Standardised assessments play an important role in many education systems as they inform decisions about students' future schooling career or entry to the labour market. Also, standardised assessments are often used for teacher performance reviews and school accountability, or to monitor learning outcomes on the national level. Various stakeholders rely on the accuracy of assessment outcomes when making decisions about students' competences, or seek to improve the quality of education. Such reliance implies a need for trust in those who design and administer standardised assessments and make decisions on the basis of the outcomes. The framework presented in this article describes the type of relational and macro-level trust that is relevant for three types of assessment systems: national, quasi-market and commercial systems. Throughout the analysis presented, examples are provided to illustrate the ways in which relational and macro-level trust can vary by who is tested and by whom they are assessed; and how trust in evaluations varies by the purpose and consequences of testing, as well as the individual agency of students, their teachers and school leaders.     

Predicting child maltreatment: A meta-analysis of the predictive validity of risk assessment instruments

Risk assessment is crucial in preventing child maltreatment since it can identify high-risk cases in need of child protection intervention. Despite widespread use of risk assessment instruments in child welfare, it is unknown how well these instruments predict maltreatment and what instrument characteristics are associated with higher levels of predictive validity. Therefore, a multilevel meta-analysis was conducted to examine the predictive accuracy of (characteristics of) risk assessment instruments. A literature search yielded 30 independent studies (N = 87,329) examining the predictive validity of 27 different risk assessment instruments. From these studies, 67 effect sizes could be extracted. Overall, a medium significant effect was found (AUC = 0.681), indicating a moderate predictive accuracy. Moderator analyses revealed that onset of maltreatment can be better predicted than recurrence of maltreatment, which is a promising finding for early detection and prevention of child maltreatment. In addition, actuarial instruments were found to outperform clinical instruments. To bring risk and needs assessment in child welfare to a higher level, actuarial instruments should be further developed and strengthened by distinguishing risk assessment from needs assessment and by integrating risk assessment with case management.     

Developing and testing a framework for evaluating the quality of comprehensive family assessment in child welfare

Over the last decade, Comprehensive Family Assessment (CFA) has become a best practice in child welfare. Comprehensive Family Assessments go beyond risk assessment to develop a full picture of the child's and family's situation. When appropriately synthesized, assessment information can lead to a clear articulation of the patterns of child or family functioning which are related to child abuse and maltreatment or which can be strengthened to facilitate change. This study defines and provides concrete examples of dimensions of quality in child welfare assessment reports that are consistent with the CFA guidelines and best practices embraced by child welfare agencies, courts, and other key stakeholders. Leveraging a random assignment design, the study compares the quality of reports written by a caseworker alone versus those written by a caseworker paired with a licensed Integrated Assessment (IA) screener. Findings are discussed in the context of the dual professional model and factors contributing to the timely completion of high quality assessment reports.     

Decision-making in child protective services: Influences at multiple levels of the social ecology

Decision-making in the child protection system is influenced by multiple factors; agency and geographic contexts, caseworker attributes, and families’ unique circumstances all likely play a role. In this study, we use the second cohort of the National Survey of Child and Adolescent Well-Being to explore how these factors are associated with two key case decisions—substantiation and removal to out-of-home care. Analyses are conducted using weighted hierarchical linear models. We find that substantiation is strongly influenced by agency factors, particularly constraints on service accessibility. Substantiation is less likely when agencies can provide services to unsubstantiated cases and when collaboration with other social institutions is high. This supports the concept that substantiation may be a gateway to services in some communities. Agency factors contributed less to the probability of removal among substantiated cases, though time resources and constraints on decision-making had some influence. For both substantiation and removal risks, county, caseworker, and child characteristics were less influential than agency characteristics and family risk factors.     

Coordination between child welfare agencies and mental health service providers,children's service use,and outcomes

ObjectiveInterorganizational relationships (IORs) between child welfare agencies and mental health service providers may facilitate mental health treatment access for vulnerable children. This study investigates whether IORs are associated with greater use of mental health services and improvement in mental health status for children served by the child welfare system.MethodsThis was a longitudinal analysis of data from a 36-month period in the National Survey of Child and Adolescent Well-Being (NSCAW). The sample consisted of 1,613 children within 75 child welfare agencies who were 2 years or older and had mental health problems at baseline. IOR intensity was measured as the number of coordination approaches between each child welfare agency and mental health service providers. Separate weighted multilevel logistic regression models tested associations between IORs and service use and outcomes, respectively.ResultsAgency-level factors accounted for 9% of the variance in the probability of service use and 12% of mental health improvement. Greater intensity of IORs was associated with higher likelihood of both service use and mental health improvement.ConclusionsHaving greater numbers of ties with mental health providers may help child welfare agencies improve children's mental health service access and outcomes.Practice implicationsPolicymakers should develop policies and initiatives to encourage a combination of different types of organizational ties between child welfare and mental health systems. For instance, information sharing at the agency level in addition to coordination at the case level may improve the coordination necessary to serve these vulnerable children.     

The effects of the ARC organizational intervention on caseworker turnover, climate, and culture in children's service systems

OBJECTIVE: This study examines the effects of the Availability, Responsiveness, and Continuity (ARC) organizational intervention strategy on caseworker turnover, climate, and culture in a child welfare and juvenile justice system. METHOD: Using a pre-post, randomized blocks, true experimental design, 10 urban and 16 rural case management teams were randomly assigned to either the ARC organizational intervention condition or to a control condition. The culture and climate of each case management team were assessed at baseline and again after the one-year organizational intervention was completed. In addition, caseworker turnover was assessed by identifying caseworkers on the sampled teams who quit their jobs during the year. RESULTS: Hierarchical Linear Models (HLM) analyses indicate that the ARC organizational intervention reduced the probability of caseworker turnover by two-thirds and improved organizational climate by reducing role conflict, role overload, emotional exhaustion, and depersonalization in both urban and rural case management teams. CONCLUSIONS: Organizational intervention strategies can be used to reduce staff turnover and improve organizational climates in urban and rural child welfare and juvenile justice systems. This is important because child welfare and juvenile justice systems in the U.S.A. are plagued by high turnover rates, and there is evidence that high staff turnover and poor organizational climates negatively affect service quality and outcomes in these systems.     

Detecting substance-related problems in narrative investigation summaries of child abuse and neglect using text mining and machine learning

BackgroundState child welfare agencies collect, store, and manage vast amounts of data. However, they often do not have the right data, or the data is problematic or difficult to inform strategies to improve services and system processes. Considerable resources are required to read and code these text data. Data science and text mining offer potentially efficient and cost-effective strategies for maximizing the value of these data.ObjectiveThe current study tests the feasibility of using text mining for extracting information from unstructured text to better understand substance-related problems among families investigated for abuse or neglect.MethodA state child welfare agency provided written summaries from investigations of child abuse and neglect. Expert human reviewers coded 2956 investigation summaries based on whether the caseworker observed a substance-related problem. These coded documents were used to develop, train, and validate computer models that could perform the coding on an automated basis.ResultsA set of computer models achieved greater than 90% accuracy when judged against expert human reviewers. Fleiss kappa estimates among computer models and expert human reviewers exceeded .80, indicating that expert human reviewer ratings are exchangeable with the computer models.ConclusionThese results provide compelling evidence that text mining procedures can be a cost-effective and efficient solution for extracting meaningful insights from unstructured text data. Additional research is necessary to understand how to extract the actionable insights from these under-utilized stores of data in child welfare.     

Caseworker-recipient interaction: welfare office differences, economic trajectories, and child outcomes

Drawing on developmental and policy research, this study examined whether 3 dimensions of caseworker-recipient interaction in welfare offices functioned as critical ecological contexts for recipient families. The sample consisted of 1,098 families from 10 welfare offices in National Evaluation of Welfare to Work Strategies (NEWWS). In multilevel analyses, caseworker support, caseload size, and emphasis on employment predicted 5-year quarterly trajectories of earnings, income, and welfare receipt. Recipients in offices characterized by high support had steeper increases in earnings and income; those in offices with high caseload size had steeper decreases in income and welfare receipt; and those in offices with high emphasis on employment had steeper decreases in welfare receipt. These economic trajectories were associated with children's reading and math achievement and internalizing behavior at ages 8-10.     

Correlates of problem recognition and intentions to change among caregivers of abused and neglected children

OBJECTIVE: To identify individual, family, and caseworker characteristics associated with problem recognition (PR) and intentions to change (ITC) in a sample of caregivers who received in-home child welfare services following substantiated reports of child abuse or neglect. METHODS: Caregivers were interviewed at 4 weeks, 16 weeks, and 1 year after referral for in-home services. In these interviews, the University of Rhode Island Change Assessment scale assessed PR and ITC in relation to caregiving practices. Additional data were obtained from administrative records and surveys of in-home services caseworkers. We used growth models to identify caregiver, family, and caseworker characteristics associated with initial levels of PR and ITC, and with changes in PR and ITC over time. RESULTS: Contrary to expectations, there were no overall increases in PR and ITC during the first 4 months of in-home services. PR and ITC scores fluctuated over time, in relation to some external events, case characteristics, and caseworker characteristics; however, we were able to account for small portions of the variance in PR and ITC. Controlling for social desirability bias and other variables in the analysis, negative life events, lack of network support, and the severity of caregiver depression were associated with greater PR; housing problems were associated with lower PR. Caregiver depression and age were associated with ITC. Caregivers whose children had been removed from their homes showed significant increases in ITC in the first few months of in-home services. The duration of caseworkers' child welfare experience predicted increases in their clients' PR in the first 4 months and more frequent contacts with an experienced caseworker predicted small, but significant increases in ITC over time. CONCLUSIONS: PR and ITC are associated with somewhat different case characteristics and may be affected by caseworkers' experience.     

Moving Toward a Comprehensive Assessment System: A Framework for Considering Interim Assessments

Local assessment systems are being marketed as formative, benchmark, predictive, and a host of other terms. Many so-called formative assessments are not at all similar to the types of assessments and strategies studied by   Black and Wiliam (1998)   but instead are interim assessments. In this article, we clarify the definition and uses of interim assessments and argue that they can be an important piece of a comprehensive assessment system that includes formative, interim, and summative assessments. Interim assessments are given on a larger scale than formative assessments, have less flexibility, and are aggregated to the school or district level to help inform policy. Interim assessments are driven by their purpose, which fall into the categories of instructional, evaluative, or predictive. Our intent is to provide a specific definition for these "interim assessments" and to develop a framework that district and state leaders can use to evaluate these systems for purchase or development. The discussion lays out some concerns with the current state of these assessments as well as hopes for future directions and suggestions for further research.     

Multi-informant assessment of treatment integrity in the classroom

Assessing the degree to which interventions are implemented in school settings is critical to making decisions about student outcomes. School psychologists may not be available to regularly conduct observations of intervention implementation, however, their data may be used alongside other methods for multi-informant assessment. Teacher self-report is a commonly used and feasible assessment method. Students have been trained to implement interventions with their peers in instances where traditional adult interventionists were unavailable. This exploratory study investigated the accuracy with which classroom teachers and middle and high school students assessed implementation of the Good Behavior Game and the impact of performance feedback on their accuracy. Results indicated that most students and teachers were able to provide accurate assessments of treatment integrity compared to researcher direct observation; however, some required performance feedback to do so. These findings suggest that multi-informant assessment may be a feasible and accurate way for school psychologists to collect formative treatment-integrity data in the classroom. Limitations and future directions are discussed.     

Development of an instrument to understand the child protective services decision-making process,with a focus on placement decisions

When children come to the attention of the child welfare system, they become involved in a decision-making process in which decisions are made that have a significant effect on their future and well-being. The decision to remove children from their families is particularly complex; yet surprisingly little is understood about this decision-making process. This paper presents the results of a study to develop an instrument to explore, at the caseworker level, the context of the removal decision, with the objective of understanding the influence of the individual and organizational factors on this decision, drawing from the Decision Making Ecology as the underlying rationale for obtaining the measures. The instrument was based on the development of decision-making scales used in prior decision-making studies and administered to child protection caseworkers in several states. Analyses included reliability analyses, principal components analyses, and inter-correlations among the resulting scales. For one scale regarding removal decisions, a principal components analysis resulted in the extraction of two components, jointly identified as caseworkers’ decision-making orientation, described as (1) an internal reference to decision-making and (2) an external reference to decision-making. Reliability analyses demonstrated acceptable to high internal consistency for 9 of the 11 scales. Full details of the reliability analyses, principal components analyses, and inter-correlations among the seven scales are discussed, along with implications for practice and the utility of this instrument to support the understanding of decision-making in child welfare.     

Models and measures of learning outcomes for non-technical skills in simulation-based medical education: Findings from an integrated scoping review of research and content analysis of curricular learning objectives

Simulations are increasingly used to enhance ‘non-technical’ skills (NTS) in medical education but little is known about the effectiveness of many simulations. The field lacks robust outcome measures and there is lack of clarity about educationally relevant conceptual and operational dimensions of NTS to support instrument development and testing. Our study addresses this dual gap through a multi-component design focusing on three key NTS: interprofessional teamwork, communication and decision-making. A Scoping Review (ScR) was utilised to synthesise instruments used in recent publications on medical/clinical simulations targeting NTS. Full-text analysis identified a sub-set of robust instruments for further analysis. Systematic qualitative content analysis of learning objectives (LOs) in UK medical education curricula identified and synthesised the conceptual dimensions underlying NTS. Finally, identified measures’ dimensions were systematically compared with the dimensions of curricular LOs to establish the extent to which instruments used in current SBL-research address desired learning goals, and to identify relevant measures and gaps. The ScR identified 225 studies from 2018−20 of which 72 met the inclusion criteria. 31/72 studies referred to a named instrument in their abstract, including 27 unique instruments. The curriculum analysis identified a set of key conceptual dimensions of each of the target NTS. Finally, comparative analysis found that while the outcome measures used in the SBL-studies addressed many of the identified curricular LO-dimensions, there are significant gaps, notably relating to evidence use and inclusive practice. Moreover, it revealed there is little conceptual overlap between the instruments, highlighting the need for further research to ensure comparability of different studies. This study contributes to our understanding and evaluation of learning outcomes in, and our readiness to generate a rigorous evidence base for, NTS-focused SBL, by describing conceptual and operational constructs of NTS-learning outcomes in medical education and identifying appropriate validated assessment instruments to evaluate SBL.     

Using art to assess environmental education outcomes

Construction of developmentally appropriate tools for assessing the environmental attitudes and awareness of young learners has proven to be challenging. Art-based assessments that encourage creativity and accommodate different modes of expression may be a particularly useful complement to conventional tools (e.g. surveys), but their efficacy and feasibility across diverse contexts has not been adequately explored. To examine the potential utility of integrating art into evaluations of environmental education outcomes, we adapted an existing drawing prompt and corresponding grading rubric to assess the environmental attitudes and awareness of children (ages 6–12) at summer camps in Athens, GA, USA (n = 285). We then compared children’s drawings with scores on a more typical survey instrument that measured similar outcomes, the Children Environmental Perception’s Scale. Results showed that a drawing prompt was a practical and unique learner-centered tool for measuring distinct components of environmental attitudes and awareness. Findings also revealed different response patterns across the two instruments, highlighting the value of using multiple approaches (e.g. art-based and survey-based) to assess cognitive and affective aspects of children’s environmental orientations.     

Welche Rolle spielt Kompetenzdiagnostik im Rahmen von Lehr-Lernprozessen?

Competence assessment can support evidence-based teaching and learning. Depending on its specific purpose, competence assessment can be applied at the individual, the classroom or school level as well as on a national level. Large-scale-assessments are typical examples for system monitoring. Competence assessment on the individual level can provide important information with respect to individual qualification and learning outcomes. Typically, competence assessment uses psychometric models from Item-Response-Theory (IRT). However, until present, competence assessment still struggles to tap its full potential regarding feedback and individual support. Teachers often have difficulties to interpret results from competence assessments that are based on IRT. Thus, it is crucial to support teachers in using these results to improve classroom teaching. One of the main challenges of educational measurement is to align everyday classroom teaching with formative and summative assessments and to establish competence models as a common ground. Process-orientated, formative assessment can inform criterion-based feedback and, consequently, promote teaching on the basis of educational standards that were defined on the system level. However, more work needs to be done with respect to research on assessments as well as their communication and implementation into teaching practice.     

Organizational climate, services, and outcomes in child welfare systems

Objective

This study examines the association of organizational climate, casework services, and youth outcomes in child welfare systems. Building on preliminary findings linking organizational climate to youth outcomes over a 3-year follow-up period, the current study extends the follow-up period to 7 years and tests main, moderating and mediating effects of organizational climate and casework services on outcomes.

Methods

The study applies hierarchical linear models (HLMs) analyses to all 5 waves of the National Survey of Child and Adolescent Well-being (NSCAW) with a US nationwide sample of 1,678 maltreated youth aged 4–16 years and 1,696 caseworkers from 88 child welfare systems. Organizational climate is assessed on 2 dimensions, Engagement and Stress, with scales from the well established measure, Organizational Social Context (OSC); youth outcomes are measured as problems in psychosocial functioning with the Child Behavior Checklist (CBCL); and casework services are assessed with original scales developed for the study and completed by the maltreated youths’ primary caregivers and caseworkers.

Results

Maltreated youth served by child welfare systems with more engaged organizational climates have significantly better outcomes. Moreover, the quantity and quality of casework services neither mediate nor interact with the effects of organizational climate on youth outcomes.

Conclusions

Organizational climate is associated with youth outcomes in child welfare systems, but a better understanding is needed of the mechanisms that link organizational climate to outcomes. In addition, there is a need for evidence-based organizational interventions that can improve the organizational climates and effectiveness of child welfare systems.     

Assessing the Organizational Social Context (OSC) of child welfare systems: Implications for research and practice

ObjectiveThe study: (1) provides the first assessment of the a priori measurement model and psychometric properties of the Organizational Social Context (OSC) measurement system in a US nationwide probability sample of child welfare systems; (2) illustrates the use of the OSC in constructing norm-based organizational culture and climate profiles for child welfare systems; and (3) estimates the association of child welfare system-level organizational culture and climate profiles with individual caseworker-level job satisfaction and organizational commitment.MethodsThe study applies confirmatory factor analysis (CFA) and hierarchical linear models (HLM) analysis to a US nationwide sample of 1,740 caseworkers from 81 child welfare systems participating in the second National Survey of Child and Adolescent Wellbeing (NSCAW II). The participating child welfare systems were selected using a national probability procedure reflecting the number of children served by child welfare systems nationwide.ResultsThe a priori OSC measurement model is confirmed in this nationwide sample of child welfare systems. In addition, caseworker responses to the OSC scales generate acceptable to high scale reliabilities, moderate to high within-system agreement, and significant between-system differences. Caseworkers in the child welfare systems with the best organizational culture and climate profiles report higher levels of job satisfaction and organizational commitment. Organizational climates characterized by high engagement and functionality, and organizational cultures characterized by low rigidity are associated with the most positive work attitudes.ConclusionsThe OSC is the first valid and reliable measure of organizational culture and climate with US national norms for child welfare systems. The OSC provides a useful measure of Organizational Social Context for child welfare service improvement and implementation research efforts which include a focus on child welfare system culture and climate.