Inter-rater reliability of the Illinois Structured Decision Support Protocol

OBJECTIVE: The purpose of this study was to determine the level of inter-rater reliability of the Illinois Structured Decision Support Protocol by examining the level of Child Protective Services (CPS) caseworkers' agreement regarding state interventions. The Protocol was designed to guide CPS workers to consistent decisions related to the level of state intervention in child maltreatment cases. METHOD: Forty-one workers recruited from three CPS field offices in Illinois participated in this study by assessing three cases using the Illinois Structured Decision Support Protocol. The three cases were developed based on actual cases investigated by the agency. The kappa statistic was used to calculate the level of workers' agreement. RESULTS: The result showed a weak reliability (kappa=.29). Further analysis showed that workers identified a wide range of risk factors and differing levels of intervention for each identified risk factor even among the workers who made the same final decisions. CONCLUSIONS: Decision support protocols are developed to provide systematic structures for workers' decision-making. However, this study suggested that workers demonstrated a low level of agreement on such a protocol. Several suggestions for developing more reliable computerized protocols include: more specified inquiries with criteria to assess the range of risk for a factor, consideration of interactions among risk factors, assessment of family strengths, and having the final decision made by a computerized program.