Developing and using a rubric for evaluating evidence-based medicine point-of-care tools

Objective:

The research sought to establish a rubric for evaluating evidence-based medicine (EBM) point-of-care tools in a health sciences library.

Methods:

The authors searched the literature for EBM tool evaluations and found that most previous reviews were designed to evaluate the ability of an EBM tool to answer a clinical question. The researchers'' goal was to develop and complete rubrics for assessing these tools based on criteria for a general evaluation of tools (reviewing content, search options, quality control, and grading) and criteria for an evaluation of clinical summaries (searching tools for treatments of common diagnoses and evaluating summaries for quality control).

Results:

Differences between EBM tools'' options, content coverage, and usability were minimal. However, the products'' methods for locating and grading evidence varied widely in transparency and process.

Conclusions:

As EBM tools are constantly updating and evolving, evaluation of these tools needs to be conducted frequently. Standards for evaluating EBM tools need to be established, with one method being the use of objective rubrics. In addition, EBM tools need to provide more information about authorship, reviewers, methods for evidence collection, and grading system employed.

Highlights

• Eleven of the fourteen previous evidence-based medicine (EBM) tool evaluations were based on clinicians evaluating tools based on their perception of the products'' ability to answer a clinical question.
• EBM tools'' evidence summaries are not updated as often as products claim.
• Although many EBM tools claim to be evidence based, only 74% of the 70 evaluated treatment summaries included graded evidence.

Implications

• To offer the best tools for users, medical libraries should evaluate EBM resources regularly, including the quality of the evidence provided.
• Medical librarians have a role to play in evaluating the quality of EBM products and can develop assessment tools to aid in this evaluation.

     

Making the Critical Appraisal for Summaries of Evidence (CASE) for evidence-based medicine (EBM): critical appraisal of summaries of evidence

Objectives:

Standards for evaluating evidence-based medicine (EBM) point-of-care (POC) summaries of research are lacking. The authors developed a “Critical Appraisal for Summaries of Evidence” (CASE) worksheet to help assess the evidence in these tools. The authors then evaluated the reliability of the worksheet.

Methods:

The CASE worksheet was developed with 10 questions covering specificity, authorship, reviewers, methods, grading, clarity, citations, currency, bias, and relevancy. Two reviewers independently assessed a random selection of 384 EBM POC summaries using the worksheet. The responses of the raters were then compared using a kappa score.

Results:

The kappa statistic demonstrated an overall moderate agreement (κ = 0.44) between the reviewers using the CASE worksheet for the 384 summaries. The 3 categories of evaluation questions in which the reviewers disagreed most often were citations (κ =  0), bias (κ = 0.11), and currency (κ = −0.18).

Conclusions:

The CASE worksheet provided an effective checklist for critically analyzing a treatment summary. While the reviewers agreed on worksheet responses for most questions, variation occurred in how the raters navigated the tool and interpreted some of the questions. Further validation of the form by other groups of users should be investigated.

Highlights

• Few critical appraisal tools have been evaluated with inter-rater reliability testing.
• The ways that users of evidence-based medicine (EBM) point-of-care (POC) tools interpret how to appraise an evidence summary—particularly when defining the grading of evidence, currency, and bias—may vary even when a standard evaluation sheet is used.
• The Critical Appraisal for Summaries of Evidence (CASE) worksheet had a moderate level of inter-rater reliability, similar to previous evaluative studies of critical appraisals tools.

Implications

• Medical librarians can develop tools useful for librarians, students, and clinicians to guide them in appraising clinical evidence summaries.
• The CASE worksheet can be a valuable tool to consider the quality of individual evidence summaries and to see patterns of overall quality in EBM POC tools.

     

Communication Practices of Professional Service Providers: Predicting Customer Satisfaction and Loyalty

This study examined customers' perceptions of personalized service communication performance in relation to their expectations as predictors of their satisfaction and loyalty intentions in four professional service contexts. Survey data were obtained from 253 customers regarding their expectations and experiences with physicians, dentists, auto mechanics, or hairdressers. Multisample covariance structure analysis procedures were performed to assess the level of support for alternative theoretical models across all four professional contexts. Results provided support for one model, specifying that customers' perceptions of communication performance accounted for significant variation in customer satisfaction, which, in turn, predicted customers' loyal behavioral intentions. Customers' expectations did not play an influential role in predicting customer satisfaction and loyalty, as proposed in an alternative model     

Planning changes to health library services on the basis of impact assessment

Background: Various methods of impact assessment for health library services exist, including a toolkit developed for the UK. The Knowledge, Resource and Information service (KRIS) for health promotion, health service commissioning and public health (Bristol area, UK) commissioned an independent team at Aberystwyth University to provide an impact assessment and evaluation of their services and to provide evidence for future planning. Objective: The review aimed to provide an action plan for KRIS through assessing the impact of the current service, extent of satisfaction with existing services and views on desirable improvements. Methods: Existing impact toolkit guidance was used, with an adapted impact questionnaire, which was distributed by the KRIS staff to 244 users (response rate 62.3%) in early 2009. The independent team analysed the questionnaire data and presented the findings. Results: Users valued the service (93% considered that relevant information was obtained). The most frequent impacts on work were advice to patients, clients or carers, and advice to colleagues. Literature searching and current awareness services saved staff time. Many users were seeking health promotion materials. Conclusion: The adapted questionnaire worked well in demonstrating the service impacts achieved by KRIS, as well as indicating desirable improvements in service delivery.