Developing and using a rubric for evaluating evidence-based medicine point-of-care tools

Objective:

The research sought to establish a rubric for evaluating evidence-based medicine (EBM) point-of-care tools in a health sciences library.

Methods:

The authors searched the literature for EBM tool evaluations and found that most previous reviews were designed to evaluate the ability of an EBM tool to answer a clinical question. The researchers'' goal was to develop and complete rubrics for assessing these tools based on criteria for a general evaluation of tools (reviewing content, search options, quality control, and grading) and criteria for an evaluation of clinical summaries (searching tools for treatments of common diagnoses and evaluating summaries for quality control).

Results:

Differences between EBM tools'' options, content coverage, and usability were minimal. However, the products'' methods for locating and grading evidence varied widely in transparency and process.

Conclusions:

As EBM tools are constantly updating and evolving, evaluation of these tools needs to be conducted frequently. Standards for evaluating EBM tools need to be established, with one method being the use of objective rubrics. In addition, EBM tools need to provide more information about authorship, reviewers, methods for evidence collection, and grading system employed.

Highlights

• Eleven of the fourteen previous evidence-based medicine (EBM) tool evaluations were based on clinicians evaluating tools based on their perception of the products'' ability to answer a clinical question.
• EBM tools'' evidence summaries are not updated as often as products claim.
• Although many EBM tools claim to be evidence based, only 74% of the 70 evaluated treatment summaries included graded evidence.

Implications

• To offer the best tools for users, medical libraries should evaluate EBM resources regularly, including the quality of the evidence provided.
• Medical librarians have a role to play in evaluating the quality of EBM products and can develop assessment tools to aid in this evaluation.

     

Exploring e-readers to support clinical medical education: two case studies

Question:

Can e-readers loaded with medical textbooks and other relevant material benefit medical students, residents, and preceptors in clinical settings?

Settings:

The settings are North Carolina community clinics served by Duke University Medical Center and St. Joseph''s Hospital in Bryan, Texas, and Scott and White Memorial Hospital in Temple, Texas.

Methods:

Duke University: Twenty second-year medical students and fourteen family medicine clerkship preceptors used Kindle e-readers in clinics during eight months of rotations. Students and preceptors provided feedback through an anonymous online survey. Texas A&M University: Nine fourth-year medical students in an elective compared medical textbooks in print, online, and on a Kindle. Six residents at a local hospital completed an anonymous online survey after a three-week loan of a Kindle loaded with medical textbooks.

Results:

The e-reader''s major advantages in clinical settings are portability and searchability. The selected e-reader''s limitations include connection speed, navigation, and display. User preferences varied, but online resources were preferred. Participants suggested additional uses for Kindles in medical education.

Conclusions:

The selected e-reader''s limitations may be resolved with further development of the device. Investigation of other e-readers is needed. Criteria for evaluating e-readers in clinical settings should include portability, searchability, speed, navigation, and display. Research comparing e-readers and mobile devices in clinical education is also warranted.     

French by osmosis

The typical nursery school in the province of Quebec, Canada, is conducted in French, the official language of schools and business. Even so, it is not unusual to find English- as well as French-speaking students at many of these schools. Both languages are spoken in the province.     

Making the Critical Appraisal for Summaries of Evidence (CASE) for evidence-based medicine (EBM): critical appraisal of summaries of evidence

Objectives:

Standards for evaluating evidence-based medicine (EBM) point-of-care (POC) summaries of research are lacking. The authors developed a “Critical Appraisal for Summaries of Evidence” (CASE) worksheet to help assess the evidence in these tools. The authors then evaluated the reliability of the worksheet.

Methods:

The CASE worksheet was developed with 10 questions covering specificity, authorship, reviewers, methods, grading, clarity, citations, currency, bias, and relevancy. Two reviewers independently assessed a random selection of 384 EBM POC summaries using the worksheet. The responses of the raters were then compared using a kappa score.

Results:

The kappa statistic demonstrated an overall moderate agreement (κ = 0.44) between the reviewers using the CASE worksheet for the 384 summaries. The 3 categories of evaluation questions in which the reviewers disagreed most often were citations (κ =  0), bias (κ = 0.11), and currency (κ = −0.18).

Conclusions:

The CASE worksheet provided an effective checklist for critically analyzing a treatment summary. While the reviewers agreed on worksheet responses for most questions, variation occurred in how the raters navigated the tool and interpreted some of the questions. Further validation of the form by other groups of users should be investigated.

Highlights

• Few critical appraisal tools have been evaluated with inter-rater reliability testing.
• The ways that users of evidence-based medicine (EBM) point-of-care (POC) tools interpret how to appraise an evidence summary—particularly when defining the grading of evidence, currency, and bias—may vary even when a standard evaluation sheet is used.
• The Critical Appraisal for Summaries of Evidence (CASE) worksheet had a moderate level of inter-rater reliability, similar to previous evaluative studies of critical appraisals tools.

Implications

• Medical librarians can develop tools useful for librarians, students, and clinicians to guide them in appraising clinical evidence summaries.
• The CASE worksheet can be a valuable tool to consider the quality of individual evidence summaries and to see patterns of overall quality in EBM POC tools.