Sensitivity and precision of adverse effects search filters in MEDLINE and EMBASE: a case study of fractures with thiazolidinediones

Background: Search filters have been developed in MEDLINE and EMBASE to help overcome the challenges of searching electronic databases for information on adverse effects. However, little evaluation of their effectiveness has been carried out. Objectives: To measure the sensitivity and precision of available adverse effects search filters in MEDLINE and EMBASE. Methods: A case study systematic review of fracture related adverse effects associated with the use of thiazolidinediones was used. Twelve MEDLINE search strategies and three EMBASE search strategies were tested. Results: Nineteen relevant references from MEDLINE and 24 from EMBASE were included in the review. Four search filters in MEDLINE achieved high sensitivity (95 or 100%) with an improved level of precision from searches without any adverse effects filter. High precision in MEDLINE could also be achieved (up to 53%) using search filters that rely on Medical Subject Headings. No search filter in EMBASE achieved high precision (all were under 5%) and the highest sensitivity in EMBASE was 83%. Conclusions: Adverse effects search filters appear to be effective in MEDLINE for achieving either high sensitivity or high precision. Search filters in EMBASE, however, do not appear as effective, particularly in improving precision.     

The NICE OECD countries’ geographic search filters: Part 1—methodology for developing the draft MEDLINE and Embase (Ovid) filters

Objective:There are no existing validated search filters for the group of 37 Organisation for Economic Co-operation and Development (OECD) countries. This study describes how information specialists from the United Kingdom''s National Institute for Health and Care Excellence (NICE) developed and evaluated novel OECD countries’ geographic search filters for MEDLINE and Embase (Ovid) to improve literature search effectiveness for evidence about OECD countries.Methods:We created the draft filters using an alternative approach to standard filter construction. They are composed entirely of geographic subject headings and are designed to retain OECD country evidence by excluding non-OECD country evidence using the NOT Boolean operator. To evaluate the draft filters’ effectiveness, we used MEDLINE and Embase literature searches for three NICE guidelines that retrieved >5,000 search results. A 10% sample of the excluded references was screened to check that OECD country evidence was not inadvertently excluded.Results:The draft MEDLINE filter reduced results for each NICE guideline by 9.5% to 12.9%. In Embase, search results were reduced by 10.7% to 14%. Of the sample references, 7 of 910 (0.8%) were excluded inadvertently. These references were from a guideline about looked-after minors that concerns both OECD and non-OECD countries.Conclusion:The draft filters look promising—they reduced search result volumes while retaining most OECD country evidence from MEDLINE and Embase. However, we advise caution when using them in topics about both non-OECD and OECD countries. We have created final versions of the search filters and will validate them in a future study.     

The NICE OECD countries' geographic search filters: Part 2—validation of the MEDLINE and Embase (Ovid) filters

Objective:We previously developed draft MEDLINE and Embase (Ovid) geographic search filters for Organisation for Economic Co-operation and Development (OECD) countries to assess their feasibility for finding evidence about the countries. Here, we describe the validation of these search filters.Methods:We identified OECD country references from thirty National Institute for Health and Care Excellence (NICE) guidelines to generate gold standard sets for MEDLINE (n=2,065) and Embase (n=2,023). We validated the filters by calculating their recall against these sets. We then applied the filters to existing search strategies for three OECD-focused NICE guideline reviews (NG103 on flu vaccination, NG140 on abortion care, and NG146 on workplace health) to calculate the filters'' impact on the number needed to read (NNR) of the searches.Results:The filters both achieved 99.95% recall against the gold standard sets. Both filters achieved 100% recall for the three NICE guideline reviews. The MEDLINE filter reduced NNR from 256 to 232 for the NG103 review, from 38 to 27 for the NG140 review, and from 631 to 591 for the NG146 review. The Embase filter reduced NNR from 373 to 341 for the NG103 review, from 101 to 76 for the NG140 review, and from 989 to 925 for the NG146 review.Conclusion:The NICE OECD countries'' search filters are the first validated filters for the countries. They can save time for research topics about OECD countries by finding the majority of evidence about OECD countries while reducing search result volumes in comparison to no filter use.     

Development of a validated search filter for Ovid Embase for degenerative cervical myelopathy

Background

Degenerative cervical myelopathy (DCM) is a recently proposed umbrella term for symptomatic cervical spinal cord compression secondary to degeneration of the spine. Currently literature searching for DCM is challenged by the inconsistent uptake of the term ‘DCM’ with many overlapping keywords and numerous synonyms.

Objectives

Here, we adapt our previous Ovid medline search filter for the Ovid embase database, to support comprehensive literature searching. Both embase and medline are recommended as a minimum for systematic reviews.

Methods

References contained within embase identified in our prior study formed a ‘development gold standard’ reference database (N = 220). The search filter was adapted for embase and checked against the reference database. The filter was then validated against the ‘validation gold standard’.

Results

A direct translation was not possible, as medline indexing for DCM and the keywords search field were not available in embase . We also used the ‘focus’ function to improve precision. The resulting search filter has 100% sensitivity in testing.

Discussion and Conclusion

We have developed a validated search filter capable of retrieving DCM references in embase with high sensitivity. In the absence of consistent terminology and indexing, this will support more efficient and robust evidence synthesis in the field.     

Developing a geographic search filter to identify randomised controlled trials in Africa: finding the optimal balance between sensitivity and precision

Background: Research on identifying trials using geographic filters is limited. Objectives:  To test the sensitivity and precision of a filter to identify African randomised controlled trials (RCTs). Methods: We searched medline and embase for RCTs published in 2004 using a Cochrane filter for RCTs. The search was limited to HIV/AIDS but irrespective of location. Two investigators independently identified African RCTs from the retrieved records forming a reference set. We then repeated the search using an African geographic filter comprising country and regional terms forming the filter set. We compared the sensitivity and precision of the sets. Results: The medline reference set comprised 1799 records with 23 African RCTs; for embase , the reference set comprised 763 records with 37 African RCTs. The medline filter set comprised 180 records with 17 African RCTs; the embase filter set comprised 98 records with 27 African RCTs. Sensitivity of the filter was 74% (medline ) and 73% (embase ). Addition of the filter improved precision from 1.3% to 9.4% (medline ) and from 5% to 28% (embase ). Conclusion: The African filter improved precision with some loss in sensitivity. Incomplete reporting of trial location in electronic bibliographic records restricts efficiency of geographic filters. Prospective trial registration should alleviate this.     

Updated generic search filters for finding studies of adverse drug effects in Ovid medline and Embase may retrieve up to 90% of relevant studies

Background

The most current objectively derived search filters for adverse drug effects are 15 years old and other strategies have not been developed and tested empirically.

Objective

To develop and validate search filters to retrieve evidence on adverse drug effects from Ovid medline and Ovid Embase.

Methods

We identified systematic reviews of adverse drug effects in Epistemonikos. From these reviews, we collated their included studies which we then randomly divided into three tests and one validation set of records. We constructed a search strategy to maximise relative recall using word frequency analysis with test set one. This search strategy was then refined using test sets two and three and validated on the final set of records.

Results

Of 107 systematic reviews which met our inclusion criteria, 1948 unique included studies were available from medline and 1980 from Embase. Generic adverse drug effects searches in medline and Embase achieved 90% and 89% relative recall, respectively. When specific adverse effects terms were added recall was improved.

Conclusion

We have derived and validated search filters that retrieve around 90% of records with adverse drug effects data in medline and Embase. The addition of specific adverse effects terms is required to achieve higher recall.     

Patient-based benefit-risk assessment of medicines: development,refinement, and validation of a content search strategy to retrieve relevant studies

Introduction:Poor indexing and inconsistent use of terms and keywords may prevent efficient retrieval of studies on the patient-based benefit-risk assessment (BRA) of medicines. We aimed to develop and validate an objectively derived content search strategy containing generic search terms that can be adapted for any search for evidence on patient-based BRA of medicines for any therapeutic area.Methods:We used a robust multistep process to develop and validate the content search strategy: (1) we developed a bank of search terms derived from screening studies on patient-based BRA of medicines in various therapeutic areas, (2) we refined the proposed content search strategy through an iterative process of testing sensitivity and precision of search terms, and (3) we validated the final search strategy in PubMed by firstly using multiple sclerosis as a case condition and secondly computing its relative performance versus a published systematic review on patient-based BRA of medicines in rheumatoid arthritis.Results:We conceptualized a final search strategy to retrieve studies on patient-based BRA containing generic search terms grouped into two domains, namely the patient and the BRA of medicines (sensitivity 84%, specificity 99.4%, precision 20.7%). The relative performance of the content search strategy was 85.7% compared with a search from a published systematic review of patient preferences in the treatment of rheumatoid arthritis. We also developed a more extended filter, with a relative performance of 93.3% when compared with a search from a published systematic review of patient preferences in lung cancer.     

The development of search filters for adverse effects of surgical interventions in medline and Embase

Background

Search filter development for adverse effects has tended to focus on retrieving studies of drug interventions. However, a different approach is required for surgical interventions.

Objective

To develop and validate search filters for medline and Embase for the adverse effects of surgical interventions.

Methods

Systematic reviews of surgical interventions where the primary focus was to evaluate adverse effect(s) were sought. The included studies within these reviews were divided randomly into a development set, evaluation set and validation set. Using word frequency analysis we constructed a sensitivity maximising search strategy and this was tested in the evaluation and validation set.

Results

Three hundred and fifty eight papers were included from 19 surgical intervention reviews. Three hundred and fifty two papers were available on medline and 348 were available on Embase. Generic adverse effects search strategies in medline and Embase could achieve approximately 90% relative recall. Recall could be further improved with the addition of specific adverse effects terms to the search strategies.

Conclusion

We have derived and validated a novel search filter that has reasonable performance for identifying adverse effects of surgical interventions in medline and Embase. However, we appreciate the limitations of our methods, and recommend further research on larger sample sizes and prospective systematic reviews.