Hilde Stromme1, Gyri H. Straumann1, Ingvild Kirkehei1, Marita Heintz1 and Elisabet Hafstad1
1Norwegian Institute of Public Health, Oslo, Norway
Corresponding author: Hilde Stromme, hilde.stromme@fhi.no
Abstract (as submitted for rewiew by the programme committee)
Background: The Cochrane Library contains, among other bibliographic databases, The Cochrane Database of Systematic Reviews (CDSR), Database of Abstracts of Reviews of Effects (DARE) and Health Technology Assessment Database (HTA), and has for many years been the major source for systematic reviews. The relatively new open access bibliographic database Epistemonikos includes broad syntheses, systematic reviews, structured summaries of systematic reviews and primary studies included in reviews. The team behind Epistemonikos regularly search several sources, including CDSR, DARE and HTA, to identify systematic reviews to populate the database. Early in 2015 we did a small study to investigate whether or not Epistemonikos could replace The Cochrane Library when searching for systematic reviews. The results, which were presented at The Cochrane Colloquium 2015, showed that there were unique relevant hits in both databases indicating that both sources should be searched. Keeping in mind the limitations to the study, a very small sample size being the most important one, and that Epistemonikos is a database under development; we have initiated a new study with a larger sample size.
Objectives: To compare the results of non-complex searches for systematic reviews in Epistemonikos and The Cochrane Library using similar search strategies to see if Epistemonikos can replace The Cochrane Library for such searches.
Methods: We will compare the results of non-complex searches using PICOs from commissioned systematic reviews. We have searched Epistemonikos and The Cochrane Library (CDSR, DARE and HTA). We used the same text words in Epistemonikos and The Cochrane Library, but adapted the search strategies making the most out of available search options in both sources. All search results have been exported to EndNote. In EndNote, we will identify duplicates, unique hits and unique relevant hits from Epistemonikos and from The Cochrane Library.
Results: Results from the previous study showed that there were many duplicate references within Epistemonikos itself. There was a substantial overlap between Epistemonikos and The Cochrane Library. However, we found unique relevant hits in both Epistemonikos and The Cochrane Library. Results from the new study will be available in time for the conference.
Keywords (MeSH): Information Storage and Retrieval; Databases, Bibliographic; Databases as Topic; Comparative study
Introduction
Since its launch in 1996 (1) The Cochrane Library has been a major source of systematic reviews. Epistemonikos, which was launched in 2012, aims to be a one-stop-shop for the most relevant information for health decision-making (2). Epistemonikos is populated by regular systematic searches for systematic reviews, broad syntheses or structured summaries in 26 different sources including The Cochrane Database of Systematic Reviews (CDSR), Database of Abstracts of Reviews of Effects (DARE), HTA Database (HTA), PubMed, LILACS Embase, and PsycINFO (3).
Early in 2015 we did a small study to investigate whether or not Epistemonikos could replace The Cochrane Library when searching for systematic reviews. The results, which were presented at The Cochrane Colloquium 2015, showed that there were unique relevant hits in both databases indicating that both sources should be searched (4). Keeping in mind the limitations to the study, a very small sample size being the most important one, and that Epistemonikos is a database under development; we concluded that further studies with larger sample sizes were needed.
Objectives
To investigate whether Epistemonikos can replace The Cochrane Library when searching for systematic reviews, by comparing the results of similar search strategies in both sources.
Methods
We compared a sample of 57 searches on a wide range of topics including rehabilitation, mental health, geriatrics, paediatrics, surgery, neurology, cancer, emergency medicine, substance abuse, migration, health promotion and organization of health care (see table 1 for full list of topics).
The searches were divided between seven information specialists who did between five and twelve each. We searched The Cochrane Library (CDSR, DARE and HTA) and Epistemonikos (results limited to the publication types Broad Synthesis, Systematic Review and Structured Summary). We used the same text words in both sources but constructed the strategies to make the most out of available search options such as proximity operators and MeSH-terms in The Cochrane Library. The searches were run in November and December 2015. Due to loss of data, one search was rerun in both databases in April 2016. All results were exported to EndNote.
Five information specialists (the authors) analysed the data. For ten searches that yielded several hundred hits, we made a cut off close to 200 hits that included all references in a time span covering full years (e.g. 2010-2015). The same time span was then used to include hits from the other database to ensure that the data were comparable.
We identified internal duplicates, overlap, unique hits and relevant hits. First, we identified internal duplicates (i.e. duplicates within Epistemonikos and within The Cochrane Library). Each systematic review was counted only once. This means that if a reference to a systematic review was found more than once in one of the databases, it was considered an internal duplicate. When we found a reference to a systematic review and a structured summary of the same systematic review in the same database, the summary was considered an internal duplicate. This method was also applied when we calculated overlap between databases and unique hits from each of the sources. Finally, we assessed all hits for relevance.
Results
After removing internal duplicates, we identified 5081 systematic reviews. Of these 1738 (34%) were unique Cochrane Library hits and 2830 (56%) were unique Epistemonikos hits. 513 of the systematic reviews were found in both sources, resulting in an overlap of 10%. See figure 1.
Of the identified systematic reviews, 626 were relevant. 181 (30%) of these were unique hits from The Cochrane Library and 246 (39%) were unique hits from Epistemonikos. The overlap was 199 (31%). See figure 2.
When looking at the 4455 irrelevant hits, 1557 (35%) were unique Cochrane hits and 2584 (58%) were unique Epistemonikos hits. The overlap was 314 (7%). See figure 3.
See table 1 for information about total hits (after removal of duplicates), hits and unique hits per database for each search.
Discussion
As in our previous study, we found a substantial overlap between The Cochrane Library and Epistemonikos, but also unique relevant hits in each of the sources. Another study performed by Rathbone et al. (5) compared the performance of seven databases including The Cochrane Library and Epistemonikos in identifying relevant systematic reviews of interventions for hypertension. They concluded that relying on one single source is not sufficient to identify all relevant systematic reviews.
We found a larger (58%) proportion of irrelevant hits in Epistemonikos than in The Cochrane Library (35%). We believe this may be due to the search functionalities in the databases. The Cochrane Library allows for proximity operators making more precise searches possible, whereas in Epistemonikos one has to search for quoted phrases or combine terms with the AND operator. Another reason for the large proportion of irrelevant hits in Epistemonikos may be that Epistemonikos, to a larger degree than Cochrane, includes systematic reviews on questions other than effect, such as prognosis and patient experiences. However, we did not investigate whether or not the irrelevant hits were systematic reviews answering other types of questions.
Based on the information that the Epistemonikos team searches CDSR, DARE and HTA regularly, one should expect to find all systematic reviews and structured summaries from these databases in Epistemonikos. However, we found a large proportion of unique hits in The Cochrane Library. We did not investigate whether this was because these references were not in Epistemonikos or if our search strategies were unable to identify them.
The main limitation to our previous study was the small sample size (N=13). The sample size of this study was much larger (N=57), but we did not perform a power analysis to estimate the sample size required to obtain statistical power. As in our previous study, assessment of relevance was not done by two people independently, and we were not blinded with regards to which database the results came from. A strength to both our studies is that we included a wide range of topics.
Conclusion
For all of our searches we found unique hits in both sources. Based on that we conclude that Epistemonikos cannot replace the Cochrane Library when searching for systematic reviews.
Tables and figures
Figure 1: Total hits (internal duplicates removed)
Figure 2: Relevant hits (internal duplicates removed)
Figure 3: Irrelevant hits (internal duplicates removed)
Table 1: Topics and hits for all searches
*Total number of systematic reviews after removal of duplicates within and between databases
**Original search had more than 200 hits in one or both sources
| Topic | Total* | Overlap | Relevant overlap | Unique Cochrane | Relevant unique Cochrane | Unique Epistemonikos | Relevant unique Epistemonikos |
|---|---|---|---|---|---|---|---|
| ADHD adults, psychoeducational intervention | 26 | 0 | 0 | 26 | 1 | 0 | 0 |
| Deductibles for patients in substance abuse treatment** | 178 | 0 | 0 | 7 | 0 | 171 | 0 |
| Effect of massage on recovery in athletes | 18 | 2 | 1 | 4 | 0 | 12 | 3 |
| Stereotactic radiotherapy versus surgery in early stage of lung cancer | 18 | 6 | 5 | 1 | 1 | 11 | 5 |
| Effect of interventions to promote occupational health in people with musculoskeletal disorders | 244 | 71 | 38 | 93 | 26 | 80 | 21 |
| The effects of psychosocial teams in primary care | 105 | 1 | 0 | 97 | 2 | 7 | 1 |
| Forced medication in mental health services | 50 | 11 | 0 | 35 | 0 | 4 | 0 |
| Effect of stretching after treatment with Botulinum toxin type A in adult patients with spasticity | 25 | 2 | 0 | 22 | 0 | 1 | 1 |
| Access to infection medicine specialists in hospitals | 117 | 1 | 0 | 115 | 0 | 1 | 0 |
| Observation of suicidal patients: patients’ experiences and effects on survival | 11 | 0 | 0 | 11 | 1 | 0 | 0 |
| Conditions with long lasting pain or fatigue** | 235 | 40 | 24 | 54 | 20 | 141 | 64 |
| Effects of interventions for elderly people with cancer or life style induced disease and comorbidity | 241 | 22 | 1 | 79 | 5 | 140 | 7 |
| Fluid bolus | 33 | 6 | 0 | 21 | 2 | 6 | 2 |
| Associations between caring for chronically ill children and parents’ immune response | 57 | 5 | 0 | 35 | 2 | 17 | 2 |
| Relationship between occupancy rate in hospitals and mortality / serious adverse events | 20 | 0 | 0 | 12 | 5 | 8 | 1 |
| Effects and cost -effectiveness of screening for colorectal cancer | 285 | 87 | 61 | 49 | 16 | 149 | 17 |
| Does financing health services with activity based funding versus block grant affect patient satisfaction? | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Effects of care pathways in mental health care | 6 | 0 | 0 | 6 | 3 | 0 | 0 |
| Prevention of neonatal Group B Streptococcus infection: universal or risk-based strategy | 75 | 5 | 3 | 66 | 8 | 4 | 1 |
| Gender differences in effects and adverse effects of drugs** | 271 | 6 | 0 | 48 | 1 | 217 | 12 |
| Early supported discharge after stroke | 35 | 7 | 7 | 11 | 2 | 17 | 11 |
| Health promotion for the elderly** | 188 | 0 | 0 | 0 | 0 | 188 | 7 |
| Effect of Positive Expiratory Pressure-therapy in critical care | 95 | 19 | 5 | 19 | 5 | 57 | 3 |
| Effects of volunteers as conversational partners in the prevention of psychosocial health problems** | 236 | 18 | 9 | 50 | 3 | 168 | 4 |
| Milieu therapy for inpatients in mental health facilities | 13 | 2 | 2 | 1 | 1 | 10 | 0 |
| Effect of orthopaedic insoles on hypermobility and gait development in children with instability of the ankle and foot joints | 5 | 0 | 0 | 4 | 0 | 1 | 0 |
| Effect of targeted preventive interventions for people with a psychiatric diagnosis** | 216 | 3 | 3 | 169 | 4 | 44 | 9 |
| Effect of deductibles on demand for health** | 178 | 0 | 0 | 7 | 1 | 171 | 0 |
| Effect of free choice of treatment provider | 57 | 0 | 0 | 18 | 0 | 39 | 0 |
| Post-exertional malaise or post exertional neuroimmune exhaustion in chronic fatigue syndrome/myalgic encephalomyelitis (CFS/ME) | 1 | 0 | 0 | 0 | 0 | 1 | 1 |
| Team based skills training | 240 | 21 | 6 | 66 | 1 | 153 | 12 |
| Interventions to preserve lung function in patients with acute cervical spinal cord injury | 30 | 2 | 2 | 28 | 5 | 0 | 0 |
| Teledermatology for diagnosis of skin cancer | 9 | 3 | 3 | 4 | 1 | 2 | 1 |
| Effect of primary / secondary preventive information campaigns about alcohol and health aimed at adults** | 264 | 51 | 1 | 69 | 3 | 144 | 8 |
| Inappropriate or unnecessary use of psyshotropic drugs | 21 | 5 | 3 | 4 | 1 | 12 | 5 |
| Volunteering – return to ordinary work | 152 | 2 | 0 | 101 | 3 | 49 | 1 |
| Health Literacy in migrants | 13 | 0 | 0 | 3 | 1 | 10 | 3 |
| Work-related heallth risks for labour migrants | 18 | 0 | 0 | 1 | 0 | 17 | 3 |
| Can MR replace CT in surveillance imaging for lymphoma | 16 | 6 | 4 | 1 | 0 | 9 | 4 |
| Does repeated CT-scanning incease the risk of other cancer types** | 241 | 12 | 1 | 10 | 1 | 219 | 0 |
| Effects of acupunture for patients with musculoskeletal disorders | 104 | 11 | 1 | 49 | 10 | 44 | 8 |
| Regeneration of spinal cord injuries | 130 | 2 | 0 | 2 | 0 | 126 | 0 |
| Physical activity and exercise after ileostomy- or reservoir surgery | 12 | 1 | 0 | 6 | 0 | 5 | 0 |
| Sexuality after ileostomy- or reservoir surgery, or for patients with gastrointestinal cancer | 82 | 4 | 1 | 19 | 0 | 59 | 3 |
| Effect of immunoglobulin for the treatment of myalgic encephalomyelitis | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| Low dose naltrexone for myalgic encephalomyelitis | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Information for increasing quality of life for people diagnosed with a rare disease | 6 | 0 | 0 | 6 | 1 | 0 | 0 |
| Effect of integrated healthcare vs. standard care for patients with multimorbidity | 37 | 7 | 0 | 3 | 0 | 27 | 1 |
| Benefit from patient organisations for cancer patients | 111 | 16 | 15 | 50 | 6 | 45 | 8 |
| Can the OQ-45 questionnaire form be applied to assess and improve the quality of treatment? | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Cancer rehabilitation in primary care | 96 | 8 | 0 | 43 | 26 | 45 | |
| The effect of co-localisation of primary care services** | 157 | 5 | 0 | 24 | 2 | 128 | 7 |
| Prevention of musculoskeletal pain in children and adolescents | 12 | 1 | 0 | 3 | 1 | 8 | 3 |
| Dental composite resins; safety and longevity | 16 | 0 | 0 | 12 | 7 | 4 | 0 |
| Liverpool Care Pathway | 8 | 1 | 0 | 1 | 1 | 6 | 4 |
| Effect of municipal emergency care beds | 115 | 1 | 0 | 90 | 1 | 24 | 1 |
| Effect of preventive use of compression stocking in people in need of care | 151 | 40 | 3 | 83 | 1 | 28 | 2 |
| Sum | 5081 | 513 | 199 | 1738 | 181 | 2830 | 246 |
REFERENCES
- History [Internet]. [s.l.]: Cochrane Community [updated December 12, 2013; cited April 25, 2016]. Available from: http://community-archive.cochrane.org/about-us/history
- Rada G, Perez D, Capurro D. Epistemonikos: a free, relational, collaborative, multilingual database of health evidence. Studies in health technology and informatics. 2013;192:486-90.
- Epistemonikos database methods [Internet]. Santiago: Epistemonikos [updated 2016; cited April 25, 2016]. Available from: http://www.epistemonikos.org/en/about_us/who_we_are
- Strømme H, Hafstad E, Heintz M, Straumann GH, Jamtvedt G. Finding systematic reviews: Can Epistemonikos replace The Cochrane Library? Poster presented at 23rd Cochrane Colloquium; Vienna, Austria 3-7. October 2015.
- Rathbone J, Carter M, Hoffmann T, Glasziou P. A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. Systematic reviews. 2016;5(1):27.