G4 – Evaluating the information retrieval quality and methodological accuracy of Systematic Reviews and Meta-analysis on congenital malformations (2004-2014)

Alicia F. Gómez-Sánchez1, Mar González-Cantalejo2, Gaétan Kerdelhué3, Pablo Iriarte4 and Rebeca Isabel-Gómez5

1Centro Nacional de Investigaciones Cardiovasculares. Madrid (Spain)
2Hospital Universitario Miguel Servet. Zaragoza (Spain)
3Rouen University Hospital / CISMeF team. Rouen (France)
4CHUV-Lausanne University Hospital. Lausanne, Switzerland
5Agencia de Evaluación de Tecnologías Sanitarias de Andalucía (AETSA). Sevilla. Spain.

Corresponding authors: Rebeca Isabel-Gómez, rebeca.isabel.ext@juntadeandalucia.es; Alicia F. Gómez-Sánchez, afgomez@cnic.es

Abstract

Introduction
Systematic reviews (SR) and meta-analysis (MA) aim to provide an in-depth summary of the literature of a research question, which must achieve some methodological requirements especially regarding how the information is retrieved and organized. There are several guidelines with recommendations for standard SRs or MAs. However, how often do those publications fulfil all the conditions to be considered SRs or MAs?
Objectives
Our aim is to check if articles using the terms ‘systematic review’ or ‘meta-analysis’ in the title accomplish the established requirements, focusing on search and methodology. The secondary objective is to observe if librarians have participated in a visible manner in the process.
Methods
We first created a checklist starting with some PRISMA points related to the literature search methodology and the documentation of the process. We added other common elements from the main methodological manuals for SR (including CRD, Cochrane, EUnetHTA, among others). Finally, we completed it with some items of the CADTH Checklist. Our final list consists of 20 evaluation criteria within the subject ‘congenital malformations’. To obtain the sample we searched in Medline/Pubmed and Embase for documents published between 2004 and 2014 and containing the terms SR or MA in their title. We limited languages to English, French, German, Italian and Spanish. We obtained 162 records after excluding duplicates and non-valid documents (letters, etc.). Once we obtain the full texts, we independently checked if the publications met our criteria. A second reviewer was consulted in cases of doubt.
Results
Among all the data, we highlight the following:

  • Around 80% do not show PICO’s questions, and around 60% specify bias
  • Information sources are explained in approximately 70% of the records. Around 60% describe the fully search strategies and nearly 50% combine electronic with manual searches
  • 20% of them use other additional sources or other types of documents
  • Around 30% use a thesaurus, and a similar number combines controlled vocabulary with natural language
  • Less than 10% of the studies mentioned a librarian

Conclusions
Although we cannot affirm that our sample is sufficiently representative, the fact remains that since most of the studies analysed are lacking in method and resources, and that is quite alarming.
Authors and publishers must bear in mind the existing guidelines. Additionally the involvement of information specialists would be a key factor in improving the quality of SR and MA.

Key words: first key word; Meta-Analysis, Systematic reviews, Congenital Abnormalities, Health Information Management, Library Services, Evaluation Research

Introduction
A systematic review (SR) is the synthesis of the best evidence available on a specific research question that meets pre-established criteria selection, in order to answer a specific research question. The planning and preparation of a SR requires the use of systematic and explicit methods to reduce the risk of bias; as well as transparent and explicit procedures to find, evaluate and synthesize the information[1]. SR methods have to provide reliable results that allow drawing conclusions and making decisions.

The main characteristics of SR[2] are:

  1. Rigorous, with a previously established systematic and reproducible methodology, which allows the evaluation of the validity of the included studies.
  2. Exhaustive, in order to identify the greatest amount of information available in different formats (not only articles but also other types of documents). This is indeed the aspect where literature search is a condition of particular relevance, given the fact that a correct and complete identification of the studies is the main point to meet the established criteria.
  3. Explicit, having a transparent, reproducible and clearly described methodology.
  4. Useful and instructive, as they must serve to answer specific questions, with a patient-centered approach or must help to develop health policies.

Many systematic reviews include meta-analysis (MA), which are statistical procedures where data from different studies are combined to provide a synthesis of the results. Both systematic reviews and meta-analysis should be written as guidelines to ensure transparency and reproducibility.

SR guidelines at-a-glance
There are several development institutions of SR and MA, such as Cochrane[3], Campbell, NICE[4] or the Centre for Reviews and Dissemination (CRD)[5]. These institutions develop their own methods and standards as reference guidelines to improve and assure the quality of SR and MA.
One of the most accepted by journals publishing SR and MA is the PRISMA Statement (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)[6], which is a list consisting of 27 items on key aspects when developing an SR or MA. However, we evaluated the guidelines from the institutes mentioned above and there are several aspects in common. The principal are:

  • Title, including the information of the type of article (systematic review, meta-analysis, or both).
  • Abstracts, that preferably should include an structured summary containing background, objectives, data sources, study eligibility criteria, participants, and interventions, methods, results including limitations, conclusions and implications of the key findings.
  • Methods, which should contain the eligibility criteria; the information sources; the search strategy for at least one database, including any limits used; the study selection and the data collection process; or the description of the methods used for assessing risk of bias of individual studies; among many others.
  • Results, where the studies screened should be fully described, with exact numbers, characteristics for the data extracted, presenting the data on risk of bias of each study; as well as other additional analysis.
  • Discussion, that should summarize the main findings, should include the limitations and conclusions.

Despite this wide range of guides, most periodicals establish compliance with PRISMA for accepting manuscripts SR and MA, and PRISMA is also adopted by an important part of the authors.

Objectives
In the recent years there has been a significant increase of articles containing the terms ‘systematic review’ or ‘meta-analysis’ in the title, in the main bibliographic databases. As an example, we performed a search in Medline looking for articles containing this terms in the title, and the results shown that the number increased exponentially from 1500 records in 2004, to around 13,000 references in 2014. The quantitative growth is obvious but we would like to know if also the qualitative criteria are accomplished.
Knowing the specific characteristics of SR and MA, and all the methodological guidelines published regarding the quality of these types of publications, our main question was if all those studies actually meet the quality criteria established by PRISMA or similar methodologies. Our first objective of this work is to clarify the extent of compliance with established standards, focusing mainly on methodological aspects and, more specifically, on those points related to the search and retrieval of bibliographic references. We also proposed other secondary objectives such as the analysis of the participation of librarians in the preparation of the articles analysed.

Methods
We first created a checklist, approved by all of us, starting with some PRISMA points related to the literature search methodology, and also including the documentation of the process. We added other common elements from the main methodological manuals for SR (including CRD and Cochrane). Finally, we completed it with some further items of the CADTH Checklist[7, 8]. Our final checklist consists of 20 evaluation criteria, that we checked applying them within the subject ‘congenital malformations’. To obtain the sample we searched in Medline/Pubmed and Embase for documents published between 2004 and 2014 and containing the terms ‘systematic review’ or ‘meta-analysis’ in the title. The corpus was limited to English, French, German, Italian and Spanish (the 5 languages the group was able to understand). We obtained 162 records after the exclusion of duplicates and non-valid documents (letters, conference abstracts, etc.). Once we obtain the full texts, we independently checked if the publications met our criteria. A second reviewer was consulted in cases of doubt.

Results
From the 162 references identified, 11.5% were identified as systematic reviews, 73% as meta-analysis, and 15.5% of the articles combine both.
As shown in table 1, the articles studied present a wide range of answers. Items regarding the transparency of methodology receive some of the most positive answers, such as the process for selecting the studies included in the MA/SR (73%), the availability of information sources (61.5%) the description, at least partial, of an electronic search strategy (59.8%). Furthermore 95.9% of all articles specify the databases used and 88.5% mention the languages used for searches. Nevertheless only a minority of articles shows a fully transparent and described search process (27%) meaning that despite the elements provided in the article, it is rarely possible to completely reproduce the search.
Risks of bias in the methodology are described in 50.8% of the articles. Missing one or several articles for an MA/SR due to the search process should also be considered as a risk of bias. In the article studied, languages bias could occur as 41.8% of the MA/SRs only use English and personal bias could also occur as only 32.8% of all MA/SRs involve at least two different people for the search and selection process.
The items regarding the quality of the search in itself, its technical aspects, receive the less positive answers such as the use of controlled vocabulary (30.3%), the use of truncation (10.7%) or the adaptation of search syntaxes to each databases (13.1%).



Table 1: Items ranked by positive answers

The different supporting documents and guidelines that serve to conduct MAs and SRs, such as PRISMA, acts as a guarantee regarding the transparency of the search process. Only 29.5% of all articles mention one. The involvement of an information specialist should improve the technical aspects of the search. They seem rarely involved as only 6.6% of the MA/SRs mention one. Figure 1 shows that articles mentioning a review protocol or an information specialist receive more positive answers than the others.



Figure 1: Percentage of positive answers for each item

Specifically regarding the searches and the databases used for them, 32.2 % of the MA or SRs run searches electronically; very few, only 1.6% use only manual searches; and manual searches frequently complete electronic searches in 56,6%. We also found 6.6% of the studies checked not giving this information.
In almost 50% of the studies only two or less databases are used and only 13% do adapt the search strategy to the different databases. Even if in almost 62% of the studies the information sources are shortly described, 68% of the studies do not describe in a transparent way the search process.



Figure 2: Databases used for the search and their distribution

Conclusions
One of the essential as1pects regarding the search for clinical evidence is that the databases and other sources that should be searched to identify evidence of clinical effectiveness depend on the review question. There are main core databases, as Medline or Embase, but additional databases and other resources should be searched, in order to achieve the best and accuracy results[4].
Although our sample is not really representative, the results obtained show that most of the studies analysed are lacking in method and resources, the search strategies are mostly not transparent and not reproducible enough. It would be highly recommendable that publishers bear in mind the existing guidelines and that they really set an evaluation system in order to ask authors to accomplish this guidelines.
Only 7% of the studies refer the collaboration of an information specialist. In our opinion, the involvement of biomedical librarians would be a key factor in improving the quality of SR and MA.

REFERENCES

  1. What is a systematic review? [Internet]. The Campbell Collaboration. [cited 2016 Apr 25]. Available from: http://www.campbellcollaboration.org/what_is_a_systematic_review/
  2. Urrútia G, Bonfill X. La declaración PRISMA: un paso adelante en la mejora de las publicaciones de la Revista Española de Salud Pública. Rev Esp Salud Pública. 2013 Apr;87(2):99–102.
  3. Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. Repr. Chichester: Wiley-Blackwell; 2012. 649 p. (Cochrane book series).
  4. The guidelines manual [Internet]. NICE Guidance. 2012 [cited 2016 Apr 25]. Available from:
    https://www.nice.org.uk/article/pmg6/chapter/1%2520introduction
  5. Undertaking systematic reviews of research on effectiveness: CRD’s guidance for carrying out or commissioning reviews [Internet]. University of York – Centre for Reviews and Dissemination. 2001 [cited 2016 Apr 25]. Available from: http://www.york.ac.uk/inst/crd/crdreports.htm
  6. PRISMA -Preferred Reporting Items for Systematic Reviews and Meta-Analyses [Internet]. 2015 [cited 2016 Apr 25]. Available from: http://prisma-statement.org/
  7. Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J. PRESS: Peer Review of Electronic Search Strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008.
  8. McGowan J, Sampson M, Lefebvre C. An Evidence Based Checklist for the Peer Review of Electronic Search Strategies (PRESS EBC). Evid Based Libr Inf Pract. 2010 Mar 17;5(1):149–54.
Print Friendly
LinkedIn0
Google+0