Systematic Reviews

Explicitly state your research question, determine which databases you will search, and determine your inclusion/exclusion criteria for studies that you find. Here is some information on writing a protocol for your systematic review study. You might want to search PROSPERO, a database of protocols, to make sure that no one else is currently working on a review on the same topic. You can also submit your protocol to PROSPERO.

• Break down your research question into smaller concepts in order to make the next few steps manageable.
• You may find it helpful to document the next few steps using a table in Word or Excel. For example, if your research question is PICO-formatted, you might start a table that looks like this:

  • Concept
    Patient or population
    Intervention or indicator
    Comparison or control
    Outcome

  • If your systematic review was on this topic: Does hormone replacement therapy increase the risk of cardiovascular disease in post-menopausal women? Your table might now look like this:

    • Concept
      Post menopausal women
      Hormone replacement therapy
      No therapy
      Cardiovascular disease

  • At this point you may want to critically look at each aspect of your PICO and remove anything that doesn't need to be a part of your search strategy. In this case, we can remove the comparison ("no therapy"). In other cases, you may need the comparison for a specific control (e.g. Drug A vs. Drug B). It's typically better to be broader at this point. Now our table might look like this:

    • Concept
      Post-menopausal women
      Hormone replacement therapy
      Cardiovascular disease

• Now we want to fill out the other columns on this table. The next step is to figure out any potential synonyms or conceptually equivalent ideas for each of our concepts. This will help us perform an exhaustive search using all possible terminology. If you have already identified any papers that you plan to include, it's helpful to scan their abstracts for any obvious keywords that should be included. Don't forget that there may be spelling variants (estrogen vs. oestrogen)!
• Here are two tricks that might save you some time.
  • You can truncate words to retrieve multiple word endings. The truncation symbol is usually an asterisk (*) but can sometimes be another character - check the database's Help documentation to determine what it is for any particular database. For example, inflam* will retrieve the following keywords in the results: inflammation, inflammatory, inflamed, inflaming...etc. This can be helpful when searching for plurals.
  • You can phrase search for exact phrases. For example, searching "post menopausal" in quotes will retrieve different results than searching post menopausal. Without quotes, most databases will look for the word post anywhere in the record and the word menopausal anywhere in the record, which may result in unwanted results. With quotes, you'll only see the exact phrase "post menopausal."
  • Now your table might look something like this:

    • Concept	Keywords
      Post-menopausal women	"post menopause" "post menopausal" postmenopause postmenopausal
      Hormone replacement therapy	"hormone therapy" "hormone replacement therapy" estrogen oestrogen progesterone
      Cardiovascular disease	cardiovascular atherosclerosis hypertension "heart failure" arrythmia stroke "myocardial infarction" "heart attack" ...etc.

• Controlled vocabularies are also often called subject headings or thesauruses. They are pre-set tags intended to organize a collection—in this case, the collection of article records inside a database. You've probably seen them before when searching for journal articles. Here are some examples of what they look like on article records (click to enlarge image):
  • ​
• At this point you'll need to decide which database you are going to search first and complete your search strategy using only that database's syntax. It can be a little confusing to try to switch back and forth between databases while still trying to finalize your search. For our example we will proceed using PubMed, but the techniques are the same in any database that offers controlled vocabulary searching.
• In PubMed, the controlled vocabulary is called MeSH. Here is a tutorial that explains how powerful and functional MeSH is. I recommend completing this tutorial to have a good understanding of the next few steps.
• Now that you've been introduced to MeSH, let's put it to work for our sample search. We're going to search for each of our concepts in the MeSH database to determine the appropriate MeSH heading for each of them. Now is also a good time to go back to any papers that you've already identified as relevant and see if there are any MeSH terms on those papers that you might want to use (to find the MeSH terms, search the papers in PubMed and look at the bottom of each article record).
• In the MeSH database, search for your concepts. Click on any relevant terms. From the MeSH database record, you can click "Add to Search Builder" to add the term into the PubMed search box. Then you can copy and paste it into your table.
• Now our table might look like this:

  • Concept	Keywords	MeSH headings
    Post-menopausal women	"post menopause" "post menopausal" postmenopause postmenopausal	"Postmenopause"[Mesh]
    Hormone replacement therapy	"hormone therapy" "hormone replacement therapy" estrogen oestrogen progesterone	"Hormone Replacement Therapy"[Mesh]
    Cardiovascular disease	cardiovascular atherosclerosis hypertension "heart failure" arrythmia stroke "myocardial infarction" "heart attack" ...etc.	"Cardiovascular Diseases"[Mesh]

• From the MeSH tutorial you'll recall that MeSH is also a taxonomy—that is, each MeSH heading has a relationship to all the other headings. You can see this in the MeSH record. In this example for "Cardiovascular Diseases", you can see that there are many more narrow terms that describe specific cardiovascular diseases. By using the broader term "Cardiovascular Diseases," we are automatically including all of those narrower concepts like "Hypertension," "Arrhymias, Cardiac," etc.
  • Note that this behavior of including narrower terms is usually called "explosion." MeSH explodes by default, but other controlled vocabularies might not. Look for the option to explode in other databases.
• Note that this whole step will need to be repeated for each database that you search, because other databases will have their own controlled vocabularies.
  • For databases that do not use controlled vocabularies (such as Web of Science), you can search with only the keyword column.

• Now we are ready to construct a search using all the terms we've listed in the table. Since we put each concept on its own row, all the terms in the same row are conceptually equivalent, so we will combine them with the Boolean operator OR. You can do this in the PubMed Advanced search builder (or any database search builder if you're not in PubMed), or you can type it out. Even if the database allows you to save search strategies, it's good to document the searches in a file separately from the database.
• At this point you will want to determine which field you want to use to search the words you've listed in the Keywords column. This will differ depending on the database. For PubMed, you will probably want to use either the Title/Abstract field (which searches those two fields) or the Text Word field (which searches title, abstract, MeSH terms and subheadings, and chemical substance names). We'll proceed using Text Word for our keywords.
• Now our table might look like this:

  • Concept	Keywords	MeSH headings	Search strategy
    Post-menopausal women	"post menopause" "post menopausal" postmenopaus*	"Postmenopause"[Mesh]	"post menopause"[Text Word] OR "post menopausal"[Text Word] OR postmenopaus*[Text Word] OR "Postmenopause"[Mesh]
    Hormone replacement therapy	"hormone therapy" "hormone replacement therapy" estrogen oestrogen progesterone	"Hormone Replacement Therapy"[Mesh]	"hormone therapy"[Text Word] OR "hormone replacement therapy"[Text Word] OR estrogen[Text Word] OR oestrogen[Text Word] OR progesterone[Text Word] OR "Hormone Replacement Therapy"[Mesh]
    Cardiovascular disease	cardiovascular atherosclerosis hypertension "heart failure" arrythmia stroke "myocardial infarction" "heart attack" ...etc.	"Cardiovascular Diseases"[Mesh]	cardiovascular[Text Word] OR atherosclerosis[Text Word] OR hypertension[Text Word] OR "heart failure"[Text Word] OR arrythmia[Text Word] OR stroke[Text Word] OR "myocardial infarction"[Text Word] OR "heart attack"[Text Word] OR "Cardiovascular Diseases"[Mesh]

• Now our search is almost ready to test. We combine each row's search strategy with the Boolean operator AND. You can do this in the search builder, or type it out (but don't forget to put parentheses around each row's search)!
  • ("post menopause"[Text Word] OR "post menopausal"[Text Word] OR postmenopaus*[Text Word] OR "Postmenopause"[Mesh]) AND ("hormone therapy"[Text Word] OR "hormone replacement therapy"[Text Word] OR estrogen[Text Word] OR oestrogen[Text Word] OR progesterone[Text Word] OR "Hormone Replacement Therapy"[Mesh]) AND (cardiovascular[Text Word] OR atherosclerosis[Text Word] OR hypertension[Text Word] OR "heart failure"[Text Word] OR arrythmia[Text Word] OR stroke[Text Word] OR "myocardial infarction"[Text Word] OR "heart attack"[Text Word] OR "Cardiovascular Diseases"[Mesh])

Here are some techniques you can use to improve your search results

1. Go back to your protocol. Can you apply any of your inclusion and exclusion criteria to your search, for instance, English-only papers or a particular date limit?
2. Use any papers that you have already identified independently of your search, and check to see if they are coming up in this search result set. If for example I had previously known about this paper, PMID 26414962 (PMID is a unique identifier for each article in PubMed), I could use the Advanced Search Builder to make sure that that particular paper showed up in my search strategy's results. By clicking "Add" next to the search strategy and combining it with AND for this paper's PMID, you can see if that PMID is included in the total set of results. Click these images to enlarge:
   
   
3. Sort your results by relevance and try to identify any particular relevant papers in the first few pages of results. If you find any good ones, scan their abstracts and MeSH terms to make sure you didn't miss anything important from your own search strategy.
4. Sort your results by date of publication and try to identify any particularly irrelevant papers. If you are seeing thousands of results that are utterly off topic, click into them and see if you can figure out why they are coming up. Some common issues include: a broad MeSH term exploding to include very irrelevant terms, keywords that are too broad and used too commonly by authors, or unexpected results of truncation. In PubMed, you can always look at the "search details" box on the results page to see how PubMed translated your search commands.