Hi again. This is Kay Dickersin in the class Systematic Reviews and Meta-Analysis. In the next section, Section B, I'm going to be talking about selection bias in doing the systematic review. Recall that we also talked about selection bias in the individual studies and this is also selection bias. But it manifests in a different way when we're doing a systematic review and meta analysis. Probably the most important selection bias or the way that selection bias manifests is a reporting bias, and there are many different types of reporting bias, and that's why I think in an earlier slide I called it reporting biases. I'm just going to talk about a few here. One of the most important reporting biases is publication bias. And this is the case where there are unpublished studies that tend to have different results than published studies. And we see that this is generally true. I'll talk a litttle bit more about it in a few minutes and why it's important to us. There's also selective outcome reporting, which we haven't known about or thought about as long as we have publication bias. That is, the results from a study can be published, but only some of the outcome's reported. For example, if one examine adverse events and there were some negative adverse events, and some that were very severe and some less severe, and only the less severe ones were reported then this would be called selective outcome reporting. In the next slide I add a slightly different type of reporting bias. And that is that studies that are easier to find may have different results from those that are harder to find. For example, we all know there are studies that are pretty easy to find, that is studies published in open-access journals, studies that are published in some of the larger journals that we all know. But it might be harder to find a journal that's not from the U.S. from conference abstracts, from books. These may be harder-to-find articles that have published but since they're harder-to-find, you have do some extra digging for your systemic review. And the final type of selection bias, that's different from a reporting bias, is what we're calling inclusion bias. That is, what if the author of the systematic review knows the results of many of the studies that are out there before beginning the systematic review, and the author sets the eligibility criteria for that systematic review. So the study is included or excluded, specifically. Or perhaps, those inclusion criteria make a difference in how the data are abstracted. This could affect the outcome of the systematic review and meta-analysis. And so, it's very important to consider inclusion bias as a potential meta-bias in systematic reviews and meta-analysis. Let me give you an example of inclusion bias. You all probably know that whether one should institute mammographic screening for breast cancer in women under 50 is an area of great controversy and has been for decades. These studies are so well known that there have been many systematic reviews and meta-analyses done by investigators who set inclusion criteria ahead of time, knowing perfectly well which randomized trials will be included or excluded. Or perhaps studies of other designs as well, because they know the results that they want to find Aand what the different studies do in terms of influencing the final result. That's a metabias that has to do with whether studies are included or excluded. So let's go back to the question of publication bias and whether all studies are published. The first question is, is there any failure to publish at all, and if so how big is that problem? So we know now from a number of similar studies, that studies regardless of their design, whether they're clinical trials or observational studies, may not always be published. You can see in this listing of various studies that have examined the likelihood of publication, a broad range of rates of publication. So for example, clinical trials that came through a Barcelona Hospital Ethics Committee show that only 21% were published by some reasonable date that they examined. On the other hand, studies that came through the Johns Hopkins School of Medicine Ethic Committee Were about 81% published, and clinical trails funded by the NIH in the late 1970s have a very high publication rate. About 93% of them were published. So you can see there's a broad range here, from 21% to 93% published that we initiated either as identified through an ethics committee or through a funding agency. People tend to use the number 50% publication. I don't think that's based on anything exact or even a summary estimate. It's just looking at the data we have and making a judgment. So for example, Johns Hopkins may have a higher publication rate than a small hospital. And so, there are differences not only in terms of where the studies were initiated, but also who funded them. There's been a wonderful systematic review done in 2009, 2010 by Fu Jen Song looking across a very large spectrum of studies examining reporting biases. And I'm just going to show you a few of his graphs and meta analyses. So what he did is he said lets look at all the studies that examine whether positive results are more likely to be published then non-positive results. And first we're going to start with what he called inception cohorts, and that is all cohorts of studies that either came through an ethics committee or came through some sort of funder. And so for example, he has about 15 studies here started either in the 80s or 90s or even 2000s. And over that period of time, you can see almost every single one has found that publication favors positive results. The one exception is the Stern study done in Australia that had a point estimate where non-positive results were favored, but the confidence interval does cross one and go into favoring positive. So, it's possible that those results include some of the results found by the other studies. Overall, the summary estimate for publication of positive results over non-positive results is about an honest ratio of 3, meaning that positive results are more likely to be published. Fu Jen Song also looked at what he called regulatory cohorts, and that is all studies that came through the FDA or other regulatory authorities. So look and see what percent of studies that were part of a package given by the drug companies to a regulatory authorities such as FDA, ultimately were published. And again publication favored positive findings by studies done by drug companies and submitted to the FDA or other regulatory authorities. The odds ratio in this case is about five although the confidence intervals are much wider because they are smaller studies and there are fewer of them. So bottom line is that there is a publication bias, and we have a tendency to publish positive results over non-positive results. That's not good news if you're doing a systematic review and meta analysis because how are you going to find those unpublished studies? Well, one might say, what if unpublished studies wouldn't influence a systematic review and meta analysis? What if they're not very good studies? What if they're so small they wouldn't have that much of an impact? Or what if there is a difference, but not that big a difference in terms of their findings? This has been examined in one study that I know of, where the authors looked at studies coming through the FDA. And they calculated the summary statistic for a series of studies of that topic using unpublished FDA data and without the FDA data. And what they found is sometimes the unpublished data increased the summary statistic and sometimes it decreased it. About 40-some percent each direction, and then, in the middle, there was about 7% with no change whatsoever. And so, one can't say that in the case of the FDA data, that the unpublished data always push the summary statistic in one direction because it's just not true. It's about half the time in one direction and half the time in the other direction. But we need to know more. This is just one study where the impact of unpublished results have been looked at a meta analysis. Most people remain quite scared that unpublished studies are going to affect their meta analysis and so the IOM standard said other standards recommend doing a comprehensive search for all studies, unpublished or not. There is a study, and we'll talk a little later in the course about clinicaltrials.gov, which is a register of clinical trials and observational studies to some extent that have been initiated. Since clinicaltrials.gov was authorized by law, there's also a requirement for studies that come through the FDA of drugs, biologics and devices, that they not only are registered, but their results are posted. This study came out in 2013 and showed that actually the results that are in cinicaltrials.gov are more comprehensive and easier to understand than the results published in journals for the same studies. If you look at the bottom of this slide, you'll see that half the trials with results posted on clinicaltrials.gov had no publication at all. So you would really need to go to clinicaltrials.gov to find out what the results for that study are. And that reporting was more complete at clinicaltrials.gov for a number of different topics that are important to a systematic review and meta-analysis. The flow of participants in the study, how well the intervention worked, adverse events associated with that intervention, and serious adverse events. And so bottom line here is, you want to go to the FDA and you also want to go to clinicaltrials.gov databases to see if there's any data in there about the studies that you would include in your systematic review to make sure you aren't missing important data, both for comprehensiveness and also that you aren't just getting positive results or a majority positive results by looking at published findings. Although people have been concerned about publication biases for many years, selective outcome reporting is something that's really only been noted and examined in the last decade. The first major report of selective outcome reporting came from Anlin Chan, who in 2004 published the first of a couple of studies examining whether selective outcome reporting existed. This study on this slide shows results from his JAMA study, where he followed up protocols that were submitted to two ethics committees in Denmark, one in Frederiksberg and one in Copenhagen. And he looked at the protocols and examined all that were associated with full publication. What he found was something that was shocking to all of us. And that is about two-thirds of the time the investigators changed what was in the protocol and listed as a primary outcome to what they reported as the primary outcome in the publication and vice versa. That is two-thirds of the time, the primary outcome was different when one compared the protocol and the publication. And we know from other courses you've taken and also this course that the primary outcome and designating it before you begin your study is very important, not only for selecting the sample size, but also for assuring the reader that you haven't been data dredging. So what you say is the primary outcome they believe is what you set ahead of time, which is very Important in inferences that can be drawn from your P-value, your statistical significance and other claims that you make as a result of your study. Just as importantly, he found that statistically significant findings had a higher likelihood of being reported than non-significant findings. So it appears that what one chooses as a primary outcome is related to whether it's found to be a statistically significant result or not. Well, this is scary, again, if we're talking about beneficial result such as survival. If we have a couple of different outcomes, death from all causes, death from a specific cause, and if we only report death from a specific cause, which had a positive effect of the intervention, and don't report death from all causes, which had no statistically significant benefit. So, if we change our primary outcome between the protocol and the publication, then we aren't telling the reader the truth about what we were initially looking for and what we found. So what Anlin Chan's results and other people since then have shown us is that we have to dig deeper than the publication if we want to know what was found for all the outcomes, both to find out what was the original primary outcome, and are they reporting that correctly? And what were the results for outcomes that were both reported and not reported? You can find these unreported outcomes sometimes in FDA database, which I've already mentioned. Or in clinicaltrials.gov, also mentioned. And sometimes you can find outcomes that weren't mentioned, and results that weren't mentioned in what we call the grey literature. So the grey literature we define as conference abstracts, unpublished data, that for example, appear in documents that are submitted by contractors to government institutes, book chapters, and other sources like pharmaceutical company data, letters, theses, et cetera. So, it turns out, and I'm not going to show you the data here because we looked at that in an earlier lecture, but only about half the time. Our study center reported an abstract form, such as to a professional society, reported fully in a journal article only half the time. So we want to look at those conference abstracts because it may include information about some very important studies. It turns out that one of the reasons that conference abstract studies are not reported in full, is related to the findings, whether they're positive or not. So, Sally Hopewell did a systematic review of the grey literature to look and see whether it might really make a difference in systematic reviews. Most systematic reviewers are reluctant to go back and search conference abstracts, search thesis, search book chapters because it's so much harder to do than say searching PubMed where we have databases at our fingertips. Conference abstracts often require going to actual books which are hard to locate, and book chapters, well where do you begin? What Sally Hopewell found in her systematic review is that studies that are published as opposed to remain in the grey literature tend more often to show a positive result compared to studies in the grey literature. And that this can affect the results of a meta analysis. Again, it was a small effect. But given the small studies that are done of this topic so far, many people are reluctant not to hand search conference abstracts and other grey literature when they do a meta-analysis, because it could possibly affect their systematic review and meta-analytic results. It's a lot of work, though, and something around which there's a great deal of debate. So finally when we think about selection bias, people think about the fact that there is often data missing from a particular study publication. And we already know that there's selective outcome reporting. But there may be other unpublished information as well. What we find is those doing systematic reviews often try to contact the original authors or the original investigators to get unpublished information. They may send an email. They may write them a letter in the old days. And there have been studies, as a matter of fact there has been five studies, that have examined whether it's worthwhile to contact authors to try to get unpublished information. This slide shows data from a systematic review published in the Cochran database. And it's a systematic review of five studies. A meta analysis was not done because the studies were probably too dissimilar. The study was done by Young and Hopewell. And Hopewell was the author of the previous study we looked at. What they found was this, that when a systematic review investigator writes an email to an investigator of an individual study, they're likely to get a better response than when they use a different method. There have been two studies on this topic and both showed the same thing. So, if you're doing a systematic review for this class, and you are concerned about some missing data or an outcome that was not examined apparently in the study that's published that you're including. You might want to correspond by email with the authors to ask whether they have the data. And we can talk to you, the TAs and the instructors can talk to you about how to do this in your email. And what to send them to make it as easy as possible for the original author to send you the data that you need. You could also write the original author to ask about clarification of the methods that they use. In particular, you may find if you include conference abstracts that you need a lot of information about how the study was conducted because conference abstracts can be so brief. Again, we will help you with how to write an email to ask for clarification, so that you maximize your chances for getting an answer. Authors of one study found sending repeated emails to the same person using the same methods really wasn't beneficial. That didn't increase the likelihood of getting the unpublished information you were seeking. If sending an email doesn't work, you might try telephoning. And if you have five things to ask the investigators, that theoretically won't influence their response based on the results of one study. So as long as they're getting out of their chair to get you the answer to one of your questions, they might as well get you the answer to all five. That's what it looks like. So, don't be afraid to ask for what you need when you're doing your systematic review. It's really hard to get data from drug companies about unpublished studies. And, it has been done, however, there are some methods people have used to get the data. But, I think very little is known even though there's one study that shows that knowing exactly what to ask for, what's the study. And then saying we'd like details on this particular study, does enhance your chances of getting a positive response. But just writing to or calling a drug company and saying, what are your unpublished studies on x, really doesn't work. So you have to be pretty specific, unless you know someone working in the area in which you're interested. So that's about it for selection bias. It's a topic I love, so you'll hear me talking about it quite a bit when I walk around one on one. But we're going to move on to other forms of meta-bias. Some of them you may find to be quite important in your systematic review as well.
