➡ Dr. Andrew Kaufman discusses the importance of critical appraisal in scientific research, a skill he learned during his residency at Duke University. He warns that over half of all published scientific research may contain false conclusions due to biases and inconsistencies. These biases can occur in various ways, such as through the influence of a study coordinator or the selection and enrollment of study participants. Therefore, it’s crucial to approach scientific papers with skepticism and thoroughly examine the methods section to understand the experiment conducted and ensure the conclusions match the findings.
➡ In recent years, scientific publications have started moving the methods section of their papers to supplementary sections, making it harder for casual readers to understand the full context of the research. This can lead to misunderstandings and misinterpretations of the findings. It’s important to scrutinize these papers and understand the experiments being conducted. However, it’s also crucial to be skeptical and double-check information, as even AI programs can make errors. In science, unlike engineering, there’s no built-in accountability, so reproducibility of results is key to validate research. Unfortunately, many attempts to reproduce results have failed, possibly due to sloppy work, unseen factors, or even deliberate falsification of data. It’s essential to approach research with skepticism and be aware of potential pitfalls, such as misleading statistics.
➡ When researching scientific topics, it’s important to understand the specific terminology used in scientific papers. Databases like PubChem and PubMed can be useful for finding information on chemicals and medical articles, respectively. However, understanding the limitations of different types of studies, such as in vitro (lab-based) and in vivo (living organisms) studies, is crucial. To critically appraise information, identify the claim or question, seek multiple sources of information, and examine the quality of these sources, understanding the difference between correlation and causation.
➡ When analyzing information, it’s crucial not to confuse correlation with causation, and to consider other factors that could be the cause. It’s also important to examine patterns from different perspectives, looking for potential biases, such as financial interests or special interest groups. This critical appraisal method can help make better decisions in various life areas, including health. The speaker encourages the audience to practice this method and become their own health authority.

And it shows that more than half of all published research findings in science are actually false. So any paper that you pick up and read has a more than 50% chance of having false conclusions. This is the true health report, where critical appraisal fuels true freedom. Hello everyone, and welcome. I’m Dr. Andrew Kaufman, and today I’m going to talk about how I do research for this podcast and to formulate my opinions on health and medical related topics. And this technique that I use could be loosely called a critical appraisal. Now, a critical appraisal is something that I originally actually learned in my residency training in psychiatry at Duke University.

And this was a required technique that we learned and then had to utilize on a weekly basis at a mandatory journal club. And in this journal club, we would be assigned clinical articles. Usually they were clinical studies of, for example, antidepressant drugs or perhaps of a type of psychotherapy. And we would in advance, individually have to critically appraise these articles and then present them to the group. And they would ask questions and give feedback, and we would have a discussion about the article. Now, this procedure was really for the purpose of rooting out what study that’s published is actually worth following in your own clinical practice.

Like, what are the takeaway messages? Can we actually learn something? Does this really show that a treatment is effective, for example? Now, I want to give a personal example that I also witnessed while I was at Duke University, but I’m not going to mention anyone’s name specifically, so you could understand a little bit of how bias can enter into a scientific paper. And it can be very, very difficult to read the paper and know that this bias is present. And so there are two main aspects that this can be done that I’m aware of, that I’m going to talk about today.

So one of these has to do with the study coordinator, and the other has to do with how study participants are selected and enrolled in the study. So the study coordinator is the one you know, and it’s kind of like a secretary type position, but it’s the individual that actually coordinates with all the study participants, like checks in with them, reminds them when they have to come to be evaluated as part of the study, or reminds them to take their medicine or whatever the intervention is for the study. And if this individual is like, super friendly and accommodating, then it will actually cause the study participants to have a better clinical outcome.

And if the study coordinator is more lax and hands off, then they might not even show up or might not do as well. And this is kind of a hidden bias that you can’t really tell. And you wouldn’t see it until you talk to researchers and see how the mechanisms work on a day to day basis and then realize why a researcher hiring a study coordinator for clinical research, that’s a very important position. They know they need to get someone who is good to have the right results. Now the other is a little bit more severe.

And I’ve seen this happen. And this has to do with how the participants are enrolled into the study. And this is often junior researchers that are given a chance to be involved in this type of research, they’re given quotas to recruit study participants because, you know, obviously you can’t complete the study unless you have enough people to participate. And they can only participate by being a volunteer. But they have to meet certain specific requirements. You know, they have to have the right diagnosis. They can’t have other severe illnesses in psychiatry studies many times, like they can’t have issues with addiction, for example, which is very common that people, you know, who are depressed or have anxiety issues also have substance abuse issues.

So it can be difficult to recruit someone who’s qualified for the study. And so this is what I’ve seen that there was a study, I believe it was called the SAD heart study, trying to evaluate an antidepressant drug for depression in people who had previously had heart attacks. Okay, so it’s a very specialized study, obviously, because all the participants had to have prior heart attacks. And it, you know, the idea of the study is that maybe depression and heart disease are related in some way. So in order to do that, we’d have to find people who, you know, are clinically depressed or meet the criteria for depression, as well as have this cardiac history.

And I was on a consult team. So that’s where we get called by doctors who are taking care of patients in the hospital for a psychiatric issue. So, like the patient is on a medical service, they’re there for heart disease or something medical. But then they report being depressed or they get confused and delirious and they call the psychiatrist like me to come and evaluate. And I had evaluated this gentleman and he did have a previous heart attack. And, you know, the initial reason for the consultation was depression. But when I interviewed him, I found out that he wasn’t actually depressed.

He had a difficult family situation. His kind of deadbeat son had shown up out of the blue at his trailer. He lived in a trailer with his girlfriend and they forced themselves in and were staying there. I mean, he was nice. He Said they could stay for a couple days, but they far outstayed their welcome. And they were engaging in partying lifestyle and there was all kinds of chaos. And it was essentially driving this guy crazy. And he didn’t know what to do. And he manifested some chest pain and went to the hospital and got admitted.

And they found out that his heart was okay and got us involved as a psychiatrist. Now, this is a situation where someone has a stressful crisis and they can’t deal with it. And what do they really need to help with that is the proper advice to set limits, maybe even have the police come and kick those deadbeats out of his house. But it’s not really a psychiatric issue that he has depression. It’s a, you know, it’s an interpersonal conflict issue. And you could see by talking with him that if you were able to solve the problem with a son, that there would be no need for any psychiatric treatment further than that.

And I basically told my supervising doctor about this situation, and what happened next was kind of surprising. I was told that I didn’t need to do any more follow up with that patient and that they would take care of it. And I found out what happened is that that patient was enrolled in the sad heart study, even though they clearly didn’t have depression. And what would have happened is that they would enroll in the study, then the study coordinator would help them fix the situation with their problem family members. They would get better, and then it would look like they got better as a result of an antidepressant drug, when that’s not really what happened at all.

And then this would get reported in the clinical study that would show some kind of favorable result for the antidepressant. And of course, if you’re just reading the study, you wouldn’t really be able to tell that all this went on and influenced the results. So we have to be very vigilant when we are examining scientific and medical research to make sure that we can as much as possible, detect the biases and inconsistencies and make sure that the experiment done and the conclusions that were reached actually match each other. Now, unfortunately, many of these articles don’t hold up to this level of scrutiny, which is why it’s very important to have an overall position of skepticism that whenever you are reading a paper, your starting position should be that whatever the findings are, they are not actually representative of the research conclusions.

So we can look at a paper, for example, done by the famous Stanford epidemiologist, and it was published in plos One, and it shows that more than half of all published research findings in science are actually false. So more than half. So any paper that you pick up and read has a more than 50% chance of having false conclusions. So we have to examine papers from this point of view. Now, one of the most critical things about evaluating a scientific paper is that you have to read the methods section. And I would suggest reading that first, or at least right after you read the abstract, because the methods section actually describes the experiment that was done that you’re reading about now.

It used to be that when articles were published, the methods section was right at the beginning, right after the introduction. So you’d have the abstract, which is like a one paragraph summary. @ the top you’d have the background or introduction section, which would give, you know, the background information, the context why this research study is important, and then it would go right into the method section where it describes exactly how the experiments were done. And it should be of sufficient detail that you can reproduce the experiment. And that’s a critical thing I’ll get to in a minute.

But what has happened in the last 10 years is that the scientific publications, even though the publications are not in paper, so they’re not actually taking up space anymore, but what they’ve done is that they’ve taken the method section out of the main paper in many, many instances, not universally 100%, but this has become the rule. And then they will publish the method section in another part of the journal, right, as like an appendix or a section with further details, a supplemental section. Like they’ve always had supplemental sections. And sometimes those would include like big long data tables or things that were not part of the main results of the study, but were still interesting.

But now they actually put the method section there. So the only people that really go to that supplementary section are folks that do research in that same area, such that they want that level of detail more. Casual readers generally won’t go there, so they might not even notice the method section is missing. And they’ll take away the results in the conclusion without even knowing what experiment exactly was done or how it was done. And this really can mislead you in interpreting research findings. And it is critical that you have to know what the experiment being done is.

So, for example, I’ve read many papers where they claim to be doing an experiment with the virus, for example, but in the method section even they just say that they had a vial of virus X. In other words, there was no information about where that came from or how they Know what it is. It was just stated that they have this thing. Now sometimes there’s a reference to another paper where, where it might have been described, where that came from. But if you don’t know what you’re actually using in the experiment and where it came from and what it is, how can you trust, you know, any results of that paper? So we really have to have this level of scrutiny as we are reading these things now.

It may be very intimidating to read scientific papers at all. And you may rely on other people like you may rely on mainstream media outlets, for example, because they report very, very often on medical studies. Or you might rely on experts, doctors, scientists, even people like me. And I would caution you against that. Although I do take pride in having very high standards of what I will report as being truthful or what I report as my opinion based on the evidence. But you should always be skeptical and double check things that people say. Even AI these days now a lot of people are using or experimenting with AI programs to try and find information.

This could be for medical research, legal research, many other things. And I have now come across several examples where the AI actually made up information like they made up fake legal cases. So the only way that we found that out is by going and looking up those case citations and then finding out they don’t exist. So this really can happen with almost anything in the information space, but especially in the sciences. And so let me explain one also key aspect that you may not have thought about before, which is that there’s a big difference between science and engineering with respect to their learning information.

Because with engineering, which I’ll loosely define as using scientific principles to design and build devices or technologies, because as an engineer, the thing that you’re designing and building, in the end it has to work because no one is going to pay you for a device or a technology that doesn’t actually accomplish what they would pay for. So that is a built in accountability that you know if an engineer is right or wrong because does their device or technology work in the end? Now when it comes to a scientific finding or a scientific study which generally speaks to observing and learning about cause and effect relationships in nature, there is no accountability built in because there’s no thing that you have to design and build that has to work.

You report your findings and then how do other people know if this is really true? And the main way that has been relied upon is reproducibility, so someone else can do the same experiment in the same way and get the same results. And that of course, is not the only test that tells you if the research is valid. But it’s one very important test, because if another researcher can’t do the same experiment and get the same results, then perhaps those results even are invalid. And if the results aren’t valid, then interpreting the results to learn about nature is not going to be valid.

You have to go back to the drawing board. And there have been attempts at reproducing results of many published papers, and the overwhelming majority of those attempts have failed. So what could explain this? Well, it could just be being sloppy and incompetent that the experiment was contaminated in some way, or a procedure was not followed, or some unseen factor influenced the experiment in one case but not in the other. Or it could be actually that the scientist was making things up. And this actually happens more than you would think. And it has been exposed a lot, even at the most prestigious universities like Stanford and Harvard.

So because there’s a lot of pressure to have the right results to get further research funding. And these scientists are human as well, so they can sometimes take shortcuts, especially when they realize that the chance of someone trying to reproduce this experiment is practically non existent. So it will only benefit me if I make up some of the results or fudge the data in such a way that it increases the ability to get published, or the prestige or the ability to get additional funding and progress in your career. So obviously, in this format of a video where I’m talking for around a half hour, I can’t cover every pitfall that you will encounter when you do your own research.

But I’m trying to help you develop a position of skepticism and to know about some of these aspects so that you can be on the lookout. Of course, there are many others. I mean, statistics is one big area. And we’ve all seen on the bookshelf of some famous people a book called how to Lie with Statistics. And this is done very deliberately, very commonly even by public health organizations like the CDC and others, because they are using statistical models rather than looking at actual data. And sometimes the data itself is not a very high quality. And there are just ways that you can apply different models that are not necessarily appropriate to the data, but can skew your interpretation of the data very carefully.

And there’s, you know, one famous example of this, which was part of the trick to convince you about COVID jabs, which was using the relative risk reduction instead of the absolute risk reduction. So this is a little statistical trick that if you have a rare condition in the population, like let’s say, you know, some rare disease that only occurs in one out of a hundred thousand people, for example. So if you have a intervention that reduces the amount of people from, you know, two people in 200,000 to one person in 200,000, you have 100% relative risk reduction with that intervention.

But really in reality, only one less person is sick. And so the absolute risk reduction would be extremely tiny because it would be essentially one per 100,000. So that number would be 0.00001, I believe, would be the absolute risk reduction versus 100% or one be the relative risk reduction. So you could see how that exaggerates the benefit of the actual effect of how many people would have a lower risk for this particular given intervention. So that would be one thing to look out for. And that’s something they actually taught us about specifically at Duke, in the critical appraisal method, because this is used to make psychiatric drugs and many other interventions look a lot more effective than they actually are.

So how do I actually go about this type of research? Well, I kind of do a top down approach. So I find out about a topic of interest. And many times it’s because of your comments and questions that I decide to research something. And so I start at the superficial level and see what are people saying about this, what are laypeople saying about it in comments and questions, what are Internet researchers saying about it, what’s the mainstream media saying about it? Or sometimes even the pop media talks about these things because they may be a trend.

And I start there and see what the claims are, the claims about the benefits of this or that or the other thing, or the harms of it, if that is the issue. And then after I get a sense of what the buzz is, that’s when I go into the research literature. So I do this in a couple of different ways. I usually start with just a general Internet research on the topic. Now sometimes they use specific words in the scientific papers that are not what you would think when you describe it using lay language. So when I start to look in the research literature, I try to identify the specific words that they use, the scientists use in their publications.

And then I would adjust my search terms to look search for those specific words. And sometimes it takes a little while to find out how, what word they use to describe a particular phenomenon or process in the scientific literature. But if you just are curious and keep reading what you find with your search and then willing to go back and refine it, you can definitely catch what the buzzwords are, and then that’ll help you find a lot more papers on the topic once you identify the right search terms. Now, if I don’t find a lot of scientific publications with a general Internet search, which sometimes I do and sometimes I don’t, then I go and look at a particular database.

And the two I use really are PubChem and PubMed, and those are both part of the, you know, National Library of Medicine. And they are databases that are very useful. So pubchem is a database of all chemicals, including pharmaceuticals, industrial chemicals, natural products, et cetera. So almost everything is in there. There are very few things. It’s hard to get information. And pubchem not only has a bunch of information about these substances, it has all kinds of links, including to clinical studies, toxicology studies, and a lot more. So it’s a very useful resource looking at materials that are used in healthcare.

And the other database that I frequently use, PubMed, is medical articles are. It’s a very large database of medical articles, goes back to the 1950s, I believe. So in there you can just find, you know, all the papers on a particular topic, depending on the search terms, and you can even do author searches and much more sophisticated searching on there. And many of the articles on there actually are free and have a direct link to the full text. But it can be difficult sometimes to get the full text of all these papers. Many times they are in paywalls.

And there are some free database resources, but they are not 100% reliable. So one that has been very useful, although sometimes it works better than others, is called Sci Hub. That’s S C I dash Hub. There are a variety of different websites that are all part of Sci Hub. You can try different ones and what you do is you put the DOI number, which is a special code that every published paper has, and you just put the DOI number in the search term on Sci Hub. And if you’re lucky, you will get the full text of the paper in a downloadable form.

And they also, if the paper is not there, they do have a function where you can join and you can ask or request that the paper be added to the database and then one of the other users may be able to do that for you. So it’s definitely a great resource that you want to get involved in. So there’s a lot more complexity to, you know, looking at these papers and know how to read them. Obviously there’s a difference between in vitro studies, which are done, you know, in a laboratory with molecules and chemicals and petri dishes versus in vivo studies that are actually done in living organisms.

And of course they can be done in animals. And animals are often used as a proxy for human research. But there are obvious limitations there. And it also depends on which animal species is used. But you can get a lot of useful information from animal studies and then you have human studies. And depending on which type you’re looking, there are going to be all kinds of different aspects to it. For example, when they use animals in animal studies, they have no problem killing the animals at the end of the study, or cutting them open, or doing all kinds of invasive things that they wouldn’t do to humans or they wouldn’t be allowed to do to humans in human studies.

So you can get different kinds of information. Like for example, there are studies where they would poison laboratory animals with a certain poison. Like there is a study like this that was done with formaldehyde embalming fluid. And what they did is that half the group of rats they gave vitamin C as a pre treatment and the other half they didn’t. And then they exposed them to lethal amounts of formaldehyde. And way more rats survived in the group that was exposed to vitamin C. And obviously this study gives us a lot of information that is useful, but it would not be considered moral or ethical to conduct that in humans, obviously giving them lethal doses of poisons.

So there are limitations to the research that we can do in humans, and that means that the knowledge we have about these studies on humans is more limited compared to animals. So these are just several factors to take into consideration as you’re doing your own critical appraisal. So let me lay out kind of a step by step simple guide that you can follow to do your own critical appraisal. And hopefully you can incorporate some of the tips and tricks that I mentioned. So the first step would be to identify the claim or the question. So what is it that you’re trying to actually research? And it’s very important to have this be very clear because sometimes there are some nuances that have to be parsed out in order for you to be able to find the right information to ask your question.

Or it could be some other person’s claim. There could be a claim that if you put castor oil on your eyelids, it will dissolve cataracts and make them go away. So that’s a very particular question. Does castor oil affect cataracts? If you formulate the question that way, it’ll be a lot easier to find information about that out there, if there’s any. You know, research on that, I imagine you might only find anecdotal information for a topic like that. But anecdotal information can be very useful, too, especially when there’s a lot of it and it’s very, a very clear signal.

So in other words, if there were a thousand people, even if it were comments on a blog that all said, yes, I put castor oil in my eyelids and my cataracts went away, that’s very specific anecdotal information. But if you had a thousand people that said, oh, one said, I used castor oil for my cataracts and they went away. But another person said, I use castor oil for my toothache and it got better. And somebody else said, castor oil gave me more energy and castor oil made my hair look better, and everyone was different, and they were all things that were very subjective, then that would be a lot lower quality anecdotal evidence.

So in order to overcome these limitations, once you identify the claim of question, the next step is to seek multiple sources of information. Okay, so not even if they’re scientific papers. You wouldn’t want to rely on just one paper, provided there was more than one available. So you want to get it from multiple sources, and you want to see, do those sources have external validation. In other words, do they agree with each other, or are they all saying something different? And then you want to examine the quality of those sources. So is it a published scientific paper? Do the methods match the conclusions? You know, is it someone’s opinion? Is it an opinion, like an editorial, published editorial? Is it a news article? Who is it from? What are their biases? The news media, for example, when it comes to curing or helping diseases, their bias is to exaggerate the benefits or the hope of any newly identified treatment or research.

So whenever you hear, oh, a cure for cancer could be just around the corner, know that that is going to be an exaggerated story. And of course, there already are many ways available to cure cancer. They’re just kept out of the mainstream media. You need to understand some key logic principles, like the difference between correlation and causation, because they are distinct. And I’ll briefly review that that many times we see things happen together at the same time, are associated with each other. Like, for example, I will often wear a sport coat or a suit jacket and a tie at the same time.

And if you calculated the correlation coefficient of the jacket and the tie together in my wardrobe, you would get a very high correlation. However, I don’t put on a tie because, like a Jacket doesn’t cause me to put on a tie. There are times when I don’t wear a tie with it, and a tie doesn’t caused me to put on a jacket. A occasion causes both, right? So a formal occasion, like if I to appear in court or to give a lecture or a podcast, that is the cause of me wearing both a jacket and a tie, even though they’re highly correlated.

So that’s a very simple example, but many people will read about a correlation and assume that there is a causative effect. And that would be a very problematic assumption. In fact, you want to look the opposite. If you see something two things that are highly correlated, you want to think very carefully, is there any possible mechanism of causation and what are the other associated factors that could be the cause? And step number five is you want to examine patterns across different perspectives. So if you identify a pattern in the information you want to see, what does this represent? Is this representing an unspoken bias, like, for example, a financial relationship? So many times and authors are supposed to disclose this, but they may not disclose all the details that they may have a financial interest in the outcome of the study.

So, for example, they may be conducting a drug trial and they also are a paid consultant for the drug company, or they own substantial stock in that particular drug company. So you want to look for patterns of bias like that. Also, you know you have many groups with special interests and many of them fund and publish research, but all of the research that they publish is all biased towards a particular type of conclusion, even when that’s not what the data says. Or perhaps they even design the studies such that they will have that bias within them.

So it’s important to try to look for these different bias perspectives as you are conducting your own critical appraisal. Now, I want you to know that this is a very powerful method and it can help you to make better decisions not only with respect to health, but obviously within other areas of your life, because this type of information is published in many domains. And I have definitely benefited through my health and my family’s health by learning how to not take the conclusions or the titles of these articles and papers at face value, but instead to carry out this type of critical appraisal.

In fact, this is really what led me away from psychiatry, because as I examined the research literature more and more, I saw that the evidence of a benefit from psychiatric medications was simply not present. And in fact, it was really the opposite, that there was substantial evidence that they were causing all kinds of harm. Some types of drugs more than others. But that calculus just does not match with helping people. And so a big part of my decision to leave that practice was a result really of learning about this from the research. And then I could decide that I would never consider to give any of these drugs to my children either.

So saved a lot of potential harm by using this critical appraisal. So I would like to conclude my presentation, but I want to leave you with a little bit of a challenge because I know that, you know, you often come to me and others trying to learn about these issues, but of course we can’t do the research for you on every topic. There are some things you’re going to need to look into yourself. So I would like you to and please tell me in the comments below what you’ve done and what what you’ve learned to try and pick an issue, a very small one.

Follow my five steps of the critical appraisal and tell me what you learned about it. And maybe you’ll even teach me a thing or two. And it’ll give you the practice and the experience to realize that you are capable of doing this. Now, sometimes you might come across an issue that’s very challenging and difficult to understand, but there are many, many things that you can just do this with right now. You will need to practice thinking clearly a little bit and challenging yourself. But this is the way to become your own health authority, which is really my main mantra.

So I can’t wait to see what you come up with. I hope you have taken away some good points from this lecture and I can’t wait to see you in the next true health report.
[tr:tra].

See more of Andrew Kaufman, M.D. on their Public Channel and the MPN Andrew Kaufman, M.D. channel.