Cutting through the clutter: Scientists develop an ‘auditing’ tool that can help improve the reliability of studies that explore relationships between things

It depends.

The same research question can yield vastly different answers depending on how a study is designed, which variables are measured, and how results are analyzed. Because of the hodgepodge of approaches used to decipher the interplay between variables, association studies — those that explore how one thing affects another — are notoriously prone to error or “bias.” Finding a false link where none exists or missing one if it does can thwart the pursuit of critical scientific questions and solutions, lead researchers down the wrong path and generate contradictory results that confuse peer scientists and the public alike.

To help remedy such problems, a team of computational scientists from Harvard Medical School has developed an auditing tool called vibration of effects (VoE). The tool, first described in PLoS Biology in September 2021, has now been deployed to analyze reported links between various gut microbes and six diseases in 15 previously published studies comprising samples from 2,434 patients with colon cancer, type 1 diabetes, type 2 diabetes, cardiovascular disease, inflammatory bowel disease (IBD) and cirrhosis of the liver.

The newly published research is the final installment in a three-paper series and represents the culmination of the team’s two-year journey undertaken at the start of the COVID-19 pandemic and conducted with collaborators working remotely across the country.

The results of the latest study, published March 2 in PLoS Biology, reveals that a full one-third of 581 reported microbe-disease associations were inconsistent, with outcomes changing depending on how the design was tweaked and which other variables were included in the analysis.

A particularly striking finding, the team said, was that more than 90 percent of the research findings of studies exploring the link between gut microbes and type 1 and type 2 diabetes were inconsistent. Studies exploring the link between cirrhosis of the liver and gut microbiome yielded the greatest consistency — 60 percent of these analyses showed consistent results when run through different models. Cardiovascular disease association studies showed nearly 50 percent consistency, as did one-third of IBD-microbiome association studies.

The tool is a brute-force computational approach that tests the reliability of research findings and can be used by researchers to audit their own results before submitting them for publication. It is publicly accessible and available for free online (https://github.com/chiragjp/quantvoe).

“At its most basic, the vibration of effects model analyzes how the modeling choices a researcher makes can influence what they will discover,” said Braden Tierney, one of the tool’s chief architects. A former doctoral student at HMS, Tierney is now a postdoctoral research fellow at Weill Cornell Medical College. “This approach is one way to maximize researchers’ confidence in the results they are getting from their analyses before they even publish them.”

In the latest study, the team checked each of the reported associations by running millions of modeling strategies, including the addition and subtraction of different variables. The modeling demonstrated how results could shift dramatically depending on which variables were used and which questions were asked.

Overall, studies that scored high were less reliable because their results showed a great degree of variation when run through multiple models. By contrast, studies that scored low on VoE were deemed robust because they pinpointed associations that remain consistent even when a different testing model is applied.

Going a step further, the team demonstrated how the VoE tool can be used to identify potential confounders — factors whose influence is not measured or accounted for in the study design and thus interfere with the reliability of the results. To do so, the team ran more than 6 million statistical modeling strategies on the findings of previous studies, adding and subtracting variables and testing different combinations of variables. For example, analyzing the role of the microbe F. prausnitzii in colonic disease, the researchers demonstrated how including factors such as a person’s blood sugar and cholesterol levels and body mass index can give seriously divergent results.

“The VoE tool can help researchers not only identify problems but also diagnose what may be causing them,” Tierney said. “It can help them understand why they may be getting conflicting findings on the same research question, and it can help them dig deeper and find links they may otherwise overlook.”

High stakes

When done well, robust association studies can become critical gateways to further research that builds on these initial findings. Identifying linkages between variables, such as coffee intake and memory, carrot consumption and eyesight, are important because they can inform hypotheses that scientists can test in the lab and in clinical trials to determine cause and effect and, eventually, the underlying mechanism of an observed effect.

“Understanding correlation is a prerequisite for understanding causality but is not enough,” said Chirag Patel, co-senior author on the trio of studies and associate professor of biomedical informatics in the Blavatnik Institute at HMS. “So, it’s essential for researchers to have confidence in the robustness of the observed association before they engage in further, sometimes expensive, study.”

“Knowing how strong the association between two variables is can save researchers from going down dead-end streets or it can set them on the right path toward understanding critical links in human health,” said Patel, who co-led the three studies with colleague Aleksandar Kostic, assistant professor of microbiology at HMS and assistant investigator at Joslin Diabetes Center.

To detect correlations, researchers often start out by getting observational data from humans. For example, comparing differences in microbial gut composition between people who have diabetes and those who do not can help illuminate which bacteria may reduce or enhance risk for the disease. But with that approach, Patel said, come a whole host of caveats. Especially critical among these is being aware of and testing variables that may play a role in a purported effect.

For example, when studying the relationship between a gut bacterium and heart disease, other factors that could influence the outcome could be the person’s age, sex, activity level, and so forth. Traditionally, researchers can account for the influence of these confounders — or control for them in statistics parlance — by incorporating them into the analytical model. But what happens when researchers don’t know or think about what they should control for?

Say that a team of researchers is trying to determine whether gut microbe A increases the risk for developing colorectal cancer, Patel explained. The team controls for sex, age, and family history of colorectal disease, but fails to incorporate host diet into the analysis. The findings would be polluted because the scientists did not account for the possibility that certain foods and dietary choices can also fuel the risk for this type of cancer. The researchers may erroneously attribute the observed increase in risk in a subset of participants to the presence or absence of microbe A, when in fact it may have been their diets that elevated their risk.

“You may be looking for one thing, but you also need to understand what other things may be playing a role to affect the outcome,” Patel said. “You must have a good study design that can extract signal from noise and can tell true associations from confounding ones, but how do you do that?”

One way to remedy the problem, Patel and Tierney said, is to “pressure-test” the findings. This is precisely what VoE does. The model does so by analyzing all the possible associations and influences that may have led to the result. If the finding remains consistent across all the possible scenarios, then they point to a true effect.

The researchers caution, however, that the VoE tool cannot differentiate between truly causative links and mediators of disease, those factors that may precipitate disease in the presence of other factors but that by themselves are not enough to cause illness.

The tool can, however, eliminate much of the faulty associations and help researchers focus on robust ones that are more likely to yield further insights — a critical step in biomedical discovery.

The VoE model is based on work started by Patel several years ago. Building on these efforts, Patel, Tierney, and Kostic sought to analyze how microbes in the human gut may influence health and disease traits. But the scientists wanted to dig deeper than the usual association studies flooding the field — those that assess whether the mere presence or absence of a given microbe affects host health. They wanted to determine which specific microbial species and which specific genes within a species might play a role in disease development.

The first installment in the three-part series tackles that very question, showing that rather than a microbe’s absence or presence, it is specific bacterial genes that are strongly linked with disease. The work, published May 2021 in Nature Communications, analyzes the genetic makeup of bacteria in the human gut and links groups of bacterial genes to several diseases.

In the second paper, the team introduced the VoE tool and used it to audit thousands of statistical analyses and compare their findings. The researchers assessed the results of studies analyzing the link between calcium intake and bone density, vitamin D levels and COVID-19 risk, carrot intake and eyesight, blood sugar and income, blood pressure levels and the use of a commonly prescribed antihypertension drug (lisinopril). They tested 10,000 different modeling scenarios per association to identify varying levels of “vibration” or noise. One of this paper’s most striking findings was that nearly half of all COVID-19 and vitamin D studies showed contradictory results when pressure-tested. By contrast, blood glucose levels and wealth showed consistent association throughout the analyses, with drops in wealth tracking with higher blood sugar levels.

The latest paper unifies all three publications to explain how the approach can be applied to microbiome disease associations for six prevalent and well-studied conditions. In many cases, different models yielded contradictory associations for the same microbe-disease pairing, some showing positive correlations and others negative.

Recommendations for the field

Broadly, the team’s findings speak to the need for ongoing self-analysis by researchers, who should never cease to question the basic assumptions they make in study design, to assess the role of the variables they choose and to question and test their own findings, Patel said.

“The decisions we make at the outset when we conceptualize and design a study determine everything thereafter,” Patel said.

“For me, things like age, sex, country of origin are basic things to consider in your study design, but someone else, even if they have the same hypothesis as I do, they may decide to include totally different things in their analysis, and because of that we may come to vastly different conclusions.”

What this means for the field, Patel said, is the need for researchers to consider more analytical scenarios accounting for more variables and also to be more transparent about which variables they include in their analysis and why.

“My assumptions may be different from yours, and the same set of data in your hands might result in something totally different,” Patel said.

Fundamentally, Tierney said, how you ask a question will influence the answer you get, and that’s critical in study design.

The researchers said that modeling vibration of effects is a critical step in navigating discovery in observational data because it can help discern robust associations and catalogue and adjust variables that affect study results. The researchers said their plan is to continue to evolve the tool and optimize its efficiency for maximum impact.

“In order to design diagnostic tests and treatments for any disease, including disease caused by alterations in the gut microbiome, we must understand the underlying mechanism and the exact interplay between gut microbes and dysfunction,” Patel said. “And the first step on that long journey is the ability to spot correct associations.”

Funding:

The work was supported by funding from the National Science Foundation (#1636870), the National Institute of Allergy and Infectious Diseases (R01AI127250), and the National Institute of Environmental Health Sciences (R01ES032470), as well as by the American Diabetes Association Pathway to Stop Diabetes Initiator Award (#1-17-INI-13), and the Smith Family Foundation Award for Excellence in Biomedical Research.

Relevant disclosures:

Kostic is a co-founder of FitBiomics, Inc. and a member of their scientific advisory board. Tierney consults for Seed Health on microbiome study design and analysis. Michaela Walker, co-author of the second PLoS Biology paper, is an employee of UPSIDE Foods.