Written by Catherine Bolgar*

Big data is a game-changer for medical research. The ability to analyze vast sets of information, thanks to bigger and faster computers, is helping researchers to understand diseases, tease out genetic factors and spot patterns.

More researchers are looking at big data and understanding how we can utilize [it] in a better manner,” says Ervin Sejdic, assistant professor of electrical and computer engineering at the University of Pittsburgh, U.S., and founder of its Innovative Medical Engineering Developments lab.

In the past, clinicians would get data from patients and hold it up to metrics to try to see something by looking among different patient groups. “What they’re doing is flushing out the details. But the devil lies in the details,” Dr. Sejdic says. “The details are where we start understanding things. What’s really shifting in medicine is the fact that, yes, there is data, but let’s look at whole data sets.”

At the same time, better and smaller electronics, from smartphones to sensors you can wear, can compile more information at a detailed level and over bigger populations. “Researchers are looking at the interactions between different physiological systems. Sometimes these interactions break down in people with various diseases. Sometimes you have to look at the level of a minute, or an hour, or a day,” Dr. Sejdic says. “What big data is going to enable us to do is finally look at a human system as a system, rather than as individual components put together.”

Big data also is helping doctors and researchers to view diseases in shades of gray, rather than with a purely black-and-white outlook.

In the past, diseases were viewed in a simplistic way: a person is healthy or a person has disease. We would get specific information about the two states and compare the difference,” says Sergei Krivov, research fellow at the University of Leeds, U.K., who recently published research on the monitoring of kidney-transplant patients using big data techniques.

With transplants, he says, “There are two outcomes: perfect or problems. We are trying to find a single parameter to describe where you are between these two stages and what is the prognosis.” Based on the indicator, doctors can decide at an earlier stage whether to intervene into the process.

What I would like to see in the future is the following picture,” Dr. Krivov says. “A sizable part of the population frequently gives blood for analysis, for example during regular visits to their doctors. This would go to a data center. Based on this data for five or 10 years, we could determine indicators describing the degree of progression or the likelihood to occur for different diseases. We will give back this information as numbers, which is easy to interpret. This, in turn, will encourage patients to participate.”

One indicator patients might get with this approach is their biological age. “So you’re 30 years old, but your biological age is 20—or 40,” Dr. Krivov says. “Changes in your diet, exercise or lifestyle affect biological age. You might get younger, biologically. That would be reinforcement to the patient that he or she is doing well.”

Some recent uses of big data include predicting the future of metabolic syndrome, advancing neuroscience, identifying dangerous pathogens, and conducting cancer research, among many others. DNA sequencing is getting cheaper thanks to big data, and genetic sequencing with big data is becoming a key part of epidemiology, because it helps trace chains of infection. Big data is helping researchers not only to understand the different genetic mutations in cancer, but also to personalize medicine: different mutations respond differently to treatments, and getting the right treatment straight away spares patients from side effects of treatments that aren’t effective for their particular kind of cancer.

However, challenges remain for big data to reach its full potential of analyzing many kinds of information from many patients. With computers, it’s “garbage in, garbage out,” so data needs to be structured to ensure consistency. Information often isn’t shared because organizations lack procedures or systems for communication. Advances in technology are helping to overcome some of those challenges, according to “The ‘Big Data’ Revolution in Healthcare,” a study by McKinsey & Co.

Big data is still a work in progress in medicine. “If a certain number of people have a disease, the task of searching for them will take minutes instead of days,” Dr. Sejdic says. “But for other things, it will still take days because you need to develop software first for analyzing the data.”

Too much data can be a problem, too. “When you know what you want to find out, it’s a much easier problem,” he says. “But if you’re looking for new patterns, it’s more of a fishing expedition. Whenever we do clinical trials, we are flushing out the details. There’s so much information that it’s hard to track it. Until we do that, we won’t have a good understanding. The major change will occur in the next 10 to 15 years.”

*For more from Catherine, contributors from the Economist Intelligence Unit along with industry experts, join The Future Realities discussion.