IBM and Sage Bionetworks announce winners of first phase of DREAM Digital Mammography Challenge
Challenge is aimed at helping to refine cancer detection algorithms so they can be used in routine clinical practice
IBM (@IBMNews) and Sage Bionetworks (@Sagebio) announced that the winners of the first phase of its DREAM Digital Mammography (DM) Challenge have developed algorithms that had 5% fewer false-positive errors in breast cancer screenings than recently published state of the art computerized methods. This 5 percent improvement could potentially lead to less anxiety and unnecessary procedures for an estimated two million women per year in the United States and could help reduce costs associated with follow-up exams and biopsies.
More than 120 independent teams of data experts from inside and outside the medical imaging field have participated in the challenge, which focused on developing predictive algorithms that reduce false-positive mammograms while maintaining or improving cancer detection. The goal is to enhance the predictive accuracy of algorithms so that they can be used in routine clinical practice.
In the first phase of the challenge, participants completed two tasks: They (i) developed a predictive algorithm that can analyze digital mammography images, (ii) and developed a predictive algorithm that can analyze both digital mammography images and clinical information.
Winning Teams
Yaroslav Nikulin, an engineer from the French imaging company Therapixel, and his team received top honors for their work on the first task and tied for first place in the second task. In the first task, they developed an algorithm with a predictive accuracy of 80.3 percent, which is 5 percent more accurate than the runner up. In the second task, Nikulin and his team developed an algorithm that was 80.4 percent accurate.
Tied for first place in the second task was a team led by Yuanfang Guan, Assistant Professor in the Department of Computational Medicine and Bioinformatics at the University of Michigan, Ann Arbor. The group developed an algorithm with a predictive accuracy of 77.5 percent and outperformed the runner-up by more than 2 percent. Though the difference in accuracy between Guan’s and Nikulin’s teams was 2.9 percent, their performance was indistinguishable in the other metrics used to score the algorithms. Both winning teams used “Deep Learning,” one of the most advanced artificial intelligence techniques capable of analyzing and interpreting images.
Reducing False Positives
Mammograms are widely considered the most accessible and cost-effective breast cancer screening method. However, the American Cancer Society and the United States Preventive Services Task Force recently issued changes to recommendations for when women should start having mammograms and how often they should get them. The changes are due, in part, to the large number of false-positive mammograms. One in 10 women undergoing screening mammography is recalled for a diagnostic workup, though fewer than 5 percent of the recalled women will eventually be found to have cancer. Recalled patients often experience stress and additional medical costs, and some require interventions, including unnecessary biopsies. New algorithms may eventually be used by doctors to help them customize screening regimens for patients and identify women who would benefit from more or less frequent screening.
About the Challenge
Participating teams used hundreds of thousands of de-identified mammograms and clinical trial data provided by Kaiser Permanente Washington and the Icahn School of Medicine at Mount Sinai to create algorithms that can determine a woman’s cancer status in the 12 months following her mammogram.
Eight teams with the best algorithms will now move on to the community phase of the challenge, where they will be invited to add outside expert collaborators. They will also share their source code publicly, including other challenge participants, in an attempt to foster cooperative learning. In the community phase, finalists will work together to develop an algorithm that can fully match the accuracy of an expert radiologist.
“I am extremely pleased with the results of the competitive phase of the DM Challenge,” said Gustavo Stolovitzky, Director at IBM Research and Founder of the DREAM Challenges. “By providing powerful computational resources and making available what is, to the best of our knowledge, the largest public mammography dataset ever released, we empowered hundreds of data scientists to contribute to the solution in the fight against breast cancer. Moreover, the code and methods generated during the DM Challenge are now available for anybody interested in building on these results to help solve this important public health problem.”
“The innovation in this challenge stems not only from its final output—a set of robust models to aid clinicians in detecting breast cancer—but also the structure of the challenge itself. This challenge embodies a new paradigm for data sharing and cloud-hosted collaboration to tackle important questions in biomedicine,” said Dr. Justin Guinney of Sage Bionetworks. “By working together as a community of researchers and using the best tools of science and technology, we have advanced set the framework for clinicians in the field of breast cancer detection.”
If by the end of the community phase, the top eight teams—including Nikulin’s, Guan’s, and six others—can develop an algorithm that matches the expert radiologist performance of about 87.9 percent accuracy, they will receive a prize of up to $1 million.
The DM Challenge was born out of the White House’s Cancer Moonshot initiative and is funded in part by the Laura and John Arnold Foundation. It was designed by an organizing committee that includes IBM, Sage Bionetworks, Kaiser Permanente Washington, the Icahn School of Medicine at Mount Sinai, and the U.S. Food and Drug Administration. The challenge relied on the technological advances of Sage Bionetworks and IBM—Sage provided Synapse, a collaborative platform to host the challenge, as well as science and engineering expertise; IBM research teams in the United States, Israel, and Australia built the infrastructure for the challenge within the IBM Watson Health SoftLayer cloud and contributed further engineering and data science expertise.
Read additional information about the DM Challenge.
About DREAM Challenges
First conceived by IBM in 2006, DREAM Challenges have addressed objectives that range from predictive models for disease progression to developing models for cell signaling networks. Designed and run by a community of researchers, DREAM Challenges invite participants to propose solutions, fostering collaboration and building communities in the process. The DREAM Challenges community shares a vision of open collaboration to leverage the “wisdom of the crowd” to improve human health and sciences.
About IBM Research
For more than seven decades, IBM Research has defined the future of information technology, with more than 3,000 researchers in 12 labs located across six continents. Scientists from IBM Research have been awarded six Nobel prizes, a U.S. Presidential Medal of Freedom, ten U.S. National Medals of Technology, five U.S. National Medals of Science, and six Turing Awards. The teams have also included 19 inductees into the U.S. National Academy of Sciences and 20 inductees into the U.S. National Inventors Hall of Fame.
About Sage Bionetworks
Founded in 2009, Sage Bionetworks is a nonprofit biomedical research organization that promotes innovations in personalized medicine by enabling a community-based approach to scientific inquiries and discoveries. In pursuit of this mission, Sage Bionetworks has assembled an information commons for biomedicine supported by Synapse, an open compute space. The commons facilitates open research collaborations and innovative DREAM Challenges; it also empowers citizens and patients to share data and partner with researchers through Sage’s BRIDGE platform.
