MEG-derived results from a CamCAN volunteer are superimposed on MRI images. The images from left to right move upward through the brain. Activity in the right inferior parietal cortex (yellow markers) is higher (red) than that in the adjacent white matter rim (blue), p<10-16. Surprisingly, the opposite is true for the right middle frontal cortex (green markers) whose activity is lower (blue) than the adjacent white matter (red), p<10-16. Referee consensus processing is the only method with sufficient sensitivity and resolution to detect this result.
Head Injury in the Cloud
Using “Cloud”-Based “Backfill Cycles” on XSEDE Machines Enables Very High Resolution Functional Brain Imaging
Jan. 3, 2018
From children’s playing fields to professional stadiums to battlefields, doctors are more and more worried about traumatic brain injury (TBI) that lurks after a seemingly minor concussion. Don Krieger is part of a team of clinician-researchers at the University of Pittsburgh who are studying TBI. They produce very high resolution functional brain images from magnetoencephalographic (MEG) recordings using a powerful new method called “referee consensus processing.” Their calculations rely on opportunistic use of “backfill” and other unused cycles on XSEDE computing resources. Their results show great promise in providing high resolution functional images of normal and TBI-affected brain activity.
Why It’s Important
From children’s playing fields to professional stadiums to battlefields, doctors are more and more worried about brain trauma that lurks after a seemingly minor concussion. Kids and adults may walk off the field and suffer from headaches, difficulty thinking, memory problems, attention deficits and mood swings for weeks, months or longer. A key problem in finding better ways to diagnose and treat concussion is that imaging studies show no abnormalities in more than 80 percent of TBI patients. For most, doctors don’t know whether the imaging methods aren’t sensitive enough or even whether there is any structural damage to detect.
“We’re trying to understand concussion. Even when nothing can be found in standard brain imaging studies, about 20 percent of those with concussion experience persistent symptoms for months or years. A detailed functional exam almost always reveals real problems, but we typically cannot identify the neurologic cause.”—Don Krieger, University of Pittsburgh
Don Krieger is part of a team of clinician-researchers at the University of Pittsburgh who are studying TBI. They produce very high resolution functional brain images from magnetoencephalographic (MEG) recordings using a powerful new method called “referee consensus processing.” MEG measures the magnetic fields caused by cooperative nerve-cell activity. The technology is noninvasive, silent and safe. But it’s also computationally expensive, requiring supercomputers to generate images.
How PSC and XSEDE Helped
To carry out their computations the Pitt team used the Open Science Grid (OSG). A member organization of XSEDE, OSG is a loosely coupled supercomputing resource composed of compute cycles donated by government laboratory and academic computing centers throughout the Americas. Using the OSG reduced the time required for the calculations from many decades to one to two years. To reduce that time further, they turned to two additional XSEDE systems that work very efficiently with the referee consensus solver the Pitt scientists had developed: Bridges, at the Pittsburgh Supercomputing Center (PSC), and Comet, at the San Diego Supercomputer Center (SDSC). Employing “backfill” and other unused cycles on the two supercomputers made more computing time available, did not impact other researchers using the same machines, and required only a few changes in the Pitt team’s software. As part of XSEDE’s Novel and Innovative Projects program, XSEDE Extended Collaborative Support Service experts Anirban Jana and Derek Simmel of PSC and Mahidhar Tatineni of SDSC helped the team make these minor adjustments in just a few days. The work was supported by XSEDE grants and with continuing support from OSG operations and University of Southern California Viterbi School of Engineering’s Mats Rynge.
“The calculations required for our work would simply not get done without the support of XSEDE, the Open Science Grid, SDSC, PSC, and others. Processing the most important fraction of the control data from the CamCAN cohort required 15 million core hours. That would conservatively cost at least $120,000 on a commercial cloud-computing resource.” —Don Krieger, University of Pittsburgh
Krieger and his colleagues analyzed MEG data from 64 volunteers with persistent symptoms of TBI, most of whom were combat veterans. They compared the scans with MEG data from 414 individuals who were similar to the TBI volunteers other than not having TBI symptoms. This second group of scans had been collected by the Cambridge (UK) Centre for Ageing and Neuroscience (CamCAN). The latter served as “controls,” providing the researchers with recordings from asymptomatic people to compare with the volunteers’ recordings. Together, the two XSEDE supercomputers reduced the computational time for the critical CamCAN control recordings from an estimated 20 months to seven.
With the results from the CamCAN cohort, scientists have a picture with unprecedented resolution of patterns of cooperative neural activity from brains that are unaffected by TBI. Comparing these results with those from symptomatic patients will help the scientists identify the mechanisms which cause symptoms in TBI. It may also help them find better treatments.
Deeper Dive: Using Backfill Cycles
The workflow development for the Pitt team’s MEG computations has continued as both Comet and Bridges have become progressively busier. The referee consensus solver proved to be an ideal application for utilizing idle computing cycles, either in bulk or in short “backfill” bursts during periods when resources are being drained in preparation for large multi-node jobs.
“This work is based on resource sharing and on the complementary principle, opportunism. The key elements of the work are shared, i.e. the CamCAN lifespan normative dataset, the OSG with boosts from Bridges and Comet and the results of the work. Absent the data or the supercomputing resources to process it, the work could not go forward. Absent sharing the results [with the research community], the value of the work is stunted, since these results provide a characterization of the brain activity of a neurologically normal population with unprecedented detail. The availability of this dataset and the processed results provides controls to the wide community of workers using MEG to study both normal and pathological human brain function.” —Don Krieger, University of Pittsburgh
The use of the Open Science Grid with booster allocations from Comet and Bridges enables completion of the time-sensitive calculations for a single volunteer in less than 24 hours, which is important for patients visiting Pittsburgh from a distance. The referee consensus solver developed by Krieger and his colleagues enables reliable measures of regional brain activity, excitability, and network coupling strength. The high information yield of the solver enables comparison of each of these measures from a single patient with control values with sufficient statistical power to draw formal inferences. That means that the Pitt investigators have much more reliable measures than ever before to help in diagnosis of illness related to brain activity in each individual patient.