THE TERAGYROID PROJECT TIED TOGETHER MORE THAN 6,000 PROCESSORS AND 17 TERAFLOPS OF COMPUTING AT SIX DIFFERENT FACILITIES ON TWO CONTINENTS.
Ketchup. Anticipation. You want the tasty tomato essence to flow freely from the mouth of the bottle. Turn it upside down and wait. Nothing happens. Shake. Shake again.
This classic ketchup flow problem is one among many applications of research by an Anglo-American team of scientists interested in little understood phenomena that occur in liquid mixtures, like ketchup, when they act like solids. They have developed an innovative and powerful computational approach that allows them — with big help from the most advanced computing resources available — to simulate these mixtures, including what happens when you apply force. Shake again, harder, and glop, the ketchup remembers its liquid nature and deposits about six times as much tomatoey essence as you need.
In recent work, dubbed the TeraGyroid Project, this Boston-U.K. team pushed research in their field to where it hasn’t gone before, and they’ve done it using the Grid, a much talked-about but still new style of scientific computing that links resources without regard to location. During SC2003 in Phoenix, TeraGyroid tied together more than 6,000 processors and 17 teraflops of computing at six different facilities on two continents. With this ability to flex the young muscles of Grid technology — including the National Science Foundation’s TeraGrid — they’ve arrived at important scientific results that couldn’t have been accomplished by now except for their Grid-based approach.
“Because of the Grid,” says Bruce Boghosian of Tufts University, “we made progress in three months that would have taken more than a year by conventional methods.” Boghosian, a professor of mathematics, leads the stateside contingent of the TeraGyroid effort.
“The Grid creates a tremendously powerful environment,” says University of London chemist Peter Coveney, who leads TeraGyroid on the U.K. side. Using the Grid and a sophisticated approach called computational steering, the scientists “steer” their simulations in real time, so that the heavy-duty computing can focus where the dynamics are most interesting. As a result, says Coveney, “our productivity skyrocketed.”
The TeraGyroid group’s Grid sophistication received a 2004 ISC Award, the major supercomputing award in Europe, for “Integrated Data and Information Management.” At SC2003 in Phoenix last November, TeraGyroid was recognized as the “Most Innovative Data-Intensive Application,” which understates what is probably the most impressive feat yet of wide-based Grid computing. Linking the TeraGrid with the U.K. E-Science Grid via dedicated trans-Atlantic fiber, the intercontinental team relied on U.K. resources at Daresbury Lab and Manchester along with computing, storage and visualization facilities at four TeraGrid sites: PSC, NCSA, SDSC and Argonne.
Their simulations rely on an approach called the lattice-Boltzmann (LB) method, and — with PSC’s LeMieux — they performed the largest LB simulation to date. These November computations were prepared by two months of work at U.S. and U.K. facilities and continued with follow-up computations into early February 2004, work which in sum yielded three terabytes of useful data.
To gather and collate this quantity of data from multiple sites was itself a large task, and the TeraGyroid team — as of April — is still analyzing results, with several papers in preparation. Their preliminary analysis points to new understanding of the somewhat bizarre liquid crystalline materials they are studying, and a strange shape known to mathematicians as a gyroid.
Amphiphiles & Gyroids
We’ve all noticed the line between oil and vinegar in a bottle of salad dressing. In chemical terms, the two liquids are immiscible. Add a third substance called an amphiphile to this kind of mix and interesting things happen. An amphiphile, by definition, is a chemical species that goes both ways. In vinegar and oil, for instance, one end of the molecule likes vinegar, the other oil.
Naturally, the amphiphile migrates to the interface between the two fluids, where both ends are happy. And if you add more and more amphiphiles, eventually there’s no space at the interface. If you then keep adding amphiphiles, they work hard chemically to create more interface area, which they do by twisting and rippling the surface until, eventually, the surface breaks apart into oil-vinegar droplets of various contorted shapes that maximize the interface area.
Some of these contorted shapes — called mesophases — have features that resemble solid materials, hence the stuck-in-the-bottle consistency of ketchup. They also lead to many industrial applications. Soap, detergent and shampoo, for instance, contain surfactants, another name for amphiphiles. Sophisticated chemical use of amphiphilic fluids has led to high-gloss liquid waxes and environmentally-friendly dry-cleaning solvents.
One mesophase in particular, a unique shape called a gyroid, hence TeraGyroid, is the focus of the team’s recent work. This convoluted and, at the same time, regularly patterned shape appears widely in biological systems. The endoplasmic reticulum, for instance, the organelle that manufactures proteins inside the cells of plants and animals, is gyroidal in structure, and gyroids have important applications in controlled drug release and biosensors.
One of the ways that a gyroid resembles the lattice-like, crystalline structure of a solid is that defects occur, disruptions to the regular pattern. These defects have a large effect on the mechanical properties of the fluid, such as rigidity — as when ketchup is stubborn. The objective of the TeraGyroid simulations at SC2003 was to look at these defects for the first time. To address questions such as: Under what conditions do defects appear? How do they change when force is applied?
“Dislocations in liquid crystals aren’t well studied,” says Boghosian, “how they develop at various temperatures, how they tend to move, how they behave under shear.”
Joysticking on the TeraGrid
In the middle ground between modeling each atom of a molecule and the high ground of computational fluid dynamics, the lattice-Boltzmann model maps fluids as particles on a 3D lattice and follows their mass and momentum changes over time. Since the 1990s, the LB method has proven itself to be both highly efficient on parallel systems (linear scaling on all platforms on which it runs) and an accurate method for modeling the dynamical properties of mixed fluids.
In the late 90s, Boghosian and Coveney developed the first LB code capable of modeling “ternary amphiphilic fluids” such as oil, water and surfactant. With this code, called LB3D, Coveney and Ph.D student Nelido González-Segredo in 2003 for the first time “captured” the self-assembly of a gyroid mesophase in an amphiphilic fluid, a result that caught the attention of other scientists and laid the groundwork for the TeraGyroid Project.
To go beyond being able to see a gyroid to looking at gyroid defects as they form and change demanded both a much grander scale of simulation and — to use the resources efficiently — the sophistication of computational steering.
The gyroid is elusive, a mesophase that forms only under a narrow and not yet well understood set of conditions. The LB model includes many parameters — temperature, relative concentrations, how much the two immiscible fluids “dislike” each other, how strongly the surfactant interacts with each of them, and others — all of which affect whether a gyroid forms and the nature of its defects. With computational steering, the scientists are able to visualize a simulation in progress, make decisions to change how it evolves, and zero in on what they want to see.
“We like to joystick around within this complex parameter space,” says Boghosian, “searching for these exotic phases. There are many parameters, and it takes human intelligence to look at a given phase and say ‘OK, I can push that toward the gyroid, by lowering the temperature, etc.’”
ABOUT 30 ORGANIZATIONS PLAYED A PART AND PROBABLY MORE THAN A 100 INDIVIDUALS.
The U.K. side of the team, with large involvement of Ph.D. student Jonathan Chin, carried most of the load in Grid-enabling LB3D. Remote researchers at Tufts (grad student Lucas Finn) and at British Telecommunications, Ipswich, U.K. collaboratively steered the simulation. Coveney emphasizes that the vision of the Grid — “transparent” access to resources without regard to location — is not yet reality, and it was a large task to coordinate this ambitious project.
“About 30 organizations played a part,” he says, “and probably more than a 100 individuals. As one example, we had to negotiate agreements between six different certificate authorities to use all these intercontinental resources.” The achievement of the TeraGyroid team in overcoming these and many other obstacles is in showing how Grid computing can dramatically reduce the time to insight in large-scale computational science.
Once the steering team zeroed in on a defect phase, the researchers enlarged the simulation — from two million lattice sites (1283) on smaller runs to more than a billion sites (1,0243) — using LeMieux, PSC’s terascale system. “The idea of Grid computing,” says Coveney, “is to be able to look at problems that haven’t been accessible before. To capture these defects, you have to simulate extremely large amounts of material, and this takes you into the terascale domain. LeMieux is the only machine we have access to that could accommodate these extremely large simulations.”
While the analysis of many terabytes of data is still in progress, the results show — among other things — that ternary amphiphilic fluids are non-Newtonian. In other words, they behave like ketchup, not like water. The viscosity depends on the shear rate. At rest, the fluid reacts to external force like a solid. Shake it vigorously, and it flows like a liquid. “We’ve been able to confirm,” says Coveney, “that our gyroid behaves like a shear-thinning, non-Newtonian fluid.”
“Being able to do these simulations,” says Boghosian, “allows us to study the dynamics of the fluid in more detail than is currently possible by experiment.” With further analysis, the TeraGyroid team hopes to suggest directions for laboratory work. As this research develops better understanding of these phenomena, it will undoubtedly lead to useful products and technologies impossible to anticipate — who knows, maybe someday a tasty ketchup that flows from the bottle smoothly and easily.
Support for This Research:
Revised: April 21, 2004
The Pittsburgh Supercomputing Center is a joint effort of Carnegie Mellon University and the University of Pittsburgh together with the Westinghouse Electric Company. It was established in 1986 and is supported by several federal agencies, the Commonwealth of Pennsylvania and private industry.
© Pittsburgh Supercomputing Center.