PSC News Center

Download the PDF version of this article as it appeared in Projects in Scientific Computing, 2002.

Better Storm Forecasting Finding the Doubly Charmed Baryon
A Free Energy Landscape on the Grid Setting a Trap for Light
Blood Flow and Artificial Organs Recipes for Amorphous Metal

Better Storm Forecasting

Severe thunderstorms inflict death and property loss adding up to billions of dollars a year. New weather-forecast technologies are on the horizon that will reduce this loss by giving improved warning — a few hours compared to current forecasting, which if we're lucky lets us know a half-hour before a severe storm hits.

The Center for Analysis and Prediction of Storms (CAPS) at the University of Oklahoma, Norman has over the last decade developed a number of technologies — including innovative use of Doppler radar — that are the foundation of the new forecast technologies. Their award-winning numerical forecast system, the Advanced Regional Prediction System (ARPS), has already been deployed by some airlines and insurance companies.

PHOTO: Jim Kasdorf

One of this spring's best forecast days occurred on May 15. "This is probably the best 12-hour forecast I've ever seen come out of a continent-scale model," reported CAPS analyst Bill Martin. "If it worked this way all the time, we'd soon be out of jobs." The ARPS radar forecast showed stunning agreement with the 7 pm North American satellite image.

This year, from May 13 to June 25, CAPS used LeMieux, PSC's terascale system, to produce daily forecasts at three different scales — continental U.S., the south-central Great Plains states, and a storm-scale forecast for Oklahoma and southern Kansas. This experiment, carried out as part of a program called The International H2O Project, focused on implementing ARPS in the LeMieux environment. By the end of the experiment, the team had fully ported the ARPS software to LeMieux, achieving impressive daily turnarounds of little over an hour to generate a storm-scale forecast for 12 hours ahead.


PSC staff worked closely with the CAPS team to debug, optimize, benchmark and prepare software, to tune file-handling procedures, and to provide dedicated time and reservation-based scheduling. "As far as the ability of what we can do," says CAPS director Kelvin Droegemeier, "the Pittsburgh environment has been phenomenal."

More information:

A Free Energy Landscape on the Grid

For about 36 hours on May 29 and 30, researchers in California attacked a problem in protein biology with four different computer systems in three parts of the country. Special software called Legion, developed at the University of Virginia, harnessed the four diverse systems like a trained team of horses pulling one wagon. This "grid" approach to large-scale computing is the wave of the future, and this large-scale experiment - which produced important research results — helped to test some of the tools and demonstrate feasibility.

Scientists Charlie Brooks and Mike Crowley at Scripps Research Institute near San Diego used the experiment to study protein folding for a protein fragment called "the engrailed homeodomain." From the fruitfly, this fragment contains much of the genetic information of much larger proteins in which it resides. Because of this, the engrailed homeodomain is of great interest to the theory of protein folding, one of the grand problems of contemporary biology.

IMAGE: Simutalation
						     of protein folding in the engrailed homeodomain

Stages in the unfolding of the engrailed homeodomain, as simulated by Charles Brooks and Mike Crowley.

On May 29 and 30, Brooks and Crowley carried out long sampling simulations of specific parts of the homeodomain's folding pathway to generate its "free-energy landscape," a map of how a protein changes energy in relation to changes in its 3D shape. They employed the IBM-SP "Blue Horizon" system at San Diego Supercomputer Center, a workstation cluster at the University of Virginia and two PSC systems — Jaromir, the CRAY T3E, and LeMieux, PSC's terascale system. The Legion software provided an interface that allowed them to submit simulations to the system most available and to gather output from all the systems.

PSC's Collaboratory Research Project participated in the experiment. Supported through NIH's National Center for Research Resources, this project studies technologies to improve the ability of biomedical scientists at separate locations to collaborate effectively. Brooks and Crowley in California coordinated with computer scientists in Virginia, who developed the Legion software, and with consultants at PSC using collaboratory tools tested through PSC's program, including an AOL Instant Messenger chatroom.

Blood Flow and Artificial Organs

By recent estimates, 80,000 Americans are waiting for organ transplants, and 8 percent of them — about 6,500 people — will die waiting. "Artificial organs represent the only real hope for most of these people," says Omar Ghattas, professor of biomedical engineering and civil and environmental engineering at Carnegie Mellon. Ghattas leads the Sangria Project, a team of researchers from several universities working to overcome some of the obstacles that hinder the design of artificial heart devices.

IMAGE: Blood cells
IMAGE: Sangria blood
							flow simulations mesh.

A microscope view of red blood cells and a 2D dynamic computational mesh developed by the Sangria Project for simulating blood flow at the microstructural level.

One of the main obstacles is that current computational models of blood flow see only the forest and not the trees of this vital fluid, which up close is a mix of hemoglobin-filled cell membranes and fluid plasma. Experiments show that these up-close details make all the difference in design of artificial organs. As blood flows through artificial devices, red blood cells are often damaged, and clotting occurs that can drastically change flows.

Models of blood flow that account for these factors could avoid many false trails and potentially save years of design time, but current models, which treat blood as a homogeneous fluid, won't do the job. With an NSF Information Technology Research grant, Ghattas and co-principal investigators Guy Blelloch, Gary Miller and Noel Walkington of Carnegie Mellon and James Antaki of the University of Pittsburgh Medical Center are using LeMieux, PSC's terascale system, to do what no one has done before — realistically simulate blood flow at the microstructural level.

"We're making steady progress," says Ghattas, "on this very challenging problem." Initially, the group's calculations were limited to 2D approximations of a cell. They've since incorporated an elastic model of the cell membrane, and they've simulated the mechanism that's responsible for the bi-concave disk-like shape of a red blood cell in its low-energy or "resting" state. They're now directing their work toward modeling the dynamics of many thousands of cells within a fluid flow in 3D.

"We've solved the main conceptual problems," says Ghattas. With access to LeMieux, he expects such simulations will be routine within two years. "Teraflop supercomputing is essential and critical to the viability of this enterprise."

More information:

Finding the Doubly Charmed Baryon

Two charm quarks plus one up or down — that's the flavoring for Mother Nature's recipe to create the "doubly charmed baryon." Predicted by modern physics' theory of fundamental particles, The Standard Model (see The Strange Flavor of Quarks), this elusive particle remained a theoretical possibility, not an observed reality, until Carnegie Mellon doctoral student Mark Mattson (now a post-doc at Wayne State) noticed something in the massive data from an experiment at Fermilab in Illinois.


Carnegie Mellon physicist Jim Russ announced the new particle findings in May. Prior to that, he and colleagues from the international team participating in the experiment — called SELEX (Segmented Large X Baryon Spectrometer) — spent months analyzing data to verify the result. So far it holds up to scrutiny. "By the standards of the field," says Russ, "this ranks as a discovery."


Photo: Mark Mattson
							with vertex detectors

Mark Mattson checks silicon vertex detectors, which allow for precise measurement of particle tracks, in the SELEX experiment.

Nevertheless, Russ is collaborating with PSC to provide massive data archiving for further analysis. Steadily over six months, at an average rate of a megabyte per second, PSC transferred 15 terabytes of data into tape storage, indexed by unique file names for retrieval. "There are ways these particles decay," says Russ, "that we have to come back to with more sophisticated analysis. We have an international team, including people in Russia and Mexico. We have to package the information and get it to people at remote sites. PSC provides us with a unique capability to accomplish this."

Setting a Trap for Light

The technology of semiconductors brought us home entertainment, computers and the information age. An emerging technology called "photonic crystals" is to light what semiconductors are to electricity, and these fascinating devices promise to make an impact on 21st century communications.

IMAGE: 3D model of
							photonic micropolis

This image depicts a "photonic micropolis," a collage of photonic-crystal devices. The golden building-like structures represent a 3D photonic crystal Joannopoulos' group has proposed. The clear buildings with blue balls depict a metallodielectric structure, with metal balls embedded in glass in a diamond lattice. The red roads with holes are one-dimensionally periodic crystals. The green forests show 2D photonic crystals. The ring resonators (at the corners) are non-photonic crystal devices. Although in wide use, they have several disadvantages compared to photonic crystals.

"Photonic crystals," says MIT physicist John Joannopoulos, "provide what is essentially a new mechanism for controlling, confining and manipulating the properties of light." Computer chips work by exploiting energy "gaps" in layered crystals of silicon to regulate the flow of electrons, and photonic crystals do much the same for photons, the quantum constituents of light. Layers of material with differing indexes of refraction create conditions in which light can't exist at certain wavelengths and therefore is in effect trapped within the crystal. These properties can overcome some current limitations in the speed and capacity of optical fibers, and they could play an important role in creating hyperfast computing that uses light rather than electrons.

Joannopoulos and his colleagues at MIT carry out a program of laboratory research in these materials, and they have developed software designed especially to simulate their properties. Since April 2002, they've used LeMieux, PSC's terascale system, to advance their research on several photonic-crystal projects. Among this work is simulations of a phenomenon called "superlensing." Taking advantage of LeMieux, Joannopoulos' team for the first time carried out a numerical experiment demonstrating that superlensing is possible in three dimensions.

With a similar simulation in 2001, Joannopoulos and colleagues also demonstrated superlensing for the first time in two dimensions. They had been unable to do the more complex 3D simulation until LeMieux became available. "We simply couldn't do this without LeMieux," says Joannopoulos. "We tried and couldn't make it work. This machine has been absolutely fantastic, enabling us to do calculations that were previously impossible."

More information:

Recipes for Amorphous Metal

Amorphous metal — isn't that an oxymoron? Metals in bulk, meaning usable forms like tin cans and steel hammers, are close-packed, regular arrangements of atoms. Like members of a three-dimensional marching band, the atoms line up at precise spacing and angles from each other, repeated over and over in a pattern called a lattice.

Modern science finds ways to do many things Mother Nature hasn't done yet and among them is creating bulk metals without a lattice structure. Called amorphous metals — also known as "metallic glasses" — these materials have a random atomic structure that, for reasons not well understood, gives them unique properties — notably including a quality called "soft" magnetism and rust resistance. Many of them also combine strength and hardness with flexibility. Their trampoline-like springiness, for instance, makes amorphous metals the choice for the heads of high-performance golf clubs.

IMAGE: 3D rendering
							from simulation of a metallic glass

This representation of a metallic glass of iron (orange), zirconium (blue) and boron (silver) was rendered by PSC visualization specialist Greg Foss using data from LSMS simulations by Yang Wang.

PSC solid-state physicist Yang Wang collaborates with Carnegie Mellon physicist Mike Widom and Oak Ridge National Lab material scientist Don Nicholson on computational simulations of these materials. Their approach uses LeMieux, PSC's terascale system, like a cooking pot to stir in a little of this, a little of that — some iron, zirconium, boron, a dash of manganese and maybe some carbon. Cook it up and see what happens.


Wang employs software he helped to develop, called LSMS (Locally Self-Consistent Multiple Scattering Method), a quantum-theory based approach to calculating the electronic, magnetic and other properties of metal systems. LSMS is designed to exploit massively parallel systems like LeMieux, and in 1998 it was the first research software to achieve sustained performance exceeding a teraflop (a trillion calculations a second). On LeMieux, it runs well above four teraflops.

In collaboration with Widom and Nicholson, Wang uses LSMS to simulate various amorphous metal "recipes" and to compare the results from simulation with experiments on the same materials. Results to date show good agreement, indicating that the simulations accurately reflect reality. With further work, Wang and Widom expect to make progress toward answering deeper questions that experiments can't resolve, such as why do amorphous metals form and why do they have such interesting properties?


PSC News Center
All of PSC