Fresh Fish meets Sherlock at SC12

Where do you think I met Steve Brandt, of Louisiana State University’s Center for Computation & Technology?

We’ll get back to that; in the meantime, I guess I should say who I am. I’m Ken Chiacchia, and in January I’ll start as PSC’s senior science writer. I was trained as a biologist, but never really liked the laboratory life. So I chucked it all to pursue literary pretensions: science writing, newspaper reporting, even a little science fiction …

Bottom line is, “computer scientist” ain’t anywhere in there. I’m here at SC12 to soak up as much as I can, fresh fish that I am. On Monday, I went to Steve’s workshop with Thomas Sterling of Indiana University, “A Beginners Gude to Supercomputing.” I also talked with Nick Nystrom, PSC’s director of strategic applications, about our new machine, Sherlock.  (See some neat pics of Sherlock’s installation, via Robin Flaus, here.

No, the workshop isn’t where I met Steve. Bear with me.

Supercomputers fit on a spectrum defined in part by how much memory is available to each processor. The simplest kind of distributed memory machine, the most common type of supercomputer, has many core processors, each of which has access to only a small part of the dataset. They generally work best on problems that can be split into many parts that can be worked on separately. Shared memory machines have fewer processors that have access to the whole shebang, and are best for problems that can’t be split up efficiently.

Now that’s an oversimplification: there are distributed memory machines with powerful connections between their processor/memory units, but at a certain point the dataset is so hard to split up that you need shared memory to avoid the computer spending lots of time just trading information between its processors rather than calculating.

Sherlock is a shared memory machine. But it does something very important in addition — and it’s all about relationships.

Let’s say you wanted to find out who I have in common with Steve Brandt by searching our Google contacts. It rapidly becomes a big problem, because of the “seven degrees of separation” rule (exclude the quick connects via supercomputing folks; that isn’t it). Worse, the connections between people are arbitrary, making the dataset notoriously difficult to split up. If you try, you inevitably break connections that you need to follow, which makes them difficult for a distributed memory machine to work efficiently.

Graph datasets consist of nodes — people, in our example — and edges — connections via Google contacts. On a vastly larger scale than our example, graph analysis is the same problem faced by a molecular biologist trying to overlap thousands of tiny DNA fragments to amass an organism’s genome. Or a government disease control agency, trying to analyze how social networks can spread misinformation that undermines their efforts to minimize an epidemic. Or an astrophysicist, trying to find clusters of galaxies in the Sloan Digital Sky Survey.

But the trick is, even a traditional shared memory machine will have some trouble with this kind of calculation. Every time it chases down a connection — every person I know in turn knows multiple other people, and each is a lead that needs to be followed. Sherlock tackles this with something called massive multithreading — each core processor can take on many leads at once.

It’s much like, having found that a person I know in turn knows five people, you called in five friends, each to chase down one of those leads. A typical shared memory machine has two to at most four hardware threads per processor; Sherlock’s 64 paired processors have 128 each.

So let’s say you run though Steve and my Google contacts and find an interesting entry for each — in mine, it’s St. Gregory Roman Catholic Church, Zelienople, PA. In Steve’s, you find a parish in the Baton Rouge area.

Aha. Following up on this lead, Google maps tells you that Sacred Heart Catholic Church is on East 9th Street South in Salt Lake City, about two miles from the Salt Palace Convention Center. A quick check on our smartphone GPS logs, and you’d find us converging at Sacred Heart at 9:30 on Sunday morning.

Now you know where we met; you also know something else we have in common. (OK — you could have just looked at the GPS logs to begin with. Still, you get the idea.) Importantly, it came through checking out a nonexplicit potential connection in the dataset — which is the kind of problem, writ vastly larger, that Sherlock has been designed to tackle.

This entry was posted in Bioinformatics, HPC Research, People and tagged , , , , , , , , , , . Bookmark the permalink.

Comments are closed.

User Information

PSC Passwords

Connect to PSC systems:

PSC Policies

For technical questions:
Call the PSC hotline: 412-268-6350 / 800-221-1641 or mail to

Other services PSC provides:

Advanced Networking: High-speed network design, testing and tuning

3ROX: High-speed network access

Biomedical Applications: Computational biomedical research and training