Blacklight comes to PSC

With a $2.8M award from the National Science Foundation, PSC introduces the world’s largest coherent shared-memory system..

In July, the National Science Foundation (NSF) awarded $2.8 million to help PSC acquire a system that features Silicon Graphics’ newest scalable, shared-memory computing system and associated disks. Called Blacklight, the new system became available for research through the NSF TeraGrid in October; its extremely large, coherent shared-memory opens a new computational capability for U.S. scientists and engineers.

PHOTO: Blacklight

Featuring 512 eight-core Intel Xeon (Nehalem) processors (4,096 cores) with 32 terabytes of memory, Blacklight is partitioned into two connected 16-terabyte coherent shared-memory systems — creating the two largest coherent shared-memory systems in the world.

“Because of the extraordinary memory size and relative ease of programming made possible by this system’s shared-memory structure, scientists and engineers will be able to solve problems that were heretofore intractable,” said PSC scientific directors Michael Levine and Ralph Roskies. “For many research communities — including data-analysis and many areas of computer science — it will open the door to use of high-performance computation and thereby expand the abilities of scientists to ask and answer questions.”


Jim Kasdorf, PSC director of special projects

David Moses

David Moses, PSC Executive Director

In computer terms, “shared memory” means that a system’s memory can be directly accessed from all of its processors, as opposed to distributed memory (in which each processor’s memory is directly accessed only by that processor). Because all processors share a single view of data, a shared memory system is, relatively speaking, easy to program and use. Coherence, a feature related to the synchrony of read-write operations by different processors within the system, is an important feature in many large data-analysis tasks.

“We are extremely excited about extending Silicon Graphics’ long-standing history of working with PSC on systems that deliver breakthrough scientific research and engineering results,” said Mark Barranchea, president and CEO of Silicon Graphics. “For over 15 years, PSC has used Silicon Graphics Data Migration Facility for hierarchical storage management, and over the past years they have used two of our current SGI Altix 4700 scalable shared-memory systems to conduct breakthrough research in chemistry and life sciences. We view PSC as a special partner in the development and delivery of our next generation, scalable shared-memory systems, which will bring new performance, flexibility and power efficiency to the supercomputing industry.”

The 4,096 processor cores and 32 terabytes of shared memory are interconnected using Silicon Graphics’ next-generation high-bandwidth, low-latency NUMAlink 5 interconnect. This interconnect enables scalable shared-memory or message-passing applications to run with higher levels of parallel efficiency, which means researchers can assign more processor cores simultaneously to the same task — to address larger problems and solve them more quickly.


The TeraGrid is the world’s most comprehensive distributed cyberinfrastructure for open scientific research. As a partner in this National Science Foundation program, PSC helps to shape the vision and progress of the TeraGrid.

PSC and TeraGrid PSC is actively involved in TeraGrid leadership. Co-scientific director Michael Levine is PSC’s representative to the TeraGrid Forum — TeraGrid’s principal decision-making group. Other PSC staff with TeraGrid leadership roles include Sergiu Sanielevici, Area Director for User Support and Jim Marsteller, head of PSC’s security, who chairs the TeraGrid Security Working Group. Laura McGinnis plays a lead role in TeraGrid education, outreach and training activities. PSC director of systems and operations, J. Ray Scott, leads the TeraGrid effort in Data Movement, and PSC director of strategic applications, Nick Nystrom, leads the TeraGrid Extreme Scalability Working Group, which fosters planning to meet the challenges of deploying extreme-scale resources into the TeraGrid. PSC staff members serve on all of TeraGrid’s working groups.



Row 1 (l to r): R. Reddy, Ken Hackworth, Derek Simmel, Anjana Kar, Pallavi Ishwad, Ralph Roskies, Rich Raymond, Jack Chang, Boris Kaminsky, Markus Dittrich, Marcela Madrid. Row 2: Christal Banks (in white), Patricia Sudac, Art Wetzel, Deb Nigra, Bob Stock, Lucille Jarzynka, Robin Flaus-Scibek, Shandra Williams. Row 3: Kathy Benninger, J. Ray Scott, Andy Adams, Jenda Domaracki, Michael Lambert, Jason Sommerfield, Dustin Sorge, David Moses, Brian Johanson, Ed Berger, Ryan Omecene, Rick Costa. Row 4: Adam Fest, Burt Cubbison, Greg Foss, Hugh Nicholas, Ken Goodwin, Dave Graham, Michael Schneider, Zhihui Zhang, Jared Yanovich, Roberto Gomez, Yang Wang, Joel Welling. Row 5: John Kochmar, Ed Wozniak, Shane Filus, Steve Cunningham, Ray Nardozzi, Dave Kapcin, Chad Vizino, Nathan Stone, James Marsteller, Kevin Sullivan, Nick Nystrom, Clint Perrone. (Not pictured: Beth Albert, Erica Anderson, Cheryl Begandy, Vivian Benton, Phillip Blood, Janet Brown, Shawn T. Brown, Robert Budden, Brian Chen, Brian Gill, Ed Hanna, Greg Hood, Wendy Huntoon, Anirban Jana, James Keener, Joe Lappa, Michael Levine, Rob Light, Susan Litzinger, Jeremy Lipson, Mahin Mahmoodi, Tom Maiden, Laura McGinnis, Paul Nowoczynski, David O' Neal, Josephine Palencia, Stephen Petko, Chris Rapier, Alex Ropelewski, Sergiu Sanielevici, Joel Stiles, John Urbanic, Bryan Webb, Troy Wymore.)