New Methods for Developing Peta-scalable Codes

How to develop for a massively parallel, petascale future.
Pittsburgh Supercomputing Center
Pittsburgh, PA

On May 3 & 4, 2004 , sixty-two of North America's finest computational scientists gathered in Pittsburgh, Pennsylvania to discuss the future of high-performance computing. Sponsored by the National Science Foundation, the Department of Energy, the Department of Defense and the Hewlett-Packard Corporation, New Methods for Developing Peta-scalable Codes introduced the tools and techniques that will be required to effciently exploit the next generation of supercomputers.

This workshop provided an opportunity for computational scientists to consider parallel programming methods other than the currently prevalent one in which they explicitly and directly manage all parallelism via MPI. Specifically, the question is how best to program the upcoming generation of computer systems that will use massive parallelism and complex memory hierarchies to reach from the terascale into the petascale regime over the next five years.

Nathan Stone presenting at the Petascale Workshop

The presentations, by leading computer scientists, focused on languages, runtimes and libraries, tool collections and I/O methods. Talks are provided below.

The three discussion sessions provided a forum for all participants to explore issues raised by the presentations, in light of what they perceive to be the problems and requirements of petascale application development.

Repeatedly emphasized as critical were:

  • Locality of data accesses
  • Memory bandwidth and global network performance balanced with processor performance;
  • Support for latency hiding, including overlapping communication with computation, dynamic load balancing, and irregular dynamic communication patterns;
  • Support for asynchronous and one-sided communications, with data driven computation;
  • Support for fault tolerance, checkpoint/restart and optimized I/O;
  • Support for programmers to control and understand the behavior and performance of their code down to the granularity provided by MPI;
  • Support for debugging and profiling tools that enable programmers to cut through the complexity of thousands of interacting program objects;
  • Need to rethink algorithms as processor counts grow, and to provide optimized scientific libraries for large processor count systems.
  • The need for HPC system vendors to take responsibility for integrating the best solutions to these problems into their "petascale" product paths.
Discussion during Petascale Workshop

Progress reported by the presenters can be very beneficial towards resolving these problems, if exploited and integrated efficiently. For example, languages such as CAF, UPC and Titanium, which allow users to write code almost as shared memory, make it easier for compilers and runtime systems to provide the required functionality (e.g. prefetch, and optimized low-level data movement). Implementations of MPI and MPI-2 would thereby become more efficient themselves. Presenters showed that performance in such languages is already comparable to direct coding with MPI. However, the results presented were only for relatively small processor counts.

There should be synergy between these languages and the runtime systems discussed at this meeting, such as CHARM++ and KeLP. Kale stressed that virtualization is key to scaling performance, especially for dynamic load balancing, and presented data for up to 3000 processors. Producing more entities than processors may seem daunting as number of processors grows, but clever choices of the individual entities can defeat this. For example in particle models, the entity might be pairs of particles (which grows quadratically with problem size), rather than individual particles. These runtime systems should motivate the further development of solvers and frameworks built on top of them. They also provide "hooks" for performance analysis tools.

It was also suggested that the increasing processor-memory gap requires new algorithms which replace expensive communications by less expensive computations. Baden presented one such example for a Poisson solver.

Talks

Presentations from New Methods for Developing Peta-scalable Codes are available for download in either PowerPoint (PPT) or Portable Document Format (PDF) files.

Topic Presentation
Co-Array Fortran Introduction to Co-Array Fortran [ PPT] [ PDF], Robert W. Numrich
Co-array Fortran: Compilation, Performance, Languages Issues [ PPT] [ PDF], John Mellor-Crummey
Unified Parallel C UPC: Introduction and Recent News [ PDF], Bill Carlson
Unified Parallel C (UPC) [ PPT] [ PDF], Kathy Yelick
Titanium Titanium: A Java Dialect for High Performance Computing [PPT] [PDF], Katherine Yelick
CHARM++, AMPI, FEM Petascale Programming with Virtual Processors: Charm++, AMPI, and domain-specific frameworks [PPT], Laxmikant Kale
SCALLOP Applications, scalability, and technological change [PPT] [PDF], Scott B. Baden
ACTS Collection The ACTS Collection: A Software Infrastructure for Scientific Computations on High-End Computers [PPT] [PDF], Osni Marques
The DOE ACTS Collection: Interfaces, Functionality and Interoperability [PPT] [PDF], Tony Drummond
Large Scale Parallel I/O and Checkpointing CPR [PPT] [PDF], Nathan Stone
Large Scale Parallel I/O and Checkpointing [PPT] [PDF], John Urbanic

Attendees

  1. Dr. Maria C Babiuc
    Physics and Astronomy
    University of Pittsburgh
    3941 O'Hara Street
    100 Allen Hall
    Pittsburgh, PA 15260
    maria@einstein.phyast.pitt.edu
    phone (412) 624-9089
    fax (412) 683-2756
  2. Prof. Scott B Baden
    Computer Science & Engineering
    Univ of California San Diego
    9500 Gilman Drive
    MS 0114
    La Jolla, CA 92093-0114
    baden@ucsd.edu
    phone (858) 534-8861
  3. Mr. Edward D. Berger
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    Pittsburgh, PA 15213
    eberger@psc.edu
    phone 412-268-6481
  4. Dr. Paul W Bode
    Astrophysical Sciences
    Princeton University
    Peyton Hall
    Ivy Lane
    Princeton, NJ 08544
    bode@astro.princeton.edu
    phone 609-258-3702
    fax 609-688-1184
  5. Dr. Paul F Buerger
    HPC
    Ohio Supercomputer Center
    1224 Kinnear Road
    Columbus, OH 43212
    paul@osc.edu
    phone 614.292.4170
  6. Dr. William W Carlson
    Center for Computing Sciences
    17100 Science Drive
    Bowie, MD 20715
    wwc@super.org
    phone 301-805-7486
  7. Dr. See-Wing Chiu
    National Center for Supercomputing Applications
    University of Illinois
    4027 Beckman Institute
    405 N. Mathews
    Urbana, IL 61801
    schiu@ncsa.uiuc.edu
    phone 217-244-5818
    fax 217-328-4921
  8. Prof. Hugh MP Couchman
    Physics & Astronomy
    McMaster University
    1280 Main St West
    Hamilton, Ontario L8S 4M1
    Canada
    couchman@mcmaster.ca
    phone 905 525 9140 x27860
    fax 416 654 8580
  9. Dr. Suchuan Dong
    Applied Mathematics
    Brown University
    182 George Street
    Providence, RI 2912
    sdong@dam.brown.edu
    phone 401-863-3694
  10. Mr. Tomasz Drozda
    Mechanical Engineering
    University of Pittsburgh
    524 Benedum Engineering Hall
    Pittsburgh, PA 15261
    tgd4@pitt.edu
    phone 412-624-4568
  11. Dr. Leroy A Drummond
    Computational Research Division
    Lawrence Berkeley National Lab
    One Cyclotron Road
    MS 50F 1650
    Berkeley, CA 94720
    LADrummond@lbl.gov
    phone 510 486-7624
  12. Mr. Cem Ersahin
    Mechanical and Aerospace Engineering
    West Virginia University
    Morgantown, WV 26506
    cersahin@mix.wvu.edu
    phone 304-293-3111 x2466
    fax 304-319-0395
  13. Mr. Rohit C Fernandes
    Computer Science
    490 Rhodes Hall
    Cornell University
    Ithaca, NY 14853
    rohitf@cs.cornell.edu
    phone 607-254-8830
    fax 607-256-8119
  14. Dr. Jeffrey P Gardner
    Pittsburgh Supercomputing Center
    4400 5th Ave
    Pittsburgh, PA 15213
    gardnerj@psc.edu
    phone 412-268-5115
  15. Dr. Aytekin Gel.
    Aeolus Research, Inc.
    11447 S 46th St.
    Phoenix, AZ 85044
    aike@aeolusresearch.com
    phone 480-782-9474
  16. Dr. Roberto O Gomez
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    Pittsburgh, PA 15213
    gomez@psc.edu
    phone 412-268-6885
    fax 412-630-8813
  17. Prof. Steven A Gottlieb
    Indiana University
    Swain Hall West 117
    Department of Physics
    Bloomington, IN 47405
    sg@indiana.edu
    phone 812-855-0243
    fax 812-334-4058
  18. Dr. Matthew J Grismer
    Computational Sciences Branch
    Air Vehicles Directorate
    Air Force Research Laboratory
    AFRL/VAAC
    2210 Eighth Street
    Wright-Patterson AFB, OH 45433-7512
    matthew.grismer@wpafb.af.mil
    phone 937 255-3876
  19. Mr. John R Gulick
    AFRL
    USAF
    2591 K St
    WPAFB, OH 45433
    john.gulick@wpafb.af.mil
    phone 937-255-9258
  20. Mr. Miles Hurvitz
    Department of Defense
    DOD High Performance Computing
    Modernization Program Office
    1010 N Glebe Road
    Arlington, VA 22201
    phone 703-812-8205
  1. Mr. Greg Hood
    Pittsburgh Supercomputing Center
    MI 218A
    4400 Fifth Ave.
    Pittsburgh, PA 15213
    ghood@psc.edu
    phone 412-268-7063
    fax 412-244-1977
  2. Prof. Chi Hu
    Califronia State University at Long Beach
    Department of Physics
    Calif.State Univ. Long Beach
    1250 Bellflower Blvd
    Long Beach, CA 90840
    chihu@csulb.edu
    phone 562-985-4922
    fax 714 -962-3729
  3. Prof. Laxmikant V Kale
    University of Illinois at Urbana-Champaign
    Dept. of Computer Science
    1304 W. Springfield Ave.
    Urbana, IL 61801
    kale@uiuc.edu
    phone (217) 244-0094
  4. Dr. Tahsin Kurc
    Biomedical Informatics
    The Ohio State University
    3184 Graves Hall
    333 W 10th Avenue
    Columbus, OH 43210
    kurc.1@osu.edu
    phone 614-292-6568
  5. Dr. Igor V Kurnikov
    Chemistry
    Northwestern University
    2145 Sheridan Rd
    Evanston, IL 60208
    igor@kurnikov.org
    phone 414-241-5511
  6. Mrs. Carrie L Leach
    Computational Science and Engineering
    DoD HPCMP; Computer Sciences Corporation
    P.O. Box 820186
    Vicksburg, MS 39182
    Carrie.L.Leach@erdc.usace.army.mil
    phone 601.634.3372
  7. Prof. Michael Levine
    Pittsburgh Supercomputing Center
    4400 Fifth Ave
    Pittsburgh, PA 15213
    levine@psc.edu
    phone 412-268-4960
  8. Dr. Junwoo Lim
    Pittsburgh Supercomputing Center
    4400 5th Avenue
    Pittsburgh, PA 15213
    jlim@psc.edu
    phone 412-268-6467
  9. Mr. Greg Lindahl
    PathScale, Inc.
    477 N. Mathilda Ave
    Sunnyvale, CA 94085
    lindahl@pathscale.com
    phone 408 746 9100 x 261
  10. Mr. Ken MacInnis
    Pittsburgh Supercomputing Center
    4400 5th Ave
    Pittsburgh, PA 15213
    kcm@psc.edu
    phone 4122689833
  11. Dr. Marcela Madrid
    Pittsburgh Supercomputing Center
    4400 Fifth Ave
    Pittsburgh, PA 15213
    mmadrid@psc.edu
    phone 412-268-5135
  12. Mr. Thomas S Maiden
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    MI - 190B
    Pittsburgh, PA 15213
    tmaiden@psc.edu
    phone 412-268-5004
  13. Dr. Osni Marques
    High Performance Computing Research
    Lawrence Berkeley National Laboratory
    1 Cyclotron Road
    MS 50F-1650
    Berkeley, CA 94720-8139
    oamarques@lbl.gov
    phone (510) 486-5290
  14. Mr. Cameron McBride
    Physics and Astronomy
    University of Pittsburgh
    3941 O'Hara Street
    PIttsburgh, PA 15260
    cameron@phyast.pitt.edu
    phone 412-624-1826
  15. Dr. Kenneth McLain
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    Pittsburgh, PA 15213
    mclain@psc.edu
    phone 412-268-1558
    fax 412-561-6139
  16. Dr. John M Mellor-Crummey
    Rice University
    Dept of Computer Science
    MS 132
    P.O. Box 1892
    Houston, TX 77251-1892
    johnmc@cs.rice.edu
    phone 713-348-5179
  17. Dr. Robert W Numrich
    Minnesota Supercomputing Institute
    University of Minnesota
    599 Walter Library
    117 Pleasant St. SE
    Minneapolis, MN 55455
    rwn@msi.umn.edu
    phone 612-624-4341
  18. Dr. Nick Nystrom
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    Pittsburgh, PA 15213
    nystrom@psc.edu
    phone 412-268-1592
  19. Mr. David C O'Neal
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    Pittsburgh, PA 15213
    oneal@psc.edu
    phone 412-268-6801
    fax 412-364-6299
  20. Dr. John K Prentice
    Wild Rose Biophysics
    505 Hapgood St.
    Boulder, CO 80302-6965
    john@wildrosebiophysics.com
    phone 303-819-7256
    fax 720-565-6167
  21. Mr. Michael T Prinkey
    Aeolus Research, Inc.
    18 Cecil Driv3
    Dunbar, PA 15431
    mprinkey@aeolusresearch.com
    phone 724-438-5165
  1. Mr. Raghu N Reddy
    Pittsburgh Supercomputing Center
    4400 Fifth Ave
    Pittsburgh, PA 15044
    rreddy@psc.edu
    phone 412-268-8159
    fax 724-443-1256
  2. Mr. Steven P Reinhardt
    Server Products Group
    SGI
    2750 Blue Water Road
    Eagan, MN 55121
    spr@sgi.com
    phone 651.683.5664
    fax 651.683.0631
  3. Prof. Ralph Roskies
    Pittsburgh Supercomputing Center
    4400 Fifth Ave
    Pittsburgh, PA 15213
    roskies@psc.edu
    phone 412-268-4960
  4. Dr. Sergiu Sanielevici
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    Pittsburgh, PA 15213
    sergiu@psc.edu
    phone 412 268 5240
  5. Mr. Ray Scott
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    Pittsburgh, PA 15213
    scott@psc.edu
    phone 412 268 6175
  6. Mr. Reza Sheikhi
    Mechanical Engineering
    University of Pittsburgh
    524 Benedum Engineering Hall
    Pittsburgh, PA 15261
    moh3@pitt.edu
    phone 412-624-4568
  7. Mr. Aaron R Shelmire
    Pittsburgh Supercomputing Center
    4400 Fifth Ave
    Pittsburgh, PA 15213
    shelmire@psc.edu
    phone 412.268.4216
  8. Dr. Niraj Srivastava
    HPTCD
    Hewlett-Packard
    7 Elliot Circle
    Shrewsbury, MA 1545
    niraj.srivastava@hp.com
    phone 508-421-6489
    fax 508-421-6489
  9. Mr. Robert Stock
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    Pittsburgh, PA 15213
    stock@psc.edu
    phone 412-268-6351
  10. Dr. Nathan T.B. Stone
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    Pittsburgh, PA 15213
    stone@psc.edu
    phone 412-268-4367
  11. Mr. Daniel W Stubbs
    McMaster University/Sharcnet
    Dept. of Physics and Astronomy
    Hamilton, Ontario L8S 4M1
    CANADA
    stubbsd@mcmaster.ca
    phone (905) 525-9140 ext. 27663
    fax (519) 763-4311
  12. Mr. Vasileios Symeonidis
    Division of Applied Mathematics
    Brown University
    182 George St
    Box F
    Providence, RI 2912
    sjoh0341@cfm.brown.edu
    phone (401) 863 3694
    fax (401) 499 4379
  13. Dr. Robert J Thacker
    McMaster University/SHARCNET
    Dept of Physics and Astronomy
    Hamilton, Ontario L8S 4M1
    Canada
    thacker@physics.mcmaster.ca
    phone 905 525 9140 x 24009
  14. Mr. John Urbanic
    Pittsburgh Supercomputing Center
    4400 Fifth Avenue
    Pittsburgh, PA 15213
    urbanic@psc.edu
    phone 412-268-7861
  15. Dr. Yang Wang
    Pittsburgh Supercomputing Center
    Mellon Institute Building
    4400 Fifth Ave
    Pittsburgh, PA 15213
    ywg@psc.edu
    phone 412-268-2795
  16. Dr. Daniel B Weber
    Center for Analysis and Prediction of Storms
    University of Oklahoma
    100 East Boyd Street Room 1110
    Norman, OK 73019
    dweber@ou.edu
    phone 405-325-1932
  17. Mr. Ron L Westfall
    CTO
    OctigaBay Systems Corp
    301-4621 Canada Way
    Burnaby, BC V7H 2C2
    Canada
    westfall@octigabay.com
    phone 604-484-2249
  18. Mr. Kevin L Wohlever
    HPC
    Ohio Supercomputer Center
    1224 Kinnear Rd.
    Columbus, OH 43235
    kevin@osc.edu
    phone 614-292-4167
    fax 614-440-4640
  19. Dr. Ibrahim Yavuz
    West Virginia Univeristy
    Mechanical and Aerospace Engineering
    Morgantown, WV 26506
    Ibrahim.Yavuz@mail.wvu.edu
    phone 304-293-3111
    fax 304-599-4985
  20. Prof. Katherine Yelick
    UC Berkeley
    yelick@cs.berkeley.edu
    phone (510) 642-8900
    fax (510) 642-5775
  21. Mr. Server L Yilmaz
    Mechanical Engineering
    University of Pittsburgh
    524 Benedum Engineering Hall
    Pittsburgh, PA 15261
    sly5@pitt.edu
    phone 412-624-4568
    fax 412-422-9356