Table of Contents

• Overview

• Production Platforms

• Friendly User Platforms

• Software Packages

• Allocations and Usage

• Applications Development

Overview

• Platforms

  Cray J90 , C90 , T3D , T3E
  DEC AlphaServer
  SGI Indigo2
  

• Software Packages

  Fluid Dynamics
  Structural Analysis
  Electromagnetic Analysis
  Graphical Support Software
  
  

• Allocations and Usage

  PVP Usage (C90, J90, YMP)
  MPP Usage
  Summary
  
  

• Applications Development

  Parallel Spectrum Port
  Parallel File I/O Library (PFIO)
  Parallel Jacobi Eigensolver Library (PJAC)
  Parallel Finite Element Method (PFEM/ANSYS)
  

Production Platforms

PSC's CRAY J90s each have 8 processors. Each processor has a peak speed of 200 MFlops. Each machine has 128 MWords of memory, with 85 MWords available for users, 25 MWords for ldcache, and the remainder used for system operation.

The CRAY C90-16/512 has 16 processors with a peak aggregate speed of 16 Gflops and a main memory of 512 MWords or 4 GBytes. A Solid-state Storage Device (SSD) serves as an extension of memory to provide an additional 512 million words or 4 Gbytes. UNICOS has access to the SSD for use as a fast disk cache for the /tmp file system, and automatically moves large files to and from the device as needed.

The CRAY T3D-512 has 512 DEC Alpha processing elements (PEs) with a peak aggregate rate of 76.8 Gflops. Each of the processors is a DEC Alpha 64-bit microprocessor running at 150 MHz, with a theoretical peak of 150 MFlops. The T3D's network topology is that of 3-dimensional torus. Each processor runs a Mach-derived microkernel. The memory is logically shared and physically distributed, with each PE having 64 MBytes (or 8 MWords).

Friendly User Platforms

jaromir and tourney

PSC's CRAY T3E currently has 256 DEC Alpha processing elements (PEs) with a peak aggregate rate of 76.8 Gflops. Each of the processors is a DEC Alpha 64-bit microprocessor running at 300 MHz, with a theoretical peak of 600 MFlops. The T3E's network topology is that of 3-dimensional torus. Each processor runs a UNICOS microkernel. The memory is logically shared and physically distributed, with each PE having 128 MBytes (or 16 MWords).

The Pittsburgh Supercomputing Center is currently running a configuration consisting of one AlphaServer 8400 SMP server and four AlphaServer 2100 SMP servers. The AlphaServer 8400 has four 300 MHz processors and 4 GByte of memory. The 8400 system has five fast-wide-differential SCSI controllers that connect to thirty-five 4 GByte SCSI disk drives. Each AlphaServer 2100 has two 250 MHz processors and 256 MByte memory. They are clustered via Digital's new low-latency (less than 5 microseconds) high-bandwidth (100MB/sec) memory-to-memory interconnect, called Memory Channel. All of the AlphaServers are running 64-bit Digital Unix, High Performance Fortran, C, C++, and numerous other applications.

The aggregate memory space of the noted platforms is about 76 GBytes.

The peak aggregate floating point rate is around 250 GFlops (1/4 TeraFlop).

---

Software Packages

• Fluid Dynamics

  ADINA , CFDS Flow3D , FIDAP , FLUENT , KIVA , NEKTON

  
  

• Structural Analysis

  ABAQUS , ADINA , ANSYS , LS-DYNA , DYTRAN , NASTRAN , NIKE , TOPAZ

  
  

• Electromagnetic Analysis

  ANSYS , EMAS , SPICE

  
  

• Graphical Support

  ADINA , ANSYS , FACET , FIDAP , INGRID , MAZE , ORION , PATRAN , TAURUS
  

  
  

All packages, with the exception of PATRAN, are available on the C90. PATRAN is installed on a two-processor AlphaServer with 128 MB memory. Only LS-DYNA is available for the T3D.

New items for 1996 include a full commercial version of NASTRAN for the C90, the installations of the FIDAP and ANSYS GUIs on a J90 production platform, and the move of PATRAN from a DECstation box to one of the new AlphaServer front ends.

The Parallel Applications Group is in the process of porting Spectrum to the T3E.

Allocations and Usage

Current staff allocation is about 2 FTEs

PVP Engineering Usage (YMP, J90, C90)

MPP Engineering Usage

Summary

PVP Engineering Usage

---

MPP Engineering Usage

---

Allocations and Usage Summary

Proposals

Year	1992	1993	1994	1995	1996*
All Proposals	1043	907	815	698	636
Engineering	144 (14%)	180 (20%)	191 (23%)	171 (25%)	157 (25%)

All 1996 figures in this section include estimated counts for the fourth quarter.

Trend towards larger proposals in smaller numbers from increasingly sophisticated groups.

Percentage of proposals received from engineering disciplines increased from 14% in 1992 to 25% in 1996.

Engineering usage of the PVP platforms has increased steadily over the past five years. Since the introduction of the T3D in 1994, engineering MPP allocations have grown dramatically. MPP usage also inceased from near zero in 1994 to about 8% in 1996.

---

Applications Development

Parallel Spectrum Port

Parallel File I/O Library (PFIO)

Parallel Jacobi Eigensolver Library (PJAC)

Parallel Finite Element Method (PFEM/ANSYS)

Parallel Spectrum

Description

Design Basics

Capabilities

Development

Description of Parallel Spectrum

Commercial finite element solver for analyzing multiple and interacting physical phenomena (full spectrum multiphysics code).

Capable of resolving fluids and solids problems, as well as fluid-solid and solid-solid interactions. Both linear and nonlinear solution strategies are featured.

Full support for static and transient solutions, compressible and incompressible flows, turbulence models, and ALE fluid mesh movement.

Domain decomposition performed by a derivative of the Metis code (developed at the University of Minnesota) coupled with a staggered solution control mechanism allows each physical region, or subdomain thereof, to be processed concurrently.

Visualization functionality for fluid-solid-thermal analyses. Interactive 3D graphics and animations driven by point-and-click GUI.

Parallel Spectrum Design Basics

Both direct and iterative linear solvers are available. Direct methods include profile and frontal methods. Iterative methods include conjugate gradient and generalized minimum residual.

Two types of eigensolvers are provided for the analysis of vibration and buckling problems. The subspace iteration method can be used to compute a small number of solutions to vibration and buckling problems. The Lanczos method is used for larger vibration problems.

A multistagger strategy is used to break complete systems of coupled equations into smaller subsystems of equations. These subsystems represent a partial decoupling of the full system. A solution to the global system is obtained by iterating over all "staggers" until mutual convergence has been achieved.

The arbitrary Lagrangian-Eulerian description for evolving the computational domain is employed. It applies to a wide range of problems including fixed boundary, specified boundary motion, fluid-solid interaction, and free surface.

Client-server arrangement utilizes N+2 processes, where N is the specified number of subdomains. The other two processes are associated with the Server and the coordination Client.

A majority of the source code is written in C. Numerical kernels are written in Fortran. Remote memory is accessed via PVM.

Spectrum Capabilities

External Flow about the Pantograph Cover of a High-Speed Train

Prediction of acoustic noise generated from unsteady flow phenomena around the pantograph cover of a high-speed train. Although the fairing effectively reduces the wind noise generated by a pantograph, the shroud itself generates noise over a different frequency range.

Train Tunnel Entry Simulation

The form of the compression wave generated by a high-speed train as it enters a tunnel is highly dependent on the ratio of cross-sectional areas of the train and the tunnel, as well as the shape of the nose of the lead car. Such pressure waves can cause uneasiness to the traveler's ear, and may cause the vehicle to rock and sway on its suspension. This study provided design loadings due to tunnel entry for the undercarraige and shell of the lead car.

Aeroelastic Deformation of a Supercritical Wing

Analysis of the aeroelastic response of an experimental wing configuration under in-flight conditions reveals an instability that traditional analyses were unable to predict.

Flow Simulations in Deformable Arteries

Investigation of blood flow conditions in critical human arteries shows that at low speed, lipid particles are not properly cleared from the abdominal aorta and carotid arteries, encouraging plaque to accumulate in vessel sinus regions.

Spectrum Development

The port of Parallel Spectrum to the T3E was undertaken by PSC in late August, 1996.

Construction of the T3E code uncovered problems in both the compiler and loader. The former problem revealed a pattern matching discrepancy that is currently outstanding. The latter problem was determined to be memory allocation error. It has been resolved.

Optimized single processor versions of Spectrum have been ported to the J90 and T3E at PSC. Validation on both platforms completed.

Multiprocessor codes for the J90 and T3E have also been built. The J90 validation has been completed. The T3E version is currently in testing. Two odd problems arose. The first involved taking the square root of SYS_REAL_MAX. The other was caused by the unusual way in which Cray Network PVM handles 64-bit integers (PvmDataDefault).

While testing a number of larger problems, we discovered that in its current configuration, we can not wire-up large models due to insufficient memory. Additional memory for the set of command processors has been taken under consideration (it appears likely that it will be doubled).

We're completing a demonstration that will be run on the floor at SC'96.

Parallel File I/O Library (PFIO)

Description

Development

Performance

Summary

PFIO Description

The PFIO library provides a pair of highly optimized Fortran I/O routines for moving data between MPP memory and disk.

PSC News Article

PSC Web Documentation

PFIO Development

T3E port of the PFIO library was completed shortly after the arrival of PSC's small prototype machine in June. Only the makefiles required modification.

Development has been suspended, pending the arrival of a new I/O subsystem for jaromir.

PFIO Performance

• PVP Platforms

  J90 , C90

• MPP Platforms

  T3D , T3E

PFIO Summary

The PFIO library is currently available for PVP (J90/C90) and MPP (T3D/T3E) platforms.

System monitoring indicates frequent and regular use by PSC clients.

Performance of T3E code is severely limited by the current architecture. A single GigaRing with one I/O processor is responsible for brokering all I/O requests. Final configuration of PSC's T3E will feature additional GigaRings and disk striping, which should alleviate bandwidth problems.

Requests from other T3E sites for distribution copies of this library have already been received and processed.

Transfer rates for PFIO routines represent peak achievable rates for the subject I/O systems, hence it is a staple of PSC's test suite.

Parallel Jacobi Eigensolver Library (PJAC)

Description

Development

Performance

Summary

PJAC Description

The PJAC library provides scalable eigensolvers for real/complex, symmetric/hermitian matrices. File handling routines for creating input systems and viewing output systems are bundled with the solver.

PSC News Article

PSC Web Documentation

PJAC Development

Added complex hermitian solver and initiated work on the general complex case.

Constructed validator codes for all supported platforms.

T3E port required minimal effort. Some intrinsics were moved into the shared memory library (SHMEM) and given new names. Makefile changes also required.

Modifications required to support arbitrary partition sizes completed in late October (power-of-two restriction removed on T3E). The form of the rotation scheme was revised to eliminate a range error that occurs for three specific cases and only on 256 processor T3E partitions. The changes are currently in testing.

PJAC Performance

T3D

T3E

PJAC Summary

The PJAC code is based on an opportunistic algorithm that exploits the performance of SGEMM/CGEMM (matrix-matrix multiply) and bandwidth associated with block communications.

The PJAC kernels demonstrate the second highest floating point rate recorded for a T3D application. Their performance is representative of the peak rates one might expect to achieve on an MPP platform, hence the PJAC library examples have been incorporated into PSC's MPP test suite.

PJAC solvers have been incorporated into major chemistry and material science research codes in the U.S. and Europe. The underlying algorithm does not suffer from round-off problems associated with bisection methods.

Ratios of T3D time to T3E time range from 2 to 4. Smaller problems that fit entirely into secondary cache show the most improvement.

Timings for the complex hermitian solver are 3-4 times the corresponding real symmetric timings, as would be expected.

A detailed study of primary kernel components revealed that some BLAS routines on the T3E perform at levels below the T3D. We identified three routines: SGEMV/CGEMV, SAXPBY/CAXPBY, SCOPY/CCOPY. A Fortran version of SGEMV was found to be faster than the library code, so it was substituted. Current performance is around 61 GFlop/s. Expected improvements should yield a rate of close to 100 GFlop/s on the current machine (256 PEs).

Parallel Finite Element Method (PFEM)

Description

Development

Performance

Summary

PFEM Description

The PFEM application is a research-level finite element code for suitable for solving Dirichlet-Poisson and Neumann boundary value problems on rectangular domains in two dimensions.

PSC News Article

Cray User Group Article

PFEM Development

Constructed an HPF port for the purpose of testing alpha and beta releases of the DEC Fortran 90 compiler and Parallel Software Environment targeting the new multiprocessor AlphaServer platforms and Memory Channel architectures.

Collaborating with a biomedical group at the University of Pittsburgh on a dipole model of the human head. Small grant ($100K) awarded for postdoc support. Larger grant proposal ($500K) seeking partial support for the PIs in progress.

Progress on ANSYS/PFEM hybrid has been slow. Generalizations required of PFEM are wide-ranging. Complexity level is high. Incorporation of Neumann boundary conditions and cubic elements in 2D a step in the right direction. The code responsible for creating the necessary element meshes and permutation mappings has also been split away from the kernel.

Initiated testing of a variation of the core algorithm in which a free-matrix implementation of the conjugate gradient method is used to handle element couplings resulting from imperfect colorings.

T3E port of existing code required minimal effort. Some intrinsics were moved into the shared memory library (SHMEM) and given new names. Makefile and job file changes were also required.

PFEM Performance

T3D/C90

T3E

PFEM Summary

Performance of the PFEM code is highly dependent on network latency. On the T3D, about half the time is spent in communications routines, with the other half spent in computational loops. It is an excellent code for stressing MPP network designs.

Early reports indicated that overheads and bandwidth associated with the network would be the same on the T3E as they were on the T3D. Coupled with the preceding statement, this implies that performance gains for PFEM should be bounded by 2, but measurements are consistently about twice this. This demonstrates that latencies associated with remote memory accesses have been reduced by more than half.

Scaling characteristics of the PFEM code are nearly linear on the T3E, as they were on the T3D and C90. Estimates based on Amdahl's Law indicate that the work performed in parallel regions of the kernel exceeds 99.8% of the total work.