Bridges User Guide

 

Using Bridges' GPU nodes

A standard NVIDIA accelerator environment is installed on  Bridges' GPU nodes.  If you have programmed using GPUs before, you should find this familiar.   Please contact This email address is being protected from spambots. You need JavaScript enabled to view it. for more help.

GPU Nodes

There are two types of GPU nodes on Bridges: the Phase 1 nodes use NVIDIA K80 GPUs;  the Phase 2 nodes use NVIDIA P100 GPUs.

Phase 1 nodes: The 16 Phase 1 GPU nodes are HPE Apollo 2000s, each with 2 NVIDIA K80 GPUs, 2 Intel Xeon E5-2695 v3 CPUs (14 cores per CPU) and 128GB RAM.
 Details

Phase 2 nodes: The 32 Phase 2 GPU nodes are HPE Apollo 2000s, each with 2 NVIDIA P100 GPUs,  2 Intel Xeon E5-2683 v4 CPUs (16 cores per CPU) and 128GB RAM.
 Details

File Systems

The /home, /pylon1 and /pylon2 file systems are available on all of these nodes.  See the File Spaces section of the User Guide for more information on these file systems.

Compiling and Running jobs

Use the GPU partition, either in batch or interactively, to compile your code and run your jobs.  See the Running Jobs section of the User Guide for more information on Bridges' partitions and how to run jobs. 

 

CUDA

To use CUDA, first you must load the CUDA module.  To see all versions of CUDA that are available, type:

module avail cuda

Then choose the version that you need and load the module for it.

module load cuda

loads the default CUDA.   To load a different version, use the full module name.

module load cuda/8.0

 CUDA 8 codes should run on both types of Bridges' GPU nodes with no issues.  CUDA 7 should only be used on the  K80 GPUs (Phase 1).  Performance may suffer with CUDA 7 on the P100 nodes (Phase 2).

 

OpenACC

Our primary GPU programming environment is OpenACC.

The PGI compilers are available on all GPU nodes. To set up the appropriate environment for the PGI compilers, use the  module  command:

module load pgi

Read more about the module command at PSC.  

If you will be using these compilers often, it will be useful to add this command to your shell initialization script.

There are many options available with these compilers. See the online man pages (“man pgf90”,”man pgcc”,”man pgCC”) for detailed information.  You may find these basic OpenACC options a good place to start:

pgcc –acc yourcode.c
pgf90 –acc yourcode.f90

 

P100 node users  should add the “-ta=tesla,cuda8.0” option to the compile command, for example:

pgcc -acc -ta=tesla,cuda8.0 yourcode.c
 

Adding the “-Minfo=accel” flag to the compile command (whether pgf90, pgcc or pgCC) will provide useful feedback regarding compiler errors or success with your OpenACC commands.

pgf90 -acc -Minfo=accel yourcode.f90

Hybrid MPI/GPU Jobs

To run a hybrid MPI/GPU job use the following commands for compiling your program:

module load cuda
module load mpi/pgi_openmpi
mpicc -acc yourcode.c

When you execute your program you must first issue the above two module load commands.

Profiling and Debugging

The environment variables PGI_ACC_TIME, PGI_ACC_NOTIFY and PGI_ACC_DEBUG can provide profiling and debugging information for your job. Specific commands depend on the shell you are using.
 Unix shells

Send email to This email address is being protected from spambots. You need JavaScript enabled to view it. to request that additional CUDA-oriented debugging tools be installed.

 

  • bash

  • csh

Performance profiling

Enable runtime GPU performance profiling by setting the PGI_ACC_TIME environment variable.

export PGI_ACC_TIME=1

 

Debugging

Basic debugging can be accomplished by setting the PGI_ACC_NOTIFY environment variable. For more detail, set PGI_ACC_NOTIFY to 3.

export PGI_ACC_NOTIFY=1

 

A further level of debugging is available with the PGI_ACC_DEBUG environment variable.

export PGI_ACC_DEBUG=1

Performance profiling

Enable runtime GPU performance profiling by setting the PGI_ACC_TIME environment variable.

setenv PGI_ACC_TIME 1

 

Debugging

Basic debugging can be accomplished by setting the PGI_ACC_NOTIFY environment variable. For more detail, set PGI_ACC_NOTIFY to 3.

setenv PGI_ACC_NOTIFY 1

 

A further level of debugging is available with the PGI_ACC_DEBUG environment variable.

setenv PGI_ACC_DEBUG 1

 

 

 

New on Bridges

DDT version 7.0 installed
DDT documentation

Runtime limit on LM nodes now 2 weeks
See partition limits

Omni-Path User Group

The Intel Omni-Path Architecture User Group is open to all interested users of Intel's Omni-Path technology.

More information on OPUG