Bridges-2 User Guide

This User Guide is being updated constantly as Bridges-2 is brought into production. Please check back often for the latest information. If you have questions at anytime, please email help@psc.edu

We take security very seriously. Please take a minute now to read PSC policies on passwords, security guidelines, resource use, and privacy. You are expected to comply with these policies at all times when using PSC systems. If you have questions at any time, you can send email to help@psc.edu.

Are you new to HPC?

If you are new to high performance computing, please read Getting Started with HPC before you begin your research on Bridges-2. It explains HPC concepts which may be unfamiliar. You can also check the Introduction to Unix or the Glossary for quick definitions of terms that may be new to you.

We hope that that information along with the Bridges-2 User Guide will have you diving into your work on Bridges-2. But if you have any questions, don’t hesitate to email us for help at help@psc.edu.

Questions?

PSC User Services is here to help you get your research started and keep it on track. If you have questions at any time, you can send email to help@psc.edu.

Set your PSC password

Before you can connect to Bridges-2, you must have a PSC password.

If you have an active account on any other PSC system: PSC usernames and passwords are the same across all PSC systems. You will use the same username and password on Bridges-2 as for your other PSC account.

If you do not have an active account on any other PSC system: You must create a PSC password. Go to the web-based PSC password change utility  at  apr.psc.edu  to set your PSC password.

PSC password policies

Computer security depends heavily on maintaining secrecy of passwords.

PSC uses Kerberos authentication on all its production systems, and your PSC password (also known as your Kerberos password) is the same on all PSC machines.

Set your initial PSC password

When you receive a PSC account, go to the web-based PSC password change utility to set your password.  For security, you should use a unique password for your PSC account, not one that you use for other sites.

Change your PSC password

Changing your password changes it on all PSC systems. To change your Kerberos password, use the web-based PSC password change utility .

Please note that changing your password on the XSEDE Portal does not change it on PSC systems and will not prevent your PSC password from expiring.

PSC password requirements

Your password must:

  • be at least eight characters long
  • contain characters from at least three of the following groups:
    • lower-case letters
    • upper-case letters
    • digits
    • special characters, excluding apostrophes (') and quotes (")
  • be different from the last three PSC passwords you have used
  • be changed at least once per year

Password safety

Under NO circumstances does PSC reveal any passwords over the telephone, FAX them to any location, send them through email, set them to a requested string, or perform any other action that could reveal a password.

If someone claiming to represent PSC contacts you and requests information that in any manner would reveal a password, be assured that the request is invalid and do NOT comply.

System Configuration

Bridges-2 is designed for converged HPC + AI + Data. Its custom topology is optimized for data-centric HPC, AI, and HPDA (High Performance Data Analytics). An extremely flexible software environment along with community data collections and BDaaS (Big Data as a Service) provide the tools necessary for modern pioneering research. The data management system, Ocean, consists of two-tiers, disk and tape, transparently managed as a single, highly usable namespace.

Compute nodes

Bridges-2 will hold three types of nodes: GPU, “Regular Memory” and “Extreme Memory”.

 

GPU nodes

Bridges-2’s GPU nodes provide exceptional performance and scalability for deep learning and accelerated computing, with a total of 40, 960 CUDA cores and 5,120 tensor cores.

GPU nodes
Number 24
GPUs per node 8 NVIDIA Tesla V100-32GB SXM2
GPU performance 1 Pf/s tensor
CPUs 2 Intel Xeon Gold 6248 “Cascade Lake” CPUs
20 cores, 40 threads/CPU
2.50 – 3.90 GHz
RAM 512GB, DDR4-2933
Cache 27.5MB LLC, 6 memory channels
Node-local storage 7.68TB NVMe SSD
Network 2 Mellanox ConnectX-6 HDR Infiniband 200 Gbs/s Adapters

 

Regular Memory nodes

Regular Memory (RM) nodes provide extremely powerful general-purpose computing, machine learning and data analytics, AI inferencing, and pre- and post-processing.

RM nodes
Number 488 16
CPU 2 AMD EPYC 7742 CPUs
64 cores, 128 threads each
2.25-3.40 GHz
2 AMD EPYC 7742 CPUs
64 cores, 128 threads each
2.25-3.40 GHz
RAM 256GB 512GB
Cache 256MB L3, 8 memory channels 256MB L3, 8 memory channels
Node-local storage 3.84TB NVMe SSD 3.84TB NVMe SSD
Network Mellanox ConnectX-6-HDR Infiniband 200Gb/s Adapter Mellanox ConnectX-6-HDR Infiniband 200Gb/s Adapter

 

Extreme memory nodes

Extreme Memory (EM) nodes provide 4TB of shared memory for genome sequence assembly, graph analytics, statistics, and other applications requiring a large amount of memory for which distributed-memory implementations are not available.

EM nodes
CPU 4 Intel Xeon Platinum 8260M “Cascade lake” CPUs
24 cores, 48 threads each
2.40-3.90 GHz
RAM 4TB, DDR4-2933
Cache 37.75MB LLC, 6 memory channels
Node-local storage 7.68TB NVMe SSD
Network Mellanox ConnectX-6-HDR Infiniband 200Gb/s Adapter

 

Data Management

Data management on Bridges-2 is accomplished through a unified, high-performance filesystem for active project data, archive, and resilience, named Ocean.

Ocean consists of two tiers, disk and tape, transparently managed as a single, highly usable namespace.

Ocean’s disk subsystem, for active project data, is a high-performance, internally resilient Lustre parallel filesystem with 15PB of usable capacity, configured to deliver up to 129GB/s and 142GB/s of read and write bandwidth, respectively.

Ocean’s tape subsystem, for archive and additional resilience, is a high-performance tape library with 7.2PB of uncompressed capacity, configured to deliver 50TB/hour. Data compression occurs in hardware, transparently, with no performance overhead.

Connecting to Bridges-2

Bridges-2 contains two broad categories of nodes: compute nodes, which handle the production research computing, and login nodes, which are used for managing files, submitting batch jobs and launching interactive sessions. Login nodes are not suited for production computing.

When you connect to Bridges-2, you are connecting to a Bridges-2 login node.

See the Running Jobs section of this User Guide for information on production computing on Bridges-2. –>

You can connect to Bridges-2 via a web browser or through a command line interface.

Connect in a web browser

You can access Bridges-2 through a web browser by using the OnDemand software. You will still need to understand Bridges-2’s partition structure and the options which specify job limits, like time and memory use, but OnDemand provides a more modern, graphical interface to Bridges-2.

See the OnDemand section for more information.

Connect to a command line interface

You can connect to a traditional command line interface by logging in via one of these:

  • ssh, using either XSEDE or PSC credentials. If you are registered with XSEDE for DUO Multi-Factor Authentication (MFA), you can use this security feature in connecting to Bridges-2. See the XSEDE instructions to set up DUO for MFA.
  • XSEDE Single Sign On, including using Multi-Factor authentication if you are an XSEDE user

SSH

You can use an ssh client from your local machine to connect to Bridges-2 using either your PSC or XSEDE credentials.

SSH  is a program that enables secure logins over an unsecure network. It encrypts the data passing both ways so that if it is intercepted it cannot be read.

SSH is client-server software, which means that both the user’s local computer and the remote computer must have it installed. SSH server software is installed on all the PSC machines. You must install SSH client software on your local machine.

Free ssh clients for  Macs, Windows machines and many versions of Unix are available. Popular ssh clients (GUI) include PuTTY for Windows and Cyberduck for Macs. A command line version of ssh is installed on Macs  by default; if you prefer that, you can use it in the Terminal application. You can also check with your university to see if there is an ssh client that they recommend.

Once you have an ssh client installed, you can use either your PSC credentials or XSEDE credentials (optionally with DUO MFA) to connect to Bridges-2. Note that you must have created your PSC password before you can use ssh to connect to Bridges-2.

Use ssh to connect to Bridges-2 using XSEDE credentials and (optionally) DUO MFA:

  1. Using your ssh client, connect to hostname bridges2.psc.xsede.org  or bridges2.psc.edu  using port 2222.
    Either hostname will connect you to Bridges-2, but you must specify port 2222.
  2. Enter your XSEDE username and password when prompted.
  3. (Optional) If you are registered with XSEDE DUO, you will receive a prompt on your phone.  Once you have answered it, you will be logged in.

Use ssh to connect to Bridges-2 using PSC credentials:

  1. Using your ssh client, connect to hostname bridges2.psc.xsede.org  or bridges2.psc.edu  using the default port (22).
    Either hostname will connect you to Bridges-2. You do not have to specify the port.
  2. Enter your PSC username and password when prompted.

Read more about using SSH to connect to PSC systems

Public-private keys

You can also use public-private key pairs to connect to Bridges-2. To do so, you must first fill out this form to register your keys with PSC.

XSEDE single sign on

XSEDE users can use their XSEDE usernames and passwords in the XSEDE User Portal Single Sign On Login Hub (SSO Hub) to access bridges2.psc.xsede.org  or bridges2.psc.edu.

You must use DUO Multi-Factor Authentication in the SSO Hub.

See the XSEDE instructions to set up DUO for Multi-Factor Authentication.

Account Administration

Changing your PSC password

There are two ways to change or reset your PSC password:

When you change your PSC password, whether you do it via the online utility or via the kpasswd command on one PSC system, you change it on all PSC systems.

See PSC password policies.

Remember that your PSC password is separate from your XSEDE Portal password. Resetting one password does not change the other password.

Managing your XSEDE allocation

Most account management functions for your XSEDE grant are handled through the XSEDE User Portal. You can search the XSEDE user portal to get  help. Some common questions:

Changing your default shell

The change_shell command allows you to change your default shell. This command is only available on the login nodes.

To see which shells are available, type

change_shell -l

To change your default shell, type

change_shell newshell

where newshell is one of the choices output by the change_shell -l command. You must use the entire path output by change_shell -l, e.g. /usr/psc/shells/bash. You must log out and back in again for the new shell to take effect.

PSC account policies

The policies documented here are evaluated regularly to assure adequate and responsible administration of PSC systems for users. As such, they are subject to change at any time.

File Retention

PSC provides storage resources, for long-term storage and file management.

Files in a PSC storage system are retained for 3 months after the affiliated grant has expired.

Requesting a Refund

When appropriate, PSC provides refunds for jobs that failed due to circumstances beyond your control.

To request a refund, contact a PSC consultant or email help@psc.edu. In the case of batch jobs, we require the standard error and output files produced by the job. These contain information needed in order to refund the job.

Maintenance Downtimes

The following times are reserved for system maintenance.

Note that the machines will not always be down at these times. However, PSC reserves the right to use these scheduled times for maintenance as necessary.

Computing resources Wed. 9 am – 3 pm
Storage resources Wed. 9 am – 3 pm
Gigapop Tue. & Fri. 7 – 8 am
PSC production network Tue. & Fri. 7 – 8 am

File Spaces

There are several distinct file spaces available on Bridges-2, each serving a different function.

  • Home ($HOME), your home directory on Bridges-2
  • Ocean,  a system for persistent file storage.
  • Node-local storage, scratch storage on the local SSD on the node running a job
  • Memory storage, scratch storage in the local memory associated with a running job

File expiration

Three months after your grant expires all of your Bridges-2 files associated with that grant will be deleted, no matter which file space they are in. You will be able to login during this 3-month period to transfer files, but you will not be able to run jobs or create new files.

File permissions

Access to files in any Bridges-2 space is governed by Unix file permissions. If  your data has additional security or compliance requirements, please contact compliance@psc.edu.

Unix file permissions

For detailed information on Unix file protections, see the man page for the chmod (change mode) command.

To share files with your group, give the group read and execute access for each directory from your top-level directory down to the directory that contains the files you want to share.

chmod g+rx directory-name

Then give the group read and execute access to each file you want to share.

chmod g+rx filename

To give the group the ability to edit or change a file, add write access to the group:

chmod g+rwx filename

Access Control Lists

If you want more fine-grained control than Unix file permissions allow —for example, if you want to give only certain members of a group access to  a file, but not all members—then you need to use Access Control Lists (ACLs). Suppose, for example, that you want to give janeuser read access to a file in a directory, but no one else in the group.

Use the setfacl (set file acl) command to give janeuser read and execute access on the directory:

setfacl -m user:janeuser:rx directory-name

for each directory from your top-level directory down to the directory that contains the file you want to share with janeuser. Then give janeuser access to a specific file with

setfacl -m user:janeuser:r filename

User janeuser will now be able to read this file, but no one else in the group will have access to it.

To see what ACLs are set on a file, use the getfacl (get file acl) command.

There are man pages for chmod, setfacl and getfacl.

Home ($HOME)

This is your Bridges-2 home directory. It is the usual location for your batch scripts, source code and parameter files. Its path is /home/username, where  username is your PSC username. You can refer to your home directory with the environment variable $HOME. Your home directory is visible to all of Bridges-2’s nodes.

Your home directory is backed up daily, although it is still a good idea to store copies of your important  files in another location, such as the Ocean file system or on a local file system at your site. If you need to recover a home directory file from backup send email to help@psc.edu. The process of recovery will take 3 to 4 days.

$HOME quota

Your home directory has a 10GB quota. You can check your home directory usage using the quota command or the command du -sh. To improve the access speed to your home directory files you should stay as far below your home directory quota as you can.

Grant expiration

Three months after a grant expires, the files in your home directory associated with that grant will be deleted.

Ocean

The path of your Ocean home directory is /ocean/groupname/username, where groupname is the Unix group id associated with your grant. Use the id command to find your group name.

The command id -Gn will list all the Unix groups you belong to.

The command id -gn will list the Unix group associated with your current session.

If you have more than one grant, you will have an Ocean directory for each grant. Be sure to use the appropriate directory when working with multiple grants.

Ocean quota

Your usage quota for each of your grants is the Ocean storage allocation you received when your proposal was approved. If your total use in Ocean exceeds this quota you won’t be able to run jobs on Bridges-2 until you are under quota again.

Use the du -sh  or projects command to check your Ocean usage. You can also check your usage on the XSEDE User Portal.

If you have multiple grants, it is very important that you store your files in the correct Ocean directory.

Grant expiration

Three months after a grant expires, the files in any Ocean directories associated with that grant will be deleted.

Node-local

Each of Bridges-2’s nodes has a local file system attached to it. This local file system is only visible to the node to which it is attached. The local file system provides fast access to local storage.

If your application performs a lot of small reads and writes, then you could benefit from using this space.

Node-local storage is only available when your job is running, and can only be used as working space for a running job. Once your job finishes your local files are inaccessible and deleted. To use local space, copy files to it at the beginning of your job and back out to a persistent file space before your job ends.

If a node crashes all the node-local files are lost. You should checkpoint theses files by copying them to Ocean during long runs.

Memory files

You can also use the memory allocated for your job for IO rather than using disk space.

The amount of memory space available to you depends on the size of the memory on the nodes and the number of nodes you are using. You can only perform IO to the memory of nodes assigned to your job.

If you do not use all of the cores on a node, you are allocated memory in proportion to the number of cores you are using. Note that you cannot use 100% of a node’s memory for IO; some is needed for program and data usage.

This space is only available to you while your job is running, and can only be used as working space for a running job. Once your job ends this space is inaccessible. To use memory files, copy files to it at the beginning of your job and back out to a permanent space before your job ends. If your job terminates abnormally your memory files are lost.

Transferring Files

Several methods are available to transfer files into and from Bridges-2.

Paths for Bridges-2 file spaces

To copy files into any of your Bridges-2 spaces, you need to know the path to that space on Bridges-2. The start of the full paths for your Bridges-2 directories are:

Home directory     /home/username

Ocean directory   /ocean/groupname/username

To find your groupname, use the command id -Gn. All of your valid groupnames will be listed. You have an Ocean directory for each grant you have.

Commands to transfer files

You can use rsync, scp or sftp to copy files to and from Bridges-2.

rsync

You can use the rsync command to copy files to and from Bridges-2. A sample rsync command to copy to a Bridges-2 directory is

rsync -rltpDvp -e 'ssh -l username' source_directory data.bridges2.psc.edu:target_directory

Substitute your username for ‘username‘. Make sure you use the correct groupname in your target directory. By default, rsync will not copy older files with the same name in place of newer files in the target directory. It will overwrite older files.

We recommend the rsync options -rltDvp. See the rsync man page for information on these options and other options you might want to use. We also recommend the option

-oMACS=umac-64@openssh.com

If you use this option, your transfer will use a faster data validation algorithm.

You may want to put your rsync command in a loop to insure that it completes. A sample loop is

RC=1
 n=0
 while [[ $RC -ne 0 && $n -lt 20 ]] do
     rsync source-file target-file
     RC = $?
     let n = n + 1
     sleep 10
 done

This loop will try your rsync command 20 times. If it succeeds it will exit. If an rsync invocation is unsuccessful the system will try again and pick up where it left off. It will copy only those files that have not already been transferred. You can put this loop, with your rsync command, into a batch script and run it with sbatch.

scp

To use scp for a file transfer you must specify a source and destination for your transfer. The format for either source or destination is

username@machine-name:path/filename

For transfers involving Bridges-2,  username is your PSC username. The machine-name should be given as data.bridges2.psc.edu. This is the name for a high-speed data connector at PSC. We recommend using it for all file transfers using scp involving Bridges-2. Using it prevents file transfers from disrupting interactive use on Bridges-2’s login nodes.

File transfers using scp must specify full paths for Bridges-2 file systems. See Paths for Bridges-2 file spaces for details.

sftp

To use sftp, first connect to the remote machine:

sftp username@machine-name

When  Bridges-2 is the remote machine, use your PSC userid as  username. The Bridges-2 machine-name should be specified as data.bridges2.psc.edu. This is the name for a high-speed data connector at PSC.  We recommend using it for all file transfers using sftp involving Bridges-2. Using it prevents file transfers from disrupting interactive use on Bridges-2’s login nodes.

You will be prompted for your password on the remote machine. If Bridges-2 is the remote machine, enter your PSC password.

You can then enter sftp subcommands, like put to copy a file from the local system to the remote system, or get to copy a file from the remote system to the local system.

To copy files into Bridges-2, you must either cd to the proper directory or use full pathnames in your file transfer commands. See Paths for Bridges-2 file spaces for details.

Transferring files using Two-Factor Authentication

If you are required to use Two-Factor Authentication (TFA) to access Bridges-2’s filesystems, you must enroll in XSEDE DUO. Once that is complete, use scp or sftp to transfer files to and from Bridges-2.

TFA users must use port 2222 and XSEDE Portal usernames and passwords, not PSC usernames and passwords. The machine name for these transfers is data.bridges2.psc.edu.

In the examples below, myfile is the local filename, XSEDE-username is your XSEDE Portal username and /path/to/file is the full path to the file on a Bridges-2 filesystem. Note that -P (capital P) is necessary.

scp

Transfer a file from a local machine to Bridges-2:

scp -P 2222 myfile XSEDE-username@data.bridges2.psc.edu:/path/to/file

Transfer a file from Bridges-2 to a local machine:

scp -P 2222 XSEDE-username@data.bridges2.psc.edu:/path/to/file myfile

sftp

Use sftp interactively:

sftp -P 2222 XSEDE-username@data.bridges2.psc.edu

Then use the put command to copy a file from the local machine to Bridges-2, or the get command to transfer a file from Bridges-2 to the local machine.

Graphical SSH client

If you are using a graphical SSH client, configure it to connect to data.bridges2.psc.edu on port 2222/TCP. Login using your XSEDE Portal username and password.

Globus

Globus can be used for any file transfer to Bridges-2. It tracks the progress of the transfer and retries when there is a failure; this makes it especially useful for transfers involving large files or many files.

To use Globus to transfer files you must authenticate either via a Globus account or with InCommon credentials.

To use a Globus account for file transfer, set up a Globus account at the Globus site.

To use InCommon credentials to transfer files to/from Bridges-2, you must first provide your CILogin Certificate Subject information to PSC. Follow these steps:

  1. Find your Certificate Subject string
    1. Navigate your web browser to https://cilogon.org/.
    2. Select your institution from the ‘Select an Identity Provider’ list.
    3. Click the ‘Log On’ button.  You will be taken to the web login page for your institution.
    4. Login with your username and password for your institution.
      • If your institution has an additional login requirement (e.g., Duo), authenticate to that as well.
    5. After successfully authenticating to your institution’s web login interface, you will be returned to the CILogon webpage.
    6. Click on the Certificate Information drop down link to find the ‘Certificate Subject’.
  2. Send your Certificate Subject string to PSC
    1. In the CILogon webpage, select and copy the Certificate Subject text. Take care to get the entire text string if it is broken up onto multiple lines.
    2. Send email to support@psc.edu.  Paste your Certificate Subject field into the message, asking that it be mapped to your PSC username.

Your CILogin Certificate Subject information will be added within one business day, and you will be able to begin transferring files to and from Bridges-2.

Globus endpoints

Once you have the proper authentication you can initiate file transfers from the Globus site. A Globus transfer requires a Globus endpoint, a file path and a file name for both the source and destination. The endpoints for Bridges-2 are:

  • psc#bridges2-xsede if you are using an XSEDE User Portal account for authentication
  • psc#bridges2-cilogon if you are using InCommon for authentication

These endpoints are owned by psc@globusid.org. If you use DUO MFA for your XSEDE authentication, you do not need to because you cannot use it with Globus. You must always specify a full path for the Bridges-2 file systems. See Paths for Bridges-2 file spaces for details.

Globus-url-copy

The globus-url-copy command can be used if you have access to Globus client software. Both the globus-url-copy and myproxy-logon commands are available on Bridges-2, and can be used for file transfers internal to the PSC.

To use globus-url-copy you must have a current user proxy certificate. The command grid-proxy-info will tell you if you have a current user proxy certificate and what the remaining life of your certificate is.

Use the myproxy-logon command to get a valid user proxy certificate if any one of these applies:

  • you get an error from the grid-proxy-info command
  • you do not have a current user proxy certificate
  • the remaining life of your certificate is not sufficient for your planned file transfer

When prompted for your MyProxy passphrase enter your XSEDE Portal password.

To use globus-url-copy for transfers to a machine you must know the Grid FTP server address. The Grid FTP server address for Bridges-2 is

gsiftp://gridftp.bridges2.psc.edu

The use of globus-url-copy always requires full paths. See Paths for Bridges-2 file spaces for details.

Transfer rates

PSC maintains a Web page at http://speedpage.psc.edu that lists average data transfer rates between XSEDE resources.  If your data transfer rates are lower than these average rates or you believe that your file transfer performance is subpar, send email to help@psc.edu.  We will examine approaches for improving your file transfer performance.

Programming Environment

Bridges-2 provides a rich programming environment for the development of applications.

C, C++ and Fortran

AMD (AOCC), Intel, Gnu and PGI compilers for C, C++ and Fortan are available on Bridges-2. The compiler commands are:

C C++ Fortran
AMD clang clang flang
Intel icc icpc ifort
Gnu gcc g++ gfortran
PGI pgcc pgc++ pgfortran

 

There are man pages for each of the compilers.

See also:

OpenMP programming

To compile OpenMP programs you must add an option to your compile command:

Compiler Option
Intel -qopenmp
for example: icc -qopenmp yprog.c
Gnu -fopenmp
for example: gcc -fopenmp myprog.c
PGI -mp
for example: pgcc -mp myprog.c

See also:

MPI programming

Three types of MPI are supported on Bridges-2: MVAPICH2, OpenMPI and Intel MPI.

To compile an MPI program, you must do these things:

  • Load the module for the MPI type you want to use.
  • Load the module for the compiler that you want to use.
  • Issue the appropriate MPI wrapper command to compile your program.

The three MPI types  may perform differently on different problems or in different programming environments. If you are having trouble with one type of MPI, please try using another type. Contact help@psc.edu for more help.

Note: When using Intel MPI, you must set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT to 0. Otherwise the mpirun task placement settings you give will be ignored. The command to do this is:

For the BASH shell:

export I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0

For the CSH shell:

setenv I_MPI_JOB_RESPECT_PROCESS_PLACEMENT 0


See also:

Other languages

Other languages, including Java, Python, R,  and MATLAB, are available. See the software page for information.

Debugging and performance analysis

DDT

DDT is a debugging tool for C, C++ and Fortran 90 threaded and parallel codes. It is client-server software. Install the client on your local machine and then you can access the GUI on Bridges-2 to debug your code.

See the DDT page for more information.

Software

Bridges-2 has a broad collection of applications installed. See the list of software installed on Bridges-2.

Additional software may be installed by request. Request that additional applications be installed by using the Software Request form.

OnDemand

The OnDemand interface allows you to conduct your research on Bridges-2 through a web browser. You can manage files – create, edit and move them – submit and track jobs, see job output, check the status of the queues, run a Jupyter notebook through JupyterHub and more, without logging in to Bridges-2 via traditional interfaces.

OnDemand was created by the Ohio Supercomputer Center (OSC). In addition to this document, you can check the extensive documentation for OnDemand created by OSC, including many video tutorials, or email help@psc.edu.

More information on using OnDemand on Bridges-2 will be available soon.

GPU nodes

Bridges-2’s GPU nodes provide substantial, complementary computational power for deep learning, simulations and other applications.

A standard NVIDIA accelerator environment is installed on  Bridges-2’s GPU nodes. If you have programmed using GPUs before, you should find this familiar. Please contact help@psc.edu for more help.

The GPU nodes on Bridges-2 are available to those with a Bridges-2 “GPU” allocation. You can see which of Bridges-2’s resources that you have been allocated with the projects command.

File Systems

The /home and Ocean file systems are available on all of these nodes.  See the File Spaces section of this User Guide for more information on these file systems.

Compiling and Running jobs

Use the GPU partition, either in batch or interactively, to compile your code and run your jobs. See the Running Jobs section of this User Guide for more information on Bridges-2’s partitions and how to run jobs. –>

CUDA

More information on using CUDA on Bridges-2 can be found in the CUDA document.

To use CUDA, first you must load the CUDA module. To see all versions of CUDA that are available, type:

module avail cuda

Then choose the version that you need and load the module for it.

module load cuda

loads the default CUDA.   To load a different version, use the full module name.

module load cuda/8.0

OpenACC

Our primary GPU programming environment is OpenACC.

The PGI compilers are available on all GPU nodes. To set up the appropriate environment for the PGI compilers, use the  module  command:

module load pgi

Read more about the module command at PSC.

If you will be using these compilers often, it will be useful to add this command to your shell initialization script.

There are many options available with these compilers. See the online man pages  (“man pgf90”,”man pgcc”,”man pgCC”) for detailed information.  You may find these basic OpenACC options a good place to start:

pgcc –acc yourcode.c  
pgf90 –acc yourcode.f90

Adding the “-Minfo=accel” flag to the compile command (whether pgf90, pgcc or pgCC) will provide useful feedback regarding compiler errors or success with your OpenACC commands.

pgf90 -acc -Minfo=accel yourcode.f90

Hybrid MPI/GPU Jobs

To run a hybrid MPI/GPU job use the following commands for compiling your program:

module load cuda
 module load mpi/pgi_openmpi
 mpicc -acc yourcode.c

When you execute your program you must first issue the above two module load commands.

Profiling and Debugging

For CUDA codes, use the command line profiler nvprof. See the CUDA document for more information.

For OpenACC codes, the environment variables PGI_ACC_TIME, PGI_ACC_NOTIFY and PGI_ACC_DEBUG can provide profiling and debugging information for your job. Specific commands depend on the shell you are using.

Bash shell C shell
Performance profiling
Enable runtime GPU performance profiling export PGI_ACC_TIME=1 setenv PGI_ACC_TIME 1
Debugging
Basic debugging
For data transfer information, set PGI_ACC_NOTIFY to 3
export PGI_ACC_NOTIFY=1 setenv PGI_ACC_NOTIFY 1
More detailed debugging export PGI_ACC_DEBUG=1 setenv PGI_ACC_DEBUG 1

Containers

Containers are stand-alone packages holding the software needed to create a very specific computing environment. If you need a very specialized environment, you can create  your own container or use one that is already installed on Bridges-2. Singularity is the only type of container supported on  Bridges-2.

Creating a container

Singularity is the only container software supported on Bridges-2. You can create a Singularity container, copy it to Bridges-2 and then execute your container on Bridges-2, where it can use Bridges-2’s compute nodes and filesystems. In your container you can use any software required by your application: a different version of CentOS,  a different Unix operating system, any software in any specific version needed. You can install your Singularity container without any intervention from PSC staff.

See the PSC documentation on Singularity for more details on producing your own container and Singularity use on Bridges-2.

Publicly available containers on Bridges-2

We have installed many containers from the NVIDIA GPU Cloud (NGC) on Bridges-2. These containers are fully optimized, GPU-accelerated environments for AI, machine learning and HPC. They can only be used on the Bridges-2 GPU nodes.

These include containers for:

  • Caffe and Caffe2
  • Microsoft Cognitive Toolkit
  • DIGITS
  • Inference Server
  • MATLAB
  • MXNet
  • PyTorch
  • Tensorflow
  • TensorRT
  • Theano
  • Torch

See the PSC documentation on Singularity for more details on Singularity use on Bridges-2.

Datasets

A community dataset space allows Bridges-2’s users from different grants to share data in a common space. Bridges-2 hosts both community (public) and private datasets, providing rapid access for individuals, collaborations and communities with appropriate protections.

If there is a dataset which you need for your research on Bridges-2 and you believe it would be of widespread interest, you can request that it be installed for public use by using the Dataset Request form.

Public datasets

Some datasets are available to anyone with a Bridges-2 account. They include:

2019nCoVR: 2019 Novel Coronavirus Resource

The 2019 Novel Coronavirus Resource concerns the outbreak of novel coronavirus in Wuhan, China since December 2019. For more details about the statistics, metadata, publications, and visualizations of the data, please visit https://bigd.big.ac.cn/nco.

COCO 

COCO (Common Objects in Context) is a large scale image dataset designed for object detection, segmentation, person keypoints detection, stuff segmentation, and caption generation. Please visit http://cocodataset.org/ for more information on COCO, including details about the data, paper, and tutorials.

PREVENT-AD 

The PREVENT-AD (Pre-symptomatic Evaluation of Experimental or Novel Treatments for Alzheimer Disease) cohort is composed of cognitively healthy participants over 55 years old, at risk of developing Alzheimer Disease (AD) as their parents and/or siblings were/are affected by the disease. These ‘at-risk’ participants have been followed for a naturalistic study of the presymptomatic phase of AD since 2011 using multimodal measurements of various disease indicators. Two clinical trials intended to test pharmaco-preventive agents have also been conducted. The PREVENT-AD research group is now releasing data openly with the intention to contribute to the community’s growing understanding of AD pathogenesis.

ImageNet

ImageNet is an image dataset organized according to WordNet hierarchy. See the ImageNet website for complete information.

Natural Languge Tool Kit Data

NLTK comes with many corpora, toy grammars, trained models, etc. A complete list of the available data is posted at: http://nltk.org/nltk_data/

MNIST

Dataset of handwritten digits used to train image processing systems.

BLAST

The BLAST databases can be accessed through the environment variable $BLASTDB after loading the BLAST module.

Other genomics datasets

Other available datasets are typically used with a particular genomics package. These include:

  • Homer
  • MetaPhlAn2

Gateways

Bridges-2 hosts a number of gateways – web-based, domain-specific user interfaces to applications, functionality and resources that allow users to focus on their research rather than programming and submitting jobs. Gateways  provide intuitive, easy-to-use interfaces to complex functionality and data-intensive workflows.

Gateways can manage large numbers of jobs and provide collaborative features, security constraints and provenance tracking, so that you can concentrate on your analyses instead of on the mechanics of accomplishing them.

Security Guidelines and Policies

PSC policies regarding privacy, security and the acceptable use of PSC resources are documented here. Questions about any of these policies should be directed to PSC User Services.

See also policies for:

Security Measures

Security is very important to PSC. These policies are intended to ensure that our machines are not misused and that your data is secure.

What You Can Do:

You play a significant role in security!  To keep your account and PSC resources secure, please:

  • Be aware of and comply with PSC’s policies on security, use and privacy found in this document
  • Choose strong passwords and don’t share them between accounts or with others. More information can be found in the PSC password policies.
  • Utilize your local security team for advice and assistance
  • Take the online XSEDE Cybersecurity Tutorial. Go to Online Training and click on “XSEDE Cybersecurity (CI-Tutor)
  • Keep your computer properly patched and protected
  • Report any security concerns to the PSC help desk ASAP by calling the PSC hotline at: 412-268-6350 or email help@psc.edu
What We Will Never Do:
  • PSC will never send you unsolicited emails requesting confidential information.
  • We will also never ask you for your password via an unsolicited email or phone call.

Remember that the PSC help desk is always a  phone call away to confirm any correspondence at  412-267-6350.

If you have replied to an email appearing to be from PSC and supplied your password or other sensitive information, please contact the help desk immediately.

What You Can Expect:
  • We will send you email when we need to communicate with you about service outages, new HPC resources, and the like.
  • We will send you email when your password is about to expire and ask you to change it by using the web-based PSC password change utility.

Other Security Policies

  • PSC password policies
  • Users must connect to PSC machines using ssh in order to avoid remote logins with clear text passwords
  • We vigilantly monitor our computer systems and network connections for security violations
  • We are in close contact with the CERT Coordination Project with regard to possible Internet security violations

Reporting Security Incidents

To report a security incident you should contact our Hotline at 412-268-6350. To report non-emergency security incidents you can send email to help@psc.edu.

PSC Acceptable Use Policy

PSC’s resources are vital to the scientific community, and we have a responsibility to ensure that all resources are utilized in a responsible manner. PSC has legal and other obligations to protect its services, resources, and the intellectual property of users. Users share this responsibility by observing the rules of acceptable use that are outlined in this document. Your on-line assent to this Acceptable Use Policy is your acknowledgment that you have read and understand your responsibilities as a user of PSC Services and Resources, which refers to all computers owned or operated by PSC and all hardware, data, software, storage systems and communications networks associated with these computers. If you have questions, please contact PSC User Services.

By using PSC Services and Resources associated with your allocation, you agree to comply with the following conditions of use:

  1. You will protect any access credentials (e.g., private keys, tokens & passwords) that are issued for your sole use by PSC and not knowingly allow any other person to impersonate or share any of your identities.
  2. You will not use PSC Services and Resources for any unauthorized purpose, including but not limited to:
    1. Financial gain
    2. Tampering with or obstructing PSC operations
    3. Breaching, circumventing administrative, or security controls
    4. Inspecting, modifying, distributing, or copying privileged data or software without proper authorization, or attempting to do so
    5. Supplying, or attempting to supply, false or misleading information or identification in order to access PSC Services and Resources
  3. You will comply with all applicable laws and relevant regulations, such as export control law or HIPAA.
  4. You will immediately report any known or suspected security breach or misuse of PSC access credentials to help@psc.edu.
  5. Use of PSC Services and Resources is at your own risk. There are no guarantees that PSC Services and Resources will be available, that they will suit every purpose, or that data will never be lost or corrupted. Users are responsible for backing up critical data.
  6. Logged information, including information provided by you for registration purposes, will be used solely for administrative, operational, accounting, monitoring and security purposes.
  7. Violations of this Acceptable Use Policy and/or abuse of PSC Services and Resources may result in loss of access to PSC Services and Resources. Abuse will be referred to the PSC User Services manager and/or the appropriate local, state and federal authorities, at PSC’s discretion.
  8. PSC may terminate or restrict any user’s access to PSC Services and Resources, without prior notice, if such action is necessary to maintain computing availability and security for other users of the systems.
  9. Allocations are awarded solely for open research, intended for publication. You will only use PSC Computing Resources to perform work consistent with the stated allocation request goals and conditions of use as defined by your approved PSC project and this Acceptable Use Policy.
  10. PSC is entitled to regulate, suspend or terminate your access, and you will immediately comply with their instructions.

Privacy

Pittsburgh Supercomputing Center is committed to preserving your privacy. This privacy policy explains exactly what information is collected when you visit our site and how it is used.

This policy may be modified as new features are added to the site. Any changes to the policy will be posted on this page.

  • Any data automatically collected from our site visitors – domain name, browser types, etc. – are used only in aggregate to help us better meet site visitors’ needs.
  • There is no identification of individuals from our aggregate data. Therefore, unless you choose otherwise, you are totally anonymous when visiting our site.
  • We do not share data with anyone for commercial purposes.
  • If you choose to submit personally identifiable information to us electronically via the PSC feedback page, email, etc., we will treat it with the same respect for privacy afforded to mailed submissions. Submission of such information is always optional.

PSC respects individual privacy and takes great effort in supporting web site privacy policy outlined above. Please be aware, however, that we may publish URLs of other sites on our web site that may not adhere to the same policy.

 

Reporting a Problem

To report a problem on Bridges-2, please email help@psc.edu. Please report only one problem per email; it will help us to track and solve any issues more quickly and efficiently.

Be sure to include

  • an informative subject line
  • your username

If the question concerns a particular job, include these in addition:

  • the JobID
  • any error messages you received
  • the date and time the job ran
  • link to job scripts, output and data files
  • the software being used, and versions when appropriate
  • a screenshot of the error or the output file showing the error, if possible

 

Acknowledgement in Publications

All publications, copyrighted or not, resulting from an allocation of computing time on Bridges-2 should include an acknowledgement. Please acknowledge both the funding source that supported your access to PSC and the specific PSC resources that you used.

Please also acknowledge support provided by XSEDE’s ECSS program and/or PSC staff when appropriate.

Proper acknowledgment is critical for our ability to solicit continued funding to support these projects and next generation hardware.

Suggested text and citations follow.

XSEDE supported research on Bridges-2

We ask that you use the following text:

This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges-2 system, which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC).

Please include these citations:

Towns, J., Cockerill, T., Dahan, M., Foster, I., Gaither, K., Grimshaw, A., Hazlewood, V., Lathrop, S., Lifka, D., Peterson, G.D., Roskies, R., Scott, J.R. and Wilkens-Diehr, N. 2014. XSEDE: Accelerating Scientific Discovery. Computing in Science & Engineering. 16(5):62-74. http://doi.ieeecomputersociety.org/10.1109/MCSE.2014.80.

XSEDE ECSS Support

To acknowledge support provided through XSEDE’s Extended Collaborative Support Services (ECSS), please use this text:

We thank [consultant name(s)] for [his/her/their] assistance with [describe tasks such as porting code, optimization, visualization, etc.], which was made possible through the XSEDE Extended Collaborative Support Service (ECSS) program.

Please include this citation:

Wilkins-Diehr, N and S Sanielevici, J Alameda, J Cazes, L Crosby, M Pierce, R Roskies. High Performance Computer Applications 6th International Conference, ISUM 2015, Mexico City, Mexico, March 9-13, 2015, Revised Selected Papers Gitler, Isidoro, Klapp, Jaime (Eds.) Springer International Publishing. ISBN 978-3-319-32243-8, 3-13, 2016. 10.1007/978-3-319-32243-8.

Research on Bridges-2 not supported by XSEDE

For research on Bridges-2 supported by programs other than XSEDE, we ask that you use the following text:

This work used the Bridges-2 system, which is supported by NSF award number OAC-1928147 at the Pittsburgh Supercomputing Center (PSC).

PSC Support

Please also acknowledge any support provided by PSC staff.

If PSC staff contributed substantially to software development, optimization, or other aspects of the research, they should be considered as coauthors.

When PSC staff contributions do not warrant coauthorship, please acknowledge their support with the following text:

We thank [consultant name(s)] for [his/her/their] assistance with [describe tasks such as porting code, optimization, visualization, etc.]