Bridges-2 User Guide
This User Guide is being updated constantly as Bridges-2 is brought into production. Please check back often for the latest information. If you have questions at anytime, please email firstname.lastname@example.org
We take security very seriously. Please take a minute now to read PSC policies on passwords, security guidelines, resource use, and privacy. You are expected to comply with these policies at all times when using PSC systems. If you have questions at any time, you can send email to email@example.com.
Are you new to HPC?
If you are new to high performance computing, please read Getting Started with HPC before you begin your research on Bridges-2. It explains HPC concepts which may be unfamiliar. You can also check the Introduction to Unix or the Glossary for quick definitions of terms that may be new to you.
We hope that that information along with the Bridges-2 User Guide will have you diving into your work on Bridges-2. But if you have any questions, don’t hesitate to email us for help at firstname.lastname@example.org.
PSC User Services is here to help you get your research started and keep it on track. If you have questions at any time, you can send email to email@example.com.
Set your PSC password
Before you can connect to Bridges-2, you must have a PSC password.
If you have an active account on any other PSC system: PSC usernames and passwords are the same across all PSC systems. You will use the same username and password on Bridges-2 as for your other PSC account.
PSC password policies
Computer security depends heavily on maintaining secrecy of passwords.
PSC uses Kerberos authentication on all its production systems, and your PSC password (also known as your Kerberos password) is the same on all PSC machines.
Set your initial PSC password
When you receive a PSC account, go to the web-based PSC password change utility to set your password. For security, you should use a unique password for your PSC account, not one that you use for other sites.
Change your PSC password
Changing your password changes it on all PSC systems. To change your Kerberos password, use the web-based PSC password change utility .
Please note that changing your password on the XSEDE Portal does not change it on PSC systems and will not prevent your PSC password from expiring.
PSC password requirements
Your password must:
- be at least eight characters long
- contain characters from at least three of the following groups:
- lower-case letters
- upper-case letters
- special characters, excluding apostrophes (') and quotes (")
- be different from the last three PSC passwords you have used
- be changed at least once per year
Under NO circumstances does PSC reveal any passwords over the telephone, FAX them to any location, send them through email, set them to a requested string, or perform any other action that could reveal a password.
If someone claiming to represent PSC contacts you and requests information that in any manner would reveal a password, be assured that the request is invalid and do NOT comply.
Bridges-2 is designed for converged HPC + AI + Data. Its custom topology is optimized for data-centric HPC, AI, and HPDA (High Performance Data Analytics). An extremely flexible software environment along with community data collections and BDaaS (Big Data as a Service) provide the tools necessary for modern pioneering research. The data management system, Ocean, consists of two-tiers, disk and tape, transparently managed as a single, highly usable namespace.
Bridges-2 will hold three types of nodes: GPU, “Regular Memory” and “Extreme Memory”.
Bridges-2’s GPU nodes provide exceptional performance and scalability for deep learning and accelerated computing, with a total of 40, 960 CUDA cores and 5,120 tensor cores.
|GPUs per node||8 NVIDIA Tesla V100-32GB SXM2|
|GPU performance||1 Pf/s tensor|
|CPUs||2 Intel Xeon Gold 6248 “Cascade Lake” CPUs
20 cores, 40 threads/CPU
2.50 – 3.90 GHz
|Cache||27.5MB LLC, 6 memory channels|
|Node-local storage||7.68TB NVMe SSD|
|Network||2 Mellanox ConnectX-6 HDR Infiniband 200 Gbs/s Adapters|
Regular Memory nodes
Regular Memory (RM) nodes provide extremely powerful general-purpose computing, machine learning and data analytics, AI inferencing, and pre- and post-processing.
|CPU||2 AMD EPYC 7742 CPUs
64 cores, 128 threads each
|2 AMD EPYC 7742 CPUs
64 cores, 128 threads each
|Cache||256MB L3, 8 memory channels||256MB L3, 8 memory channels|
|Node-local storage||3.84TB NVMe SSD||3.84TB NVMe SSD|
|Network||Mellanox ConnectX-6-HDR Infiniband 200Gb/s Adapter||Mellanox ConnectX-6-HDR Infiniband 200Gb/s Adapter|
Extreme memory nodes
Extreme Memory (EM) nodes provide 4TB of shared memory for genome sequence assembly, graph analytics, statistics, and other applications requiring a large amount of memory for which distributed-memory implementations are not available.
|CPU||4 Intel Xeon Platinum 8260M “Cascade lake” CPUs
24 cores, 48 threads each
|Cache||37.75MB LLC, 6 memory channels|
|Node-local storage||7.68TB NVMe SSD|
|Network||Mellanox ConnectX-6-HDR Infiniband 200Gb/s Adapter|
Data management on Bridges-2 is accomplished through a unified, high-performance filesystem for active project data, archive, and resilience, named Ocean.
Ocean consists of two tiers, disk and tape, transparently managed as a single, highly usable namespace.
Ocean’s disk subsystem, for active project data, is a high-performance, internally resilient Lustre parallel filesystem with 15PB of usable capacity, configured to deliver up to 129GB/s and 142GB/s of read and write bandwidth, respectively.
Ocean’s tape subsystem, for archive and additional resilience, is a high-performance tape library with 7.2PB of uncompressed capacity, configured to deliver 50TB/hour. Data compression occurs in hardware, transparently, with no performance overhead.
Connecting to Bridges-2
Bridges-2 contains two broad categories of nodes: compute nodes, which handle the production research computing, and login nodes, which are used for managing files, submitting batch jobs and launching interactive sessions. Login nodes are not suited for production computing.
When you connect to Bridges-2, you are connecting to a Bridges-2 login node.
You can connect to Bridges-2 via a web browser or through a command line interface.
Connect in a web browser
You can access Bridges-2 through a web browser by using the OnDemand software. You will still need to understand Bridges-2’s partition structure and the options which specify job limits, like time and memory use, but OnDemand provides a more modern, graphical interface to Bridges-2.
Connect to a command line interface
You can connect to a traditional command line interface by logging in via one of these:
- ssh, using either XSEDE or PSC credentials. If you are registered with XSEDE for DUO Multi-Factor Authentication (MFA), you can use this security feature in connecting to Bridges-2. See the XSEDE instructions to set up DUO for MFA.
- XSEDE Single Sign On, including using Multi-Factor authentication if you are an XSEDE user
You can use an ssh client from your local machine to connect to Bridges-2 using either your PSC or XSEDE credentials.
SSH is a program that enables secure logins over an unsecure network. It encrypts the data passing both ways so that if it is intercepted it cannot be read.
SSH is client-server software, which means that both the user’s local computer and the remote computer must have it installed. SSH server software is installed on all the PSC machines. You must install SSH client software on your local machine.
Free ssh clients for Macs, Windows machines and many versions of Unix are available. Popular ssh clients (GUI) include PuTTY for Windows and Cyberduck for Macs. A command line version of ssh is installed on Macs by default; if you prefer that, you can use it in the Terminal application. You can also check with your university to see if there is an ssh client that they recommend.
Once you have an ssh client installed, you can use either your PSC credentials or XSEDE credentials (optionally with DUO MFA) to connect to Bridges-2. Note that you must have created your PSC password before you can use ssh to connect to Bridges-2.
Use ssh to connect to Bridges-2 using XSEDE credentials and (optionally) DUO MFA:
- Using your ssh client, connect to hostname bridges-2.psc.xsede.org or bridges-2.psc.edu using port 2222.
Either hostname will connect you to Bridges-2, but you must specify port 2222.
- Enter your XSEDE username and password when prompted.
- (Optional) If you are registered with XSEDE DUO, you will receive a prompt on your phone. Once you have answered it, you will be logged in.
Use ssh to connect to Bridges-2 using PSC credentials:
- Using your ssh client, connect to hostname bridges2.psc.xsede.org or bridges2.psc.edu using the default port (22).
Either hostname will connect you to Bridges-2. You do not have to specify the port.
- Enter your PSC username and password when prompted.
You can also use public-private key pairs to connect to Bridges-2. To do so, you must first fill out this form to register your keys with PSC.
XSEDE single sign on
XSEDE users can use their XSEDE usernames and passwords in the XSEDE User Portal Single Sign On Login Hub (SSO Hub) to access bridges2.psc.xsede.org or bridges2.psc.edu.
You must use DUO Multi-Factor Authentication in the SSO Hub.
Changing your PSC password
There are two ways to change or reset your PSC password:
- Use the web-based PSC password change utility
- Use the kpasswd command when logged into a PSC system. Do not use the passwd command.
When you change your PSC password, whether you do it via the online utility or via the kpasswd command on one PSC system, you change it on all PSC systems.
Remember that your PSC password is separate from your XSEDE Portal password. Resetting one password does not change the other password.
Managing your XSEDE allocation
Most account management functions for your XSEDE grant are handled through the XSEDE User Portal. You can search the XSEDE user portal to get help. Some common questions:
- How do I add a user to my grant?
- How can I transfer SUs from Bridges-2 Extreme Memory to Bridges-2 Regular Memory (or vice versa)?
- How can I request more SUs or storage space on Bridges-2?
Changing your default shell
The change_shell command allows you to change your default shell. This command is only available on the login nodes.
To see which shells are available, type
To change your default shell, type
where newshell is one of the choices output by the
change_shell -l command. You must use the entire path output by
change_shell -l, e.g. /usr/psc/shells/bash. You must log out and back in again for the new shell to take effect.
PSC account policies
The policies documented here are evaluated regularly to assure adequate and responsible administration of PSC systems for users. As such, they are subject to change at any time.
PSC provides storage resources, for long-term storage and file management.
Files in a PSC storage system are retained for 3 months after the affiliated grant has expired.
Requesting a Refund
When appropriate, PSC provides refunds for jobs that failed due to circumstances beyond your control.
To request a refund, contact a PSC consultant or email firstname.lastname@example.org. In the case of batch jobs, we require the standard error and output files produced by the job. These contain information needed in order to refund the job.
The following times are reserved for system maintenance.
Note that the machines will not always be down at these times. However, PSC reserves the right to use these scheduled times for maintenance as necessary.
|Computing resources||Wed. 9 am – 3 pm|
|Storage resources||Wed. 9 am – 3 pm|
|Gigapop||Tue. & Fri. 7 – 8 am|
|PSC production network||Tue. & Fri. 7 – 8 am|
There are several distinct file spaces available on Bridges-2, each serving a different function.
- Home ($HOME), your home directory on Bridges-2
- Ocean, a system for persistent file storage.
- Node-local storage, scratch storage on the local SSD on the node running a job
- Memory storage, scratch storage in the local memory associated with a running job
Three months after your grant expires all of your Bridges-2 files associated with that grant will be deleted, no matter which file space they are in. You will be able to login during this 3-month period to transfer files, but you will not be able to run jobs or create new files.
Access to files in any Bridges-2 space is governed by Unix file permissions. If your data has additional security or compliance requirements, please contact email@example.com.
Unix file permissions
For detailed information on Unix file protections, see the man page for the chmod (change mode) command.
To share files with your group, give the group read and execute access for each directory from your top-level directory down to the directory that contains the files you want to share.
chmod g+rx directory-name
Then give the group read and execute access to each file you want to share.
chmod g+rx filename
To give the group the ability to edit or change a file, add write access to the group:
chmod g+rwx filename
Access Control Lists
If you want more fine-grained control than Unix file permissions allow —for example, if you want to give only certain members of a group access to a file, but not all members—then you need to use Access Control Lists (ACLs). Suppose, for example, that you want to give janeuser read access to a file in a directory, but no one else in the group.
Use the setfacl (set file acl) command to give janeuser read and execute access on the directory:
setfacl -m user:janeuser:rx directory-name
for each directory from your top-level directory down to the directory that contains the file you want to share with janeuser. Then give janeuser access to a specific file with
setfacl -m user:janeuser:r filename
User janeuser will now be able to read this file, but no one else in the group will have access to it.
To see what ACLs are set on a file, use the
getfacl (get file acl) command.
There are man pages for chmod, setfacl and getfacl.
This is your Bridges-2 home directory. It is the usual location for your batch scripts, source code and parameter files. Its path is /home/username, where username is your PSC username. You can refer to your home directory with the environment variable $HOME. Your home directory is visible to all of Bridges-2’s nodes.
Your home directory is backed up daily, although it is still a good idea to store copies of your important files in another location, such as the Ocean file system or on a local file system at your site. If you need to recover a home directory file from backup send email to firstname.lastname@example.org. The process of recovery will take 3 to 4 days.
Your home directory has a 10GB quota. You can check your home directory usage using the
quota command or the command
du -sh. To improve the access speed to your home directory files you should stay as far below your home directory quota as you can.
Three months after a grant expires, the files in your home directory associated with that grant will be deleted.
The path of your Ocean home directory is /ocean/groupname/username, where groupname is the Unix group id associated with your grant. Use the
id command to find your group name.
id -Gn will list all the Unix groups you belong to.
id -gn will list the Unix group associated with your current session.
If you have more than one grant, you will have an Ocean directory for each grant. Be sure to use the appropriate directory when working with multiple grants.
Your usage quota for each of your grants is the Ocean storage allocation you received when your proposal was approved. If your total use in Ocean exceeds this quota you won’t be able to run jobs on Bridges-2 until you are under quota again.
du -sh or
projects command to check your Ocean usage. You can also check your usage on the XSEDE User Portal.
If you have multiple grants, it is very important that you store your files in the correct Ocean directory.
Three months after a grant expires, the files in any Ocean directories associated with that grant will be deleted.
Each of Bridges-2’s nodes has a local file system attached to it. This local file system is only visible to the node to which it is attached. The local file system provides fast access to local storage.
If your application performs a lot of small reads and writes, then you could benefit from using this space.
Node-local storage is only available when your job is running, and can only be used as working space for a running job. Once your job finishes your local files are inaccessible and deleted. To use local space, copy files to it at the beginning of your job and back out to a persistent file space before your job ends.
If a node crashes all the node-local files are lost. You should checkpoint theses files by copying them to Ocean during long runs.
You can also use the memory allocated for your job for IO rather than using disk space.
The amount of memory space available to you depends on the size of the memory on the nodes and the number of nodes you are using. You can only perform IO to the memory of nodes assigned to your job.
If you do not use all of the cores on a node, you are allocated memory in proportion to the number of cores you are using. Note that you cannot use 100% of a node’s memory for IO; some is needed for program and data usage.
This space is only available to you while your job is running, and can only be used as working space for a running job. Once your job ends this space is inaccessible. To use memory files, copy files to it at the beginning of your job and back out to a permanent space before your job ends. If your job terminates abnormally your memory files are lost.
Several methods are available to transfer files into and from Bridges-2.
Paths for Bridges-2 file spaces
To copy files into any of your Bridges-2 spaces, you need to know the path to that space on Bridges-2. The start of the full paths for your Bridges-2 directories are:
Home directory /home/username
Ocean directory /ocean/groupname/username
To find your groupname, use the command
id -Gn. All of your valid groupnames will be listed. You have an Ocean directory for each grant you have.
Commands to transfer files
You can use rsync, scp or sftp to copy files to and from Bridges-2.
You can use the rsync command to copy files to and from Bridges-2. A sample rsync command to copy to a Bridges-2 directory is
rsync -rltpDvp -e 'ssh -l username' source_directory data.bridges2.psc.edu:target_directory
Substitute your username for ‘username‘. Make sure you use the correct groupname in your target directory. By default, rsync will not copy older files with the same name in place of newer files in the target directory. It will overwrite older files.
We recommend the rsync options -rltDvp. See the rsync man page for information on these options and other options you might want to use. We also recommend the option
If you use this option, your transfer will use a faster data validation algorithm.
You may want to put your rsync command in a loop to insure that it completes. A sample loop is
RC=1 n=0 while [[ $RC -ne 0 && $n -lt 20 ]] do rsync source-file target-file RC = $? let n = n + 1 sleep 10 done
This loop will try your rsync command 20 times. If it succeeds it will exit. If an rsync invocation is unsuccessful the system will try again and pick up where it left off. It will copy only those files that have not already been transferred. You can put this loop, with your rsync command, into a batch script and run it with
To use scp for a file transfer you must specify a source and destination for your transfer. The format for either source or destination is
For transfers involving Bridges-2, username is your PSC username. The machine-name should be given as data.bridges2.psc.edu. This is the name for a high-speed data connector at PSC. We recommend using it for all file transfers using scp involving Bridges-2. Using it prevents file transfers from disrupting interactive use on Bridges-2’s login nodes.
File transfers using
scp must specify full paths for Bridges-2 file systems. See Paths for Bridges-2 file spaces for details.
To use sftp, first connect to the remote machine:
When Bridges-2 is the remote machine, use your PSC userid as username. The Bridges-2 machine-name should be specified as data.bridges2.psc.edu. This is the name for a high-speed data connector at PSC. We recommend using it for all file transfers using sftp involving Bridges-2. Using it prevents file transfers from disrupting interactive use on Bridges-2’s login nodes.
You will be prompted for your password on the remote machine. If Bridges-2 is the remote machine, enter your PSC password.
You can then enter
sftp subcommands, like
put to copy a file from the local system to the remote system, or
get to copy a file from the remote system to the local system.
To copy files into Bridges-2, you must either
cd to the proper directory or use full pathnames in your file transfer commands. See Paths for Bridges-2 file spaces for details.
Transferring files using Two-Factor Authentication
If you are required to use Two-Factor Authentication (TFA) to access Bridges-2’s filesystems, you must enroll in XSEDE DUO. Once that is complete, use scp or
sftp to transfer files to and from Bridges-2.
TFA users must use port 2222 and XSEDE Portal usernames and passwords, not PSC usernames and passwords. The machine name for these transfers is data.bridges2.psc.edu.
In the examples below, myfile is the local filename, XSEDE-username is your XSEDE Portal username and /path/to/file is the full path to the file on a Bridges-2 filesystem. Note that -P (capital P) is necessary.
Transfer a file from a local machine to Bridges-2:
scp -P 2222 myfile XSEDEemail@example.com:/path/to/file
Transfer a file from Bridges-2 to a local machine:
scp -P 2222 XSEDEfirstname.lastname@example.org:/path/to/file myfile
Use sftp interactively:
sftp -P 2222 XSEDEemail@example.com
Then use the
put command to copy a file from the local machine to Bridges-2, or the
get command to transfer a file from Bridges-2 to the local machine.
Graphical SSH client
If you are using a graphical SSH client, configure it to connect to data.bridges2.psc.edu on port 2222/TCP. Login using your XSEDE Portal username and password.
Globus can be used for any file transfer to Bridges-2. It tracks the progress of the transfer and retries when there is a failure; this makes it especially useful for transfers involving large files or many files.
To use Globus to transfer files you must authenticate either via a Globus account or with InCommon credentials.
To use a Globus account for file transfer, set up a Globus account at the Globus site.
To use InCommon credentials to transfer files to/from Bridges-2, you must first provide your CILogin Certificate Subject information to PSC. Follow these steps:
- Find your Certificate Subject string
- Navigate your web browser to https://cilogon.org/.
- Select your institution from the ‘Select an Identity Provider’ list.
- Click the ‘Log On’ button. You will be taken to the web login page for your institution.
- Login with your username and password for your institution.
- If your institution has an additional login requirement (e.g., Duo), authenticate to that as well.
- After successfully authenticating to your institution’s web login interface, you will be returned to the CILogon webpage.
- Click on the Certificate Information drop down link to find the ‘Certificate Subject’.
- Send your Certificate Subject string to PSC
- In the CILogon webpage, select and copy the Certificate Subject text. Take care to get the entire text string if it is broken up onto multiple lines.
- Send email to firstname.lastname@example.org. Paste your Certificate Subject field into the message, asking that it be mapped to your PSC username.
Your CILogin Certificate Subject information will be added within one business day, and you will be able to begin transferring files to and from Bridges-2.
Once you have the proper authentication you can initiate file transfers from the Globus site. A Globus transfer requires a Globus endpoint, a file path and a file name for both the source and destination. The endpoints for Bridges-2 are:
- psc#bridges2-xsede if you are using an XSEDE User Portal account for authentication
- psc#bridges2-cilogon if you are using InCommon for authentication
These endpoints are owned by email@example.com. If you use DUO MFA for your XSEDE authentication, you do not need to because you cannot use it with Globus. You must always specify a full path for the Bridges-2 file systems. See Paths for Bridges-2 file spaces for details.
globus-url-copy command can be used if you have access to Globus client software. Both the
myproxy-logon commands are available on Bridges-2, and can be used for file transfers internal to the PSC.
globus-url-copy you must have a current user proxy certificate. The command
grid-proxy-info will tell you if you have a current user proxy certificate and what the remaining life of your certificate is.
myproxy-logon command to get a valid user proxy certificate if any one of these applies:
- you get an error from the
- you do not have a current user proxy certificate
- the remaining life of your certificate is not sufficient for your planned file transfer
When prompted for your MyProxy passphrase enter your XSEDE Portal password.
globus-url-copy for transfers to a machine you must know the Grid FTP server address. The Grid FTP server address for Bridges-2 is
The use of
globus-url-copy always requires full paths. See Paths for Bridges-2 file spaces for details.
PSC maintains a Web page at http://speedpage.psc.edu that lists average data transfer rates between XSEDE resources. If your data transfer rates are lower than these average rates or you believe that your file transfer performance is subpar, send email to firstname.lastname@example.org. We will examine approaches for improving your file transfer performance.
Bridges-2 provides a rich programming environment for the development of applications.
C, C++ and Fortran
AMD (AOCC), Intel, Gnu and PGI compilers for C, C++ and Fortan are available on Bridges-2. The compiler commands are:
There are man pages for each of the compilers.
- AMD Optimizing C/C++ Compiler (AOCC)
- PGI web site
- GNU compilers web site
- Module documentation for information on what modules are available and how to use them.
To compile OpenMP programs you must add an option to your compile command:
for example: icc -qopenmp yprog.c
for example: gcc -fopenmp myprog.c
for example: pgcc -mp myprog.c
Three types of MPI are supported on Bridges-2: MVAPICH2, OpenMPI and Intel MPI.
To compile an MPI program, you must do these things:
- Load the module for the MPI type you want to use.
- Load the module for the compiler that you want to use.
- Issue the appropriate MPI wrapper command to compile your program.
The three MPI types may perform differently on different problems or in different programming environments. If you are having trouble with one type of MPI, please try using another type. Contact email@example.com for more help.
Note: When using Intel MPI, you must set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT to 0. Otherwise the mpirun task placement settings you give will be ignored. The command to do this is:
For the BASH shell:
For the CSH shell:
setenv I_MPI_JOB_RESPECT_PROCESS_PLACEMENT 0
- Intel MPI web site
- MVAPICH2 web site
- OpenMPI web site
- Module documentation for information on what modules are available and how to use them.
Debugging and performance analysis
DDT is a debugging tool for C, C++ and Fortran 90 threaded and parallel codes. It is client-server software. Install the client on your local machine and then you can access the GUI on Bridges-2 to debug your code.
See the DDT page for more information.
Bridges-2 has a broad collection of applications installed. See the list of software installed on Bridges-2.
Additional software may be installed by request. Request that additional applications be installed by using the Software Request form.
The OnDemand interface allows you to conduct your research on Bridges-2 through a web browser. You can manage files – create, edit and move them – submit and track jobs, see job output, check the status of the queues, run a Jupyter notebook through JupyterHub and more, without logging in to Bridges-2 via traditional interfaces.
OnDemand was created by the Ohio Supercomputer Center (OSC). In addition to this document, you can check the extensive documentation for OnDemand created by OSC, including many video tutorials, or email firstname.lastname@example.org.
More information on using OnDemand on Bridges-2 will be available soon.
Bridges-2’s GPU nodes provide substantial, complementary computational power for deep learning, simulations and other applications.
A standard NVIDIA accelerator environment is installed on Bridges-2’s GPU nodes. If you have programmed using GPUs before, you should find this familiar. Please contact email@example.com for more help.
The GPU nodes on Bridges-2 are available to those with a Bridges-2 “GPU” allocation. You can see which of Bridges-2’s resources that you have been allocated with the
The /home and Ocean file systems are available on all of these nodes. See the File Spaces section of this User Guide for more information on these file systems.
Compiling and Running jobs
Use the GPU partition, either in batch or interactively, to compile your code and run your jobs.
More information on using CUDA on Bridges-2 can be found in the CUDA document.
To use CUDA, first you must load the CUDA module. To see all versions of CUDA that are available, type:
module avail cuda
Then choose the version that you need and load the module for it.
module load cuda
loads the default CUDA. To load a different version, use the full module name.
module load cuda/8.0
Our primary GPU programming environment is OpenACC.
The PGI compilers are available on all GPU nodes. To set up the appropriate environment for the PGI compilers, use the module command:
module load pgi
If you will be using these compilers often, it will be useful to add this command to your shell initialization script.
There are many options available with these compilers. See the online man pages (“man pgf90”,”man pgcc”,”man pgCC”) for detailed information. You may find these basic OpenACC options a good place to start:
pgcc –acc yourcode.c pgf90 –acc yourcode.f90
Adding the “-Minfo=accel” flag to the compile command (whether pgf90, pgcc or pgCC) will provide useful feedback regarding compiler errors or success with your OpenACC commands.
pgf90 -acc -Minfo=accel yourcode.f90
Hybrid MPI/GPU Jobs
To run a hybrid MPI/GPU job use the following commands for compiling your program:
module load cuda module load mpi/pgi_openmpi mpicc -acc yourcode.c
When you execute your program you must first issue the above two module load commands.
Profiling and Debugging
For CUDA codes, use the command line profiler nvprof. See the CUDA document for more information.
For OpenACC codes, the environment variables PGI_ACC_TIME, PGI_ACC_NOTIFY and PGI_ACC_DEBUG can provide profiling and debugging information for your job. Specific commands depend on the shell you are using.
|Bash shell||C shell|
|Enable runtime GPU performance profiling||export PGI_ACC_TIME=1||setenv PGI_ACC_TIME 1|
For data transfer information, set PGI_ACC_NOTIFY to 3
|export PGI_ACC_NOTIFY=1||setenv PGI_ACC_NOTIFY 1|
|More detailed debugging||export PGI_ACC_DEBUG=1||setenv PGI_ACC_DEBUG 1|
Containers are stand-alone packages holding the software needed to create a very specific computing environment. If you need a very specialized environment, you can create your own container or use one that is already installed on Bridges-2. Singularity is the only type of container supported on Bridges-2.
Creating a container
Singularity is the only container software supported on Bridges-2. You can create a Singularity container, copy it to Bridges-2 and then execute your container on Bridges-2, where it can use Bridges-2’s compute nodes and filesystems. In your container you can use any software required by your application: a different version of CentOS, a different Unix operating system, any software in any specific version needed. You can install your Singularity container without any intervention from PSC staff.
See the PSC documentation on Singularity for more details on producing your own container and Singularity use on Bridges-2.
Publicly available containers on Bridges-2
We have installed many containers from the NVIDIA GPU Cloud (NGC) on Bridges-2. These containers are fully optimized, GPU-accelerated environments for AI, machine learning and HPC. They can only be used on the Bridges-2 GPU nodes.
These include containers for:
- Caffe and Caffe2
- Microsoft Cognitive Toolkit
- Inference Server
See the PSC documentation on Singularity for more details on Singularity use on Bridges-2.
A community dataset space allows Bridges-2’s users from different grants to share data in a common space. Bridges-2 hosts both community (public) and private datasets, providing rapid access for individuals, collaborations and communities with appropriate protections.
If there is a dataset which you need for your research on Bridges-2 and you believe it would be of widespread interest, you can request that it be installed for public use by using the Dataset Request form.
Some datasets are available to anyone with a Bridges-2 account. They include:
2019nCoVR: 2019 Novel Coronavirus Resource
The 2019 Novel Coronavirus Resource concerns the outbreak of novel coronavirus in Wuhan, China since December 2019. For more details about the statistics, metadata, publications, and visualizations of the data, please visit https://bigd.big.ac.cn/nco.
COCO (Common Objects in Context) is a large scale image dataset designed for object detection, segmentation, person keypoints detection, stuff segmentation, and caption generation. Please visit http://cocodataset.org/ for more information on COCO, including details about the data, paper, and tutorials.
The PREVENT-AD (Pre-symptomatic Evaluation of Experimental or Novel Treatments for Alzheimer Disease) cohort is composed of cognitively healthy participants over 55 years old, at risk of developing Alzheimer Disease (AD) as their parents and/or siblings were/are affected by the disease. These ‘at-risk’ participants have been followed for a naturalistic study of the presymptomatic phase of AD since 2011 using multimodal measurements of various disease indicators. Two clinical trials intended to test pharmaco-preventive agents have also been conducted. The PREVENT-AD research group is now releasing data openly with the intention to contribute to the community’s growing understanding of AD pathogenesis.
ImageNet is an image dataset organized according to WordNet hierarchy. See the ImageNet website for complete information.
Natural Languge Tool Kit Data
NLTK comes with many corpora, toy grammars, trained models, etc. A complete list of the available data is posted at: http://nltk.org/nltk_data/
Dataset of handwritten digits used to train image processing systems.
The BLAST databases can be accessed through the environment variable $BLASTDB after loading the BLAST module.
Other genomics datasets
Other available datasets are typically used with a particular genomics package. These include:
Bridges-2 hosts a number of gateways – web-based, domain-specific user interfaces to applications, functionality and resources that allow users to focus on their research rather than programming and submitting jobs. Gateways provide intuitive, easy-to-use interfaces to complex functionality and data-intensive workflows.
Gateways can manage large numbers of jobs and provide collaborative features, security constraints and provenance tracking, so that you can concentrate on your analyses instead of on the mechanics of accomplishing them.
Security Guidelines and Policies
See also policies for:
Security is very important to PSC. These policies are intended to ensure that our machines are not misused and that your data is secure.
What You Can Do:
You play a significant role in security! To keep your account and PSC resources secure, please:
- Be aware of and comply with PSC’s policies on security, use and privacy found in this document
- Choose strong passwords and don’t share them between accounts or with others. More information can be found in the PSC password policies.
- Utilize your local security team for advice and assistance
- Take the online XSEDE Cybersecurity Tutorial. Go to Online Training and click on “XSEDE Cybersecurity (CI-Tutor)
- Keep your computer properly patched and protected
- Report any security concerns to the PSC help desk ASAP by calling the PSC hotline at: 412-268-6350 or email firstname.lastname@example.org
What We Will Never Do:
- PSC will never send you unsolicited emails requesting confidential information.
- We will also never ask you for your password via an unsolicited email or phone call.
Remember that the PSC help desk is always a phone call away to confirm any correspondence at 412-267-6350.
If you have replied to an email appearing to be from PSC and supplied your password or other sensitive information, please contact the help desk immediately.
What You Can Expect:
- We will send you email when we need to communicate with you about service outages, new HPC resources, and the like.
- We will send you email when your password is about to expire and ask you to change it by using the web-based PSC password change utility.
Other Security Policies
- PSC password policies
- Users must connect to PSC machines using ssh in order to avoid remote logins with clear text passwords
- We vigilantly monitor our computer systems and network connections for security violations
- We are in close contact with the CERT Coordination Project with regard to possible Internet security violations
Reporting Security Incidents
To report a security incident you should contact our Hotline at 412-268-6350. To report non-emergency security incidents you can send email to email@example.com.
PSC Acceptable Use Policy
PSC’s resources are vital to the scientific community, and we have a responsibility to ensure that all resources are utilized in a responsible manner. PSC has legal and other obligations to protect its services, resources, and the intellectual property of users. Users share this responsibility by observing the rules of acceptable use that are outlined in this document. Your on-line assent to this Acceptable Use Policy is your acknowledgment that you have read and understand your responsibilities as a user of PSC Services and Resources, which refers to all computers owned or operated by PSC and all hardware, data, software, storage systems and communications networks associated with these computers. If you have questions, please contact PSC User Services.
By using PSC Services and Resources associated with your allocation, you agree to comply with the following conditions of use:
- You will protect any access credentials (e.g., private keys, tokens & passwords) that are issued for your sole use by PSC and not knowingly allow any other person to impersonate or share any of your identities.
- You will not use PSC Services and Resources for any unauthorized purpose, including but not limited to:
- Financial gain
- Tampering with or obstructing PSC operations
- Breaching, circumventing administrative, or security controls
- Inspecting, modifying, distributing, or copying privileged data or software without proper authorization, or attempting to do so
- Supplying, or attempting to supply, false or misleading information or identification in order to access PSC Services and Resources
- You will comply with all applicable laws and relevant regulations, such as export control law or HIPAA.
- You will immediately report any known or suspected security breach or misuse of PSC access credentials to firstname.lastname@example.org.
- Use of PSC Services and Resources is at your own risk. There are no guarantees that PSC Services and Resources will be available, that they will suit every purpose, or that data will never be lost or corrupted. Users are responsible for backing up critical data.
- Logged information, including information provided by you for registration purposes, will be used solely for administrative, operational, accounting, monitoring and security purposes.
- Violations of this Acceptable Use Policy and/or abuse of PSC Services and Resources may result in loss of access to PSC Services and Resources. Abuse will be referred to the PSC User Services manager and/or the appropriate local, state and federal authorities, at PSC’s discretion.
- PSC may terminate or restrict any user’s access to PSC Services and Resources, without prior notice, if such action is necessary to maintain computing availability and security for other users of the systems.
- Allocations are awarded solely for open research, intended for publication. You will only use PSC Computing Resources to perform work consistent with the stated allocation request goals and conditions of use as defined by your approved PSC project and this Acceptable Use Policy.
- PSC is entitled to regulate, suspend or terminate your access, and you will immediately comply with their instructions.
This policy may be modified as new features are added to the site. Any changes to the policy will be posted on this page.
- Any data automatically collected from our site visitors – domain name, browser types, etc. – are used only in aggregate to help us better meet site visitors’ needs.
- There is no identification of individuals from our aggregate data. Therefore, unless you choose otherwise, you are totally anonymous when visiting our site.
- We do not share data with anyone for commercial purposes.
- If you choose to submit personally identifiable information to us electronically via the PSC feedback page, email, etc., we will treat it with the same respect for privacy afforded to mailed submissions. Submission of such information is always optional.
Reporting a Problem
To report a problem on Bridges-2, please email email@example.com. Please report only one problem per email; it will help us to track and solve any issues more quickly and efficiently.
Be sure to include
- an informative subject line
- your username
If the question concerns a particular job, include these in addition:
- the JobID
- any error messages you received
- the date and time the job ran
- link to job scripts, output and data files
- the software being used, and versions when appropriate
- a screenshot of the error or the output file showing the error, if possible
Acknowledgement in Publications
All publications, copyrighted or not, resulting from an allocation of computing time on Bridges-2 should include an acknowledgement. Please acknowledge both the funding source that supported your access to PSC and the specific PSC resources that you used.
Please also acknowledge support provided by XSEDE’s ECSS program and/or PSC staff when appropriate.
Proper acknowledgment is critical for our ability to solicit continued funding to support these projects and next generation hardware.
Suggested text and citations follow.
XSEDE supported research on Bridges-2
We ask that you use the following text:
This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges-2 system, which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC).
Please include these citations:
Towns, J., Cockerill, T., Dahan, M., Foster, I., Gaither, K., Grimshaw, A., Hazlewood, V., Lathrop, S., Lifka, D., Peterson, G.D., Roskies, R., Scott, J.R. and Wilkens-Diehr, N. 2014. XSEDE: Accelerating Scientific Discovery. Computing in Science & Engineering. 16(5):62-74. http://doi.ieeecomputersociety.org/10.1109/MCSE.2014.80.
XSEDE ECSS Support
To acknowledge support provided through XSEDE’s Extended Collaborative Support Services (ECSS), please use this text:
We thank [consultant name(s)] for [his/her/their] assistance with [describe tasks such as porting code, optimization, visualization, etc.], which was made possible through the XSEDE Extended Collaborative Support Service (ECSS) program.
Please include this citation:
Wilkins-Diehr, N and S Sanielevici, J Alameda, J Cazes, L Crosby, M Pierce, R Roskies. High Performance Computer Applications 6th International Conference, ISUM 2015, Mexico City, Mexico, March 9-13, 2015, Revised Selected Papers Gitler, Isidoro, Klapp, Jaime (Eds.) Springer International Publishing. ISBN 978-3-319-32243-8, 3-13, 2016. 10.1007/978-3-319-32243-8.
Research on Bridges-2 not supported by XSEDE
For research on Bridges-2 supported by programs other than XSEDE, we ask that you use the following text:
This work used the Bridges-2 system, which is supported by NSF award number OAC-1928147 at the Pittsburgh Supercomputing Center (PSC).
Please also acknowledge any support provided by PSC staff.
If PSC staff contributed substantially to software development, optimization, or other aspects of the research, they should be considered as coauthors.
When PSC staff contributions do not warrant coauthorship, please acknowledge their support with the following text:
We thank [consultant name(s)] for [his/her/their] assistance with [describe tasks such as porting code, optimization, visualization, etc.]