Bridges to Bridges-2 Migration Guide
You will find working on Bridges-2 to be similar to working on Bridges. This document provides guidance on moving to Bridges-2 and some differences you should be familiar with. Be sure to check the Bridges-2 User Guide for full information.
Transferring files to Bridges-2
You can use any of the file transfer methods outlined in the Transferring Files section of the Bridges-2 User Guide to move files into Bridges-2, but consider these recommendations:
To copy files from the pylon5 filesystem on Bridges to the Ocean filesystem on Bridges-2, PSC has created a tool, filemover, to help. Filemover will copy entire directories from pylon5 to the corresponding Ocean directory in Bridges-2. For more information, see the filemover documentation.
To copy files from your home directory on Bridges, a simple rsync command should work well. A sample command to transfer your entire Bridges home directory tree into a subdirectory named “from-bridges” of your Bridges-2 home directory is given here. Run this command from Bridges-2.
rsync -zvrh username@data.bridges.psc.edu:/home/username $HOME/from-bridges
Node differences
Bridges-2 has the same types of compute nodes as you found on Bridges, but the nodes are more powerful. You should be aware of how many cores or GPUs you are requesting, as that will affect the SUs a job consumes. Some differences are:
- Bridges-2 RM nodes have 128 cores each, Bridges RM nodes had 28 cores each
- Bridges-2 RM nodes have at least 256GB of RAM, and some have 512GB; Bridges had 128GB
- Bridges-2 EM nodes with 4TB RAM replace Bridges LM nodes with either 3TB or 12TB
- Bridges-2 GPU nodes are all Volta architecture; Bridges contained some K80 and P100 architecture GPUs
- Bridges-2 GPU nodes all have 8 GPUs, some Bridges GPU nodes had only 2 GPUs
- Bridges-2 GPU nodes have 512GB RAM, Bridges GPU nodes had 128GB or 192 GB
- Note that soon after Bridges is decommissioned, its GPU-AI resources will be migrated to Bridges-2. Watch for an announcement.
Charging algorithms
On the RM nodes, SUs are still calculated by core-hours as they were on Bridges. Be aware, however, that since the Bridges-2 RM nodes have 128 cores, using an entire node for one hour will generate 128 SUs.
On the EM nodes, SUs are now calculated by core-hours, instead of by TB-hours as they were on Bridges.
On the GPU nodes, SUs are still calculated by GPU-hours. Be aware that all of Bridges-2 GPU nodes contain 8 GPUs, while some of Bridges only had 2.
Shared file space for grant members
All grants on Bridges-2 are given a shared directory, open to all who are on the grant. This directory is /ocean/projects/groupname/shared, where groupname is the project id for your grant.
Module environment
The module environment has been updated. In particular, the command module spider topic
will list all modules pertaining to topic along with some descriptive help information for each.
Module names do not always reflect if they were built for CPU or GPU use. Be sure to check the help text by typing module help module-name
when you need to know this.
Module names now include the compiler type and its version when relevant. For example, modules are available for mvapich built with either the GNU or the intel compilers.
[user@bridges2-login013 ~]$ module avail mvapich ------------------------------- /opt/modulefiles ------------------------------- mvapich2/2.3.5-gcc8.3.1 mvapich2/2.3.5-intel20.4 (D) Where: D: Default Module
$PROJECT space
You have persistent storage space in the Ocean file system at /ocean/projects/groupname/username, accessible by the environment variable $PROJECT. $SCRATCH is not defined on Bridges-2.
Porting your job scripts
There are several things you should be aware of when porting your job scripts to Bridges-2.
Filepaths
Be sure to change any explicit file paths you in your job scripts from the Bridges pylon5 filesystem to use Bridges-2 Ocean filesystem.
Replace any references to $SCRATCH, as $SCRATCH is not defined on Bridges-2.
GPU use
The flags used to request GPUs have changed.
In an interactive session, use –gres=gpu:n, where n is the number of GPUs to use per node.
In a batch job, use –gpus=n, where n is the number of GPUs to use per node.
Modules
Check for the correct names of modules, as they will be different from the Bridges modules. See the section on Module Environment in this document for more information.
Account administration
Charge ids
On Bridges-2, most charge ids are identical to the grant number. This may not be the case for some special types of grants. You can confirm your charge id for any grant with the projects
command.
projects command
A new option to the projects
command, -s, will show a condensed version of the output of the projects command, and may be all the information that you need.
my_quotas command
A new command, my_quotas
, shows the Ocean quota for all of your grants.
Getting help
Questions about Bridges-2 usage should be sent to help@psc.edu.