Bridges User Guide

 

File Spaces

There are several distinct file spaces available on Bridges, each serving a different function. Three months after your grant expires all of your Bridges files associated with that grant will be deleted. You will be able to login during this 3-month period to transfer files, but you will not be able to run jobs or create files.

 

Home ($HOME)

This is your home directory. It is the usual location for your batch scripts, source code and parameter files. Its path is /home/username, where  username is your PSC userid. You can refer to your home directory with the environment variable $HOME. Your home directory is visible to all of Bridges's nodes.

Your home directory is backed up daily, although it is still a good idea to store copies of your important  files in another location, such as the /pylon2 file system or on a local file system at your site.

$HOME quota

Your home directory has a 10GB quota. You can check your home directory usage using the quota command or the command du -sh.

Grant expiration

Three months after your grant expires, the files in your home directory associated with that grant will be deleted.

 

/pylon1

The /pylon1 file system is a scratch file system, to be used as working space for your running jobs. It provides fast, temporary file space for data read or written by running jobs. It provides much faster IO than your home directory. The /pylon1 file system is shared across all of Bridges nodes.

The /pylon1 file system is not currently available on Bridges' 12-TB nodes.

pylon1 directories

The path of your /pylon1 home directory is /pylon1/groupname/username, where groupname is the name for the PSC group associated with your grant.  Use the id command to find your group name.

The command id -Gn will list all the groups you belong to.
The command id -gn will list the group associated with your current session.

If you have more than one grant, you will have a /pylon1 directory for each grant. Be sure to use the appropriate directory when working with multiple grants.

/pylon1 wiper

The /pylon1 file system is not a persistent file system. Files are wiped after 30 days.  It is also not backed up.  Be sure to move copies of your important /pylon1 files to another location as soon as you can after you create them.

If you have a compelling reason that your files should not be wiped after 30 days, email This email address is being protected from spambots. You need JavaScript enabled to view it. and request an exemption.   Be sure to explain the need for the exemption and supply a deadline after which the files can be safely wiped.

/pylon1 quota

Your usage quota for each of your grants is the Pylon storage allocation you received when your proposal was approved.  Files stored under a grant in both /pylon1 and /pylon2 count towards this storage allocation.  If your total use in /pylon1 and /pylon2 exceeds this quota your write access to /pylon1 and /pylon2 will be shut off until you are under quota.

Use the du -sh  or projects command to check your total pylon usage (/pylon1 plus /pylon2). You can also check your usage on the XSEDE User Portal.

If you have multiple grants, it is very important that you store your files in the correct /pylon1 directory.

Grant expiration

Three months afer your grant expires, the files in your /pylon1 directories associated with that grant will be deleted, although the wiper will probably have deleted them already.

 

/pylon2

The /pylon2 file system is a persistent file system. You should use it for long-term storage of your files and not for working space for running jobs.  The /pylon2 file system is shared across all of Bridges' nodes.

/pylon2 directories

The path of your /pylon2 home directory is /pylon2/groupname/username, where groupname is the name for the PSC group associated with your grant. The id command can be used to find your group name.

The command id -Gn will list all the groups you belong to.
The command id -gn will list the group associated with your current session.

If you have more than one grant, you will have a /pylon2 home directory for each grant. Be sure to use the appropriate directory when working with multiple grants.

The /pylon2 file system is not backed up. You should therefore store copies of your important /pylon2 files in another location.

/pylon2 quota

Your usage quota for each of your grants is the Pylon storage allocation you received when your proposal was approved.  Files stored under a grant in both /pylon1 and /pylon2 count towards this storage allocation.  If your total use in /pylon1 and /pylon2 exceeds this quota your write access to /pylon1 and /pylon2 will be shut off until you are under quota.

Use the du -sh  or projects command to check your total pylon usage (/pylon1 plus /pylon2). You can also check your usage on the XSEDE User Portal.

If you have multiple grants, it is very important that you store your files in the correct /pylon2 directory.

Grant expiration

 Three months after your grant expires, the files in your /pylon2 directories associated with that grant will be deleted.

 

Node-local ($LOCAL)

Each of Bridges's nodes has a local file system attached to it. This local file system is only visible to the node to which it is attached.  The local file system provides fast access to local storage.

This file space is available on all nodes as $LOCAL.

$LOCAL is only available when your job is running, and can only be used as working space for a running job. Once your job finishes your local files are inaccessible and deleted. To use local space, copy files to $LOCAL at the beginning of your job and back out to a persistent file space before your job ends.

If a node crashes all the $LOCAL files are lost. Therefore, you should checkpoint your $LOCAL files by copying them to pylon1 during long runs.

Multi-node jobs

If you are running a multi-node job the variable $LOCAL points to the local file space on the node that is running your rank 0 process.

You can use the srun command to copy files between $LOCAL on the nodes in a multi-node job.  See the MPI job script in the Running Jobs section of this User  Guide for details.

$LOCAL size

The maximum amount of local space varies by node type. The RSM (128GB) nodes have a maximum of 3.7TB.  The LSM (3TB) nodes have a maximum of 15TB and the  ESM (12TB) nodes have a maximum of 64TB.

To check on your local file space usage type:

du -sh

 

 There is  no charge for the use of $LOCAL.

 

Memory files ($RAMDISK)

You can also use the memory allocated for your job for IO rather than using disk space. This will offer the fastest IO on Bridges.

In a running job the environment variable $RAMDISK will refer to the memory associated with the nodes in use.

The amount of memory space available to you depends on the size of the memory on the nodes and the number of nodes you are using. You can only perform IO to the memory of nodes assigned to your job.

If you do not use all of the cores on a node, you are allocated memory in proportion to the number of cores you are using.  Note that you cannot use 100% of a node's memory for IO; some is needed for program and data usage.

$RAMDISK is only available to you while your job is running, and can only be used as working space for a running job. Once your job ends this space is inaccessible. To use memory files, copy files to $RAMDISK at the beginning of your job and back out to a permanent space before your job ends.  If your job terminates abnormally your memory files are lost.

 Within your job you can cd to $RAMDISK, copy files to and from it, and use it to open files.  Use the command du -sh to see how much space you are using.

If you are running a multi-node job the $RAMDISK variable points to the memory space on your node that is running your rank 0 process.

 

 

 

Omni-Path User Group

The Intel Omni-Path Architecture User Group is open to all interested users of Intel's Omni-Path technology.

More information on OPUG