Bridges User Guide

 

Transferring Files

There are a variety of file transfer methods available for Bridges:

  • Globus - recommended for all file transfers of over 10 Gbytes
  • globus-url-copy - recommended for file transfers of over 10 Gbytes, if you have access to Globus client software but cannot use Globus
  • scp or sftp - recommended for external transfers under 10 Gbytes or for external transfers if you do not have access to Globus or Globus client software
  • cp - recommended for use between Bridges' filesystems if the transfer is under 10 Gbytes

 

Transfers into your Bridges home directory 

Since your Bridges home directory quota is 10 Gbytes, you should use scp or sftp for all file transfers involving your home directory.    Large files cannot be stored there; they should be copied into one of your pylon file spaces instead.

Exceeding your home directory quota will prevent you from writing more data into your home directory and will adversely impact other operations you might want to perform.  

Paths for Bridges file spaces

No matter which file transfer method you use, other than cp, the default directory for file transfers into a Bridges file system is a non-writeable directory. You will not be able to store any files in the default directory. This means that for transfers involving Bridges file systems, in which you are not using cp, you must always use the full path for your Bridges files.  The full paths for your Bridges directories are:

Home directory     /home/username

Pylon1 directory   /pylon1/groupname/username

Pylon2 directory   /pylon2/groupname/username

The command id -Gn will tell you your groupnames.  You  have a pylon1 and pylon2 directory for each grant you have.

 

Transfer rates

PSC maintains a Web page at http://speedpage.psc.edu that lists average data transfer rates between all XSEDE resources.  If your data transfer rates are lower than these average rates or you believe that your file transfer performance is subpar, send email to This email address is being protected from spambots. You need JavaScript enabled to view it..  We will examine approaches for improving your file transfer performance.

 

Globus

To use Globus to transfer files you must obtain proper authentication either by setting up a Globus account or by using your InCommon credentials if you have them.  You can set up a Globus account at the Globus site.

Once you have the proper authentication you can initiate your file transfers using the Globus site.  A Globus transfer requires a Globus endpoint, a file path and a file name for both your source and destination.

If you are using your Globus account for authentication, the Bridges endpoint is psc#bridges-xsede.

If you are using InCommon for your authentication the Bridges endpoint is psc#bridges-cilogon.

You must always specify a full path for the Bridges file systems.

Globus is recommended for file transfers internal to the PSC that are over 10 Gbytes. This includes transfers between the Data Supercell and Bridges' pylon1 and pylon2 file systems and between pylon1 and pylon2.

 

Globus-url-copy

The globus-url-copy command can be used for file transfer if you cannot use Globus, but do have access to Globus client software. 

To use globus-url-copy you must have a current user proxy certificate.  The command grid-proxy-info will tell you if you have current user proxy certificate and if so, what the remaining life of your certificate is.

If

  • you get an error from the grid-proxy-info command,
  • you do not have a current user proxy certificate, or  
  • the remaining life of your certificate is not sufficient for your planned file transfer

use the myproxy-logon command to get a valid user proxy certificate.

When prompted for your MyProxy passphrase enter your XSEDE Portal password.

To use globus-url-copy for transfers to a machine you must know the Grid FTP server address.  The Grid FTP server address for Bridges is

gsiftp://gridftp.bridges.psc.edu

The use of globus-url-copy always requires full paths.

If you have file transfers internal to the PSC that are over 10 Gbytes, but you cannot use Globus, you can use globus-url-copy. The globus-url-copy and myproxy-logon commands are available on Bridges.

 

scp

To use scp for a file transfer you must specify a source and destination for your transfer.  The format for either source or destination is

username@machine-name:path/filename

For transfers involving Bridges,  username is your PSC username.  The machine-name should be specified as follows:

If your transfer isUse machine-name
10 Gbytes or less bridges.psc.edu
Over 10 Gbytes and between Bridges and another XSEDE platform data.bridges.psc.edu
Over 10 Gbytes and is not between Bridges and another XSEDE platform data.bridges.psc.edu

Data.bridges.psc.edu is the name for a high-speed data connector located at the PSC.   All large file transfers should use this data connector.

File transfers using scp must specify full paths for Bridges file systems.

sftp

To use sftp, first connect to  your destination machine:

    sftp username@machine-name

When  Bridges is your destination,  use your PSC userid as  username. The Bridges machine-name should be specified as follows:

If your transfer isUse machine-name
10 Gbytes or less bridges.psc.edu
Over 10 Gbytes and between Bridges and another XSEDE platform data.bridges.psc.edu
Over 10 Gbytes and is not between Bridges and another XSEDE platform data.bridges.psc.edu

Data.bridges.psc.edu is the name for a high-speed data connector located at the PSC.   All  large file transfers should use this data connector.    

You will be prompted for your password on the destination machine. If Bridges is your destination machine enter your PSC password.

You can then enter sftp subcommands, like put to copy a file from your local system to the destination system, or get to copy a file from the destination system to your local system.  

To copy files into Bridges you must either cd to the proper directory or use full pathnames in your file transfer commands.

 

New on Bridges

DDT version 7.0 installed
DDT documentation

Runtime limit on LM nodes now 2 weeks
See partition limits

Omni-Path User Group

The Intel Omni-Path Architecture User Group is open to all interested users of Intel's Omni-Path technology.

More information on OPUG