Transferring Files

Pylon2 to be decommissioned June 19

All of your pylon2 files should be moved to other spaces and deleted from pylon2. Instructions to move your files to pylon5 are below.

If you have questions or run into any issues in moving your files from pylon2 to pylon5, please let us know by emailing bridges@psc.edu.

Instructions for transferring files from pylon2 to pylon5

Pylon2 has been unmounted from the Bridges login nodes. This means that you cannot see or access any pylon2 directory from a login node.  You will get a "No such file or directory" error if you try.

You can use the rsync command or the Globus web application to transfer your files from pylon2 to pylon5.  We suggest you use rsync.

Remember to delete your pylon2 files once your transfers have finished.

rsync

The rsync command can be run on Bridges' compute nodes in an interactive session or a batch job, or by using ssh on one of Bridges' high-speed data transfer nodes.  An advantage of rsync is that if the transfer does not complete, you can rerun the rsync command, and rsync will copy only those files which have not already been transferred.

PSC has created a shell script that you can use to move files from pylon2 to pylon5.  The shell script can be used in an interactive session or in a batch job.

Note that rsync will overwrite files in the destination directory if a file of the same name with a more recent modified time exists in the source directory.    To prevent this, the examples below copy pylon2 files into a new subdirectory on pylon5.  Once transferred, please examine the files and move them to the directory where you want them.

In all the examples given here, change groupname, username and new-directory to be your charging group, userid and name of the new subdirectory (if you like) to store the files.

Shell script 

The PSC-provided shell script is /opt/packages/utilities/pylon2to5. It will:

  • Copy all files from your pylon2 home directory (/pylon2/groupname/username) to a subdirectory named "from_pylon2" under your pylon5 home directory (/pylon5/groupname/username/from_pylon2)
  • Loop until it succeeds for all files or is killed (e.g. due to timeout).  This could use a lot of SUs if failures persist.
  • Skip copying older files from pylon2 on top of newer files in pylon5 with the same name. This is default rsync behavior.

Interactive session

Start an interactive session, type

interact

To use the PSC-supplied shell script, when your interactive session begins type

/opt/packages/utilities/pylon2to5

If you prefer not to use the PSC-supplied shell script, when your session begins you can use a command like

rsync -av /pylon2/groupname/username/    /pylon5/groupname/username/new-directory

Batch job

To run the PSC-supplied shell script in a batch job, create a batch script with the following content:

#!/bin/bash
#SBATCH -c 4
#SBATCH -p RM-shared
#SBATCH -
## newgrp my_other_grant
date 
## The next line runs the shell script
/opt/packages/utilities/pylon2to5

To move files from a grant that is not your default, uncomment the newgrp command in the script by removing "##" and substitute the correct group name for "my_other_grant".

If you prefer not to use the PSC-supplied shell script, you can create a batch script  which runs rsync like the one shown here.  Change groupname, username and new-directory to be your charging group, userid and name of the new subdirectory (if you like) to store the files.

#!/bin/bash
#SBATCH -p RM-shared
rsync -av /pylon2/groupname/username/    /pylon5/groupname/username/new-directory

Submit your batch script by typing (where script-name is the name of your script file)

sbatch script-name

Data transfer node

Use Bridges' high-speed data transfer nodes to move your files from pylon2 to pylon5.  At the Bridges' prompt, type

ssh data.bridges.psc.edu "rsync -av /pylon2/groupname/username/    /pylon5/groupname/username/new-directory"

Globus

To use the Globus web application, visit www.globus.org.

  • Choose “PSC Bridges with XSEDE Authentication” as each endpoint (you will need to authenticate with your XSEDE login and password).
  • For the first path, choose the pylon2 directory that you wish to transfer files from. For example: /pylon2/chargeid/userid
  • For the second path, choose the appropriate pylon5 target directory: /pylon5/chargeid/userid

To find your chargeid on Bridges, use the projects command to see all of the allocations that you have access to.

  • At the bottom of the Globus transfer page, choose the Transfer Settings that you wish to use (e.g. “preserve source file modification times”) and transfer your files as you would through any other web application.

Content of this document

 

Paths for Bridges file spaces

For all file transfer methods other than cp, you must always use the full path for your Bridges files.  The start of the full paths for your Bridges directories are:

Home directory     /home/username

Pylon2 directory   /pylon2/groupname/username

Pylon5 directory   /pylon5/groupname/username

The command id -Gn will show all of your valid groupnames.  You  have a pylon2 and pylon5 directory for each grant you have.

 

Transfers into your Bridges home directory 

Your home directory quota is 10GB, so large files cannot be stored there; they should be copied into one of your pylon file spaces instead. Exceeding your home directory quota will prevent you from writing more data into your home directory and will adversely impact other operations you might want to perform.  

 

rsync

 

You can use the rsync command to copy files to and from Bridges. A sample rsync command to copy to a Bridges directory is

rsync -rltpDvp -e 'ssh -l joeuser' source_directory data.bridges.psc.edu:target_directory

 

Substitute your userid for 'joeuser'. Make sure you use the correct group name in your target directory. By default, rsync will not copy older files with the same name in place of newer files in the target directory. It will overwrite older files.

 

We recommend the rsync options -rltDvp. See the rsync man page for information on these options and other you options you might want to use. We also recommend the option

-oMACS=umac-64@openssh.com

If you use this option your transfer will use a faster data validation algorithm.

 

You may to want to put your rsync command in a loop to insure that it completes. A sample loop is

RC=1
n=0
while [[ $RC -ne 0 && $n -lt 20 ]] do
    rsync ...
    RC = $?
    let n = n + 1
    sleep 10
done

 

This loop will try your rsync command 20 times. If it succeeds it will exit. If an rsync invocation is unsuccessful the system will try again and pick up where it left off. It will copy only those files that have not already been transferred. You can put this loop, with your rsync command, into a batch script and run it with sbatch.

 

Globus

Globus can be used for any file transfer to Bridges. It tracks the progress of the transfer and retries when there is a failure; this makes it especially useful for transfers involving large files or many files. This includes transfers between the pylon2 and pylon5 filesystems and file transfers between the Data Supercell and pylon2 or pylon5.  

To use Globus to transfer files you must authenticate either via a Globus account or with InCommon credentials

Globus account

You can set up a Globus account at the Globus site.

InCommon credentials

If you wish to use InCommon credentials to transfer files to/from Bridges, you must first provide your CI Login Certificate Subject information to PSC.  Follow these steps:

  1. Find your Certificate Subject string
    1. Navigate your web browser to https://cilogon.org/.
    2. Select your institution from the 'Select an Identity Provider' list.
    3. Click the 'Log On' button.  You will be taken to the web login page for your institution.
    4. Login with your username and password for your institution.
      • If your institution has an additional login requirement (e.g., Duo), authenticate to that as well.

      After successfully authenticating to your institution's web login interface, you will be returned to the CILogon webpage.  Note the boxed section near the top that lists a field named 'Certificate Subject'.

      Certificate Subject
  2. Send your Certificate Subject string to PSC
    1. In the CILogon webpage, select and copy the Certificate Subject text. Take care to get the entire text string if it is broken up onto multiple lines.
    2. Send email to support@psc.edu.  Paste your Certificate Subject field into the message, asking that it be mapped to your PSC username.

Your CI Login Certificate Subject information will be added within one business day, and you will be able to begin transferring files to and from Bridges.

 

Globus endpoints

Once you have the proper authentication you can initiate file transfers from the Globus site.  A Globus transfer requires a Globus endpoint, a file path and a file name for both the source and destination.  The endpoints for Bridges are:

  • psc#bridges-xsede if you are using an XSEDE User Portal account for authentication
  • psc#bridges-cilogon if you are using InCommon for authentication

These endpoints are owned by psc@globusid.org. If you use DUO MFA for your XSEDE authentication, you do not need to because you cannot use it with Globus. You must always specify a full path for the Bridges file systems.  See Paths for Bridges file spaces for details.

 

Globus-url-copy

The globus-url-copy command can be used if you have access to Globus client software.  Both the globus-url-copy and myproxy-logon commands are available on Bridges, and can be used for file transfers internal to the PSC.

To use globus-url-copy you must have a current user proxy certificate.  The command grid-proxy-info will tell you if you have current user proxy certificate and if so, what the remaining life of your certificate is.

Use the myproxy-logon command to get a valid user proxy certificate if any one of these applies:

  • you get an error from the grid-proxy-info command
  • you do not have a current user proxy certificate
  • the remaining life of your certificate is not sufficient for your planned file transfer

When prompted for your MyProxy passphrase enter your XSEDE Portal password.

To use globus-url-copy for transfers to a machine you must know the Grid FTP server address.  The Grid FTP server address for Bridges is

gsiftp://gridftp.bridges.psc.edu

The use of globus-url-copy always requires full paths. See Paths for Bridges file spaces for details.

 

scp

To use scp for a file transfer you must specify a source and destination for your transfer.  The format for either source or destination is

username@machine-name:path/filename

For transfers involving Bridges,  username is your PSC username.  The machine-name should be given as data.bridges.psc.edu. This is the name for a high-speed data connector at PSC. We recommend using it for all file transfers using scp involving Bridges.  Using it prevents file transfers from disrupting interactive use on Bridges' login nodes.

File transfers using scp must specify full paths for Bridges file systems. See Paths for Bridges file spaces for details.

sftp

To use sftp, first connect to the remote machine:

    sftp username@machine-name

When  Bridges is the remote machine, use your PSC userid as  username. The Bridges machine-name should be specified as data.bridges.psc.edu. This is the name for a high-speed data connector at PSC.  We recommend using it for all file transfers using sftp involving Bridges.  Using it prevents file transfers from disrupting interactive use on Bridges' login nodes.

You will be prompted for your password on the remote machine. If Bridges is the remote machine enter your PSC password.

You can then enter sftp subcommands, like put to copy a file from the local system to the remote system, or get to copy a file from the remote system to the local system.  

To copy files into Bridges you must either cd to the proper directory or use full pathnames in your file transfer commands. See Paths for Bridges file spaces for details.

 

Two-factor Authentication

If you are required to use two-factor authentication (TFA) to access Bridges' filesystems, you must enroll in XSEDE DUO.  Once that is complete, use scp or sftp to transfer files to/from Bridges.

TFA users must use port 2222 and XSEDE Portal usernames and passwords.  The machine name for these transfers is data.bridges.psc.edu.

In the examples below, myfile is the local filename, XSEDE-username is your XSEDE Portal username and /path/to/file is the full path to the file on a Bridges filesystem. Note that -P ( capital P) is necessary.

scp

Transfer a file from a local machine to Bridges:

scp -P 2222 myfile XSEDE-username@data.bridges.psc.edu:/path/to/file

Transfer a file from Bridges to a local machine:

scp -P 2222 XSEDE-username@data.bridges.psc.edu:/path/to/file myfile

sftp

Use sftp interactively:

sftp -P 2222 XSEDE-username@data.bridges.psc.edu

Then use the put command to copy a file from the local machine to Bridges, or the get command to transfer a file from Bridges to the local machine.

 

Graphical SSH client

If you are using a graphic SSH client, configure it to connect to data.bridges.psc.edu on port 2222/TCP. Login using your XSEDE Portal username and password.

 

Transfer rates

PSC maintains a Web page at http://speedpage.psc.edu that lists average data transfer rates between all XSEDE resources.  If your data transfer rates are lower than these average rates or you believe that your file transfer performance is subpar, send email to bridges@psc.edu.  We will examine approaches for improving your file transfer performance.

System Status

  • Bridges is up

     

     Scheduled downtime June 26-28

    Read more

Pylon2 Discontinued June 19

The pylon2 filesystem will be discontinued on June 19. If you have not already done so, move your files to pylon5.

See how to move your files.

New on Bridges

Filesystems upgrades mean changes in usage for pylon5 and pylon2.
Read more

Omni-Path User Group

The Intel Omni-Path Architecture User Group is open to all interested users of Intel's Omni-Path technology.

More information on OPUG