Filemover

Filemover is a tool developed by PSC to copy data between Bridges’ pylon5 filesystem and Bridges-2’s Ocean filesystem. It allows you to select which directories from Bridges you wish to copy to Bridges-2, creates a batch file containing commands which do the copying, and submits the job to  the SLURM scheduler. Note that filemover copies the entire contents of the directories you select. You cannot select the individual files from a directory to be copied to Ocean.

Filemover finds the Bridges groups that you belong to and determines the corresponding Bridges-2 group. It then creates a SLURM job to copy the directories you choose from Bridges to the corresponding  directory on Bridges-2.  This means that if you have a Bridges allocation that does not continue to Bridges-2, you cannot use filemover to copy those files.  Filemover will be unable to find the matching Bridges-2 group and the job will fail. See the Transferring Files section of the Bridges-2 User Guide for other file transfer methods that you can use in this case.

Filemover will not delete files from pylon5; it only creates a new copy on Ocean.

The filemover command is available on Bridges-2. Filemover cannot be run from Bridges.

Running Filemover

When logged in to Bridges-2, to run filemover, type

filemover

When filemover is executed,  it finds your username and all of the groups that you belong to. From that, filemover finds the list of directories you own.

For each group you are in, filemover will ask you which subdirectories you want to be copied to Bridges-2’s  Ocean filesystem.

To have all subdirectories copied,  hit return.

To copy just some of the subdirectories, enter a list of those directories, separated by commas.

To have no directories from this group copied, enter the word  “skip”.

Filemover will verify the directories exist, create a valid SLURM batch job, write this to your home directory, and then submit the job to SLURM (unless run with the -b option). Output from the job will can be found in filemover_jobId.log, where jobId is the  job id of the SLURM job that copied your directories to Ocean.

Options for the filemover command

The filemover command has one option, -b.  If you type

filemover -b

filemover will create the SLURM batch job file but will not submit it to SLURM to run.  To copy your files to Ocean, you will have to submit the batch file yourself.

Example

The root file path for a user’s directory on pylon5 is /pylon5/groupname/username.

For each group you belong to, filemover will identify the root file path (as illustrated above) and ask you to enter a comma delimited list of subdirectories you would like to be copied to Bridges-2.

For example, the user Count Chocula’s username is cchocula. They belong to three user groups: cereal, milky, and breakfast, and cereal is their default user group..

Their default source directory would be /pylon5/cereal/cchocula.  They may also have files in /pylon5/milky/cchocula and  /pylon5/breakfast/cchocula.

When they run the filemover command they would see the following. Count Chocula’s responses are shown in bold.

[cchocula@br011 ~]$ filemover
Welcome to the PSC filemover. This application will copy your files
to the Bridges 2 file storage system as a scheduled slurm job. We can
provide no estimate on when the job will run or how long it will take
to copy your files.
Enter a comma delimited list of the directories you would like to transfer
from the following base directory. If you would like to transfer everything
enter a blank line. To skip this path enter 'skip'

Base directory: /pylon5/cereal/cchocula

Enter a comma delimited list of the directories you would like to transfer
from the following base directory. If you would like to transfer everything
enter a blank line. To skip this path enter 'skip'

Base directory: /pylon5/milky/cchocula

data1, data4, data10

Enter a comma delimited list of the directories you would like to transfer
from the following base directory. If you would like to transfer everything
enter a blank line. To skip this path enter 'skip'

Base directory: /pylon5/breakfast/cchocula

skip

In this example, filemover would copy everything found in the /pyon5/cereal/cchocula directory, only the directories data1, data4, and data10  under /pylon5/milky/cchocula, and nothing from /pylon5/breakfast/cchocula.

Error reporting from filemover

Fatal errors

If filemover cannot create a job to copy your files,  you will receive an email from Filemover with the subject line “FATAL Error from Filemover process”.  The most likely cause is a mismatch between your Bridges grants and your Bridges-2 grants. If this happens, email help@psc.edu for assistance.

Non-fatal errors

If filemover is able to create and submit a job. you will receive an email from filemover when your job is created, when it begins to run, and when it finishes. 

If the filemover job is successful, the last email will contain  a link to the log file from the job.  Success does not mean that all files were transferred, however. You should check the log file to see if there were any errors in the filemover process so that you can try again to copy any files which failed.

The top of the log file will list errors in these categories:

• Vanished: these files existed when the filemover job began to run, but were deleted from Bridges before they could be copied to Bridges-2.
• Missing: these files existed when filemover created the list of files to be copied, but were deleted from Bridges before the job could run.
• Denied: you do not have permission to access these files. A common cause of this error are temporary files created by the sed editor, with names that begin with “sed”.  
• Quota: you do not have enough space in  your Ocean directory for these files. Delete some files from Ocean or use the XSEDE XRAS system to request more space on Ocean for your grant.