Filemover

Filemover is a tool developed by PSC to copy data between Bridges’ pylon5 filesystem and Bridges-2’s Ocean filesystem. It allows you to select which directories from Bridges you wish to copy to Bridges-2, creates a batch file containing commands which do the copying, and submits the job to  the SLURM scheduler. Note that filemover copies the entire contents of the directories you select. You cannot select the individual files from a directory to be copied to Ocean.

Filemover will not delete files from pylon5; it only creates a new copy on Ocean.

The filemover command is available on Bridges-2. Filemover cannot be run from Bridges.

Running Filemover

When logged in to Bridges-2, to run filemover, type

filemover

When filemover is executed,  it finds your username and all of the groups that you belong to. From that, filemover finds the list of directories you own.

For each group you are in, filemover will ask you which subdirectories you want to be copied to Bridges-2’s  Ocean filesystem.

To have all subdirectories copied,  hit return.

To copy just some of the subdirectories, enter a list of those directories, separated by commas.

To have no directories from this group copied, enter the word  “skip”.

Filemover will verify the directories exist, create a valid SLURM batch job, write this to your home directory, and then submit the job to SLURM (unless run with the -b option). Output from the job will can be found in filemover_jobId.log, where jobId is the  job id of the SLURM job that copied your directories to Ocean.

Options for the filemover command

The filemover command has one option, -b.  If you type

filemover -b

filemover will create the SLURM batch job file but will not submit it to SLURM to run.  To copy your files to Ocean, you will have to submit the batch file yourself.

Example

The root file path for a user’s directory on pylon5 is /pylon5/groupname/username.

For each group you belong to, filemover will identify the root file path (as illustrated above) and ask you to enter a comma delimited list of subdirectories you would like to be copied to Bridges-2.

For example, the user Count Chocula’s username is cchocula. They belong to three user groups: cereal, milky, and breakfast, and cereal is their default user group..

Their default source directory would be /pylon5/cereal/cchocula.  They may also have files in /pylon5/milky/cchocula and  /pylon5/breakfast/cchocula.

When they run the filemover command they would see the following. Count Chocula’s responses are shown in bold.

[cchocula@br011 ~]$ filemover
Welcome to the PSC filemover. This application will copy your files
to the Bridges 2 file storage system as a scheduled slurm job. We can
provide no estimate on when the job will run or how long it will take
to copy your files.
Enter a comma delimited list of the directories you would like to transfer
from the following base directory. If you would like to transfer everything
enter a blank line. To skip this path enter 'skip'

Base directory: /pylon5/cereal/cchocula

Enter a comma delimited list of the directories you would like to transfer
from the following base directory. If you would like to transfer everything
enter a blank line. To skip this path enter 'skip'

Base directory: /pylon5/milky/cchocula

data1, data4, data10

Enter a comma delimited list of the directories you would like to transfer
from the following base directory. If you would like to transfer everything
enter a blank line. To skip this path enter 'skip'

Base directory: /pylon5/breakfast/cchocula

skip

In this example, filemover would copy everything found in the /pyon5/cereal/cchocula directory, only the directories data1, data4, and data10  under /pylon5/milky/cchocula, and nothing from /pylon5/breakfast/cchocula.