Filemover
Filemover is a tool developed by PSC to copy data between Bridges’ pylon5 filesystem and Bridges-2’s Ocean filesystem. It allows you to select which directories from Bridges you wish to copy to Bridges-2, creates a batch file containing commands which do the copying, and submits the job to the SLURM scheduler. Note that filemover copies the entire contents of the directories you select. You cannot select the individual files from a directory to be copied to Ocean.
Filemover will not delete files from pylon5; it only creates a new copy on Ocean.
The filemover command is available on Bridges-2. Filemover cannot be run from Bridges.
Running Filemover
When logged in to Bridges-2, to run filemover, type
filemover
When filemover is executed, it finds your username and all of the groups that you belong to. From that, filemover finds the list of directories you own.
For each group you are in, filemover will ask you which subdirectories you want to be copied to Bridges-2’s Ocean filesystem.
To have all subdirectories copied, hit return.
To copy just some of the subdirectories, enter a list of those directories, separated by commas.
To have no directories from this group copied, enter the word “skip”.
Filemover will verify the directories exist, create a valid SLURM batch job, write this to your home directory, and then submit the job to SLURM (unless run with the -b option). Output from the job will can be found in filemover_jobId.log, where jobId is the job id of the SLURM job that copied your directories to Ocean.
Options for the filemover command
The filemover command has one option, -b. If you type
filemover -b
filemover will create the SLURM batch job file but will not submit it to SLURM to run. To copy your files to Ocean, you will have to submit the batch file yourself.
Example
The root file path for a user’s directory on pylon5 is /pylon5/groupname/username.
For each group you belong to, filemover will identify the root file path (as illustrated above) and ask you to enter a comma delimited list of subdirectories you would like to be copied to Bridges-2.
For example, the user Count Chocula’s username is cchocula. They belong to three user groups: cereal, milky, and breakfast, and cereal is their default user group..
Their default source directory would be /pylon5/cereal/cchocula. They may also have files in /pylon5/milky/cchocula and /pylon5/breakfast/cchocula.
When they run the filemover command they would see the following. Count Chocula’s responses are shown in bold.
[cchocula@br011 ~]$ filemover Welcome to the PSC filemover. This application will copy your files to the Bridges 2 file storage system as a scheduled slurm job. We can provide no estimate on when the job will run or how long it will take to copy your files. Enter a comma delimited list of the directories you would like to transfer from the following base directory. If you would like to transfer everything enter a blank line. To skip this path enter 'skip' Base directory: /pylon5/cereal/cchocula Enter a comma delimited list of the directories you would like to transfer from the following base directory. If you would like to transfer everything enter a blank line. To skip this path enter 'skip' Base directory: /pylon5/milky/cchocula data1, data4, data10 Enter a comma delimited list of the directories you would like to transfer from the following base directory. If you would like to transfer everything enter a blank line. To skip this path enter 'skip' Base directory: /pylon5/breakfast/cchocula skip
In this example, filemover would copy everything found in the /pyon5/cereal/cchocula directory, only the directories data1, data4, and data10 under /pylon5/milky/cchocula, and nothing from /pylon5/breakfast/cchocula.