DMOVER

Scheduled Data Transfer for Distributed Computational Workflows

DMOVER is used to schedule bulk data transfers between GridFTP nodes. DMOVER jobs contain the locations and paths of the files to be transferred. If a stripe count is specified, file transfers will occur in a striped mode.

Using DMOVER

1. Ssh to a login node where DMOVER is available. Currently, DMOVER is only available on pople. Also, since DMOVER is designed for bulk data transfers, you should only use DMOVER to transfer files to and from bessemer on pople. 

    ssh tg-login.pople.psc.teragrid.org

2. Load the dmover module to make the DMOVER client programs available in your PATH. 

    module load dmover

3. Create a transfer file which contains your list of file transfer source and destination pairs. An example transfer file is 

# Comment: Transfer Example File 1
file:///bessemer/janedoe/test/5GB   gsiftp://gridftp.mercury.ncsa.teragrid.org/gpfs_scratch1/janedoe/5GBtest
gsiftp://gridftp.mercury.ncsa.teragrid.org/gpfs_scratch1/janedoe/10GB   file:///bessemer/janedoe/test/10GBtest
file:///bessemer/janedoe/test/100MB   gsiscp://tg-login.ncsa.teragrid.org/gpfs_scratch1/janedoe/100MBtest
gsiscp://tg-login.ncsa.teragrid.org/gpfs_scratch1/janedoe/10MB   file:///bessemer/janedoe/test/10MBtest

The first URL on each line in your transfer file is the source URL and the second is the destination URL. URLs beginning with 'file://' specify full file paths on the login host. You can also use fully qualified GridFTP or GSI-OpenSSH URLs to specify your files. A list of GridFTP nodes which you can use as either source or destination nodes with DMOVER is available at 

http://www.teragrid.org/web/user-support/transfer_location#servers

4. Obtain a user proxy certificate for authentication to all file source and destination hosts by issuing the myproxy-login command 

    myproxy-login -s myproxy.teragrid.org

5. Issue the dsub command to submit your DMOVER job. 

    dsub -n 4 -s 2 example1

In this example the name of your transfer file is example1. The -n and -s options are used respectively to specify how many cores and how many stripes per core you want your DMOVER transfer to use. The -L option can be used to specify a directory where your DMOVER transfer logs will be written. If you do not use the -L option your transfer logs are written to a directory in your home directory named 'dmover' followed by a timestamp corresponding to when your DMOVER job finished. If you use the -L option these 'dmover' directories are written to the directory named in the -L option. These transfer logs contain the messages which indicate whether your transfers succeeded or failed.

User Configuration File

You can create an SSH-style configuration file in ~/.dmover/config to define shortcut aliases for long host URLs. An example user configuration file is 

# Example ~/.dmover/config user configuration file
Host mercury
	URI gsiftp
		Hostname gridftp.mercury.ncsa.teragrid.org
	URI gsiscp
		Hostname tg-login.ncsa.teragrid.org
	URIOrder gsiftp, gsiscp

If you use this configuration file, the transfer file shown above can be written as 

# Comment: Transfer Example File 2
file:///bessemer/janedoe/test/5GB   mercury:/gpfs_scratch1/janedoe/5GBtest
mercury:/gpfs_scratch1/janedoe/10GB   file:///bessemer/janedoe/test/10GBtest
file:///bessemer/janedoe/test/100MB   mercury:/gpfs_scratch1/janedoe/100MBtest
mercury:/gpfs_scratch1/janedoe/10MB   file:///bessemer/janedoe/test/10MBtest

Given the URIOrder specified for the mercury alias in the configuration file, each transfer in the example2 transfer file will first be attempted using GridFTP. If transfers attempted using GridFTP fail, DMOVER will retry using the gsiscp method instead.

To learn more about creating a DMOVER user configuration file, see the dmover_config man page.

More information

Man pages

Several man pages are available for DMOVER on pople. You must first load the dmover module to display these man pages using the man command. PDF files of the man pages are linked below:

  • dsub — DMOVER job submission
  • ddel — DMOVER job cancellation
  • dstat — DMOVER job submission status in DMOVER queue
  • dmover_config — the DMOVER user configuration file

If you have questions about DMOVER send email to remarks@psc.edu.

Papers and presentations

Related work

  • Chris Rapier & Benjamin Bennett, PSC, Michael Stevens, CMU, High Performance SSH/SCP - HPN-SSH Performance improvements provided by HPN-SSH are incorporated into GSI-OpenSSH.