High performance SSH/SCP



HPN-SSH is a research project based at the Pittsburgh Supercomputing Center


(PI) Chris Rapier PSC, Michael Stevens CMU, Benjamin Bennett PSC, Mike Tasota PSC/CMU, Mitch Dorrell PSC
email: hpn-ssh@psc.edu

Configuration Options

These options can be used on the command line in the typical '-o Option ' format or embedded in the ssh_config or sshd_config files.

LibreSSL Support

Changes in LibreSSL version 3.5 and 3.6 prevent the use of the threaded AES CTR cipher. In those cases HPNSSH will fallback to the serial versionof the AES CTR cipher. A warning is printed to stderr. This shoudl be resolved in LibreSSL 3.7.

Automatic Port Fallback

Starting with version 17v3 the hpnssh client now uses TCP port 2222 to connect automatically as this is the default hpnsshd port. However, we understand that many users will be end up connecting standard SSH servers on port 22. To make the easier for users the client will fall back to port 22 in the event that there is no hpnssh server running on port 2222. The behaviour can be modifed as follows:

Enable or disable port fallback. Default is yes.
N is the port number that should be used for fall back. Default is 22.
Example Usage:
Stop hpnssh from using sshd on port 22 if hpnsshd isn't found on port 2222

hpnssh -oFallback=no

Have hpnssh fall back to port 2022 if hpnsshd isn't on port 2222

hpnssh -oFallbackPort=2022

Metrics options

This features allows the client to request tcp networking information from the TCP_INFO struct. This includes data on retransmits, round trip time, lost packets, data transferred, and the like. The metrics are polled periodically through the life of the connection. By default this is every 5 seconds but users can pick different polling periods. The resulting data is stored in two distinct files; one for local metrics and one for remote metrics. Remote metrics are only available if the remote supports this feature. This feature will not diagnose a poorly performing connection but may provide insight into what is happening during the connection.

This option will enable metrics polling. Default: No.
N is the polling period in seconds. Default: 5 seconds.
This is the path to the files where the remote and local data will be stored. Default: `./ssh\_stack\_metrics.[local|remote]`. Any other option chosen by the user will have a .local or .remote suffix appended to it.
Example usage:

ssh -oMetrics=yes -oMetricsInterval=1 -oMetricsPath=/tmp/scp-test-results

SCP resume options

This feature allows SCP to resume failed transfers. In the event of a failed transfer issues the same scp command with the '-Z' option. For example - if you issued:

scp myhugefile me@host:~

and it dies halfway through the transfer issuing

scp -Z myhugefile me@host:~ will resume the transfer at the point where it left off.

SCP however, will use the first scp in the user's path. This might not support the resume function and the attempt will fail. In those cases the user can explicitly define the path to the resume enabled scp with the '-z' option. This would be the path on the remote host. For example:

scp -Z -z /opt/hpnssh/usr/bin/scp myhugefile me@host:~

None cipher options

To use the NONE option you must have the NoneEnabled switch set on the server and you must have both NoneEnabled and NoneSwitch set to yes on the client. The NONE feature works with ALL ssh subsystems (as far as we can tell) as long as a tty is not spawned. If a user uses the -T switch to prevent a tty being created the NONE cipher will be disabled.

Default is no. Set to "yes" to enable the switch to no encryption after authentication. This option must be set to "yes" on the server as well as the client.
Default is no. Set to "yes" to disable the message authentication cipher after authentication. This must be set to "yes" on both the client and server.
Default is no. If the NoneEnabled option is set to yes then the connection will disable encryption after authentication. If NoneMacEnabled and NoneEnabled is set to yes then both the cipher and MAC will be disabled after authentication.
Example usage:

scp -oNoneEnabled=yes -oNoneCipherSwitch=yes hugefile me@remote:~

will, after authentication disable encryption while transferring hugefile from the local host to the remote host. The remote host must have NoneEnabled=yes set the in the sshd_config file.

scp -oNoneEnabled=yes -oNoneMacEnabled=yes -oNoneCipherSwitch=yes hugefile me@remote:~

will, after authentication disable both encryption and the MAC while transferring hugefile from the local host to the remote host. All 3 options must be set to yes on the client to disabled the MAC. The remote host must have both NoneEnabled=yes and NoneMacEnabled=yes set the in the sshd_config file.

HPN specific options


TcpRcvBuf=[int]KB client
Set the TCP socket receive buffer to int Kilobytes. It can be set up to the maximum socket size allowed by the system. This is useful in situations where the TCP receive window is set low but the maximum buffer size is set higher (as is typical). This works on a per TCP connection basis. You can also use this to artificially limit the transfer rate of the connection. In these cases the throughput will be no more than n/RTT. The minimum buffer size is 1KB . Default is the current system wide TCP receive buffer size.
TcpRcvBufPoll=[yes/no] client/server
Enable or disable the polling of the TCP receive buffer through the life of the connection. You would want to make sure that this option is enabled for systems making use of autotuning kernels (linux 2.4.24, 2.6, MS Vista). Default is no.
NoneEnabled=[yes/no] client/server
Enable or disable the use of the None cipher. Care must always be used when enabling this as it will allow users to send data in the clear. However, it is important to note that authentication information remains encrypted even if this option is enabled. Default is no.
NoneSwitch=[yes/no] client
Switch the encryption cipher being used to the None cipher after authentication takes place. NoneEnabled must be enabled on both the client and server side of the connection. When the connection switches to the NONE cipher a warning is sent to STDERR. The connection attempt will fail with an error if a client requests a NoneSwitch from the server that does not explicitly have NoneEnabled set to yes. Note: The NONE cipher cannot be used in interactive (shell) sessions and it will fail silently.  Default is no.
HPNDisabled=[yes/no] client/server
In some situations, such as transfers on a local area network, the impact of the HPN code produces a net decrease in performance. In these cases it is helpful to disable the HPN functionality. Default is no.
HPNBufferSize=[int]KB client/server
This is the default buffer size the HPN functionality uses when interacting with nonHPN SSH installations. Conceptually this is similar to the TcpRcvBuf option as applied to the internal SSH flow control. This value can range from 1KB to 14MB (1-14336). Use of oversized or undersized buffers can cause performance problems depending on the length of the network path. The default size of this buffer is 2MB.
DisableMTAES=[yes/no] client/server
Switch the encryption cipher being used from the multithreaded MT-AES-CTR cipher back to the stock single-threaded AES-CTR cipher. Useful on modern processors with AES-NI instructions which make the stock single-threaded AES-CTR cipher faster than the multithreaded MT-AES-CTR cipher. Set to no by default.


This patch was conceived, designed, and led by Chris Rapier (rapier@psc.edu) The majority of the actual coding for versions up to HPN12v1 was performed by Michael Stevens (mstevens@andrew.cmu.edu). The MT-AES-CTR cipher was implemented by Ben Bennet (ben@psc.edu) and improved by Mike Tasota (tasota@gmail.com) an NSF REU grant recipient for 2013. Mitchell Dorrell (mwd@psc.edu) has provided invaluable assistance on improving the performance of ChaCha20 and Poly1305 along with other critical work. Allan Jude provided the code for the NoneMac and buffer normalization. This work was financed, in part, by Cisco System, Inc., the National Library of Medicine, and the National Science Foundation.


Thanks to Niklas Hambuchen for being the first sponsor of HPN-SSH via github's sponsor program! Join our sponsors at our github, view our funding page, or contact at hpnssh@psc.edu.

Notes and News

HPN-SSH 18.3.0 Released

We are pleased to announce that HPN-SSH 18.3.0 has been released and is now available on https://github.com/rapier1/hpn-ssh. This release incorporates OpenSSH 9.6p1. Fedora and Debian packages are also available.

HPN-SSH 18.2.0 Released

We are pleased to announce that HPN-SSH 18.2.0 has been released and is now available on https://github.com/rapier1/hpn-ssh. This release incorporates OpenSSH 9.5p1. Fedora and Debian packages are also available.

HPN-SSH 18.1.0 Released

We are pleased to announce that HPN-SSH 18.1.0 has been released and is now available on https://github.com/rapier1/hpn-ssh. This version introduces a parallel ChaCha20-Poly1305 cipher that is 59% faster than OpenSSH 9.4. Packages for Ubuntu and Fedora will be...

Binary Packages Released for 17v11

Binary packages for Debian and Fedora are now available. Commands to add these package sources are:For Debian: sudo add-apt-repository ppa:rapier1/hpnsshFor Fedora: sudo dnf copr enable rapier1/hpnssh

HPN-SSH 17v11 Released

We are pleased to announce that HPN-SSH 17v11 has been released and is now available on https://github.com/rapier1/openssh-portable. This version brings HPN-SSH up to parity with OpenSSH 9.2 and resolves a number of minor bugs. Patches and binary packages will be...

HPN17v0 Released

We are pleased to announce HPN17v0. This version brings us up to date with OpenSSH 8.9. HPN17 also brings a big change to how we are naming things. From now all executables will have an "hpn" prefix attached to them. So "ssh" is now "hpnssh" and "scp" becomes...

Patch sets on SourceForge

All patch sets from 4.4p1 to 8.1p1 are now available on SourceForge at https://sourceforge.net/projects/hpnssh/. The entire codebase (merged with OpenSSH) is also available as a git repo from https://github.com/rapier1/openssh-portable. The SourceForge location now...


We are proud to announce that the HPN-SSH development team has received a grant from the National Science Foundation (Award#: 2004012) to continue development on HPN-SSH. This grant will be used to develop and incorporate new features and optimizations. This grant...

This work was made possible in part by grants from Cisco Systems, Inc., the National Science Foundation, and the National Library of Medicine.