Native Parallel Client

API documentation for the ssh2-python (libssh2) based parallel client.

class pssh.clients.native.parallel.ParallelSSHClient(hosts, user=None, password=None, port=22, pkey=None, num_retries=3, timeout=None, pool_size=10, allow_agent=True, host_config=None, retry_delay=5, proxy_host=None, proxy_port=22, proxy_user=None, proxy_password=None, proxy_pkey=None, forward_ssh_agent=False, tunnel_timeout=None, keepalive_seconds=60)

ssh2-python based parallel client.

Parameters:
  • hosts (list(str)) – Hosts to connect to
  • user (str) – (Optional) User to login as. Defaults to logged in user
  • password (str) – (Optional) Password to use for login. Defaults to no password
  • port (int) – (Optional) Port number to use for SSH connection. Defaults to 22.
  • pkey (str) – Private key file path to use. Path must be either absolute path or relative to user home directory like ~/<path>.
  • num_retries (int) – (Optional) Number of connection and authentication attempts before the client gives up. Defaults to 3.
  • retry_delay (int) – Number of seconds to wait between retries. Defaults to pssh.constants.RETRY_DELAY
  • timeout (float) – (Optional) SSH session timeout setting in seconds. This controls timeout setting of socket operations used for SSH sessions. Defaults to OS default - usually 60 seconds.
  • pool_size (int) – (Optional) Greenlet pool size. Controls concurrency, on how many hosts to execute tasks in parallel. Defaults to 10. Overhead in event loop will determine how high this can be set to, see scaling guide lines in project’s readme.
  • host_config (dict) – (Optional) Per-host configuration for cases where not all hosts use the same configuration.
  • allow_agent (bool) – (Optional) set to False to disable connecting to the system’s SSH agent.
  • proxy_host (str) – (Optional) SSH host to tunnel connection through so that SSH clients connect to host via client -> proxy_host -> host
  • proxy_port (int) – (Optional) SSH port to use to login to proxy host if set. Defaults to 22.
  • proxy_user (str) – (Optional) User to login to proxy_host as. Defaults to logged in user.
  • proxy_password (str) – (Optional) Password to login to proxy_host with. Defaults to no password.
  • proxy_pkey (Private key file path to use.) – (Optional) Private key file to be used for authentication with proxy_host. Defaults to available keys from SSHAgent and user’s SSH identities.
  • forward_ssh_agent (bool) – (Optional) Turn on SSH agent forwarding - equivalent to ssh -A from the ssh command line utility. Defaults to False if not set. Requires agent forwarding implementation in libssh2 version used.
  • tunnel_timeout (float) – (Optional) Timeout setting for proxy tunnel connections.
Raises:

pssh.exceptions.PKeyFileError on errors finding provided private key.

copy_file(local_file, remote_file, recurse=False, copy_args=None)

Copy local file to remote file in parallel via SFTP.

This function returns a list of greenlets which can be join-ed on to wait for completion.

gevent.joinall() function may be used to join on all greenlets and will also raise exceptions from them if called with raise_error=True - default is False.

Alternatively call .get() on each greenlet to raise any exceptions from it.

Exceptions listed here are raised when either gevent.joinall(<greenlets>, raise_error=True) or .get() on each greenlet are called, not this function itself.

Parameters:
  • local_file (str) – Local filepath to copy to remote host
  • remote_file (str) – Remote filepath on remote host to copy file to
  • recurse (bool) – Whether or not to descend into directories recursively.
  • copy_args (tuple or list) – (Optional) format local_file and remote_file strings with per-host arguments in copy_args. copy_args length must equal length of host list - pssh.exceptions.HostArgumentException is raised otherwise
Return type:

list(gevent.Greenlet) of greenlets for remote copy commands

Raises:

ValueError when a directory is supplied to local_file and recurse is not set

Raises:

pssh.exceptions.HostArgumentException on number of per-host copy arguments not equal to number of hosts

Raises:

pss.exceptions.SFTPError on SFTP initialisation errors

Raises:

pssh.exceptions.SFTPIOError on I/O errors writing via SFTP

Raises:

OSError on local OS errors like permission denied

Note

Remote directories in remote_file that do not exist will be created as long as permissions allow.

copy_remote_file(remote_file, local_file, recurse=False, suffix_separator='_', copy_args=None, encoding='utf-8')

Copy remote file(s) in parallel via SFTP as <local_file><suffix_separator><host>

With a local_file value of myfile and default separator _ the resulting filename will be myfile_myhost for the file from host myhost.

This function, like ParallelSSHClient.copy_file(), returns a list of greenlets which can be join-ed on to wait for completion.

gevent.joinall() function may be used to join on all greenlets and will also raise exceptions if called with raise_error=True - default is False.

Alternatively call .get on each greenlet to raise any exceptions from it.

Exceptions listed here are raised when either gevent.joinall(<greenlets>, raise_error=True) is called or .get is called on each greenlet, not this function itself.

Parameters:
  • remote_file (str) – remote filepath to copy to local host
  • local_file (str) – local filepath on local host to copy file to
  • recurse (bool) – whether or not to recurse
  • suffix_separator (str) – (Optional) Separator string between filename and host, defaults to _. For example, for a local_file value of myfile and default separator the resulting filename will be myfile_myhost for the file from host myhost. suffix_separator has no meaning if copy_args is provided
  • copy_args (tuple or list) – (Optional) format remote_file and local_file strings with per-host arguments in copy_args. copy_args length must equal length of host list - pssh.exceptions.HostArgumentException is raised otherwise
  • encoding (str) – Encoding to use for file paths.
Return type:

list(gevent.Greenlet) of greenlets for remote copy commands

Raises:

ValueError when a directory is supplied to local_file and recurse is not set

Raises:

pssh.exceptions.HostArgumentException on number of per-host copy arguments not equal to number of hosts

Raises:

pss.exceptions.SFTPError on SFTP initialisation errors

Raises:

pssh.exceptions.SFTPIOError on I/O errors reading from SFTP

Raises:

OSError on local OS errors like permission denied

Note

Local directories in local_file that do not exist will be created as long as permissions allow.

Note

File names will be de-duplicated by appending the hostname to the filepath separated by suffix_separator.

join(output, consume_output=False, timeout=None)

Wait until all remote commands in output have finished and retrieve exit codes. Does not block other commands from running in parallel.

Parameters:
  • output (dict as returned by pssh.pssh_client.ParallelSSHClient.get_output()) – Output of commands to join on
  • consume_output (bool) – Whether or not join should consume output buffers. Output buffers will be empty after join if set to True. Must be set to True to allow host logger to log output on call to join when host logger has been enabled.
  • timeout (int) – Timeout in seconds if remote command is not yet finished. Note that use of timeout forces consume_output=True otherwise the channel output pending to be consumed always results in the channel not being finished.
Raises:

pssh.exceptions.Timeout on timeout requested and reached with commands still running.

Return type:

None

reset_output_generators(host_out, timeout=None, client=None, channel=None, encoding='utf-8')

Reset output generators for host output.

Parameters:
  • host_out (pssh.output.HostOutput) – Host output
  • client (pssh.ssh2_client.SSHClient) – (Optional) SSH client
  • channel (ssh2.channel.Channel) – (Optional) SSH channel
  • timeout (int) – (Optional) Timeout setting
  • encoding (str) – (Optional) Encoding to use for output. Must be valid Python codec
Return type:

tuple(stdout, stderr)

run_command(command, sudo=False, user=None, stop_on_errors=True, use_pty=False, host_args=None, shell=None, encoding='utf-8', timeout=None, greenlet_timeout=None)

Run command on all hosts in parallel, honoring self.pool_size, and return output dictionary.

This function will block until all commands have been received by remote servers and then return immediately.

More explicitly, function will return after connection and authentication establishment and after commands have been accepted by successfully established SSH channels.

Any connection and/or authentication exceptions will be raised here and need catching unless run_command is called with stop_on_errors=False in which case exceptions are added to individual host output instead.

Parameters:
  • command (str) – Command to run
  • sudo (bool) – (Optional) Run with sudo. Defaults to False
  • user (str) – (Optional) User to run command as. Requires sudo access for that user from the logged in user account.
  • stop_on_errors (bool) – (Optional) Raise exception on errors running command. Defaults to True. With stop_on_errors set to False, exceptions are instead added to output of run_command. See example usage below.
  • shell (str) – (Optional) Override shell to use to run command with. Defaults to login user’s defined shell. Use the shell’s command syntax, eg shell=’bash -c’ or shell=’zsh -c’.
  • use_pty (bool) – (Optional) Enable/Disable use of pseudo terminal emulation. Disabling it will prohibit capturing standard input/output. This is required in majority of cases, exceptions being where a shell is not used and/or input/output is not required. In particular when running a command which deliberately closes input/output pipes, such as a daemon process, you may want to disable use_pty. Defaults to True
  • host_args (tuple or list) – (Optional) Format command string with per-host arguments in host_args. host_args length must equal length of host list - pssh.exceptions.HostArgumentException is raised otherwise
  • encoding (str) –

    Encoding to use for output. Must be valid Python codec

  • timeout (int) – (Optional) Timeout in seconds for reading from stdout or stderr. Defaults to no timeout. Reading from stdout/stderr will raise pssh.exceptions.Timeout after timeout number seconds if remote output is not ready.
  • greenlet_timeout (float) – (Optional) Greenlet timeout setting. Defaults to no timeout. If set, this function will raise gevent.Timeout after greenlet_timeout seconds if no result is available from greenlets. In some cases, such as when using proxy hosts, connection timeout is controlled by proxy server and getting result from greenlets may hang indefinitely if remote server is unavailable. Use this setting to avoid blocking in such circumstances. Note that gevent.Timeout is a special class that inherits from BaseException and thus can not be caught by stop_on_errors=False.
Return type:

Dictionary with host as key and pssh.output.HostOutput as value as per pssh.pssh_client.ParallelSSHClient.get_output()

Raises:

pssh.exceptions.AuthenticationException on authentication error

Raises:

pssh.exceptions.UnknownHostException on DNS resolution error

Raises:

pssh.exceptions.ConnectionErrorException on error connecting

Raises:

pssh.exceptions.HostArgumentException on number of host arguments not equal to number of hosts

Raises:

TypeError on not enough host arguments for cmd string format

Raises:

KeyError on no host argument key in arguments dict for cmd string format

Raises:

pssh.exceptions.ProxyError on errors connecting to proxy if a proxy host has been set.

Raises:

gevent.Timeout on greenlet timeout. Gevent timeout can not be caught by stop_on_errors=False.

Raises:

Exceptions from ssh2.exceptions for all other specific errors such as ssh2.exceptions.SocketDisconnectError et al.

scp_recv(remote_file, local_file, recurse=False, copy_args=None, suffix_separator='_')

Copy remote file(s) in parallel via SCP as <local_file><suffix_separator><host> or as per copy_args argument.

With a local_file value of myfile and default separator _ the resulting filename will be myfile_myhost for the file from host myhost.

De-duplication behaviour is configurable by providing copy_args argument, see below.

This function, like ParallelSSHClient.scp_send(), returns a list of greenlets which can be join-ed on to wait for completion.

gevent.joinall() function may be used to join on all greenlets and will also raise exceptions if called with raise_error=True - default is False.

Alternatively call .get on each greenlet to raise any exceptions from it.

Exceptions listed here are raised when either gevent.joinall(<greenlets>, raise_error=True) is called or .get is called on each greenlet, not this function itself.

Parameters:
  • remote_file (str) – remote filepath to copy to local host
  • local_file (str) – local filepath on local host to copy file to
  • recurse (bool) – whether or not to recurse
  • suffix_separator (str) – (Optional) Separator string between filename and host, defaults to _. For example, for a local_file value of myfile and default separator the resulting filename will be myfile_myhost for the file from host myhost. suffix_separator has no meaning if copy_args is provided
  • copy_args (tuple or list) – (Optional) format remote_file and local_file strings with per-host arguments in copy_args. copy_args length must equal length of host list - pssh.exceptions.HostArgumentException is raised otherwise
Return type:

list(gevent.Greenlet) of greenlets for remote copy commands.

Raises:

ValueError when a directory is supplied to local_file and recurse is not set.

Raises:

pssh.exceptions.HostArgumentException on number of per-host copy arguments not equal to number of hosts.

Raises:

pss.exceptions.SCPError on errors copying file.

Raises:

OSError on local OS errors like permission denied.

Note

Local directories in local_file that do not exist will be created as long as permissions allow.

Note

File names will be de-duplicated by appending the hostname to the filepath separated by suffix_separator or as per copy_args argument if provided.

scp_send(local_file, remote_file, recurse=False)

Copy local file to remote file in parallel via SCP.

This function returns a list of greenlets which can be join-ed on to wait for completion.

gevent.joinall() function may be used to join on all greenlets and will also raise exceptions from them if called with raise_error=True - default is False.

Alternatively call .get() on each greenlet to raise any exceptions from it.

Note

Creating remote directories when either remote_file contains directory paths or recurse is enabled requires SFTP support on the server as libssh2 SCP implementation lacks directory creation support.

Parameters:
  • local_file (str) – Local filepath to copy to remote host
  • remote_file (str) – Remote filepath on remote host to copy file to
  • recurse (bool) – Whether or not to descend into directories recursively.
Return type:

list(gevent.Greenlet) of greenlets for remote copy commands.

Raises:

pss.exceptions.SCPError on errors copying file.

Raises:

OSError on local OS errors like permission denied.