Jobs running on the DNAnexus Platform can be optionally configured to allow
SSH connections to the DNAnexus worker executing the job. This can be used
to monitor the job underway, to troubleshoot a failed job, or to employ the worker as a workstation in the cloud.
By default, jobs are firewalled from the Internet, and only have network access to the DNAnexus API. Outbound access can
be configured using
network access permissions. Inbound access is
restricted to SSH connectivity only, and must be enabled explicitly by the user at run time.
Logging and reproducibility guarantees
A quick note about reproducibility guarantees: the DNAnexus platform provides features in support of high-level reproducibility of analyses. In general, apps given the
same inputs produce the same outputs when run again. Features such as network access (other than to the DNAnexus API)
are incompatible with this reproducibility support. Please be aware that when you connect to a job, reproducibility can
no longer be ensured.
Setting up your environment for SSH access
One-time setup of the user's account is required to allow use of SSH connections. Open a command shell and use
dx ssh_config to perform this setup.
If you haven't set up a public SSH key on the machine you're using before, use option 0 ("Generate a new SSH key pair using ssh-keygen").
If you have previously set up a public SSH key on the machine you're using, select one of the SSH key pairs listed, or option 0 if you wish to generate a new key pair.
If you have previously set up a public SSH key on another computer, you'll have to copy the keys into your current environment. The keys are found (by default) in
If you already have a key and you generate a new one, the old one will be overwritten. Note that you won't be able to SSH into jobs from any computer that has an older key.
Connecting to jobs via ssh
Once you have setup your ssh keypair, you can connect to individual jobs you launch on DNAnexus.
SSH connections to jobs can be established by running
dx ssh job-xxxx
Alternatively, you can launch a job and immediately ssh into that job by running:
dx run executable --ssh.
Another good use of ssh features is in debugging failing jobs. The platform allows you to specify conditions under which a failed job should be held for debugging using
dx run --debug-on. If the job encounters an error, the worker will not be terminated as usual, but will remain open for 72 hours, giving the user time to ssh into the worker and debug the underlying issue. See the
Debug Hold section below.
Finally, a great use of ssh features on the platform is to launch a cloud workstation.
For more information about configuring a job to employ a worker as a
workstation in the cloud, please see our
tutorial on cloud workstations.
Setting up SSH on Windows
If you are using Windows, we highly recommend installing OpenSSH, an open source connectivity tool for remote login with the SSH protocol.
Once OpenSSH is installed, you should be able to follow the instructions above.
For PuTTY users
If you would like to use PuTTY (external link) as your ssh terminal, you will first need to generate your public/private key pair as described above. Once that is done, you can do the following:
Use PuTTYgen to import your private key. You can do this by:
- Clicking on "Load" in PuTTYgen, changing the filetype from "PuTTY Private Key Files" to "All Files", and selecting the
ssh_id file found in the
.dnanexus_config folder inside of your Windows User folder.
- Then click on "Save private key" and save as "ssh_id.ppk"
Get the URL of the job you would like to ssh into by running
dx describe job-<id> and making note of the url found in the
- Place the URL found above in the "Host Name" field
- In Connection->Data, place
dnanexus in the "Auto-login username" field.
- In Connection->Data->SSH->Auth, click "Browse" to select your "ssh_id.ppk" file you generated in step 1.
Change "Window->Translation->Remote character set" session setting from UTF-8 to ISO-8859-1.
Set “Terminal->Keyboard->The Function keys and keypad” to Xterm R6 to allow Byobu hotkeys to work.
Open button at the bottom of the screen and you will login to your job!
Setting up SSH access via proxy
If you would like to use a proxy during SSH connection, run
dx ssh --ssh-proxy <proxy_address>:<proxy_port> or
dx run --ssh --ssh-proxy <proxy_address>:<proxy_port>. For
dx run with SSH
access, you must specify both
--ssh-proxy at runtime. If you do
not specify the proxy with
dx run --ssh, you will have to exit out of the
current SSH session and initiate a new session using
dx ssh --ssh-proxy <proxy_address>:<proxy_port> to access the job via proxy.
The following details describe the API functionality used by the high-level
dx commands described above.
SSH connection authentication is supported using SSH keys only. To configure a user's account to allow SSH connections
to jobs, the
sshPublicKey field is passed to
/user-xxxx/update. This field is returned in
/user-xxxx/describe only when called by that
user; other users cannot see the user's SSH public key.
To configure the job to allow SSH connections, the
allowSSH field is passed to
/job-xxxx/update methods. A user
can connect to a job only if all of the following conditions are satisfied:
- The job was started by that user.
- The job's
allowSSH field is set
and includes a hostmask that matches the IP address from which the
connection is being made.
- The job was started by running an applet, an open-source
app, or a non-open-source app that the user is listed as a developer for.
- The SSH client uses a private key that matches the public key stored in their user account.
Once connected, the terminal uses a byobu/tmux terminal window
manager, allowing the terminal to persist even if you get disconnected. Further information on how to use the terminal
is presented in a banner when you log in.
If a job is terminated while you are logged in via SSH, your connection will be terminated as well.
Jobs can be optionally configured to hold the execution environment for debugging when certain types of errors happen
(debug hold). When jobs are held for debugging, the user can log in to their execution environment using
dx ssh (as
long as it has been configured via
When entering debug hold, the job transitions into the
state, and any extra information about the failure reason is set in
debug.failureMessage fields. Jobs in
this state are held for up to 2 days, and terminated by the system
afterwards. Jobs in debug hold can also be terminated by the user
either using the API (e.g. via
dx terminate), or by terminating the
process tree in the job's execution environment. In all cases, the job
can only transition into the
When a job's
debug.debugOn field contains one of the eligible failure reasons (
typically specified via
dx run --debug-on AppError --debug-on AppInternalError), the job can enter debug hold via the
following two conditions:
- The job can quit with a non-zero exit status. In this case the job's original process tree is replaced with a
- The job can create the file named
/.dx-hold. This will induce the execution environment to transition into debug
hold without altering the job's process tree, allowing any running processes to remain intact.