Helpstrings of SDK Command-Line Utilities

Below is a list of some of the various command-line utilities available in the SDK and some brief documentation for their usage.

General purpose dx utilities

dx

usage: dx [-h] [--version] command ...

DNAnexus Command-Line Client, API v1.0.0, client v0.240.1

dx is a command-line client for interacting with the DNAnexus platform.  You
can log in, navigate, upload, organize and share your data, launch analyses,
and more.  For a quick tour of what the tool can do, see

  https://wiki.dnanexus.com/Command-Line-Client/Quickstart

For a breakdown of dx commands by category, run "dx help".

dx exits with exit code 3 if invalid input is provided or an invalid operation
is requested, and exit code 1 if an internal error is encountered.  The latter
usually indicate bugs in dx; please report them at

  https://github.com/dnanexus/dx-toolkit/issues

optional arguments:
  -h, --help  show this help message and exit
  --env-help  Display help message for overriding environment
              variables
  --version   show program's version number and exit

dx-app-wizard

usage: dx-app-wizard [-h] [--json-file JSON_FILE] [--language LANGUAGE]
                     [--template {basic,parallelized,scatter-process-gather}]
                     [name]

Create a source code directory for a DNAnexus app. You will be prompted for
various metadata for the app as well as for its input and output
specifications.

positional arguments:
  name                  Name of your app

optional arguments:
  -h, --help            show this help message and exit
  --json-file JSON_FILE
                        Use the metadata and IO spec found in the given file
  --language LANGUAGE   Programming language of your app
  --template {basic,parallelized,scatter-process-gather}
                        Execution pattern of your app

dx-build-asset

usage: dx-build-asset [-h] [src_dir]

Builds a DNAnexus Asset.

positional arguments:
  src_dir     Asset source directory (default: current directory)

optional arguments:
  -h, --help  show this help message and exit

dx-build-report-html

usage: dx-build-report-html [-h] [-r REMOTE] [--local LOCAL] [-w WIDTH]
                            [-g HEIGHT]
                            src [src ...]

Constructs and saves/uploads an HTML report from HTML and/or linked images

positional arguments:
  src                   Source image or HTML file(s)

optional arguments:
  -h, --help            show this help message and exit
  -r REMOTE, --remote REMOTE
                        Destination route. Can be: (1) a project ID, (2) a
                        path, with or without object name (e.g.
                        /PATH/REPORT_NAME), (3) project ID + path (e.g.
                        PROJECT:/PATH/REPORT_NAME)
  --local LOCAL         Local file to save baked HTML to
  -w WIDTH, --width WIDTH
                        Width of the final report, in pixels
  -g HEIGHT, --height HEIGHT
                        Height of the final report, in pixels

dx-docker

usage: dx-docker [-h] {pull,run,add-to-applet,create-asset} ...

positional arguments:
  {pull,run,add-to-applet,create-asset}
    pull                Pulls a docker image for use in DNAnexus
    run                 Runs a docker image in a container
    add-to-applet       Adds a local Docker image to an applet
    create-asset        Caches a local Docker image as an asset in the
                        DNAnexus platform (subject to change)

optional arguments:
  -h, --help            show this help message and exit

dx-fetch-bundled-depends

usage: dx-fetch-bundled-depends [-h]

Downloads the contents of runSpec.bundledDepends of a job running in the
execution environment.

optional arguments:
  -h, --help  show this help message and exit

dx-jobutil-add-output

usage: dx-jobutil-add-output [-h] [--class [CLASSNAME]] [--array] name value

Reads and modifies job_output.json in your home directory to be a JSON hash
with key *name* and value  *value*.

If --class is not provided or is set to "auto", auto-detection of the output
format will occur.  In particular, it will treat it as a number, hash, or
boolean if it can be successfully parsed as such.  If it is a string which
matches the pattern for a data object ID, it will encapsulate it in a DNAnexus
link hash; otherwise it is treated as a simple string.

Use --array to append to an array of values or prepend "array:" to the --class
argument.

To use the output of another job as part of your output, use --class=jobref
(which will throw an error if it is not formatted correctly), or use the
automatic parsing which will recognize anything starting with a job ID as a
job-based object reference.  You should format the value as follows:

  Format: <job ID>:<output field name>
  Example: dx-jobutil-add-output myoutput --class=jobref \
             job-B2JKYqK4Zg2K915yQxPQ0024:other_output

Analysis references can be specified similarly with --class=analysisref and
formatted as:

  Format: <analysis ID>:<stage ID>.<output field name>
          <analysis ID>:<exported output field name>
  Example: dx-jobutil-add-output myoutput --class=analysisref \
             analysis-B2JKYqK4Zg2K915yQxPQ0024:some_output

positional arguments:
  name                 Name of the output field name
  value                Value of the output field

optional arguments:
  -h, --help           show this help message and exit
  --class [CLASSNAME]  Class of output for formatting purposes, e.g. "int";
                       default "auto"
  --array              Output field is an array

dx-jobutil-dxlink

usage: dx-jobutil-dxlink [-h] object

Creates a DNAnexus link from an object ID or "<project ID>:<object ID>"
string. The result is of the form {"$dnanexus_link": "<object ID>"} or
{"$dnanexus_link": {"project": <project ID>, "id": <object ID>}}, as
appropriate.

positional arguments:
  object      Data object ID or "<Project ID>:<Data object ID>" to package
              into a DNAnexus link

optional arguments:
  -h, --help  show this help message and exit

dx-jobutil-new-job

usage: dx-jobutil-new-job [-h] [-i INPUT] [-j INPUT_JSON] [-f FILENAME]
                          [--instance-type INSTANCE_TYPE_OR_MAPPING]
                          [--instance-type-help] [--extra-args EXTRA_ARGS]
                          [--property KEY=VALUE] [--tag TAG] [--name NAME]
                          [--depends-on [JOB_OR_OBJECT_ID [JOB_OR_OBJECT_ID ...]]]
                          function

Creates a new job to run the named function with the specified input. If
successful, prints the ID of the new job.

positional arguments:
  function              Name of the function to run

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        An input to be added using "<input
                        name>[:<class>]=<input value>" (provide "class" if
                        there is no input spec; it can be any job IO class,
                        e.g. "string", "array:string", or "array"; if "class"
                        is "array" or not specified, the value will be
                        attempted to be parsed as JSON and is otherwise
                        treated as a string)
  -j INPUT_JSON, --input-json INPUT_JSON
                        The full input JSON (keys=input field names,
                        values=input field values)
  -f FILENAME, --input-json-file FILENAME
                        Load input JSON from FILENAME ("-" to use stdin)
  --instance-type INSTANCE_TYPE_OR_MAPPING
                        Specify instance type(s) for jobs this executable will
                        run; see --instance-type-help for more details
  --instance-type-help  Print help for specifying instance types
  --extra-args EXTRA_ARGS
                        Arguments (in JSON format) to pass to the underlying
                        API method, overriding the default settings
  --property KEY=VALUE  Key-value pair to add as a property; repeat as
                        necessary, e.g. "--property key1=val1 --property
                        key2=val2"
  --tag TAG             Tag for the resulting execution; repeat as necessary,
                        e.g. "--tag tag1 --tag tag2"
  --name NAME           Name for the new job (default is the current job name,
                        plus ":<function>")
  --depends-on [JOB_OR_OBJECT_ID [JOB_OR_OBJECT_ID ...]]
                        Job and/or data object IDs that must finish or close
                        before the new job should be run. WARNING: For proper
                        parsing, do not use this flag directly before the
                        *function* parameter.

dx-jobutil-parse-link

usage: dx-jobutil-parse-link [-h] [--no-project] dxlink

Parse a dxlink JSON hash into an object ID or project:object-id tuple

positional arguments:
  dxlink        Link to parse

optional arguments:
  -h, --help    show this help message and exit
  --no-project  Ignore project ID in an extended dxlink - just print the
                object ID

dx-jobutil-report-error

usage: dx-jobutil-report-error [-h] message [{AppInternalError,AppError}]

Creates job_error.json in your home directory, a JSON file to include the
error type and message for the running job. There are two types of errors you
may report: 1) AppError (the default) for recognized actionable errors, and 2)
AppInternalError for unexpected application errors.

positional arguments:
  message               Error message for the job
  {AppInternalError,AppError}
                        Error type

optional arguments:
  -h, --help            show this help message and exit

dx-log-stream

usage: dx-log-stream [-h] [-l {critical,error,warning,info,debug}]
                     [-s {DX_APP,DX_APP_STREAM}]

Redirects stdin to a DNAnexus log socket in the execution environment.

Valid logging levels:

┌─────────────────────────┬────────────────┬────────────┐
│ --source                │ --level        │ Appears as │
├─────────────────────────┼────────────────┼────────────┤
│ DX_APP_STREAM (default) │ info (default) │ STDOUT     │
│ DX_APP_STREAM (default) │ error          │ STDERR     │
├─────────────────────────┼────────────────┼────────────┤
│ DX_APP                  │ debug          │ DEBUG      │
│ DX_APP                  │ info (default) │ INFO       │
│ DX_APP                  │ warning        │ WARNING    │
│ DX_APP                  │ error          │ ERROR      │
│ DX_APP                  │ critical       │ CRITICAL   │
└─────────────────────────┴────────────────┴────────────┘

optional arguments:
  -h, --help            show this help message and exit
  -l {critical,error,warning,info,debug}, --level {critical,error,warning,info,debug}
                        Logging level to use
  -s {DX_APP,DX_APP_STREAM}, --source {DX_APP,DX_APP_STREAM}
                        Source ID to use

dx-print-bash-vars

usage: dx-print-bash-vars [-h]

Parses $HOME/job_input.json and prints the bash variables that would be
available in the execution environment.

optional arguments:
  -h, --help  show this help message and exit

dx-verify-file

Usage: dx-verify-file [options] -r <remote_file1_id> -l <local_file1> [-r <remote_file2_id> -l <local_file2> ...]

Available options:
  -h [ --help ]            Produce a help message
  --version                Print the version
  -e [ --env ]             Print environment information
  -a [ --auth-token ] arg  Specify the authentication token
  -r [ --remote-file ] arg ID of the remote file
  -l [ --local-file ] arg  Local file path
  --read-threads arg (=1)  Number of parallel disk read threads
  --md5-threads arg (=1)   Number of parallel MD5 compute threads
  -v [ --verbose ]         Verbose logging

dx-workflow-to-applet

usage: dx-workflow-to-applet [-h] [--name NAME] [--overwrite] path

WARNING: dx-workflow-to-applet is deprecated, and will be removed
in a future release. Your workflows have been migrated. Please use
dx run <workflow> to run them. See dx run <workflow> --help.

Takes a workflow on the DNAnexus platform and makes a local directory out of
it which can be built as an applet using the "dx build" command.

When built, any referenced applets and data objects will be bundled with the
applet (copying the applet from one project to another will also copy any
dependencies).  If the applet is run without its dependencies in the same
project, it may fail.

You may also choose to build an app instead of an applet using the command "dx
build --create-app --no-temp-build-project", but make sure all applet and data
dependencies are present in your current project at the time you run the
command.  Otherwise, the dependencies will not be packaged with the app.

positional arguments:
  path             DNAnexus path to a workflow, e.g.
                   projectName:/folder/workflowname

optional arguments:
  -h, --help       show this help message and exit
  --name NAME      Name of applet to be created (default is the workflow
                   name)
  --overwrite, -f  Whether to overwrite a local directory with the same
                   name

Utilities useful in writing bash apps and applets

dx-download-all-inputs

usage: dx-download-all-inputs [-h] [--except EXCLUDE] [--parallel]
                              [--sequential]

Note: this is a utility for use by bash apps running in the DNAnexus Platform.

Downloads all files that were supplied as inputs to the app.  By convention,
if an input parameter "FOO" has value

    {"$dnanexus_link": "file-xxxx"}

and filename INPUT.TXT, then the linked file will be downloaded into the path:

    $HOME/in/FOO/INPUT.TXT

If an input is an array of files, then all files will be placed into numbered
subdirectories under a parent directory named for the input. For example, if
the input key is FOO, and the inputs are {A, B, C}.vcf then, the directory
structure will be:

    $HOME/in/FOO/0/A.vcf
                 1/B.vcf
                 2/C.vcf

Zero padding is used to ensure argument order. For example, if there are 12
input files {A, B, C, D, E, F, G, H, I, J, K, L}.txt, the directory structure
will be:

    $HOME/in/FOO/00/A.vcf
                 ...
                 11/L.vcf

This allows using shell globbing (FOO/*/*.vcf) to get all the files in the
input order.

optional arguments:
  -h, --help        show this help message and exit
  --except EXCLUDE  Do not download the input with this name. (May be used
                    multiple times.)
  --parallel        Download the files in parallel
  --sequential      Download the files sequentially

dx-upload-all-outputs

usage: dx-upload-all-outputs [-h] [--except EXCLUDE] [--parallel]
                             [--sequential] [--clearJSON CLEARJSON]
                             [--wait-on-close]

Note: this is a utility for use by bash apps running in the DNAnexus Platform.

Uploads all files and subdirectories in the directory $HOME/out, as described
below. It also adds relevant entries into the job_output.json file.

By convention, only directories with names equal to output parameter names are
expected to be found in the output directory, and any file(s) found in those
subdirectories will be uploaded as the corresponding outputs.  For example, a
file with the path

    $HOME/out/FOO/OUTPUT.TXT

will be uploaded, and the key "FOO" will be added to the job_output.json file
with value

    {"$dnanexus_link": "file-xxxx"}

where "file-xxxx" is the ID of the newly uploaded file. If multiple files are
found, they will be added as an array output (in unspecified order). If
subdirectories are found under $HOME/out/FOO, then they are uploaded in their
entirety to the workspace, and values are added to FOO in the job_output.json
file. For example, the path:

    $HOME/out/FOO/BAR/XXX.TXT

will be uploaded to /BAR/XXX.TXT.

optional arguments:
  -h, --help            show this help message and exit
  --except EXCLUDE      Do not upload the input with this name. (May be used
                        multiple times.)
  --parallel            Upload the files in parallel
  --sequential          Upload the files sequentially
  --clearJSON CLEARJSON
                        Clears the output JSON file prior to starting upload.
  --wait-on-close       Wait for files to close, default is not to wait

Last edited by Kurt Jensen, 2017-11-15 21:30:15

 Feedback