Monitoring Executions

__TOC__

This page covers how to monitor executions (jobs and analyses) on the DNAnexus platform. Monitoring includes the ability to see a job’s progress or a list of current or past executions.

On this page, dx watch is the main command used for monitoring job logs of current or past jobs, while dx find executions is the main command for listing executions. If you type dx watch -h or dx find executions -h into the command line, it will give you a list of options that we cover in more detail below.

By default, dx find executions will return up to ten of the most recent executions in your current project in order of execution creation time.

Monitoring a Job

You can use dx watch to view the log of a running job or any past jobs, which may have finished successfully, failed, or been terminated. To view additional options of dx watch, navigate to the "Advanced Options" section of this page.

Monitor a currently running job

If you'd like to view the job's log stream while it runs, you can use dx watch. The log stream includes a log of stdout, stderr, and additional information the worker outputs as it executes the job.

$dx watch job-xxxx Watching job job-xxxx. Press Ctrl+C to stop. * Sample Prints (sample_prints:main) (running) job-xxxx amy 2017-01-01 09:00:00 (running for 0:00:37) 2017-01-01 09:06:00 Sample Prints INFO Logging initialized (priority) 2017-01-01 09:06:37 Sample Prints INFO CPU: 4% (4 cores) * Memory: 547/7479MB * Storage: 74GB free * Net: 0↓/0↑MBps 2017-01-01 09:06:37 Sample Prints STDOUT dxpy/0.227.1 (Linux-3.13.0-125-generic-x86_64-with-Ubuntu-14.04-trusty) 2017-01-01 09:06:37 Sample Prints INFO Installing apt packages dx-toolkit 2017-01-01 09:06:37 Sample Prints INFO Setting SSH public key 2017-01-01 09:06:37 Sample Prints STDOUT dxpy/0.227.1 (Linux-3.13.0-125-generic-x86_64-with-Ubuntu-14.04-trusty) 2017-01-01 09:06:37 Sample Prints STDOUT /usr/sbin/sshd already running. 2017-01-01 09:06:37 Sample Prints STDOUT Invoking main with {} 2017-01-01 09:06:37 Sample Prints STDOUT 0 ...

Monitor past jobs

If you'd like to view any jobs that have finished running, you can use the dx watch command. The log stream includes a log of stdout, stderr, and additional information the worker outputs as it executed the job.

$dx watch job-xxxx Watching job job-xxxx. Press Ctrl+C to stop. * Sample Prints (sample_prints:main) (running) job-xxxx amy 2017-01-01 09:00:00 (running for 0:00:37) 2017-01-01 09:06:00 Sample Prints INFO Logging initialized (priority) 2017-01-01 09:06:37 Sample Prints INFO CPU: 4% (4 cores) * Memory: 547/7479MB * Storage: 74GB free * Net: 0↓/0↑MBps 2017-01-01 09:06:37 Sample Prints STDOUT dxpy/0.227.1 (Linux-3.13.0-125-generic-x86_64-with-Ubuntu-14.04-trusty) 2017-01-01 09:06:37 Sample Prints INFO Installing apt packages dx-toolkit 2017-01-01 09:06:37 Sample Prints INFO Setting SSH public key 2017-01-01 09:06:37 Sample Prints STDOUT dxpy/0.227.1 (Linux-3.13.0-125-generic-x86_64-with-Ubuntu-14.04-trusty) 2017-01-01 09:06:37 Sample Prints STDOUT /usr/sbin/sshd already running. 2017-01-01 09:06:37 Sample Prints STDOUT Invoking main with {} 2017-01-01 09:06:37 Sample Prints STDOUT 0 2017-01-01 09:06:37 Sample Prints STDOUT 1 2017-01-01 09:06:37 Sample Prints STDOUT 2 2017-01-01 09:06:37 Sample Prints STDOUT 3 * Sample Prints (sample_prints:main) (done) job-xxxx amy 2017-01-01 09:08:11 (runtime 0:02:11) Output: -

Finding Executions

You can use dx find executions to return the ten most recent executions in your current project. You can specify the number of executions you wish to view by running dx find executions -n <specified number>. The output from dx find executions will be similar to the information shown in the "Monitor" tab on the DNAnexus web UI.

In order to learn how to view your current project, select a different project, or view objects in your projects, see the Project and Data Object Navigation page.

Below is an example of dx find executions; in this case, only two executions have been run in the current project. There is an individual job, LoFreq Variant Caller, and a workflow consisting of two stages, Variant Calling Workflow. A stage is represented by either another analysis (if running a workflow) or a job (if running an app(let)).

The job running the LoFreq Variant Caller executable is running and has been running for 10 minutes and 28 seconds. The analysis running the Variant Calling Workflow consists of 2 stages, Freebayes Variant Caller, which is waiting on input, and BWA-MEM FASTQ Read Mapper, which is has been running for 10 minutes and 18 seconds.

$ dx find executions * LoFreq Variant Caller (lofreq:main) (running) job-xxxx amy 2017-01-01 09:00:18 (running for 0:10:18) * Variant Calling Workflow (in_progress) analysis-xxxx │ amy 2017-01-01 09:00:18 ├── * FreeBayes Variant Caller (freebayes:main) (waiting_on_input) job-yyyy │ amy 2017-01-01 09:00:18 └── * BWA-MEM FASTQ Read Mapper (bwa_mem_fastq_read_mapper:main) (running) job-zzzz amy 2017-01-01 09:00:18 (running for 0:10:18)

What does dx find executions specifically search for?

By default, the dx find executions operation will search for jobs or analyses created when an app or applet is run by a user. If a job is part of an analysis, the results will be returned in a tree representation linking all of the jobs in an analysis together. By default, the command will not return subjobs, or jobs launched by the master jobs.

However, a user can also filter the returned executions by job type. Using the flag --origin-jobs in conjunction with the dx find executions command returns only original jobs, whereas the flag --all-jobs will also include subjobs. For more information on the different types of jobs, see the guide to the Job Lifecycle.

Finding Analyses

We can choose to monitor only analyses by running the command dx find analyses. Analyses are executions of workflows and consist of one or more app(let)s being run. When using dx find analyses, the command will return only the top-level analyses, not any of the jobs contained therein.

Below is an example of dx find analyses:

$ dx find analyses * Variant Calling Workflow (in_progress) analysis-xxxx amy 2017-01-01 09:00:18

Finding Jobs

Jobs are runs of an individual app(let) and make up analyses. We can choose to monitor only jobs by running the command dx find jobs, which will return a flat list of jobs. If a job is in an analysis, all jobs within the analysis are also returned.

Below is an example of dx find jobs:

$ dx find jobs * LoFreq Variant Caller (lofreq:main) (running) job-xxxx amy 2017-01-01 09:10:00 (running for 0:00:28) * FreeBayes Variant Caller (freebayes:main) (waiting_on_input) job-yyyy amy 2017-01-01 09:00:18 * BWA-MEM FASTQ Read Mapper (bwa_mem_fastq_read_mapper:main) (running) job-zzzz amy 2017-01-01 09:00:18 (running for 0:10:18)

Advanced Options

Searches for executions can be restricted to specific parameters.

View stdout and/or stderr from a job log

  • To extract stdout only from this job, we can run the command dx watch job-xxxx --get-stdout.
  • To extract stderr only from this job, we can run the command dx watch job-xxxx --get-stderr.
  • To extract only stdout and stderr from this job, we can run the command dx watch job-xxxx --get-streams.

Below is an example of viewing stdout lines of a job log:

$ dx watch job-xxxx --get-streams Watching job job-xxxx. Press Ctrl+C to stop. dxpy/0.227.1 (Linux-3.13.0-125-generic-x86_64-with-Ubuntu-14.04-trusty) dxpy/0.227.1 (Linux-3.13.0-125-generic-x86_64-with-Ubuntu-14.04-trusty) /usr/sbin/sshd already running. Invoking main with {} 0 1 2 3 4 5 6 7 8 9 10

View subjobs

To view the entire job tree, including both main jobs and subjobs, use the command dx watch job-xxxx --tree.

View first n messages of a job log

To view the entire job tree, main jobs and subjobs, use the command dx watch job-xxxx -n 8. If the job already ran, the output is displayed as well.

In the example below, the app Sample Prints doesn’t have any output.

$ dx watch job-F5vPQg807yxPJ3KP16Ff1zyG -n 8 Watching job job-xxxx. Press Ctrl+C to stop. * Sample Prints (sample_prints:main) (done) job-xxxx amy 2017-01-01 09:00:00 (runtime 0:02:11) 2017-01-01 09:06:00 Sample Prints INFO Logging initialized (priority) 2017-01-01 09:08:11 Sample Prints INFO CPU: 4% (4 cores) * Memory: 547/7479MB * Storage: 74GB free * Net: 0↓/0↑MBps 2017-01-01 09:08:11 Sample Prints STDOUT dxpy/0.227.1 (Linux-3.13.0-125-generic-x86_64-with-Ubuntu-14.04-trusty) 2017-01-01 09:08:11 Sample Prints INFO Installing apt packages dx-toolkit 2017-01-01 09:08:11 Sample Prints INFO Setting SSH public key 2017-01-01 09:08:11 Sample Prints STDOUT dxpy/0.227.1 (Linux-3.13.0-125-generic-x86_64-with-Ubuntu-14.04-trusty) 2017-01-01 09:08:11 Sample Prints STDOUT /usr/sbin/sshd already running. * Sample Prints (sample_prints:main) (done) job-F5vPQg807yxPJ3KP16Ff1zyG amy 2017-01-01 09:00:00 (runtime 0:02:11) Output: -

Search across all projects

By default, dx find will restrict your search to only your current project context. To search across all the projects to which you have access, use the --all-projects flag.

$ dx find executions -n 3 --all-projects * Sample Prints (sample_prints:main) (done) job-xxxx amy 2017-01-01 09:15:00 (runtime 0:02:11) * Sample Applet (sample_applet:main) (done) job-yyyy ben 2017-01-01 09:10:00 (runtime 0:00:28) * Sample Applet (sample_applet:main) (failed) job-zzzz amy 2017-01-01 09:00:00 (runtime 0:19:02)

Return more than ten results

By default, dx find will only return up to ten of the most recently launched executions matching your search query. To change the number of executions returned, you can use the -n option.

# Find the 100 most recently launched jobs in your project $ dx find executions -n 100

Search by executable

A user can search for only executions of a specific app(let) or workflow based on its entity ID.

# Find most recent executions running app-lofreq in the current project $ dx find executions --executable app-lofreq * LoFreq Variant Caller (lofreq:main) (running) job-xxxx amy 2017-01-01 09:00:18 (running for 0:10:18)

Search by execution start time

Users can also use the --created-before and --created-after options to search based on when the execution began.

Search by date

# Find executions run on January 2, 2017 $ dx find executions --created-after=2017-01-01 --created-before=2017-01-03

Search by time

# Find executions created in the last 2 hours $ dx find executions --created-after=-2h # Find analyses created in the last 5 days $ dx find analyses --created-after=-5d

Search by execution state Users can also restrict the search to a specific state, e.g. "done", "failed", "terminated". The full list of states can be found in the API documentation.

# Find failed jobs in the current project $ dx find jobs --state failed

Scripting

Delimiters

The --delim flag will tab-delimit the output. This allows the output to be passed into other shell commands.

$ dx find jobs --delim * Cloud Workstation (cloud_workstation:main) done job-xxxx amy 2017-01-07 09:00:00 (runtime 1:00:00) * GATK3 Human Exome Pipeline(gatk3_human_exome_pipeline:main) done job-yyyy amy 2017-01-07 09:00:00 (runtime 0:21:16)

Return only IDs

You can use the --brief flag to return only the object IDs for the objects returned by your search query. The ‑‑origin‑jobs flag will omit the subjob information.

Below is an example usage of the --brief flag:

$ dx find jobs executions -n 3 --delim job-xxxx job-yyyy job-zzzz

Below is an example of using the flags --origin-jobs and --brief. In the example below, we describe the last job run in the current default project.

$ dx describe $(dx find jobs -n 1 --origin-jobs --brief) Result 1: ID job-xxxx Class job Job name BWA-MEM FASTQ Read Mapper Executable name bwa_mem_fastq_read_mapper Project context project-xxxx Billed to amy Workspace container-xxxx Cache workspace container-yyyy Resources container-zzzz App app-xxxx Instance Type mem1_ssd1_x8 Priority high State done Root execution job-zzzz Origin job job-zzzz Parent job - Function main Input genomeindex_targz = file-xxxx reads_fastqgz = file-xxxx [read_group_library = "1"] [mark_as_secondary = true] [read_group_platform = "ILLUMINA"] [read_group_sample = "1"] [add_read_group = true] [read_group_id = {"$dnanexus_link": {"input": "reads_fastqgz", "metadata": "name"}}] [read_group_platform_unit = "None"] Output - Output folder / Launched by amy Created Sun Jan 1 09:00:17 2017 Started running Sun Jan 1 09:00:10 2017 Stopped running Sun Jan 1 09:00:27 2017 (Runtime: 0:00:16) Last modified Sun Jan 1 09:00:28 2017 Depends on - Sys Requirements {"main": {"instanceType": "mem1_ssd1_x8"}} Tags - Properties -

Rerun time-specific failed jobs with updated instance type

# Find failed jobs in the current project from a time period $ dx find jobs --state failed --created-after=2017-01-01 --created-before=2017-02-01 * BWA-MEM FASTQ Read Mapper (bwa_mem_fastq_read_mapper:main) (failed) job-xxxx amy 2017-01-22 09:00:00 (runtime 0:02:12) * BWA-MEM FASTQ Read Mapper (bwa_mem_fastq_read_mapper:main) (done) job-yyyy amy 2017-01-07 06:00:00 (runtime 0:11:22)

Rerun failed executions with an updated executable

# Find all failed executions of specified executable $ dx find executions --state failed --executable app-bwa_mem_fastq_read_mapper * BWA-MEM FASTQ Read Mapper (bwa_mem_fastq_read_mapper:main) (failed) job-xxxx amy 2017-01-01 09:00:00 (runtime 0:02:12) # Update the app and navigate to within app directory $dx build -a INFO:dxpy:Archived app app-xxxx to project-xxxx:"/.App_archive/bwa_mem_fastq_read_mapper (Sun Jan 1 09:00:00 2017)" {"id": "app-yyyy"} # Rerun job with updated app $dx run bwa_mem_fastq_read_mapper --clone job-xxxx

Find the status of jobs related to a given tag

$dx find jobs --tag

Last edited by Samantha Zarate, 2017-08-10 17:46:00

 Feedback