dxworkflow.json (Workflow metadata)

The file dxworkflow.json is a DNAnexus workflow metadata file. If a dxworkflow.json file is detected in the directory provided to dx build, the toolkit will attempt to build a workflow in the platform according to the workflow specification in the JSON file.

The format of the file closely resembles that of the corresponding calls to /workflow/new.

The next section shows a detailed example of the fields used in the file. The specification for the possible options can be found in the Specification section.

Annotated Example

The following lists the contents of an example dxworkflow.json that should be provided in a directory for use with the dx build command.

Note that comments as shown below are not valid in the JSON format but are provided here for easy reference.

{
 "name": "exome_variant_calling",     # (optional for regular, project-based workflows;
                                      #   required for global workflows) Workflow name
 "title" "Exome Variant Calling",     # (optional) Title of a workflow, it is a label
                                      #   used when displaying, searching, or listing the workflow
                                      #   in the UI or CLI
 "version" "1.0.0",                   # (optional for regular, project-based workflows;
                                      #   required for global workflows) Version of the global workflow
 "summary" "A simple exome calling pipeline",
                                      # (optional) A short description of the workflow
 "outputFolder": "output",            # (optional) Folder for the workflow's output
 "inputs": [                          # (optional) Workflow level input specification (see API documentation)
   {
    "name": "reads",                  # Name of the workflow-level input
    "class": "file"                   # Class of the workflow-level input
   }
 ],
 "outputs": [                         # (optional) Workflow level output specification (see API documentation)
   {
    "name": "variants",               # Name of the workflow-level output
    "class": "file",                  # Class of the workflow-level output
    "outputSource": {                 # Link to the output of the stage which
       "$dnanexus_link": {            #   provides the output of the workflow
         "outputField": "variants",
         "stage": "call_variants"
       }
     }
   }
 ], 
 "stages": [                          # (optional) A list of stages
  {
   "id": "map_reads",                 # Unique ID of the first stage
   "name": "BWA MEM",                 # (optional) Display name of the first stage
   "executable": "app-bwa_mem_fastq_read_mapper/1.5.0",
                                      # Name or ID of the app or ID of the applet run in this stage
   "folder": "map_reads_output"       # The output folder into which outputs should be
                                      #   cloned for the stage
   "input": {                         # (optional) Input of the first stage
    "genomeindex_targz": {            # Input field name
     "$dnanexus_link": {              # Link to a reference genome file
      "project": "project-xxxx",
      "id": "file-aaaa"
     }
    },
    "reads_fastqgz": {                # Input field name
     "$dnanexus_link": {              # Link to the workflow level input; the input passed to "reads" on the
      "workflowInputField": "reads"   #   workflow level will be consumed by the "reads_fastqgz" input
     }
    }
   },
   "systemRequirements": {            # (optional) Request different instance types for different entry
                                      #   points
     "main": {                        # "main" is the name of the entry point that is called when a
       "instanceType": "mem2_hdd2_x4" #   stage is run
     }
   },
   "executionPolicy": {               # (optional) Options governing job restart policy
     "restartOn": {
       "*": 3                         # Restart automatically up to 3 times for system errors
     }
   }
  },
  {
   "id": "call_variants",
                                     # Unique ID of the second stage
   "name": "Vendor Human Exome GATK-Lite",
                                     # (optional) Display name of the second stage
   "executable": "app-vendor_human_exome_gatk_lite_pipeline/1.1.6",
                                     # Name or ID of the app or ID of the applet run in this stage
   "folder": "call_variants_output"  # The output folder into which outputs should be
                                     #   cloned for the stage
   "input": {                        # (optional) Input of the second stage which is linked
     "sorted_bam": {                 # to the output of "sorted_bam" of the first stage
       "$dnanexus_link": {
         "outputField": "sorted_bam",
         "stage": "map_reads"
       }
     }
   }
  }
 ]
}

Other options for the /workflow/new call, such as specifying in which project or folder to create a workflow, are populated via command-line flags of dx build.

Specification

name

string The name of the workflow. If it is not provided, the auto-generated workflow ID will be used. When a global workflow is built (with dx build --globalworkflow) the name is required and more strict formatting rules apply: the name can have lower case letters, numbers, -, ., and _ but cannot have spaces.

Example

{
...
  "name": "exome_variant_calling",
...
}

title

string The title of the workflow. It is a label displayed to the users in the Web Interface. If it is not provided, the name of the workflow will be used.

Example

{
...
  "title": "Exome Variant Calling",
...
}

version

string (Global workflows only) The version of the workflow. This version must be unique from all other versions of the global workflow (published or not).

We recommend following the Semantic Versioning (external link) conventions for numbering the versions of your global workflow. Semantic Versioning also specifies how you should change the version number for various kinds of updates to your global workflow (that is, bug-fix only, backwards compatible, or backwards incompatible). Using the Semantic Versioning guidelines will help users and other developers to understand when it is safe to move between different versions of your global workflow.

Example

{
...
  "version": "1.0.0",
...
}

summary

string A short description of the workflow.

Example

{
...
  "summary": "A simple exome calling pipeline",
...
}

outputFolder

string (optional) The default output folder for the workflow; see the Customizing Output Folders specification for more details on how it interacts with stages' output folders.

Example

{
...
  "outputFolder": "output",
}

inputs

array of mappings (optional) JSON array containing the specifications for each input to the workflow. The full input specification documentation can be found here.

Example

[
  {
    "name": "reads",
    "class": "file",
    "default": {"$dnanexus_link": "file-xxxx"}
   }
]

outputs

array of mappings (optional) JSON array containing the specifications for each output of the workflow. The specification is the same as the output specification of an app(let) with an addition of the "outputSource" field, which allows the workflow developer to link specific stage outputs to workflow outputs.

Example

[
  {
    "name": "reads",
    "class": "file",
    "outputSource": {"$dnanexus_link": {
        "stage": "stage_id",
        "outputField": "executable_output_fieldname"
      }
    }
  }
]

stages

string (optional) A list of stages to add to the workflow. See the stages input field of the /workflow/new call for a detailed specification.

Example

{
...
"stages": [
 {
  "id": "map_reads",
  "name": "BWA MEM",
  "executable": "app-bwa_mem_fastq_read_mapper/1.5.0",
  "input": {
   "genomeindex_targz": {
    "$dnanexus_link": {
     "project": "project-xxxx",
     "id": "file-aaaa"
    }
   },
   "reads_fastqgz": {
    "$dnanexus_link": {
     "project": "project-yyyy",
     "id": "file-bbbb"
    }
   }
  },
  "systemRequirements": {
    "main": {
      "instanceType": "mem2_hdd2_x4"
    }
  },
  "executionPolicy": {
    "restartOn": {
      "*": 3
    }
  }
 },
 {
  "id": "call_variants",
  "name": "Vendor Human Exome GATK-Lite",
  "executable": "app-vendor_human_exome_gatk_lite_pipeline/1.1.6",
  "folder": "call_variants_output"
  "input": {
    "sorted_bam": {
      "$dnanexus_link": {
        "outputField": "sorted_bam",
        "stage": "map_reads"
      }
    },
    "vendor_exome": "agilent_sureselect_human_all_exon_v2"
  }
 }
]
}
...

Next steps

See the following guides for details on how to build a workflow from a dxworkflow.json specification:

Last edited by Aleksandra Zalcman, 2018-06-07 02:35:45

 Feedback