Applets and Entry Points

Applets

Applets are executable data objects. Like other data objects, applets exist inside projects and can be cloned between projects. By default, applets also have VIEW permissions into the projects from which they are run. Applets can be used to create private, customized scripts for specialized needs, or for testing and developing more general apps that may be of interest to the community at large.

Applets must be created with a run specification so that the system knows how to run it, but they can be created with or without input and output specifications. Providing I/O specifications is encouraged because it allows the system to validate the input arguments when an applet is launched, and validate the produced output when an applet is finished. Also, when an applet has an I/O specification, the DNAnexus website can automatically render a configuration form for users who want to launch the applet (since the system is aware of the names and types of inputs that the applet requires). Finally, other developers who want to invoke the applet from their applets can look at the I/O specification for help on how to launch the applet programmatically (i.e., what to give as inputs, or what to expect from the outputs).

If I/O specifications are not provided, users can launch the applet with any input (which the system will not validate, except for DNAnexus links and job-based object references), and the applet can produce any output. Developers of such applets are responsible for documenting what their applet expects in the input and what outputs their applet can produce. Likewise, they are responsible for providing a user interface for someone to configure the applet’s input on the DNAnexus’ website. This allows for building powerful applets whose input or output might be hard to formally describe, perhaps because it is variable or polymorphic, or would otherwise be constrained by what DNAnexus allows in its input/output definition grammar.

Running an applet: Project context

Running an applet is slightly different depending on whether the applet is launched by a user from outside of the DNAnexus platform, or by another running job.

Launching an applet outside of the platform requires associating a project with it. As mentioned earlier, this project context is important for the following reasons:

  • Any charges related to the execution of the applet are associated with that project.
  • Jobs (such as the job created by launching the applet, as well as any other jobs created by the applet itself while running) will be given VIEW access to that project.
  • Any objects output by the applet will be placed into that project.

When launching an applet from another running job, this parent job is already associated with a project. This project is carried forward to the launched master job; more specifically:

  • Any charges related to the execution of the master job are associated with that project.
  • Jobs (such as the job created by launching the master job, as well as any other descendant jobs of the master job) will be given VIEW access to that project.
  • Any objects output by the master job will be placed into the workspace of the parent job.

When running an applet, the applet object does not need to be inside the project context. Whoever runs the applet needs to have VIEW access to the applet object and CONTRIBUTE access to the project context. However, the generated job will have VIEW access to the project context and CONTRIBUTE access to the workspace only, and the system "acts as that job" for the purposes of accessing any references to file objects mandated by the applet spec, (such as assets mentioned under bundledDepends, etc.); therefore, these objects need to be accessible by the job, and this can only be done if they happen to be in the project context or in a public project. Thus, if an applet object’s dependencies are not in the project context, then running it will eventually fail unless it is in a public project, because the system will not be able to fetch associated files. As such, when an applet is cloned, all objects linked to in the bundledDepends of its run specification will also be cloned with it.

Running Entry Points

Any job can issue the /job/new API call to create a subjob that will run a particular entry point. Running an entry point is like calling a function in your code that will then execute on its own worker in the cloud. Because each job runs on a separate worker, any communication between jobs under the same master job must be via their input, output, and/or stored data objects in their shared temporary workspace. When an applet is first run, it automatically runs the entry point called "main". See Code Interpreters for more information on how to create entry points in your code given the interpreter of your choice.

Requesting Instance Types

Different instance types can be requested for different entry points of an applet using the systemRequirements field. This can be specified in an applet's run specification and partially (or fully) overridden at runtime (using the same syntax as in the run specification). When computing the effective value of systemRequirements, the keys are retrieved in the following order (using the first one found):

  • The systemRequirements value requested for the job's master job
    • If the job was run via /app-xxxx/run or /applet-xxxx/run, then this is the value of systemRequirements in the API call
    • If the job was run as a stage in a workflow, then the effective value is a combination of fields provided in /workflow-xxxx/run and of any default values stored in the workflow itself
  • The systemRequirements field provided to the /job/new method
  • The runSpec.systemRequirements field in the applet or app
  • If none of these values are present (or contain a relevant entry point), system-wide defaults are used

The "*" entry point refers to all other entry points that are not already named in systemRequirements. Note that if an applet has a run specification that includes an instance type X for entry point "main", and the applet is run with systemRequirements specifying that "*" should have a different instance type Y, the "main" entry point will be run on instance type Y because the "main" key has been implicitly overridden by the "*" option.

Specifying Job Timeouts

Job timeout policies (configurable by executable ID and then by entry point) may be specified to request timeouts for jobs running specific executables at specific entry points. When a job times out, the job is either terminated or restarted (depending on the job restart policy). Job timeout policies may be specified upon creation of the executable (app or applet; configurable by entry point) in the executable's run specification and may be overridden (in part or in full) at runtime (via an input to /app-xxxx/run, /applet-xxxx/run, /workflow-xxxx/run, or /job/new; configurable by executable ID and then by entry point). The effective timeout of a job running a specific executable at a specific entry point is determined from the following sources (in decreasing priority):

  • The runtime input timeoutPolicyByExecutable.<executable_id>.<entry_point> field. Note that this field overrides both user-specified and system default timeout policies, and it propagates down the entire resulting job execution tree and merges with all downstream runtime inputs, with explicit upstream inputs taking precedence.
  • The timeoutPolicy.<entry_point> field provided in the executable's run specification upon executable creation. This field serves as the user-specified default timeout policy and will override the system default timeout policy.
  • The system default timeout policy, which is effectively no timeout.

The '*' entry point refers to all other entry points that are not named in timeoutPolicyByExecutable.<executable_id> or the timeoutPolicy field of the executable's run specification.

Note that setting the timeout of a specific executable at a specific entry point to 0 is the same as not setting a timeout for that executable at that entry point at all (i.e. there exists no runtime or default run specification entry for that executable at that entry point).

Example: Suppose that an applet with ID <applet_id> has a run specification that includes a timeout policy for entry point '*' that specifies a timeout of 5 hours, and suppose that that applet is run with a runtime policy for <applet_id> for entry point 'main' that specifies a timeout of 2 hours. The resulting job will have a timeout of 2 hours. Now, suppose that the resulting job runs a subjob with no runtime policies at entry point 'foo'. The resulting subjob will have a timeout of 5 hours.

List of API Methods

Applet API Methods

The following are API methods specific to Applets.

Common data object API methods

The following are API methods methods common to all data objects and are defined where the methods are discussed.

Job API Methods

Applet API Method Specifications

API method: /applet/new

Specification

Creates a new applet object with the given applet specification.

Links specified in "bundledDepends" contribute to the "links" array returned by a describe call and are always cloned together with the applet regardless of their visibility.

Note that an applet that is not written in a supported interpreted language can still be run if a separate file object is created containing the compiled code. It is suggested that the user mark this file object as "hidden", and create an applet object with a link to it in the "bundledDepends" and code which specifically runs the compiled binary. Note that the file object does not need to be hidden for it to be bundled with the applet object, but for this purpose, it may be more useful to hide the file object if it is unlikely to be used in isolation. See the Execution Environment Reference for more details on how the contents of "bundledDepends" are handled.

The applet object does not receive special permissions for any referenced data objects (such as "id" in "bundledDepends"); these entries are accessed every time the applet is run, with the same permissions as the job. This means that if some referenced file is later deleted or is not present in the project context, the applet will not be able to run. However, the system will not automatically "invalidate" the applet object for any broken links; if the referenced file is reinstated from a copy existing in another project, the applet can then be run.

Inputs

  • project string ID of the project or container to which the applet should belong (e.g. the string "project-xxxx")
  • name string (optional, default is the new ID) The name of the object
  • title string (optional, default "") Title of the applet, e.g. "Micro Map"
  • summary string (optional, default "") A short description of the applet
  • description string (optional, default "") A longer description about the applet
  • developerNotes string (optional, default "") More detailed notes about the applet
  • tags array of strings (optional) Tags to associate with the object
  • types array of strings (optional) Types to associate with the object
  • hidden boolean (optional, default false) Whether the object should be hidden
  • properties mapping (optional) Properties to associate with the object
    • key Property name
    • value string Property value
  • details mapping or array (optional, default { }) JSON object or array that is to be associated with the object; see the Object Details section for details on valid input
  • folder string (optional, default "/") Full path of the folder that is to contain the new object
  • parents boolean (optional, default false) Whether all folders in the path provided in folder should be created if they do not exist
  • inputSpec array of mappings (optional) An input specification as described in the Input Specification section
  • outputSpec array of mappings (optional) An output specification as described in the Output Specification section
  • runSpec mapping A run specification as described in the Run Specification section
  • dxapi string The version of the API that the applet was developed with, for example, "1.0.0"
  • access mapping (optional) Access requirements as described in the Access Requirements section
  • nonce string (optional) Unique identifier for this request. Ensures that even if multiple requests fail and are retried, only a single applet is created. For more information, see Nonces.

Outputs

  • id string ID of the created applet object (i.e. a string in the form "applet-xxxx")

Errors

  • InvalidInput
    • A reserved linking string ("$dnanexus_link") appears as a key in a mapping in details but is not the only key in the hash.
    • A reserved linking string ("$dnanexus_link") appears as the only key in a hash in details but has value other than a string.
    • The spec is invalid.
    • All specified bundled dependencies must be in the same region as the specified project.
    • A nonce was reused in a request but some of the other inputs had changed signifying a new and different request
    • A nonce may not exceed 128 bytes
  • PermissionDenied
    • UPLOAD access is required to the specified project.
  • ResourceNotFound
    • The route in folder does not exist while parents is false.

API method: /applet-xxxx/describe

Specification

Describes an applet object.

Alternatively, you can use the /system/describeDataObjects method to describe a large number of data objects at once.

Inputs

  • project string (optional) Project or container ID to be used as a hint for finding the object in an accessible project
  • defaultFields boolean (optional, default false if fields is supplied, true otherwise) whether to include the default set of fields in the output (the default fields are described in the "Outputs" section below). The selections are overridden by any fields explicitly named in fields.
  • fields mapping (optional) include or exclude the specified fields from the output. These selections override the settings in defaultFields.
    • key Desired output field; see the "Outputs" section below for valid values here
    • value boolean whether to include the field

The following options are deprecated (and will not be respected if fields is present):

  • properties boolean (optional, default false) Whether the properties should be returned
  • details boolean (optional, default false) Whether the details should also be returned

Outputs

  • id string The object ID (i.e. the string "applet-xxxx")

The following fields are included by default (but can be disabled using fields or defaultFields):

  • project string ID of the project or container in which the object was found
  • class string The value "applet"
  • types array of strings Types associated with the object
  • created timestamp Time at which this object was created
  • state string Either "open" or "closed"
  • hidden boolean Whether the object is hidden or not
  • links array of strings The object IDs that are pointed to from this object, including links found in both the details and in bundledDepends (if it exists) of the applet
  • name string The name of the object
  • folder string The full path to the folder containing the object
  • sponsored boolean Whether the object is sponsored by DNAnexus
  • tags array of strings Tags associated with the object
  • modified timestamp Time at which the user-provided metadata of the object was last modified
  • createdBy mapping How the object was created

    • user string ID of the user who created the object or launched an execution which created the object
    • job string present if a job created the object ID of the job that created the object
    • executable string present if a job created the object ID of the app or applet that the job was running
  • runSpec mapping The run specification of the applet but without the code field (use the /applet-xxxx/get method for obtaining the source code)

  • dxapi string The version of the API used

  • access mappings The access requirements of the applet

  • title string The title of the applet

  • summary string The summary of the applet

  • description string The description of the applet

  • developerNotes string The developer notes of the applet

The following field (included by default) is available if an input specification is specified for the applet:

  • inputSpec array of mappings The input specification of the applet

The following field (included by default) is available if an output specification is specified for the applet:

  • outputSpec array of mappings The output specification of the applet

The following field (included by default) is available if the object is sponsored by a third party:

  • sponsoredUntil timestamp Indicates the expiration time of data sponsorship (this field is only set if the object is currently sponsored, and if set, the specified time is always in the future)

The following fields are only returned if the corresponding field in the fields input is set to true:

  • properties mapping Properties associated with the object
    • key Property name
    • value string Property value
  • details mapping or array Contents of the object’s details

Errors

  • ResourceNotFound (the specified object does not exist or the specified project does not exist)
  • InvalidInput (the input is not a hash, project (if supplied) is not a string, or the value of properties (if supplied) is not a boolean)
  • PermissionDenied (VIEW access required for the project provided (if any), and VIEW access required for some project containing the specified object (not necessarily the same as the hint provided))

API method: /applet-xxxx/get

Specification

Returns the full specification of the applet, i.e. the same output as /applet-xxxx/describe but with the runSpec field left intact.

Inputs

  • None

Outputs

  • project string ID of the project or container in which the object was found
  • id string The object ID (i.e. the string "applet-xxxx")
  • class string The value "applet"
  • types array of strings Types associated with the object
  • created timestamp Time at which this object was created
  • state string Either "open" or "closed"
  • hidden boolean Whether the object is hidden or not
  • links array of strings The object IDs that are pointed to from this object, including links found in both the details and in bundledDepends (if it exists) of the applet
  • name string The name of the object
  • folder string The full path to the folder containing the object
  • sponsored boolean Whether the object is sponsored by DNAnexus
  • tags array of strings Tags associated with the object
  • modified timestamp Time at which the user-provided metadata of the object was last modified
  • createdBy mapping How the object was created

    • user string ID of the user who created the object or launched an execution which created the object
    • job string present if a job created the object ID of the job that created the object
    • executable string present if a job created the object ID of the app or applet that the job was running
  • runSpec mapping The run specification of the applet

  • dxapi string The version of the API used

  • access mappings The access requirements of the applet

  • title string The title of the applet

  • summary string The summary of the applet

  • description string The description of the applet

  • developerNotes string The developer notes of the applet

If an input specification is specified for the applet:

  • inputSpec array of mappings The input specification of the applet

If an output specification is specified for the applet:

  • outputSpec array of mappings The output specification of the applet

Errors

  • ResourceNotFound (the specified object does not exist)
  • PermissionDenied (VIEW access required)

API method: /applet-xxxx/run

Specification

Creates a new job which will execute the code of this applet. The default entry point for the applet’s interpreter (given in the runSpec.interpreter field of the applet spec) will be called.

InterpreterEntry point
bashmain() in top level scope with no args, if it exists. Also, $1 is set to "main"
python2.7Any function decorated with @dxpy.app_entry(func="main") with no args

If constraints on inputs are specified in the applet spec, and the given inputs do not satisfy those constraints at the time the API call is performed, or if the names of inputs given do not exactly match the inputs listed in the applet object, or if an input is omitted and no default is listed in the applet object, an InvalidInput error will result. For inputs given as job-based object references, an equivalent error may result at job dispatch time, in which case the job will fail.

The job might fail for the following reasons (this list is non-exhaustive):

  • A reference such as one mentioned in "bundledDepends" could not be accessed using the job’s credentials (VIEW access to project context, CONTRIBUTE access to a workspace, VIEW access to public projects)
  • A job-based object reference did not resolve successfully (invalid job, job ID not found, job not in that project, job is in failed state, field does not exist, field does not contain a valid object link).
  • An input object does not exist.
  • Permission denied accessing an input object.
  • An input object is not a data object (things like users, projects, or jobs are not data objects)
  • An input object does not satisfy the class constraints.
  • An input object does not satisfy the type constraints.
  • An input object is not in the "closed" state.
  • Insufficient credits.
  • The user has too many jobs that are in a nonterminal state.

Inputs

  • name name (optional, default is the applet's title if set and otherwise the applet's name) Name for the resulting job
  • input mapping Input that the applet is launched with
    • key Input field name. If the applet has an input specification, it must be one of the names of the inputs; otherwise, it can be any valid input field name.
    • value Input field value
  • dependsOn array of strings (optional) List of job, analysis and/or data object IDs; the applet will not begin running any of its stages until all jobs listed have transitioned to the "done" state, and all data objects listed are in the "closed" state
  • project string (required if invoked by a user; prohibited if invoked by a job) The ID of the project in which this applet will be run (i.e., the project context). If invoked by a job, then the project context is inherited from that of the invoking job. If project is specified, then all output objects will be cloned into the project context; otherwise, all output objects will be cloned into the temporary workspace of the invoking job. See The Project Context and Temporary Workspace for more information.
  • folder string (optional, default "/") The folder into which objects output by the job will be placed. If the folder does not exist when the job completes, the folder (and any parent folders necessary) will be created. The folder structure that output objects reside in is replicated within the target folder, e.g. if folder is set to "/myJobOutput" and the job outputs an object which is in the folder "/mappings/mouse" in the workspace, the object is placed into "/myJobOutput/mappings/mouse".
  • tags array of strings (optional) Tags to associate with the resulting job
  • properties mapping (optional) Properties to associate with the resulting job
    • key Property name
    • value string Property value
  • details mapping or array (optional, default { }) JSON object or array that is to be associated with the job
  • systemRequirements mapping (optional) Request specific resources for each of the executable's stages; see the Requesting Instance Types section above for more details
  • executionPolicy mapping (optional) A collection of options that govern automatic job restart upon certain types of failures. The format of this field is identical to that of the executionPolicy field in the Run Specification supplied to /applet/new and can override part or all of the executionPolicy found in the applet's run specification (if present).
  • timeoutPolicyByExecutable mapping (optional) User-specified timeout policies for all jobs in the resulting job execution tree, configurable by executable. If unspecified, it indicates that all jobs in the resulting job execution tree will have the default timeout policies present in the run specifications of their executables. If present, includes at least one of the following key-value pairs:
    • key Executable ID. If an executable is not explicitly specified in timeoutPolicyByExecutable, then any job in the resulting job execution tree that runs that executable will have the default timeout policy present in the run specification of that executable.
    • value mapping or null Timeout policy for the corresponding executable. A value of null overrides the default timeout policy present in the run specification of the corresponding executable and indicates that no job in the resulting job execution tree that runs the corresponding executable will have a timeout policy. If a mapping, includes at least one of the following key-value pairs:
      • key Entry point name or "*" to indicate all entry points not explicitly specified in this mapping. If an entry point name is not explicitly specified and "*" is not present, then any job in the resulting job execution tree that runs the corresponding executable at that entry point will have the default timeout policy present in the run specification of the corresponding executable.
      • value mapping or null Timeout for a job running the corresponding executable at the corresponding entry point. A value of null indicates that no job in the resulting job execution tree that runs the corresponding executable at the corresponding entry point will have a timeout. Includes at least one of the following key-value pairs:
        • key Unit of time; one of "days", "hours", or "minutes".
        • value number Amount of time for the corresponding time unit; must be nonnegative. The effective timeout is the sum of the units of time represented in this mapping. Note that setting the effective timeout to 0 is the same as specifying null for the corresponding executable at the corresponding entry point. Note that timeoutPolicyByExecutable (keyed by executable ID and then entry point name) will propagate down the entire job execution tree, and that explicitly specified upstream policies always take precedence.
  • delayWorkspaceDestruction boolean (optional, default false) Whether the temporary workspace created for the resulting job should be kept around for 3 days after the job either succeeds or fails
  • allowSSH array of strings (optional, default [ ]) Array of hostnames (translated into CIDR blocks) or hostmasks from which SSH access will be allowed to the user by the worker running this applet's job(s). See Connecting to Jobs for more information.
  • debug mapping (optional, default { }) Specify debugging options for running the executable; this field is only accepted when this call is made by a user (and not a job)
    • debugOn array of strings (optional, default [ ]) Array of job errors after which the job's worker should be kept running for debugging purposes, offering a chance to SSH into the worker before worker termination (assuming SSH has been enabled). This option applies to all jobs in the execution tree. Jobs in this state for longer than 2 days will be automatically terminated but can be terminated earlier. Allowed entries include "ExecutionError", "AppError", and "AppInternalError".
  • nonce string (optional) Unique identifier for this request. Ensures that even if multiple requests fail and are retried, only a single job is created. For more information, see Nonces.

Outputs

  • id string ID of the created job (i.e. a string in the form "job-xxxx")

Errors

  • ResourceNotFound
    • The specified applet object or project context does not exist.
    • One of the IDs listed in dependsOn does not exist.
  • PermissionDenied
    • The requesting user must have VIEW access to all objects listed in dependsOn, and to all project contexts of all jobs listed in dependsOn.
    • The requesting user must have VIEW access to the applet object.
    • If invoked by a user, then the requesting user must have CONTRIBUTE access to the project context.
    • The requesting user must be able to describe all jobs used in a job-based object reference -- see /job-xxxx/describe.
    • The requesting user has too many (65536, by default) nonterminal (e.g., running, runnable) jobs and must wait for some to finish before creating more.
  • InvalidInput

    • input does not satisfy the input specification of this applet; an additional field is provided in the error JSON for this error that looks like

      { "error": { "type": "InvalidInput",
                   "message": "i/o value for fieldname is not int",
                   "details": { "field": "fieldname",
                                "reason": "class",
                                "expected": "int"
                              }
                 }
      }
      

      The possible error reasons are described in Input Specification Errors.

    • If invoked by a user, then project must be specified.

    • If invoked by a job, then project must not be specified.

    • The project context must be in the same region as this applet.

    • All data object inputs that are specified directly must be in the same region as this applet.

    • All inputs that are job-based object references must refer to a job that was run in the same region as this applet.

    • A nonce was reused in a request but some of the other inputs had changed signifying a new and different request

    • A nonce may not exceed 128 bytes

  • InvalidState

    • Some specified input is not in the "closed" state.
    • Some job in dependsOn has failed or has been terminated.
Input Specification Errors

The following list describes the possible error reasons and what the fields mean:

  • "class": the specified "field" was expected to have class "expected". If the input spec required an array but it was not an array, the value for "expected" will be "array". If the input spec required an array but an element was of the wrong class, then the value for "expected" will be the actual class the entry was expected to be, e.g. "record".
  • "type": the specified "field" either needs to have the type "expected" or does not satisfy the or-condition in "expected"
  • "missing": the specified "field" was not provided but is required in the input specification
  • "unrecognized": the given "field" is not present in the input specification
  • "malformedLink": incorrect syntax was given either for a job-based object reference or for a link to a data object. Possible values for "expected" include:
    • "key "field"": the key "field" was missing in a job-based object reference
    • "only two keys": exactly two keys were expected in the hash for the job-based object reference
    • "key "$dnanexus_link"": the key "$dnanexus_link" was missing in a link for specifying a data object
  • "choices": the specified "field" must be one of the values in "expected"

Job API Method Specifications

API method: /job/new

Specification

This API call may only be made from within an executing job. This call creates a new job which will execute a particular function (from the same applet as the one the current job is running) with a particular input. The input will be checked for links, and any job-based object references will be honored. However, the input is not checked against the applet spec. Since this is done from inside another job, the new job will inherit the same workspace and project context -- no objects will be cloned, and no other modification will take place in the workspace.

The entry point for the job’s execution will be determined as follows, where f is the string given in the function parameter in the input:

InterpreterEntry point
bashf() in top level scope with no args, if it exists. Also, $1 is set to f
python2.7Any function decorated with @dxpy.entry_point("f")

See Execution Environment Reference for more info.

This call will fail if the specified OAuth2 token does not internally represent a currently executing job.

The system takes note of the job that this new job is created from; this information is available in the "parent" field when describing the new job object. Moreover, the parent-child relationship is tracked by the system, and used to advance the job state of the parent. Specifically, a job which has finished executing remains in the "waiting_on_output" state until all of its child jobs have proceeded to the "done" state. See Job Lifecycle for more information.

The new job may fail for at least the following reasons:

  • A reference such as one mentioned in "bundledDepends" could not be accessed using the job’s credentials (VIEW access to project context, CONTRIBUTE access to workspace, VIEW access to public projects)
  • A job-based object reference did not resolve successfully (invalid job, job ID not found, job not in that project, job is in failed state, field does not exist, field does not contain a valid object link).
  • Insufficient credits.

Inputs

  • name string (optional, default "<parent job's name>:<function>") Name for the resulting job
  • input mapping Input that the job is launched with; no syntax checking occurs, but the mapping will be checked for links and create dependencies on any open data objects or unfinished jobs accordingly
    • key Input field name
    • value Input field value
  • dependsOn array of strings (optional) List of job, analysis and/or data object IDs; the applet will not begin running any of its stages until all jobs listed have transitioned to the "done" state, and all data objects listed are in the "closed" state
  • function string The name of the entry point or function of the applet’s code that will be executed
  • tags array of strings (optional) Tags to associate with the resulting job
  • properties mapping (optional) Properties to associate with the resulting job
    • key Property name
    • value string Property value
  • details mapping or array (optional, default { }) JSON object or array that is to be associated with the job
  • systemRequirements mapping (optional) Request specific resources for each of the executable's stages; see the Requesting Instance Types section above for more details
  • timeoutPolicyByExecutable mapping (optional) Similar to the timeoutPolicyByExecutable field supplied to /applet-xxxx/run
  • nonce string (optional) Unique identifier for this request. Ensures that even if multiple requests fail and are retried, only a single job is created. For more information, see Nonces.

Outputs

  • id string ID of the created job (i.e. a string in the form "job-xxxx").

Errors

  • InvalidAuthentication (the usual reasons InvalidAuthentication is thrown, or the auth token used is not a token issued to a job)
  • ResourceNotFound (one of the IDs listed in dependsOn does not exist)
  • PermissionDenied (VIEW access required for any objects listed in dependsOn and for the project contexts of any jobs listed in dependsOn; ability to describe any job used in a job-based object reference -- see /job-xxxx/describe)
  • InvalidInput
    • The input is not a hash
    • input is missing or is not a hash
    • an invalid link syntax appears in the input
    • dependsOn, if given, is not an array of strings
    • "details", if given, is not a conformant JSON object
    • For each property key-value pair, the size, encoded in UTF-8, of the property key may not exceed 100 bytes and the property value may not exceed 700 bytes
    • A nonce was reused in a request but some of the other inputs had changed signifying a new and different request
    • A nonce may not exceed 128 bytes
  • InvalidState
    • some job in dependsOn has already failed or been terminated

API method: /job-xxxx/describe

Specification

Describes a job object. Each job is the result of either running an applet by calling "run" on an applet or app object, or the result of launching a job from an existing job by calling the "new" job class method. In all cases, the job is associated with either an applet or an app, and a project context reflected in the "project" field. Moreover, the ID of the user who launched the execution that this job is a part of is reflected in the "launchedBy" field (and is propagated to all child jobs as well, including any executables launched from those jobs). A job is always in a particular state; for information about job states, see the Job Lifecycle section.

Inputs

  • fields mapping (optional) Restrict the output of this method to have only the provided keys in this field
    • key Desired output field; see the "Outputs" section below for valid values here
    • value boolean The value true

The following option is deprecated (and will not be respected if fields is present):

  • io boolean (optional, default true) Whether the input and output fields (runInput, originalInput, input, and output) should be returned

Outputs

  • id string The job ID (i.e. the string "job-xxxx")

The following fields are included by default (but can be disabled using fields):

  • class string The value "job"
  • name string The name of the job
  • executableName string The name of the executable (applet or app) that the job was created to run
  • created timestamp Time at which this job was created
  • modified timestamp Time at which this job was last updated
  • startedRunning timestamp (present once the transition has occurred) Time at which this object transitioned into the "running" state (see Job Lifecycle)
  • stoppedRunning timestamp (present once the transition has occurred) Time at which this object transitioned out of the "running" state (see Job Lifecycle)
  • billTo string ID of the account to which any costs associated with this job will be billed
  • project string The project context associated with this job
  • folder string The output folder in which the outputs of this job’s master job will be placed
  • rootExecution string ID of the job or analysis at the root of the execution tree (the job or analysis created by a user's API call rather than called by a job or as a stage in an analysis)
  • parentJob string or null ID of the job which created this job, or null if this job is an origin job
  • originJob string The closest ancestor job whose parentJob is null, either because it was run by a user directly or was run as a stage in an analysis
  • parentAnalysis string or null If this is an origin job that was run as a stage in an analysis, then this is the ID of that analysis; otherwise, it is null
  • analysis string or null Null if this job was not run as part of a stage in an analysis; otherwise, the ID of the analysis this job is part of
  • stage string or null Null if this job was not run as part of a stage in an analysis; otherwise, the ID of the stage this job is part of
  • state string The job state: one of "idle", "waiting_on_input", "runnable", "running", "waiting_on_output", "done", "debug_hold", "restartable", "failed", "terminating", and "terminated"); see Job Lifecycle for more details on job states
  • stateTransitions array of mappings Each element in the list indicates a time at which the state of the job changed; the initial state of a job is always "idle" when it is created and is not included in the list. Each hash has the key/values:
    • newState string The new state, e.g. "runnable"
    • setAt timestamp Time at which the new state was set for the job
  • workspace string (present once the workspace has been allocated) ID of the temporary workspace assigned to the job (e.g. "container-xxxx")
  • launchedBy string ID of the user who launched the original job
  • function string Name of the function, or entry point, that this job is running
  • tags array of strings Tags associated with the job
  • properties mapping Properties associated with the job
    • key Property name
    • value string Property value
  • details mapping or array JSON details that were stored with this job
  • systemRequirements mapping Specific resources that were requested for each of the executable's stages; see the Requesting Instance Types section above for more details
  • executionPolicy mapping Options specified for this job to perform automatic restart upon certain types of failures. The format of this field is identical to that of the executionPolicy field in the Run Specification supplied to /applet/new. This hash is computed from the following sources, in order of precedence:
  • timeout int or undefined The effective timeout, in milliseconds, for this job. Undefined indicates that no timeout policy is specified for the executable that this job is running or the effective timeout resolved to 0ms.
  • instanceType string Instance type for this job, computed from the systemRequirements specification or the system-wide default if no instance type was requested.
  • networkAccess array of strings The computed network access list that is available to the job; this may be a subset of the requested list in the access requirements of the executable if an ancestor job had access to a more restricted list of domains.
  • delayWorkspaceDestruction boolean (present for origin and master jobs only) Whether the job's temporary workspace will be kept around for 3 days after the job either succeeds or fails
  • dependsOn array of strings If the job is in a waiting state ("waiting_on_input" or "waiting_on_output"), an array of job IDs and data object IDs that must transition to the done or closed state for the job to transition out of the waiting state to either "runnable" or "done"
  • failureReason string (present if the job is in a failing or failed state) A short string describing where the error occurred, e.g. "AppError" for errors thrown in the execution of the applet or app
  • failureMessage string (present if the job is in a failing or failed state) A more detailed message describing why the error occurred
  • failureFrom mapping (present if the job is in a failing or failed state) Metadata describing the job which caused the failure of this job (which may be the same job as the one being described):
    • id string ID of the failed job
    • name string name of the job
    • executable string ID (of the form "applet-xxxx" or "app-xxxx") of the executable the job was running
    • executableName string Name of the executable the job was running
    • function string name of the function, or entry point, the job was running
    • failureReason string failureReason of the failed job
    • failureMessage string failureMessage of the failed job
  • failureReports array of mappings (present if this job failure was reported through a support mechanism) Each item in the list has the following fields:
    • to string Email address the failure was reported to
    • by string ID of the user who reported the failure
    • at timestamp Time at which the report was made
  • runInput mapping The input field that was provided when launching this job
  • originalInput mapping The same as runInput but with default values filled in for any optional inputs that were not provided
  • input mapping The same as originalInput, except that if any job-based references have since been resolved, they are replaced with the resulting object IDs; once the job's state has transitioned to "runnable", this represents exactly the input that will be given to the job
  • output mapping or null If the job is in the "done" state, this is the output that this job has generated as output. Otherwise, this value is null. The output may contain unresolved job-based object references.
  • region string The region (e.g., "aws:us-east-1") in which the job is running.

The following fields (included by default) are only available if the requesting user has permissions to view the pricing model of the billTo of the job, the job is an origin job, and the job's price has been computed:

  • isFree boolean Whether this job will be charged to the billTo (usually set to true if the job has failed for failure reasons which are usually indicative of some system error rather than user error)
  • totalPrice number Price (in dollars) for how much this job (along with all its subjobs) costs (or would cost if isFree is true)
  • priceComputedAt timestamp Time at which totalPrice was computed. For billing purposes, the cost of the job accrues to the invoice of the month that contains priceComputedAt (in UTC).

The following fields (included by default) are only available if the requesting user has permissions to view debug options for the job (either the user launched the job, or the user has ADMINISTER permissions in the job's project context; see Connecting to Jobs for more information):

  • debugOn array of strings Array of error types (e.g. AppError) upon which the job will be held for debugging

The following fields (included by default) are only available if the requesting user has permissions to view debug options for the job (either the user launched the job, or the user has ADMINISTER permissions in the job's project context; see Connecting to Jobs for more information), and the job has been configured to allow SSH access:

  • allowSSH array of strings Array of hostnames (translated into CIDR blocks) or hostmasks from which SSH access will be allowed to the user by the worker running this job

The following fields (included by default) are only available if the requesting user has permissions to view debug options for the job (either the user launched the job, or the user has ADMINISTER permissions in the job's project context; see Connecting to Jobs for more information), the job has been configured to allow SSH access, and a worker has started executing the job:

  • sshHostKey string The worker's SSH host key
  • host string The worker's public FQDN or IP address
  • sshPort string TCP port that can be used to connect to the SSH daemon for monitoring or debugging the job

The following fields (included by default) are only available if this job is running an applet:

  • applet string ID of the applet from which the job was run

The following fields (included by default) are only available if this job is running an app:

  • app string ID of the app from which the job was run
  • resources string ID of the app’s resources container
  • projectCache string ID of the project cache

The following field is only returned if the corresponding field in the fields input is set to true, the requesting user has permissions to view the pricing model of the billTo of the job, and the job is a root execution:

  • subtotalPriceInfo mapping Information about the current costs associated with all jobs in the tree rooted at this job
    • subtotalPrice number Current cost (in dollars) of the job tree rooted at this job
    • priceComputedAt timestamp Time at which subtotalPrice was computed

Errors

  • ResourceNotFound (the specified object does not exist)
  • PermissionDenied (require either VIEW access to the job's temporary workspace, or VIEW access to the parent job's temporary workspace)
  • InvalidInput (input is not a hash, or fields if present, is not a hash or has a non-boolean key)

API method: /job-xxxx/update

Specification

Update a job. Note that most runtime options for jobs are immutable for reproducibility reasons.

Inputs

  • allowSSH array of strings (optional, default [ ]) Array of hostnames (translated into CIDR blocks) or hostmasks from which SSH access will be allowed to the user by the worker running this job. See Connecting to Jobs for more information. Changing this value after a job has started running may not take be taken into account for the running job, but it may be passed on to new child jobs created after the update.

Outputs

None

Errors

  • ResourceNotFound (the specified job does not exist)
  • PermissionDenied (the supplied token does not belong to the user who started the job or who has ADMINISTER access to the job's parent project)
  • InvalidInput (input is not a hash)

API method: /job-xxxx/addTags

Specification

Adds the specified tags to the specified job. If any of the tags are already present, no action is taken for those tags.

Inputs

  • tags array of strings Tags to be added

Outputs

  • id string ID of the manipulated job

Errors

  • InvalidInput (the input is not a hash, the key tags is missing, or its value is not an array, or the array contains at least one invalid (not a string of nonzero length) tag)
  • ResourceNotFound (the specified job does not exist)
  • PermissionDenied (CONTRIBUTE access is required for the job's project context; otherwise, the request can also be made by jobs sharing the same workspace as the specified job or the same workspace as the parent job of the specified job)

API method: /job-xxxx/removeTags

Specification

Removes the specified tags from the specified job. Ensures that the specified tags are not part of the job -- if any of the tags are already missing, no action is taken for those tags.

Inputs

  • tags array of strings Tags to be removed

Outputs

  • id string ID of the manipulated job

Errors

  • InvalidInput (the input is not a hash, or the key tags is missing, or its value is not an array, or the array contains at least one invalid (not a string of nonzero length) tag)
  • ResourceNotFound (the specified job does not exist)
  • PermissionDenied (CONTRIBUTE access is required for the job's project context; otherwise, the request can also be made by jobs sharing the same workspace as the specified job or the same workspace as the parent job of the specified job)

API method: /job-xxxx/setProperties

Specification

Sets properties on the specified job. To remove a property altogether, its value needs to be set to the JSON null (instead of a string). This call updates the properties of the job by merging any old (previously existing) ones with what is provided in the input, the newer ones taking precedence when the same key appears in the old.

Best practices: to completely "reset" properties (i.e. remove all existing key/value pairs and replace them with some new), issue a describe call to get the names of all properties, then issue a setProperties request to set the values of those properties to null.

Inputs

  • properties mapping Properties to modify
    • key Name of property to modify
    • value string or null Either a new string value for the property, or null to unset the property

Outputs

  • id string ID of the manipulated job

Errors

  • InvalidInput (the input is not a hash, properties is missing or is not a hash, or there exists at least one value in properties which is neither a string nor the JSON null)
  • ResourceNotFound (the specified job does not exist)
  • PermissionDenied (CONTRIBUTE access is required for the job's project context; otherwise, the request can also be made by jobs sharing the same workspace as the specified job or the same workspace as the parent job of the specified job)

API method: /job-xxxx/getLog

Specification

Provides a persistent WebSocket connection to monitor messages produced by jobs in real time, as well as get all logged messages after the jobs complete. Unlike other DNAnexus API methods, which use HTTP, this method uses the SockJS API on top of the WebSocket protocol. The raw WebSocket URL is /job-xxxx/getLog/websocket.

Jobs running on the platform can produce messages either by printing to their standard output and standard error, or by using a syslog-like socket, available in the Execution Environment under /opt/dnanexus/log/priority and /opt/dnanexus/log/bulk. DNAnexus language bindings have functions to make this easier. See the Execution Environment Reference for more info.

Log levels

The following levels are defined for job logs: "STDERR", "STDOUT", "EMERG", "ALERT", "CRITICAL", "ERROR", "WARNING", "NOTICE", "INFO", "DEBUG".

Protocol notes

In a browser environment, the SockJS client library can be used. Otherwise, an RFC 6455 compliant WebSocket client should be used to open the URL /job-xxxx/getLog/websocket.

Upon WebSocket handshake completion, the client should send the JSON hash with contents specified in Inputs below. The server will then send log messages formatted with the format given in Outputs below. The server will keep the socket open until all the jobs complete (or fail) and all the messages are transmitted, then close it with the WebSocket status code and reason listed in Errors below.

Inputs

  • access_token: The user's OAuth2 authentication token. See Authentication.
    • Note: Unlike other DNAnexus API methods, this method uses the body of the message for the token, not the Authorization header.
  • token_type: Must be set to the string "Bearer".
  • numRecentMessages: The number of previously stored messages to be streamed (optional; default is 10). If the job has completed, this is the number of messages that will be returned; otherwise, the server will continue to stream messages until the job is complete.
  • recurseJobs: Whether or not to return logs for all child jobs in addition to those of the given job (optional; default is false).
  • tail: Whether or not to wait for jobs to finish running (optional; default is true).
  • levels: An array of strings from the Log levels list above. Only messages with these levels will be returned. Optional; by default all levels are returned.

Outputs

Each log message is a JSON hash with the following fields:

  • source Can be set to APP or SYSTEM
  • msg Body of the message (string)
  • timestamp Timestamp of the message (integer)
  • level Logging level of the message (string). Only set when source is APP.
  • job ID of the job that produced the message (string). Only set when source is APP.
  • line Unique sequential counter identifying the message (integer). Only set when source is APP. (TODO: bulk/priority separation)

Note: When using the SockJS client library, the serialized hash above is given to the onmessage handler in the data field (e.data in the example).

System messages

When the source field of the message is set to SYSTEM, the client should examine msg and take appropriate action.

System message type Content of msg Notes
End of logs END_LOG After this message, no more messages will be sent by the server, and the client should close the connection (normally done by calling close() on the connection object).

Errors

If an error occurs, the server will close the socket with an error code defined by the WebSocket protocol.

WebSocket Code Meaning
1000 Normal closure
1008 Policy violation

If the code is 1008, the close reason contains a JSON hash formatted as described in Errors:

  • PermissionDenied (the user must have at least VIEW access to the project in which the job was run)
  • InvalidInput (Input is not a hash, numRecentMessages is not a non-negative integer, recurseJobs is not a boolean value)

API method: /job-xxxx/terminate

Specification

Terminates a job and its job tree. If the job is already in a terminal state such as "terminated", "failed", or "done", no action will be taken. Otherwise, all jobs left in the job tree that have not reached a terminal state will eventually be put into the "terminated" state with failure reason "Terminated". Any authentication tokens generated for this execution will be invalidated, and any running jobs will be stopped. See Job Lifecycle for more details on job states.

Jobs can only be terminated by the user who launched the job or by any user with ADMINISTER access to the project context.

Inputs

  • None

Outputs

  • id string ID of the terminated job (i.e., the string "job-xxxx")

Errors

  • ResourceNotFound (the specified object does not exist)
  • PermissionDenied
    • Either (1) the user must match the "launchedBy" entry of the job object, and CONTRIBUTE access is required to the project context of the job, or (2) ADMINISTER access is required to the project context of the job

Last edited by Phil Sung, 2017-04-09 03:23:25

 Feedback