Projects

Projects are special containers intended for collaboration; access permissions are controlled by the users who have ADMINISTER access to the project.

Projects vs Containers

Users can create projects, either through the website or using the API method /project/new. Unless they are creating a project on behalf of an organization, they will be responsible for the storage costs of the data inside the project, as well as any compute time for any jobs run from the project. Projects have various metadata (name, description, tags, properties). The project can also have project-wide boolean flags that place restrictions on how the data can be modified or copied to other projects:

  • PROTECTED: If set to true, only members with "ADMINISTER" permission level can delete objects. Otherwise, members with "CONTRIBUTE" permission level can also delete objects from the project. To aid in the reproducibility of results from running apps in the project, project caches are not persistent and are recreated for each time an app is run.
  • RESTRICTED: If set to true, the objects of this project cannot be cloned, and any running applet that reads from this project cannot write results to any other project (see later discussion for what this means, too).
  • DOWNLOAD RESTRICTED If set to true, data in this project cannot be downloaded outside of the platform.
  • containsPHI: If set to true, data in this project is considered to be protected health information (PHI) and will be subject to additional restrictions to ensure security and compliance, such as:
    • data in this project cannot be cloned to other projects that do not have containsPHI set to true
    • any jobs that run in this project will be specially isolated from other jobs in the cloud
    • any jobs that run in non-PHI projects will not be able to access any data that can only be found in PHI projects

In contrast, users cannot create generic data containers directly. Containers are generated automatically for different purposes involved in running an analysis, sharing data, etc. and will always be associated with a project or user/organization that will be responsible for its storage costs.

Project Metadata

Projects can have names, descriptions, properties, and tags. Properties may be automatically set by the website front-end and are used for facilitating user experience.

List of API Methods

Project API Methods

The following methods are specified on this page:

A related method:

Additional Project API Methods

The following methods are specified with the appropriate section.

For managing data in the project:

For administering access permissions on a project:

For finding projects accessible to a user or job:

Project API Method Specifications

API method: /project/new

Specification

Creates a new project. Initially only the user performing this action will be in the permissions/member list, with ADMINISTER access.

Inputs

  • name string The name of the project
  • summary string (optional, default "") A short summary of what the project contains
  • description string (optional, default "") A longer description of the project
  • protected boolean (optional, default false) Whether the project should be PROTECTED (see more details in the section Projects vs Containers)
  • restricted boolean (optional, default false) Whether the project should be RESTRICTED (see more details in the section Projects vs Containers)
  • downloadRestricted boolean (optional, default false) Whether data in this project can be downloaded outside of the platform.
  • containsPHI boolean (optional, default false) Whether the project should be marked as containing protected health information (PHI) (see more details in the section Projects vs Containers)
  • tags array of strings (optional) Tags to associate with the project
  • properties mapping (optional) Properties to associate with the project
    • key Property name
    • value string Property value
  • billTo string (optional, default is the billTo of the requesting user) ID of the entity to which any costs associated with this project will be billed; must be the ID of the requesting user or an org of which the requesting user is a member with allowBillableActivities permission
  • region string (optional; defaults to the defaultRegion of the billTo of the project) The region that this project will be created in. The region must be among the permitted regions of the project's billTo. For more information about regions, see Region.

Outputs

  • id string ID of the created project

Errors

  • InvalidInput
    • name may not be empty and may not match the regex [\x00-\x1F]
    • containsPHI may not be set to false if the project is currently marked as containing PHI
  • InvalidState
    • containsPHI is true, but the project is in a region that does not support PHI features
  • SpendingLimitExceeded
    • billTo has reached its spending limit
  • PermissionDenied
    • Must be logged in
    • One of the following must be supplied: a full scope token, a token with ADMINISTER access in all projects ('*'), or a token with the projectCreation scope set
    • If containsPHI was set to true, the billTo of the project must have PHI features enabled
    • billTo must be either the ID of the requesting user or the ID of an org in which the requesting user is a member with allowBillableActivities permission
    • The billTo of the invoking user is an org and the user is no longer a member of that org
    • The region that was selected was not among the permittedRegions of the billTo

API method: /project-xxxx/update

Specification

Updates the specified project with the specified options. Options that are not supplied in the input hash are not changed, e.g. if the project originally has the PROTECTED flag set and this method is called to update the title or description and does not include the "protected" field, the PROTECTED flag will remain unchanged. Only members with ADMINISTER access can update the project. Note that updating the billTo field specifically will reset both totalSponsoredEgressBytes and consumedSponsoredEgressBytes for the project.

Inputs

  • name string (optional) The new name of the project
  • summary string (optional) The new summary of the project
  • description string (optional) The new description of the project
  • protected boolean (optional) The new value of the PROTECTED property (see more details in the section Projects vs Containers)
  • restricted boolean (optional) The new value of the RESTRICTED property (see more details in the section Projects vs Containers)
  • downloadRestricted boolean (optional) If set to true, disables the downloading of data outside of the platform.
  • containsPHI boolean (optional) Set to true to mark the project as containing protected health information (PHI) (see more details in the section Projects vs Containers)
  • version int (optional) If provided, the value must match the current project's version; otherwise, the update will be applied unconditionally
  • billTo string (optional) ID of the entity to which any costs associated with this project will be billed; must be the ID of the requesting user or an org of which the requesting user is a member with allowBillableActivities permission

Outputs

  • id string ID of the manipulated project

Errors

  • ResourceNotFound
    • The specified project does not exist
  • InvalidInput
    • The input is not a hash
    • name (if provided) is not a nonempty string
    • summary (if provided) is not a string
    • description (if provided) is not a string
    • protected restricted, downloadRestrictedor containsPHI (if provided) are not booleans
    • containsPHI (if provided) cannot be false -- marking a project as no longer containing PHI is currently unsupported
    • containsPHI is true, but the project is in a region that does not support PHI features
  • InvalidState
    • version was provided and does not match the current version number found in the project
  • PermissionDenied
    • ADMINISTER access required
    • containsPHI was set to true, but the billTo of the project does not have PHI features enabled
    • billTo must be either the ID of the requesting user or the ID of an org of which the requesting user is a member with allowBillableActivities permission
    • billTo must have the region of this project in its permittedRegions
    • If billTo is specified and the project is currently billed to an org, then the requesting user must be a member of the org; depending on the restrictProjectTransfer org policy, the requesting user may be required to be an ADMIN of the org.
  • SpendingLimitExceeded
    • billTo has reached its spending limit

API method: /project-xxxx/updateSponsorship

Specification

This method modifies various forms of sponsorship in the project. It can be used to set or modify data egress sponsorship, or used to set or unset third-party data storage sponsorship in the project.

When data egress sponsorship has been set to a specified number of bytes, the billTo of the project may be billed for up to that amount of data transferred outside of the system for that project (i.e. downloaded by a user). The rate at which the data is billed may vary depending on the IP range of the downloader. If the amount of data egress from this project to outside the system exceeds the number of data egress sponsored bytes the downloader will then incur any remaining and future costs of downloading data from project.

When the project is sponsored for data storage, the billTo of the project is billed for any storage charges for all objects in this project. Any other users or orgs that have copies of any of these data objects in other projects can store them for free until the expiration of the sponsorship period.

When the project is sponsored, adding new data objects to the project (or removing them) causes those objects to be sponsored (unsponsored, respectively). An object may only be sponsored in one project at a time, and cloning an object into a sponsored project will fail if the object is already sponsored by virtue of its presence in another sponsored project.

If the project is already sponsored, you can call this method again to update the expiration time or to terminate sponsorship.

Inputs

  • sponsoredUntil timestamp or null (optional) Sets the expiration time of data storage sponsorship. Specifying null (or any time in the past) terminates the sponsorship effective immediately.
  • totalSponsoredEgressBytes int (optional) Sets the number of bytes to be sponsored for downloads (data egress) from the DNAnexus Platform to an external IP for this project. Users may download up to the specified number of bytes from the project at no cost to themselves, with the cost being borne by the project's billTo. If the new value is less than the number of bytes already consumed the sponsor only owes charges for downloads that have already completed and its liabilites end effective immediately. Note that setting this value overrides the previous value and does not increment it, so to effectively extend the byte limit this method must be invoked with a value higher than the previous value it was invoked with.

Outputs

  • id string ID of the project

Errors

  • InvalidInput
    • totalSponsoredEgressBytes must be a non-negative integer
  • InvalidState
    • A data object in the project has already been third-party sponsored in another project; the data object(s) need to be removed from this project (or from the other project) before they can be third-party sponsored again. When this error is thrown, the error details is of the form {alreadySponsored: [{id: ..., project: ...}, ...]} indicating the ID and sponsoring project of each object that was already third-party sponsored.
    • A data object in the project has already been DNAnexus-sponsored
    • The project is archived
    • The project has a pending transfer
  • PermissionDenied
    • If sponsoredUntil is supplied, the billTo of the project needs to have data sponsorship features enabled. Please contact us to enable data storage sponsorship features for your account.
    • ADMINISTER access to the project is required
    • The user's billTo must match the billTo of the project

API method: /project-xxxx/setProperties

Specification

Sets properties on the specified project. To remove a property altogether, its value needs to be set to the JSON null (instead of a string). This call updates the properties of the project by merging any old (previously existing) ones with what is provided in the input, the newer ones taking precedence when the same key appears in the old.

Best practices: to completely "reset" properties (i.e. remove all existing key/value pairs and replace them with some new), issue a describe call to get the names of all properties, then issue a setProperties request to set the values of those properties to null.

Inputs

  • properties mapping Properties to modify
    • key Name of property to modify
    • value string or null Either a new string value for the property, or null to unset the property

Outputs

  • id string ID of the manipulated project (i.e. a string of the form "project-xxxx")

Errors

  • InvalidInput (the input is not a hash, properties is missing or is not a hash, or there exists at least one value in properties which is neither a string nor the JSON null)
  • ResourceNotFound (the specified project does not exist)
  • PermissionDenied (CONTRIBUTE access required for the specified project)

API method: /project-xxxx/describe

Specification

Describes the specified project.

Inputs

  • fields mapping (optional) Restrict the output of this method to have only the provided keys in this field
    • key Desired output field; see the "Outputs" section below for valid values here
    • value boolean The value true

Outputs

  • id string ID of the project (i.e. the string "project-xxxx")

The following fields are included by default (but can be disabled using fields):

  • class string The value "project"
  • name string The name of the project
  • region string The region this project resides in. For more information about regions, see Regions.
  • summary string The summary of the project
  • description string The description of the project
  • version int The current version number of the project used for tracking changes to its metadata
  • tags array of strings Tags associated with the project
  • billTo string ID of the account to which any costs associated with this project will be billed
  • protected boolean The value of the PROTECTED flag
  • restricted boolean The value of the RESTRICTED flag
  • downloadRestriced boolean The value of the DOWNLOAD RESTRICTED flag
  • containsPHI boolean Whether the project is marked as containing protected health information (PHI) and will be subject to additional restrictions to ensure security and compliance
  • created timestamp Time at which this project was created
  • createdBy mapping How the project was created
    • user string ID of the user who created the project
    • job string (present if applicable) ID of the job that created the project
    • executable string (present if job is present) ID of the applet or app that the job was running
  • modified timestamp Time at which this project was last updated
  • level string The highest permission level that the requesting user has
  • dataUsage number Data usage in GB (not including sponsored data). A short amount of time may elapse between changes to the project and when this number is updated.
  • defaultInstanceType string Default instance type for apps to run on
  • sponsoredDataUsage number sum of DNAnexus-sponsored and third-party sponsored data usage in GB. If the current project is itself third-party sponsored, then objects in that project will be reflected in dataUsage rather than sponsoredDataUsage since the billTo of the project does get billed for them. A short amount of time may elapse between changes to the container and when this number is updated.
  • sponsoredUntil timestamp If present, indicates that the project is sponsored by its billTo until the specified time (this field is only set if the project is currently sponsored, and if set, the specified time is always in the future)
  • pendingTransfer string or null If applicable, account ID that has been invited to take the project; otherwise null
  • totalSponsoredEgressBytes int the total number of bytes sponsored for this project over the lifetime of the project's current billTo. If the current value is less than the number of bytes already consumed, consumedSponsoredEgressBytes, the sponsor is not billed for any future downloads and the cost is billed to the downloader.
  • consumedSponsoredEgressBytes int the number of sponsored bytes already used to transfer files from this project out of the DNAnexus platform, over the lifetime of the project's current billTo.

The following fields (included by default) are available if the requesting user has UPLOAD+ access to the project:

  • atSpendingLimit boolean Indicates whether the billTo of the project is at their spending limit (and therefore functionality is restricted)

The following fields (included by default) are available if the requesting user has permissions to view the pricing model of the billTo of the project:

  • storageCost number Storage cost (in dollars) of the project per month given the current data usage (note that the total cost of an account's projects may be smaller than the sum of this value over all projects if any objects are shared between the projects)

The following fields are only returned if the corresponding field in the fields input is set to true:

  • folders array of strings List of all folders in the project
  • objects int The number of visible objects in the project
  • permissions mapping Who has permissions to access this project
    • key Entity ID; a user or org ID
    • value string One of the values "VIEW", "UPLOAD", "CONTRIBUTE", "ADMINISTER"
  • appCaches mapping List of containers attached to the project that have been created for running apps
    • key App ID
    • value string ID of the app ID's project cache container
  • properties mapping Properties associated with the project
    • key Property name
    • value string Property value
  • fileUploadParameters mapping Information about what part sizes and numbers should be used to upload files in this project. See the section Limits on Parts of the Files API for more information about interpreting this. Mapping with the key/values:
    • minimumPartSize int minimum part size, in bytes, that applies to all parts except the part with the highest index. (Clients may assume that if emptyLastPartAllowed is false, then minimumPartSize will be at least 1, that is, the constraint on the last part is no stronger than the constraint on previous parts.)
    • maximumPartSize int maximum part size, in bytes
    • emptyLastPartAllowed boolean If true, then the the minimum number of parts is 1 and the part with the highest index may contain 0 bytes. If false, then the minimum number of parts is 0 and the part with the highest index must contain at least 1 byte. Note that all parts other than the part with the highest index must still have minimum size given by minimumPartSize. (If true, then the client can upload a 0-byte file by invoking /file-xxxx/upload once with a part of size 0. If false, then the client can upload a 0-byte file by not invoking /file-xxxx/upload at all.)
    • maximumNumParts int the maximum number of parts that may be uploaded (also equal to the largest permissible part index)
    • maximumFileSize int the maximum size of the file, in bytes
  • availableInstanceTypes mapping Instances available in the project for apps and applets to run on
    • key Instance type
    • value mapping Contains information on the instance type
      • numCores int Number of cores
      • totalMemoryMB int Memory size in MB
      • ephemeralStorageGB int Storage size in GB
      • os array of mappings Supported operating systems. Each value is a mapping with the following fields, indicating one of the supported operating systems:
        • distribution string The Linux operating system distribution; must match the distribution of the run specifications of the app or the applet to be run
        • release string The Linux operating system release; must match the release of the run specifications of the app or the applet to be run

Errors

  • ResourceNotFound (the specified project does not exist)
  • InvalidInput (input is not a hash, or fields if present, is not a hash or has a non-boolean key)
  • PermissionDenied
    • Must have at least VIEW access to the project, be the billTo with a full scope token (if the billTo is a user), or be an admin of the billTo with a full scope token (if the billTo is an org)

API method: /project-xxxx/destroy

Specification

Destroys the specified project. All objects in the project are removed.

If there are any jobs in the project which are not in a terminal state (i.e. not done, failed, or terminated), then an InvalidState error will be thrown. Set the terminateJobs option to force termination of jobs.

Inputs

  • terminateJobs boolean (optional, default false) Whether to force the destruction of the project and terminate all non-terminal jobs

Outputs

  • id string ID of the manipulated project

Errors

  • ResourceNotFound (the specified project does not exist)
  • PermissionDenied (ADMINISTER access required)
  • InvalidInput (terminateJobs (if present) is not a boolean)
  • InvalidState
    • terminateJobs was not set to true, and there still remain jobs which are not in a terminal state
    • The project is sponsored (sponsorship must be terminated before the project may be destroyed)

API method: /project-xxxx/addTags

Specification

Tags the project with the given strings. There is no effect for tags that already exist for the project.

Inputs

  • tags array of strings Tags to be added

Outputs

  • id string ID of the manipulated project

Errors

  • ResourceNotFound (the specified project does not exist)
  • InvalidInput (tags is not present or is not a list of nonempty strings)
  • PermissionDenied (CONTRIBUTE access required)

API method: /project-xxxx/removeTags

Specification

Removes the given tags from the project. There is no effect if the tags do not exist.

Inputs

  • tags array of strings Tags to be removed

Outputs

  • id string ID of the manipulated project

Errors

  • ResourceNotFound (the specified project does not exist)
  • InvalidInput (tags is not present or is not a list of nonempty strings)
  • PermissionDenied (CONTRIBUTE access required)

API method: /project-xxxx/countRootExecutions

Specification

Counts the root executions in this project by state. Returns an object whose keys are execution states, and whose values are the number of root executions that are in that state.

Inputs

  • None

Outputs

  • key State
  • value int Number of root executions in that state

Errors

  • ResourceNotFound (the specified project does not exist)
  • PermissionDenied (VIEW access required)

API method: /system/getProjectTags

Specification

Returns the tags on all visible projects. Note: Initial implementation will only include public projects.

Inputs

  • None

Outputs

  • key Tag
  • value int Number of projects with the tag

Errors

Last edited by nicolasbockg, 2017-07-26 21:15:06

 Feedback