Developer Tutorials/Asset Build Process

This document shows you how to get started building an asset bundle and describes the structure and format of an asset source directory. Please refer to the page Asset Bundle document for an introduction if you merely wish to use an asset bundle in your applet/app.

Assets can target either of the supported Ubuntu releases: 12.04 or 14.04.

Asset Directory Structure

A DNAnexus asset can be built using the dx build_asset utility in the DNAnexus SDK, which reads an asset configuration, builds it in an isolated environment, and creates an asset bundle object. The build_asset utility expects a directory with a specific structure, which looks like this:

MyAssetDirectory
├── dxasset.json
├── resources
│   └── ...
└── Makefile

Each of these components is described in detail below.

dxasset.json (Asset metadata)

The file dxasset.json contains essential metadata that defines the asset. This file must be present in the directory provided as input to the build_asset utility. The following is a sample dxasset.json file:

{
  "name": "EMBOSSNeedleAsset",                # Asset bundle name

  "title": "EMBOSS Needle reads align Asset", # Human-readable name

  "description": "Libraries required for the EMBOSS Needle reads alignment tool",
                                              # A detailed description of the asset bundle and its
                                              # contents

  "version": "0.0.1",                         # Version number

  "distribution": "Ubuntu",                   # The flavor and version of Linux that the asset is
                                              # targeted to (default Ubuntu)

  "instanceType": "mem2_ssd1_x4",             # The instance type on which the asset bundle
                                              # will be built

  "release": "12.04",                         # The version of Linux flavor, Ubuntu 12.04 (default)
                                              # and 14.04 are supported

  "excludeResource": [                        # Files and directories that should not be included
    "/src/my.ccp",                            # in the asset bundle (optional)
    "/scripts"
  ],

  "execDepends": [                            # The list of packages that the asset depends on
    {"name": "samtools", "package_manager": "apt"},
    {"name": "bedtools", "package_manager": "apt"},
    {"name": "libfreetype6", "package_manager": "apt"},
    {"name": "libfreetype6-dev", "package_manager": "apt"},
    {"name": "emboss", "package_manager": "apt"},
    {"name": "cutadapt","package_manager": "pip"},
    {"name": "pandas","package_manager": "pip"},
    {"name": "cython","package_manager": "pip"},
    {"name": "matplotlib","package_manager": "pip"},
    {"name": "pybedtools","package_manager": "pip"},
    {"name": "pysam","package_manager": "pip"},
    {"name": "twobitreader","package_manager": "pip"},
    {"name": "xlwt","package_manager": "pip"},
    {"name": "biopython","package_manager": "pip"}
  ]

}

resources/ subdirectory

Anything that is present in the resources/ subdirectory of the asset directory will be archived by dx build_asset and then unpacked into the root of a clean Ubuntu installation during the asset build process. Files here can be propagated directly into the asset, or they can be used to build other artifacts that are themselves propagated into the asset.

For example, if you have some C or C++ source files that should be compiled so that their binary output can be included in the asset bundle, you can (1) put your source files somewhere in the resources/ directory, say, resources/mySrc/, (2) invoke your build process from the Makefile included in the asset source directory (described below). In the asset build environment, your source files are available in the directory /mySrc.

By default, the PATH in the execution environment is the following (it is inherited from Ubuntu's defaults):

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Makefile

The Makefile optionally contains instructions to build and copy additional resources required by the asset. If one appears in the asset source directory, it is copied to the working directory /home/dnanexus of the asset build environment and executed. Here is an example Makefile that downloads and installs various resources:

SHELL=/bin/bash -e
EA_UTILS="https://storage.googleapis.com/google-code-archive-source/v2/code.google.com/ea-utils/source-archive.zip"
EA_FILE="ea_utils.zip"

all:
	# Trust the signing key for this repo. Reference: http://cran.rstudio.com/bin/linux/ubuntu/README.html
	sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
	sudo rm -f /etc/apt/apt.conf.d/99dnanexus
	# install bioconductor and it's packages
	R -e "install.packages(c('ape', 'biom', 'optparse', 'RColorBrewer', 'randomForest', 'vegan'), repos='http://cran.rstudio.com/'); source('http://bioconductor.org/biocLite.R'); biocLite(); biocLite(c('DESeq2', 'metagenomeSeq'))"
	sudo pip install future click
	git clone https://github.com/pycogent/pycogent.git
	cd pycogent; python setup.py build; sudo python setup.py install; cd ..; sudo rm -rf pycogent
	sudo pip install qiime
	echo "Upgrading numexpr......."
	sudo pip install -U numexpr
	sudo apt-get --yes install libgsl0-dev
	echo "Downloading and installing ea-utils........."
	sudo curl -L $(EA_UTILS) > $(EA_FILE)
	sudo unzip $(EA_FILE)
	echo "Building and making ea-utils.........."
	sudo $(MAKE) -C ea-utils/trunk/clipper
	sudo $(MAKE) -C ea-utils/trunk/clipper install

Building an asset

You can use the following command to build your asset:

dx build_asset MyAssetDirectory

(assuming the directory MyAssetDirectory contains a dxasset.json file).

A new DNAnexus job is started on a virtualized worker in order to provide an isolated build environment in which the following steps are performed:

  • A snapshot is taken of the worker's filesystem. During this process all the directories, files and symbolic links are recorded, except the ones mentioned in the excludeResource field of the dxasset.json. Also, files from the following directories are excluded: /tmp, /proc, /boot, /sys, /home/dnanexus
  • Any packages mentioned in the execDepends of the dxasset.json are installed in the worker's execution environment.
  • If a resources/ directory is present in the asset source directory, its contents are copied to the root directory / of the worker's execution environment.
  • If a Makefile is present in the asset source directory, it is copied to the /home/dnanexus on the worker and executed as: sudo make -C /home/dnanexus
  • A second snapshot of the worker's execution environment is taken.
  • All new and changed files (i.e. files with different timestamp than the earlier step) are packaged in the resulting asset bundle.

Last edited by Andrew Carroll (acarroll), 2016-06-15 23:53:55

 Feedback