Upload Agent

Introduction

The DNAnexus Upload Agent is a fast and convenient command-line client that can be used to upload files to DNAnexus. For uploading multiple or large files, Upload Agent is particularly recommended due to its ability to resume previously interrupted uploads.

Installing Upload Agent: Follow the instructions on the Upload Agent download page to download the Upload Agent executable.

Note: For the rest of this document, we will use ua to represent the Upload Agent executable, but you should replace it with the path to where you have saved the Upload Agent executable on your local file system.

This page is a basic guide to Upload Agent. For more detailed information, see the Advanced Guide to Upload Agent.

Basic Usage

Synopsis

$ ./ua [options] [...]

Usage

In the examples below, we assume that you have appropriately set your environment variables. Specifically, we assume that the authentication token and current workspace (project) are set. Remember that you can always override these environment variables by using the --auth-token and --project command-line options.

To see the current environment variables being used by ua, run:

$ ./ua --env API server protocol: https API server host: api.dnanexus.com API server port: --- Auth token: <TOKEN> Current Project: my_project (project-xxxx)

Running a simple diagnostic test

Running the Upload Agent with the --test flag will run a simple test to verify that ua is correctly configured. The output of a successful configuration will look similar to the output below. Upload Agent will print any errors as part of the output.

$ ./ua --test Upload Agent Version: 1.5.19 git version: v0.201.0+g8dd2d2f libboost version: 1.55.0 libcurl version: 7.45.0 Upload Agent v1.5.19, environment info: API server protocol: https API server host: api.dnanexus.com API server port: 443 Current User: user-xxxx Current Project: (project-xxxx) Proxy Settings: No proxy set in environment. Operating System: Name: Linux Release: 4.4.0-59-generic Version: #80~14.04.1-Ubuntu SMP Machine: x86_64 CA Certificate: Resolving Amazon S3: Resolved to Testing connection: Sucessfully contacted google.com over http: (200) Sucessfully contacted google.com over https: (200)

Uploading a single file

You can upload a single file using the Upload Agent. In the following example, we upload a local file named my_file.txt to the project called "my_project".

$ ./ua --auth-token <TOKEN> --project my_project my_file.txt Uploading file my_file.txt to file object file-xxxx File "my_file.txt" was uploaded successfully. Closing… file-xxxx

Now, if the project's files are viewed from the web UI or via the command line using the dx ls command, there is a new file with the name my-file.txt.gz. Upload Agent automatically compressed the file my-file.txt and appended the .gz extension during upload.

Uploading multiple files to the same project

Upload Agent can upload multiple files to the same project. In the following example, we upload two local files, my_file_1.txt and my_file_2.txt, to the project "my_project".

$ ./ua --auth-token <TOKEN> --project my_project my_file_1.txt my_file_2.txt Uploading file my_file_1.txt to file object file-xxxx Uploading file my-file_2.txt to file object file-yyyy File "my_file_1.txt" was uploaded successfully. Closing... File "my_file_2.txt" was uploaded successfully. Closing... file-xxxx file-yyyy

File IDs output in the same command-line input order. In the above example, the first and second lines correspond to the new file IDs generated by uploading my_file_1.txt and my_file_2.txt, respectively.

Uploading directories

Upload Agent can upload all the files in a given directory. The destination of the files depends on the directory name given as input. If the name contains a trailing /, the Upload Agent doesn’t create the directory, it just copies the contents of the folder to the destination path in the platform.

$ ./ua --auth-token <TOKEN> dir_name/ Uploading file dir_name/test_file.txt to file object file-xxxx File "dir_name/test_file.txt" was uploaded successfully. Closing... file-xxxx

Without a trailing /, a new remote directory will be created and the files will be uploaded to the new directory (dir_name).

$ ./ua --auth-token <TOKEN> dir_name Uploading file dir_name/test_file.txt to file object file-xxxx File "dir_name/test_file.txt" was uploaded successfully. Closing... file-xxxx
WARNING ! You can upload at most 1000 files in a single operation.

Uploading directories recursively

Upload Agent can upload a directory recursively using the --recursive flag. The destination directory follows the same rules as above; that is, with a trailing /, ua will assume that the destination directory exists, and without the trailing /, Upload Agent will create a new directory, if the directory doesn’t exist.

$ ./ua --auth-token <TOKEN> dir_name --recursive Uploading file dir_name/first_file.txt to file object file-xxxx Uploading file dir_name/second_file.txt to file object file-yyyy Uploading file dir_name/log/log_file.txt to file object file-zzzz File "dir_name/first_file.txt" was uploaded successfully. Closing... File "dir_name/second_file.txt" was uploaded successfully. Closing... File "dir_name/log/log_file.txt" was uploaded successfully. Closing... file-xxxx file-yyyy file-zzzz

Help String

-h [ --help ] Produce a help message --version Print the version -e [ --env ] Print environment information -a [ --auth-token ] arg Specify the authentication token -p [ --project ] arg Name or ID of the destination project -f [ --folder ] arg (=/) Name of the destination folder -n [ --name ] arg Name of the remote file (Note: Extension ".gz" will be appended if the file is compressed before uploading) --visibility arg (=visible) Use "--visibility hidden" to set the file's visibility as hidden. --property arg Key-value pair to add as a property; repeat as necessary, e.g. "--property key1=val1 --property key2=val2" --type arg Type of the data object; repeat as necessary, e.g. "--type type1 --type type2" --tag arg Tag of the data object; repeat as necessary, e.g. "--tag tag1 --tag tag2" --details arg JSON to store as details --recursive Recursively upload the directories --read-threads arg (=2) Number of parallel disk read threads -c [ --compress-threads ] arg (=3) Number of parallel compression threads -u [ --upload-threads ] arg (=8) Number of parallel upload threads -s [ --chunk-size ] arg (=75M) Size of chunks in which the file should be uploaded. Specify an integer size in bytes or append optional units (B, K, M, G). E.g., '50M' sets chunk size to 50 megabytes. --throttle arg Limit maximum upload speed. Specify an integer to set speed in bytes/second or append optional units (B, K, M, G). E.g., '3M' limits upload speed to 3 megabytes/second. If not set, uploads are not throttled. -r [ --tries ] arg (=3) Number of tries to upload each chunk --do-not-compress Do not compress file(s) before upload -g [ --progress ] Report upload progress -v [ --verbose ] Verbose logging --wait-on-close Wait for file objects to be closed before exiting --do-not-resume Do not attempt to resume any incomplete uploads --test Test upload agent settings -i [ --read-from-stdin ] Read file content from stdin
Note: The number of compress threads is optimized and would depend on the system used.

Last edited by Samantha Zarate, 2017-11-09 21:16:50

 Feedback