XNAT REST API Manuals, Artemis (HPC) and examples

1. RESTAPI Basic Operation Manual

List projects (will return only the projects the user has access to)

curl -u username "https://xnat.sydney.edu.au/data/archive/projects"

^data is returned in JSON format by default. Modify format using the format parameter with html, xml, json, csv.

^username is the XNAT logon username, password will be prompt.

For example, below command downloads to dicomfiles.zip that contains all files for existing scan "3980" of existing session "29", that is for existing subject "MH_Brain" of existing project "fang1".

curl -u xufang "https://dev.xnat.sydney.edu.au/data/projects/fang1/subjects/MH_Brain/experiments/29/scans/3980/files?format=zip" -o dicomfiles.zip

Upload all files into the scan

curl -u username -X PUT "https://xnat.sydney.edu.au/data/projects/[insert project name here]/subjects/[insert subject name here]/experiments/[insert session name here]/scans/[insert scan id here]/resources/[insert a custom folder name here]/files?extract=true" -F “file=@compresses-file.zip”

For example, below command will decompress test.zip and save all DICOM files into "TEST" folder under existing scan "3980" of existing session "29", that is for existing subject "MH_Brain" of existing project "fang1". You can specify the name of folder, if not existed, it'll be created.

curl -u xufang -X PUT "https://dev.xnat.sydney.edu.au/data/projects/fang1/subjects/MH_Brain/experiments/29/scans/3980/resources/TEST/files?extract=true" -F "file=@test.zip"

Create folder

curl -u username -X PUT "https://xnat.sydney.edu.au/data/projects/[insert project name here]/subjects/[insert subject name here]/experiments/[insert session name here]/scans/[insert scan id here]/resources/[insert a custom folder name here]"

For example, below command creates a sub-folder "TEST3" under scan named "3980" of existing session "29", that is for existing subject "MH_Brain" of existing project "fang1". Note that no further sub-directories are allowed.

curl -u xufang -X PUT "https://xnat.sydney.edu.au/data/projects/fang1/subjects/MH_Brain/experiments/29/scans/3980/resources/TEST3"

2. How can we determine the project, subject, session/experiments, scan name via the XNAT Web Interface ?

It is troublesome in using the RESTAPI command to list project, as they are in HTML/XML/JSON format. Another straightforward way to retrieve the information is via the XNAT web interface.

From the web interface, we can see the project name is : PVT100, Subject name is: "Haofei01", and session/experiment name is "Haofei01_MRI", Scan name is "4" .

Then we would like to know the directory structure below scan name, so we click on the "Manage Files" on the right side of the "Actions" Page.

So we know that the "DICOM" files have all the DICOM images we need. So it is easy for us to figure out the URL path mentioned above in the RESTAPI manuals.

https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/resources/DICOM/files?format=zip

2.1 What is "scans" vs "Resources" ?

We noticed in the File Manager screenshot above, there are "scans" vs "Resources".

Scans are the folder created by XNAT for the DICOM data. Therefore, all the DICOM related files (NIFTI, snapshot, description, report) can be placed into the folder.

Resources are the folder to keep all the non-DICOM data. By default it is not created. It has to be manually created.

We can click "Add Folder" in the FileManager to create new folder

Click "Create"

3. Example on how to run a DICOM to NIFTI conversion on Artemis (University of Sydney High Performance Computing).

In order to run on the Artemis, you need to have an Artemis account. For more information, please refer to below guide to logon to Artemis.

https://sydney.edu.au/research_support/hpc/access/index.shtml#newacct

For Artemis user guide, please refer to the University of Sydney Artemis User Guide

Artemis User Guide

Step 1. Logon to Artemis

ddd-mbp:Deployment$ ssh unikey@hpc.sydney.edu.au

unikey@hpc.sydney.edu.au's password:

Last login: Wed May 9 13:53:09 2018 from vlan-2689-10-17-108-122.staff.wireless.sydney.edu.au

------------------------------------------------------------------------------

***************************** ARTEMIS 3.0 ******************************

------------------------------------------------------------------------------

*** !!! LOGIN NODES ARE FOR JOB SETUP & SUBMISSION ONLY !!! ***

------------------------------------------------------------------------------

Use Interactive flag (-I) or defaultQ or small-express queue for job testing

NO ACTUAL COMPUTE WORK IS TO BE CONDUCTED ON LOGIN NODES

Any sustained compute or multicore/memory intensive process on login nodes

will be subject to being killed without notice.

------------------------------------------------------------------------------

!!! MEMORY MUST BE SPECIFIED FOR ALL JOBS - use the mem= PBS directive !!!

------------------------------------------------------------------------------

Please refer to the Quick Start User Guide to get started with Artemis 2.0:

https://informatics.sydney.edu.au/services/artemis/

-------------------------------------------------------------------------------

Step 2. Download the XNAT data into Artemus directory into the folder. For data in big size, it is recommended to put in to the Artemis /scratch directory.

As the example is only a small set of data (32MB), we are putting it into the home folder on the Artemis Logging node.

[unikey@login2 ~]$ curl -u username "https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/resources/DICOM/files?format=zip" -o dicomfiles.zip

Enter host password for user 'unikey':

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 22.8M 0 22.8M 0 0 5827k 0 --:--:-- 0:00:04 --:--:-- 5994k

[unikey@login2 ~]$ ls -ltr

-rw-r--r-- 1 unikey linuxusers 23963682 May 11 09:22 dicomfiles.zip

[unikey@login2 ~]$ unzip dicomfiles.zip

Archive: dicomfiles.zip

extracting: Haofei01_MR1/scans/4-Ax_3D_T1/resources/DICOM/files/1.3.6.1.4.1.14301.53.4.1079983.1-4-134-4qu0ab.dcm

extracting: Haofei01_MR1/scans/4-Ax_3D_T1/resources/DICOM/files/1.3.6.1.4.1.14301.53.4.1079983.1-4-34-4qtzjm.dcm

...

[unikey@login2 ~]$ ls -ltr

total 23480

-rw-r--r-- 1 unikey linuxusers 23963682 May 11 09:22 dicomfiles.zip

drwxr-xr-x 3 unikey linuxusers 4096 May 11 09:24 Haofei01_MR1

Step 3. Create the Artemis Scripts

For information about Artemis Job Submission, monitoring and management, please refer to the information page about Artemis.

https://sydneyuni.atlassian.net/wiki/spaces/RC/pages/190284213/Job+Submission

https://sydneyuni.atlassian.net/wiki/spaces/RC/pages/213090348/Job+Monitoring+and+Management

[unikey@login2 ~]$ vi run.pbs

#!/bin/bash

#PBS -P SASTEST

#PBS -l select=1:ncpus=1:mem=1gb

#PBS -l walltime=1:00:00

#PBS -M emailemail@sydney.edu.au

#PBS -m abe

#PBS -j oe

module load dcm2niix

mkdir -p /home/unikey/Haofei01_MR1/scans/4-Ax_3D_T1/resources/NIFTI

dcm2niix -o /home/unikey/Haofei01_MR1/scans/4-Ax_3D_T1/resources/NIFTI/ /home/hfen0483/Haofei01_MR1/scans/4-Ax_3D_T1/resources/DICOM/files

Step 4. Run the Artemis Script

[unikey@login2 ~]$ qsub run.pbs

2238022.pbsserver

Step 5. Upload the result back to XNAT

[unikey@login2 ~]$ cd /home/unikey/Haofei01_MR1/scans/4-Ax_3D_T1/resources/NIFTI/

[unikey@login2 NIFTI]$ ls -ltr

total 21000

-rw-r--r-- 1 hfen0483 RDS-ICT-SASTEST-RW 21496160 May 11 09:29 files_Ax_3D_T1_20140604082236_4.nii

-rw-r--r-- 1 hfen0483 RDS-ICT-SASTEST-RW 751 May 11 09:29 files_Ax_3D_T1_20140604082236_4.json

Note, below command create the "unikey" folder and upload the file in one command in XNAT

[unikey@login2 NIFTI]$ curl -u username -X PUT "https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/resources/unikey/files" -F "file=@files_Ax_3D_T1_20140604082236_4.nii"

Enter host password for user 'unikey':

[unikey@login2 NIFTI]$ curl -u username -X PUT "https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/resources/unikey/files" -F "file=@files_Ax_3D_T1_20140604082236_4.json"

Enter host password for user 'unikey':

Step 5. Check from XNAT

4. Use Cookie to perform XNAT RESTAPI call

The reason for using this method is in the AAF, there wont be a "permanent“ username and password. But in order for us to use the AAF with RESTAPI credentials, Chapter 4 and Chapter 5.

4.1 Get A New User Session

This call allows for manual management of HTTP sessions within XNAT. This is particularly useful when you want to re-use the same session across multiple curl or XNATDataClient calls.

POST - /data/JSESSION

Parameters: Enter user credentials via calling mechanism, i.e:

$ curl -u username -X POST https://yourxnatdomain/data/JSESSION

Response: A new JSESSION ID string, for example:

3940FAAC4CB8DD368A9A1575372ECBBA

You can then use this JSESSION ID in a cookie in lieu of user credentials to authenticate subsequent calls, e.g.:

4.1.1 Get the SESSION Number via curl command line

[unikey@login2 NIFTI]$ curl -u username "https://xnat.sydney.edu.au/data/JSESSION/"

Enter host password for user 'username':

628EE7A19C8B9CEF7AE46E3744F6CXX

4.1.2 Get the SESSION Number via Web interface

Just logon the URL://data/JSESSION

List the project

[unikey@login2 NIFTI]$curl --cookie JSESSIONID=628EE7A19C8B9CEF7AE46E3744F6CXX https://xnat.sydney.edu.au/data/projects

{"ResultSet":{"Result":[{"pi_firstname":"","secondary_ID":"PTV000","pi_lastname":"","name":"PTV000","description":"","ID":"PTV000","URI":"/data/projects/PTV000"},{"pi_firstname":"","secondary_ID":"MS project","pi_lastname":"","name":"A prospective study investigating the clinical correlates of brain magnetic resonance imaging; T1, T2, Gd-DTPA, DWI (ADC/FA), perfusion and spectroscopy parameters in enhancing and non-enhancing MSl","description":"","ID":"2012_1047","URI":"/data/projects/2012_1047"},{"pi_firstname":"","secondary_ID":"ForeFront","pi_lastname":"","name":"ForeFront Ageing and Neurodegeneration","description":"Data from various studies run by the ForeFront group.","ID":"FF","URI":"/data/projects/FF"},{"pi_firstname":"","secondary_ID":"Preclinical Imaging","pi_lastname":"","name":"Preclinical Imaging","description":"Preclincial Imaging Core Facility Multimodal Projects","ID":"1","URI":"/data/projects/1"}], "totalRecords": "4"}}

Download the file

[unikey@login2 NIFTI]$curl --cookie JSESSIONID=628EE7A19C8B9CEF7AE46E3744F6CXX

"https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/resources/DICOM/files?format=zip" -o dicomfiles.zip

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 22.8M 0 22.8M 0 0 5513k 0 --:--:-- 0:00:04 --:--:-- 5514k

5. User can generate a user-access token which is a temporary username and password to logon via RESTAPI or XNAT Upload Assistant

The Chapter 4 mentioned a way to use JSESSION ID to logon to XNAT. But if we are to use software such as XNAT Upload Assistant, we might still need to put in the username and password. Below outlines how we can get a temporary username and password from XNAT.

5.1 use command line to get the XNAT temporary username and password

mbp:$ curl -u username "https://test.xnat.sydney.edu.au/data/services/tokens/issue"

Enter host password for user 'username':

{"secret":"gPlzgvaPip6oDGAlJXLypLHBKW3HVbriBSCIZVJcZPic90KDrFkaeMdfefsasef","estimatedExpirationTime":1526514625443,"alias":"451a00c9-441d-4674-84b3-baafb5ab1122","xdatUserId":"username","singleUse":false,"id":3,"disabled":0,"created":1526341825056,"enabled":true,"timestamp":1526341825056}

"alias" follows the temporary username

"secrect" follows the temporary password

mbp:$ curl -u 451a00c9-441d-4674-84b3-baafb5ab1122 "https://test.xnat.sydney.edu.au/data/archive/projects"

Enter host password for user 451a00c9-441d-4674-84b3-baafb5ab1122:

{"ResultSet":{"Result":[{"pi_firstname":"","secondary_ID":"ForeFront","pi_lastname":"","name":"ForeFront Ageing and Neurodegeneration","description":"","ID":"FF","URI":"/data/projects/FF"}], "totalRecords": "1"}}

5.2 use Web Browswer to get the XNAT temporary username and password

Logon to https://URL/data/services/tokens/issue

alias: is the temporary username. (temporary username will be reset for every 6 hours)

secret: is the passwrod

mbp:$ curl -u 451a00c9-441d-4674-84b3-baafb5ab1122 "https://URL/data/archive/projects"

Enter host password for user 451a00c9-441d-4674-84b3-baafb5ab1122:

{"ResultSet":{"Result":[{"pi_firstname":"","secondary_ID":"ForeFront","pi_lastname":"","name":"ForeFront Ageing and Neurodegeneration","description":"","ID":"FF","URI":"/data/projects/FF"}], "totalRecords": "1"}}

Reference

https://wiki.xnat.org/docs16/4-developer-documentation/using-the-xnat-rest-api

https://wiki.xnat.org/docs16/4-developer-documentation/using-the-xnat-rest-api/downloading-files-via-xnat-rest-api

User Session Management API

XNAT User API (provides the ability to manage user accounts, access, and permissions)

XNAT API Documentation

Artemis User Guide

https://sydney.edu.au/research_support/hpc/access/index.shtml#newacct

[data-colorid=z3lhrj5qwu]{color:#333333} html[data-color-mode=dark] [data-colorid=z3lhrj5qwu]{color:#cccccc}[data-colorid=uxoywhqz0a]{color:#333333} html[data-color-mode=dark] [data-colorid=uxoywhqz0a]{color:#cccccc}1. RESTAPI Basic Operation Manual

List projects (will return only the projects the user has access to)

List projects (in XML format and save to a file)

List files for an experiment (in XML format and save to a file)

Download all DICOM files for a scan (in zip format)

Upload all files into the scan

Create folder

2. How can we determine the project, subject, session/experiments, scan name via the XNAT Web Interface ?

2.1 What is "scans" vs "Resources" ?

3. Example on how to run a DICOM to NIFTI conversion on Artemis (University of Sydney High Performance Computing).

For Artemis user guide, please refer to the University of Sydney Artemis User Guide

Step 1. Logon to Artemis

Step 2. Download the XNAT data into Artemus directory into the folder. For data in big size, it is recommended to put in to the Artemis /scratch directory.

Step 3. Create the Artemis Scripts

Step 4. Run the Artemis Script

Step 5. Upload the result back to XNAT

Note, below command create the "unikey" folder and upload the file in one command in XNAT

Step 5. Check from XNAT

4. Use Cookie to perform XNAT RESTAPI call

4.1 Get A New User Session

4.1.1 Get the SESSION Number via curl command line

4.1.2 Get the SESSION Number via Web interface

List the project

Download the file

5. User can generate a user-access token which is a temporary username and password to logon via RESTAPI or XNAT Upload Assistant

5.1 use command line to get the XNAT temporary username and password

5.2 use Web Browswer to get the XNAT temporary username and password

Reference

1. RESTAPI Basic Operation Manual