1. RESTAPI Basic Operation Manual
List projects (will return only the projects the user has access to)
curl -u username "https://xnat.sydney.edu.au/data/archive/projects"
^data is returned in JSON format by default. Modify format using the format parameter with html, xml, json, csv.
^username is the XNAT logon username, password will be prompt.
List projects (in XML format and save to a file)
curl -u username "https://xnat.sydney.edu.au/data/archive/projects/?format=xml" -o projects.xml
List files for an experiment (in XML format and save to a file)
curl -u username "https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/files?format=xml" -o filelist.xml
Download all DICOM files for a scan (in zip format)
curl -u username "https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/DICOM/files?format=zip" -o dicomfiles.zip
Note: In this above example, PVT000 is the project name, Haofei01 is the subject name, "Haofei01_MRI" is the session/experiment name, "4" is the scan name.
Upload all files into the scan
curl -u username -X PUT "https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/ABC/files" -F "file=@filelist"
Note: In this above example, PVT000 is the project name, Haofei01 is the subject name, "Haofei01_MRI" is the session/experiment name, "4" is the scan name, "ABC" is the subfolder name, and "files" are necessary to be added.
Create folder
curl -u username -X PUT "https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/test"
Creates a subfolder under scans/4 named test. Note that no further subdirectories are allowed.
2. How can we determine the project, subject, session/experiments, scan name via the XNAT Web Interface ?
It is troublesome in using the RESTAPI command to list project, as they are in HTML/XML/JSON format. Another straightforward way to retrieve the information is via the XNAT web interface.
From the web interface, we can see the project name is : PVT100, Subject name is: "Haofei01", and session/experiment name is "Haofei01_MRI", Scan name is "4" .
Then we would like to know the directory structure below scan name, so we click on the "Manage Files" on the right side of the "Actions" Page.
So we know that the "DICOM" files have all the DICOM images we need. So it is easy for us to figure out the URL path mentioned above in the RESTAPI manuals.
2.1 What is "scans" vs "Resources" ?
We noticed in the File Manager screenshot above, there are "scans" vs "Resources".
Scans are the folder created by XNAT for the DICOM data. Therefore, all the DICOM related files (NIFTI, snapshot, description, report) can be placed into the folder.
Resources are the folder to keep all the non-DICOM data. By default it is not created. It has to be manually created.
We can click "Add Folder" in the FileManager to create new folder
Click "Create"
3. Example on how to run a DICOM to NIFTI convertion on Artemis (University of Sydney High Performance Computing).
In order to run on the Artemis, you need to have an Artemis account. For more information, please refer to below guide to logon to Artemis.
https://sydney.edu.au/research_support/hpc/access/index.shtml#newacct
For Artemis user guide, please refer to the University of Sydney Artemis User Guide
Step 1. Logon to Artemis
ddd-mbp:Deployment$ ssh unikey@hpc.sydney.edu.au
unikey@hpc.sydney.edu.au's password:
Last login: Wed May 9 13:53:09 2018 from vlan-2689-10-17-108-122.staff.wireless.sydney.edu.au
------------------------------------------------------------------------------
***************************** ARTEMIS 3.0 ******************************
------------------------------------------------------------------------------
*** !!! LOGIN NODES ARE FOR JOB SETUP & SUBMISSION ONLY !!! ***
------------------------------------------------------------------------------
Use Interactive flag (-I) or defaultQ or small-express queue for job testing
NO ACTUAL COMPUTE WORK IS TO BE CONDUCTED ON LOGIN NODES
Any sustained compute or multicore/memory intensive process on login nodes
will be subject to being killed without notice.
------------------------------------------------------------------------------
!!! MEMORY MUST BE SPECIFIED FOR ALL JOBS - use the mem= PBS directive !!!
------------------------------------------------------------------------------
Please refer to the Quick Start User Guide to get started with Artemis 2.0:
https://informatics.sydney.edu.au/services/artemis/
-------------------------------------------------------------------------------
Step 2. Download the XNAT data into Artemus directory into the folder. For data in big size, it is recommended to put in to the Artemis /scratch directory.
As the example is only a small set of data (32MB), we are putting it into the home folder on the Artemis Logging node.
[unikey@login2 ~]$ curl -u username "https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/resources/DICOM/files?format=zip" -o dicomfiles.zip
Enter host password for user 'unikey':
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 22.8M 0 22.8M 0 0 5827k 0 --:--:-- 0:00:04 --:--:-- 5994k
[unikey@login2 ~]$ ls -ltr
-rw-r--r-- 1 unikey linuxusers 23963682 May 11 09:22 dicomfiles.zip
[unikey@login2 ~]$ unzip dicomfiles.zip
Archive: dicomfiles.zip
extracting: Haofei01_MR1/scans/4-Ax_3D_T1/resources/DICOM/files/1.3.6.1.4.1.14301.53.4.1079983.1-4-134-4qu0ab.dcm
extracting: Haofei01_MR1/scans/4-Ax_3D_T1/resources/DICOM/files/1.3.6.1.4.1.14301.53.4.1079983.1-4-34-4qtzjm.dcm
...
[unikey@login2 ~]$ ls -ltr
total 23480
-rw-r--r-- 1 unikey linuxusers 23963682 May 11 09:22 dicomfiles.zip
drwxr-xr-x 3 unikey linuxusers 4096 May 11 09:24 Haofei01_MR1
Step 3. Create the Artemis Scripts
For information about Artemis Job Submission, monitoring and management, please refer to the information page about Artemis.
https://sydneyuni.atlassian.net/wiki/spaces/RC/pages/190284213/Job+Submission
https://sydneyuni.atlassian.net/wiki/spaces/RC/pages/213090348/Job+Monitoring+and+Management
[unikey@login2 ~]$ vi run.pbs
#!/bin/bash
#PBS -P SASTEST
#PBS -l select=1:ncpus=1:mem=1gb
#PBS -l walltime=1:00:00
#PBS -M emailemail@sydney.edu.au
#PBS -m abe
#PBS -j oe
module load dcm2niix
mkdir -p /home/unikey/Haofei01_MR1/scans/4-Ax_3D_T1/resources/NIFTI
dcm2niix -o /home/unikey/Haofei01_MR1/scans/4-Ax_3D_T1/resources/NIFTI/ /home/hfen0483/Haofei01_MR1/scans/4-Ax_3D_T1/resources/DICOM/files
Step 4. Run the Artemis Script
[unikey@login2 ~]$ qsub run.pbs
2238022.pbsserver
Step 5. Upload the result back to XNAT
[unikey@login2 ~]$ cd /home/unikey/Haofei01_MR1/scans/4-Ax_3D_T1/resources/NIFTI/
[unikey@login2 NIFTI]$ ls -ltr
total 21000
-rw-r--r-- 1 hfen0483 RDS-ICT-SASTEST-RW 21496160 May 11 09:29 files_Ax_3D_T1_20140604082236_4.nii
-rw-r--r-- 1 hfen0483 RDS-ICT-SASTEST-RW 751 May 11 09:29 files_Ax_3D_T1_20140604082236_4.json
Note, below command create the "unikey" folder and upload the file in one command in XNAT
[unikey@login2 NIFTI]$ curl -u username -X PUT "https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/resources/unikey/files" -F "file=@files_Ax_3D_T1_20140604082236_4.nii"
Enter host password for user 'unikey':
[unikey@login2 NIFTI]$ curl -u username -X PUT "https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/resources/unikey/files" -F "file=@files_Ax_3D_T1_20140604082236_4.json"
Enter host password for user 'unikey':
Step 5. Check from XNAT
4. Use Cookie to perform XNAT RESTAPI call
The reason for using this method is in the AAF, there wont be a "permanent“ username and password. But in order for us to use the AAF with RESTAPI credentials, Chapter 4 and Chapter 5.
4.1 Get A New User Session
This call allows for manual management of HTTP sessions within XNAT. This is particularly useful when you want to re-use the same session across multiple curl or XNATDataClient calls.
POST - /data/JSESSION |
Parameters: Enter user credentials via calling mechanism, i.e:
$ curl -u username -X POST https://yourxnatdomain/data/JSESSION |
Response: A new JSESSION ID string, for example:
3940FAAC4CB8DD368A9A1575372ECBBA |
You can then use this JSESSION ID in a cookie in lieu of user credentials to authenticate subsequent calls, e.g.:
4.1.1 Get the SESSION Number via curl command line
[unikey@login2 NIFTI]$ curl -u username "https://xnat.sydney.edu.au/data/JSESSION/"
Enter host password for user 'username':
628EE7A19C8B9CEF7AE46E3744F6CXX
4.1.2 Get the SESSION Number via Web interface
Just logon the URL://data/JSESSION
List the project
[unikey@login2 NIFTI]$curl --cookie JSESSIONID=628EE7A19C8B9CEF7AE46E3744F6CXX https://xnat.sydney.edu.au/data/projects
{"ResultSet":{"Result":[{"pi_firstname":"","secondary_ID":"PTV000","pi_lastname":"","name":"PTV000","description":"","ID":"PTV000","URI":"/data/projects/PTV000"},{"pi_firstname":"","secondary_ID":"MS project","pi_lastname":"","name":"A prospective study investigating the clinical correlates of brain magnetic resonance imaging; T1, T2, Gd-DTPA, DWI (ADC/FA), perfusion and spectroscopy parameters in enhancing and non-enhancing MSl","description":"","ID":"2012_1047","URI":"/data/projects/2012_1047"},{"pi_firstname":"","secondary_ID":"ForeFront","pi_lastname":"","name":"ForeFront Ageing and Neurodegeneration","description":"Data from various studies run by the ForeFront group.","ID":"FF","URI":"/data/projects/FF"},{"pi_firstname":"","secondary_ID":"Preclinical Imaging","pi_lastname":"","name":"Preclinical Imaging","description":"Preclincial Imaging Core Facility Multimodal Projects","ID":"1","URI":"/data/projects/1"}], "totalRecords": "4"}}
Download the file
[unikey@login2 NIFTI]$curl --cookie JSESSIONID=628EE7A19C8B9CEF7AE46E3744F6CXX
"https://xnat.sydney.edu.au/data/archive/projects/PTV000/subjects/Haofei01/experiments/Haofei01_MR1/scans/4/resources/DICOM/files?format=zip" -o dicomfiles.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 22.8M 0 22.8M 0 0 5513k 0 --:--:-- 0:00:04 --:--:-- 5514k
5. User can generate a user-access token which is a temporary username and password to logon via RESTAPI or XNAT Upload Assistant
The Chapter 4 mentioned a way to use JSESSION ID to logon to XNAT. But if we are to use software such as XNAT Upload Assistant, we might still need to put in the username and password. Below outlines how we can get a temporary username and password from XNAT.
5.1 use command line to get the XNAT temporary username and password
mbp:$ curl -u username "https://test.xnat.sydney.edu.au/data/services/tokens/issue"
Enter host password for user 'username':
{"secret":"gPlzgvaPip6oDGAlJXLypLHBKW3HVbriBSCIZVJcZPic90KDrFkaeMdfefsasef","estimatedExpirationTime":1526514625443,"alias":"451a00c9-441d-4674-84b3-baafb5ab1122","xdatUserId":"username","singleUse":false,"id":3,"disabled":0,"created":1526341825056,"enabled":true,"timestamp":1526341825056}
"alias" follows the temporary username
"secrect" follows the temporary password
mbp:$ curl -u 451a00c9-441d-4674-84b3-baafb5ab1122 "https://test.xnat.sydney.edu.au/data/archive/projects"
Enter host password for user 451a00c9-441d-4674-84b3-baafb5ab1122:
{"ResultSet":{"Result":[{"pi_firstname":"","secondary_ID":"ForeFront","pi_lastname":"","name":"ForeFront Ageing and Neurodegeneration","description":"","ID":"FF","URI":"/data/projects/FF"}], "totalRecords": "1"}}
5.2 use Web Browswer to get the XNAT temporary username and password
Logon to https://URL/data/services/tokens/issue
alias: is the temporary username. (temporary username will be reset for every 6 hours)
secret: is the passwrod
mbp:$ curl -u 451a00c9-441d-4674-84b3-baafb5ab1122 "https://URL/data/archive/projects"
Enter host password for user 451a00c9-441d-4674-84b3-baafb5ab1122:
{"ResultSet":{"Result":[{"pi_firstname":"","secondary_ID":"ForeFront","pi_lastname":"","name":"ForeFront Ageing and Neurodegeneration","description":"","ID":"FF","URI":"/data/projects/FF"}], "totalRecords": "1"}}
Reference
https://wiki.xnat.org/docs16/4-developer-documentation/using-the-xnat-rest-api
XNAT User API (provides the ability to manage user accounts, access, and permissions)
https://sydney.edu.au/research_support/hpc/access/index.shtml#newacct