Attention: : Confluence is not suitable for the storage of highly confidential data. Please ensure that any data classified as Highly Protected is stored using a more secure platform.
If you have any questions, please refer to the University's data classification guide or contact ict.askcyber@sydney.edu.au

Downloading files from XNAT

Owned by Sophie Matis
Last updated: Jan 08, 2024
6 min read

Downloading your files from XNAT into RDS or Scratch using the XML:

You can either download your files using the XNAT desktop client (Option 1 below) or by submitting a job to the data transfer que in Artemis (Option 2 below). If you would like to download straight into scratch, you will need to do this through Artemis (Option 2). Using the desktop client may be simpler or more intuitive, however, it can be incredibly slow. Using Artemis is recommended as you don’t need to use your computers own resources to download.

Option 1: XNAT Desktop Client (only to RDS).

If you have the RDS mapped to your desktop (you can find out how to do that here if you haven’t already: https://sydneyuni.atlassian.net/wiki/spaces/RC/pages/228589620) you can use the XNAT Desktop Client to download directly into an RDS folder. Once you have downloaded and installed the client (https://download.xnat.org/desktop-client/), you’ll need to open it and enter your XNAT credentials. You can save these so that you only need to enter your password on subsequent logins.

 

Next, select “Download files” from the main menu, select your XML file and set the destination as the path to your RDS folder. The XML file is the file that is downloaded after selecting option 3 on the XNAT download page (you can follow the filtering in. Once you press download the files specified in the XML will be downloaded into the RDS folder you have selected.

 

Alternatively, you can just select Option 1: Download from Desktop client when selecting your files to download from XNAT.

You can track the progress of the download in “Transfer History”.

Option 2: Submit a job through Artemis (recommended).

1. Move the XML file from your local downloads to an Artemis accessible location

The XML file is the file that is downloaded after selecting option 3 on the XNAT download page. This is the option you will need to select to download the data via Artemis. You can easily just drag and drop the file into your RDS folder it if you have it mapped to your local device (https://sydneyuni.atlassian.net/wiki/spaces/RC/pages/228589620). You can also move the XML straight into Artemis from your downloads folder if this is quicker for you.

2. The PBS script

To get Artemis to download scans from XNAT, we can use a tool called xnat-get from a collection of command line tools: XNAT-Utils (). To do so we will submit a job request with a pbs script. To create the pbs script copy and paste the below into a text editor:

#!/bin/bash #PBS -P Project #PBS -N Job Name #PBS -l select=1:ncpus=1:mem=16GB #PBS -l walltime=00:20:00 #PBS -q dtq # Load Modules module load python/3.6.5 source ~/xnatutils.venv/bin/activate # Job Commands # If you would like the scans in rds rather than scratch, specify the target as the rds folder. # If you would like the sessions grouped under separate subject directories add --subject-dirs xnat-get /rds/PRJ-Project/YourFolder/nameofXMLfile.xml --target /scratch/Project/DestinationFolder

You will need to fill in some information for each of the directives:

  1. Replace “Project” with the name of the project in Artemis.

  2. Replace “Job Name” with a name for your job.

  3. Adjust the wall time to reflect the size of the job. This will depend on the number, type, and file type of the scans you are downloading. You may want to try downloading whatever you are after from only one session first, see how long it took and then multiply that time by the number of sessions you are after + a little extra.

You can use the data transfer queue (dtq) for this job so your project’s fair share will not be affected.

Next, edit the arguments for the xnat-get command:

  1. The first argument is the location and file name of the XML file.

  2. The second argument (after --target) is the location you would like the XNAT files downloaded to.

  3. Apply any options you would like, outlined in the comments of the script. **Please note the --convert_to options are not currently working.

Now your script is ready to go. Make sure to save it with the extension as pbs and that it’s Unix compatible.

3. Submit the Job in Artemis

First, navigate to your login node (the directory you are in when you first log in to Artemis). If you haven’t used xnatutils before, you will need to first create an xnatutils environment. Please enter each line one at a time:

module load python/3.6.5 # python -m venv ~/xnatutils.venv # source ~/xnatutils.venv/bin/activate # pip install xnatutils

you can then load the xnatutils module:

source ~/xnatutils.venv/bin/activate

The first time you access XNAT through xnatutils, it will prompt you to enter the server, username, and password. You will need to do this now so that your credentials are able to be accessed when the job is executed.  The easiest way to do this is to run “xnat-ls”:

Now your credentials are saved in a .netrc file in your login node for 2 days and you can submit your XNAT-get job. First enter “deactivate” to leave the XNAT environment, and then submit.

Each session will be downloaded into its own folder in the directory you specified as the target. If you specified – subject dirs, the session folders will be in subject folders.

 

*If your job will take longer than 2 days, you should break it up into multiple jobs, and re-enter your XNAT credentials.