Attention: Confluence is not suitable for the storage of highly confidential data. Please ensure that any data classified as Highly Protected is stored using a more secure platform.
If you have any questions, please refer to the University's data classification guide or contact ict.askcyber@sydney.edu.au
TensorFlow
Tensorflow is installed by default in several python modules available on Artemis. Simply executing, e.g. module load python/3.8.2
will give you a python environment with Tensorflow installed.Â
Python module | TensorFlow Version |
---|---|
python/3.9.15 | - |
python/3.8.2 | 2.1.0 |
python/3.7.7 | 2.3.0 |
python/3.7.2 | 1.13.1 |
python/3.6.5 | 1.14.1 |
python/3.5.1 | 1.12.0 |
Custom TensorFlow versions
If you need a specific version you can install following this guide, which is an adaption of the README file provided by Bernie Kirby in the directory /usr/local/tensorflow/v100/cuda10
.2 on Artemis. For the most up-to-date information, have a look for newer README files in /usr/local/tensorflow/v100 directories.
Keep in mind: only the Tensorflow packages provided by us in /usr/local/tensorflow/v100 will work on Artemis. Tensorflow downloaded from the internet will not work. If you would like to use a new version of Tensorflow, submit a High Performance Computing request for it to be installed.
Tensorflow needs to be installed in a Python virtual environment in a directory you have access to on Artemis. To create your own Python virtual environment and install Tensorflow, enter these commands from a login node, to start an interactive job and set up your virtual environment. Note that here the virtual environment name is "tf". You can replace this with any name you like. And replace <project> with your actual project shortcode:
<unikey>@login$ qsub -I -P <project> qsub: waiting for job 466227.pbstraining to start qsub: job 466227.pbstraining ready <unikey>@hpc213 ~$ cd /project/<project> <unikey>@hpc213 <project>$ module load python/3.7.7 cuda/10.2.89 openmpi-gcc/4.1.1 <unikey>@hpc213 <project>$ virtualenv --system-site-packages /project/<project>/tf <unikey>@hpc213 <project>$ source tf/bin/activate (tf) <unikey>@hpc213 <project>$ pip install -U /usr/local/tensorflow/v100/cuda10.2/tensorflow-2.7.0-cp37-cp37-linux_x86_64.whl
There are severl combinations of python and tensorflow version you can install. See available versions of Tensorflow by executing:
ls /usr/local/tensorflow/v100/cuda10.2
You can optionally install most other packages as normal (an interactive job is recommended though). For example, to install/upgrade Keras, run:
(tf) $ pip install -U keras
GPU versions of TensorFlow only work on GPU nodes. You cannot run GPU versions of TensorFlow on any other Artemis node.
Example job script
An example job script for Tensorflow:
#!/bin/bash #PBS -P <project> #PBS -l select=1:ncpus=1:mem=4gb:ngpus=1 #PBS -l walltime=1:00:00 module load python/3.8.2 cuda/10.2.89 source /project/<project>/tf/bin/activate cd "$PBS_O_WORKDIR" python myscript.py