Attention: Confluence is not suitable for the storage of highly confidential data. Please ensure that any data classified as Highly Protected is stored using a more secure platform.
If you have any questions, please refer to the University's data classification guide or contact ict.askcyber@sydney.edu.au
PyTorch
- nathan butterworth
PyTorch is machine learning framework used in the Python Language. It can utilise GPUs. Newer versions of PyTorch require a version of glibc not easily configurable on the Artemis base operating system Centos 6.9.
Only versions of PyTorch pre-compiled for ARTEMIS will work, “Internet” version will not.
There are multiple certain versions of pytorch installed with the different python versions. Simply loading the preconfigured python module within your pbs script should enable you to run python with pytorch. For example python/3.7.7 comes with torch 1.12 (and note you must load other libraries for torch to work):
module load python/3.7.7 magma/2.5.3 openmpi-gcc/3.1.5
python mytorchscript.pySee the README files located in /usr/local/pytorch/ (and subdirectories for different cuda versions) for more version options available to install. For a specific combination, you can install one of the artemis-built wheels in a virtual environment. For example:
module load python/3.9.15 magma/2.5.3 openmpi-gcc/3.1.5 cuda/10.2.89
virtualenv --system-site-packages /project/<YOURPROJECT>/torchenv
source /project/<YOURPROJECT>/torchenv/bin/activate
pip install /usr/local/pytorch/cuda10.2/torch-1.12.0a0+git664058f-cp39-cp39-linux_x86_64.whl
python testGPU.py*NOTE* You will have to run the install in an interactive job so the installation has enough cpu/ram to complete. (See TensorFlow documentation for similar configuration).
Alternatively, an approach is to using PyTorch on Artemis is by wrapping your workflow in a Singularity container. It is still necessary to balance your container OS and the underlying Host OS (i.e. Artemis).
An example workflow and container recipe file is available from here: https://github.com/Sydney-Informatics-Hub/pytorch-contained and for general Artemis containerisation strategies seeRunning containers with Singularity .
For more information help please submit a Sydney Informatics Hub project support request or contact us at sih.info@sydney.edu.au
Pre-installed versions
PyTorch is installed by default in several python modules available on Artemis. Simply executing, e.g. module load python/3.8.2 will give you a python environment with PyTorch installed.
Python module | PyTorch Version |
|---|---|
python/3.9.15 | - |
python/3.8.2 | 1.9.0 |
python/3.7.7 | 1.6.0 |
python/3.7.2 | 1.1.0 |
python/3.6.5 | 1.1.0 |