Attention: : Confluence is not suitable for the storage of highly confidential data. Please ensure that any data classified as Highly Protected is stored using a more secure platform.
If you have any questions, please refer to the University's data classification guide or contact ict.askcyber@sydney.edu.au

PyTorch

Owned by nathan butterworth
Last updated: Mar 29, 2023
1 min read

PyTorch is machine learning framework used in the Python Language. It can utilise GPUs. Newer versions of PyTorch require a version of glibc not easily configurable on the Artemis base operating system Centos 6.9.

Only versions of PyTorch pre-compiled for ARTEMIS will work, “Internet” version will not.

There are multiple certain versions of pytorch installed with the different python versions. Simply loading the preconfigured python module within your pbs script should enable you to run python with pytorch. For example python/3.7.7 comes with torch 1.12 (and note you must load other libraries for torch to work):

module load python/3.7.7 magma/2.5.3 openmpi-gcc/3.1.5 python mytorchscript.py

See the README files located in /usr/local/pytorch/ (and subdirectories for different cuda versions) for more version options available to install. For a specific combination, you can install one of the artemis-built wheels in a virtual environment. For example:

module load python/3.9.15 magma/2.5.3 openmpi-gcc/3.1.5 cuda/10.2.89 virtualenv --system-site-packages /project/<YOURPROJECT>/torchenv source /project/<YOURPROJECT>/torchenv/bin/activate pip install /usr/local/pytorch/cuda10.2/torch-1.12.0a0+git664058f-cp39-cp39-linux_x86_64.whl python testGPU.py

*NOTE* You will have to run the install in an interactive job so the installation has enough cpu/ram to complete. (See TensorFlow documentation for similar configuration).

Alternatively, an approach is to using PyTorch on Artemis is by wrapping your workflow in a Singularity container. It is still necessary to balance your container OS and the underlying Host OS (i.e. Artemis).

An example workflow and container recipe file is available from here: https://github.com/Sydney-Informatics-Hub/pytorch-contained and for general Artemis containerisation strategies seehttps://sydneyuni.atlassian.net/wiki/spaces/RC/pages/761200644 .

For more information help please submit a Sydney Informatics Hub project support request or contact us at sih.info@sydney.edu.au