The information here is of a detailed, technical nature and is not required knowledge to use Artemis. If you are an advanced user, the information here may prove useful if you wish to compile specifically for Artemis.
Artemis is made up of a number of components, including the login nodes, compute nodes, storage system, and management nodes. The compute nodes, Lustre storage and management nodes are all connected by a high performance, low latency “interconnect” based on Mellanox InfiniBand (IB), which is a proprietary networking infrastructure. There is also a 10 Gbps Ethernet management network interconnecting the compute, login and management nodes for compute node and batch job management.
Artemis has 56 “Standard” Haswell compute nodes and 80 “Standard” Broadwell compute nodes based on dual socket servers.
The key features of the Haswell nodes are:
Attribute | Value |
|---|---|
Base vendor model | Dell PowerEdge R630 Server |
CPU model | Intel Xeon E5-2680 V3 @ 2.50 GHz |
CPU generation | Haswell |
Number of cores per node | 24 (2 x 12) |
Resident RAM | 128 GB (8 x 16 GB) DDR3 DIMMs |
Disk storage | 2 x 1 TB 7k NL-SAS in RAID 1 |
Number of 10 Gbps interfaces | 2 |
Number of 1 Gbps interfaces | 2 |
InfiniBand interface | FDR InfiniBand |
The key features of the Broadwell nodes are:
Attribute | Value |
|---|---|
Base vendor model | Dell PowerEdge C6320 Server |
CPU model | Intel Xeon E5-2697A V4 @ 2.6 GHz |
CPU generation | Broadwell |
Number of cores per node | 32 (2 x 16) |
Resident RAM | 128 GB (8 x 16 GB) DDR3 DIMMs |
Disk storage | 2 x 1 TB 10k SAS in RAID 1 |
Number of 10 Gbps interfaces | 2 |
Number of 1 Gbps interfaces | 2 |
InfiniBand interface | FDR InfiniBand |
High Memory Nodes
Artemis has three very high memory compute nodes based on a quad socket server with 6 TB of RAM:
Attribute | Value |
|---|---|
Base vendor model | Dell PowerEdge R930 Server |
CPU model | Intel Xeon E7-8860 V3 @ 2.20 GHz |
CPU generation | Haswell |
Number of cores per node | 64 (4 x 16) |
Resident RAM | 6144 GB (96 x 64 GB) DDR4 DIMMs |
Disk storage | 2 x 200 GB (12 Gbps) SAS SSD, 5 x 2 TB (2 Gbps) SAS SSD |
Number of 10 Gbps interfaces | 2 |
Number of 1 Gbps interfaces | 2 |
InfiniBand interface | FDR InfiniBand |
There are also two high memory nodes based on a dual socket server with 512 GB of RAM:
Attribute | Value |
|---|---|
Base vendor model | Dell PowerEdge R630 Server |
CPU model | Intel Xeon E5-2680 V3 @ 2.50 GHz |
CPU generation | Haswell |
Number of cores per node | 24 (2 x 12) |
Resident RAM | 512 GB (16 x 32 GB) DDR4 DIMMs |
Disk storage | 4 x 1 TB 7k NL-SAS in RAID 10 |
Number of 10 Gbps interfaces | 2 |
Number of 1 Gbps interfaces | 2 |
InfiniBand interface | FDR InfiniBand |
GPU Nodes
There are 5 GPU compute nodes, based on Dell PowerEdge R730 servers. Each is fitted with dual 12 core CPUs and 2 NVIDIA K40 GPUs, giving 10 GPUs in total.
High Performance File System
Artemis has globally accessible, high-performance “Lustre” file system with 56 Gbps FDR InfiniBand interconnect to the rest of the cluster.
The /home, /scratch and /project directories are all stored on Lustre.
Lustre performs best when doing I/O in large chunks (greater than 1 MB). If you are performing I/O constantly on very small files (less than 1 MB), the performance of Lustre filesystem will be poor for yourself and all other all Artemis users. You may want to consider using /tmp, which uses local node disk if you have a program that does lots of small I/O.
Management and Control Nodes
The management nodes are accessible to the vendor’s system administrators only and are used to manage workflow within the cluster.
These nodes consist of two Dell R630 servers with dual 16 core CPUs and 512 GB of memory with a Compellant storage subsystem running virtualised VMware ESXI 6.
Node Interconnect
All nodes and the Lustre filesystem are connected using 56 Gbps FDR InfiniBand interconnect in a 2:1 blocking fat tree configuration. The InfiniBand interconnect provides low latency communications between compute nodes for maximum MPI bandwidth.
Ethernet Networking
10 Gbps redundant switched connections are available between the MPLS switches provided by the University and the Ethernet fabric. The 10 Gbps Ethernet provides access to the login nodes from AARnet and the management network. The 1 Gbps Ethernet provides out-of-band connectivity to the compute nodes.