...
Artemis is made up of a number of components, including the login nodes, compute nodes, storage system, and management nodes. The compute nodes, Lustre storage and management nodes are all connected by a high performance, low-latency InfiniBand (IB) interconnect. There is also a 10 Gbps Ethernet management network interconnecting the compute, login and management nodes for compute node and batch job management.
Artemis has 56 Haswell compute nodes and 80 Broadwell a mix of Haswell, Broadwell and Skylake compute nodes based on dual socket servers.
The key features of the Haswell nodes are:
Attribute | Value | |
|---|---|---|
Base vendor Server model | Dell EMC PowerEdge R630 Server | |
CPU model | Intel Xeon E5-2680 V3 @ 2.50 GHz | |
CPU generation | Haswell | |
Number of cores per node | 24 (2 x 12) | |
Resident RAM | 128 GB (8 x 16 GB) DDR3 DIMMs | |
Disk storage | 2 x 1 TB 7k NL-SAS in RAID 1 | |
Number of Ethernet interface | 10 Gbps interfaces | 2 |
Number of 1 Gbps interfaces | 2 | |
InfiniBand interface | 56 Gbps FDR InfiniBand |
The key features of the Broadwell nodes are:
Attribute | Value | |
|---|---|---|
Base vendor Server model | Dell EMC PowerEdge C6320 Server | |
CPU model | Intel Xeon E5-2697A V4 @ 2.6 GHz | |
CPU generation | Broadwell | |
Number of cores Cores per node | 32 (2 x 16) | |
Resident RAM | 128 GB (8 x 16 GB) DDR3 DIMMs | |
Disk storage | 2 x 1 TB 10k SAS in RAID 1 | |
Number of 10 Gbps interfaces | 2 | |
Number of 1 Gbps interfaces | 2 | |
InfiniBand interface | Ethernet interface | 10 Gbps |
InfiniBand interface | 56 Gbps FDR InfiniBand |
The key features of the Skylake nodes are:
| Attribute | Value |
|---|---|
| Server model | Dell EMC PowerEdge C6420 |
| CPU model | Intel Xeon Platinum 8160 @ 2.10GHz |
| CPU generation | Skylake |
| Cores per node | 48 (2 x 24) |
| Resident RAM | 192 GB |
| Disk storage | |
| Ethernet interface | 10 Gbps |
| InfiniBand interface | 56 Gbps FDR InfiniBand |
High Memory Nodes
Artemis has three very high memory compute nodes based on a quad socket server with 6 TB of RAM:
Attribute | Value | |
|---|---|---|
Base vendor Server model | Dell EMC PowerEdge R930 Server | |
CPU model | Intel Xeon E7-8860 V3 @ 2.20 GHz | |
CPU generation | Haswell | |
Number of cores per node | 64 (4 x 16) | |
Resident RAM | 6144 GB (96 x 64 GB) DDR4 DIMMs | |
Disk storage | 2 x 200 GB (12 Gbps) SAS SSD, 5 x 2 TB (2 Gbps) SAS SSD | |
Ethernet interface | Number of 10 Gbps interfaces | 2 |
Number of 1 Gbps interfaces | 2 | |
InfiniBand interface | 56 Gbps FDR InfiniBand |
There are also two high memory nodes based on a dual socket server with 512 GB of RAM:
Attribute | Value | |
|---|---|---|
Base vendor Server model | Dell EMC PowerEdge R630 Server | |
CPU model | Intel Xeon E5-2680 V3 @ 2.50 GHz | |
CPU generation | Haswell | |
Number of cores per node | 24 (2 x 12) | |
Resident RAM | 512 GB (16 x 32 GB) DDR4 DIMMs | |
Disk storage | 4 x 1 TB 7k NL-SAS in RAID 10 | |
Number of Ethernet interface | 10 Gbps interfaces | 2 |
Number of 1 Gbps interfaces | 2 | |
InfiniBand interface | 56 Gbps FDR InfiniBand |
GPU Nodes
There are 5 GPU compute nodes, based on Dell PowerEdge R730 servers. Each is fitted with dual 12 core CPUs and 2 NVIDIA K40 GPUs, giving 10 GPUs in total.seven Dell C4140 GPU nodes with four Nvidia V100 SXM2 16 GB GPGPUs and two x 16 core CPUs available to all Artemis users. The four GPUs in each compute node are connected using 300 Gbit/s NVLINK2. Technical details about these nodes are shown below:
| Attribute | Value |
|---|---|
| Server model | Dell EMC PowerEdge C4140 |
| CPU model | Intel Xeon Gold 6140 @ 2.30GHz |
| CPU generation | Skylake |
| Cores per node | 36 (2 x 18) |
| Resident RAM | 192 GB |
| GPUs | NVIDIA V100 SXM2 (4 per server) |
| GPU RAM | 16 GB HBM2 per GPU |
| Local GPU-GPU interconnect | NVLINK2 (300 Gbps) |
| Disk Storage | |
| Ethernet interface | 10 Gbps |
| Infiniband interface | 56 Gbps FDR Infiniband |
High Performance File System
Artemis has globally accessible, high-performance “Lustre” Lustre file system with 56 Gbps FDR InfiniBand interconnect to the rest of the cluster.
The /home, /scratch and scratch and /project directories project directories are all stored on Lustre.
| Info |
|---|
Lustre performs best when doing I/O in large chunks (greater than 1 MB). If you are performing I/O constantly on very small files (less than 1 MB), the performance of Lustre filesystem will be poor for yourself and all other all Artemis users. You may want to consider using /tmp, which uses local node disk if you have a program that does lots of small I/O. |
Management and Control Nodes
The management nodes are accessible to the vendor’s system administrators only and are used to manage workflow within the cluster.
These nodes consist of two Dell R630 servers with dual 16 core CPUs and 512 GB of memory with a Compellant storage subsystem running virtualised VMware ESXI 6.
Node Interconnect
All nodes and the Lustre filesystem are connected using 56 Gbps FDR InfiniBand interconnect in a 2:1 blocking fat tree configuration. The InfiniBand interconnect provides low latency communications between compute nodes for maximum MPI bandwidth.
Ethernet Networking
10 Gbps redundant switched connections are available between the MPLS switches provided by the University and the Ethernet fabric. The 10 Gbps Ethernet provides access to the login nodes from AARnet and the management network. The 1 Gbps Ethernet provides out-of-band connectivity to the compute nodes.