Using the Xeon Phi accelerators

qsub -I -q test -l nodes=1:ppn=9:mics=3

allocated-mics.pl > ${HOME}/mics.list
mpirun -n 120 -hostfile ${HOME}/mics.list ./program.mic

Each card has 60 cores and can have up to four threads per core, so all in all 240 threads per card can be created. But from my experience it might be better to use only 120 threads per card.

• No communication between Xeon Phi accelerators of different nodes is possible at the moment This has been fixed with the reinstallation with CentOS7

 qsub submit.cmd 

, where submit.cmd is the name of the submit-file.

To use the Infiniband interfaces, append a "-ib" to the hostnames:

cat $PBS_MICFILE | awk '{print $0"-ib"}' > micfile-ib.$PBS_JOBID

Batch system

Creating submit-files

Example of a submit-file of a MPI-job:

#PBS -o output.dat
#PBS -l walltime=01:00:00,nodes=2:ppn=24
#PBS -M username@uni-muenster.de
#PBS -m ae
#PBS -q default
#PBS -N job_name
#PBS -j oe
cd $PBS_O_WORKDIR
mpdboot  -n 2 -f $PBS_NODEFILE  -v
mpirun -machinefile $PBS_NODEFILE -np 48 ./executable

An MPI-job with 48 processes is started.

Further Information:

• username: Replace by own username
• job_directory: Replace by the path, where the executable can be found
• executable: Enter the name of the executable
• walltime: The time needed for a whole run. At the moment, maximal 48 hours are possible

When no MPI is needed, the submit-file can be simpler.

Example for a job using openMP:

#PBS -o output.dat
#PBS -l walltime=01:00:00,nodes=1:ppn=24
#PBS -M username@uni-muenster.de
#PBS -m ae
#PBS -q default
#PBS -N job_name
#PBS -j oe
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=12
./executable

Choosing the nodes

The cluster consists of the following nodes:

To choose the node type that you want to use, you have to use the correct queue. So if you want to run a large computation which needs more than 128 GB of RAM for a single process, the dl560 is right for you. In this case, you have to use the bigsmp queue:

#PBS -q bigsmp

Submitting jobs / Managing the queue

A job is submitted by entering the command

 qsub submit.cmd 

Further commands:

• qstat: Shows the current queue
• qstat -a: As above, but with the number of requested cores
• qstat -n: Shows in detail, which nodes are used
• qdel job_number: Deletes jobs from the queue
• showbf: Shows the number of free cores

Monitoring jobs

There are different tools for monitoring

• qstat -a: Shows the queues with running and waiting jobs
• pbstop: Similar to qstat but with a text-based graphical output
• Ganglia: Shows detailed information of every node including memory and CPU usage

Storage

There is a 88 TB partition for /home and /scratch using the BeeGFS filesystem (formerly known as FHGfs). Try to store your data like on PALMA: Put your programs in /home and your data in /scratch. Due to the amount of data, there is no backup at the moment.

Most of the computational power of NWZPHI stems from the accelerators. To have some numbers: The 24 CPU cores of one node deliver some hundred GFLOPS, a single Xeon PHI accelerator has a peak performance of 1 TFLOP.

You can reserve the Xeon Phi Cards via the batch system. This would be done the following way:

#PBS -l nodes=1:ppn=24:mics=8

Some useful links

• https://software.intel.com/de-de/mic-developer
• http://www.drdobbs.com/parallel/programming-intels-xeon-phi-a-jumpstart/240144160?pgno=4
• https://software.intel.com/en-us/articles/using-the-intel-mpi-library-on-intel-xeon-phi-coprocessor-systems
• http://www.prace-ri.eu/Best-Practice-Guide-Intel-Xeon-Phi-HTML?lang=en

To use the same modules at every login, put the commands in your $HOME/.bashrc. Recommended modules are

module add compiler/intel/14
module add mpi/intel/4.1.0.024
module add tools/mic

Example: Compile a program that uses the FFTW:

module add compiler/intel/14
module add mpi/intel/4.1.0.024
module add lib/fftw/intel/3.3.3

${MPIICC} -I ${FFTW_INCLUDE_DIR} -o program program.c -g ${FLAGS_FAST} -L${FFTW_LIB_DIR} -lfftw_mpi -lfftw -lm

Hard- and Software overview

• The operating system is RedHat Enterprise Linux 6

Differences to PALMA

• In the submit file, you do not need the switch "-A"
• One node has 24 CPU cores
• The node names and properties are different
• The operating system has another version, so you have to recompile your code
• To use the Xeon Phi accelerators, more work is necessary (see below)

Starting jobs on NWZPHI

• Choose your software environment and (optionally) compile your code
• Submit your job via the batch system

Known Issues

For temporary problems please read the login messages.

Other problems are:

• No communication between Xeon Phi accelerators of different nodes is possible at the moment

NWZPHI the cluster of the IVV 4

NWZPHI is a cluster equipped with 98 Xeon Phi cards. These are PCIe based accelerators similar to GPUs, but can be used with regular programming languages.

Hardware overview

• 2 developing and debugging servers (24 CPU cores with 2.4 GHz, 64 GB RAM, 1 Xeon Phi 5110p)
• 12 accelerator nodes (24 cores with 2.4 GHz, 128 GB RAM, 8 Xeon Phi 5110p)
• 1 SMP node (32 CPU cores, 1.5 TB RAM)
• 88 TB storage (with FhGFS) for home and scratch
• FDR Infinibad as interconnect

NWZPHI for the impatient reader

The name of the login-server is NWZPHI. Allowed are all users that are members of the group u0clustr and at least one of the groups starting with p0, q0 or r0. You can register yourself for u0clstr at http://www.uni-muenster.de/MeinZIV/.

The batch and module system are working very similar to PALMA.

Environment Modules

Batch system

Using Xeon Phi Accelerators

Support

-- HolgerAngenent - 2014-07-23