Difference: Palma:rechnen:english (1 vs. 18)

Revision 182017-09-05 - JulianBigge

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 19 to 19
 
module purge Deletes all modules from current environment
Several environment variables will be set by the modules.
Changed:
<
<
For the beginning it is mostly sufficient to load the module "intel/2016a", which is a toolchain that will load further modules like Intel MPI and the MKL.
>
>
For the beginning it is mostly sufficient to load the module "intel/2016b", which is a toolchain that will load further modules like Intel MPI and the MKL.
 

Using the module command in submit scripts

If you want to use the module command in submit scripts, the line

Line: 43 to 43
 #PBS -q default #PBS -N job_name #PBS -j oe
Added:
>
>
source /etc/profile.d/modules.sh module add intel/2016b
 cd $PBS_O_WORKDIR mpdboot --rsh=ssh -n 4 -f $PBS_NODEFILE -v mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np 32 ./executable
Changed:
<
<
An MPI-job with 32 processes is started. For this purpose, 4 Westmere nodes with 32 cores each are demanded.
>
>
An MPI-job with 32 processes is started. For this purpose, 4 Westmere nodes with 32 cores each are demanded. The intel MPI implementation from 2016 is used.
  Further Information:
  • project_name: Has to be replaced by the own project, otherwise the job will not run
Line: 82 to 84
 #PBS -q default #PBS -N job_name #PBS -j oe
Added:
>
>
source /etc/profile.d/modules.sh module add intel/2016b
 export PARNODES=`wc -l $PBS_NODEFILE |gawk '{print $1}'` export UNIQNODES=`cat $PBS_NODEFILE |gawk '{print $1}' | uniq | wc -l` cd $PBS_O_WORKDIR
Line: 99 to 103
 #PBS -q default #PBS -N job_name #PBS -j oe
Added:
>
>
source /etc/profile.d/modules.sh module add foss/2016b
 export PARNODES=`wc -l $PBS_NODEFILE |gawk '{print $1}'` export UNIQNODES=`cat $PBS_NODEFILE |gawk '{print $1}' | uniq | wc -l` cd $PBS_O_WORKDIR OMP_NUM_THREADS=12 mpirun -machinefile $PBS_NODEFILE -n $UNIQNODES -x OMP_NUM_THREADS=12 ./executable
Added:
>
>
Instead of loading the "intel/2016b" module, we will use the "foss/2016b" module which includes the OpenMPI implementation.
 

Submitting jobs / Managing the queue

A job is submitted by entering the command

Revision 162016-11-28 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 10 to 10
 
Command (Short- and Long-form) Meaning
module av[ailable] Lists all available modules
Changed:
<
<
module li[st] Lists all modules in the actual enviroment
>
>
module li[st] Lists all modules in the current enviroment
 
module show modulname Lists all changes caused by a module
Changed:
<
<
module add modul1 modul2 ... Adds module to the actual environment
module rm modul1 modul2 ... Deletes module from the actual environment
module purge Deletes all modules from actual environment
>
>
module add modul1 modul2 ... Adds module to the current environment
module rm modul1 modul2 ... Deletes module from the current environment
module purge Deletes all modules from current environment
 Several environment variables will be set by the modules.

For the beginning it is mostly sufficient to load the module "intel/2016a", which is a toolchain that will load further modules like Intel MPI and the MKL.

Line: 157 to 157
 

The scratch partition

Changed:
<
<
In /scratch there are 180 TB space waiting for user data. For space and performance reasons, data generated by the codes running on palma should be stored here. The filesystem used here is lustre, which is a parallel filesystem. It is therefore optimized for writing small amounts of large files. Writing huge amounts of small files can decrease the performance drastically.
>
>
In /scratch there are 180 TB space waiting for user data. For space and performance reasons, data generated by the codes running on palma should be stored here. The filesystem used here is GPFS, which is a parallel filesystem. It is therefore optimized for writing small amounts of large files. Writing huge amounts of small files can decrease the performance.
  There is no backup of the scratch partition so please make a backup of your data!

Revision 152016-09-05 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 15 to 15
 
module add modul1 modul2 ... Adds module to the actual environment
module rm modul1 modul2 ... Deletes module from the actual environment
module purge Deletes all modules from actual environment
Deleted:
<
<
 Several environment variables will be set by the modules.
Changed:
<
<
Example: Compile a program that uses the FFTW:

module add intel/cc/11.1.059
module add mpi/intel/4.0.3.008
module add fftw/intel/3.3.3

${MPIICC} -I ${FFTW_INCLUDE_DIR} -o program program.c -g ${FLAGS_FAST} -L${FFTW_LIB_DIR} -lfftw_mpi -lfftw -lm
Explanaition: The module fftw/2.1.5 sets the environment variables FFTW2_INCLUDE_DIR and FFTW2_LIB_DIR. These can be used to shorten the compiler calls (also in makefiles).

For the best performance, we recommend the usage of Intel MPI.

>
>
For the beginning it is mostly sufficient to load the module "intel/2016a", which is a toolchain that will load further modules like Intel MPI and the MKL.
 

Using the module command in submit scripts

If you want to use the module command in submit scripts, the line

Line: 35 to 21
 

Using the module command in submit scripts

If you want to use the module command in submit scripts, the line

Deleted:
<
<
 
Changed:
<
<
source /etc/profile.d/modules.sh has to be added before.
>
>
source /etc/profile.d/modules.sh
 
Added:
>
>
has to be added before.
 

Submitting jobs

Added:
>
>
 The batch system Torque and the scheduler Maui are used to submit jobs. It is not allowed, to start jobs manually. Batch jobs should only be submitted from the server palma1.

Creating submit-files

Line: 103 to 84
 export UNIQNODES=`cat $PBS_NODEFILE |gawk '{print $1}' | uniq | wc -l` cd $PBS_O_WORKDIR mpdboot --rsh=ssh -n $UNIQNODES -f $PBS_NODEFILE -v
Changed:
<
<
OMP_NUM_THREADS=12 mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np $UNIQNODES -env OMP_NUM_THREADS=12 ./executable
>
>
OMP_NUM_THREADS=12 mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np $UNIQNODES -env OMP_NUM_THREADS=12 ./executable
  This would start 8 MPI processes on 8 nodes with 12 openMP threads each.
Line: 121 to 100
 export PARNODES=`wc -l $PBS_NODEFILE |gawk '{print $1}'` export UNIQNODES=`cat $PBS_NODEFILE |gawk '{print $1}' | uniq | wc -l` cd $PBS_O_WORKDIR
Changed:
<
<
OMP_NUM_THREADS=12 mpirun -machinefile $PBS_NODEFILE -n $UNIQNODES -x OMP_NUM_THREADS=12 ./executable
>
>
OMP_NUM_THREADS=12 mpirun -machinefile $PBS_NODEFILE -n $UNIQNODES -x OMP_NUM_THREADS=12 ./executable
 

Submitting jobs / Managing the queue

Changed:
<
<
A job is submitted by entering the command
 qsub submit.cmd 
, where submit.cmd is the name of the submit-file.
>
>
A job is submitted by entering the command
 qsub submit.cmd 

, where submit.cmd is the name of the submit-file.

  Further commands:
  • qstat: Shows the current queue

Revision 142014-08-01 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 44 to 44
 

Submitting jobs

Changed:
<
<
The batch system Torque and the scheduler Maui are used to submit jobs. It is not allowed, to start jobs manually. Batch jobs should only be submitted from the server zivcluster.
>
>
The batch system Torque and the scheduler Maui are used to submit jobs. It is not allowed, to start jobs manually. Batch jobs should only be submitted from the server palma1.
 

Creating submit-files

Revision 132013-07-30 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 31 to 31
 

For the best performance, we recommend the usage of Intel MPI.

Added:
>
>

Using the module command in submit scripts

If you want to use the module command in submit scripts, the line

source /etc/profile.d/modules.sh
has to be added before.

 

Submitting jobs

The batch system Torque and the scheduler Maui are used to submit jobs. It is not allowed, to start jobs manually. Batch jobs should only be submitted from the server zivcluster.

Revision 122013-04-08 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 141 to 141
 
  • default: maximal walltime of 48 hours
  • long: same as default, but with maximal 160 hours walltime; only 8 jobs of each user will be executed at a time
Changed:
<
<
  • mescashort: jobs will be sent to the nodes palma060-palma065 which have at least 256 GB RAM and 32 CPU cores; they are not accesible via the default queue; only 4 jobs per user will run at the same time, maximal walltime of 48 hours
>
>
  • mescashort: jobs will be sent to the nodes palma060-palma063 or palma065 which have 256 GB RAM and 32 CPU cores; they are not accesible via the default queue; only 4 jobs per user will run at the same time, maximal walltime of 48 hours
 
  • mescalong: the same as mescashort, but with maximal 2 jobs per user and a walltime of 160 hours
Changed:
<
<
  • mescatest: higher priotized queue to test jobs on the largesmp nodes; walltime is restricted to 1 hour
>
>
  • mescatest: higher priotized queue to test jobs on the mesca nodes; walltime is restricted to 1 hour
  • mescabig: jobs will be send to palma064 only, which has 512 GB memory, maximum walltime of 160 hours
  Global restrictions:

Revision 112013-03-22 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 16 to 16
 
module rm modul1 modul2 ... Deletes module from the actual environment
module purge Deletes all modules from actual environment
Changed:
<
<
To get acces to all modules, the following modules have to be added to the standard environment (this can be done by adding them to .bashrc):
>
>
Several environment variables will be set by the modules.
 
Changed:
<
<
module add shared
module add /Applic.PALMA/modules/path

Intel MPI is installed on Palma. To use, load the appropriate module:

module add intel/mpi/3.2.2.006
The compiler executables are similar to their serial versions: mpiicc, mpiicpc, mpiifort etc.

Example: Compile a program that uses FFTW2:

>
>
Example: Compile a program that uses the FFTW:
 
Deleted:
<
<
module add fftw/2.1.5
 module add intel/cc/11.1.059
Changed:
<
<
module add intel/mpi/3.2.2.006 mpiicc -I ${FFTW2_INCLUDE_DIR} -o program mpifftw2d.c -g -O3 -L${FFTW2_LIB_DIR} -lsrfftw_mpi -lsfftw_mpi -lsrfftw -lsfftw -lm
>
>
module add mpi/intel/4.0.3.008 module add fftw/intel/3.3.3

${MPIICC} -I ${FFTW_INCLUDE_DIR} -o program program.c -g ${FLAGS_FAST} -L${FFTW_LIB_DIR} -lfftw_mpi -lfftw -lm

  Explanaition: The module fftw/2.1.5 sets the environment variables FFTW2_INCLUDE_DIR and FFTW2_LIB_DIR. These can be used to shorten the compiler calls (also in makefiles).
Added:
>
>
For the best performance, we recommend the usage of Intel MPI.
 

Submitting jobs

The batch system Torque and the scheduler Maui are used to submit jobs. It is not allowed, to start jobs manually. Batch jobs should only be submitted from the server zivcluster.
Line: 99 to 91
 export UNIQNODES=`cat $PBS_NODEFILE |gawk '{print $1}' | uniq | wc -l` cd $PBS_O_WORKDIR mpdboot --rsh=ssh -n $UNIQNODES -f $PBS_NODEFILE -v
Changed:
<
<
mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np $UNIQNODES -env OMP_NUM_THREADS=12 ./executable
>
>
OMP_NUM_THREADS=12 mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np $UNIQNODES -env OMP_NUM_THREADS=12 ./executable
 

This would start 8 MPI processes on 8 nodes with 12 openMP threads each.

Line: 117 to 109
 export PARNODES=`wc -l $PBS_NODEFILE |gawk '{print $1}'` export UNIQNODES=`cat $PBS_NODEFILE |gawk '{print $1}' | uniq | wc -l` cd $PBS_O_WORKDIR
Changed:
<
<
mpirun -machinefile $PBS_NODEFILE -n $UNIQNODES -x OMP_NUM_THREADS=12 ./executable
>
>
OMP_NUM_THREADS=12 mpirun -machinefile $PBS_NODEFILE -n $UNIQNODES -x OMP_NUM_THREADS=12 ./executable
 

Submitting jobs / Managing the queue

Revision 102012-11-20 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 97 to 97
 #PBS -j oe export PARNODES=`wc -l $PBS_NODEFILE |gawk '{print $1}'` export UNIQNODES=`cat $PBS_NODEFILE |gawk '{print $1}' | uniq | wc -l`
Deleted:
<
<
export OMP_NUM_THREADS=12
 cd $PBS_O_WORKDIR mpdboot --rsh=ssh -n $UNIQNODES -f $PBS_NODEFILE -v
Changed:
<
<
mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np $UNIQNODES ./executable
>
>
mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np $UNIQNODES -env OMP_NUM_THREADS=12 ./executable
 

This would start 8 MPI processes on 8 nodes with 12 openMP threads each.

Added:
>
>
Using gcc and OpenMPI, this would read in this way:

#PBS -o output.dat
#PBS -l walltime=01:00:00,nodes=8:westmere:ppn=12
#PBS -A project_name
#PBS -M username@uni-muenster.de
#PBS -m ae
#PBS -q default
#PBS -N job_name
#PBS -j oe
export PARNODES=`wc -l $PBS_NODEFILE |gawk '{print $1}'`
export UNIQNODES=`cat $PBS_NODEFILE |gawk '{print $1}' | uniq | wc -l`
cd $PBS_O_WORKDIR
mpirun -machinefile $PBS_NODEFILE -n $UNIQNODES -x OMP_NUM_THREADS=12 ./executable
 

Submitting jobs / Managing the queue

A job is submitted by entering the command

 qsub submit.cmd 
, where submit.cmd is the name of the submit-file.

Revision 92012-09-27 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 118 to 118
 

Choosing the compute nodes

Changed:
<
<
The option "#PBS -l" determines, which resourses are required for the batch job. Due to the existence of two different kind of nodes it can be distinguished between them with the attribute "nehalem" and "westmere". The following tables shows different possibilities to reserve nodes.
>
>
The option "#PBS -l" determines, which resources are required for the batch job. Due to the existence of two different kind of nodes it can be distinguished between them with the attribute "nehalem" and "westmere". The following tables shows different possibilities to reserve nodes.
 
in node-file nodes that will be reserved
Changed:
<
<
-l nodes=10 10 arbitrary CPU cores. Not recommended
-l nodes=2:ppn=12 2 Westmere nodes
-l nodes=1:ppn=8 8 cores of a Westmere or Nehalem node, not advisable due to the unpredictable result
-l nodes=2:nehalem:ppn=8+3:westmere:ppn=12 16 cores of 2 Nehalem nodes and 36 cores of 3 Westmere nodes
-l nodes=8:ppn=1 Produces an error. ppn should be larger than 1. Due to a bug in TORQUE, the user gets 8 cores on arbitrary nodes in this case.

If in doubt, use

-l nodes=x:westmere:ppn=12
-l nodes=x:nehalem:ppn=8
with "x" being the number of nodes you want to reserve
>
>
-l nodes=10:westmere:ppn=12 10 Westmere nodes with 12 cores each
-l nodes=2:himem:ppn=12 2 Westmere himem nodes (with 48 GB memory per node)
-l nodes=1:ppn=8 8 cores of a Westmere or a Nehalem node (not recommended)
-l nodes=1:nehalem:ppn=1 1 core of a Nehalem node (recommended for serial jobs)
 
Changed:
<
<
Please send serial jobs only to the nehalem nodes and parallel jobs to the westmere nodes.
>
>
Please send serial jobs only to the Nehalem nodes and parallel jobs to the Westmere nodes.
 

The queues

Line: 164 to 158
 

The scratch partition

Changed:
<
<
In /scratch there are 180 TB space waiting for user data. For space and performance reasons, data generated by the codes running on palma should be stored here. The filesystem used here is lustre, which is a parralel filesystem.
>
>
In /scratch there are 180 TB space waiting for user data. For space and performance reasons, data generated by the codes running on palma should be stored here. The filesystem used here is lustre, which is a parallel filesystem. It is therefore optimized for writing small amounts of large files. Writing huge amounts of small files can decrease the performance drastically.
  There is no backup of the scratch partition so please make a backup of your data!

Revision 82012-03-30 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 139 to 139
 There are some diffent queues on Palma:

  • default: maximal walltime of 48 hours
Changed:
<
<
  • long: same as default, but with maximal 160 hours walltime
  • largesmp: jobs will be sent to the nodes palma060-palma065 which have at least 256 GB RAM and 32 CPU cores; they are not accesible via the default queue; only 2 jobs per user will run at the same time
  • largesmptest: higher priotized queue to test jobs on the largesmp nodes; walltime is restricted to 1 hour
>
>
  • long: same as default, but with maximal 160 hours walltime; only 8 jobs of each user will be executed at a time
  • mescashort: jobs will be sent to the nodes palma060-palma065 which have at least 256 GB RAM and 32 CPU cores; they are not accesible via the default queue; only 4 jobs per user will run at the same time, maximal walltime of 48 hours
  • mescalong: the same as mescashort, but with maximal 2 jobs per user and a walltime of 160 hours
  • mescatest: higher priotized queue to test jobs on the largesmp nodes; walltime is restricted to 1 hour
 
Changed:
<
<
Restrictions:
>
>
Global restrictions:
 
  • Jobs will only start, if they use less than 132.710.400 seconds of CPU time. This corresponds for example to 48 hours on 64 westmere nodes. If you want to use more nodes at a time, please reduce the walltime appropriately.
Changed:
<
<
  • Only 8 jobs of each user will be executed at a time in the long queue
  • Only 2 jobs of each user will be executed at a time in the largesmp queue
>
>
 

Using the batch system for jobs with a GUI

Revision 72012-03-30 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 79 to 79
 #PBS -q default #PBS -N job_name #PBS -j oe
Changed:
<
<
cd $HOME/job_directory
>
>
cd $PBS_O_WORKDIR export OMP_NUM_THREADS=12
 ./executable
Added:
>
>

Setting up hybrid jobs

To start a hybrid parallelized job, i.e with MPI and openMP, you can use the following method:

#PBS -o output.dat
#PBS -l walltime=01:00:00,nodes=8:westmere:ppn=12
#PBS -A project_name
#PBS -M username@uni-muenster.de
#PBS -m ae
#PBS -q default
#PBS -N job_name
#PBS -j oe
export PARNODES=`wc -l $PBS_NODEFILE |gawk '{print $1}'`
export UNIQNODES=`cat $PBS_NODEFILE |gawk '{print $1}' | uniq | wc -l`
export OMP_NUM_THREADS=12
cd $PBS_O_WORKDIR
mpdboot --rsh=ssh -n $UNIQNODES -f $PBS_NODEFILE  -v
mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np $UNIQNODES ./executable

This would start 8 MPI processes on 8 nodes with 12 openMP threads each.

 

Submitting jobs / Managing the queue

A job is submitted by entering the command

 qsub submit.cmd 
, where submit.cmd is the name of the submit-file.
Line: 109 to 132
 
-l nodes=x:nehalem:ppn=8
with "x" being the number of nodes you want to reserve
Added:
>
>
Please send serial jobs only to the nehalem nodes and parallel jobs to the westmere nodes.
 

The queues

There are some diffent queues on Palma:

  • default: maximal walltime of 48 hours
  • long: same as default, but with maximal 160 hours walltime
Changed:
<
<
  • largesmp: jobs will be sent to the nodes palma060-palma065 which have at least 256 GB RAM and 32 CPU cores; they are not accesible via the default queue
>
>
  • largesmp: jobs will be sent to the nodes palma060-palma065 which have at least 256 GB RAM and 32 CPU cores; they are not accesible via the default queue; only 2 jobs per user will run at the same time
  • largesmptest: higher priotized queue to test jobs on the largesmp nodes; walltime is restricted to 1 hour
  Restrictions:

  • Jobs will only start, if they use less than 132.710.400 seconds of CPU time. This corresponds for example to 48 hours on 64 westmere nodes. If you want to use more nodes at a time, please reduce the walltime appropriately.
Changed:
<
<
  • Only 4 jobs of each user will be executed at a time in the long queue
  • Only 3 jobs of each user will be executed at a time in the largesmp queue
>
>
  • Only 8 jobs of each user will be executed at a time in the long queue
  • Only 2 jobs of each user will be executed at a time in the largesmp queue
 

Using the batch system for jobs with a GUI

Revision 62011-11-09 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 100 to 100
 
in node-file nodes that will be reserved
-l nodes=10 10 arbitrary CPU cores. Not recommended
-l nodes=2:ppn=12 2 Westmere nodes
Changed:
<
<
-l nodes=1:ppn=8 8 cores of a Westmere or Nehalem node
>
>
-l nodes=1:ppn=8 8 cores of a Westmere or Nehalem node, not advisable due to the unpredictable result
 
-l nodes=2:nehalem:ppn=8+3:westmere:ppn=12 16 cores of 2 Nehalem nodes and 36 cores of 3 Westmere nodes
-l nodes=8:ppn=1 Produces an error. ppn should be larger than 1. Due to a bug in TORQUE, the user gets 8 cores on arbitrary nodes in this case.
Added:
>
>
If in doubt, use
-l nodes=x:westmere:ppn=12
-l nodes=x:nehalem:ppn=8
with "x" being the number of nodes you want to reserve
 
Changed:
<
<
To submit jobs to the SMP-system, it is neccessary to login to them directly, at the moment. They cannot be accessed via the batch system at the moment. The name of the nodes are: palma060, palma061, palma062 and palma063.
>
>

The queues

There are some diffent queues on Palma:

  • default: maximal walltime of 48 hours
  • long: same as default, but with maximal 160 hours walltime
  • largesmp: jobs will be sent to the nodes palma060-palma065 which have at least 256 GB RAM and 32 CPU cores; they are not accesible via the default queue

Restrictions:

  • Jobs will only start, if they use less than 132.710.400 seconds of CPU time. This corresponds for example to 48 hours on 64 westmere nodes. If you want to use more nodes at a time, please reduce the walltime appropriately.
  • Only 4 jobs of each user will be executed at a time in the long queue
  • Only 3 jobs of each user will be executed at a time in the largesmp queue

Using the batch system for jobs with a GUI

To start these programs with a GUI on a node with the batch system, see this guide

Monitoring jobs

There are different tools for monitoring

  • qstat -a: Shows the queues with running and waiting jobs
  • pbstop: Similar to qstat but with a text-based graphical output
  • myJAM: Webinterface; see "Rack View" to view the state of every node in the system
  • Ganglia: Shows even more information of every node including memory and CPU usage
 

The scratch partition

In /scratch there are 180 TB space waiting for user data. For space and performance reasons, data generated by the codes running on palma should be stored here. The filesystem used here is lustre, which is a parralel filesystem.

Changed:
<
<
There is no backup of the scratch partition!
>
>
There is no backup of the scratch partition so please make a backup of your data!
 

-- HolgerAngenent - 2010-08-16 \ No newline at end of file

Revision 52011-08-29 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 54 to 54
 #PBS -q default #PBS -N job_name #PBS -j oe
Changed:
<
<
cd $HOME/job_directory cp $PBS_NODEFILE $HOME/$PBS_JOBID.nodefile mpdboot --rsh=ssh -n 2 -f ~/$PBS_JOBID.nodefile -v
>
>
cd $PBS_O_WORKDIR mpdboot --rsh=ssh -n 4 -f $PBS_NODEFILE -v
 mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np 32 ./executable
Changed:
<
<
An MPI-job with 32 processes is started. For this purpose, 2 Westmere nodes with 12 cores each are demanded.
>
>
An MPI-job with 32 processes is started. For this purpose, 4 Westmere nodes with 32 cores each are demanded.
  Further Information:
  • project_name: Has to be replaced by the own project, otherwise the job will not run

Revision 42010-10-29 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 105 to 105
 
-l nodes=2:nehalem:ppn=8+3:westmere:ppn=12 16 cores of 2 Nehalem nodes and 36 cores of 3 Westmere nodes
-l nodes=8:ppn=1 Produces an error. ppn should be larger than 1. Due to a bug in TORQUE, the user gets 8 cores on arbitrary nodes in this case.
Deleted:
<
<
It is recommended to reserve complete nodes, this means the requirements should look like
nodes=x:nehalem:ppn=8
or
nodes=x:westmere:ppn=12
  To submit jobs to the SMP-system, it is neccessary to login to them directly, at the moment. They cannot be accessed via the batch system at the moment. The name of the nodes are: palma060, palma061, palma062 and palma063.

Revision 32010-08-17 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 96 to 96
 

Choosing the compute nodes

Changed:
<
<
Mit der Option -l legt man die Ressourcenanforderungen für den Batch-Job fest. Bei der Festlegung der Anzahl und Art der Nodes ist zu beachten, dass die Rechenknoten des Systems zur Unterscheidung mit den Eigenschaften "nehalem" und "westmere" gekennzeichnet sind. Außerdem gibt es pro Node mehrere CPU-Kerne, so dass die Angabe nodes zwei unterschiedliche Bedeutungen haben kann. Die folgende Tabelle soll verdeutlichen, wie sich die Angabe nodes auf die Auswahl der Rechenknoten auswirkt:
>
>
The option "#PBS -l" determines, which resourses are required for the batch job. Due to the existence of two different kind of nodes it can be distinguished between them with the attribute "nehalem" and "westmere". The following tables shows different possibilities to reserve nodes.
 
Changed:
<
<
Angabe im Batch-Job Auswahl der Rechenknoten
-l nodes=10 10 beliebige CPU-Kerne. Nicht zu empfehlen.
-l nodes=2:ppn=12 2 Westmere Knoten
-l nodes=1:ppn=8 8 Prozessorkerne eines Westmere oder Nehalem Knotens
-l nodes=2:nehalem:ppn=8+3:westmere:ppn=12 16 Prozessorkerne von 2 Nehalem Knoten und 36 Prozessorkerne von 3 Westmere Knoten
-l nodes=8:ppn=1 Fehlerhafte Zuteilung. Die Anzahl von ppn, wenn angegeben, sollte größer oder gleich 2 sein. Aufgrund eines Bugs in TORQUE erhält man für ppn=1 nicht das erwartete Ergebnis, sondern z.B. in diesem Fall 8 willkürlich über die HPC-Rechnenknoten verteilte Cores.
>
>
in node-file nodes that will be reserved
-l nodes=10 10 arbitrary CPU cores. Not recommended
-l nodes=2:ppn=12 2 Westmere nodes
-l nodes=1:ppn=8 8 cores of a Westmere or Nehalem node
-l nodes=2:nehalem:ppn=8+3:westmere:ppn=12 16 cores of 2 Nehalem nodes and 36 cores of 3 Westmere nodes
-l nodes=8:ppn=1 Produces an error. ppn should be larger than 1. Due to a bug in TORQUE, the user gets 8 cores on arbitrary nodes in this case.
 
Changed:
<
<
Generell sollten immer komplette Knoten angefordert werden, d.h. Angaben wie
nodes=x:nehalem:ppn=8
oder
nodes=x:westmere:ppn=12
gemacht werden.
>
>
It is recommended to reserve complete nodes, this means the requirements should look like
nodes=x:nehalem:ppn=8
or
nodes=x:westmere:ppn=12
 
Changed:
<
<
Bei Bedarf können die einzelnen Chassis, in denen die Knoten stecken, auch direkt angefordert werden. Dazu ist statt der Angabe #PBS -q default die Angabe des Chassis nötig. Es kann zwischen folgenden Chassis gewählt werden: chassisd1, chassisd2, chassisd3, chassisd4, chassise1, chassise2, chassise3, chassisf1, chassisf2, chassisf3, chassisg1 wobei die Chassis d1 und d2 nehalem Prozessoren besitzen und das chassis g1 48GB statt 24 GB Hauptspeicher pro Knoten hat.
>
>
To submit jobs to the SMP-system, it is neccessary to login to them directly, at the moment. They cannot be accessed via the batch system at the moment. The name of the nodes are: palma060, palma061, palma062 and palma063.
 
Changed:
<
<
Um auf den SMP-Systemen zu rechnen, ist es derzeit noch notwendig, sich direkt auf diesen einzuloggen und die Jobs interaktiv zu starten. Die Namen der Knoten sind palma060, palma061, palma062 und palma063.

Die scratch Partition

>
>

The scratch partition

 
Changed:
<
<
Paralles speichern von Daten auf /scratch... in Vorbereitung

Unter /scratch befinden sich 180 TB Speicherplatz, der genutzt werden kann, um die in den Programmen generierten Daten zu speichern. Diese sollten aus Platz- und Geschwindigkeitsgründen nicht im Homeverzeichnis gespeichert werden. Als Dateisystem kommt lustre zum Einsatz, das einige Besonderheiten besitzt.

Es wird kein Backup dieser Partition durchgeführt!

>
>
In /scratch there are 180 TB space waiting for user data. For space and performance reasons, data generated by the codes running on palma should be stored here. The filesystem used here is lustre, which is a parralel filesystem.
 
Added:
>
>
There is no backup of the scratch partition!
 

-- HolgerAngenent - 2010-08-16

Revision 22010-08-17 - HolgerAngenent

Line: 1 to 1
 
META TOPICPARENT name="PALMA:english"

Computations on Palma

Line: 23 to 23
 module add /Applic.PALMA/modules/path
Changed:
<
<
Auf Palma ist MPI von Intel installiert. Um es nutzen zu können, muss ebenfalls ein Modul geladen werden.
>
>
Intel MPI is installed on Palma. To use, load the appropriate module:
 
module add intel/mpi/3.2.2.006
Changed:
<
<
Die wichtigsten Kompileraufrufe lauten dann analog zum seriellen Intelcompiler mpiicc, mpiicpc, mpiifort usw.
>
>
The compiler executables are similar to their serial versions: mpiicc, mpiicpc, mpiifort etc.
 
Changed:
<
<
Beispiel: Übersetzen eines Programms mit der FFTW2:
>
>
Example: Compile a program that uses FFTW2:
 
module add fftw/2.1.5
module add intel/cc/11.1.059
module add intel/mpi/3.2.2.006
Changed:
<
<
mpiicc -I ${FFTW2_INCLUDE_DIR} -o Programm mpifftw2d.c -g -O3 -L${FFTW2_LIB_DIR} -lsrfftw_mpi -lsfftw_mpi -lsrfftw -lsfftw -lm
>
>
mpiicc -I ${FFTW2_INCLUDE_DIR} -o program mpifftw2d.c -g -O3 -L${FFTW2_LIB_DIR} -lsrfftw_mpi -lsfftw_mpi -lsrfftw -lsfftw -lm
 
Changed:
<
<
Zur Erklärung: Die Module shared und path müssen natürlich schon geladen sein. Durch das Modul fftw/2.1.5 werden laut "module show fftw/2.1.5" die Umgebungsvariablen FFTW2_INCLUDE_DIR und FFTW2_LIB_DIR gesetzt, die dann beim Kompilieren (auch im makefile) verwendet werden können.
>
>
Explanaition: The module fftw/2.1.5 sets the environment variables FFTW2_INCLUDE_DIR and FFTW2_LIB_DIR. These can be used to shorten the compiler calls (also in makefiles).
 
Changed:
<
<

Submittieren von Jobs

Für die Verteilung von Rechenjobs auf dem Cluster sorgen das Batchsystem TORQUE sowie der Scheduler Maui. Ein manuelles Starten von längeren Programmläufen auf den Rechenknoten oder auf PALMA am Batchsytem vorbei ist nicht erlaubt. Solche "streunenden" Prozesse werden ohne Vorwarnung vom Administrator beendet. Grundsätzlich sollten Batch-Jobs nur vom Dialogserver palma1 aus abgeschickt werden.
>
>

Submitting jobs

The batch system Torque and the scheduler Maui are used to submit jobs. It is not allowed, to start jobs manually. Batch jobs should only be submitted from the server zivcluster.
 
Changed:
<
<

Erstellen des submit-files

>
>

Creating submit-files

 
Changed:
<
<
Beispiel für ein submit-file eines MPI-Jobs:
>
>
Example of a submit-file of a MPI-job:
 
#PBS -o output.dat
#PBS -l walltime=01:00:00,nodes=4:westmere:ppn=12
Changed:
<
<
#PBS -A Projektname #PBS -M Nutzername@uni-muenster.de
>
>
#PBS -A project_name #PBS -M username@uni-muenster.de
 #PBS -m ae #PBS -q default
Changed:
<
<
#PBS -N Jobname
>
>
#PBS -N job_name
 #PBS -j oe
Changed:
<
<
cd $HOME/Jobverzeichnis
>
>
cd $HOME/job_directory
 cp $PBS_NODEFILE $HOME/$PBS_JOBID.nodefile mpdboot --rsh=ssh -n 2 -f ~/$PBS_JOBID.nodefile -v
Changed:
<
<
mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np 32 ./Programmname
>
>
mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np 32 ./executable
 
Changed:
<
<
Hier wird ein MPI-Job gestartet, der mit insgesamt 32 Prozessen läuft. Dazu werden 4 westmere-Knoten mit je 12 Prozessorkernen angefordert.
>
>
An MPI-job with 32 processes is started. For this purpose, 2 Westmere nodes with 12 cores each are demanded.
 
Changed:
<
<
Weitere Angaben:
  • Projektname: Muss unbedingt durch das eigene Projekt erstetzt werden, ansonsten startet der Job nicht.
  • Nutzername: Durch eigene Kennung ersetzen, damit Emails korrekt zugestellt werden können
  • Jobverzeichnis: Durch Pfad, in dem das Programm liegt, ersetzen.
  • Programmname: Hier den Namen der auszuführenden Datei eintragen.
  • walltime: Die nötige Zeit, die das Programm läuft, angeben. Nach Ablauf der Zeit wird der Job abgebrochen. Maximal 48 Stunden sind möglich.
>
>
Further Information:
  • project_name: Has to be replaced by the own project, otherwise the job will not run
  • username: Replace by own username
  • job_directory: Replace by the path, where the executable can be found
  • executable: Enter the name of the executable
  • walltime: The time needed for a whole run. At the moment, maximal 48 hours are possible
 
Changed:
<
<
Benötigt das Programm kein MPI, kann die Datei entsprechend vereinfacht werden.
>
>
When no MPI is needed, the submit-file can be simpler.
 
Changed:
<
<
Beispiel für ein submit-file eines openmp Jobs:
>
>
Example for a job using openMP:
 
#PBS -o output.dat
#PBS -l walltime=01:00:00,nodes=1:westmere:ppn=12
Changed:
<
<
#PBS -A Projektname #PBS -M Nutzername@uni-muenster.de
>
>
#PBS -A project_name #PBS -M username@uni-muenster.de
 #PBS -m ae #PBS -q default
Changed:
<
<
#PBS -N Jobname
>
>
#PBS -N job_name
 #PBS -j oe
Changed:
<
<
cd $HOME/Jobverzeichnis ./Programmname
>
>
cd $HOME/job_directory ./executable
 
Changed:
<
<

Abschicken des Jobs/Verwaltung der Warteschlange

>
>

Submitting jobs / Managing the queue

 
Changed:
<
<
Ein Job wird gestartet, indem die oben erzeugte submit-Datei mittels
 qsub submit.cmd 
abgeschickt wird, wobei submit.cmd der Name der submit-Datei ist.
>
>
A job is submitted by entering the command
 qsub submit.cmd 
, where submit.cmd is the name of the submit-file.
 
Changed:
<
<
Weitere Befehle:
  • qstat: Zeigt die derzeitige Warteschlange an
  • qstat -a: Selbiges, nur mit Anzahl der angeforderten Knoten
  • qstat -n: Alle Knoten, die den Jobs zugeordnet sind, werden aufgeschlüsselt
  • qdel Jobnummer: Löscht Jobs aus der Warteschlange
  • showbf: Zeigt die Anzahl der freien Prozessorkerne
>
>
Further commands:
  • qstat: Shows the current queue
  • qstat -a: As above, but with the number of requested cores
  • qstat -n: Shows in detail, which nodes are used
  • qdel job_number: Deletes jobs from the queue
  • showbf: Shows the number of free cores
 
Changed:
<
<

Die Auswahl der Rechenknoten

>
>

Choosing the compute nodes

  Mit der Option -l legt man die Ressourcenanforderungen für den Batch-Job fest. Bei der Festlegung der Anzahl und Art der Nodes ist zu beachten, dass die Rechenknoten des Systems zur Unterscheidung mit den Eigenschaften "nehalem" und "westmere" gekennzeichnet sind. Außerdem gibt es pro Node mehrere CPU-Kerne, so dass die Angabe nodes zwei unterschiedliche Bedeutungen haben kann. Die folgende Tabelle soll verdeutlichen, wie sich die Angabe nodes auf die Auswahl der Rechenknoten auswirkt:
Line: 122 to 122
 
Deleted:
<
<
-- HolgerAngenent - 2010-07-12
 -- HolgerAngenent - 2010-08-16

Revision 12010-08-16 - HolgerAngenent

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="PALMA:english"

Computations on Palma

Compiling Software

The module concept

Environment variables (like PATH, LD_LIBRARY_PATH) for compilers and libraries can be set by modules:

Command (Short- and Long-form) Meaning
module av[ailable] Lists all available modules
module li[st] Lists all modules in the actual enviroment
module show modulname Lists all changes caused by a module
module add modul1 modul2 ... Adds module to the actual environment
module rm modul1 modul2 ... Deletes module from the actual environment
module purge Deletes all modules from actual environment

To get acces to all modules, the following modules have to be added to the standard environment (this can be done by adding them to .bashrc):

module add shared
module add /Applic.PALMA/modules/path

Auf Palma ist MPI von Intel installiert. Um es nutzen zu können, muss ebenfalls ein Modul geladen werden.

module add intel/mpi/3.2.2.006
Die wichtigsten Kompileraufrufe lauten dann analog zum seriellen Intelcompiler mpiicc, mpiicpc, mpiifort usw.

Beispiel: Übersetzen eines Programms mit der FFTW2:

module add fftw/2.1.5
module add intel/cc/11.1.059
module add intel/mpi/3.2.2.006
mpiicc -I ${FFTW2_INCLUDE_DIR} -o Programm mpifftw2d.c -g -O3 -L${FFTW2_LIB_DIR} -lsrfftw_mpi -lsfftw_mpi -lsrfftw -lsfftw -lm
Zur Erklärung: Die Module shared und path müssen natürlich schon geladen sein. Durch das Modul fftw/2.1.5 werden laut "module show fftw/2.1.5" die Umgebungsvariablen FFTW2_INCLUDE_DIR und FFTW2_LIB_DIR gesetzt, die dann beim Kompilieren (auch im makefile) verwendet werden können.

Submittieren von Jobs

Für die Verteilung von Rechenjobs auf dem Cluster sorgen das Batchsystem TORQUE sowie der Scheduler Maui. Ein manuelles Starten von längeren Programmläufen auf den Rechenknoten oder auf PALMA am Batchsytem vorbei ist nicht erlaubt. Solche "streunenden" Prozesse werden ohne Vorwarnung vom Administrator beendet. Grundsätzlich sollten Batch-Jobs nur vom Dialogserver palma1 aus abgeschickt werden.

Erstellen des submit-files

Beispiel für ein submit-file eines MPI-Jobs:

#PBS -o output.dat
#PBS -l walltime=01:00:00,nodes=4:westmere:ppn=12
#PBS -A Projektname
#PBS -M Nutzername@uni-muenster.de
#PBS -m ae
#PBS -q default
#PBS -N Jobname
#PBS -j oe
cd $HOME/Jobverzeichnis
cp $PBS_NODEFILE $HOME/$PBS_JOBID.nodefile
mpdboot --rsh=ssh -n 2 -f ~/$PBS_JOBID.nodefile  -v
mpirun --rsh=ssh -machinefile $PBS_NODEFILE -np 32 ./Programmname

Hier wird ein MPI-Job gestartet, der mit insgesamt 32 Prozessen läuft. Dazu werden 4 westmere-Knoten mit je 12 Prozessorkernen angefordert.

Weitere Angaben:

  • Projektname: Muss unbedingt durch das eigene Projekt erstetzt werden, ansonsten startet der Job nicht.
  • Nutzername: Durch eigene Kennung ersetzen, damit Emails korrekt zugestellt werden können
  • Jobverzeichnis: Durch Pfad, in dem das Programm liegt, ersetzen.
  • Programmname: Hier den Namen der auszuführenden Datei eintragen.
  • walltime: Die nötige Zeit, die das Programm läuft, angeben. Nach Ablauf der Zeit wird der Job abgebrochen. Maximal 48 Stunden sind möglich.

Benötigt das Programm kein MPI, kann die Datei entsprechend vereinfacht werden.

Beispiel für ein submit-file eines openmp Jobs:

#PBS -o output.dat
#PBS -l walltime=01:00:00,nodes=1:westmere:ppn=12
#PBS -A Projektname
#PBS -M Nutzername@uni-muenster.de
#PBS -m ae
#PBS -q default
#PBS -N Jobname
#PBS -j oe
cd $HOME/Jobverzeichnis
./Programmname

Abschicken des Jobs/Verwaltung der Warteschlange

Ein Job wird gestartet, indem die oben erzeugte submit-Datei mittels

 qsub submit.cmd 
abgeschickt wird, wobei submit.cmd der Name der submit-Datei ist.

Weitere Befehle:

  • qstat: Zeigt die derzeitige Warteschlange an
  • qstat -a: Selbiges, nur mit Anzahl der angeforderten Knoten
  • qstat -n: Alle Knoten, die den Jobs zugeordnet sind, werden aufgeschlüsselt
  • qdel Jobnummer: Löscht Jobs aus der Warteschlange
  • showbf: Zeigt die Anzahl der freien Prozessorkerne

Die Auswahl der Rechenknoten

Mit der Option -l legt man die Ressourcenanforderungen für den Batch-Job fest. Bei der Festlegung der Anzahl und Art der Nodes ist zu beachten, dass die Rechenknoten des Systems zur Unterscheidung mit den Eigenschaften "nehalem" und "westmere" gekennzeichnet sind. Außerdem gibt es pro Node mehrere CPU-Kerne, so dass die Angabe nodes zwei unterschiedliche Bedeutungen haben kann. Die folgende Tabelle soll verdeutlichen, wie sich die Angabe nodes auf die Auswahl der Rechenknoten auswirkt:

Angabe im Batch-Job Auswahl der Rechenknoten
-l nodes=10 10 beliebige CPU-Kerne. Nicht zu empfehlen.
-l nodes=2:ppn=12 2 Westmere Knoten
-l nodes=1:ppn=8 8 Prozessorkerne eines Westmere oder Nehalem Knotens
-l nodes=2:nehalem:ppn=8+3:westmere:ppn=12 16 Prozessorkerne von 2 Nehalem Knoten und 36 Prozessorkerne von 3 Westmere Knoten
-l nodes=8:ppn=1 Fehlerhafte Zuteilung. Die Anzahl von ppn, wenn angegeben, sollte größer oder gleich 2 sein. Aufgrund eines Bugs in TORQUE erhält man für ppn=1 nicht das erwartete Ergebnis, sondern z.B. in diesem Fall 8 willkürlich über die HPC-Rechnenknoten verteilte Cores.

Generell sollten immer komplette Knoten angefordert werden, d.h. Angaben wie

nodes=x:nehalem:ppn=8
oder
nodes=x:westmere:ppn=12
gemacht werden.

Bei Bedarf können die einzelnen Chassis, in denen die Knoten stecken, auch direkt angefordert werden. Dazu ist statt der Angabe #PBS -q default die Angabe des Chassis nötig. Es kann zwischen folgenden Chassis gewählt werden: chassisd1, chassisd2, chassisd3, chassisd4, chassise1, chassise2, chassise3, chassisf1, chassisf2, chassisf3, chassisg1 wobei die Chassis d1 und d2 nehalem Prozessoren besitzen und das chassis g1 48GB statt 24 GB Hauptspeicher pro Knoten hat.

Um auf den SMP-Systemen zu rechnen, ist es derzeit noch notwendig, sich direkt auf diesen einzuloggen und die Jobs interaktiv zu starten. Die Namen der Knoten sind palma060, palma061, palma062 und palma063.

Die scratch Partition

Paralles speichern von Daten auf /scratch... in Vorbereitung

Unter /scratch befinden sich 180 TB Speicherplatz, der genutzt werden kann, um die in den Programmen generierten Daten zu speichern. Diese sollten aus Platz- und Geschwindigkeitsgründen nicht im Homeverzeichnis gespeichert werden. Als Dateisystem kommt lustre zum Einsatz, das einige Besonderheiten besitzt.

Es wird kein Backup dieser Partition durchgeführt!

-- HolgerAngenent - 2010-07-12

-- HolgerAngenent - 2010-08-16

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding ZIVwiki? Send feedback
Datenschutzerklärung Impressum