Search JPL

Hardware

-	Dell Xeon Clusters
	+	Software Guide
	-	User's Guide
	+	2 GB Size Limitations
+	SGI Altix 3700 Systems
+	Data Storage
+	Visualization

Using the Dell Xeon Cluster System at JPL

Introduction

The User's Guide for the JPL Dell Xeon Cluster Supercomputer is intended to provide the minimum amount of information needed by a new user of this system. As such, it assumes that the user is familiar with many of the standard aspects of supercomputing such as the Unix operating system, Fortran and C programming languages, and various standard libraries (BLAS, LAPACK, MPI, etc.).

The JPL Dell Xeon Cluster Supercomputing facility is funded by JPL and is available to users at JPL. The computer system is located in the Supercomputing Center, and is supported by JPL's Supercomputing and Visualization Systems Group.

Getting an account

A user account for this machine can be obtained by completing an application at the Account Applications page.

Getting help

User questions and support are handled online by sending e-mail to: scconsult

The Dell Xeon Cluster Hardware at JPL

The system has 512 nodes, with two Intel 3.2 GHz Xeon processors on each node, for a total of 1024 processors. There is 2GB RAM per processor, for a total or 2TB RAM. 496 nodes are available for computation, the remaining 16 nodes are used for I/O.

Interactive editing, compiling, and very simple debugging is done on the headnode, cosmos. Production computing is done on the 496 compute nodes using the batch queueing system PBS Pro, version 10.0.0. PBS Pro supports both interactive and background batch. See the "Batch" discussion below.

Operating System

The operating system is RedHat Linux Enterprise Edition 4.4. We assume that the user is familiar with Linux; if not there are many web pages available on line to help a new user get started with Linux.

Disk Details

Home directories are NFS mounted on every node. Each user has a /home quota of 1GB of disk space. Home directories are backed up nightly.

All nodes also have nfs mounted work directories from the IO nodes on the system. Currently /work00-/work07 are available. These are RAID5 disks, and each has a capacity of 255GB. The work directories have a quota of 100 GB per project, and are never backed up.

Additionally, all nodes have 53 GB of local scratch space in /lscratch. /lscratch will be scrubbed after each batch run, so if users wish to use /lscratch, they will need to stage their data in and/or out of that area during their runs.

Environment

We are using modules to switch between compilers. Here are some basic module commands:

module list: lists currently loaded modules
module avail: lists modules available to load
module help <name>: tells what the module is/does/loads
module unload <name>: unloads the specified module
module load <name>: loads the specified module
module switch <oldname> <newname>: places <newname> as the complier and removes <oldname>

When users first log in, the module "latest_intel101" is loaded automatically.

Compiling

To compile your MPI applications, use the following scripts:

mpicc <scriptname.c> using Intel's C/C++ compiler, icc

mpif90 <scriptname.cpp> using Intel's Fortran90 compiler, ifort

Batch Scheduling

As with any supercomputer, the fair and efficient use of CPU time is an important concern for users. A batch queue system is meant to address these issues. We are using Altair's Portable Batch System Professional (PBS Pro) version 10.0 for our batch queuing system. Jobs MUST be run using the PBS Pro batch system.

PBS Pro Commands

There are many commands associated with PBS Pro. Man pages are available for most of them. The most important commands for a new user to learn are:

qstat

Display status of PBS batch jobs, queues, or servers.

qstat -q

Gives information for available queues.

qstat -Q

This command displays queue status in default format.

qstat -a

Gives the status of all jobs on the system.

qstat -n

Lists nodes allocated to a running job in addition to basic information.

qstat -f <PBS_JOBID>

Gives detailed information on a particular job. Use "qstat -nf" to monitor jobs.

qdel <PBS_JOBID>

This deletes one or more unfinished batch jobs.

qsub

This command submits a job for execution. Please see below for details on qsub usage. Submit jobs with "qsub -S /bin/sh script"

PBS Pro Queues

PBS Pro allows for the placement of batch queue jobs based upon the availability of a variety of resources. The resources of most importance are: number of processors and wallclock time. Additionally, fairshare scheduling is being used to determine the order in which jobs are executed.

Currently, the following queues are set up:

Note:

The queue system has been designed to provide a fair and efficient service to all users. The queues are given a priority value and there is no pre-emption. That is, once a job starts running, it will run until completion. However, be advised that a job with a higher priority will run before a job with a lower priority, provided that both jobs are waiting to be run. The scheduler will order jobs waiting to be run according to their priority value, not arrival time. For example, if a low priority job has been waiting there for a long time and a new job arrives that has higher priority, then the new job will run first. Please note that a job with low priority can potentionally starve if higher priority jobs keep arriving before the low priority job can start running.

debug, priority=40

This queue allows the use of up to 32 processors for up to 60 minutes, and is available at all times. Very simple debugging may be done on the headnode, cosmos. However, debugging multi- processor jobs, MPI jobs of any type, and CPU intensive jobs must be done in the batch queue system. This queue can be used for "interactive batch" jobs so that the user can interact with a debugging tool (such as totalview).

shortq, priority=30

This queue allows the use of up to 128 processors for up to 3 hours, and is available at all times.

longq, priority=20

This queue allows the use of up to 128 processors for up to 12 hours.

priorityq, priority=50, restricted use

This queue is a high priority queue and can only be used by group 'gba' (admin group). This queue allows unlimited processors for unlimited time.

specialq, priority=10, all users allowed

This queue allows the use of unlimited processors for unlimited time, at the lowest priority.

Jobs submitted to this queue will run provided that there are no other jobs with higher priority waiting to be executed.

Submitting a Job

qsub: How to submit a batch job

The basic command for submitting a batch queue job to PBS Pro. Although there are a multitude of options to qsub (see the man page), there are only a few options that the average user will commonly use:

-l select=number-of-resources:ncpus=number-of-cpus

This option sets the number of processors to be used on this job and MUST BE USED on EVERY qsub command.

-l walltime=hh:mm:ss

This option sets the maximum amount of wallclock time to be used by this job and MUST BE USED on EVERY qsub command.

-q queue

Allows jobs to pick their queue criteria as specified above.

-o outfile

This option reroutes the standard output to outfile. If -o is used without -e, the standard error of the job is stored in outfile. -o and -I are mutually exclusive.

-e errfile

This option reroutes the standard error output to errfile.

-I

This option specifies that this job is to be an interactive batch job. Standard error, input, and output will be connected to your terminal. This is most useful when doing interactive debugging in the debug queue.

Here is an example of a qsub scriptfile for MPI:

 #!/bin/csh
 #PBS -l select=128:ncpus=1
 #PBS -l walltime=1:00:00
 #PBS -o /work01/gba/admin/outfile
 #PBS -e /work01/gba/admin/errfile
 #PBS -q shortq
 setenv PBS_O_WORKDIR /work01/gba/admin
 cd $PBS_O_WORKDIR
 setenv OMP_NUM_THREADS 1
 limit stacksize unlimited
 limit coredumpsize 0
 /opt/mpich/gm/intel10.1/bin/mpirun -np 128 -machinefile $PBS_NODEFILE ./executable

Launch this scriptfile as follows:

qsub scriptfile

Printer Friendly Version

Site Manager: Chris Catherasoo
Webmaster: Krista Andersen, Dana Runge
Last Modified: Wednesday, July 22, 2009