+
+
Much of the information below has been exracted from information +available through the web at ANL or IBM SP man pages by Robert Harrison +(rj_harrison@pnl.gov). + +
For NWChem support mail nwchem-support@emsl.pnl.gov or +visit the NWChem +homepage. +
+ +
For support on SP specific issues contact spsupport@mcs.anl.gov. + +
ANL use their own environment (EASY) for scheduling interactive and +batch use of the machine. You need to know about just a few facts and +commands to get going. + +
You will log into either bonnie or +clyde.mcs.anl.gov which are not really part of the SP2 - they +are 'front-ends'. Information on the SP, examples etc. can be found +on the MCS SP +Web pages. + +
The command sphelp gives a useful summary of SP specific +commands and is a good starting point ... here is the sp command +summary +
+ node# 0|1 username JID JobTerminationTime (MMDDhhmm) + 0 = node is available, 1 = in use. ++ +
To run either interactively or with batch you have to schedule +the resources using spsubmit. By mail you'll be told when the +resources become available. If running interactively, you can then +login into the nodes you've been assigned (details below) or if +running in batch you provided spsubmit a script to run (provided for +NWChem). From prior experiences, you may experience a significant +delay in receiving mail that you have the nodes. The use of xspusage +is best (see below). If you don't have X, use spq. + +
If a node is not currently assigned to you then you will not be able +to log into it. + +
Each node of the SP is a Power 1 cpu with 128 MB of physical +memory and a local scratch disk named /tmp of about 0.25 to +0.4 GB. The O/S and I/O buffers consume about 10 MB (estimate), and +the NWChem executable is 5 MB, leaving about 113 MB to be allocated. +The local scratch disk is wiped clean at the end of each job. Thus, +in order for useful files to be saved by default and to make sure that +scratch files with high bandwidth requirements are in /tmp, +you should always +
Do this first - it's a lot less confusing than running interactively. + +
In the directory /sphome/harrison/hpcci are several files +
Copy one of the example input files into your login directory +(e.g., /sphome/harrison/hpcci/examples/scf_h2o.nw - input for a +conventional SCF calculation on water) into your login directory. +(This input file is also commented and tailored to the SP). + +
To run the calculation in batch you must use spsubmit to +
+ runnwchem /tmp /sphome/harrison/test.nw ++or +
+ runnwchem /tmp /sphome/harrison/test.nw /sphome/harrison/test.out ++ +
Spsubmit prompts for the parameters it wants ... here is +a sample dialog that runs a five minute nwchem job on eight nodes in +batch (when you run this use your own username instead of harrison). +
+ /sphome/harrison % spsubmit + ********************************************************************* + 112 nodes are available + 16 nodes are allocated for I-Way use + + Please report any problems to spsupport@mcs.anl.gov + ********************************************************************* + Charge Allocation Category: [default harrison] + Maximum Wall-clock Run-Time (minutes): (1-???) 5 + Number of Nodes Required: (1 - 128) 8 + CAC: "harrison" *now* has 4397 RUs available, after committing. + (I)nteractive (B)atch: B + Job Classification (M)PL, (T)ask Farm, (P)4: M + IP over the switch [y/n]: Y + Full path to Shell Script/Program: /sphome/harrison/hpcci/runnwchem + Command Line arguments for your job: /sphome/harrison \ + /sphome/harrison/scf_h2o.nw + (C)ommit or (A)bort: C ++ +
In order to be crystal clear, I typed (comments in parentheses) +
+ spsubmit (the command) + carriage return (to accept charging to the default account) + 5 (for a 5 minute job) + 8 (for 8 processors) + B (for batch) + M (for MPL) + Y (to let IP also run over the switch) + /sphome/harrison/hpcci/runnwchem (name of script) + /sphome/harrison /sphome/harrison/scf_h2o.nw (runnwchem arguments) + C (to commit) ++ +
The script /sphome/harrison/hpcci/job runs +spsubmit from input in a file - take a copy and edit it to +automate use of spsubmit. + +
The command line arguments given to the runnwchem script +specify the work directory (/sphome/harrison) and the full +path to the input file (/sphome/harrison/scf_h2o.nw). + Runnwchem will leave the output in +/sphome/harrison/scf_h2o.out. + +
Since it can take a large elapsed time to get a job started five +minutes is the practical lower limit that should be used for the job +time. + +
The command spq will indicate the position of the job in +the queue and its status. When the job is started you'll receive mail +indicating the nodes actually allocated. At the end of the job you'll +receive another mail message providing the output of the shell script. +Only the NWChem output goes into the output file. The advantage of +doing this is that you can examine the output while the job is +executing. The unix command 'tail -f filename' is useful to +watch a job run. + +
Note that only one node (the master) executes the script +and that only the commands run with POE are actually executed in +parallel. This causes seemingly simple things to fail. For instance, +if you request to run in a subdirectory of /tmp (e.g., +/tmp/rjh) then runnwchem will make this directory +and cd into it. Subsequent POE commands will fail since this +directory will not be present except on the master node. + +
If you want to run multiple NWChem calculations in the same job +you'll have to modify the runnwchem script. Basically, this +just requires replicating the line running poe and specifying +the correct input and output file on each line. Make sure that you +leave the sprelease at the end of the file since it ensures +that you won't be charged for more time than you use. + + +
Again, you use spsubmit to reserve nodes for your use, +but this time specifying interactive access. In this instance it will +not prompt you for a script or arguments. Mail will notify you when +your nodes are available. Alternatively, you can keep looking with +spq to see when your job starts. There is also a graphical +program (xspusage) that names nodes according to who 'owns' them. + +
The mail message, or the file SPnodes.$JOBID in your login +directory (you can get the jobid from the output of spsubmit), or +xspusage tell you which nodes you have access to. Simply +telnet to one of these and login with your usual id and password, or +with xspusage you just click on a node you own to open an +xterm on that node. + +
Now that you're logged into the partition you can run NWChem using +the /sphome/harrison/hpcci/interactive shell script. This +differs from the runnwchem script only in that output comes +to the terminal and the sprelease has been removed. Thus, +you can run multiple calculations in one interactive session. + +
Don't forget to do an sprelease with the correct +jobid after completing the calculations otherwise you will be charged +for all the time that you booked, rather than just the time that you +used. Once sprelease is issued you no longer own the nodes +and you will be unable to log into them. + +
E.g., +
+ (... log into spnode ...) + interactive /tmp/ /sphome/harrison/hpcci/examples/scf_h2o.nw + interactive /tmp/ /sphome/harrison/hpcci/examples/mp2_h2o.nw + interactive /tmp/ /sphome/harrison/hpcci/examples/dft_h2o.nw + sprelease 070714025108 + exit ++ +
Xspusage is the easiest way to run interactively. Before +using xspusage make sure that your DISPLAY +environment variable is correctly set (hostname:0.0) and that +your local X server permits remote window creation (use the command +'xhost +' in a local shell). + + +
Keep the email with the runnwchem shell script output +since this can be used to identify nodes with problems. + +
I've had this happen when I've specified a work directory that is +not accessible to all processes (e.g., a subdirectory of +/tmp). This often works for a single process, but fails for +multiple processes since only the master process has access to this +directory. Rerun the job specifying either just /tmp or a +subdirectory of your login directory. + +
This was a problem with incomplete instalation of a system +Fortran upgrade. It should be fixed, but if the problem appears +contact spsupport@mcs.anl.gov sending them the shell script +output and the output in your logfile. They may ask you to rerun it +with the MP_INFO variable increased to 3 or 4 in order to +generate debug information. To do this take a copy of the runnwchem +script and edit it. + +
I haven't seen this problem but apparently it can arise from +excessive load from others on the Ethernet. Just resubmit the job. + +
Some of the lower numbered nodes apparently have some system junk +sitting in the /tmp filesystem that is not automatically +deleted. You can rely on only about 0.25 GB per node, though some +will have up to 0.4 GB. + +
Development accounts have just 125 node hours per quarter - I blew +this in two days. The command 'cacReport -u userid' reports +the time available to you. The accounting is not rigorously enforced +and a request to spsupport to increase your time will usually be +granted. + +
Only the user that currently has allocated an SP node can log +into it. This is done by restricting entries in the node's +/etc/passwd file. A side effect of this is that an SP node has no +knowledge of other users. Hence, the usual shell expansion mechanism +for specifying a login directory by prefacing a user name with a tilde +will not work, unless you are referring to yourself. The rule of +thumb is always to use fully qualified filenames. + +
Report NWChem problems, suggestions, feedback, etc., to +nwchem-support@emsl.pnl.gov or via the NWChem +home page. + +
There is a mailing list for NWChem users that can be used for +announcements and discussion amoung users. To subscribe send email to +majordomo@emsl.pnl.gov with the body 'subscribe +nwchem-users'. You can do this with the following command +
+ echo subscribe nwchem-users | mail majordomo@emsl.pnl.gov ++ +
To post a message to the mailing list send it to +nwchem-users@emsl.pnl.gov. + + +
Thanks to Dave Bernholdt and Mike Minkoff for useful input to +this document and to Jeff Tilson for an initial version of the +execution scripts. + +