Massively-Parallel Self-Consistent Field Benchmarks
+
+
The scaling of the distributed-data SCF is illustrated,
+including some very fast in-core calculations on the CRAY-T3D (e.g.,
+the total time for a 324 function, DZP basis, SCF on C14F6H8 is
+just 5.5 minutes). The excellent scaling of large in-core
+calculations also proves that the direct algorithm will benefit fully
+from the introduction of faster integral technology into NWChem.
+
+
Background
+
+
As part of our High-Performance
+Computing and Communications Collaboration we have collected a set
+of benchmark calculations to use to help in establishing design goals
+and in tracking performance. A paper is in preparation which compares
+the performance of NWChem with other computational chemistry packages
+on a variety of parallel computers. The current document picks out
+certain interesting highlights.
+
+
Si108O194H81C5N - zeolite fragment
+
+
+
2053 functions, minimal basis
+
389 atoms, no symmetry
+
+
+
This system was of interest to PNNL researchers as an early
+cluster model for reactions inside minerals. It is a fragment of the
+zeolite ZSM-5, terminated with hydrogen atoms and with an embedded
+pyridine molecule. The high sparsity and small basis caused
+the previous version of NWChem to spend most of its time accessing
+global arrays, but the current version spends less than 1.6% of the
+time in global array accesses (CRAY-T3D, 256 nodes).
+
+
This calculation would not be possible with a replicated-data
+algorithm on many massively parallel computers since the Fock and
+density matrices require at least 34 Mbytes (storing lower triangular
+matrices). The distributed-data algorithm requires just 0.26
+Mbytes/node (on 256 processors, storing full square matrices) leaving
+much more memory to optimize the speed of integral evaluation.
+
+
The following table presents wall times in hours for complete
+SCF calculations on this system on several parallel computers. The
+SP1 at Argonne National Laboratory is unusual since it has
+SP1 compute nodes but uses an SP2 switch.
+
+
This medium-sized molecule is representative of many routine
+calculations and is also about the largest molecule without symmetry
+for which it is practical to cache all integrals in memory. The
+combination of massive parallelism and in-core algorithm makes for
+very high performance. The distributed data method makes over 200
+Mbytes more memory available for caching integrals than would be
+available using a replicated-data approach.
+
+
The following table presents total wall times in seconds
+for a complete SCF calculation on this system using a fully in-core
+algorithm on the NERSC CRAY-T3D which has 64 Mbytes of memory per
+node.
+
+
This cluster was provided by our industrial collaborators as being
+representative of systems being considered as catalysts. Due to bugs,
+previous versions of NWChem obtained only first-order convergence for
+this system. This problem has been addressed and full, stable
+second-order convergence is now obtained. In addition, optimizations in the use of linear algebra has
+significantly improved the scalability of this calculation.
+
+
The following table presents total wall time in minutes for this
+problem for the old and new versions of the code running on the KSR-2
+and CRAY T3D. The drop-off in performance is typical for small
+molecules and is due to poor load-balancing - the cobalt atom has
+many more functions on it than the other atoms. The performance could be
+enhanced by splitting large atoms, but this is only necessary for small
+molecules.
+
+
+ KSR T3D
+ ------------------------- -------------------------
+No. of Old New New Old New New
+proc.s Direct Direct In-core Direct Direct In-core
+------ ------ ------ ------- ------ ------ -------
+
+ 1 - 406.0 - - - -
+ 8 - 53.2 7.5 - - -
+ 16 38.0 29.6 4.5 - 31.0 -
+ 32 31.8* 18.8 2.9 23 17.7 2.3
+ 64 21.3** 12.3 2.8 15 10.9 -
+ 128 - - - 11 7.9 -
+
+ * = 36 processor time
+ ** = 72 processor time
+
+
+
+
Acknowledgments
+
+This work was performed under the auspices of the High Performance
+Computing and Communications Program of the Office of Scientific
+Computing, U.S. Department of Energy under contract DE-AC6-76RLO 1830
+with Battelle Memorial Institute which operates the Pacific Northwest
+National Laboratory, a multiprogram national laboratory. This work
+has made extensive use of software developed by the Molecular Science
+Software Group of the Environmental Molecular Sciences Laboratory
+project managed by the Office Health and Energy Research.
+
+This research was performed in part using the Caltech Concurrent
+Supercomputing Facilities (CCSF) operated by Caltech on behalf of the
+Concurrent Supercomputing Consortium. Access to this facility was
+provided by Pacific Northwest National Laboratory.
+
+
+
+Prepared by RJ Harrison: Email: nwchem-support@emsl.pnl.gov.
+
+
+
diff --git a/web/benchmarks/ddscf/bench.jpg b/web/benchmarks/ddscf/bench.jpg
new file mode 100644
index 0000000000..d32dc72eee
Binary files /dev/null and b/web/benchmarks/ddscf/bench.jpg differ
diff --git a/web/benchmarks/ddscf/ddscf.html b/web/benchmarks/ddscf/ddscf.html
new file mode 100644
index 0000000000..67f5bc7c20
--- /dev/null
+++ b/web/benchmarks/ddscf/ddscf.html
@@ -0,0 +1,196 @@
+
+
+
+
+distributed data scf
+
+
+
+
Distributed-Data Massively-Parallel Self-Consistent Field
+
+
Adrian T. Wong and Robert J. Harrison
+
+
Recent significant optimizations to our distributed data SCF
+algorithm are described. These result in superior scaling for smaller
+molecules and an up to 7 fold increase in performance in very large
+sparse systems.
+
+
Background
+
+
The simplest N-electron wavefunction in common use is a single
+antisymmetric product (or ``Slater-determinant'') of one-electron
+functions (or molecular orbitals, MOs) which are orthonormal linear
+combinations of the atomic orbital (AO) basis functions. The expansion
+coefficients are determined by minimization of the energy in the Self
+Consistent Field (SCF) approach. This simplest of wavefunctions
+determines many properties with surprising accuracy and recovers over
+99% of the total energy.
+
+
The computational kernel of the SCF algorithm is construction of the
+Fock matrix F by contracting the integrals (ij|kl)
+against the density matrix D
+
+ F = h + (1/2) SUM [ 2 (ij|kl) - (ik|jl) ] D
+ ij ij kl kl
+
+
+
Symmetry and sparsity in the matrix of integrals (ij|kl)
+must be exploited to avoid redundant computation. The major
+computation is evaluation of the non-zero integrals and the largest
+data requirements are due to the Fock and density matrices, both
+O(N*N) with N = O(1000). There are two major problems:
+
+
distributing and accessing the matrices so as to minimize
+ communication costs
+
maintaining load balance in the presence of sparsity and
+ large variation in the cost of integral evaluation.
+
+
+
Improved distributed data algorithm
+
+
Our distributed data algorithm is modeled after that of a blocked
+matrix-multiply algorithm for a high-performance workstation with a
+small fast memory cache. This is because non-uniform memory access
+(NUMA) is the fundamental performance issue for both workstations and
+MPPs. The cost of accessing an element of the distributed density or Fock
+matrices must be offset by using this element multiple times. To
+achieve this re-use of data we simply stripmined the fourfold loop over
+the basis function indices. Since the computation is a quartic
+function of the block size while the communication is only a quadratic
+function, the block size may be adjusted to ensure the computation time
+dominates the communication time.
+
+
We previously stripmined by grouping basis functions
+according to their (usually atomic) centers. This had the advantage
+that the geometrical sparsity may be exploited in the outer
+stripmining loops. However, the granularity was fixed and was
+inadequate for calculations on large molecules in small basis sets,
+or when using machines with very high latencies. The revised
+algorithm groups atoms into blocks, looping over quartets of
+blocks of atoms, and then evaluating all integrals within a quartet of
+blocks of atoms. Sparsity may still be exploited in the outer loops
+and the performance gains include
+
+
+
many fewer, larger messages,
+
elimination of sensitivity to message-passing latency,
+
reduction of communication costs to less than one percent, and
+
the ability to vectorize integral computation over many thousands
+ of atomic quartets.
+
+
+
All calculations will benefit from the vectorization of integral
+evaluation, but most calculations benefit by only 10% from the
+reduction in communication. However, the time to solution for our
+ZSM-5 zeolite fragment test-case (389 atoms, 2053 function, STO-3G
+basis, no symmetry) on 64 nodes of the IBM SP2 was reduced from over
+40 hours to less than six. This is because the minimal basis set and
+high latency of the IBM violated all the criteria for good performance
+with the old algorithm. With the new code the Fock-build is completely
+dominated by integral computation.
+
+
The global array tools make it very easy to express
+blocked algorithms since the library provides routines that enable
+each process to access patches of arrays independently of all other
+processes. This asynchrony makes the program easier to write and also
+more efficient since there are no unnecessary synchronizations between
+processes. In addition, since the data is readily accessed by any
+process we can use full dynamic load balancing, implemented by using
+an element of a global array as a shared counter.
+
+
Conventional SCF with memory caching
+
+
Previous versions of NWChem have been fully-direct. The latest
+version includes the ability to perform conventional SCF calculations,
+caching integrals in memory where possible and storing additional
+integrals on disk. The blocking described in the above section is
+constrained to keep blocks less than 256 functions, and we can thus
+pack integral labels one per byte independent of the actual
+basis dimension. Data compression is also used to compress the
+integral values to 32 bits, maintaining a fixed overall precision
+which defaults to 10^-9. Thus, an integral and its labels
+are packed into 64 bits, a 33-100 percent saving over most other codes.
+
+
The frugal memory usage of the distributed data algorithm also
+permits much more memory to be used to cache integrals in memory. For
+instance, in the 324 function in-core calculation described in the benchmarking section, a replicated-data approach
+would have wasted over 200 Mbytes of memory that NWChem uses to
+cache integrals.
+
+
This functionality will be extended to include semi-direct algorithms.
+
+
+
Reducing linear algebra overhead
+
+
Linear algebra components in a conventional SCF algorithm include
+
+
matrix diagonalization,
+
matrix multiplication and
+
orthogonalization.
+
+To this list our quadratically convergent algorithm adds
+
+
matrix exponentiation.
+
+With PeIGS and ScaLAPACK we now have
+quite efficient matrix diagonalization and orthogonalization
+routines, but these operations still take much longer than a matrix
+multiplication. This is because the latter both parallelizes more
+readily and also drives a single processor at peak speed.
+
+
We have restructured the SCF code to rephrase as much linear
+algebra as possible as matrix multiplication operations. Most novel
+is the new matrix exponentiation routine that provides an accurate
+unitary approximation, which eliminates the necessity of
+orthogonalization and stabilizes the initial search step when large
+rotations are applied. The new matrix exponentiation algorithm (a
+modification of one due to Moler) is
+
+
scale the matrix by a power of two so the largest element is less
+ than 0.01,
+
use a 3rd order taylor series approximation to the exponential,
+
repeatedly square the result to undo the scaling, and
+
if the largest element of the original was greater than 0.001
+ apply one iteration of a quadratically convergent symmetric
+ orthogonalization (two matrix multiplications).
+
+This is tailored to the antisymmetric matrices typical of our
+applications and may not be stable for more general matrices.
+
+
The net result is that in a typical calculation
+
+
one general real symmetric diagonalization is performed (to
+ generate orthonormal, energy weighted starting guess orbitals),
+
one or two real symmetric diagonalizations (to make the
+ orbital-Hessian more diagonally dominant, and to
+ canonicalize the final orbitals),
+
one or more matrix exponentials are performed each
+ macro-iteration, and
+
no orthogonalizations are performed.
+
+As a result, small and in-core calculations are up to 30% improved on
+large numbers of processors.
+
+
Acknowledgments
+
+
This work was performed under the auspices of the High Performance
+Computing and Communications Program of the Office of Scientific
+Computing, U.S. Department of Energy under contract DE-AC6-76RLO 1830
+with Battelle Memorial Institute which operates the Pacific Northwest
+National Laboratory, a multiprogram national laboratory. This work
+has made extensive use of software developed by the Molecular Science
+Software Group of the Environmental Molecular Sciences Laboratory
+project managed by the Office Health and Energy Research.
+
+
+
+Prepared by RJ Harrison: Email: nwchem-support@emsl.pnl.gov.
+
+
+
diff --git a/web/benchmarks/dft/back.html b/web/benchmarks/dft/back.html
new file mode 100644
index 0000000000..30f843e8bf
--- /dev/null
+++ b/web/benchmarks/dft/back.html
@@ -0,0 +1,44 @@
+
+
+
+
+
+
+ back
+
+
+Help and support from the following individuals and institutions was
+ provided durring the collection of these benchmark data:
+
+The following wall clock times and scaling characteristics have been
+obtained for molecular dynamics simulations of liquid water of four
+different system sizes, 5184, 17496, 41472 and 82000 atoms respectively.
+The simulations used the SPC/E water model and a simple cutoff radius
+of 1.8 nm, and were run on the IBM SP2.
+Timings and scaling illustrate the need for sufficient computational
+load to obtain scalability. From these benchmark results, good scaling
+with the number of processors is obtained when each processor has at
+least in the order of 100 atoms.
+
+
+
+
+
+
Relative free energy of Na and K complex with 18-Crown-6
+
+>
+
+Free energy data and wall clock times to obtain them on the IBM SP2
+and CRAY T3E-900
+are given for the free energy difference calculation of Na and K in
+aqueous solution and in the complex with the 18-crown-6 crownether
+molecule in aqueous solution. The system has 6382 atoms.
+The calculated free energy difference
+of complexation is 6±4 kJ/mol from the multiconfiguration
+thermodynamic integration (MCTI) approach and 5±5 kJ/mol. The
+experimental result is 7 kJ/mol.
+
+
+
+
+
+
+
Haloalkanedehalogenase
+
+Wall clock time and speedup as a function of the number of processors
+used on the IBM SP are given for a simulation of haloalkane dehalogenase
+in aqueous solution. The system has 41,259 atoms and was simulated using
+the AMBER force field and SPC/E water model, a cutoff of 1.0 nm and
+included long range electrostatic energy and force corrections using the
+particle-mesh Ewald method on 64³ grid. To run this job, download
+the files had_md.nw,
+ had.top.gz, and
+ had_md.rst.gz.
+
+
+
+
+
+
+
1,2-Dichloroethane
+
+Wall clock time and speedup as a function of the number of processors
+used on the IBM SP and Cray T3E-900 are given for a simulation of a
+droplet of 1,2-dichloroethane in aqueous solution.
+The system has 100,369 atoms and was simulated using the force field
+parameters for chlorinated alkanes as developed by Paulsen and the
+SPC/E water model, and a simple cutoff of 2.4 nm.
+1,2-Dichloroethane is one of the main pollutants on DOE and
+other sites.
+
+
+
+
+
+
+
Liquid octanol
+
+Wall clock time and speedup as a function of the number of processors
+used on the Cray T3E-900 are given for a simulation of liquid octanol.
+The system has 216,000 atoms and was simulated using AMBER force field
+parameters and the SPC/E water model, and a simple cutoff of 2.4 nm.
+
+
+
+
+
+
+
+
+
+
SCF Benchmark
+
+Wall clock and CPU speedup as a function of the number of processors
+used on an IBM SP with 150 MHz nodes. The total disk space used is
+also plotted. This is a semi-direct
+distributed data calculation and so more disk
+space is used as it becomes available.
+
+
+
+
+
+
+
+
DFT Benchmarks
+
+Wall clock speedups as a function of the number of processors
+used on various platforms for LDA calculations of three zeolite fragments
+(347, 1687 and 3554 basis functions). If you would like to run the benchmark
+yourself, please download the files siosi3.nw,
+ siosi6.nw and siosi7.nw.
+
+
+Please note:
+The calculations must be completly in-core; to check this, you need
+to search for the string in-core and get this kind of output
+
+grep in-core siosi3.out
+ The percent of 3c 2e- integrals held in-core is: 100.00
+
+ Acknowledgments
+
+
SIOSI3 (Si8O7H18)
+
+
+
+
+
+
SIOSI6 (Si28O67H30)
+
+
+
+
+
+
SIOSI7 (Si75O148H66)
+
+
+
+
+
+
MP2 Gradient Benchmark
+
+CPU times for an MP2 gradient calculation as a function of
+the number of processors used for the
+(H2O)7 molecule.
+If you would like to run the benchmark
+yourself, please download the file h2o7.nw
+
+
+
+
+
CPU times for different portions of an MP2 gradient as a function of
+the number of processors used on an IBM SP with 120 MHz nodes for
+a potassium crown ether.
Our parallel eigensolver package is called PeIGS, and solves the
+standard and generalized, real, symmetric eigenproblems. It is based
+on bisection for eigenvalues and repeated inverse iteration and
+reorthogonalization for eigenvectors. The parallel performance is
+competitive with alternative methods for typical "chemical" problem sizes, and is
+illustrated with three applications from quantum chemistry
+
+
SCF-Hartree-Fock (Fock matrix for 389 atom zeolite fragment, 2053 functions)
+
+
density functional theory (charge-fitting matrix for (SiOSi)6, 1687)
+
+
resolution-of-the-identity Moller-Plesset theory (biphenyl
+charge-fitting matrix, 966)
+
+The distinguishing feature of the repeated inverse iteration and
+orthonormalization method used by PeIGS is that orthogonalization is
+performed across multiple processors as dictated by the spectrum.
+The correct, orthonormal result is always generated.
+
+
For a spectrum that is well spaced, there is essentially no
+orthogonalization time. Most of the time is consumed in the
+Householder reduction to tridiagonal form. For large numerically
+degenerate clusters, almost all of the time is consumed in the
+Householder reduction and in orthonormalization.
+
+
+
+
+
+
Acknowledgments
+
+
This work was performed under the auspices of the High Performance
+Computing and Communications Program of the Office of Scientific
+Computing, U.S. Department of Energy under contract DE-AC6-76RLO 1830
+with Battelle Memorial Institute which operates the Pacific Northwest
+National Laboratory, a multiprogram national laboratory. This work
+has made extensive use of software developed by the Molecular Science
+Software Group of the Environmental Molecular Sciences Laboratory
+project managed by the Office Health and Energy Research.
+
+
This research was performed in part using the Caltech Concurrent
+Supercomputing Facilities (CCSF) operated by Caltech on behalf of the
+Concurrent Supercomputing Consortium. Access to this facility was
+provided by Pacific Northwest National Laboratory.
+
+
+
+Prepared by RJ Harrison: Email: nwchem-support@emsl.pnl.gov.
+
+
+
diff --git a/web/benchmarks/peigs/pinkpig.jpg b/web/benchmarks/peigs/pinkpig.jpg
new file mode 100644
index 0000000000..440e28ec6c
Binary files /dev/null and b/web/benchmarks/peigs/pinkpig.jpg differ
diff --git a/web/benchmarks/peigs/pinkpig2.jpg b/web/benchmarks/peigs/pinkpig2.jpg
new file mode 100644
index 0000000000..6e1b1533fe
Binary files /dev/null and b/web/benchmarks/peigs/pinkpig2.jpg differ
diff --git a/web/benchmarks/peigs/pinkpig3.jpg b/web/benchmarks/peigs/pinkpig3.jpg
new file mode 100644
index 0000000000..9d7b7815a4
Binary files /dev/null and b/web/benchmarks/peigs/pinkpig3.jpg differ
diff --git a/web/benchmarks/peigs/qc_speedup.gif b/web/benchmarks/peigs/qc_speedup.gif
new file mode 100644
index 0000000000..633edba204
Binary files /dev/null and b/web/benchmarks/peigs/qc_speedup.gif differ
diff --git a/web/benchmarks/peigs/qc_times.gif b/web/benchmarks/peigs/qc_times.gif
new file mode 100644
index 0000000000..362ff6800e
Binary files /dev/null and b/web/benchmarks/peigs/qc_times.gif differ
+
>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 
+
+