From 9c4e267cea381c49c429659f08962eeaf3ff47e5 Mon Sep 17 00:00:00 2001 From: Judith M Cuta Date: Fri, 11 Dec 1998 18:57:08 +0000 Subject: [PATCH] updated with some input from Joop; plus minor edits -- jmc 12/11/98 --- doc/prog/ga.tex | 98 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 93 insertions(+), 5 deletions(-) diff --git a/doc/prog/ga.tex b/doc/prog/ga.tex index f831d1d5a6..aba57a58fe 100644 --- a/doc/prog/ga.tex +++ b/doc/prog/ga.tex @@ -15,6 +15,9 @@ model. The Global Array (GA) toolkit provides an efficient and portable Each process in a MIMD parallel program can asynchronously access logical blocks of physically distributed matrices without need for explicit cooperation by other processes. +The trade-off with this approach is that +access to shared data will be slower than access +to local data, and the programmer must be aware of this in designing modules. From the user perspective, a global array can be used as if it was stored in the shared memory. Details of the data distribution, addressing and @@ -34,6 +37,87 @@ on the web site http://www.emsl.pnl.gov:2080/docs/global/. Currently support is limited to 2-D double precision or integer arrays with block distribution, at most one block per array per processor. +\subsubsection{Interaction Between GA and MA} + +Available global (GA) +and local (MA) memory can interact within NWChem in only two ways, + +\begin{enumerate} +\item GA is allocated within MA, and GA is limited only by the available + space in MA. + +\item GA is not allocated within MA, and GA is limited at initialization + (within NWChem input this is controlled by the MEMORY directive) +\end{enumerate} + +If GA is allocated within MA, then +the available GA space is limited to the currently available MA space. This +also means that the total allocatable memory for GA {\em and} MA must be +no more than the available MA space. +If GA is not allocated within MA, then local and global arrays occupy essentially +independent space. The allocatable memory for GA is limited only by the available +space for GA, and similarly, the allocatable memory for MA is limited only +by the available local memory. + +When allocating space for GA, +some care must be exercised in the treatment of the information returned by +the routine \verb+ga_memory_avail()+, whether or not +the allocation is done in MA. The routine \verb+ga_memory_avail()+ +returns the amount of memory (in bytes) +available for use by GA in the calling process. +This returned value must be converted to double precision words when +using double precision. +If a uniformly distributed GA is desired, it is also necessary to find +the minimum of this value across all nodes. This value will in general be +a rather large number. +When running on a platform with many nodes and having a large memory, +the agreggate GA memory, even in double precision words, could be a large enough +value to overflow a +32-bit integer. Therefore, for calculations that require knowing the size of +total memory, it is advisable to first store the size of memory on each node +in a double precision +number and then sum these values across all the nodes. + +The following pseudo-code illustrates this process for an application. + +\begin{verbatim} +#include "global.fh" +#include "mafdecls.fh" + + integer avail_ma, avail_ga + + avail_ma = ma_inquire_avail(mt_dbl) + avail_ga = ga_memory_avail()/ma_sizeof(mt_dbl,1,mt_byte) + + if (ga_uses_ma()) then +c +c available GA space is limited to currently available MA space, +c and GA and MA share the same space +c + allocatable_ga + allocable_ma <= avail_ma = avail_ga + + else +c +c GA and MA are independent +c + allocatable_ga <= avail_ga + allocatable_ma <= avail_ma + + endif +c +c find the minimum value of available GA space over all nodes +c + call ga_igop(msgtype,avail_ga,1,'min') +c +c determine the total available GA space +c + double precision davail_ga + davail_ga = ga_memory_avail()/ma_sizeof(mt_dbl,1,mt_byte) + call ga_dgop(msgtype,davail_ga,1,'+') + +\end{verbatim} + + \subsubsection{List of GA Routines} The following routines are invoked for operations that are globally collective. @@ -44,7 +128,8 @@ simultaneously invoked by all processes as if in SIMD mode. \item {\tt ga\_initialize\_()} --- initialize global array internal structures \item {\tt ga\_initialize\_ltd(mem\_limit)} --- initialize global arrays and set - memory usage limits + memory usage limits (note: if \verb+mem_limit+ is less than zero specifies + unlimited memory usage.) \begin{itemize} \item integer {\tt mem\_limit} --- [input] GA total memory ( specifying less than 0 means "unlimited memory") @@ -430,12 +515,15 @@ mode where only compute processes participate \item character {\tt op} --- [input] \end{itemize} -\item {\tt ga\_igop(type, x, n, op)} --- equivalent to TCGMSG igop, for use in data-server mode -where only compute processes participate +\item {\tt ga\_igop(type, x, n, op)} --- equivalent to TCGMSG {\tt igop}, for use in data-server mode +where only compute processes participate; performs the operation specified by the input variable +{\tt op} (supported operations include addition, multiplication, maximum, minimum, +and maximum or minimum of the absolute +value), and returns the value in {\tt x}. \begin{itemize} \item integer {\tt type} --- [input] integer handle of array \item integer {\tt n} --- [input] -\item double precision {\tt x} --- [input] +\item double precision {\tt x} --- [input/output] \item character {\tt op} --- [input] \end{itemize} @@ -468,7 +556,7 @@ Other utility operations: \item {\tt ga\_inquire\_memory\_()} --- find the amount of memory in active arrays -\item {\tt ga\_memory\_avail\_()} --- find the amount of memory left for +\item {\tt ga\_memory\_avail\_()} --- find the amount of memory (in bytes) left for GA \item {\tt ga\_summarize(verbose)} --- prints summary info about allocated arrays