\section{Makefiles and Libraries}
\label{sec:makelib}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NOTE: the first part of this section (up to the discussion of the
% environmental variables NWCHEM_TOP and NWCHEM_TARGET)
% is adapted from the current (as of 9/29/98) version of
% the file Build.notes, from ~/doc/ in the CVS repository.  If Build.notes
% has been updated since, this section should be updated, too.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


The working assumption in documenting the NWChem code is that the
developers will be relatively sophisticated hackers, used to deciphering
C and Fortran source code and makefile scripts.  The steps for building
an executable version of the code are as automated as is practical, with
upward of a hundred individual makefiles in the source code directory
tree.  The developer is advised to look carefully at the top-level
makefile
{\tt ...src/config/makefile.h} to get a feel for what this package can
be built for and for some instructions about variables that must be set
before invoking {\tt make}.

The makefiles will work only with GNU {\tt make}, and it must be the one
that appears first in your path, otherwise dependent "makes" will not
invoke the right one.

The code in the CVS repository is set up to run on machines for
which working floating point precision is DOUBLE PRECISION.  If your
platform wants REAL, you need to do a {\tt make dbl\_to\_sngl} before
anything else.  (Remember to do {\tt make sngl\_to\_dbl} {\em before} checking
anything back into the repository.  Doing this before updating from
the repository helps avoid problems too.  Since code comes out of the
repository set up for DOUBLE PRECISION, you will probably have
to convert after your update anyway.)

Which routines are subject to precision conversion is governed by a
{\tt USES\_BLAS} macro in each makefile.  Only files listed in this macro
will be processed for precision changes.  If you write new routines
that use {\tt BLAS} or {\tt LAPACK}, be sure to add them to the {\tt USES\_BLAS} macro
too.

The current philosophy is that by default objects are built without
optimization (and normally with debugging on, to facilitate
development).  Only those routines which are performance hot spots are
optimized.  This helps avoid stupid compiler bugs which, by empirical
observation, are more likely to show up on things other than
hard-working number crunching.  Each makefile contains an {\tt OBJ\_OPTIMIZE}
macro which should name those routines that {\em should} be built with
optimization.  All the rest should be listed in the OBJ macro.  The
extra arguments used when compiling optimized code are the
platform-dependent [FC]OPTIMIZE macros in \verb+config/makefile.h+.  If you
want to turn off all optimization, change these and rebuild.

At present, the package is almost entirely self-contained.  The only
thing left out is George Fann's PEIGS (Parallel EIGenSolver,
pronounced "pigs") library.  For most platforms, the makefiles already
include pointers to the canonical locations for the PEIGS library for
that platform.  If your link complains about this library, contact one
of the NWChem developers for help, or send e-mail to 
{\tt nwchem-developers@emsl.pnl.gov}.

To build the package, go to the source directory ({\tt .../src}) in your
local copy of the NWChem source directory tree and type 'make'.  (NOTE:
If you are messing
around only in specific subdirectories, you can do 'make' inside them, and
then 'make nwchem\_link' in directory {\tt src}.  This will save you from traversing
{\em every} directory in the package -- but your version of these directories
had better be up to date!)
Object libraries are constructed in the {\tt lib} directory.  The executable
ends up in {\tt src} (strange but true).

Before you actually do the build, however, you must set up your own
environment properly.
In your environment, or on the {\tt make} command line, you must specify two
variables
\begin{itemize}
\item {\tt NWCHEM\_TARGET} --- the name of the machine to build for.
\item {\tt NWCHEM\_TOP} --- the full path to the top level NWChem
  directory.
\end{itemize}
Look in the {\tt README} file in the top-level NWChem directory for
information about supported target platforms.  For instance, you might
insert the following in your \verb+.cshrc+ file on a SUN with SunOS
4.1.3 
\begin{verbatim}
  setenv NWCHEM_TARGET SUN
  if (! $?NWCHEM_TOP) setenv NWCHEM_TOP $HOME/nwchem
\end{verbatim}
(The test to see if \verb+NWCHEM_TOP+ is already defined permits you
to build in an alternative directory without having to edit your
\verb+.cshrc+ file).  Given just this information the structure of
NWChem makes it easy to write a makefile to build a library for a
module (.e.g, {\tt libddscf.a} for the SCF module) or to add routines
into a library shared between multiple modules ({\tt libutil.a}
which includes at least the util, geometry, basis, global, and ma
trees).

A minimal makefile looks like this
\begin{verbatim}
      LIBRARY = libminimal.a
          OBJ = a.o b.o c.o

include ../config/makefile.h
include ../config/makelib.h
\end{verbatim}

The above specifies that the object files are to be generated by
compiling available source (C or Fortran, without optimization) and
put into the library {\tt libminimal.a} (in the NWChem library
directory).  Nothing else is necessary.  If the library source is not
located in a subdirectory of the NWChem {\tt src} directory then the
path to the included files must be modified accordingly.

A slightly more complex makefile looks like this
\begin{verbatim}
      LIBRARY = libsimple.a
          OBJ = a.o b.o c.o
 OBJ_OPTIMIZE = d.o
    USES_BLAS = c.o
      HEADERS = simple.fh
  LIB_TARGETS = test

include ../config/makefile.h
include ../config/makelib.h

test: test.o $(LIBRARY_PATH)
      $(LINK.f) -o $@ $^

a.o b.o c.o test.o: simple.fh private.fh
\end{verbatim}

This makefile builds the library {\tt libsimple.a} from four object
files of which only one ({\tt d.o}) is optimized.  The source
associated with {\tt c.o} uses FORTRAN BLAS and will be automatically
converted on machines where 64 bit reals are single precision (e.g.,
requiring {\tt sgemm()} rather than {\tt dgemm()}).  The header file
{\tt simple.fh} is exported automatically into the NWChem include
directory ({\tt src/include}) where it may be included by other
modules which reference these routines.  Associated with the module is
the executable {\tt test} (not made by default) which will be cleaned
up automatically with \verb+make clean+.  The final line
specifies a dependency of certain object files on various header
files.

At an absolute minimum, a makefile for a module must do the following:
\begin{enumerate}
\item include \verb+../config/makefile.h+ --- this will
    define {\tt TARGET} (among other things)
    from which any machine dependent actions are driven.
    Because the first rule in this file
     builds the library, there should be {\em no} targets before this.
    (NOTE: if you do not need to use {\tt TARGET} then it is best to include this
     file at the same point that {\tt makelib.h} is included.)
\item define {\tt LIBRARY} as the name of the library to be made.
\end{enumerate}

The makefile can also do the following, at the developer's option:
\begin{enumerate}
\item define {\tt OBJ} as the list of object files to be made 
    without optimization
\item define {\tt OBJ\_OPTIMIZE} as the list of object files to 
    be made with optimization.  (HINT: It is good practice to keep this list 
    short, to minimize exposure to possible compiler errors.)
\item define {\tt HEADERS} as the list of header/include files to be
    copied into the common include directory.  (WARNING: Do NOT copy {\tt include} 
    files into the {\tt include} directory by hand.  This is done
    automatically, and any separately copied files will be lost.)
\item define {\tt LIB\_TARGETS} as any additional files made in
    this subdirectory that may need cleaning up
\item define {\tt LIB\_DEFINES} as any additional defines for
   the C preprocessor (for both Fortran and C)
 \item define {\tt LIB\_INCLUDES} as any additional include
   directories
\item  define {\tt SUBDIRS} as any subdirectories to build (NOTE: If you do this,
  makefiles in subdirectories will need to modify the paths to
  the include files.)
\item define {\tt USES\_BLAS} to be the list of FORTRAN files that
    need BLAS names converting between single and double (e.g., {\tt
      ddot} to  {\tt sdot})
\item define any additional targets (e.g., test programs)
\end{enumerate}

Addtional things you will need to know how to do:
\begin{enumerate}
\item  To modify the optimization being used, specify on the command
  line {\tt C/FDEBUG} or {\tt C/FOPTIMIZE} to override the flags for the 
  {\tt OBJ} and {\tt OBJ\_OPTIMIZE} files respectively.  E.g.,
\begin{verbatim}
     make FDEBUG="-g -O1"
     make FOPTIMIZE="-O3 -Superfast -bugs" FDEBUG="-O1"
\end{verbatim}
\item  The library is put directly into the NWChem library
  directory and the full path to the library (if needed by your
  makefile) is automatically put into the variable {\tt
    LIBRARY\_PATH}.
\item The object files are put directly into the libraries and are
  not kept elsewhere.  This has several implications
  \begin{itemize}
  \item You can (apart from TCGMSG and GA which are being fixed) build
    executables and libraries for multiple platforms in the same
    source tree.
  \item To force recompilation of all source in a given directory 
    \verb+make clean+  works by deleting the object files from
    the library, and deletes the library itself only if it is empty.
    You have to
    actually either delete the corresponding library or touch the
    source files.
  \item To override the compilation options for a specifc file (e.g.,
    because of compiler errors on a specific platform) you must
    specify the dependency on the object file in the library.  Here
    are two examples.  The first one (\verb+dosymops.f+) does not need
    preprocessing, whereas the second one (\verb+sym_mo_ap_op.F+) does
    and this must be done explicity within the rule for this file.
    This preprocessing is normally done automatically.
\begin{verbatim}
  ifeq ($(TARGET),CRAY-T3D)
    FNOOPT = -dp -Ccray-t3d -Wf"-o noscalar,jump,noieeedivide"
  $(LIBRARY_PATH)(dosymops.o):  dosymops.f
        $(FC) -c $(FNOOPT) $^
  $(LIBRARY_PATH)(sym_mo_ap_op.o):  sym_mo_ap_op.F
        $(FCONVERT)
        $(FC) -c $(FNOOPT) sym_mo_ap_op.f
        @/bin/rm -f sym_mo_ap_op.f
  endif
\end{verbatim}
%$ ... to unscrew emacs latex hiliting
  \end{itemize}
\item The target {\tt clean} will recursively descend subdirectories and
  delete object files from both the directory and associated library, 
  core files and files defined in {\tt LIB\_TARGETS}.
\item The target {\tt realclean} will, in addition to the actions of
  clean, also delete the library and any emacs backup files.
\item The target {\tt cleanF} will recursively descend subdirectories
  and search for and delete \verb+.f+ files for which a corresponding
  \verb+.F+ file exists.  This is useful on machines for which the
  conversion from \verb+.F+ to \verb+.f+ is done explicitly rather
  than by the compiler.
\item The target {\tt depend} will recursively descend subdirectories
  and append onto the end of makefiles dependencies of \verb+.F+ and
  \verb+.c+ files on header files that have been included using the
  notation \verb+#include "filename"+.  File includes using angle
  brackets are assumed to be system files and dependencies are not
  generated.  If the include file is in the local directory, the
  dependency is generated upon that.  Otherwise, a dependency is
  generated upon a file in the NWChem include directory.
  Do not insert anything below the line
\begin{verbatim}
# DO NOT EDIT BENEATH THIS LINE ... GENERATED AUTOMATICALLY
\end{verbatim}
  since it will be lost the next time that \verb+make depend+
  is run.
\end{enumerate}

Typing {\tt make} in the top-level NWChem directory will traverse the
entire directory tree twice.  Once to ensure the include files are
up-to-date and then again for the libraries.  This can take a while.
Therefore, 
when working on development of a particular a module alone,
it is usually much faster to 
\begin{enumerate}
\item execute {\tt make} in the subdirectory, and
\item execute {\tt make link} in the top NWChem directory.
\end{enumerate}

Note that this approach cannot be used if recompilation of another
module is also required, since the
special target link just relinks the code and does not traverse
the directory tree.  {\em After doing a \verb+cvs update+ you should:}
\begin{enumerate}
\item do a {\tt make depend}
 if you have not recently, in any directory you
  have been working in, and
\item do a full {\tt make} from the top level to ensure that all libraries
  incorporate any changed common blocks or declarations.
\end{enumerate}

In addition, the top-level makefile has the target {\tt test} which
builds the executable \verb+nwchem_test+ in the \verb+src/+ directory
(rather than the usual \verb+$(BINDIR)/nwchem+), % $
and the target {\tt prof} 
which builds \verb+nwchem_prof+ (in \verb+src/+)
for performance profiling by linking with the -p option.