diff --git a/docs/source/devguide/index.rst b/docs/source/devguide/index.rst index 1a8252383..e40e7f635 100644 --- a/docs/source/devguide/index.rst +++ b/docs/source/devguide/index.rst @@ -10,9 +10,10 @@ as debugging. .. toctree:: :numbered: - :maxdepth: 3 + :maxdepth: 2 styleguide workflow + tests user-input docbuild diff --git a/docs/source/devguide/tests.rst b/docs/source/devguide/tests.rst new file mode 100644 index 000000000..749737b5c --- /dev/null +++ b/docs/source/devguide/tests.rst @@ -0,0 +1,73 @@ +.. _devguide_tests: + +========== +Test Suite +========== + +Running Tests +------------- + +The OpenMC test suite consists of two parts, a regression test suite and a unit +test suite. The regression test suite is based on regression or integrated +testing where different types of input files are configured and the full OpenMC +code is executed. Results from simulations are compared with expected +results. The unit tests are primarily intended to test individual +functions/classes in the OpenMC Python API. + +The test suite relies on the third-party `pytest `_ +package. To run either or both the regression and unit test suites, it is +assumed that you have OpenMC fully installed, i.e., the :ref:`scripts_openmc` +executable is available on your :envvar:`PATH` and the :mod:`openmc` Python +module is importable. In development where it would be onerous to continually +install OpenMC every time a small change is made, it is recommended to install +OpenMC in development/editable mode. With setuptools, this is accomplished by +running:: + + python setup.py develop + +or using pip (recommended):: + + pip install -e .[test] + +It is also assumed that you have cross section data available that is pointed to +by the :envvar:`OPENMC_CROSS_SECTIONS` and :envvar:`OPENMC_MULTIPOLE_LIBRARY` +environment variables. Furthermore, to run unit tests for the :mod:`openmc.data` +module, it is necessary to have ENDF/B-VII.1 data available and pointed to by +the :envvar:`OPENMC_ENDF_DATA` environment variable. All data sources can be +obtained using the ``tools/ci/travis-before-script.sh`` script. + +To execute the test suite, go to the ``tests/`` directory and run:: + + pytest + +If you want to collect information about source line coverage in the Python API, +you must have the `pytest-cov `_ plugin +installed and run:: + + pytest --cov=../openmc --cov-report=html + +Adding Tests to the Regression Suite +------------------------------------ + +To add a new test to the regression test suite, create a sub-directory in the +``tests/regression_tests/`` directory. To configure a test you need to add the +following files to your new test directory: + + * OpenMC input XML files, if they are not generated through the Python API + * **test.py** - Python test driver script; please refer to other tests to + see how to construct. Any output files that are generated during testing + must be removed at the end of this script. + * **inputs_true.dat** - ASCII file that contains Python API-generated XML + files concatenated together. When the test is run, inputs that are + generated are compared to this file. + * **results_true.dat** - ASCII file that contains the expected results from + the test. The file *results_test.dat* is compared to this file during the + execution of the python test driver script. When the above files have been + created, generate a *results_test.dat* file and copy it to this name and + commit. It should be noted that this file should be generated with basic + compiler options during openmc configuration and build (e.g., no MPI, no + debug/optimization). + +In addition to this description, please see the various types of tests that are +already included in the test suite to see how to create them. If all is +implemented correctly, the new test will automatically be discovered by pytest. diff --git a/docs/source/devguide/workflow.rst b/docs/source/devguide/workflow.rst index 942cee55b..fd1ae0455 100644 --- a/docs/source/devguide/workflow.rst +++ b/docs/source/devguide/workflow.rst @@ -89,139 +89,6 @@ features and bug fixes. The general steps for contributing are as follows: 6. After the pull request has been thoroughly vetted, it is merged back into the *develop* branch of mit-crpg/openmc. -.. _test suite: - -OpenMC Test Suite ------------------ - -The purpose of this test suite is to ensure that OpenMC compiles using various -combinations of compiler flags and options, and that all user input options can -be used successfully without breaking the code. The test suite is comprised of -regression tests where different types of input files are configured and the -full OpenMC code is executed. Results from simulations are compared with -expected results. The test suite is comprised of many build configurations -(e.g. debug, mpi, hdf5) and the actual tests which reside in sub-directories -in the tests directory. We recommend to developers to test their branches -before submitting a formal pull request using gfortran and Intel compilers -if available. - -The test suite is designed to integrate with cmake using ctest_. It is -configured to run with cross sections from NNDC_ augmented with 0 K elastic -scattering data for select nuclides as well as multipole data. To download the -proper data, run the following commands: - -.. code-block:: sh - - wget -O nndc_hdf5.tar.xz $(cat /.travis.yml | grep anl.box | awk '{print $2}') - tar xJvf nndc_hdf5.tar.xz - export OPENMC_CROSS_SECTIONS=$(pwd)/nndc_hdf5/cross_sections.xml - - git clone --branch=master git://github.com/smharper/windowed_multipole_library.git wmp_lib - tar xzvf wmp_lib/multipole_lib.tar.gz - export OPENMC_MULTIPOLE_LIBRARY=$(pwd)/multipole_lib - -The test suite can be run on an already existing build using: - -.. code-block:: sh - - cd build - make test - -or - -.. code-block:: sh - - cd build - ctest - -There are numerous ctest_ command line options that can be set to have -more control over which tests are executed. - -Before running the test suite python script, the following environmental -variables should be set if the default paths are incorrect: - - * **FC** - The command for a Fortran compiler (e.g. gfotran, ifort). - - * Default - *gfortran* - - * **CC** - The command for a C compiler (e.g. gcc, icc). - - * Default - *gcc* - - * **CXX** - The command for a C++ compiler (e.g. g++, icpc). - - * Default - *g++* - - * **MPI_DIR** - The path to the MPI directory. - - * Default - */opt/mpich/3.2-gnu* - - * **HDF5_DIR** - The path to the HDF5 directory. - - * Default - */opt/hdf5/1.8.16-gnu* - - * **PHDF5_DIR** - The path to the parallel HDF5 directory. - - * Default - */opt/phdf5/1.8.16-gnu* - -To run the full test suite, the following command can be executed in the -tests directory: - -.. code-block:: sh - - python run_tests.py - -A subset of build configurations and/or tests can be run. To see how to use -the script run: - -.. code-block:: sh - - python run_tests.py --help - -As an example, say we want to run all tests with debug flags only on tests -that have cone and plot in their name. Also, we would like to run this on -4 processors. We can run: - -.. code-block:: sh - - python run_tests.py -j 4 -C debug -R "cone|plot" - -Note that standard regular expression syntax is used for selecting build -configurations and tests. To print out a list of build configurations, we -can run: - -.. code-block:: sh - - python run_tests.py -p - -Adding tests to test suite -++++++++++++++++++++++++++ - -To add a new test to the test suite, create a sub-directory in the tests -directory that conforms to the regular expression *test_*. To configure -a test you need to add the following files to your new test directory, -*test_name* for example: - - * OpenMC input XML files - * **test_name.py** - Python test driver script, please refer to other - tests to see how to construct. Any output files that are generated - during testing must be removed at the end of this script. - * **inputs_true.dat** - ASCII file that contains Python API-generated XML - files concatenated together. When the test is run, inputs that are - generated are compared to this file. - * **results_true.dat** - ASCII file that contains the expected results - from the test. The file *results_test.dat* is compared to this file - during the execution of the python test driver script. When the - above files have been created, generate a *results_test.dat* file and - copy it to this name and commit. It should be noted that this file - should be generated with basic compiler options during openmc - configuration and build (e.g., no MPI/HDF5, no debug/optimization). - -In addition to this description, please see the various types of tests that -are already included in the test suite to see how to create them. If all is -implemented correctly, the new test directory will automatically be added -to the CTest framework. - Private Development ------------------- diff --git a/docs/source/usersguide/install.rst b/docs/source/usersguide/install.rst index b59572923..6bb685f48 100644 --- a/docs/source/usersguide/install.rst +++ b/docs/source/usersguide/install.rst @@ -364,7 +364,7 @@ Testing Build To run the test suite, you will first need to download a pre-generated cross section library along with windowed multipole data. Please refer to our -:ref:`test suite` documentation for further details. +:ref:`devguide_tests` documentation for further details. -------------------- Python Prerequisites diff --git a/docs/source/usersguide/scripts.rst b/docs/source/usersguide/scripts.rst index 90ca6886d..2f199fc25 100644 --- a/docs/source/usersguide/scripts.rst +++ b/docs/source/usersguide/scripts.rst @@ -168,8 +168,8 @@ ENDF/B-VII.1. It has the following optional arguments: This script downloads `ENDF/B-VII.1 ACE data `_ from NNDC and converts it to -an HDF5 library for use with OpenMC. This data is used for OpenMC's regression -test suite. This script has the following optional arguments: +an HDF5 library for use with OpenMC. This script has the following optional +arguments: -b, --batch Suppress standard in diff --git a/tests/readme.rst b/tests/readme.rst index f2e2bb1c1..5e4b0227b 100644 --- a/tests/readme.rst +++ b/tests/readme.rst @@ -1,58 +1 @@ -================= -OpenMC Test Suite -================= - -The purpose of this test suite is to ensure that OpenMC compiles using various -combinations of compiler flags and options and that all user input options can -be used successfully without breaking the code. The test suite is based on -regression or integrated testing where different types of input files are -configured and the full OpenMC code is executed. Results from simulations -are compared with expected results. The test suite is comprised of many -build configurations (e.g. debug, mpi, hdf5) and the actual tests which -reside in sub-directories in the tests directory. - -The test suite is designed to integrate with cmake using ctest_. To run the -full test suite run: - -.. code-block:: sh - - python run_tests.py - -The test suite is configured to run with cross sections from NNDC_. To -download these cross sections please do the following: - -.. code-block:: sh - - cd ../data - python get_nndc.py - export CROSS_SECTIONS=/nndc/cross_sections.xml - -The environmental variable **CROSS_SECTIONS** can be used to quickly switch -between the cross sections set for the test suite and cross section set for -your simulations. - -A subset of build configurations and/or tests can be run. To see how to use -the script run: - -.. code-block:: sh - - python run_tests.py --help - -As an example, say we want to run all tests with debug flags only on tests -that have cone and plot in their name. Also, we would like to run this on -4 processors. We can run: - -.. code-block:: sh - - python run_tests.py -j 4 -C debug -R "cone|plot" - -Note that standard regular expression syntax is used for selecting build -configurations and tests. To print out a list of build configurations, we -can run: - -.. code-block:: sh - - python run_tests.py -p - -.. _ctest: http://www.cmake.org/cmake/help/v2.8.12/ctest.html -.. _NNDC: http://http://www.nndc.bnl.gov/endf/b7.1/acefiles.html +See docs/source/devguide/tests.rst for information on the OpenMC test suite.