Merge pull request #4041 from akohlmey/test-fix-numdiff

Add fix numdiff based tests for bonded interactions

doc/src/Build_development.rst

@@ -122,32 +122,39 @@ Code Coverage and Unit Testing (CMake only)
 -------------------------------------------
 
 The LAMMPS code is subject to multiple levels of automated testing
-during development: integration testing (i.e. whether the code compiles
+during development:
-on various platforms and with a variety of settings), unit testing
+
-(i.e. whether certain individual parts of the code produce the expected
+- Integration testing (i.e. whether the code compiles
-results for given inputs), run testing (whether selected complete input
+  on various platforms and with a variety of compilers and settings),
-decks run without crashing for multiple configurations), and regression
+- Unit testing (i.e. whether certain functions or classes of the code
-testing (i.e. whether selected input examples reproduce the same
+  produce the expected results for given inputs),
-results over a given number of steps and operations within a given
+- Run testing (i.e. whether selected input decks can run to completion
-error margin).  The status of this automated testing can be viewed on
+  without crashing for multiple configurations),
-`https://ci.lammps.org <https://ci.lammps.org>`_.
+- Regression testing (i.e. whether selected input examples reproduce the
+  same results over a given number of steps and operations within a
+  given error margin).
+
+The status of this automated testing can be viewed on `https://ci.lammps.org
+<https://ci.lammps.org>`_.
 
 The scripts and inputs for integration, run, and regression testing
 are maintained in a
 `separate repository <https://github.com/lammps/lammps-testing>`_
-of the LAMMPS project on GitHub.
+of the LAMMPS project on GitHub.  A few tests are also run as GitHub
+Actions and their configuration files are in the ``.github/workflows/``
+folder of the LAMMPS git tree.
 
-The unit testing facility is integrated into the CMake build process
+The unit testing facility is integrated into the CMake build process of
-of the LAMMPS source code distribution itself.  It can be enabled by
+the LAMMPS source code distribution itself.  It can be enabled by
 setting ``-D ENABLE_TESTING=on`` during the CMake configuration step.
-It requires the `YAML <https://pyyaml.org/>`_ library and development
+It requires the `YAML <https://pyyaml.org/>`_ library and matching
-headers (if those are not found locally a recent version will be
+development headers to compile (if those are not found locally a recent
-downloaded and compiled along with LAMMPS and the test program) to
+version of that library will be downloaded and compiled along with
-compile and will download and compile a specific recent version of the
+LAMMPS and the test programs) and will download and compile a specific
-`Googletest <https://github.com/google/googletest/>`_ C++ test framework
+version of the `GoogleTest <https://github.com/google/googletest/>`_ C++
-for implementing the tests.
+test framework that is used to implement the tests.
 
-.. admonition:: Software version requirements for testing
+.. admonition:: Software version and LAMMPS configuration requirements
    :class: note
 
    The compiler and library version requirements for the testing
@@ -155,7 +162,7 @@ for implementing the tests.
    example the default GNU C++ and Fortran compilers of RHEL/CentOS 7.x
    (version 4.8.x) are not sufficient.  The CMake configuration will try
    to detect incompatible versions and either skip incompatible tests or
-   stop with an error.  Also the number of tests will depend on
+   stop with an error.  Also the number of available tests will depend on
    installed LAMMPS packages, development environment, operating system,
    and configuration settings.
 
@@ -234,12 +241,31 @@ will be skipped if prerequisite features are not available in LAMMPS.
    time.  Preference is given to parts of the code base that are easy to
    test or commonly used.
 
-Tests for styles of the same kind of style (e.g. pair styles or bond
+Tests as shown by the ``ctest`` program are command lines defined in the
-styles) are performed with the same test executable using different
+``CMakeLists.txt`` files in the ``unittest`` directory tree.  A few
-input files in YAML format.  So to add a test for another style of the
+tests simply execute LAMMPS with specific command line flags and check
-same kind it may be sufficient to add a suitable YAML file.
+the output to the screen for expected content.  A large number of unit
-:doc:`Detailed instructions for adding tests <Developer_unittest>` are
+tests are special tests programs using the `GoogleTest framework
-provided in the Programmer Guide part of the manual.
+<https://github.com/google/googletest/>`_ and linked to the LAMMPS
+library that test individual functions or create a LAMMPS class
+instance, execute one or more commands and check data inside the LAMMPS
+class hierarchy.  There are also tests for the C-library, Fortran, and
+Python module interfaces to LAMMPS.  The Python tests use the Python
+"unittest" module in a similar fashion than the others use `GoogleTest`.
+These special test programs are structured to perform multiple
+individual tests internally and each of those contains several checks
+(aka assertions) for internal data being changed as expected.
+
+Tests for force computing or modifying styles (e.g. styles for non-bonded
+and bonded interactions and selected fixes) are run by using a more generic
+test program that reads its input from files in YAML format. The YAML file
+provides the information on how to customized the test program to test
+a specific style and - if needed - with specific settings.
+To add a test for another, similar style (e.g. a new pair style) it is
+usually sufficient to add a suitable YAML file.  :doc:`Detailed
+instructions for adding tests <Developer_unittest>` are provided in the
+Programmer Guide part of the manual.  A description of what happens
+during the tests is given below.
 
 Unit tests for force styles
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^

doc/utils/sphinx-config/false_positives.txt

@@ -3797,6 +3797,7 @@ unimodal
 uninstall
 unitarg
 unitless
+unittest
 Universite
 unix
 unmaintained

unittest/force-styles/test_angle_style.cpp

@@ -27,6 +27,7 @@
 #include "atom.h"
 #include "compute.h"
 #include "exceptions.h"
+#include "fix.h"
 #include "fmt/format.h"
 #include "force.h"
 #include "info.h"
@@ -528,6 +529,59 @@ TEST(AngleStyle, omp)
     if (!verbose) ::testing::internal::GetCapturedStdout();
 };
 
+TEST(AngleStyle, numdiff)
+{
+    if (!LAMMPS::is_installed_pkg("EXTRA-FIX")) GTEST_SKIP();
+    if (test_config.skip_tests.count(test_info_->name())) GTEST_SKIP();
+
+    LAMMPS::argv args = {"AngleStyle", "-log", "none", "-echo", "screen", "-nocite"};
+
+    ::testing::internal::CaptureStdout();
+    LAMMPS *lmp = init_lammps(args, test_config, true);
+
+    std::string output = ::testing::internal::GetCapturedStdout();
+    if (verbose) std::cout << output;
+
+    if (!lmp) {
+        std::cerr << "One or more prerequisite styles are not available "
+                     "in this LAMMPS configuration:\n";
+        for (auto &prerequisite : test_config.prerequisites) {
+            std::cerr << prerequisite.first << "_style " << prerequisite.second << "\n";
+        }
+        GTEST_SKIP();
+    }
+
+    EXPECT_THAT(output, StartsWith("LAMMPS ("));
+    EXPECT_THAT(output, HasSubstr("Loop time"));
+
+    // abort if running in parallel and not all atoms are local
+    const int nlocal = lmp->atom->nlocal;
+    ASSERT_EQ(lmp->atom->natoms, nlocal);
+
+    if (!verbose) ::testing::internal::CaptureStdout();
+    lmp->input->one("fix diff all numdiff 2 6.05504e-6");
+    lmp->input->one("run 2 post no");
+    if (!verbose) ::testing::internal::GetCapturedStdout();
+    Fix *ifix = lmp->modify->get_fix_by_id("diff");
+    if (ifix) {
+        double epsilon = test_config.epsilon * 5.0e8;
+        ErrorStats stats;
+        double **f1 = lmp->atom->f;
+        double **f2 = ifix->array_atom;
+        SCOPED_TRACE("EXPECT FORCES: numdiff");
+        for (int i = 0; i < nlocal; ++i) {
+            EXPECT_FP_LE_WITH_EPS(f1[i][0], f2[i][0], epsilon);
+            EXPECT_FP_LE_WITH_EPS(f1[i][1], f2[i][1], epsilon);
+            EXPECT_FP_LE_WITH_EPS(f1[i][2], f2[i][2], epsilon);
+        }
+        if (print_stats)
+            std::cerr << "numdiff  stats: " << stats << " epsilon: " << epsilon << std::endl;
+    }
+    if (!verbose) ::testing::internal::CaptureStdout();
+    cleanup_lammps(lmp, test_config);
+    if (!verbose) ::testing::internal::GetCapturedStdout();
+}
+
 TEST(AngleStyle, single)
 {
     if (test_config.skip_tests.count(test_info_->name())) GTEST_SKIP();

unittest/force-styles/test_bond_style.cpp

@@ -27,6 +27,7 @@
 #include "bond.h"
 #include "compute.h"
 #include "exceptions.h"
+#include "fix.h"
 #include "fmt/format.h"
 #include "force.h"
 #include "info.h"
@@ -530,6 +531,60 @@ TEST(BondStyle, omp)
     if (!verbose) ::testing::internal::GetCapturedStdout();
 };
 
+
+TEST(BondStyle, numdiff)
+{
+    if (!LAMMPS::is_installed_pkg("EXTRA-FIX")) GTEST_SKIP();
+    if (test_config.skip_tests.count(test_info_->name())) GTEST_SKIP();
+
+    LAMMPS::argv args = {"BondStyle", "-log", "none", "-echo", "screen", "-nocite"};
+
+    ::testing::internal::CaptureStdout();
+    LAMMPS *lmp = init_lammps(args, test_config, true);
+
+    std::string output = ::testing::internal::GetCapturedStdout();
+    if (verbose) std::cout << output;
+
+    if (!lmp) {
+        std::cerr << "One or more prerequisite styles are not available "
+                     "in this LAMMPS configuration:\n";
+        for (auto &prerequisite : test_config.prerequisites) {
+            std::cerr << prerequisite.first << "_style " << prerequisite.second << "\n";
+        }
+        GTEST_SKIP();
+    }
+
+    EXPECT_THAT(output, StartsWith("LAMMPS ("));
+    EXPECT_THAT(output, HasSubstr("Loop time"));
+
+    // abort if running in parallel and not all atoms are local
+    const int nlocal = lmp->atom->nlocal;
+    ASSERT_EQ(lmp->atom->natoms, nlocal);
+
+    if (!verbose) ::testing::internal::CaptureStdout();
+    lmp->input->one("fix diff all numdiff 2 6.05504e-6");
+    lmp->input->one("run 2 post no");
+    if (!verbose) ::testing::internal::GetCapturedStdout();
+    Fix *ifix = lmp->modify->get_fix_by_id("diff");
+    if (ifix) {
+        double epsilon = test_config.epsilon * 5.0e8;
+        ErrorStats stats;
+        double **f1 = lmp->atom->f;
+        double **f2 = ifix->array_atom;
+        SCOPED_TRACE("EXPECT FORCES: numdiff");
+        for (int i = 0; i < nlocal; ++i) {
+            EXPECT_FP_LE_WITH_EPS(f1[i][0], f2[i][0], epsilon);
+            EXPECT_FP_LE_WITH_EPS(f1[i][1], f2[i][1], epsilon);
+            EXPECT_FP_LE_WITH_EPS(f1[i][2], f2[i][2], epsilon);
+        }
+        if (print_stats)
+            std::cerr << "numdiff  stats: " << stats << " epsilon: " << epsilon << std::endl;
+    }
+    if (!verbose) ::testing::internal::CaptureStdout();
+    cleanup_lammps(lmp, test_config);
+    if (!verbose) ::testing::internal::GetCapturedStdout();
+}
+
 TEST(BondStyle, single)
 {
     if (test_config.skip_tests.count(test_info_->name())) GTEST_SKIP();

unittest/force-styles/test_dihedral_style.cpp

@@ -27,6 +27,7 @@
 #include "compute.h"
 #include "dihedral.h"
 #include "exceptions.h"
+#include "fix.h"
 #include "fmt/format.h"
 #include "force.h"
 #include "info.h"
@@ -531,3 +532,57 @@ TEST(DihedralStyle, omp)
     cleanup_lammps(lmp, test_config);
     if (!verbose) ::testing::internal::GetCapturedStdout();
 };
+
+
+TEST(DihedralStyle, numdiff)
+{
+    if (!LAMMPS::is_installed_pkg("EXTRA-FIX")) GTEST_SKIP();
+    if (test_config.skip_tests.count(test_info_->name())) GTEST_SKIP();
+
+    LAMMPS::argv args = {"DihedralStyle", "-log", "none", "-echo", "screen", "-nocite"};
+
+    ::testing::internal::CaptureStdout();
+    LAMMPS *lmp = init_lammps(args, test_config, true);
+
+    std::string output = ::testing::internal::GetCapturedStdout();
+    if (verbose) std::cout << output;
+
+    if (!lmp) {
+        std::cerr << "One or more prerequisite styles are not available "
+                     "in this LAMMPS configuration:\n";
+        for (auto &prerequisite : test_config.prerequisites) {
+            std::cerr << prerequisite.first << "_style " << prerequisite.second << "\n";
+        }
+        GTEST_SKIP();
+    }
+
+    EXPECT_THAT(output, StartsWith("LAMMPS ("));
+    EXPECT_THAT(output, HasSubstr("Loop time"));
+
+    // abort if running in parallel and not all atoms are local
+    const int nlocal = lmp->atom->nlocal;
+    ASSERT_EQ(lmp->atom->natoms, nlocal);
+
+    if (!verbose) ::testing::internal::CaptureStdout();
+    lmp->input->one("fix diff all numdiff 2 6.05504e-6");
+    lmp->input->one("run 2 post no");
+    if (!verbose) ::testing::internal::GetCapturedStdout();
+    Fix *ifix = lmp->modify->get_fix_by_id("diff");
+    if (ifix) {
+        double epsilon = test_config.epsilon * 5.0e8;
+        ErrorStats stats;
+        double **f1 = lmp->atom->f;
+        double **f2 = ifix->array_atom;
+        SCOPED_TRACE("EXPECT FORCES: numdiff");
+        for (int i = 0; i < nlocal; ++i) {
+            EXPECT_FP_LE_WITH_EPS(f1[i][0], f2[i][0], epsilon);
+            EXPECT_FP_LE_WITH_EPS(f1[i][1], f2[i][1], epsilon);
+            EXPECT_FP_LE_WITH_EPS(f1[i][2], f2[i][2], epsilon);
+        }
+        if (print_stats)
+            std::cerr << "numdiff  stats: " << stats << " epsilon: " << epsilon << std::endl;
+    }
+    if (!verbose) ::testing::internal::CaptureStdout();
+    cleanup_lammps(lmp, test_config);
+    if (!verbose) ::testing::internal::GetCapturedStdout();
+}

unittest/force-styles/test_improper_style.cpp

@@ -26,6 +26,7 @@
 #include "atom.h"
 #include "compute.h"
 #include "exceptions.h"
+#include "fix.h"
 #include "fmt/format.h"
 #include "force.h"
 #include "improper.h"
@@ -524,3 +525,56 @@ TEST(ImproperStyle, omp)
     cleanup_lammps(lmp, test_config);
     if (!verbose) ::testing::internal::GetCapturedStdout();
 };
+
+TEST(ImproperStyle, numdiff)
+{
+    if (!LAMMPS::is_installed_pkg("EXTRA-FIX")) GTEST_SKIP();
+    if (test_config.skip_tests.count(test_info_->name())) GTEST_SKIP();
+
+    LAMMPS::argv args = {"ImproperStyle", "-log", "none", "-echo", "screen", "-nocite"};
+
+    ::testing::internal::CaptureStdout();
+    LAMMPS *lmp = init_lammps(args, test_config, true);
+
+    std::string output = ::testing::internal::GetCapturedStdout();
+    if (verbose) std::cout << output;
+
+    if (!lmp) {
+        std::cerr << "One or more prerequisite styles are not available "
+                     "in this LAMMPS configuration:\n";
+        for (auto &prerequisite : test_config.prerequisites) {
+            std::cerr << prerequisite.first << "_style " << prerequisite.second << "\n";
+        }
+        GTEST_SKIP();
+    }
+
+    EXPECT_THAT(output, StartsWith("LAMMPS ("));
+    EXPECT_THAT(output, HasSubstr("Loop time"));
+
+    // abort if running in parallel and not all atoms are local
+    const int nlocal = lmp->atom->nlocal;
+    ASSERT_EQ(lmp->atom->natoms, nlocal);
+
+    if (!verbose) ::testing::internal::CaptureStdout();
+    lmp->input->one("fix diff all numdiff 2 6.05504e-6");
+    lmp->input->one("run 2 post no");
+    if (!verbose) ::testing::internal::GetCapturedStdout();
+    Fix *ifix = lmp->modify->get_fix_by_id("diff");
+    if (ifix) {
+        double epsilon = test_config.epsilon * 5.0e8;
+        ErrorStats stats;
+        double **f1 = lmp->atom->f;
+        double **f2 = ifix->array_atom;
+        SCOPED_TRACE("EXPECT FORCES: numdiff");
+        for (int i = 0; i < nlocal; ++i) {
+            EXPECT_FP_LE_WITH_EPS(f1[i][0], f2[i][0], epsilon);
+            EXPECT_FP_LE_WITH_EPS(f1[i][1], f2[i][1], epsilon);
+            EXPECT_FP_LE_WITH_EPS(f1[i][2], f2[i][2], epsilon);
+        }
+        if (print_stats)
+            std::cerr << "numdiff  stats: " << stats << " epsilon: " << epsilon << std::endl;
+    }
+    if (!verbose) ::testing::internal::CaptureStdout();
+    cleanup_lammps(lmp, test_config);
+    if (!verbose) ::testing::internal::GetCapturedStdout();
+}

unittest/force-styles/test_main.cpp

@@ -47,7 +47,7 @@ void EXPECT_STRESS(const std::string &name, double *stress, const stress_t &expe
     EXPECT_FP_LE_WITH_EPS(stress[3], expected_stress.xy, epsilon);
     EXPECT_FP_LE_WITH_EPS(stress[4], expected_stress.xz, epsilon);
     EXPECT_FP_LE_WITH_EPS(stress[5], expected_stress.yz, epsilon);
-    if (print_stats) std::cerr << name << " stats" << stats << std::endl;
+    if (print_stats) std::cerr << name << " stats: " << stats << std::endl;
 }
 
 void EXPECT_FORCES(const std::string &name, Atom *atom, const std::vector<coord_t> &f_ref,
@@ -64,7 +64,7 @@ void EXPECT_FORCES(const std::string &name, Atom *atom, const std::vector<coord_
         EXPECT_FP_LE_WITH_EPS(f[i][1], f_ref[tag[i]].y, epsilon);
         EXPECT_FP_LE_WITH_EPS(f[i][2], f_ref[tag[i]].z, epsilon);
     }
-    if (print_stats) std::cerr << name << " stats" << stats << std::endl;
+    if (print_stats) std::cerr << name << " stats: " << stats << std::endl;
 }
 
 void EXPECT_POSITIONS(const std::string &name, Atom *atom, const std::vector<coord_t> &x_ref,
@@ -81,7 +81,7 @@ void EXPECT_POSITIONS(const std::string &name, Atom *atom, const std::vector<coo
         EXPECT_FP_LE_WITH_EPS(x[i][1], x_ref[tag[i]].y, epsilon);
         EXPECT_FP_LE_WITH_EPS(x[i][2], x_ref[tag[i]].z, epsilon);
     }
-    if (print_stats) std::cerr << name << " stats" << stats << std::endl;
+    if (print_stats) std::cerr << name << " stats: " << stats << std::endl;
 }
 
 void EXPECT_VELOCITIES(const std::string &name, Atom *atom, const std::vector<coord_t> &v_ref,
@@ -98,7 +98,7 @@ void EXPECT_VELOCITIES(const std::string &name, Atom *atom, const std::vector<co
         EXPECT_FP_LE_WITH_EPS(v[i][1], v_ref[tag[i]].y, epsilon);
         EXPECT_FP_LE_WITH_EPS(v[i][2], v_ref[tag[i]].z, epsilon);
     }
-    if (print_stats) std::cerr << name << " stats" << stats << std::endl;
+    if (print_stats) std::cerr << name << " stats: " << stats << std::endl;
 }
 
 // common read_yaml_file function

unittest/force-styles/tests/angle-cosine_delta.yaml

@@ -2,7 +2,7 @@
 lammps_version: 17 Feb 2022
 date_generated: Fri Mar 18 22:17:51 2022
 epsilon: 5e-13
-skip_tests:
+skip_tests: numdiff
 prerequisites: ! |
   atom full
   angle cosine/delta

unittest/force-styles/tests/angle-fourier_simple.yaml

@@ -5,6 +5,7 @@ epsilon: 2.5e-13
 prerequisites: ! |
   atom full
   angle fourier/simple
+skip_tests: numdiff
 pre_commands: ! ""
 post_commands: ! ""
 input_file: in.fourmol

unittest/force-styles/tests/angle-mm3.yaml

@@ -1,7 +1,7 @@
 ---
 lammps_version: 17 Feb 2022
 date_generated: Fri Mar 18 22:17:52 2022
-epsilon: 2.5e-13
+epsilon: 2.5e-12
 skip_tests:
 prerequisites: ! |
   atom full

unittest/force-styles/tests/angle-table_linear.yaml

@@ -2,7 +2,7 @@
 lammps_version: 17 Feb 2022
 date_generated: Fri Mar 18 22:17:52 2022
 epsilon: 5e-13
-skip_tests:
+skip_tests: numdiff
 prerequisites: ! |
   atom full
   angle table

unittest/force-styles/tests/bond-table_linear.yaml

@@ -2,7 +2,7 @@
 lammps_version: 17 Feb 2022
 date_generated: Fri Mar 18 22:17:51 2022
 epsilon: 2.5e-13
-skip_tests:
+skip_tests: numdiff
 prerequisites: ! |
   atom full
   bond table

unittest/force-styles/tests/dihedral-charmm.yaml

@@ -2,7 +2,7 @@
 lammps_version: 17 Feb 2022
 date_generated: Fri Mar 18 22:18:01 2022
 epsilon: 2.5e-13
-skip_tests:
+skip_tests: numdiff
 prerequisites: ! |
   atom full
   dihedral charmm

unittest/force-styles/tests/dihedral-charmmfsw.yaml

@@ -1,7 +1,7 @@
 ---
 lammps_version: 17 Feb 2022
 date_generated: Fri Mar 18 22:18:02 2022
-epsilon: 2.5e-13
+epsilon: 5.0e-12
 skip_tests:
 prerequisites: ! |
   atom full

unittest/force-styles/tests/dihedral-table_cut_linear.yaml

@@ -3,7 +3,7 @@ lammps_version: 17 Feb 2022
 tags: unstable
 date_generated: Fri Mar 18 22:18:02 2022
 epsilon: 1e-13
-skip_tests:
+skip_tests: numdiff
 prerequisites: ! |
   atom full
   dihedral table/cut

unittest/force-styles/tests/dihedral-table_linear.yaml

@@ -3,7 +3,7 @@ lammps_version: 22 Dec 2022
 tags: unstable
 date_generated: Mon Dec 26 16:49:31 2022
 epsilon: 7.5e-14
-skip_tests:
+skip_tests: numdiff
 prerequisites: ! |
   atom full
   dihedral table

unittest/force-styles/tests/dihedral-table_spline.yaml

@@ -1,7 +1,7 @@
 ---
 lammps_version: 22 Dec 2022
 date_generated: Mon Dec 26 16:49:37 2022
-epsilon: 1e-13
+epsilon: 1.0e-13
 skip_tests:
 prerequisites: ! |
   atom full

unittest/force-styles/tests/improper-cossq.yaml

@@ -2,7 +2,7 @@
 lammps_version: 17 Feb 2022
 date_generated: Fri Mar 18 22:18:02 2022
 epsilon: 2.5e-13
-skip_tests:
+skip_tests: numdiff
 prerequisites: ! |
   atom full
   improper cossq