lammps/doc/txt/Speed_compare.txt

"Higher level section"_Speed.html - "LAMMPS WWW Site"_lws - "LAMMPS
Documentation"_ld - "LAMMPS Commands"_lc :c

:link(lws,http://lammps.sandia.gov)
:link(ld,Manual.html)
:link(lc,Commands_all.html)

:line

Comparison of various accelerator packages :h3

The next section compares and contrasts the various accelerator
options, since there are multiple ways to perform OpenMP threading,
run on GPUs, optimize for vector units on CPUs and run on Intel
Xeon Phi (co-)processors.

All of these packages can accelerate a LAMMPS calculation taking
advantage of hardware features, but they do it in different ways
and acceleration is not always guaranteed.

As a consequence, for a particular simulation on specific hardware,
one package may be faster than the other.  We give some guidelines
below, but the best way to determine which package is faster for your
input script is to try multiple of them on your machine and experiment
with available performance tuning settings.  See the benchmarking
section below for examples where this has been done.

[Guidelines for using each package optimally:]

Both, the GPU and the KOKKOS package allows you to assign multiple
MPI ranks (= CPU cores) to the same GPU. For the GPU package, this
can lead to a speedup through better utilization of the GPU (by
overlapping computation and data transfer) and more efficient
computation of the non-GPU accelerated parts of LAMMPS through MPI
parallelization, as all system data is maintained and updated on
the host. For KOKKOS, there is less to no benefit from this, due
to its different memory management model, which tries to retain
data on the GPU.
 :ulb,l

The GPU package moves per-atom data (coordinates, forces, and
(optionally) neighbor list data, if not computed on the GPU) between
the CPU and GPU at every timestep.  The KOKKOS/CUDA package only does
this on timesteps when a CPU calculation is required (e.g. to invoke
a fix or compute that is non-GPU-ized). Hence, if you can formulate
your input script to only use GPU-ized fixes and computes, and avoid
doing I/O too often (thermo output, dump file snapshots, restart files),
then the data transfer cost of the KOKKOS/CUDA package can be very low,
causing it to run faster than the GPU package. :l

The GPU package is often faster than the KOKKOS/CUDA package, when the
number of atoms per GPU is on the smaller side.  The crossover point,
in terms of atoms/GPU at which the KOKKOS/CUDA package becomes faster
depends strongly on the pair style.  For example, for a simple Lennard Jones
system the crossover (in single precision) is often about 50K-100K
atoms per GPU.  When performing double precision calculations the
crossover point can be significantly smaller. :l

Both KOKKOS and GPU package compute bonded interactions (bonds, angles,
etc) on the CPU.  If the GPU package is running with several MPI processes
assigned to one GPU, the cost of computing the bonded interactions is
spread across more CPUs and hence the GPU package can run faster in these
cases. :l

When using LAMMPS with multiple MPI ranks assigned to the same GPU, its
performance depends to some extent on the available bandwidth between
the CPUs and the GPU. This can differ significantly based on the
available bus technology, capability of the host CPU and mainboard,
the wiring of the buses and whether switches are used to increase the
number of available bus slots, or if GPUs are housed in an external
enclosure.  This can become quite complex. :l

To achieve significant acceleration through GPUs, both KOKKOS and GPU
package require capable GPUs with fast on-device memory and efficient
data transfer rates. This requests capable upper mid-level to high-end
(desktop) GPUs. Using lower performance GPUs (e.g. on laptops) may
result in a slowdown instead. :l

For the GPU package, specifically when running in parallel with MPI,
if it often more efficient to exclude the PPPM kspace style from GPU
acceleration and instead run it - concurrently with a GPU accelerated
pair style - on the CPU. This can often be easily achieved with placing
a {suffix off} command before and a {suffix on} command after the
{kspace_style pppm} command. :l

The KOKKOS/OpenMP and USER-OMP package have different thread management
strategies, which should result in USER-OMP being more efficient for a
small number of threads with increasing overhead as the number of threads
per MPI rank grows. The KOKKOS/OpenMP kernels have less overhead in that
case, but have lower performance with few threads. :l

The USER-INTEL package contains many options and settings for achieving
additional performance on Intel hardware (CPU and accelerator cards), but
to unlock this potential, an Intel compiler is required. The package code
will compile with GNU gcc, but it will not be as efficient. :l
:ule

[Differences between the GPU and KOKKOS packages:]

The GPU package accelerates only pair force, neighbor list, and (parts
of) PPPM calculations. The KOKKOS package attempts to run most of the
calculation on the GPU, but can transparently support non-accelerated
code (with a performance penalty due to having data transfers between
host and GPU). :ulb,l

The GPU package requires neighbor lists to be built on the CPU when using
exclusion lists, or a triclinic simulation box. :l

The GPU package can be compiled for CUDA or OpenCL and thus supports
both, Nvidia and AMD GPUs well. On Nvidia hardware, using CUDA is typically
resulting in equal or better performance over OpenCL. :l

OpenCL in the GPU package does theoretically also support Intel CPUs or
Intel Xeon Phi, but the native support for those in KOKKOS (or USER-INTEL)
is superior. :l
:ule