2015-07-29 05:46:05 +08:00
|
|
|
<HTML>
|
|
|
|
|
<CENTER><A HREF = "http://lammps.sandia.gov">LAMMPS WWW Site</A> - <A HREF = "Manual.html">LAMMPS Documentation</A> - <A HREF = "Section_commands.html#comm">LAMMPS Commands</A>
|
|
|
|
|
</CENTER>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<HR>
|
|
|
|
|
|
|
|
|
|
<H3>package command
|
|
|
|
|
</H3>
|
|
|
|
|
<P><B>Syntax:</B>
|
|
|
|
|
</P>
|
|
|
|
|
<PRE>package style args
|
|
|
|
|
</PRE>
|
|
|
|
|
<UL><LI>style = <I>cuda</I> or <I>gpu</I> or <I>intel</I> or <I>kokkos</I> or <I>omp</I>
|
|
|
|
|
|
|
|
|
|
<LI>args = arguments specific to the style
|
|
|
|
|
|
|
|
|
|
<PRE> <I>cuda</I> args = Ngpu keyword value ...
|
|
|
|
|
Ngpu = # of GPUs per node
|
|
|
|
|
zero or more keyword/value pairs may be appended
|
|
|
|
|
keywords = <I>newton</I> or <I>gpuID</I> or <I>timing</I> or <I>test</I> or <I>thread</I>
|
|
|
|
|
<I>newton</I> = <I>off</I> or <I>on</I>
|
|
|
|
|
off = set Newton pairwise and bonded flags off (default)
|
|
|
|
|
on = set Newton pairwise and bonded flags on
|
|
|
|
|
<I>gpuID</I> values = gpu1 .. gpuN
|
|
|
|
|
gpu1 .. gpuN = IDs of the Ngpu GPUs to use
|
|
|
|
|
<I>timing</I> values = none
|
|
|
|
|
<I>test</I> values = id
|
|
|
|
|
id = atom-ID of a test particle
|
|
|
|
|
<I>thread</I> = auto or tpa or bpa
|
|
|
|
|
auto = test whether tpa or bpa is faster
|
|
|
|
|
tpa = one thread per atom
|
|
|
|
|
bpa = one block per atom
|
|
|
|
|
<I>gpu</I> args = Ngpu keyword value ...
|
|
|
|
|
Ngpu = # of GPUs per node
|
|
|
|
|
zero or more keyword/value pairs may be appended
|
|
|
|
|
keywords = <I>neigh</I> or <I>newton</I> or <I>binsize</I> or <I>split</I> or <I>gpuID</I> or <I>tpa</I> or <I>device</I>
|
|
|
|
|
<I>neigh</I> value = <I>yes</I> or <I>no</I>
|
|
|
|
|
yes = neighbor list build on GPU (default)
|
|
|
|
|
no = neighbor list build on CPU
|
|
|
|
|
<I>newton</I> = <I>off</I> or <I>on</I>
|
|
|
|
|
off = set Newton pairwise flag off (default and required)
|
|
|
|
|
on = set Newton pairwise flag on (currently not allowed)
|
|
|
|
|
<I>binsize</I> value = size
|
|
|
|
|
size = bin size for neighbor list construction (distance units)
|
|
|
|
|
<I>split</I> = fraction
|
|
|
|
|
fraction = fraction of atoms assigned to GPU (default = 1.0)
|
|
|
|
|
<I>gpuID</I> values = first last
|
|
|
|
|
first = ID of first GPU to be used on each node
|
|
|
|
|
last = ID of last GPU to be used on each node
|
|
|
|
|
<I>tpa</I> value = Nthreads
|
|
|
|
|
Nthreads = # of GPU threads used per atom
|
|
|
|
|
<I>device</I> value = device_type
|
|
|
|
|
device_type = <I>kepler</I> or <I>fermi</I> or <I>cypress</I> or <I>generic</I>
|
2015-08-19 23:38:23 +08:00
|
|
|
<I>blocksize</I> value = size
|
|
|
|
|
size = thread block size for pair force computation
|
2015-07-29 05:46:05 +08:00
|
|
|
<I>intel</I> args = NPhi keyword value ...
|
|
|
|
|
Nphi = # of coprocessors per node
|
|
|
|
|
zero or more keyword/value pairs may be appended
|
|
|
|
|
keywords = <I>omp</I> or <I>mode</I> or <I>balance</I> or <I>ghost</I> or <I>tpc</I> or <I>tptask</I> or <I>no_affinity</I>
|
|
|
|
|
<I>omp</I> value = Nthreads
|
|
|
|
|
Nthreads = number of OpenMP threads to use on CPU (default = 0)
|
|
|
|
|
<I>mode</I> value = <I>single</I> or <I>mixed</I> or <I>double</I>
|
|
|
|
|
single = perform force calculations in single precision
|
|
|
|
|
mixed = perform force calculations in mixed precision
|
|
|
|
|
double = perform force calculations in double precision
|
|
|
|
|
<I>balance</I> value = split
|
|
|
|
|
split = fraction of work to offload to coprocessor, -1 for dynamic
|
|
|
|
|
<I>ghost</I> value = <I>yes</I> or <I>no</I>
|
|
|
|
|
yes = include ghost atoms for offload
|
|
|
|
|
no = do not include ghost atoms for offload
|
|
|
|
|
<I>tpc</I> value = Ntpc
|
|
|
|
|
Ntpc = max number of coprocessor threads per coprocessor core (default = 4)
|
|
|
|
|
<I>tptask</I> value = Ntptask
|
|
|
|
|
Ntptask = max number of coprocessor threads per MPI task (default = 240)
|
|
|
|
|
<I>no_affinity</I> values = none
|
|
|
|
|
<I>kokkos</I> args = keyword value ...
|
|
|
|
|
zero or more keyword/value pairs may be appended
|
|
|
|
|
keywords = <I>neigh</I> or <I>newton</I> or <I>binsize</I> or <I>comm</I> or <I>comm/exchange</I> or <I>comm/forward</I>
|
|
|
|
|
<I>neigh</I> value = <I>full</I> or <I>half/thread</I> or <I>half</I> or <I>n2</I> or <I>full/cluster</I>
|
|
|
|
|
full = full neighbor list
|
|
|
|
|
half/thread = half neighbor list built in thread-safe manner
|
|
|
|
|
half = half neighbor list, not thread-safe, only use when 1 thread/MPI task
|
|
|
|
|
n2 = non-binning neighbor list build, O(N^2) algorithm
|
|
|
|
|
full/cluster = full neighbor list with clustered groups of atoms
|
|
|
|
|
<I>newton</I> = <I>off</I> or <I>on</I>
|
|
|
|
|
off = set Newton pairwise and bonded flags off (default)
|
|
|
|
|
on = set Newton pairwise and bonded flags on
|
|
|
|
|
<I>binsize</I> value = size
|
|
|
|
|
size = bin size for neighbor list construction (distance units)
|
|
|
|
|
<I>comm</I> value = <I>no</I> or <I>host</I> or <I>device</I>
|
|
|
|
|
use value for both comm/exchange and comm/forward
|
|
|
|
|
<I>comm/exchange</I> value = <I>no</I> or <I>host</I> or <I>device</I>
|
|
|
|
|
<I>comm/forward</I> value = <I>no</I> or <I>host</I> or <I>device</I>
|
|
|
|
|
no = perform communication pack/unpack in non-KOKKOS mode
|
|
|
|
|
host = perform pack/unpack on host (e.g. with OpenMP threading)
|
|
|
|
|
device = perform pack/unpack on device (e.g. on GPU)
|
|
|
|
|
<I>omp</I> args = Nthreads keyword value ...
|
|
|
|
|
Nthread = # of OpenMP threads to associate with each MPI process
|
|
|
|
|
zero or more keyword/value pairs may be appended
|
|
|
|
|
keywords = <I>neigh</I>
|
|
|
|
|
<I>neigh</I> value = <I>yes</I> or <I>no</I>
|
|
|
|
|
yes = threaded neighbor list build (default)
|
|
|
|
|
no = non-threaded neighbor list build
|
|
|
|
|
</PRE>
|
|
|
|
|
|
|
|
|
|
</UL>
|
|
|
|
|
<P><B>Examples:</B>
|
|
|
|
|
</P>
|
|
|
|
|
<PRE>package gpu 1
|
|
|
|
|
package gpu 1 split 0.75
|
|
|
|
|
package gpu 2 split -1.0
|
|
|
|
|
package cuda 2 gpuID 0 2
|
|
|
|
|
package cuda 1 test 3948
|
|
|
|
|
package kokkos neigh half/thread comm device
|
|
|
|
|
package omp 0 neigh no
|
|
|
|
|
package omp 4
|
|
|
|
|
package intel 1
|
|
|
|
|
package intel 2 omp 4 mode mixed balance 0.5
|
|
|
|
|
</PRE>
|
|
|
|
|
<P><B>Description:</B>
|
|
|
|
|
</P>
|
|
|
|
|
<P>This command invokes package-specific settings for the various
|
|
|
|
|
accelerator packages available in LAMMPS. Currently the following
|
|
|
|
|
packages use settings from this command: USER-CUDA, GPU, USER-INTEL,
|
|
|
|
|
KOKKOS, and USER-OMP.
|
|
|
|
|
</P>
|
|
|
|
|
<P>If this command is specified in an input script, it must be near the
|
|
|
|
|
top of the script, before the simulation box has been defined. This
|
|
|
|
|
is because it specifies settings that the accelerator packages use in
|
|
|
|
|
their intialization, before a simultion is defined.
|
|
|
|
|
</P>
|
|
|
|
|
<P>This command can also be specified from the command-line when
|
|
|
|
|
launching LAMMPS, using the "-pk" <A HREF = "Section_start.html#start_7">command-line
|
|
|
|
|
switch</A>. The syntax is exactly the same as
|
|
|
|
|
when used in an input script.
|
|
|
|
|
</P>
|
|
|
|
|
<P>Note that all of the accelerator packages require the package command
|
|
|
|
|
to be specified (except the OPT package), if the package is to be used
|
|
|
|
|
in a simulation (LAMMPS can be built with an accelerator package
|
|
|
|
|
without using it in a particular simulation). However, in all cases,
|
|
|
|
|
a default version of the command is typically invoked by other
|
|
|
|
|
accelerator settings.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The USER-CUDA and KOKKOS packages require a "-c on" or "-k on"
|
|
|
|
|
<A HREF = "Section_start.html#start_7">command-line switch</A> respectively, which
|
|
|
|
|
invokes a "package cuda" or "package kokkos" command with default
|
|
|
|
|
settings.
|
|
|
|
|
</P>
|
|
|
|
|
<P>For the GPU, USER-INTEL, and USER-OMP packages, if a "-sf gpu" or "-sf
|
|
|
|
|
intel" or "-sf omp" <A HREF = "Section_start.html#start_7">command-line switch</A>
|
|
|
|
|
is used to auto-append accelerator suffixes to various styles in the
|
|
|
|
|
input script, then those switches also invoke a "package gpu",
|
|
|
|
|
"package intel", or "package omp" command with default settings.
|
|
|
|
|
</P>
|
|
|
|
|
<P>IMPORTANT NOTE: A package command for a particular style can be
|
|
|
|
|
invoked multiple times when a simulation is setup, e.g. by the "-c
|
|
|
|
|
on", "-k on", "-sf", and "-pk" <A HREF = "Section_start.html#start_7">command-line
|
|
|
|
|
switches</A>, and by using this command in an
|
|
|
|
|
input script. Each time it is used all of the style options are set,
|
|
|
|
|
either to default values or to specified settings. I.e. settings from
|
|
|
|
|
previous invocations do not persist across multiple invocations.
|
|
|
|
|
</P>
|
|
|
|
|
<P>See the <A HREF = "Section_accelerate.html">Section Accelerate</A> section of the
|
|
|
|
|
manual for more details about using the various accelerator packages
|
|
|
|
|
for speeding up LAMMPS simulations.
|
|
|
|
|
</P>
|
|
|
|
|
<HR>
|
|
|
|
|
|
|
|
|
|
<P>The <I>cuda</I> style invokes settings associated with the use of the
|
|
|
|
|
USER-CUDA package.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>Ngpus</I> argument sets the number of GPUs per node. There must be
|
|
|
|
|
exactly one MPI task per GPU, as set by the mpirun or mpiexec command.
|
|
|
|
|
</P>
|
|
|
|
|
<P>Optional keyword/value pairs can also be specified. Each has a
|
|
|
|
|
default value as listed below.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>newton</I> keyword sets the Newton flags for pairwise and bonded
|
|
|
|
|
interactions to <I>off</I> or <I>on</I>, the same as the <A HREF = "newton.html">newton</A>
|
|
|
|
|
command allows. The default is <I>off</I> because this will almost always
|
|
|
|
|
give better performance for the USER-CUDA package. This means
|
|
|
|
|
more computation is done, but less communication.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>gpuID</I> keyword allows selection of which GPUs on each node will
|
|
|
|
|
be used for a simulation. GPU IDs range from 0 to N-1 where N is the
|
|
|
|
|
physical number of GPUs/node. An ID is specified for each of the
|
|
|
|
|
Ngpus being used. For example if you have three GPUs on a machine,
|
|
|
|
|
one of which is used for the X-Server (the GPU with the ID 1) while
|
|
|
|
|
the others (with IDs 0 and 2) are used for computations you would
|
|
|
|
|
specify:
|
|
|
|
|
</P>
|
|
|
|
|
<PRE>package cuda 2 gpuID 0 2
|
|
|
|
|
</PRE>
|
|
|
|
|
<P>The purpose of the <I>gpuID</I> keyword is to allow two (or more)
|
|
|
|
|
simulations to be run on one workstation. In that case one could set
|
|
|
|
|
the first simulation to use GPU 0 and the second to use GPU 1. This is
|
|
|
|
|
not necessary however, if the GPUs are in what is called <I>compute
|
|
|
|
|
exclusive</I> mode. Using that setting, every process will get its own
|
|
|
|
|
GPU automatically. This <I>compute exclusive</I> mode can be set as root
|
|
|
|
|
using the <I>nvidia-smi</I> tool which is part of the CUDA installation.
|
|
|
|
|
</P>
|
|
|
|
|
<P>Also note that if the <I>gpuID</I> keyword is not used, the USER-CUDA
|
|
|
|
|
package sorts existing GPUs on each node according to their number of
|
|
|
|
|
multiprocessors. This way, compute GPUs will be priorized over
|
|
|
|
|
X-Server GPUs.
|
|
|
|
|
</P>
|
|
|
|
|
<P>If the <I>timing</I> keyword is specified, detailed timing information for
|
|
|
|
|
various subroutines will be output.
|
|
|
|
|
</P>
|
|
|
|
|
<P>If the <I>test</I> keyword is specified, information for the specified atom
|
|
|
|
|
with atom-ID will be output at several points during each timestep.
|
|
|
|
|
This is mainly usefull for debugging purposes. Note that the
|
|
|
|
|
simulation slow down dramatically if this option is used.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>thread</I> keyword can be used to specify how GPU threads are
|
|
|
|
|
assigned work during pair style force evaluation. If the value =
|
|
|
|
|
<I>tpa</I>, one thread per atom is used. If the value = <I>bpa</I>, one block
|
|
|
|
|
per atom is used. If the value = <I>auto</I>, a short test is performed at
|
|
|
|
|
the beginning of each run to determing where <I>tpa</I> or <I>bpa</I> mode is
|
|
|
|
|
faster. The result of this test is output. Since <I>auto</I> is the
|
|
|
|
|
default value, it is usually not necessary to use this keyword.
|
|
|
|
|
</P>
|
|
|
|
|
<HR>
|
|
|
|
|
|
|
|
|
|
<P>The <I>gpu</I> style invokes settings associated with the use of the GPU
|
|
|
|
|
package.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>Ngpu</I> argument sets the number of GPUs per node. There must be
|
|
|
|
|
at least as many MPI tasks per node as GPUs, as set by the mpirun or
|
|
|
|
|
mpiexec command. If there are more MPI tasks (per node)
|
|
|
|
|
than GPUs, multiple MPI tasks will share each GPU.
|
|
|
|
|
</P>
|
|
|
|
|
<P>Optional keyword/value pairs can also be specified. Each has a
|
|
|
|
|
default value as listed below.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>neigh</I> keyword specifies where neighbor lists for pair style
|
|
|
|
|
computation will be built. If <I>neigh</I> is <I>yes</I>, which is the default,
|
|
|
|
|
neighbor list building is performed on the GPU. If <I>neigh</I> is <I>no</I>,
|
|
|
|
|
neighbor list building is performed on the CPU. GPU neighbor list
|
|
|
|
|
building currently cannot be used with a triclinic box. GPU neighbor
|
|
|
|
|
list calculation currently cannot be used with
|
|
|
|
|
<A HREF = "pair_hybrid.html">hybrid</A> pair styles. GPU neighbor lists are not
|
|
|
|
|
compatible with comannds that are not GPU-enabled. When a non-GPU
|
|
|
|
|
enabled command requires a neighbor list, it will also be built on the
|
|
|
|
|
CPU. In these cases, it will typically be more efficient to only use
|
|
|
|
|
CPU neighbor list builds.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>newton</I> keyword sets the Newton flags for pairwise (not bonded)
|
|
|
|
|
interactions to <I>off</I> or <I>on</I>, the same as the <A HREF = "newton.html">newton</A>
|
|
|
|
|
command allows. Currently, only an <I>off</I> value is allowed, since all
|
|
|
|
|
the GPU package pair styles require this setting. This means more
|
|
|
|
|
computation is done, but less communication. In the future a value of
|
|
|
|
|
<I>on</I> may be allowed, so the <I>newton</I> keyword is included as an option
|
|
|
|
|
for compatibility with the package command for other accelerator
|
|
|
|
|
styles. Note that the newton setting for bonded interactions is not
|
|
|
|
|
affected by this keyword.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>binsize</I> keyword sets the size of bins used to bin atoms in
|
|
|
|
|
neighbor list builds performed on the GPU, if <I>neigh</I> = <I>yes</I> is set.
|
|
|
|
|
If <I>binsize</I> is set to 0.0 (the default), then bins = the size of the
|
|
|
|
|
pairwise cutoff + neighbor skin distance. This is 2x larger than the
|
|
|
|
|
LAMMPS default used for neighbor list building on the CPU. This will
|
|
|
|
|
be close to optimal for the GPU, so you do not normally need to use
|
|
|
|
|
this keyword. Note that if you use a longer-than-usual pairwise
|
|
|
|
|
cutoff, e.g. to allow for a smaller fraction of KSpace work with a
|
|
|
|
|
<A HREF = "kspace_style.html">long-range Coulombic solver</A> because the GPU is
|
|
|
|
|
faster at performing pairwise interactions, then it may be optimal to
|
|
|
|
|
make the <I>binsize</I> smaller than the default. For example, with a
|
|
|
|
|
cutoff of 20*sigma in LJ <A HREF = "units.html">units</A> and a neighbor skin
|
|
|
|
|
distance of sigma, a <I>binsize</I> = 5.25*sigma can be more efficient than
|
|
|
|
|
the default.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>split</I> keyword can be used for load balancing force calculations
|
|
|
|
|
between CPU and GPU cores in GPU-enabled pair styles. If 0 < <I>split</I> <
|
|
|
|
|
1.0, a fixed fraction of particles is offloaded to the GPU while force
|
|
|
|
|
calculation for the other particles occurs simulataneously on the CPU.
|
|
|
|
|
If <I>split</I> < 0.0, the optimal fraction (based on CPU and GPU timings)
|
|
|
|
|
is calculated every 25 timesteps, i.e. dynamic load-balancing across
|
|
|
|
|
the CPU and GPU is performed. If <I>split</I> = 1.0, all force
|
|
|
|
|
calculations for GPU accelerated pair styles are performed on the GPU.
|
|
|
|
|
In this case, other <A HREF = "pair_hybrid.html">hybrid</A> pair interactions,
|
|
|
|
|
<A HREF = "bond_style.html">bond</A>, <A HREF = "angle_style.html">angle</A>,
|
|
|
|
|
<A HREF = "dihedral_style.html">dihedral</A>, <A HREF = "improper_style.html">improper</A>, and
|
|
|
|
|
<A HREF = "kspace_style.html">long-range</A> calculations can be performed on the
|
|
|
|
|
CPU while the GPU is performing force calculations for the GPU-enabled
|
|
|
|
|
pair style. If all CPU force computations complete before the GPU
|
|
|
|
|
completes, LAMMPS will block until the GPU has finished before
|
|
|
|
|
continuing the timestep.
|
|
|
|
|
</P>
|
|
|
|
|
<P>As an example, if you have two GPUs per node and 8 CPU cores per node,
|
|
|
|
|
and would like to run on 4 nodes (32 cores) with dynamic balancing of
|
|
|
|
|
force calculation across CPU and GPU cores, you could specify
|
|
|
|
|
</P>
|
|
|
|
|
<PRE>mpirun -np 32 -sf gpu -in in.script # launch command
|
|
|
|
|
package gpu 2 split -1 # input script command
|
|
|
|
|
</PRE>
|
|
|
|
|
<P>In this case, all CPU cores and GPU devices on the nodes would be
|
|
|
|
|
utilized. Each GPU device would be shared by 4 CPU cores. The CPU
|
|
|
|
|
cores would perform force calculations for some fraction of the
|
|
|
|
|
particles at the same time the GPUs performed force calculation for
|
|
|
|
|
the other particles.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>gpuID</I> keyword allows selection of which GPUs on each node will
|
|
|
|
|
be used for a simulation. The <I>first</I> and <I>last</I> values specify the
|
|
|
|
|
GPU IDs to use (from 0 to Ngpu-1). By default, first = 0 and last =
|
|
|
|
|
Ngpu-1, so that all GPUs are used, assuming Ngpu is set to the number
|
|
|
|
|
of physical GPUs. If you only wish to use a subset, set Ngpu to a
|
|
|
|
|
smaller number and first/last to a sub-range of the available GPUs.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>tpa</I> keyword sets the number of GPU thread per atom used to
|
|
|
|
|
perform force calculations. With a default value of 1, the number of
|
|
|
|
|
threads will be chosen based on the pair style, however, the value can
|
|
|
|
|
be set explicitly with this keyword to fine-tune performance. For
|
|
|
|
|
large cutoffs or with a small number of particles per GPU, increasing
|
|
|
|
|
the value can improve performance. The number of threads per atom must
|
|
|
|
|
be a power of 2 and currently cannot be greater than 32.
|
|
|
|
|
</P>
|
2015-08-29 04:01:49 +08:00
|
|
|
<P>The <I>blocksize</I> keyword allows you to tweak the number of threads used
|
2015-08-22 06:51:16 +08:00
|
|
|
per thread block. This number should be a multiple of 32 (for GPUs)
|
|
|
|
|
and its maximum depends on the specific GPU hardware. Typical choices
|
2015-08-29 04:01:49 +08:00
|
|
|
are 64, 128, or 256. A larger blocksize increases occupancy of
|
|
|
|
|
individual GPU cores, but reduces the total number of thread blocks,
|
|
|
|
|
thus may lead to load imbalance.
|
2015-08-22 06:51:16 +08:00
|
|
|
</P>
|
2015-07-29 05:46:05 +08:00
|
|
|
<P>The <I>device</I> keyword can be used to tune parameters optimized for a
|
|
|
|
|
specific accelerator, when using OpenCL. For CUDA, the <I>device</I>
|
|
|
|
|
keyword is ignored. Currently, the device type is limited to NVIDIA
|
|
|
|
|
Kepler, NVIDIA Fermi, AMD Cypress, or a generic device. More devices
|
|
|
|
|
may be added later. The default device type can be specified when
|
|
|
|
|
building LAMMPS with the GPU library, via settings in the
|
|
|
|
|
lib/gpu/Makefile that is used.
|
|
|
|
|
</P>
|
|
|
|
|
<HR>
|
|
|
|
|
|
|
|
|
|
<P>The <I>intel</I> style invokes settings associated with the use of the
|
|
|
|
|
USER-INTEL package. All of its settings, except the <I>omp</I> and <I>mode</I>
|
|
|
|
|
keywords, are ignored if LAMMPS was not built with Xeon Phi
|
|
|
|
|
coprocessor support. All of its settings, including the <I>omp</I> and
|
|
|
|
|
<I>mode</I> keyword are applicable if LAMMPS was built with coprocessor
|
|
|
|
|
support.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>Nphi</I> argument sets the number of coprocessors per node.
|
|
|
|
|
This can be set to any value, including 0, if LAMMPS was not
|
|
|
|
|
built with coprocessor support.
|
|
|
|
|
</P>
|
|
|
|
|
<P>Optional keyword/value pairs can also be specified. Each has a
|
|
|
|
|
default value as listed below.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>omp</I> keyword determines the number of OpenMP threads allocated
|
|
|
|
|
for each MPI task when any portion of the interactions computed by a
|
|
|
|
|
USER-INTEL pair style are run on the CPU. This can be the case even
|
|
|
|
|
if LAMMPS was built with coprocessor support; see the <I>balance</I>
|
|
|
|
|
keyword discussion below. If you are running with less MPI tasks/node
|
|
|
|
|
than there are CPUs, it can be advantageous to use OpenMP threading on
|
|
|
|
|
the CPUs.
|
|
|
|
|
</P>
|
|
|
|
|
<P>IMPORTANT NOTE: The <I>omp</I> keyword has nothing to do with coprocessor
|
|
|
|
|
threads on the Xeon Phi; see the <I>tpc</I> and <I>tptask</I> keywords below for
|
|
|
|
|
a discussion of coprocessor threads.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>Nthread</I> value for the <I>omp</I> keyword sets the number of OpenMP
|
|
|
|
|
threads allocated for each MPI task. Setting <I>Nthread</I> = 0 (the
|
|
|
|
|
default) instructs LAMMPS to use whatever value is the default for the
|
|
|
|
|
given OpenMP environment. This is usually determined via the
|
|
|
|
|
<I>OMP_NUM_THREADS</I> environment variable or the compiler runtime, which
|
|
|
|
|
is usually a value of 1.
|
|
|
|
|
</P>
|
|
|
|
|
<P>For more details, including examples of how to set the OMP_NUM_THREADS
|
|
|
|
|
environment variable, see the discussion of the <I>Nthreads</I> setting on
|
|
|
|
|
this doc page for the "package omp" command. Nthreads is a required
|
|
|
|
|
argument for the USER-OMP package. Its meaning is exactly the same
|
|
|
|
|
for the USER-INTEL pacakge.
|
|
|
|
|
</P>
|
|
|
|
|
<P>IMPORTANT NOTE: If you build LAMMPS with both the USER-INTEL and
|
|
|
|
|
USER-OMP packages, be aware that both packages allow setting of the
|
|
|
|
|
<I>Nthreads</I> value via their package commands, but there is only a
|
|
|
|
|
single global <I>Nthreads</I> value used by OpenMP. Thus if both package
|
|
|
|
|
commands are invoked, you should insure the two values are consistent.
|
|
|
|
|
If they are not, the last one invoked will take precedence, for both
|
2015-10-05 23:18:49 +08:00
|
|
|
packages. Also note that if the "-sf hybrid intel omp" <A HREF = <A HREF = "Section_start.html#start_7">command-line"></A>
|
2015-07-29 05:46:05 +08:00
|
|
|
switch</A> is used, it invokes a "package
|
|
|
|
|
intel" command, followed by a "package omp" command, both with a
|
|
|
|
|
setting of <I>Nthreads</I> = 0.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>mode</I> keyword determines the precision mode to use for
|
|
|
|
|
computing pair style forces, either on the CPU or on the coprocessor,
|
|
|
|
|
when using a USER-INTEL supported <A HREF = "pair_style.html">pair style</A>. It
|
|
|
|
|
can take a value of <I>single</I>, <I>mixed</I> which is the default, or
|
|
|
|
|
<I>double</I>. <I>Single</I> means single precision is used for the entire
|
|
|
|
|
force calculation. <I>Mixed</I> means forces between a pair of atoms are
|
|
|
|
|
computed in single precision, but accumulated and stored in double
|
|
|
|
|
precision, including storage of forces, torques, energies, and virial
|
|
|
|
|
quantities. <I>Double</I> means double precision is used for the entire
|
|
|
|
|
force calculation.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>balance</I> keyword sets the fraction of <A HREF = "pair_style.html">pair
|
|
|
|
|
style</A> work offloaded to the coprocessor for split
|
|
|
|
|
values between 0.0 and 1.0 inclusive. While this fraction of work is
|
|
|
|
|
running on the coprocessor, other calculations will run on the host,
|
|
|
|
|
including neighbor and pair calculations that are not offloaded, as
|
|
|
|
|
well as angle, bond, dihedral, kspace, and some MPI communications.
|
|
|
|
|
If <I>split</I> is set to -1, the fraction of work is dynamically adjusted
|
|
|
|
|
automatically throughout the run. This typically give performance
|
|
|
|
|
within 5 to 10 percent of the optimal fixed fraction.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>ghost</I> keyword determines whether or not ghost atoms, i.e. atoms
|
|
|
|
|
at the boundaries of proessor sub-domains, are offloaded for neighbor
|
|
|
|
|
and force calculations. When the value = "no", ghost atoms are not
|
|
|
|
|
offloaded. This option can reduce the amount of data transfer with
|
|
|
|
|
the coprocessor and can also overlap MPI communication of forces with
|
|
|
|
|
computation on the coprocessor when the <A HREF = "newton.html">newton pair</A>
|
|
|
|
|
setting is "on". When the value = "yes", ghost atoms are offloaded.
|
|
|
|
|
In some cases this can provide better performance, especially if the
|
|
|
|
|
<I>balance</I> fraction is high.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>tpc</I> keyword sets the max # of coprocessor threads <I>Ntpc</I> that
|
|
|
|
|
will run on each core of the coprocessor. The default value = 4,
|
|
|
|
|
which is the number of hardware threads per core supported by the
|
|
|
|
|
current generation Xeon Phi chips.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>tptask</I> keyword sets the max # of coprocessor threads (Ntptask</I>
|
|
|
|
|
assigned to each MPI task. The default value = 240, which is the
|
|
|
|
|
total # of threads an entire current generation Xeon Phi chip can run
|
|
|
|
|
(240 = 60 cores * 4 threads/core). This means each MPI task assigned
|
|
|
|
|
to the Phi will enough threads for the chip to run the max allowed,
|
|
|
|
|
even if only 1 MPI task is assigned. If 8 MPI tasks are assigned to
|
|
|
|
|
the Phi, each will run with 30 threads. If you wish to limit the
|
|
|
|
|
number of threads per MPI task, set <I>tptask</I> to a smaller value.
|
|
|
|
|
E.g. for <I>tptask</I> = 16, if 8 MPI tasks are assigned, each will run
|
|
|
|
|
with 16 threads, for a total of 128.
|
|
|
|
|
</P>
|
|
|
|
|
<P>Note that the default settings for <I>tpc</I> and <I>tptask</I> are fine for
|
|
|
|
|
most problems, regardless of how many MPI tasks you assign to a Phi.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>no_affinity</I> keyword will turn off automatic setting of core
|
|
|
|
|
affinity for MPI tasks and OpenMP threads on the host when using
|
|
|
|
|
offload to a coprocessor. Affinity settings are used when possible
|
|
|
|
|
to prevent MPI tasks and OpenMP threads from being on separate NUMA
|
|
|
|
|
domains and to prevent offload threads from interfering with other
|
|
|
|
|
processes/threads used for LAMMPS.
|
|
|
|
|
</P>
|
|
|
|
|
<HR>
|
|
|
|
|
|
|
|
|
|
<P>The <I>kokkos</I> style invokes settings associated with the use of the
|
|
|
|
|
KOKKOS package.
|
|
|
|
|
</P>
|
|
|
|
|
<P>All of the settings are optional keyword/value pairs. Each has a
|
|
|
|
|
default value as listed below.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>neigh</I> keyword determines how neighbor lists are built. A value
|
|
|
|
|
of <I>half</I> uses half-neighbor lists, the same as used by most pair
|
|
|
|
|
styles in LAMMPS. A value of <I>half/thread</I> uses a thread-safe variant
|
|
|
|
|
of the half-neighbor list. It should be used instead of <I>half</I> when
|
|
|
|
|
running with more than 1 threads per MPI task on a CPU. A value of
|
|
|
|
|
<I>n2</I> uses an O(N^2) algorithm to build the neighbor list without
|
|
|
|
|
binning, where N = # of atoms on a processor. It is typically slower
|
|
|
|
|
than the other methods, which use binning.
|
|
|
|
|
</P>
|
|
|
|
|
<P>A value of <I>full</I> uses a full neighbor lists and is the default. This
|
|
|
|
|
performs twice as much computation as the <I>half</I> option, however that
|
|
|
|
|
is often a win because it is thread-safe and doesn't require atomic
|
|
|
|
|
operations in the calculation of pair forces. For that reason, <I>full</I>
|
|
|
|
|
is the default setting. However, when running in MPI-only mode with 1
|
|
|
|
|
thread per MPI task, <I>half</I> neighbor lists will typically be faster,
|
|
|
|
|
just as it is for non-accelerated pair styles.
|
|
|
|
|
</P>
|
|
|
|
|
<P>A value of <I>full/cluster</I> is an experimental neighbor style, where
|
|
|
|
|
particles interact with all particles within a small cluster, if at
|
|
|
|
|
least one of the clusters particles is within the neighbor cutoff
|
|
|
|
|
range. This potentially allows for better vectorization on
|
|
|
|
|
architectures such as the Intel Phi. If also reduces the size of the
|
|
|
|
|
neighbor list by roughly a factor of the cluster size, thus reducing
|
|
|
|
|
the total memory footprint considerably.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>newton</I> keyword sets the Newton flags for pairwise and bonded
|
|
|
|
|
interactions to <I>off</I> or <I>on</I>, the same as the <A HREF = "newton.html">newton</A>
|
|
|
|
|
command allows. The default is <I>off</I> because this will almost always
|
|
|
|
|
give better performance for the KOKKOS package. This means more
|
|
|
|
|
computation is done, but less communication. However, when running in
|
|
|
|
|
MPI-only mode with 1 thread per MPI task, a value of <I>on</I> will
|
|
|
|
|
typically be faster, just as it is for non-accelerated pair styles.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>binsize</I> keyword sets the size of bins used to bin atoms in
|
|
|
|
|
neighbor list builds. The same value can be set by the <A HREF = "neigh_modify.html">neigh_modify
|
|
|
|
|
binsize</A> command. Making it an option in the
|
|
|
|
|
package kokkos command allows it to be set from the command line. The
|
|
|
|
|
default value is 0.0, which means the LAMMPS default will be used,
|
|
|
|
|
which is bins = 1/2 the size of the pairwise cutoff + neighbor skin
|
|
|
|
|
distance. This is fine when neighbor lists are built on the CPU. For
|
|
|
|
|
GPU builds, a 2x larger binsize equal to the pairwise cutoff +
|
|
|
|
|
neighbor skin, is often faster, which can be set by this keyword.
|
|
|
|
|
Note that if you use a longer-than-usual pairwise cutoff, e.g. to
|
|
|
|
|
allow for a smaller fraction of KSpace work with a <A HREF = "kspace_style.html">long-range
|
|
|
|
|
Coulombic solver</A> because the GPU is faster at
|
|
|
|
|
performing pairwise interactions, then this rule of thumb may give too
|
|
|
|
|
large a binsize.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>comm</I> and <I>comm/exchange</I> and <I>comm/forward</I> keywords determine
|
|
|
|
|
whether the host or device performs the packing and unpacking of data
|
|
|
|
|
when communicating per-atom data between processors. "Exchange"
|
|
|
|
|
communication happens only on timesteps that neighbor lists are
|
|
|
|
|
rebuilt. The data is only for atoms that migrate to new processors.
|
|
|
|
|
"Forward" communication happens every timestep. The data is for atom
|
|
|
|
|
coordinates and any other atom properties that needs to be updated for
|
|
|
|
|
ghost atoms owned by each processor.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>comm</I> keyword is simply a short-cut to set the same value
|
|
|
|
|
for both the <I>comm/exchange</I> and <I>comm/forward</I> keywords.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The value options for all 3 keywords are <I>no</I> or <I>host</I> or <I>device</I>.
|
|
|
|
|
A value of <I>no</I> means to use the standard non-KOKKOS method of
|
|
|
|
|
packing/unpacking data for the communication. A value of <I>host</I> means
|
|
|
|
|
to use the host, typically a multi-core CPU, and perform the
|
|
|
|
|
packing/unpacking in parallel with threads. A value of <I>device</I> means
|
|
|
|
|
to use the device, typically a GPU, to perform the packing/unpacking
|
|
|
|
|
operation.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The optimal choice for these keywords depends on the input script and
|
|
|
|
|
the hardware used. The <I>no</I> value is useful for verifying that the
|
|
|
|
|
Kokkos-based <I>host</I> and <I>device</I> values are working correctly. It may
|
|
|
|
|
also be the fastest choice when using Kokkos styles in MPI-only mode
|
|
|
|
|
(i.e. with a thread count of 1).
|
|
|
|
|
</P>
|
|
|
|
|
<P>When running on CPUs or Xeon Phi, the <I>host</I> and <I>device</I> values work
|
|
|
|
|
identically. When using GPUs, the <I>device</I> value will typically be
|
|
|
|
|
optimal if all of your styles used in your input script are supported
|
|
|
|
|
by the KOKKOS package. In this case data can stay on the GPU for many
|
|
|
|
|
timesteps without being moved between the host and GPU, if you use the
|
|
|
|
|
<I>device</I> value. This requires that your MPI is able to access GPU
|
|
|
|
|
memory directly. Currently that is true for OpenMPI 1.8 (or later
|
|
|
|
|
versions), Mvapich2 1.9 (or later), and CrayMPI. If your script uses
|
|
|
|
|
styles (e.g. fixes) which are not yet supported by the KOKKOS package,
|
|
|
|
|
then data has to be move between the host and device anyway, so it is
|
|
|
|
|
typically faster to let the host handle communication, by using the
|
|
|
|
|
<I>host</I> value. Using <I>host</I> instead of <I>no</I> will enable use of
|
|
|
|
|
multiple threads to pack/unpack communicated data.
|
|
|
|
|
</P>
|
|
|
|
|
<HR>
|
|
|
|
|
|
|
|
|
|
<P>The <I>omp</I> style invokes settings associated with the use of the
|
|
|
|
|
USER-OMP package.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>Nthread</I> argument sets the number of OpenMP threads allocated for
|
|
|
|
|
each MPI task. For example, if your system has nodes with dual
|
|
|
|
|
quad-core processors, it has a total of 8 cores per node. You could
|
|
|
|
|
use two MPI tasks per node (e.g. using the -ppn option of the mpirun
|
|
|
|
|
command in MPICH or -npernode in OpenMPI), and set <I>Nthreads</I> = 4.
|
|
|
|
|
This would use all 8 cores on each node. Note that the product of MPI
|
|
|
|
|
tasks * threads/task should not exceed the physical number of cores
|
|
|
|
|
(on a node), otherwise performance will suffer.
|
|
|
|
|
</P>
|
|
|
|
|
<P>Setting <I>Nthread</I> = 0 instructs LAMMPS to use whatever value is the
|
|
|
|
|
default for the given OpenMP environment. This is usually determined
|
|
|
|
|
via the <I>OMP_NUM_THREADS</I> environment variable or the compiler
|
|
|
|
|
runtime. Note that in most cases the default for OpenMP capable
|
|
|
|
|
compilers is to use one thread for each available CPU core when
|
|
|
|
|
<I>OMP_NUM_THREADS</I> is not explicitly set, which can lead to poor
|
|
|
|
|
performance.
|
|
|
|
|
</P>
|
|
|
|
|
<P>Here are examples of how to set the environment variable when
|
|
|
|
|
launching LAMMPS:
|
|
|
|
|
</P>
|
|
|
|
|
<PRE>env OMP_NUM_THREADS=4 lmp_machine -sf omp -in in.script
|
|
|
|
|
env OMP_NUM_THREADS=2 mpirun -np 2 lmp_machine -sf omp -in in.script
|
|
|
|
|
mpirun -x OMP_NUM_THREADS=2 -np 2 lmp_machine -sf omp -in in.script
|
|
|
|
|
</PRE>
|
|
|
|
|
<P>or you can set it permanently in your shell's start-up script.
|
|
|
|
|
All three of these examples use a total of 4 CPU cores.
|
|
|
|
|
</P>
|
|
|
|
|
<P>Note that different MPI implementations have different ways of passing
|
|
|
|
|
the OMP_NUM_THREADS environment variable to all MPI processes. The
|
|
|
|
|
2nd example line above is for MPICH; the 3rd example line with -x is
|
|
|
|
|
for OpenMPI. Check your MPI documentation for additional details.
|
|
|
|
|
</P>
|
|
|
|
|
<P>What combination of threads and MPI tasks gives the best performance
|
|
|
|
|
is difficult to predict and can depend on many components of your
|
|
|
|
|
input. Not all features of LAMMPS support OpenMP threading via the
|
|
|
|
|
USER-OMP packaage and the parallel efficiency can be very different,
|
|
|
|
|
too.
|
|
|
|
|
</P>
|
|
|
|
|
<P>Optional keyword/value pairs can also be specified. Each has a
|
|
|
|
|
default value as listed below.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The <I>neigh</I> keyword specifies whether neighbor list building will be
|
|
|
|
|
multi-threaded in addition to force calculations. If <I>neigh</I> is set
|
|
|
|
|
to <I>no</I> then neighbor list calculation is performed only by MPI tasks
|
|
|
|
|
with no OpenMP threading. If <I>mode</I> is <I>yes</I> (the default), a
|
|
|
|
|
multi-threaded neighbor list build is used. Using <I>neigh</I> = <I>yes</I> is
|
|
|
|
|
almost always faster and should produce idential neighbor lists at the
|
|
|
|
|
expense of using more memory. Specifically, neighbor list pages are
|
|
|
|
|
allocated for all threads at the same time and each thread works
|
|
|
|
|
within its own pages.
|
|
|
|
|
</P>
|
|
|
|
|
<HR>
|
|
|
|
|
|
|
|
|
|
<P><B>Restrictions:</B>
|
|
|
|
|
</P>
|
|
|
|
|
<P>This command cannot be used after the simulation box is defined by a
|
|
|
|
|
<A HREF = "read_data.html">read_data</A> or <A HREF = "create_box.html">create_box</A> command.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The cuda style of this command can only be invoked if LAMMPS was built
|
|
|
|
|
with the USER-CUDA package. See the <A HREF = "Section_start.html#start_3">Making
|
|
|
|
|
LAMMPS</A> section for more info.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The gpu style of this command can only be invoked if LAMMPS was built
|
|
|
|
|
with the GPU package. See the <A HREF = "Section_start.html#start_3">Making
|
|
|
|
|
LAMMPS</A> section for more info.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The intel style of this command can only be invoked if LAMMPS was
|
|
|
|
|
built with the USER-INTEL package. See the <A HREF = "Section_start.html#start_3">Making
|
|
|
|
|
LAMMPS</A> section for more info.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The kk style of this command can only be invoked if LAMMPS was built
|
|
|
|
|
with the KOKKOS package. See the <A HREF = "Section_start.html#start_3">Making
|
|
|
|
|
LAMMPS</A> section for more info.
|
|
|
|
|
</P>
|
|
|
|
|
<P>The omp style of this command can only be invoked if LAMMPS was built
|
|
|
|
|
with the USER-OMP package. See the <A HREF = "Section_start.html#start_3">Making
|
|
|
|
|
LAMMPS</A> section for more info.
|
|
|
|
|
</P>
|
|
|
|
|
<P><B>Related commands:</B>
|
|
|
|
|
</P>
|
|
|
|
|
<P><A HREF = "suffix.html">suffix</A>, "-pk" <A HREF = "Section_start.html#start_7">command-line
|
|
|
|
|
setting</A>
|
|
|
|
|
</P>
|
|
|
|
|
<P><B>Default:</B>
|
|
|
|
|
</P>
|
|
|
|
|
<P>For the USER-CUDA package, the default is Ngpu = 1 and the option
|
|
|
|
|
defaults are newton = off, gpuID = 0 to Ngpu-1, timing = not enabled,
|
|
|
|
|
test = not enabled, and thread = auto. These settings are made
|
|
|
|
|
automatically by the required "-c on" <A HREF = "Section_start.html#start_7">command-line
|
|
|
|
|
switch</A>. You can change them bu using the
|
|
|
|
|
package cuda command in your input script or via the "-pk cuda"
|
|
|
|
|
<A HREF = "Section_start.html#start_7">command-line switch</A>.
|
|
|
|
|
</P>
|
|
|
|
|
<P>For the GPU package, the default is Ngpu = 1 and the option defaults
|
|
|
|
|
are neigh = yes, newton = off, binsize = 0.0, split = 1.0, gpuID = 0
|
|
|
|
|
to Ngpu-1, tpa = 1, and device = not used. These settings are made
|
|
|
|
|
automatically if the "-sf gpu" <A HREF = "Section_start.html#start_7">command-line
|
|
|
|
|
switch</A> is used. If it is not used, you
|
|
|
|
|
must invoke the package gpu command in your input script or via the
|
|
|
|
|
"-pk gpu" <A HREF = "Section_start.html#start_7">command-line switch</A>.
|
|
|
|
|
</P>
|
|
|
|
|
<P>For the USER-INTEL package, the default is Nphi = 1 and the option
|
|
|
|
|
defaults are omp = 0, mode = mixed, balance = -1, tpc = 4, tptask =
|
|
|
|
|
240. The default ghost option is determined by the pair style being
|
|
|
|
|
used. This value is output to the screen in the offload report at the
|
|
|
|
|
end of each run. Note that all of these settings, except "omp" and
|
|
|
|
|
"mode", are ignored if LAMMPS was not built with Xeon Phi coprocessor
|
|
|
|
|
support. These settings are made automatically if the "-sf intel"
|
|
|
|
|
<A HREF = "Section_start.html#start_7">command-line switch</A> is used. If it is
|
|
|
|
|
not used, you must invoke the package intel command in your input
|
|
|
|
|
script or or via the "-pk intel" <A HREF = "Section_start.html#start_7">command-line
|
|
|
|
|
switch</A>.
|
|
|
|
|
</P>
|
|
|
|
|
<P>For the KOKKOS package, the option defaults neigh = full, newton =
|
|
|
|
|
off, binsize = 0.0, and comm = host. These settings are made
|
|
|
|
|
automatically by the required "-k on" <A HREF = "Section_start.html#start_7">command-line
|
|
|
|
|
switch</A>. You can change them bu using the
|
|
|
|
|
package kokkos command in your input script or via the "-pk kokkos"
|
|
|
|
|
<A HREF = "Section_start.html#start_7">command-line switch</A>.
|
|
|
|
|
</P>
|
|
|
|
|
<P>For the OMP package, the default is Nthreads = 0 and the option
|
|
|
|
|
defaults are neigh = yes. These settings are made automatically if
|
|
|
|
|
the "-sf omp" <A HREF = "Section_start.html#start_7">command-line switch</A> is
|
|
|
|
|
used. If it is not used, you must invoke the package omp command in
|
|
|
|
|
your input script or via the "-pk omp" <A HREF = "Section_start.html#start_7">command-line
|
|
|
|
|
switch</A>.
|
|
|
|
|
</P>
|
|
|
|
|
</HTML>
|
