When an OpenFOAM simulation runs in parallel, the data for decomposed fields and
mesh(es) has historically been stored in multiple files within separate
directories for each processor. Processor directories are named 'processorN',
where N is the processor number.
This commit introduces an alternative "collated" file format where the data for
each decomposed field (and mesh) is collated into a single file, which is
written and read on the master processor. The files are stored in a single
directory named 'processors'.
The new format produces significantly fewer files - one per field, instead of N
per field. For large parallel cases, this avoids the restriction on the number
of open files imposed by the operating system limits.
The file writing can be threaded allowing the simulation to continue running
while the data is being written to file. NFS (Network File System) is not
needed when using the the collated format and additionally, there is an option
to run without NFS with the original uncollated approach, known as
"masterUncollated".
The controls for the file handling are in the OptimisationSwitches of
etc/controlDict:
OptimisationSwitches
{
...
//- Parallel IO file handler
// uncollated (default), collated or masterUncollated
fileHandler uncollated;
//- collated: thread buffer size for queued file writes.
// If set to 0 or not sufficient for the file size threading is not used.
// Default: 2e9
maxThreadFileBufferSize 2e9;
//- masterUncollated: non-blocking buffer size.
// If the file exceeds this buffer size scheduled transfer is used.
// Default: 2e9
maxMasterFileBufferSize 2e9;
}
When using the collated file handling, memory is allocated for the data in the
thread. maxThreadFileBufferSize sets the maximum size of memory in bytes that
is allocated. If the data exceeds this size, the write does not use threading.
When using the masterUncollated file handling, non-blocking MPI communication
requires a sufficiently large memory buffer on the master node.
maxMasterFileBufferSize sets the maximum size in bytes of the buffer. If the
data exceeds this size, the system uses scheduled communication.
The installation defaults for the fileHandler choice, maxThreadFileBufferSize
and maxMasterFileBufferSize (set in etc/controlDict) can be over-ridden within
the case controlDict file, like other parameters. Additionally the fileHandler
can be set by:
- the "-fileHandler" command line argument;
- a FOAM_FILEHANDLER environment variable.
A foamFormatConvert utility allows users to convert files between the collated
and uncollated formats, e.g.
mpirun -np 2 foamFormatConvert -parallel -fileHandler uncollated
An example case demonstrating the file handling methods is provided in:
$FOAM_TUTORIALS/IO/fileHandling
The work was undertaken by Mattijs Janssens, in collaboration with Henry Weller.
for consistency with reactingTwoPhaseEulerFoam and to ensure correct operation
of models requiring formal boundedness of phase-fractions.
Resolves bug-report https://bugs.openfoam.org/view.php?id=2589
These new names are more consistent and logical because:
primitiveField():
primitiveFieldRef():
Provides low-level access to the Field<Type> (primitive field)
without dimension or mesh-consistency checking. This should only be
used in the low-level functions where dimensional consistency is
ensured by careful programming and computational efficiency is
paramount.
internalField():
internalFieldRef():
Provides access to the DimensionedField<Type, GeoMesh> of values on
the internal mesh-type for which the GeometricField is defined and
supports dimension and checking and mesh-consistency checking.
In order to simplify expressions involving dimensioned internal field it
is preferable to use a simpler access convention. Given that
GeometricField is derived from DimensionedField it is simply a matter of
de-referencing this underlying type unlike the boundary field which is
peripheral information. For consistency with the new convention in
"tmp" "dimensionedInteralFieldRef()" has been renamed "ref()".
Non-const access to the internal field now obtained from a specifically
named access function consistent with the new names for non-canst access
to the boundary field boundaryFieldRef() and dimensioned internal field
dimensionedInternalFieldRef().
See also commit a4e2afa4b3
When the GeometricBoundaryField template class was originally written it
was a separate class in the Foam namespace rather than a sub-class of
GeometricField as it is now. Without loss of clarity and simplifying
code which access the boundary field of GeometricFields it is better
that GeometricBoundaryField be renamed Boundary for consistency with the
new naming convention for the type of the dimensioned internal field:
Internal, see commit a25a449c9e
This is a very simple text substitution change which can be applied to
any code which compiles with the OpenFOAM-dev libraries.
Given that the type of the dimensioned internal field is encapsulated in
the GeometricField class the name need not include "Field"; the type
name is "Internal" so
volScalarField::DimensionedInternalField -> volScalarField::Internal
In addition to the ".dimensionedInternalField()" access function the
simpler "()" de-reference operator is also provided to greatly simplify
FV equation source term expressions which need not evaluate boundary
conditions. To demonstrate this kEpsilon.C has been updated to use
dimensioned internal field expressions in the k and epsilon equation
source terms.
both of which return the dimensionedInternalField for volFields only.
These will be useful in FV equation source term expressions which need
not evaluate boundary conditions.
Resolves bug-report http://www.openfoam.org/mantisbt/view.php?id=1938
Because C++ does not support overloading based on the return-type there
is a problem defining both const and non-const member functions which
are resolved based on the const-ness of the object for which they are
called rather than the intent of the programmer declared via the
const-ness of the returned type. The issue for the "boundaryField()"
member function is that the non-const version increments the
event-counter and checks the state of the stored old-time fields in case
the returned value is altered whereas the const version has no
side-effects and simply returns the reference. If the the non-const
function is called within the patch-loop the event-counter may overflow.
To resolve this it in necessary to avoid calling the non-const form of
"boundaryField()" if the results is not altered and cache the reference
outside the patch-loop when mutation of the patch fields is needed.
The most straight forward way of resolving this problem is to name the
const and non-const forms of the member functions differently e.g. the
non-const form could be named:
mutableBoundaryField()
mutBoundaryField()
nonConstBoundaryField()
boundaryFieldRef()
Given that in C++ a reference is non-const unless specified as const:
"T&" vs "const T&" the logical convention would be
boundaryFieldRef()
boundaryFieldConstRef()
and given that the const form which is more commonly used is it could
simply be named "boundaryField()" then the logical convention is
GeometricBoundaryField& boundaryFieldRef();
inline const GeometricBoundaryField& boundaryField() const;
This is also consistent with the new "tmp" class for which non-const
access to the stored object is obtained using the ".ref()" member function.
This new convention for non-const access to the components of
GeometricField will be applied to "dimensionedInternalField()" and "internalField()" in the
future, i.e. "dimensionedInternalFieldRef()" and "internalFieldRef()".
There is a need to specify const or non-const access to a non-const
object which is not currently possible with the "boundaryField()" access
function the const-ness of the return of which is defined by the
const-ness of the object for which it is called. For consistency with
the latest "tmp" storage class in which non-const access is obtained
with the "ref()" function it is proposed to replace the non-const form
of "boundaryField()" with "boundaryFieldRef()".
Thanks to Mattijs Janssens for starting the process of migration to
"boundaryFieldRef()" and providing a patch for the OpenFOAM and
finiteVolume libraries.
Foam::direction is an unsigned type which makes it easier for the
compiler to pickup and report errors in the instantiation of
VectorSpaces and associated types.
in case of tmp misuse.
Simplified tmp reuse pattern in field algebra to use tmp copy and
assignment rather than the complex delayed call to 'ptr()'.
Removed support for unused non-const 'REF' storage of non-tmp objects due to C++
limitation in constructor overloading: if both tmp(T&) and tmp(const T&)
constructors are provided resolution is ambiguous.
The turbulence libraries have been upgraded and '-DCONST_TMP' option
specified in the 'options' file to switch to the new 'tmp' behavior.
To be used instead of zeroGradientFvPatchField for temporary fields for
which zero-gradient extrapolation is use to evaluate the boundary field
but avoiding fields derived from temporary field using field algebra
inheriting the zeroGradient boundary condition by the reuse of the
temporary field storage.
zeroGradientFvPatchField should not be used as the default patch field
for any temporary fields and should be avoided for non-temporary fields
except where it is clearly appropriate;
extrapolatedCalculatedFvPatchField and calculatedFvPatchField are
generally more suitable defaults depending on the manner in which the
boundary values are specified or evaluated.
The entire OpenFOAM-dev code-base has been updated following the above
recommendations.
Henry G. Weller
CFD Direct
This resolves a whole range of issues and work-arounds with earlier
releases. This version of icpc is more or less compatible with the
latest gcc and clang compilers and only required one hack to avoid
warnings from PackedBoolList.H.
