DFT-FE 1.1.0-pre
Density Functional Theory With Finite-Elements
Loading...
Searching...
No Matches
Public Types | Public Member Functions | Private Attributes | List of all members
dftfe::utils::mpi::MPIPatternP2P< memorySpace > Class Template Reference

A class template to store the communication pattern (i.e., which entries/nodes to receive from which processor and which entries/nodes to send to which processor). More...

#include <MPIPatternP2P.h>

Public Types

using SizeTypeVector = utils::MemoryStorage<size_type, memorySpace>
 
using GlobalSizeTypeVector
 

Public Member Functions

virtual ~MPIPatternP2P ()=default
 
 MPIPatternP2P (const std::pair< global_size_type, global_size_type > &locallyOwnedRange, const std::vector< dftfe::global_size_type > &ghostIndices, const MPI_Comm &mpiComm)
 Constructor. This constructor is the typical way of creation of an MPI pattern.
 
 MPIPatternP2P (const size_type size)
 Constructor. This constructor is to create an MPI Pattern for a serial case. This is provided so that one can seamlessly use has to be used even for a serial case. In this case, all the indices are owned by the current processor.
 
std::pair< global_size_type, global_size_type > getLocallyOwnedRange () const
 
size_type localOwnedSize () const
 
size_type localGhostSize () const
 
bool inLocallyOwnedRange (const global_size_type globalId) const
 
bool isGhostEntry (const global_size_type globalId) const
 
size_type globalToLocal (const global_size_type globalId) const
 
global_size_type localToGlobal (const size_type localId) const
 
const std::vector< global_size_type > & getGhostIndices () const
 
const std::vector< size_type > & getGhostProcIds () const
 
const std::vector< size_type > & getNumGhostIndicesInProcs () const
 
size_type getNumGhostIndicesInProc (const size_type procId) const
 
SizeTypeVector getGhostLocalIndices (const size_type procId) const
 
const std::vector< size_type > & getGhostLocalIndicesRanges () const
 
const std::vector< size_type > & getTargetProcIds () const
 
const std::vector< size_type > & getNumOwnedIndicesForTargetProcs () const
 
size_type getNumOwnedIndicesForTargetProc (const size_type procId) const
 
const SizeTypeVector & getOwnedLocalIndicesForTargetProcs () const
 
SizeTypeVector getOwnedLocalIndices (const size_type procId) const
 
size_type nmpiProcesses () const
 
size_type thisProcessId () const
 
global_size_type nGlobalIndices () const
 
const MPI_Comm & mpiCommunicator () const
 
bool isCompatible (const MPIPatternP2P< memorySpace > &rhs) const
 

Private Attributes

std::pair< global_size_type, global_size_type > d_locallyOwnedRange
 
std::vector< global_size_type > d_allOwnedRanges
 
size_type d_numLocallyOwnedIndices
 
size_type d_numGhostIndices
 
std::vector< global_size_type > d_ghostIndices
 
std::set< global_size_type > d_ghostIndicesSetSTL
 
OptimizedIndexSet< global_size_type > d_ghostIndicesOptimizedIndexSet
 
size_type d_numGhostProcs
 
std::vector< size_type > d_ghostProcIds
 
std::vector< size_type > d_numGhostIndicesInGhostProcs
 
SizeTypeVector d_flattenedLocalGhostIndices
 
std::vector< size_type > d_localGhostIndicesRanges
 A vector of size 2 times the number of ghost processors to store the range of local ghost indices that are owned by the ghost processors. In other words, it stores the list $L=\{a_1,b_1, a_2, b_2, \ldots, a_G, b_G\}$, where $a_i$ and $b_i$is are the start local ghost index and one-past-the-last local ghost index of the current processor that is owned by the $i$-th ghost processor (i.e., d_ghostProcIds[i]). Put it differently, $[a_i,b_i)$ form an open interval, where $a_i$ is included but $b_i$ is not included.
 
size_type d_numTargetProcs
 
std::vector< size_type > d_targetProcIds
 
std::vector< size_type > d_numOwnedIndicesForTargetProcs
 
SizeTypeVector d_flattenedLocalTargetIndices
 
int d_nprocs
 Number of processors in the MPI Communicator.
 
int d_myRank
 Rank of the current processor.
 
global_size_type d_nGlobalIndices
 
MPI_Comm d_mpiComm
 MPI Communicator object.
 

Detailed Description

template<dftfe::utils::MemorySpace memorySpace>
class dftfe::utils::mpi::MPIPatternP2P< memorySpace >

A class template to store the communication pattern (i.e., which entries/nodes to receive from which processor and which entries/nodes to send to which processor).

  • Assumptions
    1. It assumes that a a sparse communication pattern. That is, a given processor only communicates with a few processors. This object should be avoided if the communication pattern is dense (e.g., all-to-all communication)
    2. It assumes that the each processor owns a set of continuous integers (indices). Further, the ownership is exclusive (i.e., no index is owned by more than one processor). In other words, the different sets of owning indices across all the processors are disjoint.
Template Parameters
memorySpaceDefines the MemorySpace (i.e., HOST or DEVICE) in which the various data members of this object must reside.

Member Typedef Documentation

◆ GlobalSizeTypeVector

template<dftfe::utils::MemorySpace memorySpace>
using dftfe::utils::mpi::MPIPatternP2P< memorySpace >::GlobalSizeTypeVector
Initial value:
utils::MemoryStorage<global_size_type, memorySpace>
dftfe::utils::MemoryStorage
Definition MemoryStorage.h:33

◆ SizeTypeVector

template<dftfe::utils::MemorySpace memorySpace>
using dftfe::utils::mpi::MPIPatternP2P< memorySpace >::SizeTypeVector = utils::MemoryStorage<size_type, memorySpace>

typedefs

Constructor & Destructor Documentation

◆ ~MPIPatternP2P()

template<dftfe::utils::MemorySpace memorySpace>
virtual dftfe::utils::mpi::MPIPatternP2P< memorySpace >::~MPIPatternP2P ( )
virtualdefault

◆ MPIPatternP2P() [1/2]

template<dftfe::utils::MemorySpace memorySpace>
dftfe::utils::mpi::MPIPatternP2P< memorySpace >::MPIPatternP2P ( const std::pair< global_size_type, global_size_type > & locallyOwnedRange,
const std::vector< dftfe::global_size_type > & ghostIndices,
const MPI_Comm & mpiComm )

Constructor. This constructor is the typical way of creation of an MPI pattern.

Parameters
[in]locallyOwnedRangeA pair of non-negtive integers $(a,b)$ which defines a range of indices (continuous) that are owned by the current processor.
Note
It is an open interval where $a$ is included, but $b$ is not included.
Parameters
[in]ghostIndicesAn ordered set of non-negtive indices specifyin the ghost indices for the current processor.
Note
the vector must be ordered (i.e., ordered in increasing order and non-repeating)
Parameters
[in]mpiCommThe MPI communicator object which defines the set of processors for which the MPI pattern needs to be created.
Exceptions
Throwsexception if mpiComm is in an invalid state, if the locallyOwnedRange across all the processors are not disjoint, if ghostIndices are not ordered (if it is not strictly increasing), or if some sanity checks with respect to MPI sends and receives fail.
Note
Care is taken to create a dummy MPIPatternP2P while not linking to an MPI library. This allows the user code to seamlessly link and delink an MPI library.

Constructor with MPI

◆ MPIPatternP2P() [2/2]

template<dftfe::utils::MemorySpace memorySpace>
dftfe::utils::mpi::MPIPatternP2P< memorySpace >::MPIPatternP2P ( const size_type size)

Constructor. This constructor is to create an MPI Pattern for a serial case. This is provided so that one can seamlessly use has to be used even for a serial case. In this case, all the indices are owned by the current processor.

Parameters
[in]sizeTotal number of indices.
Note
This is an explicitly serial construction (i.e., it uses MPI_COMM_SELF), which is different from the dummy MPIPatternP2P created while not linking to an MPI library. For examples, within a parallel run, one might have the need to create a serial MPIPatternP2P. A typical case is creation of a serial vector as a special case of distributed vector.
Similar to the previous constructor, care is taken to create a dummy MPIPatternP2P while not linking to an MPI library.

Constructor for a serial case

Member Function Documentation

◆ getGhostIndices()

template<dftfe::utils::MemorySpace memorySpace>
const std::vector< global_size_type > & dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getGhostIndices ( ) const

◆ getGhostLocalIndices()

template<dftfe::utils::MemorySpace memorySpace>
MPIPatternP2P< memorySpace >::SizeTypeVector dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getGhostLocalIndices ( const size_type procId) const

◆ getGhostLocalIndicesRanges()

template<dftfe::utils::MemorySpace memorySpace>
const std::vector< size_type > & dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getGhostLocalIndicesRanges ( ) const

◆ getGhostProcIds()

template<dftfe::utils::MemorySpace memorySpace>
const std::vector< size_type > & dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getGhostProcIds ( ) const

◆ getLocallyOwnedRange()

template<dftfe::utils::MemorySpace memorySpace>
std::pair< global_size_type, global_size_type > dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getLocallyOwnedRange ( ) const

◆ getNumGhostIndicesInProc()

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getNumGhostIndicesInProc ( const size_type procId) const

◆ getNumGhostIndicesInProcs()

template<dftfe::utils::MemorySpace memorySpace>
const std::vector< size_type > & dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getNumGhostIndicesInProcs ( ) const

◆ getNumOwnedIndicesForTargetProc()

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getNumOwnedIndicesForTargetProc ( const size_type procId) const

◆ getNumOwnedIndicesForTargetProcs()

template<dftfe::utils::MemorySpace memorySpace>
const std::vector< size_type > & dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getNumOwnedIndicesForTargetProcs ( ) const

◆ getOwnedLocalIndices()

template<dftfe::utils::MemorySpace memorySpace>
MPIPatternP2P< memorySpace >::SizeTypeVector dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getOwnedLocalIndices ( const size_type procId) const

◆ getOwnedLocalIndicesForTargetProcs()

template<dftfe::utils::MemorySpace memorySpace>
const MPIPatternP2P< memorySpace >::SizeTypeVector & dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getOwnedLocalIndicesForTargetProcs ( ) const

◆ getTargetProcIds()

template<dftfe::utils::MemorySpace memorySpace>
const std::vector< size_type > & dftfe::utils::mpi::MPIPatternP2P< memorySpace >::getTargetProcIds ( ) const

◆ globalToLocal()

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::globalToLocal ( const global_size_type globalId) const

◆ inLocallyOwnedRange()

template<dftfe::utils::MemorySpace memorySpace>
bool dftfe::utils::mpi::MPIPatternP2P< memorySpace >::inLocallyOwnedRange ( const global_size_type globalId) const

◆ isCompatible()

template<dftfe::utils::MemorySpace memorySpace>
bool dftfe::utils::mpi::MPIPatternP2P< memorySpace >::isCompatible ( const MPIPatternP2P< memorySpace > & rhs) const

◆ isGhostEntry()

template<dftfe::utils::MemorySpace memorySpace>
bool dftfe::utils::mpi::MPIPatternP2P< memorySpace >::isGhostEntry ( const global_size_type globalId) const

◆ localGhostSize()

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::localGhostSize ( ) const

◆ localOwnedSize()

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::localOwnedSize ( ) const

◆ localToGlobal()

template<dftfe::utils::MemorySpace memorySpace>
global_size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::localToGlobal ( const size_type localId) const

◆ mpiCommunicator()

template<dftfe::utils::MemorySpace memorySpace>
const MPI_Comm & dftfe::utils::mpi::MPIPatternP2P< memorySpace >::mpiCommunicator ( ) const

◆ nGlobalIndices()

template<dftfe::utils::MemorySpace memorySpace>
global_size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::nGlobalIndices ( ) const

◆ nmpiProcesses()

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::nmpiProcesses ( ) const

◆ thisProcessId()

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::thisProcessId ( ) const

Member Data Documentation

◆ d_allOwnedRanges

template<dftfe::utils::MemorySpace memorySpace>
std::vector<global_size_type> dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_allOwnedRanges
private

A vector of size 2 times number of processors to store the locallyOwnedRange of each processor. That is it store the list $[a_0,b_0,a_1,b_1,\ldots,a_{P-1},b_{P-1}]$, where the pair $(a_i,b_i)$ defines a range of indices (continuous) that are owned by the $i-$th processor.

Note
$a$ is included but $b$ is not included.

◆ d_flattenedLocalGhostIndices

template<dftfe::utils::MemorySpace memorySpace>
SizeTypeVector dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_flattenedLocalGhostIndices
private

A flattened vector of size number of ghosts containing the ghost indices ordered as per the list of ghost processor Ids in d_ghostProcIds In other words it stores a concatentaion of the lists $L_i = \{g^{(k_i)}_1,g^{(k_i)}_2,\ldots,g^{(k_i)}_{N_i}\}$, where $g$'s are the ghost indices, $k_i$ is the rank of the $i$-th ghost processor (i.e., d_ghostProcIds[i]) and $N_i$ is the number of ghost indices owned by the $i$-th ghost processor (i.e., d_numGhostIndicesInGhostProcs[i]).

Note
$L_i$ has to be an increasing set.
We store only the ghost index local to this processor, i.e., position of the ghost index in d_ghostIndicesSetSTL or d_ghostIndices. This is done to use size_type which is unsigned int instead of global_size_type which is long unsigned it. This helps in reducing the volume of data transfered during MPI calls.
In the case that the locally owned ranges across all the processors are ordered as per the processor Id, this vector is redundant and one can only work with d_ghostIndices and d_numGhostIndicesInGhostProcs. By locally owned range being ordered as per the processor Id, means that the ranges for processor $0, 1,\ldots,P-1$ are $[N_0,N_1), [N_1, N_2), [N_2, N_3), ..., [N_{P-1},N_P)$ with $N_0, N_1,\ldots, N_P$ beign non-decreasing. But in a more general case, where the locally owned ranges are not ordered as per the processor Id, this following array is useful.

◆ d_flattenedLocalTargetIndices

template<dftfe::utils::MemorySpace memorySpace>
SizeTypeVector dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_flattenedLocalTargetIndices
private

Vector of size $\sum_i$ d_numOwnedIndicesForTargetProcs[i] to store all thelocally owned indices which other processors need (i.e., which are ghost indices in other processors). It is stored as a concatentation of lists where the $i$-th list indices $L_i = \{o^{(k_i)}_1,o^{(k_i)}_2,\ldots,o^{(k_i)}_{N_i}\}$, where where $o$'s are indices target to other processors, $k_i$ is the rank of the $i$-th target processor (i.e., d_targetProcIds[i]) and N_i is the number of indices to be sent to i-th target processor (i.e., d_numOwnedIndicesForTargetProcs[i]).

Note
We store only the indices local to this processor, i.e., the relative position of the index in the locally owned range of this processor This is done to use size_type which is unsigned int instead of global_size_type which is long unsigned it. This helps in reducing the volume of data transfered during MPI calls.
The list $L_i$ must be ordered.

◆ d_ghostIndices

template<dftfe::utils::MemorySpace memorySpace>
std::vector<global_size_type> dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_ghostIndices
private

Vector to store an ordered set of ghost indices (ordered in increasing order and non-repeating)

◆ d_ghostIndicesOptimizedIndexSet

template<dftfe::utils::MemorySpace memorySpace>
OptimizedIndexSet<global_size_type> dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_ghostIndicesOptimizedIndexSet
private

An OptimizedIndexSet object to store the ghost indices for efficient operations. The OptimizedIndexSet internally creates contiguous sub-ranges within the set of indices and hence can optimize the finding of an index

◆ d_ghostIndicesSetSTL

template<dftfe::utils::MemorySpace memorySpace>
std::set<global_size_type> dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_ghostIndicesSetSTL
private

A copy of the above d_ghostIndices stored as an STL set

◆ d_ghostProcIds

template<dftfe::utils::MemorySpace memorySpace>
std::vector<size_type> dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_ghostProcIds
private

Vector to store the ghost processor Ids. A ghost processor is one which owns at least one of the ghost indices of this processor.

◆ d_localGhostIndicesRanges

template<dftfe::utils::MemorySpace memorySpace>
std::vector<size_type> dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_localGhostIndicesRanges
private

A vector of size 2 times the number of ghost processors to store the range of local ghost indices that are owned by the ghost processors. In other words, it stores the list $L=\{a_1,b_1, a_2, b_2, \ldots, a_G, b_G\}$, where $a_i$ and $b_i$is are the start local ghost index and one-past-the-last local ghost index of the current processor that is owned by the $i$-th ghost processor (i.e., d_ghostProcIds[i]). Put it differently, $[a_i,b_i)$ form an open interval, where $a_i$ is included but $b_i$ is not included.

Note
Given the fact that the locally owned indices of each processor are contiguous and the global ghost indices (i.e., d_ghostIndices) is ordered, it is sufficient to just store the range of local ghost indices for each ghost procId. The actual global ghost indices belonging to the $i$-th ghost processor can be fetched from d_ghostIndices (i.e., it is the subset of d_ghostIndices lying bewteen d_ghostIndices[a_i] and d_ghostIndices[b_i].

◆ d_locallyOwnedRange

template<dftfe::utils::MemorySpace memorySpace>
std::pair<global_size_type, global_size_type> dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_locallyOwnedRange
private

A pair $(a,b)$ which defines a range of indices (continuous) that are owned by the current processor.

Note
It is an open interval where $a$ is included, but $b$ is not included.

◆ d_mpiComm

template<dftfe::utils::MemorySpace memorySpace>
MPI_Comm dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_mpiComm
private

MPI Communicator object.

◆ d_myRank

template<dftfe::utils::MemorySpace memorySpace>
int dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_myRank
private

Rank of the current processor.

◆ d_nGlobalIndices

template<dftfe::utils::MemorySpace memorySpace>
global_size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_nGlobalIndices
private

Total number of unique indices across all processors

◆ d_nprocs

template<dftfe::utils::MemorySpace memorySpace>
int dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_nprocs
private

Number of processors in the MPI Communicator.

◆ d_numGhostIndices

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_numGhostIndices
private

Number of ghost indices in the current processor

◆ d_numGhostIndicesInGhostProcs

template<dftfe::utils::MemorySpace memorySpace>
std::vector<size_type> dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_numGhostIndicesInGhostProcs
private

Vector of size number of ghost processors to store how many ghost indices of this current processor are owned by a ghost processor.

◆ d_numGhostProcs

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_numGhostProcs
private

Number of ghost processors for the current processor. A ghost processor is one which owns at least one of the ghost indices of this processor.

◆ d_numLocallyOwnedIndices

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_numLocallyOwnedIndices
private

Number of locally owned indices in the current processor

◆ d_numOwnedIndicesForTargetProcs

template<dftfe::utils::MemorySpace memorySpace>
std::vector<size_type> dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_numOwnedIndicesForTargetProcs
private

Vector of size number of target processors to store how many locally owned indices of this current processor are need ghost in each of the target processors.

◆ d_numTargetProcs

template<dftfe::utils::MemorySpace memorySpace>
size_type dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_numTargetProcs
private

Number of target processors for the current processor. A target processor is one which owns at least one of the locally owned indices of this processor as its ghost index.

◆ d_targetProcIds

template<dftfe::utils::MemorySpace memorySpace>
std::vector<size_type> dftfe::utils::mpi::MPIPatternP2P< memorySpace >::d_targetProcIds
private

Vector to store the target processor Ids. A target processor is one which contains at least one of the locally owned indices of this processor as its ghost index.

The documentation for this class was generated from the following files:
Generated by doxygen 1.13.2