-
Benjamin Cumming authored
Make the execution context presented to users an opaque handle, moving all implementation of the gpu, thread and distributed contexts into the back end. * move `execution_context` and `distributed_context` definitions to the back end * create `execution_context` handle called `context` in the public API * provide `make_context` helper functions that build different context configurations (default, user-specified local resources, with MPI) * update documentation for all parts of the public API that touch contexts * move `distributed_context` docs to the developer documentation (from the public API docs)
Unverifiedd637c8bc
Distributed Context
To support running on systems from laptops and workstations to large distributed HPC clusters, Arbor uses distributed contexts to:
- Describe the distributed computer system that a simulation is to be distributed over and run on.
- Perform collective operations over the distributed system, such as gather and synchronization.
- Query information about the distributed system, such as the number of distributed processes and the index/rank of the calling process.
The global context used to run a simulation is determined at run time, not at compile time. This means that if Arbor is compiled with support for MPI enabled, then at run time the user can choose between using a non-distributed (local) context, or an distributed MPI context.
An execution context is created by a user before building and running a simulation. This context is then used to perform domain decomposition and initialize the simulation (see :ref:`cppsimulation` for more about the simulation building workflow). In the example below, a context that uses MPI is used to run a distributed simulation:
The public API does not directly expose :cpp:class:`arb::distributed_context` or any of its implementations. By default :cpp:class:`arb::context` uses only local "on-node" resources. To use an MPI communicator for distributed communication, it can be initialised with the communicator:
arb::proc_allocation resources;
my_recipe recipe;
// Create a context that uses the local resources enumerated in resources,
// and that uses the standard MPI communicator MPI_COMM_WORLD for
// distributed communication.
arb::context context = arb::make_context(resources, MPI_COMM_WORLD);
// Partition model over the distributed system.
arb::domain_decomposition decomp = arb::partition_load_balance(recipe, context);
// Instatitate the simulation over the distributed system.
arb::simulation sim(recipe, decomp, context);
// Run the simulation for 100ms over the distributed system.
sim.run(100, 0.01);In the back end :cpp:class:`arb::distributed_context` defines the interface for distributed contexts, for which two implementations are provided: :cpp:class:`arb::local_context` and :cpp:class:`arb::mpi_context`. Distributed contexts are wrapped in shared pointers:
A distributed context can then be generated using helper functions :cpp:func:`arb::make_local_context` and :cpp:func:`arb::make_mpi_context`:
// Create a context that uses only local resources (is non-distributed).
auto dist_ctx arb::make_local_context();
// Create an MPI context that uses the standared MPI_COMM_WORLD communicator.
auto dist_ctx = arb::make_mpi_context(MPI_COMM_WORLD);