Sam Yates authored 6 years ago and Benjamin Cumming committed 6 years ago

CMake and build refactoring

*   Use CUDA as first-class language (leading to CMake 3.9 minimum version requirement).

*   Use 'modern CMake' interface libraries for compiler options, include file and library dependency tracking. Interface library targets:
    * `arbor-deps`: compiler options and library requirements for the `libarbor.a` static library, as governed by configure-time options and environment.
    * `arbor-private-headers`: include path for non-installed headers, as required by unit tests and arbor itself.
    * `arbor-aux`: helper classes and utilities used across tests and examples.
    * `ext-json`, `ext-tclap`, `ext-tbb`, `ext-benchmark`, `ext-sphinx_rtd_theme`: externally maintained software that we include (directly or via submodule) in the `ext/` subdirectory.
 
*   Single static library `libarbor.a` includes all built-in modules and CUDA objects.

*   Simply configuration options:
    *  `ARB_WITH_TRACE`, `ARB_AUTORUN_MODCC_ON_CHA...

Add a new cell type, and corresponding cell_group implementation, for benchmarking the simulator library architecture.

Add an benchmark_cell_group, where each cell in the group

generates a spike train prescribed by a time_seq
takes a prescribed time interval per cell to perform the cell_group::advance method.
With this cell type, one can easily build arbitrary networks with prescribed spiking and cell update overheads.
A miniapp that uses this cell type to build a benchmark model is implemented in example/bench.

Fixes #493
Fixes #501

The built in profiler generates timings for state and current for individual multicore mechanisms.

Modcc generates and PE(advance_integrate_{state,current}_X) profiler calls (along with corresponding PL() for calls to multicore mechanism nrn_state and nrn_current API calls.

No timings are made for the gpu back end, which is not properly supported by the current profiling tools.

noraabiakar authored 6 years ago and Benjamin Cumming committed 6 years ago

Changes have been made to the simd implementation of mechansim functions: 

- The node_index array (array of indices that specifies for each mechanism the CVs where it is present), is now partitioned into 4 arrays according to the constraint on each simd_vector in node_index:
    1. contiguous array: contains the indices of all simd_vectors in node_index where the elements in simd_vector are contiguous
    2. constant array: contains the indices of all simd_vectors in node_index where the elements in simd_vector are identical
    3. independent array: contains the indices of all simd_vectors in node_index where the elements in simd_vector are independent (no repetitions) but not contiguous 
    4. none array: contains the indices of all simd_vectors in node_index where the none of the above constraints apply

    When mechanism functions are executed, they loop over each of the 4 arrays separately. This allows for optimizations in every category. 

- The modcc compiler was modified to generate code for the previous changes, including the optimizations per constraint:
    1. contiguous array: we use vector load/store and vector arithmetic. 
    2. constant array: we load only one element and broadcast it into a simd_vector; we use vector arithmetic; we reduce the result; we store one element.   
    3. indepndent array: we use vector scatter/gather and vector arithmetic. 
    4. none array: we cannot operate on the simd_vector in parallel, we loop over the elements to read, perform arithmetic and write back 

- Added a mechanism benchmark for pas, hh and expsyn

- Moved/modified some functions in simd.hpp to ensure that the correct implementation of a function is being called. 

Changes to libarbor
-------------------------

Time sequences were added in `src/time_sequence.hpp`:
- added new `time_seq` type that implements a type-erasure interface for the
  concept of a time sequence generator.
- added poisson, regular and vector-backed implementations of the time sequence
  concept.

Event generators:
- The poisson, regular and vector-backed implementations of the event generator
  concept were refactored to use the.

Cell groups:
- Removed the `dss_cell_group` and `rss_cell_group` and associated types.
- Added a generic spike source cell  that generates a sequence of spikes
  at time points specified by a `time_seq`. Using this approach, an
  additional `cell_group` specialization is not required for each type of
  sequence, and user-defined sequences can be used with minimal overhead.

Unit tests
------------

- Added unit tests for `time_seq`.
- Simplified `event_generator` unit tests, because much of the testing
  of the sequences was moved to the `time_seq` tests.
- Added unit tests for `spike_source_cell_group`.

Changes to miniapp
-------------------------

- simplified the miniapp by removing the command line options for using an input spike chain from file.
- updated the miniapp recipe to use `spike_source` cell group instead of `dss_cell_group`.

Benjamin Cumming authored 6 years ago and Sam Yates committed 6 years ago

Move from choosing the distributed communication model from a compile time choice (the old `arb::communication::communication_policy` type) to a run time decision.

* Add `arb::distributed_context` class that provides the required interface for distributed communication implementations, using type-erasure to provide value semantics.
* Add two implementations for the distributed context: `arb::mpi_context` and `arb::local_context`.
* Allow distribution over a user-supplied MPI communicator by providing it as an argument to `arb::mpi_context`.
* Add `mpi_error` exception type to wrap MPI errors.
* Move contents of the `arb::communication` namespace to the `arb` namespace.
* Add preprocessor for-each utility `ARB_PP_FOREACH`.
* Rewrite all examples and tests to use the new distributed context interface.
* Add documentation for distributed context class and semantics, and update documentation for load balancer and simulation classes accordingly.

Fixes #472

Sam Yates authored 6 years ago and Benjamin Cumming committed 6 years ago

Replace hard-coded index variable names in modcc cuda printer with ones derived from the external variable.

Uses `decode_indexed_variable`.

Benjamin Cumming authored 6 years ago and Sam Yates committed 6 years ago

* Use `sys/types.h` instead of `endian.h` for greater portability.
* Avoid use of constructor for `std::vector` in unit tests that is only available from C++14.

* Update SIMD developer docs to reflect newly merged mechanism refactor work.

Benjamin Cumming authored 6 years ago and Sam Yates committed 6 years ago

Completes CUDA printing in modcc.
* Add CudaPrinter visitor, overriding CPrinter.
* Add `ostream` `operator<<` overloads for `arb::gpu::shared_state` and `device_view` for debugging.
* Fix GPU back-end bugs.

First commit of two for mechanism refactor work (refer to PR #484 and PR #483).

FVM/mechanism code:
* Refactor mechanism data structures to decouple backend-specific implementations and mechanism metadata.
* Add mechanism catalogue for managing mechanism metadata and concrete implementation prototypes.
* Add fingerprint-checking to mechanism metadata and implementations to confirm they come from the same NMODL source (fingerprint is not yet computed, but tests are in place).
* Split FVM discretization work out from FVM integrator code.
* Use abstract base class over backend-templated FVM integrator class `fvm_lowered_cell_impl` to allow separate compilation of `mc_cell_group` and to remove the dummy backend code.
* Add a new FVM-specific scalar type `fvm_index_type` that is an alias for `int` to replace
`fvm_size_type` in fvm layouts and mechanisms. This was chosen as an alternative
to making `unsigned` versions of all our SIMD implementation classes.
* Extend `cable1d_neuron` global data to encompass: mechanism catalogue; default ion concentrations and charges; global temperature (only for Nernst); initial membrane potential.

Modcc:
* Collect printer sources in modcc under `printer/`.
* Move common functionality across printers into `printer/printerutil.{hpp,cpp}`.
* Add string to file I/O implemented in routines read_all and write_all in `io/bulkio.hpp`.
* Implement indent-friendly source code generation via a `std::streambuf` filter `io::prefixbuf` defined in `io/prefixbuf.hpp`, together with manipulators and a corresponding std::ostream-derived wrapper.
* Rewrite printers to use new infrastructure: cpu target incorporates SIMD printing options; CUDA printer at this point produces only stubs for CUDA kernel wrappers.
* Modify SIMD printing command line options for modcc: `-s` enables explicit vectorization using the SIMD classes;  `-S <N>` allows a specific data width to be prescribed.
* Fix problem in `test_ca.mod` with uninitialized ion current.
* Add infrastructure support to allow future pre-computation of SIMD index conflict cases for (hopefully) faster scatters and updates.
* Simplify `IndexedVariable` expressions in the AST, making data source explicit via a `sourceKind` enum, and leaving the indexing method and index names up to the printers.
* Allow state variables in the AST to 'shadow' an ion concentration — these are assigned in the
generated `write_ions` method.

SIMD classes:
* Add `simd_cast` operation between SIMD value types of the same width, and with `std::array`. (Note: this was tested and used in an early development version of the code, but not in this version. It was still a lacuna in the original SIMD wrappers, so it has been left in.)
* Restructure SIMD gather/scatter API to use a `simd::indirect` expression,  which encapsulates a pointer and SIMD offset.
* Add `simd::index_constraint` scoped enum to describe knowledge of contention in indirect indices, so that we can branch on this to the appropriate implementation.
* Add SIMD concrete implementation routines `reduce_add` for horizontal reduction and `element0` for access to first lane scalar value.
* Add SIMD value method `sum()` that exposes implementation `reduce_add`.
* Add SIMD concrete implementation routine `compound_indexed_add` that provides the implementation for `indirect(p, simd_indices) += simd_value` construction.
* Fix SIMD `implbase` bug where some static methods were using the `implbase` fall-back functions instead of the derived class specialized implementations.
* Move SIMD mathematical functions into friend routines of `simd_impl` in order to resolve implicit conversions from scalars in mixed SIMD-scalar operations.
* Use a templated `tag` class to dispatch on SIMD concrete implementation types, to avoid problems with incomplete types in method signatures.
* Remove old SIMD intrinsics.

CMake infrastructure:
* Downcase some variables in `CMakeLists.txt` files to  distinguish them visually from CMake keywords and variables.
* Split arbor modcc vectorization option (now `ARB_VECTORIZE`) and target-architecture optimization (now `ARB_ARCH`).
* For `arbor` and `arbormech` targets, and in particular not the `modcc` target, use `ARB_ARCH` to generate corresponding target-appropriate binaries, including, for example, appropriate SIMD support.
* Extend `CompilerOptions.cmake` to map as best as able between the various target architecture names (we use the gcc names) and the correct option to pass to the compiler based on the compiler and platform.
* Add work-around for misidentification by CMake of XL C as Clang.
* As a temporary work-around, include `arbormech` library twice on link line to resolve circular arbor–arbormech dependencies.

Unit tests:
* Extend repertoire of generic sequence equality/near equality testing support  in `common.hpp`.
* Add warning suppression for icc for the malloc instrumentation code.
* SIMD unit tests for indirect expressions, compound indirect add, reduction.
* Make some exact tests into floating point 'near' tests when comparing computed areas and lengths in swc and fvm layout tests, to account for compiler (e.g. icc) performing semantically inequivalent floating point operation reordering or fusion at `-O3`.
* Split out some of the CUDA tests into separate .cpp/.cu files for  separate-compilation purposes.

Other:
* The `padded_allocator` has been modified to propagate alignment/padding on move and copy (these semantics make their use much easier and safer in the multicore mechanism instantiation code).
* Map/table searching utilities in `util/maputil.hpp`.
* Fixes for correct sequence type categorization and `begin/end` ADL.
* Fixes for type guards for range methods that take universal references.
* Removal of some redundant code in range utilities through the use of universal references.
* Add new range view `reverse_view` for ranges delineated by bidirectional iterators.
* Add single argument form of `make_span` to count up from zero, and associated helper `count_along` that gives a span that indexes a supplied container.
* Moved `prefixbuf` to `modcc` source.
* Make sequence positive and negative tests in algorithms generic.
* Add `private`-subverting helper code/macro to `tests/unit/common.hpp` to reduce the number of public testing-only interfaces in the library code.
* Add virtual destructors for virtual base classes.
* Add new arb::math:: functions: `next_pow2` for unsigned integral types, `round_up` to round a number away from zero to next largest magnitude multiple.
* New `index_into` implementation that supports bidirectional access (moved to `util::` namespace).
* Fix problem in `test_ca.mod` with uninitialized ion current.
* Rework dangerous `memory::array(Iter, Iter)` constructor to be less dangerous (and do the expected thing).
* Allow ranges to be constructed from other ranges if the iterators are compatible.

Ben Cumming authored 6 years ago and Sam Yates committed 6 years ago

Add two new documentation pages for the C++ API

* Add domain decomposition page that covers `domain_decomposition`, `node_info` and `partition_load_balance`.
* Add simulation page that describes `arb::simulation` API interface.
* Fix some small typos elsewhere in the docs.
* Use `std::move` when adding spike callbacks to `arb::simulation` (useful if callbacks are stateful).

Ben Cumming authored 6 years ago and Sam Yates committed 6 years ago

Fixes issue #467 

* Add GPU synchronization points where required for Kepler to coordinate CPU access of managed memory.
* Use hand-rolled double precision atomic addition for Kelper targets.
* Replace `ARB_WITH_CUDA` build option with `ARB_GPU_MODEL` option that takes one of 'none', 'K20', 'K80' or 'P100', and set up source-code defines accoringly.
* Clean up of redundant compiler flags and defines no longer required now that the project uses separate compilation for CUDA sources.

Fix some typos in the SIMD documentation.

Ben Cumming authored 6 years ago and Sam Yates committed 6 years ago

Add some C++ API documentation.

* Create C++ API section in docs.
* Document `arb::recipe`: both a class reference along with more explanatory text and best practices guide.
* Add some class documentation of basic types required to understand recipe definition.
* Some in-code comment clean up.
* Change `arb::cell_kind` from a vanilla enum to a scoped enum.

Ben Cumming authored 6 years ago and Sam Yates committed 6 years ago

The name `arb::model` did not clearly describe the role of the class, while `arb::simulation` better captures that this is an instantiation of a model for the purpose of running a simulation, as distinct from the description of a model represented by an `arb::recipe` instance.

* Rename sources `model.{hpp,cpp}` to `simulation.{hpp,cpp}`.
* Rename class `arb::model` to `arb::simulation`.
* Update docs and tests to suit.

Ben Cumming authored 6 years ago and Sam Yates committed 6 years ago

Put all the SIMD docs in a single topic, to simplify the documentation tree.

Added an installation guide to the Read The Docs
Removed the outdated build/install information from README.md
Link from README to Read The Docs
Updated the splash page for Read The Docs

CUDA 9 introduced new, fine-grained, thread synchronization primitives.
In doing so, it introduced new forms of the warp intrinsics like __shfl_up, depricating the old symbols in the process.

It will be a while before we can use 9 as the default minimum, so we have to support compilers that expect the new and old behavior.

There are two options: wrap the intrinsics in question, or pass nvcc a flag to not issue warnings about depricated symbols. I go for the approach of wrapping, because I would rather keep the compiler warning turned on.

Fixes #379.

Padded vectors with run-time padding/alignment guarantees will form the basis of the storage class for the new CPU and SIMD generated mechanisms.

* Add `padded_allocator` that aligns and pads allocations.
* Make microbenchmark for `default_construct_adaptor` that overrides the allocator construct() to default- instead of value-initialization on values.
* Add `with_instrumented_malloc` class for tracking malloc, realloc, etc. calls.
* Add unit tests for `padded_allocator`.

fixes #446.

Intel compiler with default options does not guarantee correct fp behaviour with subnormals; it presumably sets the fp state to flush to zero.

Reordering a multiply and divide in the expm1 calculation avoids a transient subnormal value that was causing the routine to incorrectly return zero for very small, but normal, arguments.

This provides a bunch of SIMD intrinsic wrappers as a precursor to the SIMD printers.

The aim is that the SIMD printer can be agnostic regarding the particular vector architecture.

The design is based rather loosely on the proposal P0214R6 for C++ Parallelism TS 2. The transcendental function implementations are adapted from the existing SIMD architecture-specific code, which in turn are based on the Cephes library algorithms.

The custom CSS for the html documentation have been tweaked.

In some places our CMake scripts were attempting to check out git submodules when required, if they have not already been checked out. The code that does this was cut and pasted, and was getting unwieldy.

To minimise the responsibilities of CMake, this PR

removes calls to git
introduces a function check_git_submodule that can be used to test if a git submodule is installed, and print a helpful message that informs the user how to check it out if needed.
introduces a function add_error_target that makes a target that prints a message then quits with an error. This can be used to generate a proxy target when a problem is detected during CMake setup. This means that an error is only generated when building a target with a missing dependency, instead of an error during CMake setup.
refactors the CMake setup for the docs and ubenches targets to use these new features.

This replaces the CMake templates provided by TBB with a much more sane alternative!

The TBB CMake templates had a very strange workflow, that involved downloading the TBB source and compiling it, which made it impossible to configure the TBB build, and caused problems on systems without connection to the internet.

We replace this with a fork of the TBB repository maintained by Github user @wjakob:
https://github.com/wjakob/tbb
This fork provides a sane CMakeLists.txt that can be configured from our CMake setup.
It is added as a git submodule, so it can be downloaded with the rest of the repository, hence not requiring connection to the internet during CMake configuration.

It could be extended to use a user-provided build of TBB to use instead of building it.

fixes #332.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

Replace the profiler with a simpler design that works for nested multi-threaded regions.

* Replace hierarchical profiler accounting with strictly exclusive regions.
* Make tree grouping of profile data a presentation concern.
* Uncouple profiler semantics from Arbor classes such as `model`.
* Add thorough documentation for the new profiler to the library documentation.

Use type erasure tricks to remove abstract base class for `event_generator`.

This simplifies all the code that uses `event_generator`s (not radically, but it is simpler).

Added cell description to the Arbor namespace.

Two main contributions:

1) Implementation of LIF neuron model with no kernel and no external input (I_e=0)

The input current to each neuron is therefore just the sum of all the weights of incoming spikes. We integrate in jumps dt = min(t_final - t, t_event - t), since we know the exact solution of the differential equal describing the membrane potential.

2) Miniapp for simulating the Brunel network of LIF neurons. 

The network consists of 2 main populations: excitatory and inhibitory populations. Each neuron from the network receives a fixed number (proportional to the size of the population) of incoming connections from both of these groups. In addition to the input from excitatory and inhibitory populations, each neuron receives a fixed number of connections from the Poisson neurons producing the Poisson-like input that is integrated into the LIF cell group so that the communication of Poisson events is bypassed. 

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

* Parallelize calls to `recipe::connections_on` in communicator constructor.
Fixes #442.

Add an example miniapp that demonstrates how to use event generators in a C++ recipe.

* Replace improper use of brace initialization of vectors of classes with a deleted move constructed with a sequence of `emplace_back` invocations.

There were two latent bugs in the event generation part of `model`.

1. A segmentation fault when initializing the `event_generators` in the `model` constructor caused by using an index variable after it had been incremented.
2. Events generated during the first epoch were not delivered on time.

The first issue was simple to fix, by ensuring that the coutning variable is incremented at the end of the loop.

The second issue required refactoring the event wrangling inside `model`. Events can be introduced into a model via three sources:

1. Generated by spike exchange
2. By calling the `model::inject_events()` interface
3. `event_generator`s attached to cells.

The refactoring was required to ensure that all three sources are handled correctly. There is further opportunities for refactoring the code to make it a bit cleaner, specifically putting the wrangling code in its own type that could be tested seperately, outside `model`, but that is beyond the scope of this fix. 

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

* Enable `cell` copy constructor, remove special tag type used to guard `cell` cloning.
* Provide sane defaults for `recipe` methods.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

* Remove `cell_tree` wrapper around `tree`.
* Separate `tree` construction from tree reduction operation.
* Rename `make_parent_index` to `tree_reduce`.

Fixes #426.

A small script to compare files containing (gid, time) pairs. The list do not have to be sorted in time or gid order.
A details error is displayed on differences.

* Fix indirection in ion concentration write.
* Remove second indirection in ion write assignment.
* Extend ion write unit test to cover non-contiguous ion CV cases and verify correct ion concentration averaging.

Fixes #424.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

* Rename `miniapps` subdirectory to `example`.
* Have all example executables be built under `example` in the build directory.
* Update Travis CI to run miniapp from new path.