Sam Yates authored 6 years ago and Benjamin Cumming committed 6 years ago

CMake and build refactoring

*   Use CUDA as first-class language (leading to CMake 3.9 minimum version requirement).

*   Use 'modern CMake' interface libraries for compiler options, include file and library dependency tracking. Interface library targets:
    * `arbor-deps`: compiler options and library requirements for the `libarbor.a` static library, as governed by configure-time options and environment.
    * `arbor-private-headers`: include path for non-installed headers, as required by unit tests and arbor itself.
    * `arbor-aux`: helper classes and utilities used across tests and examples.
    * `ext-json`, `ext-tclap`, `ext-tbb`, `ext-benchmark`, `ext-sphinx_rtd_theme`: externally maintained software that we include (directly or via submodule) in the `ext/` subdirectory.
 
*   Single static library `libarbor.a` includes all built-in modules and CUDA objects.

*   Simply configuration options:
    *  `ARB_WITH_TRACE`, `ARB_AUTORUN_MODCC_ON_CHANGES` `ARB_SYSTEM_TYPE` removed.
    * External `modcc` is provided by `ARB_MODCC` configuration option; if provided `modcc` is still buildable, but is not included in the default target.
    * `ARB_PRIVATE_TBBLIB`, defaulting to `OFF`, instructs the build to make TBB from the included submodule.

*   Extend `ErrorTarget` functionality to provide a dummy target or an error target based on a condition.
*   Generate header version defines and library version variables based on git status and project version, via new script `include/git-source-id`.
*   All generated binaries now placed in `bin/` subdirectory at build.
*   Install targets installs: public headers (incomplete); static library; `modcc` tool; `lmorpho` executable; `html` documentation (examples, tests and validation data are currently not installed).
*   Executable targets have had the `.exe` suffix removed; unit tests are labelled `unit` (arbor unit tests), `unit-modcc` (modcc unit tests), `unit-local` (distributed tests with local context), `unit-mpi` (distributed tests with MPI context).
*   More graceful handling of configure-time detection of `nrniv`, Julia and required Julia modules for validation data generation.
*   Add `cmake/FindJulia.cmake`, `cmake/FindTBB.cmake`  package finders, and adjust `cmake/FindUnwind.cmake` to use link library-style properties.
*  Adjust travis script to test `unit-local` and `unit-mpi` if appropriate.
*  Simply documentation `conf.py`.

Source relocation and reorganization

* All external project sources and files moved to `ext/`.
* Source code refactoring to decouple library-using code from the configure-time definitions that govern arbor behaviour: removes conditional code in public headers that depends upon `ARB_WITH_X`-type definitions at compile time. Affected code is is in the public interfaces for MPI, the threading implementation, and the profiler.
* Remove `util/debug.hpp`; split out functionality for pretty-printing from assertion handling.
* Make FVM cell non-physical voltage check a run-time cell-group parameter.
* Move spike double buffer implementation to `simulation.cpp`.
* Make timer utility wrap POSIX `clock_gettime` independent of threading configuration.
* Make `mpi_error` derive from `system_error` and follow C++11 `system_error` semantics.
* `EXPECTS` macro replaced by `arb_assert` macro.
* JSON dependency removed from `libarbor.a` and header files: moved to auxiliary library.
* Publicly visible macros garner an `ARB_` prefix as required.
* Move SWC test file to `test/unit` directory.
* Work-in-progress splitting of public from private includes: as a convention not entirely adhered to as yet, private headers within arbor source are included with `""`, public headers with `<>`.

Modcc interface changes

* Expose via `--namespace` option the functionality that sets the namespace in generated code.
* Use `--profile` option to add profiler hooks to generated code; uses public function interface directly rather than `PE/PL` macros in order to avoid public `PE` and `PL` defines.

First commit of two for mechanism refactor work (refer to PR #484 and PR #483).

FVM/mechanism code:
* Refactor mechanism data structures to decouple backend-specific implementations and mechanism metadata.
* Add mechanism catalogue for managing mechanism metadata and concrete implementation prototypes.
* Add fingerprint-checking to mechanism metadata and implementations to confirm they come from the same NMODL source (fingerprint is not yet computed, but tests are in place).
* Split FVM discretization work out from FVM integrator code.
* Use abstract base class over backend-templated FVM integrator class `fvm_lowered_cell_impl` to allow separate compilation of `mc_cell_group` and to remove the dummy backend code.
* Add a new FVM-specific scalar type `fvm_index_type` that is an alias for `int` to replace
`fvm_size_type` in fvm layouts and mechanisms. This was chosen as an alternative
to making `unsigned` versions of all our SIMD implementation classes.
* Extend `cable1d_neuron` global data to encompass: mechanism catalogue; default ion concentrations and charges; global temperature (only for Nernst); initial membrane potential.

Modcc:
* Collect printer sources in modcc under `printer/`.
* Move common functionality across printers into `printer/printerutil.{hpp,cpp}`.
* Add string to file I/O implemented in routines read_all and write_all in `io/bulkio.hpp`.
* Implement indent-friendly source code generation via a `std::streambuf` filter `io::prefixbuf` defined in `io/prefixbuf.hpp`, together with manipulators and a corresponding std::ostream-derived wrapper.
* Rewrite printers to use new infrastructure: cpu target incorporates SIMD printing options; CUDA printer at this point produces only stubs for CUDA kernel wrappers.
* Modify SIMD printing command line options for modcc: `-s` enables explicit vectorization using the SIMD classes;  `-S <N>` allows a specific data width to be prescribed.
* Fix problem in `test_ca.mod` with uninitialized ion current.
* Add infrastructure support to allow future pre-computation of SIMD index conflict cases for (hopefully) faster scatters and updates.
* Simplify `IndexedVariable` expressions in the AST, making data source explicit via a `sourceKind` enum, and leaving the indexing method and index names up to the printers.
* Allow state variables in the AST to 'shadow' an ion concentration — these are assigned in the
generated `write_ions` method.

SIMD classes:
* Add `simd_cast` operation between SIMD value types of the same width, and with `std::array`. (Note: this was tested and used in an early development version of the code, but not in this version. It was still a lacuna in the original SIMD wrappers, so it has been left in.)
* Restructure SIMD gather/scatter API to use a `simd::indirect` expression,  which encapsulates a pointer and SIMD offset.
* Add `simd::index_constraint` scoped enum to describe knowledge of contention in indirect indices, so that we can branch on this to the appropriate implementation.
* Add SIMD concrete implementation routines `reduce_add` for horizontal reduction and `element0` for access to first lane scalar value.
* Add SIMD value method `sum()` that exposes implementation `reduce_add`.
* Add SIMD concrete implementation routine `compound_indexed_add` that provides the implementation for `indirect(p, simd_indices) += simd_value` construction.
* Fix SIMD `implbase` bug where some static methods were using the `implbase` fall-back functions instead of the derived class specialized implementations.
* Move SIMD mathematical functions into friend routines of `simd_impl` in order to resolve implicit conversions from scalars in mixed SIMD-scalar operations.
* Use a templated `tag` class to dispatch on SIMD concrete implementation types, to avoid problems with incomplete types in method signatures.
* Remove old SIMD intrinsics.

CMake infrastructure:
* Downcase some variables in `CMakeLists.txt` files to  distinguish them visually from CMake keywords and variables.
* Split arbor modcc vectorization option (now `ARB_VECTORIZE`) and target-architecture optimization (now `ARB_ARCH`).
* For `arbor` and `arbormech` targets, and in particular not the `modcc` target, use `ARB_ARCH` to generate corresponding target-appropriate binaries, including, for example, appropriate SIMD support.
* Extend `CompilerOptions.cmake` to map as best as able between the various target architecture names (we use the gcc names) and the correct option to pass to the compiler based on the compiler and platform.
* Add work-around for misidentification by CMake of XL C as Clang.
* As a temporary work-around, include `arbormech` library twice on link line to resolve circular arbor–arbormech dependencies.

Unit tests:
* Extend repertoire of generic sequence equality/near equality testing support  in `common.hpp`.
* Add warning suppression for icc for the malloc instrumentation code.
* SIMD unit tests for indirect expressions, compound indirect add, reduction.
* Make some exact tests into floating point 'near' tests when comparing computed areas and lengths in swc and fvm layout tests, to account for compiler (e.g. icc) performing semantically inequivalent floating point operation reordering or fusion at `-O3`.
* Split out some of the CUDA tests into separate .cpp/.cu files for  separate-compilation purposes.

Other:
* The `padded_allocator` has been modified to propagate alignment/padding on move and copy (these semantics make their use much easier and safer in the multicore mechanism instantiation code).
* Map/table searching utilities in `util/maputil.hpp`.
* Fixes for correct sequence type categorization and `begin/end` ADL.
* Fixes for type guards for range methods that take universal references.
* Removal of some redundant code in range utilities through the use of universal references.
* Add new range view `reverse_view` for ranges delineated by bidirectional iterators.
* Add single argument form of `make_span` to count up from zero, and associated helper `count_along` that gives a span that indexes a supplied container.
* Moved `prefixbuf` to `modcc` source.
* Make sequence positive and negative tests in algorithms generic.
* Add `private`-subverting helper code/macro to `tests/unit/common.hpp` to reduce the number of public testing-only interfaces in the library code.
* Add virtual destructors for virtual base classes.
* Add new arb::math:: functions: `next_pow2` for unsigned integral types, `round_up` to round a number away from zero to next largest magnitude multiple.
* New `index_into` implementation that supports bidirectional access (moved to `util::` namespace).
* Fix problem in `test_ca.mod` with uninitialized ion current.
* Rework dangerous `memory::array(Iter, Iter)` constructor to be less dangerous (and do the expected thing).
* Allow ranges to be constructed from other ranges if the iterators are compatible.

Change nest::mc namespaces to arb

Fixes #232.

* Try `NMC_DATADIR` environment variable for validation data path, or else if the #defined `NMC_DATADIR` does not point to a directory, try `./validation/data` and `../validation/data`.
* Don't define `NMC_DATADIR` if CMake version 3.7 or 3.8.
* Extend C++17 filesystem emulation with (POSIX) implementation of `is_directory` and supporting classes, functions and enums `filesystem_error`, `file_status`, `status(..)`, `file_type` and `perms`.

This PR is part of the gpu feature merge. The GPU implementation is not implemented here. Instead, we focus on refactoring of the original "multicore" back end so that it is ready for adding the GPU back end.

This is a big and messy change, for which I am sorry.

## build System

- A `WITH_CUDA` option has been added to the main CMakeLists. This finds the CUDA toolkit, and sets CUDA compiler flags, and will build unit tests for the gpu back end.
- The CMakeLists that generates mechanisms with modcc has been updated to generate CUDA mechanisms.
- the library is now named `libnestmc` instead of `libcellalgo`
- merge the external libraries that are optionally linked againts (tbb, libunwind, etc) into a single
  `EXTERNAL_LIBRARIES` list for ease of linking

## modcc

- the cprinter and cudaprinter have had small changes to generate mechanism files that are compatible with the refactored library.

## algorithms

- the indexes into algorithm was "rangified". An algorithm `index_into_iterator` takes two ranges as inputs to make a range that lazily generates the index of sub into super set.

## backends

- made a new path `src/backends/` for backend specific type and implementation code.
- currently:
  - complete support for the `multicore` and `gpu` backends
  - `gpu` back end is not optimized or validated
- the back end implementations are in `src/backends`
- a single `backend` class, `nest::mc::{multicore,gpu}::backend`, is provides all backend specific type and implementation details fro each backend
  - storage containers
  - Hines matrix assembly for FVM method
  - Hines matrix solver
  - mechanism "factory"

## lowered fvm cells

- removed `fvm_cell` because this can be modelled with an `fvm_multicell` with one cell.
- refactored to use backend type and implementation from `fvm_policy`
- use `std::vector` instead of containers in `nest::mc::memory::` where possible when building cells.

## memory library

Refactor the "memory" library, making it much simpler and better integrated into the rest of the application. However, it is still far from perfect. The `Coordinator` approach needs to be improved, most likely by putting target-specific wisdom into pointers (which could obviate the need for a `const_array_view` type.
1. renaming and moving
   - move from `vector/` to `src/memory`
   - move into the `nest::mc` namespace, i.e. all types and functions are now in `nest::mc::memory`
   - change from camel case nameing scheme to NestMC style naming.
2. simplification
   - remove the CRTP cruft that was used to make operator overloading work for operations like copying from one range into another, and filling a range with a constant value. These have been replaced with `memory::fill()` and `memory::copy()` helper functions. This simplified the code _a lot_, and makes code clearer in user land.
   
   ```
   // before
   vec(0, 5) = other;
   // now
   memory::copy(other, vec(0, 5));
   ```
   - add some wrappers in `src/memory/wrappers.hpp` that help with making views. These are particularly useful for passing `std::vector` through interfaces that expect a view.

## debug backtraces

Added stack traces for debugging.
- support for OSX and Linux via libunwind
- backtraces can be generated manually `nest::mc::util::backtrace().print()`
  - creates a new file and dumps trace into file
  - prints message to `stderr` with file name and instructions on how to analyse
- backtraces are also automatically generated when an assertion `EXPECTS` statement fails
- a python script in `scripts/print_backtrace` pretty prints the output with file name, line number and demangled symbols

## util simplification and consolidation

The `src/util.hpp` file was removed
- much of its contents were dead code and just removed
- useful components like `pprintf` and `make_unique` were moved into the `src/utils` path in standalone files
There was a lot of overlap between functionality provided in `src/memory/util.hpp` and existing functions/types in the `nest::mc::util` namespace. The `memory` implementations were removed, and their `nest::mc::util` counterparts used. There is still some work remaining, namely moving the rest of the `src/memory/util.hpp` into `src/util/...`

Validation of simulation results by comparing L-infinity distance and maximum peak displacement in voltage traces taken at mid-points and end-points of sections.

Note that these tests will broadly _fail_ at this point.

Reference data generation and visualization:
* Use voltage traces as validation data from NEURON; refactor generation scripts.
* Generate validation data as a dependency of `validation.exe`.
* Store validation data in a `tsplot`-compatible format.
* Extend `tsplot` to handle multiple time series per file, a non-time 'time' axis (see `-A` option), filtering of time series by meta data queries (see `-s` option).
* Remove pre-generated spike-time validation data files.

Core extensions:
* Add a single-cell `singleton_recipe` recipe wrapper.
* Allow cell descriptions to be cloned (explicitly) for use with this wrapper.
* Add `simple_sampler` which maintains a trace of scalar observables as a vector of (time, value) pairs.
* Allow cell groups to be exposed from `model` (used for post-synaptic event injection in synapse validation test.)

Utility extensions:
* Add `math::signum` function.
* Fix `value_type` determination in `util::transform_iterator` where the underlying iterator dereferences to a reference type.
* Add `util::max_element_by` and `util::max_value` sequence utilities to `rangeutil.hpp`.
* Small bug fixes for `util::optional` (one const correctness case, and proper behaviour for implicit conversion of `nothing_t` to `optional<void>`.)
* Generic `util::first` and `util::second` accessor functional objects for reference access to `std::pair` fields in the absence of generic lambda functions.
* Add `util::path` for path manipulation, corresponding to a subset of C++17 `std::filesystem::path` functionality. Only POSIX path semantics supported at this point.

Validation tests:
* Rework all validation tests as comparisons of voltage trace data against a (presumably converged) reference trace generated by NEURON.
* Add options to `validation.exe` driver for verbose output of test results (`-v`), limiting the compartment number in convergence tests (`-m`) and saving the generated traces to a `tsplot`-compatilble JSONfile (`-o`).
* Gather trace comparison metrics and analysis into `trace_analysis.hpp` and `trace_analysis.cpp`.
* Add super simple command line argument parsing utility `tinyopt.hpp`.
* Make a consistent naming of `ball_and_3stick` model across sources.
* New `ball_and_taper` model.