Sam Yates authored 6 years ago and Benjamin Cumming committed 6 years ago

CMake and build refactoring

*   Use CUDA as first-class language (leading to CMake 3.9 minimum version requirement).

*   Use 'modern CMake' interface libraries for compiler options, include file and library dependency tracking. Interface library targets:
    * `arbor-deps`: compiler options and library requirements for the `libarbor.a` static library, as governed by configure-time options and environment.
    * `arbor-private-headers`: include path for non-installed headers, as required by unit tests and arbor itself.
    * `arbor-aux`: helper classes and utilities used across tests and examples.
    * `ext-json`, `ext-tclap`, `ext-tbb`, `ext-benchmark`, `ext-sphinx_rtd_theme`: externally maintained software that we include (directly or via submodule) in the `ext/` subdirectory.
 
*   Single static library `libarbor.a` includes all built-in modules and CUDA objects.

*   Simply configuration options:
    *  `ARB_WITH_TRACE`, `ARB_AUTORUN_MODCC_ON_CHANGES` `ARB_SYSTEM_TYPE` removed.
    * External `modcc` is provided by `ARB_MODCC` configuration option; if provided `modcc` is still buildable, but is not included in the default target.
    * `ARB_PRIVATE_TBBLIB`, defaulting to `OFF`, instructs the build to make TBB from the included submodule.

*   Extend `ErrorTarget` functionality to provide a dummy target or an error target based on a condition.
*   Generate header version defines and library version variables based on git status and project version, via new script `include/git-source-id`.
*   All generated binaries now placed in `bin/` subdirectory at build.
*   Install targets installs: public headers (incomplete); static library; `modcc` tool; `lmorpho` executable; `html` documentation (examples, tests and validation data are currently not installed).
*   Executable targets have had the `.exe` suffix removed; unit tests are labelled `unit` (arbor unit tests), `unit-modcc` (modcc unit tests), `unit-local` (distributed tests with local context), `unit-mpi` (distributed tests with MPI context).
*   More graceful handling of configure-time detection of `nrniv`, Julia and required Julia modules for validation data generation.
*   Add `cmake/FindJulia.cmake`, `cmake/FindTBB.cmake`  package finders, and adjust `cmake/FindUnwind.cmake` to use link library-style properties.
*  Adjust travis script to test `unit-local` and `unit-mpi` if appropriate.
*  Simply documentation `conf.py`.

Source relocation and reorganization

* All external project sources and files moved to `ext/`.
* Source code refactoring to decouple library-using code from the configure-time definitions that govern arbor behaviour: removes conditional code in public headers that depends upon `ARB_WITH_X`-type definitions at compile time. Affected code is is in the public interfaces for MPI, the threading implementation, and the profiler.
* Remove `util/debug.hpp`; split out functionality for pretty-printing from assertion handling.
* Make FVM cell non-physical voltage check a run-time cell-group parameter.
* Move spike double buffer implementation to `simulation.cpp`.
* Make timer utility wrap POSIX `clock_gettime` independent of threading configuration.
* Make `mpi_error` derive from `system_error` and follow C++11 `system_error` semantics.
* `EXPECTS` macro replaced by `arb_assert` macro.
* JSON dependency removed from `libarbor.a` and header files: moved to auxiliary library.
* Publicly visible macros garner an `ARB_` prefix as required.
* Move SWC test file to `test/unit` directory.
* Work-in-progress splitting of public from private includes: as a convention not entirely adhered to as yet, private headers within arbor source are included with `""`, public headers with `<>`.

Modcc interface changes

* Expose via `--namespace` option the functionality that sets the namespace in generated code.
* Use `--profile` option to add profiler hooks to generated code; uses public function interface directly rather than `PE/PL` macros in order to avoid public `PE` and `PL` defines.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

* Remove `cell_tree` wrapper around `tree`.
* Separate `tree` construction from tree reduction operation.
* Rename `make_parent_index` to `tree_reduce`.

Fixes #426.

Change nest::mc namespaces to arb

`cell_tree::depth_from_root()` was incorrectly traversing only
the first branch from the tree root, leading to uninitialized
values in the returned depth array.

This leads to an incorrect maximum leaf node in `find_minimum_root()`,
which then returns `no_parent`. This gets passed to
`tree::change_root(size_t)`.

* Correct `cell_tree::depth_from_root()` implementation.
* Re-enable curiously disabled test case in `cell_tree.from_parent_index`
* Add unit tests for `cell_tree::depth_from_root`.

This PR is part of the gpu feature merge. The GPU implementation is not implemented here. Instead, we focus on refactoring of the original "multicore" back end so that it is ready for adding the GPU back end.

This is a big and messy change, for which I am sorry.

## build System

- A `WITH_CUDA` option has been added to the main CMakeLists. This finds the CUDA toolkit, and sets CUDA compiler flags, and will build unit tests for the gpu back end.
- The CMakeLists that generates mechanisms with modcc has been updated to generate CUDA mechanisms.
- the library is now named `libnestmc` instead of `libcellalgo`
- merge the external libraries that are optionally linked againts (tbb, libunwind, etc) into a single
  `EXTERNAL_LIBRARIES` list for ease of linking

## modcc

- the cprinter and cudaprinter have had small changes to generate mechanism files that are compatible with the refactored library.

## algorithms

- the indexes into algorithm was "rangified". An algorithm `index_into_iterator` takes two ranges as inputs to make a range that lazily generates the index of sub into super set.

## backends

- made a new path `src/backends/` for backend specific type and implementation code.
- currently:
  - complete support for the `multicore` and `gpu` backends
  - `gpu` back end is not optimized or validated
- the back end implementations are in `src/backends`
- a single `backend` class, `nest::mc::{multicore,gpu}::backend`, is provides all backend specific type and implementation details fro each backend
  - storage containers
  - Hines matrix assembly for FVM method
  - Hines matrix solver
  - mechanism "factory"

## lowered fvm cells

- removed `fvm_cell` because this can be modelled with an `fvm_multicell` with one cell.
- refactored to use backend type and implementation from `fvm_policy`
- use `std::vector` instead of containers in `nest::mc::memory::` where possible when building cells.

## memory library

Refactor the "memory" library, making it much simpler and better integrated into the rest of the application. However, it is still far from perfect. The `Coordinator` approach needs to be improved, most likely by putting target-specific wisdom into pointers (which could obviate the need for a `const_array_view` type.
1. renaming and moving
   - move from `vector/` to `src/memory`
   - move into the `nest::mc` namespace, i.e. all types and functions are now in `nest::mc::memory`
   - change from camel case nameing scheme to NestMC style naming.
2. simplification
   - remove the CRTP cruft that was used to make operator overloading work for operations like copying from one range into another, and filling a range with a constant value. These have been replaced with `memory::fill()` and `memory::copy()` helper functions. This simplified the code _a lot_, and makes code clearer in user land.
   
   ```
   // before
   vec(0, 5) = other;
   // now
   memory::copy(other, vec(0, 5));
   ```
   - add some wrappers in `src/memory/wrappers.hpp` that help with making views. These are particularly useful for passing `std::vector` through interfaces that expect a view.

## debug backtraces

Added stack traces for debugging.
- support for OSX and Linux via libunwind
- backtraces can be generated manually `nest::mc::util::backtrace().print()`
  - creates a new file and dumps trace into file
  - prints message to `stderr` with file name and instructions on how to analyse
- backtraces are also automatically generated when an assertion `EXPECTS` statement fails
- a python script in `scripts/print_backtrace` pretty prints the output with file name, line number and demangled symbols

## util simplification and consolidation

The `src/util.hpp` file was removed
- much of its contents were dead code and just removed
- useful components like `pprintf` and `make_unique` were moved into the `src/utils` path in standalone files
There was a lot of overlap between functionality provided in `src/memory/util.hpp` and existing functions/types in the `nest::mc::util` namespace. The `memory` implementations were removed, and their `nest::mc::util` counterparts used. There is still some work remaining, namely moving the rest of the `src/memory/util.hpp` into `src/util/...`

* Consistent use of globally defined integer types for
  indices etc. (catypes.hpp) or template type parameters
  across simulator.
* Support unsigned types in tree, cell_tree and friends.
* Use cell_member_type for probe ids (WIP)
* Rejig mechanism lookup structure (simplifies, plays nicely
  with templated index types.)
* Move spike, connection, events out of communcation namespace
  and directory.

Gtest is currently copy twice!

- improve robustness of time step and time integration
  interval selection
- changes of the type `if (!id)` to `if (id==0)`.
- fix bug selecting source cell gid when generating
  network connections
- make all-to-all network a user-specified option
    - updated README
- fix bug where an initial spike was not generated for
  the first cell on a communicator when the cell gid
  was a multiple of 20

Add path to data directory by preprocessor definition so that
unit tests can be run from any directory.

(yes, I'm one of 'those' programmers)

Use "" for local includes
Use <> for includes from include directories

+ some other minor improvements.

The line buffer is no more a pure character array but a standard string.
For reading lines from the input stream we use the higher level
std::getline() function which takes care of overflow issues.

Side improvements: String comment prefixes are now supported.

Other changes: added `-pedantic' to the compilation flags, to force
strict C++11 compliance.

The parser's semantics are now cleaner and the implementation more
robust. Cell parsing now follows the semantics of the standard input
stream (same as of standard UNIX's read), but instead of reading in
bytes or builtin types, we are reading full cell records. This has the
following implications:

1. Empty and comment lines are consumed silently.
2. End-of-file will be set on the stream only if a read operation
   returns zero. Practically, this means, eof bit will be set in the
   stream at the first attempt to read a cell record after the final
   cell record in the stream was read.
3. Exceptional conditions in the stream (e.g., eof or bad bits) stop the
   parsing process without raising an exception and return control to the
   user which is responsible to deal with them. The only exceptional
   condition that we handle internally is the presence of very long lines,
   in which case we raise an std::runtime_error().

Thanks to these cleaner semantics, the parser is now able to deal with
files not ending with a new line. The previous version would have missed
the last record.

Due to (2) the following code for reading from an input stream will
yield an additional default constructed cell record at the end:

while (!swc_input.eof()) {
    cell_record cell;
    swc_input >> cell;
    // do sth with the cell
}

According to (2), after the final record is read, eof is not yet set at
the stream, which means the while loop will be entered one more time.
The attempt to read a record from the stream then will raise the eof bit
and no cell will be read in, leaving `cell' at its default constructed
state. To avoid such a side-effect, the best way to read records from an
input stream is the following:

cell_record cell;
while( !(swc_input >> cell).eof()) {
    // do sth with the cell
}

In this case, after the final record is read, the next attempt to read
in a record will set the eof bit and won't enter the loop.

A very basic parser of SWC files. The role of the parser is to extract
cell records from an SWC file line-by-line, which will then be used to
construct the neuron cell models.

The parser currently does only a simple linewise processing of the cell
records. This practically means that the validity of the input is only
checked at the cell record level.

More advanced input processing, such as detection of duplicate cell ids,
holes in cell id numbering or multiple cell trees in a file are not
currently treated.