Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

This addresses all of the compiler warnings and errors for xlc when compiled at -O0. There are still compiler bugs when compiling with higher optimization levels, however they are more challenging.

* Ignore the incorrect `-Wno-missing-braces` warnings (similarly to Clang).
* Remove `-qhalt=e flag` inserted by CMake.
* Remove redundant comparison of char to EOF in lexer.
* The XLC compiler was crashing inexplicably on one call of the following method:
   ```void run(..., const std::vector<float>& excl={}) {...}```
  This was fixed by not having a default value for the last argument.
* Add some curly braces to silence warning for dangling else.

fixes #97

Correct pointer casting operations in `uninitialized`: issue #100

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

fixes #90

The c and cuda printer were not printing else branches in if else statements.
* added logic for printing else branches
* added if-else expressions to data/test.mod

No unit tests were added because we don't currently have a unit testing framework for the output mechanisms.

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

Fixes #68
Corresponding feature: #67

* Reproduce the hh-soma validation test on GPU.
* Reproduce the ball and stick model on GPU.
* Reproduce miniapp spike chains.
* Add `cell_group` unit test to the cuda unit tests: builds simple ball and stick model and integrates for 50ms and records how many spikes occur; it is a simple early warning that something is broken, but is no substitute for the validation tests.
* Update the `validate_soma`, `validate_ball_and_stick` and `validate_synapses` validation tests for the GPU backend:
    * refactor individual tests into test runner functions that are templated on lowered cell type;
    * for each of the original validation tests add a cuda (.cu) implementation, and write an additional "backend" field to the validation trace metadata.
* Use a `CPrinter` to generate the same `net_receive` block that is used for the multicore backend.
  Note: this is not efficient, because each read/write requires a cuda memcpy between host and device memory, however it allows us to pass all unit and validation tests. A more efficient GPU-specific implementation is left for later optimization work.
* Make paths to `gtest.h`, `test_common_cells.hpp` etc. in test sources consistent relative paths, and remove the `tests/` directory from the include path.

Fixes #80
  * Avoid ctor ambiguity by using parentheses for copy ctor invocation with `nlohmann::json` class.

Fixes #82

Bugfix: issue #80 

* Avoid ctor ambiguity by using parentheses for copy ctor invocation with `nlohmann::json` class.

bugfix for missing mechanism paths

The mechansims/gpu and mechanisms/multicore paths were not being generated
by CMake during configuration, and modcc was silently ignoring that it was
not able to write to the nonexistant target paths.
I have updated CMake to generate the missing gpu and multicore target paths.

This PR is part of the gpu feature merge. The GPU implementation is not implemented here. Instead, we focus on refactoring of the original "multicore" back end so that it is ready for adding the GPU back end.

This is a big and messy change, for which I am sorry.

## build System

- A `WITH_CUDA` option has been added to the main CMakeLists. This finds the CUDA toolkit, and sets CUDA compiler flags, and will build unit tests for the gpu back end.
- The CMakeLists that generates mechanisms with modcc has been updated to generate CUDA mechanisms.
- the library is now named `libnestmc` instead of `libcellalgo`
- merge the external libraries that are optionally linked againts (tbb, libunwind, etc) into a single
  `EXTERNAL_LIBRARIES` list for ease of linking

## modcc

- the cprinter and cudaprinter have had small changes to generate mechanism files that are compatible with the refactored library.

## algorithms

- the indexes into algorithm was "rangified". An algorithm `index_into_iterator` takes two ranges as inputs to make a range that lazily generates the index of sub into super set.

## backends

- made a new path `src/backends/` for backend specific type and implementation code.
- currently:
  - complete support for the `multicore` and `gpu` backends
  - `gpu` back end is not optimized or validated
- the back end implementations are in `src/backends`
- a single `backend` class, `nest::mc::{multicore,gpu}::backend`, is provides all backend specific type and implementation details fro each backend
  - storage containers
  - Hines matrix assembly for FVM method
  - Hines matrix solver
  - mechanism "factory"

## lowered fvm cells

- removed `fvm_cell` because this can be modelled with an `fvm_multicell` with one cell.
- refactored to use backend type and implementation from `fvm_policy`
- use `std::vector` instead of containers in `nest::mc::memory::` where possible when building cells.

## memory library

Refactor the "memory" library, making it much simpler and better integrated into the rest of the application. However, it is still far from perfect. The `Coordinator` approach needs to be improved, most likely by putting target-specific wisdom into pointers (which could obviate the need for a `const_array_view` type.
1. renaming and moving
   - move from `vector/` to `src/memory`
   - move into the `nest::mc` namespace, i.e. all types and functions are now in `nest::mc::memory`
   - change from camel case nameing scheme to NestMC style naming.
2. simplification
   - remove the CRTP cruft that was used to make operator overloading work for operations like copying from one range into another, and filling a range with a constant value. These have been replaced with `memory::fill()` and `memory::copy()` helper functions. This simplified the code _a lot_, and makes code clearer in user land.
   
   ```
   // before
   vec(0, 5) = other;
   // now
   memory::copy(other, vec(0, 5));
   ```
   - add some wrappers in `src/memory/wrappers.hpp` that help with making views. These are particularly useful for passing `std::vector` through interfaces that expect a view.

## debug backtraces

Added stack traces for debugging.
- support for OSX and Linux via libunwind
- backtraces can be generated manually `nest::mc::util::backtrace().print()`
  - creates a new file and dumps trace into file
  - prints message to `stderr` with file name and instructions on how to analyse
- backtraces are also automatically generated when an assertion `EXPECTS` statement fails
- a python script in `scripts/print_backtrace` pretty prints the output with file name, line number and demangled symbols

## util simplification and consolidation

The `src/util.hpp` file was removed
- much of its contents were dead code and just removed
- useful components like `pprintf` and `make_unique` were moved into the `src/utils` path in standalone files
There was a lot of overlap between functionality provided in `src/memory/util.hpp` and existing functions/types in the `nest::mc::util` namespace. The `memory` implementations were removed, and their `nest::mc::util` counterparts used. There is still some work remaining, namely moving the rest of the `src/memory/util.hpp` into `src/util/...`

Targets issue #62.

New tokens added:

keywords tok::kinetic, tok::conserve;
symbols tok::tilde, tok::arrow;
split tok::number into tok::real and tok::integer (stoichiometric expressions may have only integer coefficients).
New expression sub-types added, together with parsing:

IntegerExpression (derives from NumberExpression);
ReactionExpression — represents a line of the form "~ a + b <-> c (f, r)";
StoichTermExpression — represents a single term in the lhs or rhs of a reaction as an integral coefficient and identifier, for example "3c" or "-2 sp2";
StoichExpression — represents a possibly empty series of StoichTermExpressions, corresponding to the lhs or rhs of a reaction, or the lhs of a CONSERVE declaration;
ConserveExpression (derives from BinaryExpression) — represents a line of the form "CONSERVE a + 2b - c = expr".
Unit tests added for parsing and lexing of the above, with some clean up of the testing code:

verbose testing selectable at run time with -v, instead of being a compile time decision;
code duplication reduced in test_parser.cpp through the use of helper functions check_parse and check_parse_fail;
handle verbose output from lexer tests by subclassing the lexer; additionally, emit tokens from Lexer::parse() in verbose output.

* Tokenize strings of the form "7E+X", "2.3E " etc as a number
  (integer, real) followed by a new token starting with 'E'.
  Note that this makes the string "1.2E3E4" no longer cause a
  lexer error status.
* Add tests for corresponding numeric disambiguation in
  `text_lexer.cpp`.
* Add tests for stoichiometric expressions that ensure that only
  otherwise ambiguous phrases such as "7E+2F" fail to parse
  ("7E+F" should parse correctly as a stoich expression.)
* Add missing comma in list of 'good' expressions in
  the `Parser.parse_line_expressions` test.

* Extend stoichiometric expressions to allow negative coefficients.
* Check that terms in a reaction expression are all non-negative.
* Add CONSERVE keyword.
* Add `ConserveExpression` and corresponding parsing behaviour.
* Add unit tests for above and for recognition of KINETIC block
  as a `procedure`.

* Make verbose printing in test_modcc a runtime flag `-v`.
* Show results of `lexer.parse()` in test_modcc when run with `-v`.
* Split numerical lexing and parsing into real and integer categories.
* Add expression parsing for KINETIC block and enclosed reaction
  expressions.

Note: does not parse CONSERVE yet; no test for KINETIC block
(reactions are covered though.)

* first version of openmp threading back end

* adding openmp parallel sort implementation

* OpenMP sort working

* Support for units syntax within state block.

* Add soma-less cable cell to test cells.

Also:
* Ensure intrinsic and passive properties properly set on test cells.

* Change bulk resistivity default.

* Align defaults with values used in most of the NEURON
  validation scripts.
* Use consistent 100 Ω·m bulk resistivity across both
  NEURON test models and basic validation cells.

* OpenMP back end working

* Add Extrae+paraver support, needs to fix compilation warnings

* Reorganize validation data generation

* Move generation and data to top-level validation directory.
* Make BUILD_VALIDATION_DATA and VALIDATION_DATA_DIR cache vars.
* Add helper CMake functions for data generation.

Note `validation/ref/numeric/foo.sh` is just a placeholder.

* Bugfix: hh_soma.jl

* Use consistent scaling for y[1] scalar voltage in hh_soma.jl
* Also: add more reserved target names to CMakeLists.txt
  helper function.

* Refactor convergence tests; add numeric soma ref.

* Amend data directory path in validation tests.
* Enmodulate `hh_soma.jl`
* Add HH channel reference data generations script.
* Switch `validate_soma.cpp` to numeric reference data.
* Consolidate common code in `validate_ball_and_stick.cpp`
* Add (nearly) Rallpack1 validation test (see below).
* Gentle failure on absence of reference data in
  `validate_ball_and_stick.cpp`

Can't yet override mechanism default parameter values,
so the cable cell model added to `test_common_cells.hpp`
lets the default stand; validation script will have
to use the default membrane conductance rather than that
given by Rallpack1.

* Add Rallpack1 validation, plus bugfix, clean

* Implement Rallpack1 validation test (with a workaround
  for inability to set membrane conductance).
* Fix bug in L≠1 case in PassiveCable.jl (this may still be
  wrong).
* Fix bug in peak delta computation in trace analysis when
  both traces have no local maxima.
* Gentle failure on missing `numeric_soma.json`
* Allow multiple `-s` selection operations for `tsplot`,
  acting disjunctively.

* Remove errant test file.

* file's cleanup

* Remove tabs

* Use correct routine in numeric_rallpack1.jl x0.3

* Configure-time test for julia

* `math::infinity<>()` wrapper for infinity

* Use name `i_e` for Stim current density

* Use `math::infinity<>()` for infinite value

* Adds unit tests for the STATE block.

* Add "lib" to search prefixes for libtbb

* Fix quoting error in library search.
* Add "lib" to prefixes when system is "Linux".

* Address deprecated use of 'symbol' warning.

Julia 0.5 deprecates use of `symbol` instead of
`Symbol`. This patch just substitutes the
correct call.

* Address deprecated use of 'symbol' warning.

Julia 0.5 deprecates use of `symbol` instead of
`Symbol`. This patch just substitutes the
correct call.

* Addresses PR comments.

* Unit tests for math.hpp

* Tests for `math::pi`, `math::lerp`, `math::area_frustrum`
  and `math::volume_frustrum`
* Fix `math:pi<long double>()`.

* Extend range, view functionality.

* New `filter` view: lazily selects based on predicate.
* Generic `front` and `back` for sequences.
* New rangeutil STL wrappers `stable_sort_by`, `all_of`, `any_of`.
* Consolidate common utility unit testing structures into
  `tests/unit/common.hpp`

* Add `ball_and_squiggle` model; fix `ball_and_taper`.

* Make `test_common_cells.hpp` and `ball_and_taper.py` agree.
* Add `ball_and_squiggle` model that has a tapering undulating
  profile.

* Address PR#46 review comments.

* Add documentation of template parameters for `filter_iterator`.
* Document use of `uninitalized<F>` for holding functional objects
  in `filter_iterator` and `transform_iterator`

* Consolidate validation test code (issue #41)

* Simplify trace analysis and reporting code in
  `trace_analysis.hpp`
* Consolidate convergence test run procedures into
  new class `convergence_test_runner`.

* New compartment info structure for FVM.

* Make `algorithm::sum`, `algorithm::mean` more generic,
  allowing use with array types.
* Add `div_compartment` compartment representation, that
  holds geometric information for each half of a compartment
  that will then be used in calculating control volumes.
* Add three compartmentalisation schemes/policies that
  discretize a segment into `div_compartment` objects:
    * `div_compartment_by_ends` divides based only on the
      segment end points and radii.
    * `div_compartment_sampler` forms frusta by sampling
      the segment radius at each compartment boundary
    * `div_compartment_integrator` computes the compartment
      areas and volumes exactly by summing all frustra
      in the intersection of the segment and the compartmnet
      span.

* Extrae linked at execution time

* cleaning project

* Complex compartments

* Use divided compartments to determine FVM coefficients.
* Pick correct control volume in FVM from sgement position (avoids
  off-by-half error.)
* Add colour override functionality to tsplot: `--colour` option.
* Add const accessor for cell soma.
* Source formatting, comments in `math.hpp`
* Fix `range_view`: was using incorrectly named type trait.
* Add unit test for `range_view`.
* Allow points of discontinuity to be omitted from L-infinity norm
  calculations.
* Add `-d, --min-dt` option to `validate.exe` to control time
  step in validation convergence tests.
* Add validation test: confirm divided compartment policy does
  not effect results on simple frustrum dendrites.
* Change default max compartments on validation tests to 100
  (ad hoc observed convergence limit at dt circa 0.001 ms;
  finder spatial division would required much finer dt.)
* Make NEURON validation data generation scripts use CVODE by
  default, and with `secondorder=2` when non-zero `dt` is given.

* Remove division policy type parameter.

* Use only `div_compartment_integrator` for compartmentalization in
  `fvm_multicell`. The policy will later be moved to a backend
  policy class.
* For now, disable validation tests that test different division
  policies (see above).
* Tweak comments and remove redundant `using`, following comments
  on PR#54.

* Minor twicks and corrections

* Split `tok::number` token into `tok::real` and `tok::integer`
* Add new tokens: `tok::tilde`, `tok::arrow` and `tok::kinetic`
  (keyword)
* Add new expressions: `IntegerExpression` (deriving from
  `NumberExpression`), `ReactionExpression`, `StoichExpression`
  and `StoichTermExpression`.
* Add new `procedureKind`: `kinetic`.
* Add parse rules for KINETIC blocks.

Note: no real testing yet.

Complex compartments

* Use only `div_compartment_integrator` for compartmentalization in
  `fvm_multicell`. The policy will later be moved to a backend
  policy class.
* For now, disable validation tests that test different division
  policies (see above).
* Tweak comments and remove redundant `using`, following comments
  on PR#54.

* Use divided compartments to determine FVM coefficients.
* Pick correct control volume in FVM from sgement position (avoids
  off-by-half error.)
* Add colour override functionality to tsplot: `--colour` option.
* Add const accessor for cell soma.
* Source formatting, comments in `math.hpp`
* Fix `range_view`: was using incorrectly named type trait.
* Add unit test for `range_view`.
* Allow points of discontinuity to be omitted from L-infinity norm
  calculations.
* Add `-d, --min-dt` option to `validate.exe` to control time
  step in validation convergence tests.
* Add validation test: confirm divided compartment policy does
  not effect results on simple frustrum dendrites.
* Change default max compartments on validation tests to 100
  (ad hoc observed convergence limit at dt circa 0.001 ms;
  finder spatial division would required much finer dt.)
* Make NEURON validation data generation scripts use CVODE by
  default, and with `secondorder=2` when non-zero `dt` is given.

Feature/divided compartments

Consolidate validation tests (issue #41)

* Make `algorithm::sum`, `algorithm::mean` more generic,
  allowing use with array types.
* Add `div_compartment` compartment representation, that
  holds geometric information for each half of a compartment
  that will then be used in calculating control volumes.
* Add three compartmentalisation schemes/policies that
  discretize a segment into `div_compartment` objects:
    * `div_compartment_by_ends` divides based only on the
      segment end points and radii.
    * `div_compartment_sampler` forms frusta by sampling
      the segment radius at each compartment boundary
    * `div_compartment_integrator` computes the compartment
      areas and volumes exactly by summing all frustra
      in the intersection of the segment and the compartmnet
      span.

* Simplify trace analysis and reporting code in
  `trace_analysis.hpp`
* Consolidate convergence test run procedures into
  new class `convergence_test_runner`.

Feature/new test model

More range functionality, unit tests.

* Add documentation of template parameters for `filter_iterator`.
* Document use of `uninitalized<F>` for holding functional objects
  in `filter_iterator` and `transform_iterator`

* Make `test_common_cells.hpp` and `ball_and_taper.py` agree.
* Add `ball_and_squiggle` model that has a tapering undulating
  profile.

* New `filter` view: lazily selects based on predicate.
* Generic `front` and `back` for sequences.
* New rangeutil STL wrappers `stable_sort_by`, `all_of`, `any_of`.
* Consolidate common utility unit testing structures into
  `tests/unit/common.hpp`

* Tests for `math::pi`, `math::lerp`, `math::area_frustrum`
  and `math::volume_frustrum`
* Fix `math:pi<long double>()`.

modcc: Support for units syntax within a state block.

Add "lib" to search prefixes for libtbb

Address deprecated use of 'symbol' warning.

Julia 0.5 deprecates use of `symbol` instead of
`Symbol`. This patch just substitutes the
correct call.

Julia 0.5 deprecates use of `symbol` instead of
`Symbol`. This patch just substitutes the
correct call.

* Fix quoting error in library search.
* Add "lib" to prefixes when system is "Linux".

Feature/cable validation