Disambiguate e symbol in statements like the following
  for (auto& e: e->terms())
This caused GCC 5 to give an error.

Build images.dir when building report.pdf
Add outputs to .gitignore

Incorporate symbolic GE code from prototype (with some simplifications) in msparse.hpp, symge.hpp and symge.cpp, together with unit tests.

Add two kinetic scheme test cases for validation: test_kin1 (simple exponential scheme) and test_kinlva (combination of exponential gate and a three-species kinetic scheme, modelling a low voltage-activated Calcium channel from Wang, X. J. et al., J. Neurophys. 1991).

Adapt numeric HH validation data generation to LVA Ca channel, with explicit stopping at stimulus discontinuities.

Add two new validation tests based on above: kinetic.kin1_numeric_ref and kinetic.kinlva_numeric_ref (multicore backend only).

Introduce a BlockRewriterBase visitor base class, as an aid for visitors that transform/rewrite procedure bodies; refactor KineticRewriter over this class.

Introduce common error_stack mixin class for common functionality across Module and the various procedure rewriters.

Implement visitors and public-facing convenience wrappers in symdiff.hpp and symdiff.cpp:

involves_identifer for testing if an expression contains given identifiers.
constant_simplify for constant folding with removal of trivial terms arising from a NumberExpression of zero or one.
expr_value to extract the numerical value of a NumberExpression, or NaN othereise.
is_zero to test if an expression is numerically zero.
symbolic_pdiff to perform symbolic partial differentiation; this adds a new (not parseable) expression subclass to represent opaque partial differential terms.
substitute to substitute identifiers for other expressions within an expression.
linear_test for linearity, diagonality and homogeneity testing (this is probably redundant, given ExpressionClassifier already exists).
Simplify unnecessary uses of make_unique with Vistor subclasses.

Make SOLVE statement rewriting more generic, through the use of solve-rewriter visitors CnexpSolverVisitor, SparseSolverVisitor, and DirectSolverVisitor; implementations in solvers.hpp and solvers.cpp. Supports multiple SOLVE statements for independent subsets of state variables with the BREAKPOINT block.

Add block rewriter for the removal of unused local variables, with convenience wrapper remove_unused_locals.

Generalize is_in utility in modccutil.hpp.

Simplify expression comparison in modcc unit tests with EXPECT_EXPR_EQ macro added to tests/modcc/test.hpp, that operates by comparing expression text representations.

Simplify and consolidate verbose printing in modcc unit tests with verbose_print function that tests the global verbose flag and handles expression_ptr and similar which have to_string methods.

fix bug introduced when global policy was refactored.

The reproducible problem is now green on two separate systems

The MPI global gather_all unit test now compiles.

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

Bug: `modcc` was generating invalid code when generating optimized kernels.

The optimized kernels use raw pointers instead of views, and the generated code was using view semantics.

* Use appropriate `memory::copy` invocation for the optimized kernel case.

Add a dry run mode, inspired by the dry run mode implemented in NEST. A dry run
of a model simulates running a large distributed model by running only the work
of one of the ranks, with artificial spike input from the other "dummy" ranks.

This is implemented as a new global communication back end, dryrun_global_policy,
the implementation of which is straightforward:

a new implementation of gather_spikes that takes the local spikes and
replicates them n times where n is the total number of simulated
ranks.
the global_policy::size() method returns the number of ranks in the
simulated run
the new back end has to store some state that records the number
of simulated ranks and cells per rank, which are set using the new
global_policy::set_sizes() method
Some CMake modificatins were required:

make the selection of the global communication backend have the same
interface as that for selecting the threading back end.
small improvements to the selection of the threading back end to
make the cthread option visible in ccmake, and have consistent
CMake variable naming.
Command line options were also extended:

a --dry-run-size or -D option can be used to supple the number
of dry run ranks on the command line.
the miniapp driver was updated to set the dry run size and cell
count via the new global_policy::set_sizes() interface.
 

merge mainline into bcumming fork

Add some finer grained profiling to track exactly what time is spent inside of mpi calls.

Alexander Peyser authored 8 years ago and Sam Yates committed 8 years ago

Add threading pool built on `std::thread`

* Provide new threading model 'cthread' for nestmc based on a pool of `std::thread` objects.
* Unify duplicated timer class provided by `serial`, `omp` and now `cthread` threading models.

Builds and runs on BGQ up 16k nodes

Remove redundant MPI set

* Fix CMakeLists to handle build as a subproject

When several CMake generated projects are build together, it is common
practice to have a 'superproject' CMakeLists that uses
  add_subdir(proj1)
  add_subdir(proj2)
  ...
where each subproject is a self contained CMake based project
(Example proj1=HPX, proj2=nestmc, proj3=another, ...)

CMAKE_SOURCE_DIR always points to the top level directory which
is the superproject dir in this case, whereas PROJECT_SOURCE_DIR
always points to the root of the current project() in the CMakeLists
so one shouod use PROJECT_SOURCE_DIR as this gets the relative paths
correct.

* Add option to turn off auto generation from *.mod files

* Fix #134 : Change CMake WITH_OPTION to NMC_WITH_OPTION, compiler #define to NMC_HAVE_OPTION

1) The user may select an option by saying NMC_WITH_XXX

2) This may trigger CMake to use find_package(...) or setup some
other variables. CMake can then set variable NMC_HAVE_XXX and add a
what has actually been used.

3) Code should use #ifdef NMC_HAVE_XXX to check for a feature

Old CMake/define      New CMake                 Compiler #define
----------------      ---------                 ----------------
THREADING_MODEL       NMC_THREAD_MODEL
    WITH_TBB          NMC_WITH_TBB              NMC_HAVE_TBB
    WITH_OMP          NMC_WITH_OMP              NMC_HAVE_OMP
    WITH_SERIAL       NMC_WITH_SERiAL           NMC_HAVE_SERIAL

WITH_MPI              NMC_WITH_MPI              NMC_HAVE_MPI
WITH_CUDA             NMC_WITH_CUDA             NMC_HAVE_CUDA
WITH_GPU                                        NMC_HAVE_GPU
WITH_ASSERTIONS       NMC_WITH_ASSERTIONS       NMC_HAVE_ASSERTIONS
WITH_TRACE            NMC_WITH_TRACE            NMC_HAVE_TRACE
WITH_PROFILING        NMC_WITH_PROFILING        NMC_HAVE_PROFILING

Other user visible CMake vars
-----------------------------
VECTORIZE_TARGET            -> NMC_VECTORIZE_TARGET
USE_OPTIIZED_KERNELS        -> NMC_USE_OPTIIZED_KERNELS
BUILD_VALIDATION_DATA       -> NMC_BUILD_VALIDATION_DATA
BUILD_JULIA_VALIDATION_DATA -> NMC_BUILD_JULIA_VALIDATION_DATA
BUILD_NRN_VALIDATION_DATA   -> NMC_BUILD_NRN_VALIDATION_DATA
VALIDATION_DATA_DIR         -> NMC_VALIDATION_DATA_DIR

Variables such as NMC_THREADING_MODEL and NMC_VECTORIZE_TARGET now use
enumerated cmake values so you can toggle between them in ccmake gui.
SYSTEM_TYPE_CRAY/BGQ        -> NMC_SYSTEM_TYPE (Generic/Cray/BGQ)

* Use generator expression for modcc path

Some IDE's (like Xcode for example), override the CMake binary paths
and add /Debug or /Release etc so rules that have hard coded paths
to binaries will fail.

make_unique has been moved into nest:mc::util namespace to avoid a conflict with std::make_unique
util:: cbegin and cend clash with std:: equivalents

* fix signed-vs-unsigned warning in mechanisms unit test
* fix linking error caused by out of date linkage description in CMake for the performance test

* Add `target_link_libraries` for external libs in `tests/global_communication/CMakeLists.txt`.

Fixes issue #132

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

feature: #67 (GPU Support)

Implement stimulii as part of the mechanism framework, as described here: #87 (comment)

* A hand-written stimulus point process derived from `mechanisms::mechanism` was written for each back end. 
* The lowered `fvm_multicell` type explicitly constructs a stimulus mechanism if there are any stimuli attached to the its cells.
* This mechanism is added to the other mechanisms in the lowered cell, so that the update of current is performed in the current update loop (i.e. via the `nrn_current()`) method.

This isn't an ideal solution: we still have a hard coded stimulus type in the lowered cell, however the stimulus is now "in the right spot", and we can refine this better when we work on a better design for generalised mechanisms (i.e. when we have figured out what we going to do).

fixes #104.

Vasileios Karakasis authored 8 years ago and Sam Yates committed 8 years ago

These tests are intended to test the sanity of the `modcc` generated code for the individual mechanisms. The don't have any physical background. Potentially optimized CPU-targeted mechanisms generated in the build are compared with unoptimized mechanisms generated from the reference modules.

* Add generic unit tests for individual mechanisms.
* Make unit tests exercise potential problems with aliased indexes (point processes).
* Ensure unit tests correspond to multiple low level vector operations.
* Ensure unit tests run with voltage, current and indices initialized with varying values.
* Refactor CMake code for module compilation to reduce cut-and-paste code and build complexity.

Compiles on bgq

define MPICH2_CONST const for BGQ

Add SYSTEM_BGQ to cmake to handle mpi headers

Fixes #85

* Modify `parse_expression` to take a controlling (parent) precedence.
* `parse_expression` folds left over sequences of sub-expressions with decreasing operator precedence (accumulates in `lhs`).
* Use recursion rather than accumulator for left fold in `parse_binop` to simplify code logic.
* Extend parser unit test to cover more complicated, multi-level expression.
* Remove (now) redundant parenthesis from derivative check block in kinetic rewriter test.

Fixes #94

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

This PR addresses two issues that were closely related:
* correctly accounting for the current contribution of density mechanisms to CVs at branch points, where the density mechnanism is not present on all branches. This was discussed in issue #20.
* adding support for weighting of current densities calculated from density mechanism. This is required to weight the current contribution to CVs in issue #20.

## small updates

* update CMake rules for finding libunwind, because it broke for some reason.
* add `binary_find` and unit tests to the algorithms library
    * returns an iterator, as opposed to `std::binary_search`, which returns a boolean.
    * works with ranges.
* added `subrange_view` specialization that takes a subrange specified by a pair of indexes
* added `assign_from` to range utils
    * a helper function that returns a proxy type that can be copied into a container
    * evaluate a range and store contents in a container, with minimal verbosity in user code
    * simple syntax for initializnig a container where it is declared, e.g. `std::vector<int> vec = util::assign_from(...);`
* update the LaTeX documentation for the FVM scheme
* fix bug when an event has to be delivered exactly 0 ms in the future in `cell_group`
    * avoid divide by zero on the diagonal of the linear system in the new formulation.

## updated FVM formulation

Most of the changes were in the `fvm_multicell` type.
* The FVM formulation was changed slightly, moving parameters (e.g. dividing both sides of equation by dt)
    * to ensure symmetric positive definate matrix property with new partial weights
    * to give the terms in the linear syste, i.e. the matrix, solution and rhs vectors more natural units    * the system is no `G*v = i`, where `G` is conductance matrix with units [uS], `v` is voltage [mV] and `i` is current [nA].
    * change names of fields from non-descriptive things like `face_alpha` to `face_conductance`
    * add more comments that explicitly give the expected units of fields inside the back end (to help future generationas trying to understand the code... and to help me understand it three weeks from now) 
* keep additional information about the surface area of sub-control-volumes at the start and end of each segment when calcuating CV areas, capacitances and face conductances
* use this information to ensure that current contributions from density channels on branching points are properly accounted for by weighting
* remove weighting from point process currents, because they are calculated with the correct units of nA
* plumbing work to add support for user-supplied weights
    * the backend code for multicore and gpu now supports weights for mechanism generation
    * update `cprinter` and `cudaprinter` to generate kernels that use user-supplied weights for density channels
* backend modifications to generate matrices and RHS vectors according to the new formulation

fixes #20
fixes #120

Vasileios Karakasis authored 8 years ago and Sam Yates committed 8 years ago

Add cyclic iterators and cyclic range view.

Cyclic iterators wrap around when the reach the end of the underlying range over which they are defined.

Alexander Peyser authored 8 years ago and Sam Yates committed 8 years ago

Fix `is_in` compilation error with initializer lists and clang 3.7.1.

Addresses part of issue #121: `template <typename T, int N> bool is_in(T thing, const T (&list)[N])` fails to match against an initializer list second argument with clang-3.7.1 in `modcc/cprinter.cpp`.

* Add overload `template <typename T> bool is_in(T thing, const std::initializer_list<T> list) ` for `is_in` in `modccutil.hpp`. Fixes clang issue and verified to work with a version of gcc as well.

Addresses in part issue #113.

* Make compatibility wrappers/functions that dance around xlC
  bugs. Wrappers are provied in `util/compat.hpp` and live in
  the `compat` namespace.
      - `compat::end` reimplements `std::end` but in a way that
	apparently does not trigger the xlC bug.
      - `compat::compiler_barrier_if_xlc_leq()` inserts a compiler
	reordering barrier if the compiler is xlC and the version
	less than or equal to that specified. Name is deliberately
	verbose.
      - `compat::isinf()` is an inline wrapper around `std::isinf()`,
	which apparently is sufficient to defuse an evaluation
	order bug with `std::isinf()` in switch statements.
* Use `compat::compiler_barrier_if_xlc_leq()` in `util::unitialized`
  reference access methods to avoid improper reordering with -O2.
* Use `compat::isinf()` in `test_math.cpp` to defuse improper
  reordering within `EXPECT_EQ` gtest macro of `std::isinf()`.
* Use `compat::end()` in `util::back()` and `util::cend()` to avoid
  incorrect `std::end()` behaviour with -O2.
* Use `util::cend()` in `algorithms::sum()`, again to avoid
  incorrect `std::end()` behaviour with -O2.

Vasileios Karakasis authored 8 years ago and Sam Yates committed 8 years ago

Fix compilation issue with uninitialized introduced in PR #101.
* Use `const void*` in `static_cast` for `cptr()` member function.

Adds a new KineticRewriter visitor that transforms (after semantic analysis) a parsed KINETIC procedure into an equivalent DERIVATIVE procedure. The visitor takes a ProcedureExpression and composes the equivalent procedure, available via the as_procedure() method on the visitor object.

Move common functinality for 'local' variable insertion during transformation phase to new files astmanip.?pp.
Add Expression method for directly setting scope.
Use scope_ptr type alias widely.
Implement correct clone() behaviour for DerivativeExpression
Implement KineticRewriter transforming visitor class.
Add equivalence test for KineticRewriter: the test incorporates a simple ad-hoc algebraic expression simplifier.
Add unit test to Parser.parse_binop to exercise bug #94

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

This addresses all of the compiler warnings and errors for xlc when compiled at -O0. There are still compiler bugs when compiling with higher optimization levels, however they are more challenging.

* Ignore the incorrect `-Wno-missing-braces` warnings (similarly to Clang).
* Remove `-qhalt=e flag` inserted by CMake.
* Remove redundant comparison of char to EOF in lexer.
* The XLC compiler was crashing inexplicably on one call of the following method:
   ```void run(..., const std::vector<float>& excl={}) {...}```
  This was fixed by not having a default value for the last argument.
* Add some curly braces to silence warning for dangling else.

fixes #97

Correct pointer casting operations in `uninitialized`: issue #100

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

fixes #90

The c and cuda printer were not printing else branches in if else statements.
* added logic for printing else branches
* added if-else expressions to data/test.mod

No unit tests were added because we don't currently have a unit testing framework for the output mechanisms.

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

Fixes #68
Corresponding feature: #67

* Reproduce the hh-soma validation test on GPU.
* Reproduce the ball and stick model on GPU.
* Reproduce miniapp spike chains.
* Add `cell_group` unit test to the cuda unit tests: builds simple ball and stick model and integrates for 50ms and records how many spikes occur; it is a simple early warning that something is broken, but is no substitute for the validation tests.
* Update the `validate_soma`, `validate_ball_and_stick` and `validate_synapses` validation tests for the GPU backend:
    * refactor individual tests into test runner functions that are templated on lowered cell type;
    * for each of the original validation tests add a cuda (.cu) implementation, and write an additional "backend" field to the validation trace metadata.
* Use a `CPrinter` to generate the same `net_receive` block that is used for the multicore backend.
  Note: this is not efficient, because each read/write requires a cuda memcpy between host and device memory, however it allows us to pass all unit and validation tests. A more efficient GPU-specific implementation is left for later optimization work.
* Make paths to `gtest.h`, `test_common_cells.hpp` etc. in test sources consistent relative paths, and remove the `tests/` directory from the include path.

Fixes #80
  * Avoid ctor ambiguity by using parentheses for copy ctor invocation with `nlohmann::json` class.

Fixes #82

Bugfix: issue #80 

* Avoid ctor ambiguity by using parentheses for copy ctor invocation with `nlohmann::json` class.

bugfix for missing mechanism paths

The mechansims/gpu and mechanisms/multicore paths were not being generated
by CMake during configuration, and modcc was silently ignoring that it was
not able to write to the nonexistant target paths.
I have updated CMake to generate the missing gpu and multicore target paths.

This PR is part of the gpu feature merge. The GPU implementation is not implemented here. Instead, we focus on refactoring of the original "multicore" back end so that it is ready for adding the GPU back end.

This is a big and messy change, for which I am sorry.

## build System

- A `WITH_CUDA` option has been added to the main CMakeLists. This finds the CUDA toolkit, and sets CUDA compiler flags, and will build unit tests for the gpu back end.
- The CMakeLists that generates mechanisms with modcc has been updated to generate CUDA mechanisms.
- the library is now named `libnestmc` instead of `libcellalgo`
- merge the external libraries that are optionally linked againts (tbb, libunwind, etc) into a single
  `EXTERNAL_LIBRARIES` list for ease of linking

## modcc

- the cprinter and cudaprinter have had small changes to generate mechanism files that are compatible with the refactored library.

## algorithms

- the indexes into algorithm was "rangified". An algorithm `index_into_iterator` takes two ranges as inputs to make a range that lazily generates the index of sub into super set.

## backends

- made a new path `src/backends/` for backend specific type and implementation code.
- currently:
  - complete support for the `multicore` and `gpu` backends
  - `gpu` back end is not optimized or validated
- the back end implementations are in `src/backends`
- a single `backend` class, `nest::mc::{multicore,gpu}::backend`, is provides all backend specific type and implementation details fro each backend
  - storage containers
  - Hines matrix assembly for FVM method
  - Hines matrix solver
  - mechanism "factory"

## lowered fvm cells

- removed `fvm_cell` because this can be modelled with an `fvm_multicell` with one cell.
- refactored to use backend type and implementation from `fvm_policy`
- use `std::vector` instead of containers in `nest::mc::memory::` where possible when building cells.

## memory library

Refactor the "memory" library, making it much simpler and better integrated into the rest of the application. However, it is still far from perfect. The `Coordinator` approach needs to be improved, most likely by putting target-specific wisdom into pointers (which could obviate the need for a `const_array_view` type.
1. renaming and moving
   - move from `vector/` to `src/memory`
   - move into the `nest::mc` namespace, i.e. all types and functions are now in `nest::mc::memory`
   - change from camel case nameing scheme to NestMC style naming.
2. simplification
   - remove the CRTP cruft that was used to make operator overloading work for operations like copying from one range into another, and filling a range with a constant value. These have been replaced with `memory::fill()` and `memory::copy()` helper functions. This simplified the code _a lot_, and makes code clearer in user land.
   
   ```
   // before
   vec(0, 5) = other;
   // now
   memory::copy(other, vec(0, 5));
   ```
   - add some wrappers in `src/memory/wrappers.hpp` that help with making views. These are particularly useful for passing `std::vector` through interfaces that expect a view.

## debug backtraces

Added stack traces for debugging.
- support for OSX and Linux via libunwind
- backtraces can be generated manually `nest::mc::util::backtrace().print()`
  - creates a new file and dumps trace into file
  - prints message to `stderr` with file name and instructions on how to analyse
- backtraces are also automatically generated when an assertion `EXPECTS` statement fails
- a python script in `scripts/print_backtrace` pretty prints the output with file name, line number and demangled symbols

## util simplification and consolidation

The `src/util.hpp` file was removed
- much of its contents were dead code and just removed
- useful components like `pprintf` and `make_unique` were moved into the `src/utils` path in standalone files
There was a lot of overlap between functionality provided in `src/memory/util.hpp` and existing functions/types in the `nest::mc::util` namespace. The `memory` implementations were removed, and their `nest::mc::util` counterparts used. There is still some work remaining, namely moving the rest of the `src/memory/util.hpp` into `src/util/...`