Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

Fixes #212.

* Update the main `CMakeLists.txt` file to select the cthread back end by default, and present the threading options in the order: cthread, tbb, serial.

Fixes #196.

Correct treatment of missing coefficients in `cnexp` solver.

* Extend `EXPECT_EXPR_EQ` functionality with wrapper that works with `Expression *` and `expression_ptr` arguments.
* Replace string comparison checks in `test_symdiff.cpp` with equivalents that use `EXPECT_EXPR_EQ`.
* Check explicitly for missing coefficient in `cnexp` solver, which should be treated equivalently to zero.

* Add class for managing state associated with binning event times across integration periods.
* Include support for no or fixed 'regular' binning.
* Add a gtest-assertion compatible test for comparing sequences of floating point numbers: `testing::seq_almost_eq` in `tests/unit/common.hpp`.
* Rename `cell_` in `cell_group` to `lowered_`, to clarify intent (i.e. lowered cell state is very different from a `cell` object, and maintains state for many cells).
* Reformat some comments for consistency.

Note that the `event_binner` class is not used in this commit for actual binning: the original logic is still in place.

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

Fixes #152

* Remove OpenMP back end implementation.
* Remove CMake options for OpenMP.
* Simplify the threading model selection code in CMake.
* Remove Extrae benchmarking scripts, which require OpenMP.

Includes a fix to the readme which applies to any external modcc, and a way to keep from rebuilding locally an external modcc.

The requirements for event types for use in event_queue were restrictive and a bit 'special'.

  * Allow event times of any time which is well ordered by operator>.
  * Allow any event type with a public time member containing the time value.
  * Provide customization point event_time() via ADL for extracting the time value for event types that do not have a time member.
  * Simplify interface: push(begin, end) was only ever used in the unit test; add empty() const method.
  * Add unit test for more flexible functionality.

Differing classes had their own time_type and other classes were parameterized on it in ways that were compatibile, but only by chance. 

With these changes, modifying the time type used in spike will propagate through to all dependent classes.

  * Rename generic spike<I, T> as basic_spike<I, T>. 
  * Use spike = basic_spike<cell_member_type, float> as the common spike type.
  * Replace instances of spike_type aliases with just spike.
  * time_type aliases are defined in terms of spike::time_type.
  * Remove time_type parameterization in connection.
  * Remove time_type parameterization in communicator.
  * Remove time_type parameterization in exporter classes.

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

fixes #197

* Add functions that can find thread affinity and the number of available cores on linux systems via `sched_getaffinity`.
* On other systems they default to "unknown affinity" and return 0 to indicate that the number of cores is unknown.
* Set the default number of threads according to the new functions above if no environment variable explicitly setting the number of threads is set.
* Validate environment variable value against regex and range check; terminate if improper.
* Terminate if no number of threads is provided and the library is unable to determine a sensible number automatically.

Basic features:

* Compile with -t avx512
* Automatically set up by CMake if USE_OPTIMIZED_KERNELS is on and VECTORIZE_TARGET is set to KNL 
* Generic SIMD printer that contacts a SIMD backend for emitting the actual SIMD intrinsics

Note: compilation for the avx512 target requires the Intel compiler. 

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

* Add an additional exchange at the end of the time stepping loop to ensure that any spikes still in spike buffers are exchanged and saved to disk.
* Remove the parallel sort from the cthread back end to fix compilation errors with CUDA and cthreads.
* Fix `nvcc` warning concerning initialization of `std::size_t` variable with literal `-1`.

Fixes #183

Use a device kernel for net_receive state updates.
Note: very naive, but gives about a 30% speed up on the 1000 cell miniapp test. All the fun optimization will end up under issue #184.

This also incorporates PR #192, so this PR will be amended if that one is rejected.

When NMC_WITH_CUDA is off, still allow the generation
of CUDA mechanisms with modcc by exposing the
`build_all_gpu_mods` target; this can be performed
independently of the presence (or otherwise) of a
CUDA development environment.

This eases development of GPU-related modcc tasks,
as preliminary work can be performed and checked on
machines without a CUDA environment.

Using latexmk simplifies the makefile structure and avoids many of the problems with redundant pdflatex invocation.

Rewrite docs/model/makefile and docs/model/images/makefile to use latexmk for building and cleaning.
Remove generated report.bbl from repo.

Fixes #166

Debackwardsate math::signum().
Unit tests in test_math.cpp for same.

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

The storage of matrix data, and the operations on matrices (i.e. matrix assembly and matrix solution), have not been implemented in a consistent manner.

The main problem was that matrix assembly was managed by a `matrix_assembler` type provided by the back end, which had views on information that it required to perform assembly. Specifically, the views were on properties like `face_conductance`, model state like `voltage` and `current`, and on the underlying matrix storage `d`, `u` and `p`.

This was not a good solution because 
  * there was a hidden dependence of the assembly on model data. e.g. if the voltage storage was reallocated, the reference in the `matrix_assembler` would become stale.
  * if we want back end specific optimizations that require a different data layout to that used elsewhere in the back end, this layout should be shared with the solver, but there is no obvious mechanism for doing that.

This patch addresses this by making a `matrix_state` type in the back end.
  * stores the matrix state opaquely, allowing back-end specific optimizations
  * provides interface for performing operations on the state, namely `assemble`, `solve` and `get_solution`.
  * stores fields such as `face_conductance` and `cv_capacitance` that were stored in the `fvm_multicell`, despite being used only in matrix assembly.
  * takes `voltage` and `current` as parameters to the `assemble` interface, removing the hidden reference to model state.

The actual data layout has not been changed in this PR. Instead, the interface has been refactored and hidden references removed so that it is now possible to implement back-end specific optimizations cleanly.

Fix morphology section ctor bug.
Add morphology pools for miniapp from which morphologies are drawn in the recipe.
Add command-line options to expose above.
Add option --report-compartments to (slowly) check the min, mean, and max number of compartments in the generated cells across the simulation.
Use morphology::add_section to do all the heavy lifting; no need to make section_geometry objects by hand, unless you really, really want to.

This PR is a prelude to closer integration of the random morphology generation with the miniapp, with the first step being support for recipes that create cells from morphologies generated off-line. It aims to use nest::mc::morphology as the flat morphology-only representation that can be used to construct nest::mc::cell objects and which can exist as a target for SWC conversion and random morphology generation.

Simplify swc io implementation:
Avoid throwing exceptions in istream parsing and swc_record constructors — only throw when explicitly checking consistency, or when parsing a full sequence of records.
Allow direct access to record members.
Separate parsing considerations from canonicalization (renumbering, sorting) of a sequence of records.
Move lmorpho morphology classes into src/
Add invariant check procedure for morphology.
Make cells via swc -> morphology -> cell building, rather than direct swc -> cell.
Allow option to use 'natural' discretization in morphology to determine number of compartments in built cell.
Update test_swcio.cpp to accommodate new API.

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

This patch removes the many small `cudaMemcpy` calls for single values, except for those from calling `net_receive` in event delivery.

The small copies during initialization were from when the upper diagonal and time invariant component of the diagonal were computed on the host. There were many small reads/writes to device memory accessing the `p` and `u` vectors.

* Remove many small device copies in matrix setup by copying required data to host, computing, and then copying back in one copy.
* Add `constexpr` test `is_debug_mode()` for having been compiled in debug mode (tests `NDEBUG`).
* Only perform `is_physical_solution` test if `is_debug_mode()` is true. (The `is_physical_solution` test triggers a single copy from device to host on each time step to test whether the voltage has exceeded some "reasonable" physical bounds.)

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

Fixes #125

* Add `cuda_atomic_add` and `cuda_atomic_sub` wrappers for atomic addition.
* Choose native atomic add for Pascal and later architectures.
* Choose CAS workaround for devices earlier than Pascal.
* Add unit test for wrappers.
* Change default CUDA architecture target to `sm_60` in `CMakeLists.txt`.

Vasileios Karakasis authored 8 years ago and Sam Yates committed 8 years ago

* Add missing `<string>` header to `modcc/msparse.hpp`

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

Moves spike detection to the back end, which is required to have fast spike detection on the gpu.

Fixes #106.

Backend:
* Move spike detection into the backend.
* Add a `threshold_watcher` type to each backend that is initialized with a reference to the field that it is to watch, along with indexes and thresholds for each variable in the field that is being watched. This class presents a simple host-side interface:
   * `test(t)` tests for crossings at time `t`.
   *  `crossings()` returns all threshold crossings.
   * `clear_crossings()` resets (clears) the collected crossing events.
* Implement the multicore back end detector directly from the original code.
* Implement a `gpu_stack` for use with the gpu back end, that lets threads in a kernel conditionally push back into a flat array.

Miniapp:
* Run a single-step dummy run of model before starting the profiler, when profiling is enabled.
* Initialize the spike output callback functions _after_ the dummy run so that spikes from the dummy step are not output.

`cell_group`:
* Pass responsibility for spike detection to the lowered cells (`fvm_multicell`) and associated back ends.

`memory`:
* Add a new allocator for CUDA managed memory.
* Implement `managed_ptr` and `make_managed_ptr`, which are managed memory equivalents of `std::unique_ptr` and `std::make_unique_ptr`.

Tests:
* Improve host-side spike detection unit tests.
* Add device-side spike detection unit tests.
* Add unit tests for `gpu_stack`.

Building:
* Surpress CMP0023 CMake warning.

* Ignore dendrite branches with negative radii arising from correlated child diameter distribution.
* Fix fencepost errors in morphology discretization.
* Rename `tip.p` to `tip.point`.

Adds a stand-alone program for the generation of random morphologies form a L-system description. The algorithm is that of Burke (1992), with some of the extensions provided by Ascoli et al. (2001).

Two sets of L-system parameters have been included, corresponding to alpha motoneurons and Purkinje cells, but there is certainly something wrong with the data for the latter, and more correct numbers will probably need to be synthesized from existing Purkinje cell morphological information.

Documentation for `lmorpho` is incomplete, but the command line help (`--help`) goes some way to explain the usage. In order to get output, one must specify `--swc` or `--pvec` (or both) to emit SWC files or the structural parent vectors. Coarser discretization can be obtained with the `--segment` option.

Some minor modifications have been included in other parts of the source repo:
* Added copy constructor for `TextBuffer` in `modcc/textbuffer.hpp`, required to keep clang++ 3.5 happy.
* Disabled 'maybe-uninitialized' warnings from gcc, as these get raised improperly with some uses of `util::optional`. (Seems that this is also an issue for `boost::optional`.)
* Moved `tinyopt.hpp` into the `src/` directory, and extended the functionality a bit to support keyword arguments.
* `validate.cpp` modified slightly to accommodate new `tinyopt`.

fix #164

Disambiguate e symbol in statements like the following
  for (auto& e: e->terms())
This caused GCC 5 to give an error.

Build images.dir when building report.pdf
Add outputs to .gitignore

Incorporate symbolic GE code from prototype (with some simplifications) in msparse.hpp, symge.hpp and symge.cpp, together with unit tests.

Add two kinetic scheme test cases for validation: test_kin1 (simple exponential scheme) and test_kinlva (combination of exponential gate and a three-species kinetic scheme, modelling a low voltage-activated Calcium channel from Wang, X. J. et al., J. Neurophys. 1991).

Adapt numeric HH validation data generation to LVA Ca channel, with explicit stopping at stimulus discontinuities.

Add two new validation tests based on above: kinetic.kin1_numeric_ref and kinetic.kinlva_numeric_ref (multicore backend only).

Introduce a BlockRewriterBase visitor base class, as an aid for visitors that transform/rewrite procedure bodies; refactor KineticRewriter over this class.

Introduce common error_stack mixin class for common functionality across Module and the various procedure rewriters.

Implement visitors and public-facing convenience wrappers in symdiff.hpp and symdiff.cpp:

involves_identifer for testing if an expression contains given identifiers.
constant_simplify for constant folding with removal of trivial terms arising from a NumberExpression of zero or one.
expr_value to extract the numerical value of a NumberExpression, or NaN othereise.
is_zero to test if an expression is numerically zero.
symbolic_pdiff to perform symbolic partial differentiation; this adds a new (not parseable) expression subclass to represent opaque partial differential terms.
substitute to substitute identifiers for other expressions within an expression.
linear_test for linearity, diagonality and homogeneity testing (this is probably redundant, given ExpressionClassifier already exists).
Simplify unnecessary uses of make_unique with Vistor subclasses.

Make SOLVE statement rewriting more generic, through the use of solve-rewriter visitors CnexpSolverVisitor, SparseSolverVisitor, and DirectSolverVisitor; implementations in solvers.hpp and solvers.cpp. Supports multiple SOLVE statements for independent subsets of state variables with the BREAKPOINT block.

Add block rewriter for the removal of unused local variables, with convenience wrapper remove_unused_locals.

Generalize is_in utility in modccutil.hpp.

Simplify expression comparison in modcc unit tests with EXPECT_EXPR_EQ macro added to tests/modcc/test.hpp, that operates by comparing expression text representations.

Simplify and consolidate verbose printing in modcc unit tests with verbose_print function that tests the global verbose flag and handles expression_ptr and similar which have to_string methods.

fix bug introduced when global policy was refactored.

The reproducible problem is now green on two separate systems

The MPI global gather_all unit test now compiles.

Ben Cumming authored 8 years ago and Sam Yates committed 8 years ago

Bug: `modcc` was generating invalid code when generating optimized kernels.

The optimized kernels use raw pointers instead of views, and the generated code was using view semantics.

* Use appropriate `memory::copy` invocation for the optimized kernel case.

Add a dry run mode, inspired by the dry run mode implemented in NEST. A dry run
of a model simulates running a large distributed model by running only the work
of one of the ranks, with artificial spike input from the other "dummy" ranks.

This is implemented as a new global communication back end, dryrun_global_policy,
the implementation of which is straightforward:

a new implementation of gather_spikes that takes the local spikes and
replicates them n times where n is the total number of simulated
ranks.
the global_policy::size() method returns the number of ranks in the
simulated run
the new back end has to store some state that records the number
of simulated ranks and cells per rank, which are set using the new
global_policy::set_sizes() method
Some CMake modificatins were required:

make the selection of the global communication backend have the same
interface as that for selecting the threading back end.
small improvements to the selection of the threading back end to
make the cthread option visible in ccmake, and have consistent
CMake variable naming.
Command line options were also extended:

a --dry-run-size or -D option can be used to supple the number
of dry run ranks on the command line.
the miniapp driver was updated to set the dry run size and cell
count via the new global_policy::set_sizes() interface.
 

merge mainline into bcumming fork

Add some finer grained profiling to track exactly what time is spent inside of mpi calls.

Alexander Peyser authored 8 years ago and Sam Yates committed 8 years ago

Add threading pool built on `std::thread`

* Provide new threading model 'cthread' for nestmc based on a pool of `std::thread` objects.
* Unify duplicated timer class provided by `serial`, `omp` and now `cthread` threading models.

Builds and runs on BGQ up 16k nodes

Remove redundant MPI set

* Fix CMakeLists to handle build as a subproject

When several CMake generated projects are build together, it is common
practice to have a 'superproject' CMakeLists that uses
  add_subdir(proj1)
  add_subdir(proj2)
  ...
where each subproject is a self contained CMake based project
(Example proj1=HPX, proj2=nestmc, proj3=another, ...)

CMAKE_SOURCE_DIR always points to the top level directory which
is the superproject dir in this case, whereas PROJECT_SOURCE_DIR
always points to the root of the current project() in the CMakeLists
so one shouod use PROJECT_SOURCE_DIR as this gets the relative paths
correct.

* Add option to turn off auto generation from *.mod files

* Fix #134 : Change CMake WITH_OPTION to NMC_WITH_OPTION, compiler #define to NMC_HAVE_OPTION

1) The user may select an option by saying NMC_WITH_XXX

2) This may trigger CMake to use find_package(...) or setup some
other variables. CMake can then set variable NMC_HAVE_XXX and add a
what has actually been used.

3) Code should use #ifdef NMC_HAVE_XXX to check for a feature

Old CMake/define      New CMake                 Compiler #define
----------------      ---------                 ----------------
THREADING_MODEL       NMC_THREAD_MODEL
    WITH_TBB          NMC_WITH_TBB              NMC_HAVE_TBB
    WITH_OMP          NMC_WITH_OMP              NMC_HAVE_OMP
    WITH_SERIAL       NMC_WITH_SERiAL           NMC_HAVE_SERIAL

WITH_MPI              NMC_WITH_MPI              NMC_HAVE_MPI
WITH_CUDA             NMC_WITH_CUDA             NMC_HAVE_CUDA
WITH_GPU                                        NMC_HAVE_GPU
WITH_ASSERTIONS       NMC_WITH_ASSERTIONS       NMC_HAVE_ASSERTIONS
WITH_TRACE            NMC_WITH_TRACE            NMC_HAVE_TRACE
WITH_PROFILING        NMC_WITH_PROFILING        NMC_HAVE_PROFILING

Other user visible CMake vars
-----------------------------
VECTORIZE_TARGET            -> NMC_VECTORIZE_TARGET
USE_OPTIIZED_KERNELS        -> NMC_USE_OPTIIZED_KERNELS
BUILD_VALIDATION_DATA       -> NMC_BUILD_VALIDATION_DATA
BUILD_JULIA_VALIDATION_DATA -> NMC_BUILD_JULIA_VALIDATION_DATA
BUILD_NRN_VALIDATION_DATA   -> NMC_BUILD_NRN_VALIDATION_DATA
VALIDATION_DATA_DIR         -> NMC_VALIDATION_DATA_DIR

Variables such as NMC_THREADING_MODEL and NMC_VECTORIZE_TARGET now use
enumerated cmake values so you can toggle between them in ccmake gui.
SYSTEM_TYPE_CRAY/BGQ        -> NMC_SYSTEM_TYPE (Generic/Cray/BGQ)

* Use generator expression for modcc path

Some IDE's (like Xcode for example), override the CMake binary paths
and add /Debug or /Release etc so rules that have hard coded paths
to binaries will fail.