This PR adds a bunch of high-level docs for concepts related to single cell models.
  * morphologies
  * labels (region and locset expressions)
  * mechanisms
  * cable cells

It adds some python scripts to the `doc` path, which are called during documentation building to generate images of cell morphologies, regions and locsets that are generated in a separate script that uses the Arbor Python API.

The docs are not complete: the cable cell and mechanism docs are unfinished, and the tutorial section needs to build on these concepts to give more detailed step by step examples.

New PR post master rollback; squashed and rebased, but reprises #1111.

* Add (forward) ordered forest implementation, tests.
* Add `segment` region expression; to ease implementation, `msegment` now also knows its own id.
* Add `stitch_builder` and `stitched_morphology`. A stitch corresponds to a labelled, linearly-interpolated segment which can be attached at any point along a parent stitch. A `stitched_morphology` takes a `stitch_builder` object and constructs a segment tree and morphology, and provides a dictionary of stitch labels to segment indices and region expressions.
* Add `import` method for `label_dict`, so that the label dictionary returned by `stitched_morphology` can be merged with an existing dictionary.
* Add section on stitch builder etc. to cable cell docs.
* Update cable cell docs to remove out of date info and to provide some context.
* Describe ordered forest datastructure and interface in a long comment at the beginning of the header file.

Update the docs landing page and add pip install from git.

Fixes #1118

Guide Python users to the Python installation steps early in the installation guide.

* Move `cv_policy` classes to new source files.
* Remove `cv_policy_flag::single_root_cv`.
* Allow all supplied `cv_policy` makers to be restricted a domain specified by a region: the policy is then applied per component, with the boundary points of the completion of the components included.
* Expose `cv_policy` domain in public interface.
* Add `components` function for splitting an extent into disconnected components. Add comment describing what is meant by connected in this context.
* Rename `super` region function to `complete`.
* Add region expression functions `complement` and `difference`.
* Add new locset expression function `boundary` which returns the extrmal points of each component of a region.
* Add new locset expression `cboundary` which returns the extremal points of the completion of each component of the region, which is what we need to close off CVs generated by region-restricted CV policies.
* Add locset expression `support` which returns the multiset support of the points in a locset, i.e. all points in the locset but with multiplicity one.
* Replace hodgepodge of `join` expressions in `cv_policy` implementations with locset `support` and `sum`.
* Inline `mcable::prox_loc` and `mcable::dist_loc`.
* Add cv policy combinators +_and |.
* Add `util::foldl` which acts like `std::accumulate` for ranges, except it explicitly allows mutation of the sequence, and pass the accumulation value by rvalue.
* Add unit tests for mextent components, new locset expressions.
* Remove spherical soma cases from `cv_policy` testing.
* Extent `cv_policy` unit tests to cover new functionality.
* Rejig `cv_policy` unit tests to use the morphology testing predicates.
* Add `cv_policy` documentation to cable cell docs.

Fixes #1058.

* Make recipe return probes as a vector.
* Remove `probe_id` from `probe_info`.
* Rename `fvm_probe_info` to `fvm_probe_data` (everything was being called info, and it was getting confusing).
* Make `probe_association_map` specific to `mc_cell_group`/`fvm_lowered_cell`.
* Change probe_association_map representation: unordered map for probe_id -> tag; unordered multimap for probe_id -> fvm_probe_data. This allows multiple probes to be associated with the same probe id.
* Call sampler callback separately for each probe with the same probe_id.
* Replace location-based probes with locset equivalents.
* Add index for probes sharing a probe id.
* Bundle all probe metadata (id, tag, index, probe-specific meta) into `probe_metadata` struct for passing to sampler callbacks.
* Change simple_sampler to work on `trace_vector`, a vector of `trace_data`. The _i_th element is the data from probe with index i.
* Consolidate hash composition and `std::hash` specialization code in new header.
* Update python lib for new API.
* Update tests and examples for new recipe, internal probe, and simple_sampler APIs.
* Update docs to suit.

Fixes #1037. 

* Add fake unmatchable spike events to event queue for each exact sample request, so that integration boundaries align with exact sample times.
* Update probe-demo for exact sampling.
* Add copy assignment for `cable_cell`.
* Extend max args for ARB_PP_FOREACH to 20.
* Add unit test for ARB_PP_FOREACH.
* Use ARB_PP_FOREACH to instantiate backend-specific probe unit tests. This also fixes the GPU probe tests that were using multicore backend.
* Add unit test that exercises exact sampling requests combined with regular events in a gap junction context.

Fixes #589 and #730, providing cell-wide probes and a correct total trans-membrane current.
Fixes #822, providing spatial interpolation of membrane voltages as governed by cable resistivity and diameter.

* Add partition of point mechanism target vector by cell index to `fvm_mechanism_config`.
* Fix sign error in axial current interpolation.
* Replace `fvm_probe_info` with class that wraps a variant type capturing one of a set of backend-cellgroup probe translation representations: `fvm_probe_scalar`, `fvm_probe_interpolated`, `fvm_probe_multi`, `fvm_probe_weighted_multi`, `fvm_probe_membrance_currents`.
* Refactor `fvm_lowered_cell_impl::resolve_probe_address` so that it avoids the long chain of if-else tests on wrapped `any` type with an invocation of `util::any_visitor`. Each specific cable cell probe address representation is translated into one of the intermediate `fvm_probe_info` variants via an overload of `resolve_probe`.
* Add new non-scalar cell probe address types for all-of-cell values, including point mechanism state, density mechanism state, ion currents and concentrations, membrane voltage by CV cable, and total trans-mebrane currents. The latter are ultimately calculated by computing the net axial current fluxes in each CV.
* Add `util::any_ptr` field to the sampling function interface, pointing to constant-per-simulation-run metadata associated with a probe address. Underlying metadata type varies between probe types.
* Extend `simple_sampler` so that it optionally captures sampler metadata.
* Add special case to `simple_sampler` for vector-valued sample data, allowing it to translate from a pointer-pair range to a std::vector.
* Fix minor bugs in `util::variant`.
* Add to `mc_cell_group` the logic for translating each intermediate probe representation to a sampler callback invocation, reusing temporary scratch vectors across calls.
* Add unit comments to `embed_pwlin_data`.
* Add probe variety demonstration `probe-demo` to examples, covering most of the new cable cell probe classes.
* Add `util::any_visitor` runtime-type dependent dispatcher class.
* Add `util::overload` helper for generating overloaded functional objects from individual functions, lambdas or functionals.
* Extend probe unit tests in `test_probe.cpp` to cover new probe address types.
* Add unit tests for `any_visitor` and `overload`.

- Add option to compile Arbor for HIP/CUDA backend using Clang. 
- Add new CMake option `ARB_GPU_COMPILE_TYPE` to distinguish between three possible gpu platforms/builds: `cuda-nvcc`; `cuda-clang`; `hip-clang`
- Add gpu wrapper functions in `arbor/backends/gpu/gpu_api.hpp` and `arborenv/gpu_api.hpp` which call HIP/CUDA functions depending on the platform
- Rename functions and files: cuda -> gpu
- Add downgraded warp primitives for HIP
- Implement `uuid` workaround for HIP
- Set correct alignment and warp size for AMD gpus
- Update installation guide and docs.
- Update pip/setuptools to support new gpu targets

Fixes #833

Some small changes to facilitate smooth PyPi integration:
* update the Python installation guide
* update the license file
* add a `setup.cfg`
* add a stub for the Python readme

Add a `setup.py` for installing Arbor directly with pip/setuptools.

Implement a setuptools extension for CMake in `setup.py`. To be honest, I don't understand entirely how it works, but setuptools is obtuse enough that I don't feel at all guilty about this.

Define additional flags for optionally enabling GPUs, MPI, Vectorization and micro-architecture targets, for more adventurous users.

The documentation is updated with a "howto pip" for more casual users who don't want anything to do with CMake.

Fixes #958 .

Improve the Python wrapper generation and installation:
  - install a proper module that can be extended with Python code;
  - give the user more control over where to install the module (e.g. as a user package or in a virtualenv).

During building, the following sub-directory is built in the build director (`CMAKE_BINARY_DIR`)
```
└── python
    └── arbor
            ├── __init__.py
            ├── arbor.so
            └── VERSION
```
This path can then be copied VERBATIM to the target installation path. By default this will be in `CMAKE_INSTALL_PREFIX/lib/python%d.%d/site-packages`.
An additional CMake parameter `ARB_PYTHON_PREFIX` can be used to specify an alternative destination for installing the Python module.

The Python part of the wrapper, implemented in `__init__.py` is currently very limited, only providing `__version__` and `__config__` variables.

The installation guide was updated to cover the Python installation.

A big update that wraps over 90% of the public C++ API, with enough functionality to let Python users to perform useful modelling.

Key features
- wrapping of cable cell functionality
  - user-defined explicit compartmentalisation not supported
- `single_cell_model` abstraction unique to the Python wrapper that simplifies developing and testing single cell models
- one-dimensional cell builder for Python wrapper that simplfies building cells that 
  - in and of itself limited in scope, but a very useful example of mapping a richer morphology builder onto `sample_tree`s.
- parsing of region and location expressions from strings
  - implements a generic s-expression parser that we can use for other purposes later

* Add logo to read the docs as HTML header.
* Lighten nav bar background so as to not clash with logo.

Adds

* to `recipe.cpp`: 
`num_probes` and `get_probe` with helper function `cable_probe(kind, id, location)`
* `sample.cpp`: 
`sample` type with attributes `time` and `value`; a `sampler` with `samples()` and `clear()` and helper functions `attach_sampler(sim, dt)` for adding all probes as well as `attach_sampler(sim, dt, probe_id)` for adding one probe
* `morphology.cpp`: 
`location` as `arb::mlocation` with `branch` and `position` 
* corresponding docs and example
* update to new cable cell semantics in c++ validation test

fixes #762

* Add new "embedded morphology" `em_morphology`, constructed from a `morphology`, that: caches derived information such as terminal points, locations of sample points, branch lengths; provides services for canonicalizing locations and interpolation; and is used for the realisation of regions and locsets (see below).
* Add simple DSL for describing locsets and regions, which are abstract descriptions of sets of locations and regions on morphologies, together with a set of operations.
* Add a new class `label_dict` that associates names with regions and locsets.
* Extend `cable_cell` to hold a `label_dict` argument, which is used by a new method `paint` for associating mechanisms with regions.

Fixes #845.

Sam Yates authored 5 years ago and Benjamin Cumming committed 5 years ago

* Replace derivation of cylindrical cable equation formula with S–L approach.
* Relate analysis of cylindrical form to Rallpack 1 in a separate section.
* Add analysis of tapered cable case.
* Unify discussion of cable with constant electrical properties but varying geometry.
* Add section describing how one can well-estimate the inter-cell flux given point samples or CV means for the membrane voltage.

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

- wraps `partition_hint` struct for python
- `partition_hint`struct in `load_balance.hpp` is adjusted in a way that in case of zero unsigned int `cpu_group_size`/`gpu_group_size` it is set to the default value (via setter/getter)
- adds documentation for `partition_hint`  
- adds testing for `domain_decomposition` including `partition_hint` (in unit and unit_distributed)
- adds `partition_hint` in example `ring.py`
- corrects `config.cpp` (and doc) to test for `ARB_GPU_ENABLED` instead of `ARB_WITH_GPU`

Fixes #776 
Addresses #799, #769

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

* add interface for Python users to query schedule event times
* add schedule event tests and assertions
* add schedule events to doc

* Collect cable cell parameter setting structures into `include/cable_cell_param.hpp`.
* Restructure electrical specifications (axial resistance, membrane capacitance) and ionic concentrations and reversal potentials on cable cells, so that these can be specified with a global default, per-cell defaults, and per-segment values.
* Allow reversal potentials to be set by a mechanism of a new kind 'revpot', which are prohibited from maintaining state or writing to any shared state other than ionic reversal potentials.
* Specify reversal potential mechanisms as global defaults or per-cell. Reversal potential mechanisms may not be specified at the level of a segment in order to avoid non-linearities arising from the discretization.
* Supply default cable cell parameter data that matches NEURON values (this is _not_ used by default).
* Replace the d_lambda calculation with one that approximates more faithfully the effect of tapered segments, and which will use the electrical values inherited by cell or global defaults.
* Supply a bundled mechanism 'nernst' that replaces the previous hard-coded Nernst potential calculation, for use (together with ion rebinding) in the new ion description scheme.
* All ions used in a cable_cell model must be present in the top level global parameter table, together with their charge.
* Extend semantics of external variables in modcc, to permit direct assignment (as opposed to just += or -=.)
* Extend `util::binary_search_index` to allow for a projection functional analagous to other range utilites.
* Add documentation on the cable cell API, sketching over many of the details, but describing in particular the interface changes for default parameters and reversal potential behaviour.
* Adjust pyarb for new API
* Hard code global cable cell properties in the python recipe shim to useneuron default parameters.

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

* correct doc for meter_report and small word fixes
* shift make_report to next section

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Wraps the `meter_manager` with
- the constructor
- `start(ctx)`
- `checkpoint(name, ctx)`
- `checkpoint_names`
- `times`

and the `meter_report` with
- `make_meter_report(meter_manager, ctx)`

Fixes #765

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Add docs for Python wrapper to ReadTheDocs:

- Overview, Common Types, Hardware Management, Recipes, Domain Decomposition, Simulations, Metering
- Installing Arbor: Optional Requirements (Python), Buidling and Installing (Python Frontend), and Installation (Python Module)

Missing (, since not implemented yet): 

- probes
- arbor-sup 
- hint maps in domain_decomposition
- reset, events, empty schedule in event_generator
Also does not cover unit testing (since doc is user-centric).

Makes also defaults and units in wrapper consistent.

Fixes  #766

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Domain decomposition in python wrapper
* functions: `partition_load_balance`
*  types: `group_description` and `domain_decomposition`

Also includes
* remove comment that `gid` in a `group_description` must be sorted.
* a host of small clean ups of python wrapper

Fixes #761

Add citation information to the readme.

Benjamin Cumming authored 6 years ago and Sam Yates committed 6 years ago

* Rename `arb:simd::simd_detail` namespace to `arb::simd::detail`.
* Update SIMD documentation to suit.

Fixes #468.

noraabiakar authored 6 years ago and Benjamin Cumming committed 6 years ago

Add gap junctions and improve general model overview in docs.

* Replace Arbor Model::Common Types with Arbor Model::Concepts, that introduces the concept of cells, connections and gap junctions. 
* Add gap junction documentation to cpp:recipe
* Document the implicit rule forcing cells connected by gap junctions to be in the same cell group.
* Add documentation that discusses core concepts in Arbor models.

Update Hardware API documentation

* split the domain decomposition and hardware API docs into separate pages
* update hardware API to reflect new *libarbor* and *libarborenv*
* add basic documentation for `optional`, `any` and `unique_any` types.

Update the installation guide to reflect the latest supported tool,
compiler and library versions.

Remove the Python docs, because they documented features that have not
been implemented yet. The Python docs can be added incrementally as
features are implemented.

Start work on oupdating the documenetation for hardware interfaces and
domain decomposition.

akuesters authored 6 years ago and Benjamin Cumming committed 6 years ago

Update documentation for Python.

    splits the conceptual model ideas from the C++ docs into their own section
    has C++ and Python docs for recipes, domain decomposition, etc.

fixes #667

Added the following documentation (structure):

GETTING STARTED:

    Installing Arbor/Requirements/Optional Requirements/Python
    Installing Arbor/Building and Installing Arbor/Python Front End

MODEL BASICS:

    Overview
    Common Types
    Recipes
    Domain Decomposition
    Simulations

PYTHON:

    Overview
    Common Types
    Recipes
    Domain Decomposition
    Simulations

DEVELOPERS:

    Python Profiler
    Python Unit Testing

GETTING STARTED has two added sections of optional requirements using python and how to build the python front end.

MODEL BASICS describes Arbor's concepts in general (independent of programming language), thus general information on concepts in C++ API was moved here/ added.

PYTHON describes Arbor's python frontend in the same structure as MODEL BASICS and C++ API ( needs updates as soon as features are added/changed in new python PR).

DEVELOPERS section has two added sections for meter management and unit testing with python front end.

Further, some corrections in existing documentation (for obvious errors, e.g. duplicate text, not ending sentences) and referencing sections were done.

akuesters authored 6 years ago and Benjamin Cumming committed 6 years ago

First step towards the Python front end.

This commit sets up the structure of the python implementation
* directory structure
* git submodule for pybind11
* best practices for making bindings with pybind11
* unit testing for the python front end

It implements the following features in the Python front end
* execution contexts
* gpu detection
* thread count detection
* MPI initialization helpers.

Fixes #667.

noraabiakar authored 6 years ago and Benjamin Cumming committed 6 years ago

Updates the install docs. Fixes #604

akuesters authored 6 years ago and Benjamin Cumming committed 6 years ago

changes: 
- .travis.yml:
  - added matrix for different osx's, since enumeration style only works for `env` and `compiler`

- scripts/travis/build.sh:
  - changed getting compiler version from ``${CXX} -dumpversion`` to ``${CXX} --version | grep -m1 ""`` 
  - added `--oversubscribe` flag to `mpiexec` on Mac to allow more processes on a node than processing elements
  - added `--mca btl tcp,self` flag for Open MPI to use the "tcp" and "self" BTLs for transporting MPI messages on Mac

noraabiakar authored 6 years ago and Benjamin Cumming committed 6 years ago

Dry-run mode: 
* An implementation of distributed_context that is used to mimic the performance of running an MPI distributed simulation with n ranks.
* Verifiable against an MPI run with the same parameters. 

Implementation: 
* Describe the model on a single domain (tile) and translate it to however many domains we want to mimic using arb::tile and arb::symmetric_recipe. This allows us to know the exact behavior of the entire system by only running the simulation on a single node.
* Mimic communication between domains using arb::dry_run_context

Example: 
* dryrun in example/ is a verifiable example of using dry-run mode with mc_cells

Other:
* Documentation of dry-run mode 
* unit test for dry_run_context 

Sam Yates authored 6 years ago and Benjamin Cumming committed 6 years ago

Fixes #587.

* Eliminate Clang warnings from GCC-tree-optimization bug work-around.
* Error with static-assert if simd type is used with a missing simd abi.
* Clarify install documentation regarding use of ARB_VECTORIZE with ARB_ARCH.

Make the execution context presented to users an opaque handle, moving all implementation of the gpu, thread and distributed contexts into the back end.

* move `execution_context` and `distributed_context` definitions to the back end
* create `execution_context` handle called `context` in the public API
* provide `make_context` helper functions that build different context configurations (default, user-specified local resources, with MPI)
* update documentation for all parts of the public API that touch contexts
* move `distributed_context` docs to the developer documentation (from the public API docs)

Benjamin Cumming authored 6 years ago and Sam Yates committed 6 years ago

Fixes #564

Sam Yates authored 6 years ago and Benjamin Cumming committed 6 years ago

* Update the install docs for architecture build options
* Update to reflect new install target