1. All s-exp parsing has migrated to `arborio`
2. CV policies can now be read from string as per #1334. This is needed for the GUI project.
2. No longer can `arb::locset` and `arb::region` be constructed from mere strings
2. We have `arborio::literals` to construct `region`s, `locset`s, and `cv_policy`s in the form of `_rg`, `_ls`, and `_cvp`.
3. The `_lab` suffix now constructs an object that converts to `named` morphologies, rather than a simple string.
4. Constructors of `region` and `locset` are now properly guarded
5. Consequently, a lot of missing inheritance from `region_tag` and `locset_tag` was retrofitted.

Closes #1334 
Closes #1419 

New structs and types:
* `cell_tag_type` (std::string): for labelling placeable items on a cell. The label refers to a number of items placed on a locset, equal to the number of locations in a locset. The number of locations in not always known to the user, so the previous way of using indices for items was no longer sufficient. 
* `lid_selection_policy`: for allowing a user to select a single item from a group of items sharing a label. Currently only `round_robin` and `assert_univalent` are supported. 
* `cell_local_label_type` and `cell_global_label_type`: for identifying the target and source of a connection or gap_junction connection. 
* `cell_label_ranges`, and `cell_labels_and_gids`: for propagating information about the labelled items on the cell from the cell groups back to the simulation and communicator. 
* `label_resolution_map` and `resolver`: for selecting an item (and retaining state) from a labelled group of items on a cell according to a user-selected policy.

Changes to the model-initialization: 
* The `communicator` now needs `label_resolution_maps` constructed from the cell group data in order to build the `connections` vectors. 
* The `simulation_state` object handles the transfer of the information when it is constructed. 
* Spike exchange at runtime remains unchanged, because `communicator::connections` remains unchanged. 

Changes to cells, cell_groups and recipe:
* `decor::place` expects a third label parameter, no longer returns an `lid_range`.
* `lif`, `source`, and `benchmark` cells need source/target labels in their constructors. 
* A `cell_group` needs to save data about the gid/labels/lid_ranges of each cell, to propagate back to the `communicator` constructor. 
* Connections/gap junction connections are formed between {label, policy} pairs on cells instead of indices. 
* `num_sources`, `num_targets`, `num_gap_junction_sites` deleted from `recipe`.

Additional changes:
* Add MPI wrapper for exchanging vectors of strings. 
* Corresponding updates to unit tests, Python wrapper, C++ and Python examples, documentation.

Fixes #1394

* Raise KeyError in catalogue.
* Raise KeyError in label_dict.

* `i_clamp` constructors take new optional phase argument.
* Pulse or box envelope clamps are now provided by the static method `i_clamp::box`, taking onset, duration, and amplitude parameters.
* Bump pybind11 version so we can have keyword-only arguments in the Python API.
* Make frequency and phase keyword-only arguments to Python `iclamp`.
* Change frequency units for clamps from Hz to kHz for consistency with other interfaces.

Fixes #1454.

* Change implementation of current clamp from a builtin mechanism to
  functionality provided by the back-end shared state classes.
* Extend clamp functionality to cover more use cases: it now supports DC
  and AC stimuli governed by a piecewise-linear envelope.
* Update Python iclamp wrapper to suit; add unit test.
* Account for total applied stimulus currents separately, and remove
  their contribution to the reported transmembrane current and
  ionic (i.e. non-capacitive) transmembrane current.
* Add new probe specifically for tracking total current stimulus in a cell.
* Update docs, and add an entry for current clamps to the concepts doc.
* Remove entirely the builtin catalogue.

Update `set_ion` method of `cable_global_properties` to allow the addition of new ions

* Remove redundant `initial_ion_data` specification.

* add neuroML example
* no test, because GH actions arbor is built without neuroML support

* Move contents of `arbornml` to to `arborio` 
* `arborio` will be built with NeuroML2 morphology support if `ARB_CMAKE_NEUROML` is set. 
* Wrap `arborio::neuroml` and `arborio::morphology_data` for pyarb.
* Add documentation.
* add `neuroml` flag to `setup.py`

Fixes #1256
Fixes #1234

Extend Ptyhon recipe wrapper to support setting of global cell properties.

* Rename `pyrecipe::get_probes` to `py_recipe::probes`
* Remove `global_properties_shim` : It's easy to make mistakes with the class, as it holds a `mechanism_catalogue` and a `cable_cell_global_properties` which holds a pointer to the catalogue. This would likely have caused issues with the users.
* Expose `py_recipe::global_properties`

Make the `cable_cell` interface read only by passing the decorations
to its constructor in a new `decor` type.

C++ library
* Remove the paint/place/set_default interface from `cable_cell`
* Create a `decor` type that:
  * stores lists of paintings and placings
  * stores a set of cable cell parameters
  * uses the `paint`, `place`, `set_default` interface that was deprecated
    from `cable_cell`.
* Create `paintable`, `placeable` and `defaultable` variants that are sum types
  over the respective types that can be painted, placed and defaulted.
* Remove the overloaded `cable_cell::paint`, `cable_cell::place`,
  `cable_cell::set_default` methods to single methods that consume the sum types.

Unit Tests
* Many small changes because many many tests use cable cell API.
* There were no `cable_cell` unit tests! Not such a big deal, since cable_cell
  is tested implicitly in so many other tests
  * but I added cable_cell tests: not much at the moment but will quickly fill
    up as we work on reading, writing, etc.                                                            

Python
* Wrap the new decor type.
* Update cable cell type to reflect read only API.
* Wrap the cv_policy types and implement the same operator overloading.

Fix a latent bug in the Python wrapper that was triggered in pybind11 v2.6.1

These changes ensure that the GIL is acquired before calling functions in C++ that
may consume a Python object with reference count 1 as an argument, in which case
the object's destructor is called at the end of the function, after any GIL acquired inside
the function would have been released.

Fixes #1271.

Major changes in Python library and documentation:

* Add global state (immutable after initialization) for the Python module that
  manages the mapping between probe metadata types and the method by which the
  corresponding [sample data can be collected and presented as Python objects.
  This is necessary for decoupling the implementation of the simulation wrapper
  from that of the various cable cell probe types.

* Wrap each of the C++ cable cell probe types with a Python function that
  returns a `probe_info` object.

* Add code for converting and accumulating C++ probe metadata and sample
  data, registered in the module global state against the metadata type info.

* Make the `arbor.simulator` object responsible for recording all spike and
  sample data, with corresponding new methods.

* Use NumPy arrays and structured datatypes for returning spike and
  sample data.

* Place Python schedule proxies under an abstract interface so that consumers
  of the proxies can be made generic over schedule types.

* Add unit tests for the `arbor.simulator` class and its new functionality,
  including distributed spike collection.

* Rework all Python API documentation concerning probes, sampling, and spike
  recording; add information on newly exposed cable cell probe addresses.

* Add new python example `single_cell_recipe.py` which is a generic recipe
  version of `single_cell_model.py`.

* Adjust other code in the wrapper and examples to accommodate these changes.

Minor changes in Python library:

* Add equality tests for `arbor.location` and `arbor.cable` Python classes.

* Use the `py::` namespace shorthand more often in the Python wrapping sources.

* Add an implicit conversion for a pair of numbers to `arbor.cell_member`, so
  that e.g. probe ids can be specified simply as `(gid, index)`.

* Add an implicit conversion from a tuple to `arbor.mpoint` so that a
  morphological point can be specified simply as `(x, y, z, radius)`.

* Modify the interface to `arbor.regular_schedule` so that the overloads
  correspond more closely to the C++ API. In particular,
  `arbor.regular_schedule(dt)` now describes a schedule with times 0, dt, 2*dt,
  etc.

Minor changes in C++ library:

* Change test in FVM lowered cell implementation and exception message for
  `bad_source_description`: assert number of detectors is exactly the number
  of sources promised by the recipe.

* Add comment describing `probe_metadata` object in `sampling.hpp` which
  asserts that the type and value of probe-specific metadata completely
  determines the correct interpretation of sampled data provided to a callback.

* Substitute `std::optional<T>` for `arb::util::optional<T>` for non-reference types T.
* Remove reference-deducing `util::value_by_key`; replace usages with new function `util::ptr_by_key`.
* Add some missing header includes that were required but included only transitively.
* Remove `operator<<` overload for optional in python/strprintf.hpp; replace with utility wrapper class that catches `std::optional<T>`.
* Wrap some `std::optional` values with `to_string` in ostream output in python lib.

Fixes #1154.

* Move label and s-expr parsing code and unit tests from Python to arbor C++ library.
* Update `locset` and `region` constructors that take string arguments to parse strings as s-expressions or "quoted" labels.
* Modify the input stream modifier used to parse asc files to use a lookup table for substitutions, renamed it transmogrifier.                                                             
* Replace `hopefully` type implemented in python headers with an `arb::util::expected`.
* Add `ARBDEV_COLOR` CMake option that forces gcc and clang to always output color output.
* Allow arbitrary strings in labels in region and locset expressions.
* Add `parse_region_expression` and `parse_locset_expression` functions alongside the existing `parse_label_expression` function for use when a region or locset is expected. These calls will promote a quoted string `"label"` to `(region "label")` or 
 `(locset "label")` respectively.
* Ad...

* Replace usages of `util::any` with `std::any`.
* Modify `util::any_cast` so that it forwards `std::any` arguments to `std::any_cast`; move into own header.
* Prevent `std::unique_any` from implicitly encapsulating `std::any` objects.
* Replace `test_any.cpp` unit tests with `test_any_cast.cpp` unit tests.
* Replace some usages of `std::is_same<X, Y>::value` with `std::is_same_v<X, Y>` as part of general C++17ification.
* Remove `util/variant.hpp`, `test_variant.cpp`.
* Replace occurrences of `util::variant` with `std::variant`.
* Replace `util::variant::get<N>` with `std::get<N>`.
* Replace `util::get_if<N>` with `std::get_if<N>`, passing a pointer to variant instead of a ref or value.
* Add some standard #includes where they were required in the source, but only included transitively.
* Some minor comment tidying and fix-ups.
* Update C++ API docs.

Fixes #1144.

* Make the members of `cable_cell_ion_data`  optional.
* Add separate methods for painting `init_int_concentration` , `init_ext_concentration` and `init_reversal_potential` on the cell.

* Remove spherical root/soma from morphology representations and logic.
* Replace `sample_tree` with `segment_tree`, which represents sample points in pairs and allows the representation of detached branches and cable segments.
* Remove sample point properties and predicates.
* Remove `cv_policy_flag::single_root_cv` as somata with a single attached dendrite will have a representation in the morphology of a single branch.
* Remove `arb::math::{volume/area}_sphere`, which are no longer being used.
* Remove spherical root special cases from `embed_pwlin` and `place_pwlin` objects.
* Add textual representation of `mnpos` in morphology exception text.
* Rename `sample_tree_from_swc` to `segement_tree_from_swc` and remove any special casing for one point somata.
* Remove `msample` and `mbranch` types.
* Update unit tests and python wrapper to reflect the changes.

Update core library:
* support derivation of named mechanisms in catalogue from a string description of the parent.
* fix bug in radois_ge region expression that was finding region_lt.
* fix test for radius_ge region expression

Add support for mechanism, global properties and catalogue shenanigans to python wrapper

Add a `restrict` locset expression type that returns all locations in a locset that are in a specified region.

Also allow region and locset names with hyphens.

Fixes #1031 
Fixes #1032 

This PR removes the requirement that all cover points are included in a region.
The motivation is to allow more flexible definion of regions, particularly the proximal and distal sets thereof.
The other motivation is that the author finds it much simpler to reason about, however others find the existing approach easier to reason about.

The changes:
 * `mextent` does not always include points on the cover at fork points.
   * it still enforces that its constituent cables have no intersections and are ordered.
 * a `super` region expression adds the cover points to a region.
 * update `most_proximal` to return the `minset` of the proximal points in a region's cables.
 * fix some cut and paste errors in comments and printing of locset expressions.

Add `cable_cell::concrete_region(region)` method that returns a the cable segments in a region.

This is required by the Arbor GUI #953 so that it can illustrate regions (and locsets) defined by users on a concrete cell morphology.

Note that the `concrete_region` and `concrete_locset` interfaces will need to be extended with random number seed information when we add full support for that, but shouldn't need to be replaced or changed otherwise.

Fixes #960.

Use locset names to set probe sites in single cell model.

1. Extend C++ cable_cell interface to provide a concrete list of locations associated with a locset on the cell.
2. Extend Python single_cell_model to allow placement of probes on locsets.
3. Remove "debug" output in Python ring example.

The single cell model maintains a trace callback for each probe location, and needs to know the explicit locations before starting the simulation. The first step is required to enumerate and record the locations associated with a locset when attaching probes.

Open issues: 

1. The extension to `cable_cell` won't sufficiently support locsets that rely on random number generators with global seeds (it would require passing additional meta data required to fully instantiate the locset)
2. Enhancing the probe recipe interface to be able to associate one `probe_id` with all locations in a locset (the number of which is not specified at the time the `probe_id` is created) would support the interface implemented for the `single_cell_model`.

Note: The richer interface on the C++ side will be implemented in time (see #957).

A big update that wraps over 90% of the public C++ API, with enough functionality to let Python users to perform useful modelling.

Key features
- wrapping of cable cell functionality
  - user-defined explicit compartmentalisation not supported
- `single_cell_model` abstraction unique to the Python wrapper that simplifies developing and testing single cell models
- one-dimensional cell builder for Python wrapper that simplfies building cells that 
  - in and of itself limited in scope, but a very useful example of mapping a richer morphology builder onto `sample_tree`s.
- parsing of region and location expressions from strings
  - implements a generic s-expression parser that we can use for other purposes later

Replace branch/cable-based cell representation and discretization with
free specification of control volumes over the cell morphology.

Implements functionality required for issues #859 and #732, and the
reworking of inter-CV conductance computation addresses issue #635,
although there is possible room for improvement by treating CV voltages
as averages rather than point estimates.

* Remove arbor/segment.hpp and segment-based representation in `cable_cell`.
* Add a new discretization policy `cv_policy_every_sample` that replaces
  the `compartments_from_discretization` flag in the `cable_cell`
  constructor.
* Remove use of `cable_cell::num_compartments()` in example code (see
  issue #940).
* Add command line argument to `single_cell` example to specify number
  of CVs per branch.
* Replace fvm layout `fvm_discretization` with new representation
  `fvm_cv_discretization`, which in turn wraps a `cv_geometry` object
  representing CVs as cables within cells in a cell group.
* Provide `cv_geometry_from_ends` that produces `cv_geometry`
  discretization for a cell given a locset comprising CV boundary points.
* Provide interface and functionality for merging discretization
  structures `cv_geometry`, `fvm_cv_discretization`, `fvm_mechanism_data`
  from multiple cells, in order to support (in the future) parallelized
  discretization across a cell group.
* Provide complete reimplementation of `fvm_discretize(...)`.
* Remove now redundant `arbor/fvm_compartment.hpp`.
* Add new locset expression `on_branches`, representing a point on every
  branch at a given relative position.
* Fix implementation bug in `pw_elements` `zip` function.
* Add `util::equal` range-based analogue of `std::equal`.
* Remove the compartments_from_discretization argument from cable cell
  made in python module. (Does this need to begiven a discretization policy?
  It is not clear.)

* Make access to `segment`s in a `cable_cell` read only
* Change cell constructor to require morphology and label dictionary; `make_cable_cell` no longer required.
* Consolidate paint/place functionality on `cable_cell`.
* Add simple cell builder wrapping new API for use in unit tests.
* Remove validation test code: appropriate validation tests will need to be implemented in nsuite.

Fixes #871 

WIP on implementation of new cable cell API.

* Hide implementation and state using PIMPL
* Add consistent `place` methods for adding synapse, stimuls, gap junction site and spike detectors to `cable_cell`.
* Add default constructor for `region` and `locset` that create empty `nil` instances.

* Collect cable cell parameter setting structures into `include/cable_cell_param.hpp`.
* Restructure electrical specifications (axial resistance, membrane capacitance) and ionic concentrations and reversal potentials on cable cells, so that these can be specified with a global default, per-cell defaults, and per-segment values.
* Allow reversal potentials to be set by a mechanism of a new kind 'revpot', which are prohibited from maintaining state or writing to any shared state other than ionic reversal potentials.
* Specify reversal potential mechanisms as global defaults or per-cell. Reversal potential mechanisms may not be specified at the level of a segment in order to avoid non-linearities arising from the discretization.
* Supply default cable cell parameter data that matches NEURON values (this is _not_ used by default).
* Replace the d_lambda calculation with one that approximates more faithfully the effect of tapered segments, and which will ...

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Add docs for Python wrapper to ReadTheDocs:

- Overview, Common Types, Hardware Management, Recipes, Domain Decomposition, Simulations, Metering
- Installing Arbor: Optional Requirements (Python), Buidling and Installing (Python Frontend), and Installation (Python Module)

Missing (, since not implemented yet): 

- probes
- arbor-sup 
- hint maps in domain_decomposition
- reset, events, empty schedule in event_generator
Also does not cover unit testing (since doc is user-centric).

Makes also defaults and units in wrapper consistent.

Fixes  #766

Support for recording spikes generated by a simulation in the Python wrapper
* Implement a `spike_recorder` that holds a shared pointer to a `std::vector` of spikes, and a callback for the `arb::simulation` spike recording API.
* Add `python/example/ring.py` that creates a ring network, then records and prints spikes.
* Some fixes to get the full `recipe` -> `domain_decomposition` -> `simulation` -> `spikes` workflow to work
  * always use default `global_parameters`: user customization of global parameters for cable cells can wait until the ion species interface is finished.
  * change the Python recipe interface for `recipe::connections_on` to use `pybind11::objects` because of shim.
* Some small improvements to error and help messages.

Fixes #764 

Add full wrappers for

- `lif_cell`
- `spike_source_cell`
- `benchmark_cell

And an opaque wrapper for cable_cell with a helper function that generates a random morphology identical to the ones in the nsuite benchmarks.

Fixes #767.