Remove `gid` attribute of target of a connection, and local gap-junction site of a gap-junction connection (#1467)

* Change the type of the target of a `connection` from `cell_member_type` to `cell_lid_type`. The target cell's gid is always known in both the recipe and simulation.
* Change the type of the local site of a `gap_junction_connection` from `cell_member_type` to `cell_lid_type`, and make it such that the first argument of the connection is the peer site, and the second the local site. The local cell's gid is always known in both the recipe and simulation.
* Change the type of the target of an `event_generator` from `cell_member_type` to `cell_lid_type`. The target cell's gid is always known in both the recipe and simulation.
* New `cell_spike_events` and `cse_vector` for `simulation::inject_events`.
* Simplify recipe sanity checks and remove associated exception types. 
* Fix unit tests, examples, docs

Extend Ptyhon recipe wrapper to support setting of global cell properties.

* Rename `pyrecipe::get_probes` to `py_recipe::probes`
* Remove `global_properties_shim` : It's easy to make mistakes with the class, as it holds a `mechanism_catalogue` and a `cable_cell_global_properties` which holds a pointer to the catalogue. This would likely have caused issues with the users.
* Expose `py_recipe::global_properties`

Fix a latent bug in the Python wrapper that was triggered in pybind11 v2.6.1

These changes ensure that the GIL is acquired before calling functions in C++ that
may consume a Python object with reference count 1 as an argument, in which case
the object's destructor is called at the end of the function, after any GIL acquired inside
the function would have been released.

Fixes #1271.

Major changes in Python library and documentation:

* Add global state (immutable after initialization) for the Python module that
  manages the mapping between probe metadata types and the method by which the
  corresponding [sample data can be collected and presented as Python objects.
  This is necessary for decoupling the implementation of the simulation wrapper
  from that of the various cable cell probe types.

* Wrap each of the C++ cable cell probe types with a Python function that
  returns a `probe_info` object.

* Add code for converting and accumulating C++ probe metadata and sample
  data, registered in the module global state against the metadata type info.

* Make the `arbor.simulator` object responsible for recording all spike and
  sample data, with corresponding new methods.

* Use NumPy arrays and structured datatypes for returning spike and
  sample data.

* Place Python schedule proxies under an abstract interface so that consumers
  of the proxies can be made generic over schedule types.

* Add unit tests for the `arbor.simulator` class and its new functionality,
  including distributed spike collection.

* Rework all Python API documentation concerning probes, sampling, and spike
  recording; add information on newly exposed cable cell probe addresses.

* Add new python example `single_cell_recipe.py` which is a generic recipe
  version of `single_cell_model.py`.

* Adjust other code in the wrapper and examples to accommodate these changes.

Minor changes in Python library:

* Add equality tests for `arbor.location` and `arbor.cable` Python classes.

* Use the `py::` namespace shorthand more often in the Python wrapping sources.

* Add an implicit conversion for a pair of numbers to `arbor.cell_member`, so
  that e.g. probe ids can be specified simply as `(gid, index)`.

* Add an implicit conversion from a tuple to `arbor.mpoint` so that a
  morphological point can be specified simply as `(x, y, z, radius)`.

* Modify the interface to `arbor.regular_schedule` so that the overloads
  correspond more closely to the C++ API. In particular,
  `arbor.regular_schedule(dt)` now describes a schedule with times 0, dt, 2*dt,
  etc.

Minor changes in C++ library:

* Change test in FVM lowered cell implementation and exception message for
  `bad_source_description`: assert number of detectors is exactly the number
  of sources promised by the recipe.

* Add comment describing `probe_metadata` object in `sampling.hpp` which
  asserts that the type and value of probe-specific metadata completely
  determines the correct interpretation of sampled data provided to a callback.

* Update `dryrun` to generate some spikes again, add a voltage printer.
* Fix `ring.py` recipe.
* Fix labels in `single_cell_builder.py` and `single_cell_swc.py`.

* Make recipe return probes as a vector.
* Remove `probe_id` from `probe_info`.
* Rename `fvm_probe_info` to `fvm_probe_data` (everything was being called info, and it was getting confusing).
* Make `probe_association_map` specific to `mc_cell_group`/`fvm_lowered_cell`.
* Change probe_association_map representation: unordered map for probe_id -> tag; unordered multimap for probe_id -> fvm_probe_data. This allows multiple probes to be associated with the same probe id.
* Call sampler callback separately for each probe with the same probe_id.
* Replace location-based probes with locset equivalents.
* Add index for probes sharing a probe id.
* Bundle all probe metadata (id, tag, index, probe-specific meta) into `probe_metadata` struct for passing to sampler callbacks.
* Change simple_sampler to work on `trace_vector`, a vector of `trace_data`. The _i_th element is the data from probe with index i.
* Consolidate hash composition and `std::hash` specialization code in new header.
* Update python lib for new API.
* Update tests and examples for new recipe, internal probe, and simple_sampler APIs.
* Update docs to suit.

Fixes #589 and #730, providing cell-wide probes and a correct total trans-membrane current.
Fixes #822, providing spatial interpolation of membrane voltages as governed by cable resistivity and diameter.

* Add partition of point mechanism target vector by cell index to `fvm_mechanism_config`.
* Fix sign error in axial current interpolation.
* Replace `fvm_probe_info` with class that wraps a variant type capturing one of a set of backend-cellgroup probe translation representations: `fvm_probe_scalar`, `fvm_probe_interpolated`, `fvm_probe_multi`, `fvm_probe_weighted_multi`, `fvm_probe_membrance_currents`.
* Refactor `fvm_lowered_cell_impl::resolve_probe_address` so that it avoids the long chain of if-else tests on wrapped `any` type with an invocation of `util::any_visitor`. Each specific cable cell probe address representation is translated into one of the intermediate `fvm_probe_info` variants via an overload of `resolve_probe`.
* Add new non-scalar cell probe address types for all-of-cell values, including point mechanism state, density mechanism state, ion currents and concentrations, membrane voltage by CV cable, and total trans-mebrane currents. The latter are ultimately calculated by computing the net axial current fluxes in each CV.
* Add `util::any_ptr` field to the sampling function interface, pointing to constant-per-simulation-run metadata associated with a probe address. Underlying metadata type varies between probe types.
* Extend `simple_sampler` so that it optionally captures sampler metadata.
* Add special case to `simple_sampler` for vector-valued sample data, allowing it to translate from a pointer-pair range to a std::vector.
* Fix minor bugs in `util::variant`.
* Add to `mc_cell_group` the logic for translating each intermediate probe representation to a sampler callback invocation, reusing temporary scratch vectors across calls.
* Add unit comments to `embed_pwlin_data`.
* Add probe variety demonstration `probe-demo` to examples, covering most of the new cable cell probe classes.
* Add `util::any_visitor` runtime-type dependent dispatcher class.
* Add `util::overload` helper for generating overloaded functional objects from individual functions, lambdas or functionals.
* Extend probe unit tests in `test_probe.cpp` to cover new probe address types.
* Add unit tests for `any_visitor` and `overload`.

* Extend repertoire of cable cell probes.
* Add the following scalar (i.e. corresponding to just one raw value)
probe address types for cable cells:
   - `cell_probe_membrane_voltage`, replacing `cell_probe_address::membrane_voltage`;
   - `cell_probe_total_ionic_current_density`, replaing `cell_probe_address::membrane_current`;
   - `cell_probe_density_state`;
   - `cell_probe_point_state`;
   - `cell_probe_ion_current_density`;
   - `cell_prone_ion_int_concentration`;
   - `cell_prone_ion_ext_concentration`.
* Extend backend class interface with `mechanism_field_data` method, that returns a raw (device) pointer to the per-instance values of a mechanism state variable.
* Make unit test for probes backend-generic.
* Add unit tests for new probes.
* Modify `fixed_ica_current.mod` unit test NMODL module to be more generic in naming, so it can be better used in probe unit tests.
* Update examples for new cable cell probes.
* Update Python interface for new probe types

Fix warning generated when `delay` and `weight` were accidentally swapped in #984 
Match `cell_connection` delay type to `connection` delay type

A big update that wraps over 90% of the public C++ API, with enough functionality to let Python users to perform useful modelling.

Key features
- wrapping of cable cell functionality
  - user-defined explicit compartmentalisation not supported
- `single_cell_model` abstraction unique to the Python wrapper that simplifies developing and testing single cell models
- one-dimensional cell builder for Python wrapper that simplfies building cells that 
  - in and of itself limited in scope, but a very useful example of mapping a richer morphology builder onto `sample_tree`s.
- parsing of region and location expressions from strings
  - implements a generic s-expression parser that we can use for other purposes later

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Ensure that errors in Python callbacks that are called from multithreaded C++ code propogate the correct Python error back to the parent Python callback site, and that no callbacks are called from other threads if an error has already ocurred.
- protects each recipe callback with a mutex, stores python exception, catches and throws python exception if occured
- methods calling recipe (in simulation and partition_load_balance) are protected as well by try catch, resets and rethrows python exception (if occured) or else throws C++ exception

fixes #792

Adds

* to `recipe.cpp`: 
`num_probes` and `get_probe` with helper function `cable_probe(kind, id, location)`
* `sample.cpp`: 
`sample` type with attributes `time` and `value`; a `sampler` with `samples()` and `clear()` and helper functions `attach_sampler(sim, dt)` for adding all probes as well as `attach_sampler(sim, dt, probe_id)` for adding one probe
* `morphology.cpp`: 
`location` as `arb::mlocation` with `branch` and `position` 
* corresponding docs and example
* update to new cable cell semantics in c++ validation test

fixes #762

Remove the dead `cell_connection_shim` from the python wrapper.

Fix the connection printer to take an `arb::cell_connection` argument, to fix a runtime python crash caused by passing an `arb::cell_connection` where a `cell_connection_shim` was expected.

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Add docs for Python wrapper to ReadTheDocs:

- Overview, Common Types, Hardware Management, Recipes, Domain Decomposition, Simulations, Metering
- Installing Arbor: Optional Requirements (Python), Buidling and Installing (Python Frontend), and Installation (Python Module)

Missing (, since not implemented yet): 

- probes
- arbor-sup 
- hint maps in domain_decomposition
- reset, events, empty schedule in event_generator
Also does not cover unit testing (since doc is user-centric).

Makes also defaults and units in wrapper consistent.

Fixes  #766

Support for recording spikes generated by a simulation in the Python wrapper
* Implement a `spike_recorder` that holds a shared pointer to a `std::vector` of spikes, and a callback for the `arb::simulation` spike recording API.
* Add `python/example/ring.py` that creates a ring network, then records and prints spikes.
* Some fixes to get the full `recipe` -> `domain_decomposition` -> `simulation` -> `spikes` workflow to work
  * always use default `global_parameters`: user customization of global parameters for cable cells can wait until the ion species interface is finished.
  * change the Python recipe interface for `recipe::connections_on` to use `pybind11::objects` because of shim.
* Some small improvements to error and help messages.

Fixes #764 

Fixes #782 
* consistent formatting of strings returned by `__repr__` and `__str__` for wrapped types
* rename enum wrappers to be less verbose, where it makes sense.

Add full wrappers for

- `lif_cell`
- `spike_source_cell`
- `benchmark_cell

And an opaque wrapper for cable_cell with a helper function that generates a random morphology identical to the ones in the nsuite benchmarks.

Fixes #767.

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Domain decomposition in python wrapper
* functions: `partition_load_balance`
*  types: `group_description` and `domain_decomposition`

Also includes
* remove comment that `gid` in a `group_description` must be sorted.
* a host of small clean ups of python wrapper

Fixes #761

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Fixes #760

Wraps arbor recipe (without probes, i.e. num_probes, probe_info, get_probe) including 

- cell_connection, 
- gap_junction_connection, 
- recipe with
  - num_cells
  - cell_description
  - cell_kind
  - num_sources
  - num_targets
  - num_gap_junctions_sites
  - event_generators
  - connections_on
  - gap_junctions_on
  - global_properties
- enum cell_kind in `identifiers.cpp`