Fix a latent bug in the Python wrapper that was triggered in pybind11 v2.6.1

These changes ensure that the GIL is acquired before calling functions in C++ that
may consume a Python object with reference count 1 as an argument, in which case
the object's destructor is called at the end of the function, after any GIL acquired inside
the function would have been released.

Fixes #1271.

Major changes in Python library and documentation:

* Add global state (immutable after initialization) for the Python module that
  manages the mapping between probe metadata types and the method by which the
  corresponding [sample data can be collected and presented as Python objects.
  This is necessary for decoupling the implementation of the simulation wrapper
  from that of the various cable cell probe types.

* Wrap each of the C++ cable cell probe types with a Python function that
  returns a `probe_info` object.

* Add code for converting and accumulating C++ probe metadata and sample
  data, registered in the module global state against the metadata type info.

* Make the `arbor.simulator` object responsible for recording all spike and
  sample data, with corresponding new methods.

* Use NumPy arrays and structured datatypes for returning spike and
  sample data.

* Place Python schedule proxies under an abstract interface so tha...

* Update `dryrun` to generate some spikes again, add a voltage printer.
* Fix `ring.py` recipe.
* Fix labels in `single_cell_builder.py` and `single_cell_swc.py`.

* Make recipe return probes as a vector.
* Remove `probe_id` from `probe_info`.
* Rename `fvm_probe_info` to `fvm_probe_data` (everything was being called info, and it was getting confusing).
* Make `probe_association_map` specific to `mc_cell_group`/`fvm_lowered_cell`.
* Change probe_association_map representation: unordered map for probe_id -> tag; unordered multimap for probe_id -> fvm_probe_data. This allows multiple probes to be associated with the same probe id.
* Call sampler callback separately for each probe with the same probe_id.
* Replace location-based probes with locset equivalents.
* Add index for probes sharing a probe id.
* Bundle all probe metadata (id, tag, index, probe-specific meta) into `probe_metadata` struct for passing to sampler callbacks.
* Change simple_sampler to work on `trace_vector`, a vector of `trace_data`. The _i_th element is the data from probe with index i.
* Consolidate hash composition and `std::hash` specialization code in new header.
* Update python lib for new API.
* Update tests and examples for new recipe, internal probe, and simple_sampler APIs.
* Update docs to suit.

Fixes #589 and #730, providing cell-wide probes and a correct total trans-membrane current.
Fixes #822, providing spatial interpolation of membrane voltages as governed by cable resistivity and diameter.

* Add partition of point mechanism target vector by cell index to `fvm_mechanism_config`.
* Fix sign error in axial current interpolation.
* Replace `fvm_probe_info` with class that wraps a variant type capturing one of a set of backend-cellgroup probe translation representations: `fvm_probe_scalar`, `fvm_probe_interpolated`, `fvm_probe_multi`, `fvm_probe_weighted_multi`, `fvm_probe_membrance_currents`.
* Refactor `fvm_lowered_cell_impl::resolve_probe_address` so that it avoids the long chain of if-else tests on wrapped `any` type with an invocation of `util::any_visitor`. Each specific cable cell probe address representation is translated into one of the intermediate `fvm_probe_info` variants via an overload of `resolve_probe`.
* Add new non-scalar cell probe address types for all-of-cell values, including point mechanism state, density mechanism state, ion currents and concentrations, membrane voltage by CV cable, and total trans-mebrane currents. The latter are ultimately calculated by computing the net axial current fluxes in each CV.
* Add `util::any_ptr` field to the sampling function interface, pointing to constant-per-simulation-run metadata associated with a probe address. Underlying metadata type varies between probe types.
* Extend `simple_sampler` so that it optionally captures sampler metadata.
* Add special case to `simple_sampler` for vector-valued sample data, allowing it to translate from a pointer-pair range to a std::vector.
* Fix minor bugs in `util::variant`.
* Add to `mc_cell_group` the logic for translating each intermediate probe representation to a sampler callback invocation, reusing temporary scratch vectors across calls.
* Add unit comments to `embed_pwlin_data`.
* Add probe variety demonstration `probe-demo` to examples, covering most of the new cable cell probe classes.
* Add `util::any_visitor` runtime-type dependent dispatcher class.
* Add `util::overload` helper for generating overloaded functional objects from individual functions, lambdas or functionals.
* Extend probe unit tests in `test_probe.cpp` to cover new probe address types.
* Add unit tests for `any_visitor` and `overload`.

* Extend repertoire of cable cell probes.
* Add the following scalar (i.e. corresponding to just one raw value)
probe address types for cable cells:
   - `cell_probe_membrane_voltage`, replacing `cell_probe_address::membrane_voltage`;
   - `cell_probe_total_ionic_current_density`, replaing `cell_probe_address::membrane_current`;
   - `cell_probe_density_state`;
   - `cell_probe_point_state`;
   - `cell_probe_ion_current_density`;
   - `cell_prone_ion_int_concentration`;
   - `cell_prone_ion_ext_concentration`.
* Extend backend class interface with `mechanism_field_data` method, that returns a raw (device) pointer to the per-instance values of a mechanism state variable.
* Make unit test for probes backend-generic.
* Add unit tests for new probes.
* Modify `fixed_ica_current.mod` unit test NMODL module to be more generic in naming, so it can be better used in probe unit tests.
* Update examples for new cable cell probes.
* Update Python interface for new probe types

Fix warning generated when `delay` and `weight` were accidentally swapped in #984 
Match `cell_connection` delay type to `connection` delay type

A big update that wraps over 90% of the public C++ API, with enough functionality to let Python users to perform useful modelling.

Key features
- wrapping of cable cell functionality
  - user-defined explicit compartmentalisation not supported
- `single_cell_model` abstraction unique to the Python wrapper that simplifies developing and testing single cell models
- one-dimensional cell builder for Python wrapper that simplfies building cells that 
  - in and of itself limited in scope, but a very useful example of mapping a richer morphology builder onto `sample_tree`s.
- parsing of region and location expressions from strings
  - implements a generic s-expression parser that we can use for other purposes later

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Ensure that errors in Python callbacks that are called from multithreaded C++ code propogate the correct Python error back to the parent Python callback site, and that no callbacks are called from other threads if an error has already ocurred.
- protects each recipe callback with a mutex, stores python exception, catches and throws python exception if occured
- methods calling recipe (in simulation and partition_load_balance) are protected as well by try catch, resets and rethrows python exception (if occured) or else throws C++ exception

fixes #792

Adds

* to `recipe.cpp`: 
`num_probes` and `get_probe` with helper function `cable_probe(kind, id, location)`
* `sample.cpp`: 
`sample` type with attributes `time` and `value`; a `sampler` with `samples()` and `clear()` and helper functions `attach_sampler(sim, dt)` for adding all probes as well as `attach_sampler(sim, dt, probe_id)` for adding one probe
* `morphology.cpp`: 
`location` as `arb::mlocation` with `branch` and `position` 
* corresponding docs and example
* update to new cable cell semantics in c++ validation test

fixes #762

Remove the dead `cell_connection_shim` from the python wrapper.

Fix the connection printer to take an `arb::cell_connection` argument, to fix a runtime python crash caused by passing an `arb::cell_connection` where a `cell_connection_shim` was expected.

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Add docs for Python wrapper to ReadTheDocs:

- Overview, Common Types, Hardware Management, Recipes, Domain Decomposition, Simulations, Metering
- Installing Arbor: Optional Requirements (Python), Buidling and Installing (Python Frontend), and Installation (Python Module)

Missing (, since not implemented yet): 

- probes
- arbor-sup 
- hint maps in domain_decomposition
- reset, events, empty schedule in event_generator
Also does not cover unit testing (since doc is user-centric).

Makes also defaults and units in wrapper consistent.

Fixes  #766

Support for recording spikes generated by a simulation in the Python wrapper
* Implement a `spike_recorder` that holds a shared pointer to a `std::vector` of spikes, and a callback for the `arb::simulation` spike recording API.
* Add `python/example/ring.py` that creates a ring network, then records and prints spikes.
* Some fixes to get the full `recipe` -> `domain_decomposition` -> `simulation` -> `spikes` workflow to work
  * always use default `global_parameters`: user customization of global parameters for cable cells can wait until the ion species interface is finished.
  * change the Python recipe interface for `recipe::connections_on` to use `pybind11::objects` because of shim.
* Some small improvements to error and help messages.

Fixes #764 

Fixes #782 
* consistent formatting of strings returned by `__repr__` and `__str__` for wrapped types
* rename enum wrappers to be less verbose, where it makes sense.

Add full wrappers for

- `lif_cell`
- `spike_source_cell`
- `benchmark_cell

And an opaque wrapper for cable_cell with a helper function that generates a random morphology identical to the ones in the nsuite benchmarks.

Fixes #767.

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Domain decomposition in python wrapper
* functions: `partition_load_balance`
*  types: `group_description` and `domain_decomposition`

Also includes
* remove comment that `gid` in a `group_description` must be sorted.
* a host of small clean ups of python wrapper

Fixes #761

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Fixes #760

Wraps arbor recipe (without probes, i.e. num_probes, probe_info, get_probe) including 

- cell_connection, 
- gap_junction_connection, 
- recipe with
  - num_cells
  - cell_description
  - cell_kind
  - num_sources
  - num_targets
  - num_gap_junctions_sites
  - event_generators
  - connections_on
  - gap_junctions_on
  - global_properties
- enum cell_kind in `identifiers.cpp`