akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Add docs for Python wrapper to ReadTheDocs:

- Overview, Common Types, Hardware Management, Recipes, Domain Decomposition, Simulations, Metering
- Installing Arbor: Optional Requirements (Python), Buidling and Installing (Python Frontend), and Installation (Python Module)

Missing (, since not implemented yet): 

- probes
- arbor-sup 
- hint maps in domain_decomposition
- reset, events, empty schedule in event_generator
Also does not cover unit testing (since doc is user-centric).

Makes also defaults and units in wrapper consistent.

Fixes  #766

akuesters authored 5 years ago and Benjamin Cumming committed 5 years ago

Domain decomposition in python wrapper
* functions: `partition_load_balance`
*  types: `group_description` and `domain_decomposition`

Also includes
* remove comment that `gid` in a `group_description` must be sorted.
* a host of small clean ups of python wrapper

Fixes #761

Add citation information to the readme.

Benjamin Cumming authored 6 years ago and Sam Yates committed 6 years ago

* Rename `arb:simd::simd_detail` namespace to `arb::simd::detail`.
* Update SIMD documentation to suit.

Fixes #468.

noraabiakar authored 6 years ago and Benjamin Cumming committed 6 years ago

Add gap junctions and improve general model overview in docs.

* Replace Arbor Model::Common Types with Arbor Model::Concepts, that introduces the concept of cells, connections and gap junctions. 
* Add gap junction documentation to cpp:recipe
* Document the implicit rule forcing cells connected by gap junctions to be in the same cell group.
* Add documentation that discusses core concepts in Arbor models.

Update Hardware API documentation

* split the domain decomposition and hardware API docs into separate pages
* update hardware API to reflect new *libarbor* and *libarborenv*
* add basic documentation for `optional`, `any` and `unique_any` types.

Update the installation guide to reflect the latest supported tool,
compiler and library versions.

Remove the Python docs, because they documented features that have not
been implemented yet. The Python docs can be added incrementally as
features are implemented.

Start work on oupdating the documenetation for hardware interfaces and
domain decomposition.

akuesters authored 6 years ago and Benjamin Cumming committed 6 years ago

Update documentation for Python.

    splits the conceptual model ideas from the C++ docs into their own section
    has C++ and Python docs for recipes, domain decomposition, etc.

fixes #667

Added the following documentation (structure):

GETTING STARTED:

    Installing Arbor/Requirements/Optional Requirements/Python
    Installing Arbor/Building and Installing Arbor/Python Front End

MODEL BASICS:

    Overview
    Common Types
    Recipes
    Domain Decomposition
    Simulations

PYTHON:

    Overview
    Common Types
    Recipes
    Domain Decomposition
    Simulations

DEVELOPERS:

    Python Profiler
    Python Unit Testing

GETTING STARTED has two added sections of optional requirements using python and how to build the python front end.

MODEL BASICS describes Arbor's concepts in general (independent of programming language), thus general information on concepts in C++ API was moved here/ added.

PYTHON describes Arbor's python frontend in the same structure as MODEL BASICS and C++ API ( needs updates as soon as features are added/changed in new python PR).

DEVELOPERS section has two added sections for meter management and unit testing with python front end.

Further, some corrections in existing documentation (for obvious errors, e.g. duplicate text, not ending sentences) and referencing sections were done.

akuesters authored 6 years ago and Benjamin Cumming committed 6 years ago

First step towards the Python front end.

This commit sets up the structure of the python implementation
* directory structure
* git submodule for pybind11
* best practices for making bindings with pybind11
* unit testing for the python front end

It implements the following features in the Python front end
* execution contexts
* gpu detection
* thread count detection
* MPI initialization helpers.

Fixes #667.

noraabiakar authored 6 years ago and Benjamin Cumming committed 6 years ago

Updates the install docs. Fixes #604

akuesters authored 6 years ago and Benjamin Cumming committed 6 years ago

changes: 
- .travis.yml:
  - added matrix for different osx's, since enumeration style only works for `env` and `compiler`

- scripts/travis/build.sh:
  - changed getting compiler version from ``${CXX} -dumpversion`` to ``${CXX} --version | grep -m1 ""`` 
  - added `--oversubscribe` flag to `mpiexec` on Mac to allow more processes on a node than processing elements
  - added `--mca btl tcp,self` flag for Open MPI to use the "tcp" and "self" BTLs for transporting MPI messages on Mac

noraabiakar authored 6 years ago and Benjamin Cumming committed 6 years ago

Dry-run mode: 
* An implementation of distributed_context that is used to mimic the performance of running an MPI distributed simulation with n ranks.
* Verifiable against an MPI run with the same parameters. 

Implementation: 
* Describe the model on a single domain (tile) and translate it to however many domains we want to mimic using arb::tile and arb::symmetric_recipe. This allows us to know the exact behavior of the entire system by only running the simulation on a single node.
* Mimic communication between domains using arb::dry_run_context

Example: 
* dryrun in example/ is a verifiable example of using dry-run mode with mc_cells

Other:
* Documentation of dry-run mode 
* unit test for dry_run_context 

Sam Yates authored 6 years ago and Benjamin Cumming committed 6 years ago

Fixes #587.

* Eliminate Clang warnings from GCC-tree-optimization bug work-around.
* Error with static-assert if simd type is used with a missing simd abi.
* Clarify install documentation regarding use of ARB_VECTORIZE with ARB_ARCH.

Make the execution context presented to users an opaque handle, moving all implementation of the gpu, thread and distributed contexts into the back end.

* move `execution_context` and `distributed_context` definitions to the back end
* create `execution_context` handle called `context` in the public API
* provide `make_context` helper functions that build different context configurations (default, user-specified local resources, with MPI)
* update documentation for all parts of the public API that touch contexts
* move `distributed_context` docs to the developer documentation (from the public API docs)

Benjamin Cumming authored 6 years ago and Sam Yates committed 6 years ago

Fixes #564

Sam Yates authored 6 years ago and Benjamin Cumming committed 6 years ago

* Update the install docs for architecture build options
* Update to reflect new install target

Reduce redundant functionality across event_generator, time_seq
and schedule by providing a low-heap overhead interface to schedule
and using that for time sequences in event_generator and specialized
cell groups.

* Have schedule return pair of pointers as view to generated times.
* Fix missing DEBUG/TRACE functionality.
* Use rate instead of mean_dt for Poisson schedule.
* Move merge_events() functionality to simulator.cpp.
* Migrate event_generator to event span interface.
* Migrate tourney_tree to event span interface.
* Only invoke tourney_tree merge if generators have events in the epoch.
* Use schedule for times in event_generator implementations.
* Replace seq_generator with explicit_generator that keeps a copy of events.
* Replace vector_backed_generator and poisson_generator with schedule_generator.
* Replace time_seq uses with schedule.
* Add default empty schedule.
* Move rounding error test for regular time sequence into schedule test.
* Update sampling API documentation for schedule.

noraabiakar authored 6 years ago and Benjamin Cumming committed 6 years ago

Use correct profiling and mpi flags.
Fix profiler doc

Sam Yates authored 6 years ago and Benjamin Cumming committed 6 years ago

CMake and build refactoring

*   Use CUDA as first-class language (leading to CMake 3.9 minimum version requirement).

*   Use 'modern CMake' interface libraries for compiler options, include file and library dependency tracking. Interface library targets:
    * `arbor-deps`: compiler options and library requirements for the `libarbor.a` static library, as governed by configure-time options and environment.
    * `arbor-private-headers`: include path for non-installed headers, as required by unit tests and arbor itself.
    * `arbor-aux`: helper classes and utilities used across tests and examples.
    * `ext-json`, `ext-tclap`, `ext-tbb`, `ext-benchmark`, `ext-sphinx_rtd_theme`: externally maintained software that we include (directly or via submodule) in the `ext/` subdirectory.
 
*   Single static library `libarbor.a` includes all built-in modules and CUDA objects.

*   Simply configuration options:
    *  `ARB_WITH_TRACE`, `ARB_AUTORUN_MODCC_ON_CHANGES` `ARB_SYSTEM_TYPE` removed.
    * External `modcc` is provided by `ARB_MODCC` configuration option; if provided `modcc` is still buildable, but is not included in the default target.
    * `ARB_PRIVATE_TBBLIB`, defaulting to `OFF`, instructs the build to make TBB from the included submodule.

*   Extend `ErrorTarget` functionality to provide a dummy target or an error target based on a condition.
*   Generate header version defines and library version variables based on git status and project version, via new script `include/git-source-id`.
*   All generated binaries now placed in `bin/` subdirectory at build.
*   Install targets installs: public headers (incomplete); static library; `modcc` tool; `lmorpho` executable; `html` documentation (examples, tests and validation data are currently not installed).
*   Executable targets have had the `.exe` suffix removed; unit tests are labelled `unit` (arbor unit tests), `unit-modcc` (modcc unit tests), `unit-local` (distributed tests with local context), `unit-mpi` (distributed tests with MPI context).
*   More graceful handling of configure-time detection of `nrniv`, Julia and required Julia modules for validation data generation.
*   Add `cmake/FindJulia.cmake`, `cmake/FindTBB.cmake`  package finders, and adjust `cmake/FindUnwind.cmake` to use link library-style properties.
*  Adjust travis script to test `unit-local` and `unit-mpi` if appropriate.
*  Simply documentation `conf.py`.

Source relocation and reorganization

* All external project sources and files moved to `ext/`.
* Source code refactoring to decouple library-using code from the configure-time definitions that govern arbor behaviour: removes conditional code in public headers that depends upon `ARB_WITH_X`-type definitions at compile time. Affected code is is in the public interfaces for MPI, the threading implementation, and the profiler.
* Remove `util/debug.hpp`; split out functionality for pretty-printing from assertion handling.
* Make FVM cell non-physical voltage check a run-time cell-group parameter.
* Move spike double buffer implementation to `simulation.cpp`.
* Make timer utility wrap POSIX `clock_gettime` independent of threading configuration.
* Make `mpi_error` derive from `system_error` and follow C++11 `system_error` semantics.
* `EXPECTS` macro replaced by `arb_assert` macro.
* JSON dependency removed from `libarbor.a` and header files: moved to auxiliary library.
* Publicly visible macros garner an `ARB_` prefix as required.
* Move SWC test file to `test/unit` directory.
* Work-in-progress splitting of public from private includes: as a convention not entirely adhered to as yet, private headers within arbor source are included with `""`, public headers with `<>`.

Modcc interface changes

* Expose via `--namespace` option the functionality that sets the namespace in generated code.
* Use `--profile` option to add profiler hooks to generated code; uses public function interface directly rather than `PE/PL` macros in order to avoid public `PE` and `PL` defines.

Benjamin Cumming authored 6 years ago and Sam Yates committed 6 years ago

Move from choosing the distributed communication model from a compile time choice (the old `arb::communication::communication_policy` type) to a run time decision.

* Add `arb::distributed_context` class that provides the required interface for distributed communication implementations, using type-erasure to provide value semantics.
* Add two implementations for the distributed context: `arb::mpi_context` and `arb::local_context`.
* Allow distribution over a user-supplied MPI communicator by providing it as an argument to `arb::mpi_context`.
* Add `mpi_error` exception type to wrap MPI errors.
* Move contents of the `arb::communication` namespace to the `arb` namespace.
* Add preprocessor for-each utility `ARB_PP_FOREACH`.
* Rewrite all examples and tests to use the new distributed context interface.
* Add documentation for distributed context class and semantics, and update documentation for load balancer and simulation classes accordingly.

Fixes #472

* Update SIMD developer docs to reflect newly merged mechanism refactor work.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

Add two new documentation pages for the C++ API

* Add domain decomposition page that covers `domain_decomposition`, `node_info` and `partition_load_balance`.
* Add simulation page that describes `arb::simulation` API interface.
* Fix some small typos elsewhere in the docs.
* Use `std::move` when adding spike callbacks to `arb::simulation` (useful if callbacks are stateful).

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

Fixes issue #467 

* Add GPU synchronization points where required for Kepler to coordinate CPU access of managed memory.
* Use hand-rolled double precision atomic addition for Kelper targets.
* Replace `ARB_WITH_CUDA` build option with `ARB_GPU_MODEL` option that takes one of 'none', 'K20', 'K80' or 'P100', and set up source-code defines accoringly.
* Clean up of redundant compiler flags and defines no longer required now that the project uses separate compilation for CUDA sources.

Fix some typos in the SIMD documentation.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

Add some C++ API documentation.

* Create C++ API section in docs.
* Document `arb::recipe`: both a class reference along with more explanatory text and best practices guide.
* Add some class documentation of basic types required to understand recipe definition.
* Some in-code comment clean up.
* Change `arb::cell_kind` from a vanilla enum to a scoped enum.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

The name `arb::model` did not clearly describe the role of the class, while `arb::simulation` better captures that this is an instantiation of a model for the purpose of running a simulation, as distinct from the description of a model represented by an `arb::recipe` instance.

* Rename sources `model.{hpp,cpp}` to `simulation.{hpp,cpp}`.
* Rename class `arb::model` to `arb::simulation`.
* Update docs and tests to suit.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

Put all the SIMD docs in a single topic, to simplify the documentation tree.

Added an installation guide to the Read The Docs
Removed the outdated build/install information from README.md
Link from README to Read The Docs
Updated the splash page for Read The Docs

This provides a bunch of SIMD intrinsic wrappers as a precursor to the SIMD printers.

The aim is that the SIMD printer can be agnostic regarding the particular vector architecture.

The design is based rather loosely on the proposal P0214R6 for C++ Parallelism TS 2. The transcendental function implementations are adapted from the existing SIMD architecture-specific code, which in turn are based on the Cephes library algorithms.

The custom CSS for the html documentation have been tweaked.

In some places our CMake scripts were attempting to check out git submodules when required, if they have not already been checked out. The code that does this was cut and pasted, and was getting unwieldy.

To minimise the responsibilities of CMake, this PR

removes calls to git
introduces a function check_git_submodule that can be used to test if a git submodule is installed, and print a helpful message that informs the user how to check it out if needed.
introduces a function add_error_target that makes a target that prints a message then quits with an error. This can be used to generate a proxy target when a problem is detected during CMake setup. This means that an error is only generated when building a target with a missing dependency, instead of an error during CMake setup.
refactors the CMake setup for the docs and ubenches targets to use these new features.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

Replace the profiler with a simpler design that works for nested multi-threaded regions.

* Replace hierarchical profiler accounting with strictly exclusive regions.
* Make tree grouping of profile data a presentation concern.
* Uncouple profiler semantics from Arbor classes such as `model`.
* Add thorough documentation for the new profiler to the library documentation.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

Update the FVM formulation to use current densities instead of currents.

Modifications to modcc:
* Update printers to store and use weights for point process mechanisms,
* Scale ion species current contributions by area proportion, similarly to contributions to the accumulated current.

Changes to FVM code:
* Update weights calculation for density and point processes mechanisms:
    * density channels use relative proportion of CV area, i.e. "density",
    * point processes use the reciprocal of the CV area to convert to a density.
* Add `cv_area` parameter for matrix constructor, which is used by matrix assembly to convert current densities to currents.
* Update stimulus implementations (gpu and cpu backends) to contribute current densities.

Other changes:
* Update unit tests to use new interfaces.
* Update units section in LaTeX docs.

Fixes #374.

* Use ARB_ and arb_ as variable prefixes in place of NMC_ and nmc_.
* Replace references to 'NestMC' and 'NEST MC' to refer instead to Arbor.
* Use 'arbor' as the sim name in generated validation data.
* Reflow long-line paragraphs in `tests/ubench/README.md`.
* Change names of CUDA mechanism and CUDA kernel libraries to include arbor name.

Collect samples in FVM lowered cells across an integration period, prior to delivery to sampler callbacks.

Fixes #283.

* Make `multi_event_stream` a generic data structure, based on the event type. These are now used for both `deliverable_event` objects and `sample_event` objects.
* Add generic accessors for event data, separating access to event metadata (cell index, time) from payload (weight, sampling info etc.).
* Make a flat, CUDA-friendly data structure describing the `multi_event_queue` marked event state, for passing to mechanisms and kernels. This also brings the underlying representations of the `multicore` and `gpu` versions of `multi_event_queue` closer.
* Implement batched sampling kernel for GPU.
* Additional utility function: `util::is_sorted_by`.

Overhead of implementation and sampling process is negligible for CPU, and with no samples less than 1% on GPU. Running the miniapp with 10'000 cells and sampling 1% of the somata at 0.01ms adds approximately 10% to simulation time. 

Towards resolution of issue #283.

* Replace probe/sample infrastructure with new API as outlined in the `sampling_api.rst` documentation.
* Separate `cell` probe information from `cell` description object.
* Add `--list` option to `tsplot`, to summarize available time series for plotting together with their metadata.
* Add `--sample-dt` option to validation tests.
* Change validation time series comparison behaviour: linf distance metric now estimated by comparing reference data points against linearly interpreted simulation samples, rather than the other way around.
* Add utility class `any_ptr` which allows lightweight and type-checked access to a type-erased pointer.
* Replace `singleton_recipe` with two simple recipe base classes used for unit and validation tests, in `tests/simple_recipes.hpp`.
* Simplify RSS cell implementation.
* Add statistical functions for testing of probabilistic methods in `tests/unit/stats.hpp` (and .cpp):
   * simple summary stats class;
   * Kolmogorov–Smirnov one-sided statistic and CDF;
   * approximate Poisson CDF.
* Simplify and refactor miniapp `sample_trace` code.
* Add new utility class `handle_set` for managing a collection of integer handles. (Near-simplest implementation.)
* Relax const-ness constraints on `transform_iterator`.

Ben Cumming authored 7 years ago and Sam Yates committed 7 years ago

Adds support for building documentation with Sphinx from reStructuredText-formatted files in the `doc` subdirectory. Automatic building has been verified with ReadTheDocs.

* Add basic documentation to the `doc` path.
* Use a git submodule and associated CMake to pull in ReadTheDocs theme at configuration time.