We probably don't want to build EXTOLL stuff on remote (e.g. EBRAINS)
infrastructure that connects via quiggeldy anyways.

Change-Id: I68ac5044df6bd7250686ad0e924fd36c69599737

We probably don't want to build HostARQ stuff on remote (e.g. EBRAINS)
infrastructure that connects via quiggeldy anyways.

Depends-On: c/20318
Change-Id: I999496913993a662bb93dfad9754695f9da41d80

sborduka authored 3 years ago and Philipp Spilger committed 2 years ago

Depends-On: 17913,17759,18733,19543,19544
Change-Id: I15656b7d1da952890d01cf4be15919c7553e16e7

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

(Adapted from c/9752. Change moved from fisch to hxcomm.)

* QuiggeldyClient
  * Implemented as yet another connection thanks to the new architecture
    rework.
  * Since quiggeldy is only to be used for "complete" PlaybackPrograms
    sent to the FPGA, the corresponding Stream instance only supports a
    `submit_blocking`-method that - as the name suggests - blocks until
    results are ready.
  * A correspdonding `submit_future`-method that returns a future to the
    results is planned (but currently only in proof-of-concept stage).

* Provide serialization to SF via translation function that uses
  PortableBinaryArchives from cereal.

* Generic versions of `Quiggeldy{Connection,Server,Worker}` implemented in
  `hxcomm/common/{,,detail/}quiggeldy_{rcf,common,connection,server}.*`
  * then specialized in
    `hxcomm/vx/{,,detail/}quiggeldy_{rcf,common,connection,server}.*`
    -> users only have to include specialized-headers (see tests)
  * allows straight-forward port to different sets of
    connectionparamters

* `hxcomm/{common,vx}/quiggeldy_rcf.{h,tcc}`
  * have single RCF-interface for all types of connections per
    architecture
  * SF serialization via cereal

* `hxcomm/common/quiggeldy_common.{h,tcc}`
  * needed munge auth routines for connection (client) and server
    (defined inline, should be changed if hxcomm ceases to be
    header-only)

* Seperate client and server implementation:
  * client-side implementation in `hxcomm/{common,vx}/quiggeldy_connection.{h,tcc}`
  * if not given, client-side infers remote connection parameters from
    `QUIGGELDY_{IP,PORT}` environment variables
  * server-side implemtation in
    `hxcomm/{common,vx}/detail/quiggeldy_server.{h,tcc}`

* automatic detection checks for "QUIGGELDY_ENABLED" defined in
  environment to be a non-zero integer

* add simple mock-test verifying that quiggeldy can be started and
  connected to

* add StreamRC-class that implements a stream interface with Reduced
  Capabilities for Remote exeCution

NOTE:
* slurm allocation is not tested yet!

Depends-On: 12362

Change-Id: I7600b3925c56597aedf16cdbf4147c0846a00b1b

Yannik Stradmann authored 4 years ago and Philipp Spilger committed 4 years ago

* Full software connection which responds on reads with zero and shall ignore all other messages
* Adjustable ns_per_message can be used for performance testing of higher software layers
** defaults to 125M Messages/s, matching the 8 GBit/s FPGA-chip links for our parameterization
   for HX

Change-Id: I845be742f853786aaf5bdedc41db7a7120fc8e95

Not all connections will support the full direct-access interface that
the default implementation of Stream provides but rather only the subset
needed for execute_messages()-invocations.

* Introduce `ConnectionFullStreamInterfaceVariant` that is merely a
  filtered version of the full `ConnectionVariant` with those types
  removed that do not support the full(= default) Stream interface.

* Introduce `get_connection_full_stream_interface_from_env()` that gets
  the connection defined by the environment but checks if it can support
  the full_stream_interface. If so the connection is wrapped in an
  `optional`, otherwise nullopt is returned.

* Tests are then skipped at runtime via `GTEST_SKIP` if the optional is
  empty.

Change-Id: Ib3e196723e9c2beb8a0b00f57372f5b7cb359369

Philipp Spilger authored 4 years ago and Oliver Breitwieser committed 4 years ago

* autogeneration is not possible if there exist connections which
  can't generically be defined as a templated class which only takes
  the specific ConnectionParameter, which is the case e.g. if
  specializations are to be made depending on the actual usage

Depends-On: 11765

Change-Id: I1ff982ae4f03e1f99e926fe9a91a2235f3cdcab3

* As discussed on 2020-04-07 after the Softie's meeting (ECM, YS, PSP,
  CM, OJB), we want to have a streamlined connection interfaces for all
  the possible ways we turn FPGA words into other FPGA words in
  hardware.

* C++-API: Split old Connection objects into two parts:
  * The connection object that the user can only create and destroy but
    which holds state necessary for communication with the backend (hw,
    CoSim, QuiggelyClient, etc) and allows some state to be set.
  * Introduce hxcomm/common/stream.h:
    * The Stream-object featuring a uniform interface handling how to
      feed and read words to/from the backend.
    * The interface is defined via templated `Stream`-class that has to
      be befriended in the corresponding connection implementations
    * The Stream-interface is verified at compile time via the
      Streamable helper-struct.

* Move `execute_words` from fisch as more generic
  `execute_messages`-function that is the entry point for all upper
  layers when communcating with backends.
  * Overloading is possible by fully or partially-specializing the
    detail::ExecutorMessages-helper struct's operator() method (not
    needed once partial function specialization arrives, so never)
  * default implementation in common namespace by adding explicit
    `send_halt_message_type` to ConnectionParameter template argument

* Rework hxcomm/common/{arq,sim}connection.{h,tcc} to use the new Stream
  interface.

* Update all tests to use streams for interacting with the hardware.

* Rename init_parameters_t -> init_parameters_type to be consistent
  (they will be used by quiggeldy but that change is not merged -> no
  API break)

* Provide get_connection_from_env-function that returns a connection
  variant over all possible connections (and is used in pyhxcomm, see
  below).

* Introduce seperate `pyhxcomm`-namespace that is private to hxcomm and
  only to be accessed from Python. It adds convenience context-wrappers
  for connections ensuring connections get closed upon exiting contexts
  in Python
  * pyhxcomm::Managed<Connection> provides enter/exit-methods
  * enter() returns pyhxcom::Handle<Connection> holding the connection
  * if Handle<Connection> gets destroyed, connection gets freed (i.e.
    disconnected)
  * Managed<Connection> also holds a weak-ptr to the returned
    ConnectionHandle object and disconnects the Handle upon exit()-ing
  * Expose union over all Connections as `pyhxcomm_vx.ConnectionHandle`
    that can be used in type annotations in Python.
  * Expose special context manager `pyhxcomm_vx.ManagedConnection` that
    automatically creates a new connection from the environment and
    returns `ConnectionHandle` (which is a concrete `<Connection>Handle`
    in python rather than the cumbersome variant type).

* Introduce hxcomm::visit_connection that allows for applying operations
  on connections wrapped in Handle-objects such as variants for hxcomm
  and pyhxcomm::ConnectionHandles. It is used in
  hxcomm::execute_messages as well as stadls::run-pybind11 bindings.
  * Functions templated over connections can be easily specialized for
    all wrapped connections via:
    * `ConnectionIsPlainGuard<Connection> = 0`
    * `ConnectionIsWrappedGuard<Connection> = 0`
  * Another possibility is to perform compile time checks using
    `ConnectionIsPlain<Connection>::value` (see `run`-implementation in
    haldls for example).

* Add `static char const name[]` attributes for all connection for more
  informative logging.

Depends-On: 10346,10335,10703,10978,11027

Change-Id: I3ff901db53a5cb94a1faa50b18dc36786246f389