Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

* Add FPGA timeout nofication excplicitly to ConnectionParameters for
  immediate access.
* Introduces QuiggeldyWorker<T>::check_for_timeout(response) checking
  for responses that are timeouts; triggers a reconnect if encountered.

Change-Id: Ib290317b6fe1c36a44e24600a535964ef0ebf52f

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

Change-Id: I92b6a3ab17de0ad0dbbaff7dfe1dfd0ef3eeed64

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

Add tool to query remote version information from quiggeldy connection
via RCF-interface.

```
Query remote version information of running quiggeldy instance.

Allowed options:
  -h [ --help ]         produce help message
  -i [ --ip ] arg       IP to connect to
  -p [ --port ] arg     Port to connect to
```

Depends-On: 13092
Change-Id: I68ab5f2e6e6abf21ceaec27065a75c0d2925d0e7

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

For non-munge use, display the user supplied username. For munge usage,
convert uid to string.

Makes for much more informative log messages especially when not using
munge.

Unfortunately, despite avoiding copying, using strings as user-id seems
to cause memory corruption after about five minutes of intense usage.
Investigation pending.

Change-Id: If4626e078a41ce8d4b7059492fea1137db94bfa4

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

The error messages resulting from misconfigured munge-support are rather
cryptic. Therefore, we have the client ask the server if it expects
munge to be used and then:

* deactivate munge support on the client if server does not use it.
* throw an error if client has no munge support but server expects it.

Change-Id: I833b141c6658c6db95a0d06e924a4467c260935c

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

Right now, get_unique_identifier() requires an active connection because
it is not implemented as a static method that checks the environment
rather than an existing connection.

Because hxtorch depends on this information during set up, we need to
ensure the connection is established.

Depends-On: 13932

Change-Id: I167c5ced640921cde62d6d40effe6f68b04b8f51

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

This change allows clients to specify a custom username when using
quiggeldy: QUIGGELDY_USER_NO_MUNGE

As the name indicates, this will disable munge-based authentication to
quiggeldy and hence requires that munge support is disabled on the
running quiggeldy.

This is useful if several users are emulated by a single unix user, as
was the case at the NICE2021 demo.

Depends-On: 13915

Change-Id: Iadacd1fbf25a57d3c3745bfb26140cd44291181a

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

* Add templated hxcomm::ReinitStackEntry
  * Allows registration of UTMessage-vectors to be executed after
    relinquishing control of remote hardware resources.
  * Is a no-op for all connections but QuiggeldyClient as only quiggeldy
    supports this mode of operation.
  * In Python, entries should be .pop()-ed from the stack because we
    cannot rely on the garbage collector to call the destructor in time.

* Add specialized hxcomm::vx::RegisterReinit

* Add helpers to wrap RegisterReinit in upper layers that can be
  constructed from Handles

Dependendcies:

Real dependency:
Depends-On: 11390,14013

We should also ensure the tests get built:
Depends-On: 11391

Change-Id: I33d640aa569aa4d10802469542bc41f40018ed5b

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

Quiggeldy is planned to be auto-deployed in a nightly fashion. Because
from time to time there are bugfixes/new features added, we need a way
to track in which state quiggeldy was compiled.

This is meant for expert users to ensure a specific version of quiggeldy
is deployed upon debugging.

At a later time, we will add a versioned interface, preventing users
from connecting when versions mismatch.

* Add `-v`/`--version` switch to quiggeldy that prints out with which
  repository at what commit-state it was compiled.
* Add `get_version_string`-method to RCF-interface to allow for remote
  querying of build information.

Depends-On: 13086
Change-Id: I9734f1d1d0214c4953f97e3835534f0e4d93520c

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

* Add `wriggeldy`, a wrapper to run processeses with a local quiggeldy.

* Wriggeldy is meant for debugging purposes to see if adding quiggeldy
  inbetween hardware and executing script has any detrimental effects
  (hint: it should not).

* Additionally, also add `wrap_with_quiggeldy`, a more verbose symlink
  to wriggeldy (that noone wants to type).

Change-Id: I0996832348a3a3132f9bb9adc84a96a53c6a27b5

Change-Id: I1c8b6aa70aea911a0351372ca540c3fa58eda8f2

Oliver Breitwieser authored 4 years ago and Eric Müller committed 3 years ago

(Adapted from c/9752. Change moved from fisch to hxcomm.)

* QuiggeldyClient
  * Implemented as yet another connection thanks to the new architecture
    rework.
  * Since quiggeldy is only to be used for "complete" PlaybackPrograms
    sent to the FPGA, the corresponding Stream instance only supports a
    `submit_blocking`-method that - as the name suggests - blocks until
    results are ready.
  * A correspdonding `submit_future`-method that returns a future to the
    results is planned (but currently only in proof-of-concept stage).

* Provide serialization to SF via translation function that uses
  PortableBinaryArchives from cereal.

* Generic versions of `Quiggeldy{Connection,Server,Worker}` implemented in
  `hxcomm/common/{,,detail/}quiggeldy_{rcf,common,connection,server}.*`
  * then specialized in
    `hxcomm/vx/{,,detail/}quiggeldy_{rcf,common,connection,server}.*`
    -> users only have to include specialized-headers (see tests)
  * allows straight-forward port to different sets of
    connectionparamters

* `hxcomm/{common,vx}/quiggeldy_rcf.{h,tcc}`
  * have single RCF-interface for all types of connections per
    architecture
  * SF serialization via cereal

* `hxcomm/common/quiggeldy_common.{h,tcc}`
  * needed munge auth routines for connection (client) and server
    (defined inline, should be changed if hxcomm ceases to be
    header-only)

* Seperate client and server implementation:
  * client-side implementation in `hxcomm/{common,vx}/quiggeldy_connection.{h,tcc}`
  * if not given, client-side infers remote connection parameters from
    `QUIGGELDY_{IP,PORT}` environment variables
  * server-side implemtation in
    `hxcomm/{common,vx}/detail/quiggeldy_server.{h,tcc}`

* automatic detection checks for "QUIGGELDY_ENABLED" defined in
  environment to be a non-zero integer

* add simple mock-test verifying that quiggeldy can be started and
  connected to

* add StreamRC-class that implements a stream interface with Reduced
  Capabilities for Remote exeCution

NOTE:
* slurm allocation is not tested yet!

Depends-On: 12362

Change-Id: I7600b3925c56597aedf16cdbf4147c0846a00b1b

Change-Id: I880ed4cfcda3fb8f5ddb0adea4822468fc47c5d6

Change-Id: Ifd638d035f30c0599f757ba7befd72182ba055f1

Change-Id: I55cabdbda5ede7807820efe53f9f09e4410ab4f6

Change-Id: Ie56ff04878cdd0c79bece3292b831988d1b8d409

Change-Id: Ic9d697fd2cbe1b012eb8e29f70afa8c02f808e55

Change-Id: I06221708e9a6ad77ef233a37be5d534a60dbb364

Change-Id: I7e0e82a66ca18848c109642a5e801868bd8e45d5

Also modifies target name… not sure if okay ;).

Depends-On: 14548
Change-Id: I074b12a76b529f820cbff9eb2f97b5ae9eb14393

Change-Id: I195afe5d9a94e0c0b5945dffb9147a086ec05765

Change-Id: Ib37a8c589209f0068f56a8f9b2e22149093032f4

Change-Id: I55bf737cd273776977d7663b60e47127f1a3c483

Yannik Stradmann authored 3 years ago and Yannik Stradmann committed 3 years ago

The default timeout is too tight for test execution
on loaded ASIC machines.

Change-Id: I1de3714d845feb8bfb6a113152877e74faaeff11

Change-Id: I53dcca93c4b4c1e07fbf90ed2b35cf74503ee36c

* Version check will already have been done in moved from object

Change-Id: Iff0d870a92a03f98423295f71cc43926b84dab3e

Depends-On: 12902
Change-Id: I4dcc82641d99befb9483af17fc32b8edaf154f65

const-ness is correctly forwarded into variant.

See: https://godbolt.org/z/KcPYxe
Change-Id: I136fae42464cf3b23649b733388ac0dd8da5a569

Change-Id: I60edea54c1264b01ae123c5777c3500f08047c27

Yannik Stradmann authored 4 years ago and Philipp Spilger committed 4 years ago

* Full software connection which responds on reads with zero and shall ignore all other messages
* Adjustable ns_per_message can be used for performance testing of higher software layers
** defaults to 125M Messages/s, matching the 8 GBit/s FPGA-chip links for our parameterization
   for HX

Change-Id: I845be742f853786aaf5bdedc41db7a7120fc8e95

Change-Id: Id474f222c65207402e0676106ca32496c22a3929

Change-Id: If5c2561b2d2587adc63333d57131b2065c4c174d

Based on a discussion between PSP and OJB:

We only want one `friend` declaration in each connection; and that is
for `Stream`.

Stream then has a nested Streamable struct, that performs that checks
because it has a right to access private/protected members. It is
implemented as a series of static_asserts that give a helpful error
message indicating what failed exactly. But, that means that when we
instantiate the default-constructed Stream it is effectively a
declaration of intend (i.e., "we intend this connection instance to
support the full Stream interface and hard-fail if it doesn't"). We
cannot perform a "check" if the Connection is even supposed to fulfill
the full Stream interface beforehand. A separate checker would require
to also be friends with all connections..

Hence, up until now we provide a manual hint
("has_full_stream_interface") that can be used to separate the
connections into those that are supposed to fulfill the full stream
interface (and are then used for in-depth hxcomm-tests) and those that
aren't.

Better:
We should have the indication if a Connection is supposed to be
Streamable in a separate type so that Stream only ever gets instantiated
via its default implementation where a failure to fulfil Streamable
should result in a compile error with helpful error message.

This struct is supports_full_stream_interface.

Change-Id: I402df2115373d7becfd3d5a3c76c60b3f1e94c66

Change-Id: I716c34a9f48755c6b1b35fa1cddb9e2a097c6e48

Change-Id: I4a09a8111a6ad57bf2a94c4be8b6dff91c1544bf

Depends-On: 10801
Change-Id: I527b9abff8913165c8370ab548604a377b883464

Depends-On: 12737

Change-Id: I7375eea63e4272b06282835f133e8478ac2afd57

* replace tbb::concurrent_queue by std::vector and mutex
  because the concurrent queue does not support thread-safe swap
* allows fast acquisition of _all_ messages from the connection
* in the typical use case of sending, waiting until halt and receiving
  this leads to less time spend with exclusive access to the connection
  and therefore more possible parallelism
* remove single-message receive() and try_receive() interface

Depends-On: 11113
Change-Id: Ifd6cebfcea3b9bc56699aaf9db741579ca074a81

* Stream::add(), Encoder::operator(), Decoder::operator()

Change-Id: Icda158123eed0f9c7af0a7aa2ef06ace4856cb43

* same performance
* newer implementation, full C++17 support, constexpr variant & visit
* less non-std dependencies

Depends-On: 11711
Change-Id: I721a383221492ef1370ddc1fa06a1097f1eb1657