diff --git a/docs/.gitbook/assets/cache-interaction.drawio.png b/docs/.gitbook/assets/cache-interaction.drawio.png
new file mode 100644
index 0000000000000000000000000000000000000000..c2972ad28b75c2c15823a2dee32b6fa78a3af3e6
Binary files /dev/null and b/docs/.gitbook/assets/cache-interaction.drawio.png differ
diff --git a/README.md b/docs/README.md
similarity index 100%
rename from README.md
rename to docs/README.md
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index e8b3900e00d0a3ddfc6c10129513291f7cbf877f..3cab1bb5b1246624cb54e0fac4d8fe95becacd6d 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -1,6 +1,6 @@
# Table of contents
-* [ðŸ Home](../README.md)
+* [ðŸ Home](README.md)
## For Developers
@@ -9,11 +9,14 @@
* [Setup development environment](for-developers/get-started/Setup-development-environment.md)
* [Configuration](for-developers/configuration/README.md)
* [Gateway](for-developers/configuration/gateway.md)
-* [Connectors](for-developers/connector/README.md)
- * [Parsing response with JSONdata](for-developers/connector/Parsing-response-with-JSONata.md)
* [Gateway](for-developers/gateway/README.md)
* [🔑 Authentication](for-developers/gateway/authentication.md)
* [👥 Users](for-developers/gateway/users.md)
+ * [🗃 Static files](for-developers/gateway/static-files.md)
+ * [📠Cache](for-developers/gateway/cache.md)
+ * [🔌 Connectors](for-developers/gateway/connector/README.md)
+ * [Create a connector](for-developers/gateway/connector/create-a-connector.md)
+ * [Parsing response with JSONdata](for-developers/gateway/connector/parsing-response-with-jsonata.md)
* [Frontend](for-developers/frontend/README.md)
* [Update GraphQL Queries](for-developers/frontend/Update-queries-GrahpQL-in-the-frontend.md)
* [📊 Visualisations](for-developers/frontend/visualisations.md)
diff --git a/docs/for-developers/configuration/gateway.md b/docs/for-developers/configuration/gateway.md
index 7f879ca3369f383b0f2b26fbacc31d35d889258b..d38ab154e55f1e150760153a28f33425d5df9811 100644
--- a/docs/for-developers/configuration/gateway.md
+++ b/docs/for-developers/configuration/gateway.md
@@ -27,7 +27,7 @@ description: >-
| AUTH\_SKIP | boolean | false | Allow to skip authentication. Warn: all routes will be accessible without authentication. |
| AUTH\_JWT\_SECRET | string | N/A | Secret that should be used to generate JWT token |
| AUTH\_JWT\_TOKEN\_EXPIRES\_IN | string | '2d' | <p>JWT token time to live.</p><p>Expressed in seconds or a string describing a time span <a href="https://github.com/vercel/ms">vercel/ms</a></p> |
-| AUTH\_COOKIE\_SAME\_SITE | string | 'strict' | Specify the cookie same site option. Value can be `lax`, `strict` or `none` |
+| AUTH\_COOKIE\_SAME\_SITE | string | 'strict' | Specify the cookie same site option. Value can be `lax`, `strict` or `none` |
| AUTH\_COOKIE\_SECURE | boolean | true | Specify the cookie secure option. Should be set to true if same site is not set to `strict`. |
| AUTH\_ENABLE\_SSO | boolean | false | Enable SSO login process, this variable will be provided to the frontend in order to perform the login. |
@@ -43,7 +43,7 @@ description: >-
#### Matomo
-Matomo is an open source alternative to Google Analytics. The gateway provide this configuration in order to be used by any frontend. The real implementation is left to the frontend.
+Matomo is an open source alternative to Google Analytics. The Gateway provide this configuration in order to be used by any frontend. The real implementation is left to the frontend.
| name | type | default | description |
| ---------------- | ------------------- | --------- | --------------------------------------------------------------------------------------------------- |
@@ -51,6 +51,16 @@ Matomo is an open source alternative to Google Analytics. The gateway provide th
| MATOMO\_URL | string \| undefined | undefined | Base url for matomo scripts and data reporting. This parameter is `required` if Matomo is `enabled` |
| MATOMO\_SITE\_ID | string \| undefined | undefined | Matomo Website ID. This parameter is required if `Matomo` is `enabled`. |
+#### Cache
+
+The Gateway offers the possibility to cache some of the most used queries (domains and algorithms queries). This cache use In-Memory data store.
+
+| name | type | default | description |
+| ----------------- | ------- | ------- | ------------------------------------------------------ |
+| CACHE\_ENABLED | boolean | true | Enable or disable the cache |
+| CACHE\_TTL | number | 1800 | Define (in seconds) time to live for cached elements. |
+| CACHE\_MAX\_ITEMS | number | 100 | Max items that can be cached at the same time |
+
### Overwrite parameters
These parameters can be overwrite by either
diff --git a/docs/for-developers/connector/README.md b/docs/for-developers/connector/README.md
deleted file mode 100644
index ac68aea26c481fac80b2d2d781254c095177f7c7..0000000000000000000000000000000000000000
--- a/docs/for-developers/connector/README.md
+++ /dev/null
@@ -1,2 +0,0 @@
-# Connector
-
diff --git a/docs/for-developers/frontend/visualisations.md b/docs/for-developers/frontend/visualisations.md
index 56395ce99f2db62122194abc64401a12c204838d..3b6e066e2afadcce097be8c10bba2aa5b321a7b0 100644
--- a/docs/for-developers/frontend/visualisations.md
+++ b/docs/for-developers/frontend/visualisations.md
@@ -1,6 +1,6 @@
# 📊 Visualisations
-To see the different possible visualisations in the frontend we have integrated [storybook.js](https://storybook.js.org) directly in the frontend.
+To see the different possible visualisations in the frontend we have integrated [storybook.js](https://storybook.js.org/) directly in the frontend.
Start the storybook by launching this command in the frontend folder
diff --git a/docs/for-developers/gateway/authentication.md b/docs/for-developers/gateway/authentication.md
index 6c8a63fa9d5e7bf3a606cca9069b135d6301f6f9..df126d6d6970db552e7c0b6f343da1ea49450fd2 100644
--- a/docs/for-developers/gateway/authentication.md
+++ b/docs/for-developers/gateway/authentication.md
@@ -1,6 +1,6 @@
# 🔑 Authentication
-The authentication implementation is based on [passport.js](https://www.passportjs.org) it allows a flexible way to implement different strategies inside the gateway. 
+The authentication implementation is based on [passport.js](https://www.passportjs.org/) it allows a flexible way to implement different strategies inside the gateway. 
For now the authentication system is quite simple and only use JWT. The real implementation of authorization and authentication is left to the connector/engine. 
@@ -10,7 +10,7 @@ The communication between the frontend and the gateway is handled by JWT token w
.png>)
-The gateway will handle the authentication process with the frontend in a unique fashion always using a JWT token. This token can contains information specific to some connector. For that purpose the user model contains a field `extraFields` which basically a dictionary. 
+The gateway will handle the authentication process with the frontend in a unique fashion always using a JWT token. This token can contains information specific to some connector. For that purpose the user model contains a field `extraFields` which is basically a dictionary. 
{% code title="user.model.ts" %}
```typescript
@@ -90,3 +90,18 @@ export interface IEngineService {
```
{% endcode %}
+#### How to get the user 
+
+Whether you use the local login or a 3rd party system, there is a unique way to access the user inside the Gateway. This method through the request : 
+
+```typescript
+request.user
+```
+
+This request's attribute is feed by strategy policies defined in the Gateway. Currently the following strategies are applied 
+
+1. JWT cookies
+2. JWT bearer
+3. Engine (use the connector to retrieve the user)
+
+Even if the `AUTH_SKIP` is defined you should be able to retrieve the user through the request.
diff --git a/docs/for-developers/gateway/cache.md b/docs/for-developers/gateway/cache.md
new file mode 100644
index 0000000000000000000000000000000000000000..f8ce081527c84be00f16aecf30b86d82d704cf4c
--- /dev/null
+++ b/docs/for-developers/gateway/cache.md
@@ -0,0 +1,27 @@
+---
+description: This page describe how the cache is managed inside the Gateway.
+---
+
+# 📠Cache
+
+The cache system is a In-Memory cache (but it can be changed to another cache system like Redis, see [nest documentation](https://docs.nestjs.com/techniques/caching)).
+
+In the following picture will see an overview of how a request is handled in the Gateway. 
+
+
+
+The engine service is like a proxy class for the connector, it implements the `Connector` interface as each connector does but it adds some overall functionality to all the connector like the cache.
+
+#### What is cached ?
+
+Currently only `domains` and `algorithms` are cached as they are the most used functions inside the Gateway.
+
+#### Cache life cycle
+
+The cache is create whenever a user makes a query to `domain` or `algorithms` and is user based, so a cache entry is created for every pair of `domains|algorithms-user.id`. 
+
+When the user logout, all entries linked to the user are cleared.
+
+
+
+ 
diff --git a/docs/for-developers/gateway/connector/README.md b/docs/for-developers/gateway/connector/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..e7dc8cf6e87c0e3d881700b99d06cc14e112a3a1
--- /dev/null
+++ b/docs/for-developers/gateway/connector/README.md
@@ -0,0 +1,2 @@
+# 🔌 Connectors
+
diff --git a/docs/for-developers/gateway/connector/create-a-connector.md b/docs/for-developers/gateway/connector/create-a-connector.md
new file mode 100644
index 0000000000000000000000000000000000000000..0e82f4ceab76c6be0e7a8d0ec4d41cc1c3a4fbd6
--- /dev/null
+++ b/docs/for-developers/gateway/connector/create-a-connector.md
@@ -0,0 +1,153 @@
+---
+description: This page describe how to create a connector
+---
+
+# Create a connector
+
+All connectors have a folder under `src/engine/connectors/`. If you want to create a new connector, the first thing to do is to create a folder with the name of your connector. Take care that the connector name should be unique. 
+
+Inside your folder `src/engine/connectors/example/` you need a file name after your connector for e.g. `src/engine/connectors/example/example.connector.ts`. The connector should have the same name as your folder.
+
+The connector instantiation is managed by the `engine.service.ts`.
+
+Here is a minimal implementation of a connector
+
+{% code title="example.connector.ts" %}
+```typescript
+import { NotImplementedException } from '@nestjs/common';
+import { ExperimentResult } from 'src/common/interfaces/utilities.interface';
+import Connector from 'src/engine/interfaces/connector.interface';
+import { Domain } from 'src/engine/models/domain.model';
+import { Algorithm } from 'src/engine/models/experiment/algorithm.model';
+import { User } from 'src/users/models/user.model';
+
+export default class LocalConnector implements Connector {
+ async login(): Promise<User> {
+ throw new NotImplementedException();
+ }
+
+ async getAlgorithms(): Promise<Algorithm[]> {
+ throw new NotImplementedException();
+ }
+
+ async runExperiment(): Promise<ExperimentResult[]> {
+ throw new NotImplementedException();
+ }
+
+ async getDomains(): Promise<Domain[]> {
+ throw new NotImplementedException();
+ }
+
+ async getActiveUser(): Promise<User> {
+ throw new NotImplementedException();
+ }
+}
+```
+{% endcode %}
+
+### Constructor
+
+The engine service will inject some properties into the connector through the constructor that the connector is free to use or not.
+
+```typescript
+ constructor(
+ private readonly options: EngineOptions,
+ private readonly httpService: HttpService,
+ private readonly engineService: EngineService,
+ ) {}
+```
+
+The first parameter, `options`, is a key-value store that contains the `ENGINE_TYPE` and the `ENGINE_BASE_URL`.
+
+The second parameter, `httpService`, is an `Axios` instance shared between all request.
+
+The third parameter, `engineService`, is the engine service which inject itself into the connector. This can be useful because engine service provide utility functions and use the cache system. So you should always call engine service when you want to access to `algorithms` and `domains` to avoid to access external resources.
+
+### Experiments
+
+The Gateway offers two possibilities to manage experiment 
+
+1. Experiments are managed (save, edit, delete, etc...) directly internally
+2. Experiments are management by the external engine.
+
+#### Case 1
+
+In case 1, we assume that the external engine will only offers the possibility to run experiment (not save them). In this case the connector should only implements `runExperiment`. The gateway will recognize that `runExperiment` is implemented and that it needs to manage the experiments internally.
+
+This is done by using a `PostgreSQL` database.
+
+#### Case 2
+
+In case 2, we let the external engine manage experiments by itself. This required that the connector implements `createExperiment` and not `runExperiment`.
+
+This also implies that the connector need to implements these functions
+
+* `getExperiment`
+* `listExperiments`
+* `removeExperiment`
+* `editExperiment`
+
+This will delegate all the CRUD work to the external engine. 
+
+### Authentication & users
+
+There are two possibilities of authentication both relay on a external resource. The only difference will be about how the user token is managed.
+
+1. Inside the Gateway
+2. Outside the Gateway
+
+#### Case 1
+
+In case 1, the identification is managed inside the gateway and the connector is responsible for making the authentication through the function `login`. 
+
+The Gateway will then manage the token between the Frontend and the Gateway with a Json Web Token.
+
+The logout is managed by the method `logout`. After calling `logout` from the connector, the gateway will remove the token from the user's headers.
+
+#### Case 2
+
+In case 2, identification and authentication will both be managed externally through a resource that is located under the URL `/services/sso` for example a [KeyCloak](https://www.keycloak.org/) (exareme is using this strategy). This implies that the `editActiveUser` is implemented and allows updating the user (needed if ToS is enabled).
+
+In both cases, the `getActiveUser` will retrieve the current user logged. 
+
+### Configuration
+
+The connector has it's own part of configuration, it's mainly parameters that are closely related to the connector and not really with the overall configuration.
+
+| name | default | description |
+| ----------- | ------- | ------------------------------------------------------------------------------------------------------- |
+| hasGrouping | false | Define if the connector is able to make query grouping by nominal variable (mainly used for histograms) |
+| hasGalaxy | false | `Deprecated`. Only used by Exareme engine |
+
+These elements can be configured by the function `getConfiguration`.
+
+#### Filter
+
+Filter configuration describe the types of variables that can be considered as number. These types are closely related to the connector as types is depending on the engine used. 
+
+The types that should be considered as numbers can be configured by defining the function `getFilterConfiguration`. 
+
+`['real', 'integer']` are the default types.
+
+#### Formula
+
+Formula configuration give a list of available variable operations. Each element contains two properties, the variable type and an operation's list.
+
+```json
+[
+ {
+ variableType: 'real',
+ operationTypes: ['log', 'center', ...]
+ },
+ ...
+]
+```
+
+To define operations, the connector should implements `getFormulaConfiguration`.
+
+
+
+ 
+
+ 
+
diff --git a/docs/for-developers/connector/Parsing-response-with-JSONata.md b/docs/for-developers/gateway/connector/parsing-response-with-jsonata.md
similarity index 100%
rename from docs/for-developers/connector/Parsing-response-with-JSONata.md
rename to docs/for-developers/gateway/connector/parsing-response-with-jsonata.md
diff --git a/docs/for-developers/gateway/static-files.md b/docs/for-developers/gateway/static-files.md
new file mode 100644
index 0000000000000000000000000000000000000000..22e6e2990acb2a8f7158092bebb306d82ddce6b9
--- /dev/null
+++ b/docs/for-developers/gateway/static-files.md
@@ -0,0 +1,20 @@
+---
+description: This page describe how the static files are managed
+---
+
+# 🗃 Static files
+
+As different connector implies different context, there is some cases where the Frontend needed to change depending on the connector currently used, the changes can be related to 
+
+1. Custom CSS
+2. Favicon / logo
+3. Terms of Service 
+4. Login page 
+
+A module so a module `files` has been create. It allows to have dynamic resources based on the connector currently used. 
+
+All static assets are stored under `assets/engines/*`, in this path you have one folder for each connector. By default, all resources are retrieve in the `default` folder. If you want to override a specific file for your connector you just need to create a file with the same name under your connector's folder `assets/engines/yourconnector`.
+
+#### Markdown files and static files
+
+For markdown files, there is a specific process which allows to define a placeholder for relative path. For example, if you want to make a reference to a static image inside the Gateway you cannot just use an absolute link to the resource as you don't know the domain that will be used. So for this specific case, you can use the placeholder `$ASSETS_URL$`. Which will be replaced dynamically at the markdown file's rendering.
diff --git a/docs/for-developers/gateway/users.md b/docs/for-developers/gateway/users.md
index 40dd3a1cd50388ab5dec4d11b2eee1a139d6a81a..c6ed6fd54ccb19a93f1394c80f02c16a8cc34993 100644
--- a/docs/for-developers/gateway/users.md
+++ b/docs/for-developers/gateway/users.md
@@ -9,7 +9,7 @@ The gateway is not meant to manage users directly. This is the engine's role to
### How it works ?
-Let's say we want to retrieve the current user, the gateway will ask through the connector for the user's data in the same time the gateway will look in his own database if it has some data for this user. Then both data are merged to fit the User model. Data from the engine have precedence over the gateway data in case of conflict.
+Let's say we want to retrieve the current user, the Gateway will ask through the connector for the user's data in the same time the gateway will look in his own database if it has some data for this user. Then both data are merged to fit the User model. Data from the engine have precedence over the gateway data in case of conflict.
{% code title="user.model.ts" %}
```typescript
@@ -42,27 +42,4 @@ After merging data from both source and make some integrity check the gateway wi
#### Update user profile
-So now we know that the data can be retrieve through two different sources, how will we handle updating our user profile ? The system is simple, the gateway will ask the connector if he can handle the user's update by looking if the function `updateUser` is defined in the connector. If it's defined it means that the engine can handle at least some part of the update, so we delay the work to the engine. Now if the engine cannot handle all the update data, the connector can decide to return some attributes back to the gateway. 
-
-{% code title="example return update data" %}
-```typescript
- async updateUser(
- request: Request,
- userId: string,
- data: UpdateUserInput
- ): Promise<UpdateUserInput | undefined> {
- const path = this.options.baseurl + 'user';
- const response = await firstValueFrom(
- this.post<string>(request, path, {
- prop1: data.attrib1,
- prop2: data.attrib2
- }),
- );
-
- const { attrib1, attrib2, ...subset } = UpdateUserInput // Subset of updateData
- return subset;
- }
-```
-{% endcode %}
-
-The returned attributes will be provided back to the gateway and will be handle internally as far as it can do it.
+So now we know that the data can be retrieve through two different sources, how will we handle updating our user profile ? The system is simple, the gateway will check the connector if he can handle the user's update by looking at the function `updateUser` is defined in the connector. If it's defined it means that the connector can handle the update, so we delay the work to the connector. If the connector cannot handle the update, the Gateway will handle it by itself.
diff --git a/docs/for-developers/get-started/Introduction.md b/docs/for-developers/get-started/Introduction.md
index 8e731eb6b1fa6edd7da4e56ffe8d4aee96d1d0d0..988fe00eea7fe0e67a353558e10493477b001dd5 100644
--- a/docs/for-developers/get-started/Introduction.md
+++ b/docs/for-developers/get-started/Introduction.md
@@ -6,7 +6,7 @@ description: Introduction for developers
The MIP is mainly composed by 3 components
-.png>)
+.png>)
* **Frontend** : user interface (React.js)
* **Gateway** : middleware used to abstract calls from an engine (Nest.js and GraphQL)
@@ -16,7 +16,7 @@ The MIP is mainly composed by 3 components
The Frontend and the Gateway are closely related and their communication are abstracted from the engine. This abstraction is performed by the Gateway.
-
+
The Gateway is in charge of communications with various engines, it could make simple calls and pass it to the Frontend or make any transformations needed in order to fit the need in the Frontend.
@@ -51,7 +51,7 @@ The code above is an example of a connector.
### Visualizations
-.png>)
+.png>)
With the Frontend we will introduce a new way to deal with visualizations. Previously the visualizations were completely manage by the engine. As a part of abstraction from a specific engine we want to be able to delegate this task to the visualization components.
diff --git a/docs/for-developers/get-started/Setup-development-environment.md b/docs/for-developers/get-started/Setup-development-environment.md
index 287f73db845ec46cee62b5c768b778e30602e02d..0d3984048dd1d71249f998d8ccc13c20240d4403 100644
--- a/docs/for-developers/get-started/Setup-development-environment.md
+++ b/docs/for-developers/get-started/Setup-development-environment.md
@@ -21,9 +21,9 @@ In this guide we will see how to setup the last two elements.
Make sure to have
-* [Node.js](https://nodejs.org)
-* [NPM](https://npmjs.com)
-* [Yarn](https://yarnpkg.com)
+* [Node.js](https://nodejs.org) (16.x)
+* [NPM](https://npmjs.com) (8.x)
+* [Yarn](https://yarnpkg.com) (1.22.x)
* [Docker](https://docs.docker.com/get-docker/) and [docker-compose](https://docs.docker.com/compose/install/)
installed in your computer.
@@ -44,7 +44,7 @@ npm install
#### Run the DB
-The gateway need a DB in order to work. [TypeORM](https://typeorm.io) is used to make the DB calls agnostic from the real implementation. 
+The gateway need a DB in order to work. [TypeORM](https://typeorm.io/) is used to make the DB calls agnostic from the real implementation. 
We provide a docker-compose to run a `postgres` DB, you can use it by running the following command
@@ -66,7 +66,7 @@ npm run start:dev
Once you have started the Gateway, you can play with the GraphQL playground that is automatically integrated within the gateway, follow this link : [http://127.0.0.1:8081/graphql](http://127.0.0.1:8081/graphql). You should be able to see something like this :
-.png>)
+.png>)
This environment is a tool provided by GraphQL to play with queries, mutations, etc...