Skip to content

Dev/mip 17/parallel processing node tasks

Kostas FILIPPOPOLITIS requested to merge dev/MIP-17/parallel_processing_node_tasks into master

Created by: KFilippopolitis

Changelog:

NODE Celery

  1. The Celery Pool is no longer Prefork(multiprocessing), it was changed to an eventlet pool (Greenlets).
  2. The NODE runs with 16 greenlets by default. More testing is needed, blocked by ticket.

Monetdb Connection The MonetDBConnection was refactored, instead of a Singleton, to a Connection Pool. Each query uses a connection from the pool.

Query Execution On the query execution level locks have been added.

  1. create_remote_table_query_lock is needed in order to avoid a bug found in MonetDB. Jira-614
  2. create_function_query_lock is needed in order to avoid the exception of 'CREATE OR REPLACE FUNCTION: transaction conflict detected'.
  3. insert_query_lock is needed to avoid the parallel execution of udfs in the database. This is needed in order to preserve the zero cost that MonetDB is providing.

Logging A logger will be initialized for each task, instead of being a singleton, uniquely identified by the celery task id.

Merge request reports