{
"cells": [
{
"cell_type": "markdown",
"id": "050880d9-4f02-4c91-92ab-d8c05c08e094",
"metadata": {},
"source": [
"# Install TVB \n",
"This notebook will set up TVB in the Ebrains environment directly and will setup TVB remotely on the HPC using Pyunicore. The notebooks have been tested on the JUSUF supercomputer in Juelich. Getting an account at this computing site is preferred. Important for the latter is a LDAP account which can be requested [here](https://judoor.fz-juelich.de/register). After acquiring an account and [project at fenix](https://wiki.ebrains.eu/bin/view/Collabs/fenix-icei/) you can setup TVB remotely as well. For local simulations with TVB you can stop after the 3rd step. The next 3 step will isntall TVB locally. The steps thereafter will install TVB on an HPC site and copy necessary files for running it. We start by installing it in the Ebrains environment. \n",
"\n",
"#### 1 Cloning TVB\n",
"First step is to clone a branch of the main tvb repo with the necessary files. The cloning is necessary till this [PR](https://github.com/the-virtual-brain/tvb-root/pull/564) is accepted. When accepted one can simply do: \"!pip install tvb-library\""
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "b4737901-8351-4d66-ae08-f3c3ef892f90",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cloning into 'main/tvb-root'...\n",
"POST git-upload-pack (177 bytes)\n",
"remote: Enumerating objects: 84104, done.\u001b[K\n",
"remote: Counting objects: 100% (110/110), done.\u001b[K\n",
"remote: Compressing objects: 100% (33/33), done.\u001b[K\n",
"remote: Total 84104 (delta 94), reused 80 (delta 77), pack-reused 83994\u001b[K\n",
"Receiving objects: 100% (84104/84104), 570.92 MiB | 15.34 MiB/s, done.\n",
"Resolving deltas: 100% (64840/64840), done.\n",
"Updating files: 100% (3747/3747), done.\n"
]
}
],
"source": [
"os.chdir(\"/mnt/user/drive/My Libraries/\")\n",
"!git clone -b rateml_ebrains_fix --single-branch -v https://github.com/the-virtual-brain/tvb-root.git main/tvb-root\n",
"\n",
"# in case of an update\n",
"# os.chdir(\"/mnt/user/drive/My Libraries/tvb-root\")\n",
"# !git pull"
]
},
{
"cell_type": "markdown",
"id": "36298051-a63e-48d6-964c-67b0069e09bd",
"metadata": {},
"source": [
"#### 2 Install TVB \n",
"Lets install TVB from the cloned repository. (Again this would not be necessary is the pip install would work)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "7451b2bd-139e-40c8-a3f4-2631fd50602a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Processing /mnt/user/drive/My Libraries/tvb-root/tvb_library\n",
"Requirement already satisfied: autopep8 in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (1.7.0)\n",
"Requirement already satisfied: lxml in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (4.7.1)\n",
"Requirement already satisfied: mako>=1.1.4 in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (1.2.2)\n",
"Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (3.5.1)\n",
"Requirement already satisfied: networkx in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (2.8.6)\n",
"Requirement already satisfied: numba in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (0.54.1)\n",
"Requirement already satisfied: numexpr in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (2.8.1)\n",
"Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (1.20.3)\n",
"Requirement already satisfied: pylems in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (0.5.9)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (2.28.1)\n",
"Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (1.7.3)\n",
"Requirement already satisfied: six in /usr/local/lib/python3.8/dist-packages (from tvb-library==2.5) (1.16.0)\n",
"Collecting typing\n",
" Downloading typing-3.7.4.3.tar.gz (78 kB)\n",
"\u001b[K |████████████████████████████████| 78 kB 6.8 MB/s eta 0:00:01\n",
"\u001b[?25hRequirement already satisfied: pycodestyle>=2.9.1 in /usr/local/lib/python3.8/dist-packages (from autopep8->tvb-library==2.5) (2.9.1)\n",
"Requirement already satisfied: toml in /usr/local/lib/python3.8/dist-packages (from autopep8->tvb-library==2.5) (0.10.2)\n",
"Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.8/dist-packages (from mako>=1.1.4->tvb-library==2.5) (2.1.1)\n",
"Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->tvb-library==2.5) (4.37.1)\n",
"Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.8/dist-packages (from matplotlib->tvb-library==2.5) (2.8.2)\n",
"Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->tvb-library==2.5) (21.3)\n",
"Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->tvb-library==2.5) (3.0.9)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->tvb-library==2.5) (1.4.4)\n",
"Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->tvb-library==2.5) (0.11.0)\n",
"Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->tvb-library==2.5) (9.2.0)\n",
"Requirement already satisfied: llvmlite<0.38,>=0.37.0rc1 in /usr/local/lib/python3.8/dist-packages (from numba->tvb-library==2.5) (0.37.0)\n",
"Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from numba->tvb-library==2.5) (45.2.0)\n",
"Requirement already satisfied: charset-normalizer<3,>=2 in /usr/local/lib/python3.8/dist-packages (from requests->tvb-library==2.5) (2.1.1)\n",
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->tvb-library==2.5) (1.26.12)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->tvb-library==2.5) (2022.6.15)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->tvb-library==2.5) (3.3)\n",
"Building wheels for collected packages: tvb-library, typing\n",
" Building wheel for tvb-library (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25h Created wheel for tvb-library: filename=tvb_library-2.5-py3-none-any.whl size=649121 sha256=8aa86d1591faf45420528f26602344ee964343f61be135e54e107ca316980888\n",
" Stored in directory: /tmp/pip-ephem-wheel-cache-byksa18_/wheels/d2/77/cd/d7ae3c811f8aa547302071ec1d172dbcb4bbb252f89f1a2251\n",
" Building wheel for typing (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25h Created wheel for typing: filename=typing-3.7.4.3-py3-none-any.whl size=26308 sha256=395c99fce6aa4dc0057f544acdd312445f4289b571fa6b24461994a15956c98b\n",
" Stored in directory: /tmp/cache/pip/wheels/5e/5d/01/3083e091b57809dad979ea543def62d9d878950e3e74f0c930\n",
"Successfully built tvb-library typing\n",
"Installing collected packages: typing, tvb-library\n",
"Successfully installed tvb-library-2.5 typing-3.7.4.3\n"
]
}
],
"source": [
"import os\n",
"os.chdir(\"/mnt/user/drive/My Libraries/tvb-root/tvb_library/\")\n",
"!pip install ."
]
},
{
"cell_type": "markdown",
"id": "a6729c2d-0416-4889-9429-5d6764ad920a",
"metadata": {},
"source": [
"#### 3 Installing a dependancy: TVB data"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "91eb377c-7116-4e90-9cbe-d7d9c4264429",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collecting tvb-data\n",
" Downloading tvb_data-2.0-py3-none-any.whl (51.9 MB)\n",
"\u001b[K |████████████████████████████████| 51.9 MB 15.8 MB/s eta 0:00:01 | 6.1 MB 15.8 MB/s eta 0:00:03/s eta 0:00:02��███████▋ | 28.5 MB 15.8 MB/s eta 0:00:02��█████████████████████ | 35.7 MB 15.8 MB/s eta 0:00:02 | 43.1 MB 15.8 MB/s eta 0:00:01\n",
"\u001b[?25hInstalling collected packages: tvb-data\n",
"Successfully installed tvb-data-2.0\n"
]
}
],
"source": [
"!pip install tvb-data"
]
},
{
"cell_type": "markdown",
"id": "924cc504-7fa4-4728-93a1-fc6d72906346",
"metadata": {},
"source": [
"### Remote install on HPC \n",
"These need to be followed for doing the parameter sweeps on the GPU. If you only want local simulations you can stop here"
]
},
{
"cell_type": "markdown",
"id": "b29434fd-487f-49d6-a01c-096f249c8185",
"metadata": {},
"source": [
"#### 4 Setup PyUnicore\n",
"The following steps will setup pyunicore needed to transfer and execute data and script on HPC. Be sure to set your project at the PROJECT variable. "
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "ad3170da-c191-48d1-9e28-640487859cc4",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'role': {'selected': 'user', 'availableRoles': ['user']},\n",
" 'queues': {'availableQueues': ['batch', 'gpus', 'develgpus'],\n",
" 'selected': 'batch'},\n",
" 'dn': 'UID=m.van.der.vlag@fz-juelich.de',\n",
" 'xlogin': {'UID': 'vandervlag1',\n",
" 'availableGroups': ['icei-hbp-2021-0003', 'icei-hbp-2021-0012', 'cslns'],\n",
" 'availableUIDs': ['vandervlag1'],\n",
" 'group': 'icei-hbp-2021-0003'}}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import json\n",
"import os\n",
"import pyunicore.client as unicore_client\n",
"# !pip install pyunicore --upgrade\n",
"\n",
"token = clb_oauth.get_token()\n",
"tr = unicore_client.Transport(token)\n",
"r = unicore_client.Registry(tr, unicore_client._HBP_REGISTRY_URL)\n",
"HPC_LOC = \"https://zam2125.zam.kfa-juelich.de:9112/JUSUF/rest/core\"\n",
"\n",
"# Enter your project below, 'icei-hbp-2021-0003' is an example LDAP project\n",
"# PROJECT = '!!enter your project here!!'\n",
"PROJECT = 'icei-hbp-2021-0003'\n",
"\n",
"tr.preferences=\"group:\" + PROJECT\n",
"site = unicore_client.Client(transport=tr,site_url=HPC_LOC)\n",
"USER = site.access_info()['xlogin']['UID']\n",
"\n",
"STOR_LOC = 'https://zam2125.zam.kfa-juelich.de:9112/JUDAC/rest/core'\n",
"base_url = STOR_LOC + \"/storages/PROJECT/\"\n",
"storage = unicore_client.Storage(tr, base_url)\n",
"\n",
"site.access_info()\n",
"# r.site_urls\n",
"# site.get_storages()"
]
},
{
"cell_type": "markdown",
"id": "f2740a7f-b6d4-41ed-b541-0f6fa3f4afda",
"metadata": {},
"source": [
"#### 5 Copy TVB files to the remote site\n",
"\n",
"This will execute a setup script on HPC cloning TVB, installing dependancies and setting PATHs correctly. First we copy 3 necessary files to: \\\\$yourproject/$youruser. Before copying these files be sure that 'jusuf.sh' is set for with personal project. You will find this file in the root of the main directory."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "dcd4c13a-00a2-4691-b589-707541ac5e18",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"copied: /opt/app-root/src/drive/My Libraries/main/activate.sh\n",
"copied: /opt/app-root/src/drive/My Libraries/main/jusuf.sh\n",
"copied: /opt/app-root/src/drive/My Libraries/main/setup.sh\n"
]
}
],
"source": [
"LOCFILES = \"/opt/app-root/src/drive/My Libraries/main/\"\n",
"files = ['activate.sh', 'jusuf.sh', 'setup.sh']\n",
"for sfile in files:\n",
" source = LOCFILES + sfile\n",
" destination = USER + '/' + sfile\n",
" storage.upload(source, destination)\n",
" print('copied:', source)"
]
},
{
"cell_type": "markdown",
"id": "edb66647-235d-4a10-ad2e-5d3922340283",
"metadata": {},
"source": [
"#### 6 Setting up the job for remote TVB installation\n",
"The following will setup the pyunicore job for remote execution of setup.sh which will install TVB on the remote site. It will also activate the PROJECT by \"jutil env activate - p *yourproject*"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "0aac7288-5ffc-4b68-b655-22be087a0efa",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'Executable': 'jutil env activate -p icei-hbp-2021-0003; bash $PROJECT/$USER/setup.sh',\n",
" 'RunOnLoginNode': 'true',\n",
" 'Job type': 'interactive',\n",
" 'Imports': [],\n",
" 'Exports': [],\n",
" 'Resources': {}}"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# setting up the parameters sweep \n",
"setup = {}\n",
"# setting up the environment and executing the sh file in 1 go\n",
"setup['Executable'] = \"jutil env activate -p \" + PROJECT +\"; bash $PROJECT/$USER/setup.sh\"\n",
"setup['RunOnLoginNode'] = \"true\"\n",
"setup['Job type'] = \"interactive\"\n",
"setup['Imports'] = []\n",
"setup['Exports'] = []\n",
"setup['Resources'] = {}\n",
"setup"
]
},
{
"cell_type": "markdown",
"id": "de24eaf3-e7d8-4ecb-8c79-1549b8f95f54",
"metadata": {},
"source": [
"#### 7 Submit the previously defined job\n",
"Executing the next code will install TVB remotely."
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "b14d7104-011e-4716-b4c5-b60213d2e9aa",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Job finished!\n"
]
},
{
"data": {
"text/plain": [
"{'stderr': [\"Cloning into '/p/project/icei-hbp-2021-0003/vandervlag1/tvb-root'...\\n\",\n",
" 'POST git-upload-pack (436 bytes)\\n',\n",
" 'POST git-upload-pack (182 bytes)\\n',\n",
" 'Updating files: 6% (227/3747)\\rUpdating files: 7% (263/3747)\\rUpdating files: 8% (300/3747)\\rUpdating files: 9% (338/3747)\\rUpdating files: 10% (375/3747)\\rUpdating files: 11% (413/3747)\\rUpdating files: 11% (437/3747)\\rUpdating files: 12% (450/3747)\\rUpdating files: 13% (488/3747)\\rUpdating files: 14% (525/3747)\\rUpdating files: 15% (563/3747)\\rUpdating files: 16% (600/3747)\\rUpdating files: 17% (637/3747)\\rUpdating files: 17% (659/3747)\\rUpdating files: 18% (675/3747)\\rUpdating files: 19% (712/3747)\\rUpdating files: 20% (750/3747)\\rUpdating files: 20% (775/3747)\\rUpdating files: 21% (787/3747)\\rUpdating files: 22% (825/3747)\\rUpdating files: 23% (862/3747)\\rUpdating files: 24% (900/3747)\\rUpdating files: 25% (937/3747)\\rUpdating files: 26% (975/3747)\\rUpdating files: 26% (987/3747)\\rUpdating files: 27% (1012/3747)\\rUpdating files: 28% (1050/3747)\\rUpdating files: 29% (1087/3747)\\rUpdating files: 30% (1125/3747)\\rUpdating files: 31% (1162/3747)\\rUpdating files: 31% (1176/3747)\\rUpdating files: 32% (1200/3747)\\rUpdating files: 33% (1237/3747)\\rUpdating files: 34% (1274/3747)\\rUpdating files: 35% (1312/3747)\\rUpdating files: 35% (1313/3747)\\rUpdating files: 36% (1349/3747)\\rUpdating files: 37% (1387/3747)\\rUpdating files: 38% (1424/3747)\\rUpdating files: 39% (1462/3747)\\rUpdating files: 39% (1479/3747)\\rUpdating files: 40% (1499/3747)\\rUpdating files: 41% (1537/3747)\\rUpdating files: 42% (1574/3747)\\rUpdating files: 43% (1612/3747)\\rUpdating files: 44% (1649/3747)\\rUpdating files: 45% (1687/3747)\\rUpdating files: 46% (1724/3747)\\rUpdating files: 47% (1762/3747)\\rUpdating files: 47% (1781/3747)\\rUpdating files: 48% (1799/3747)\\rUpdating files: 49% (1837/3747)\\rUpdating files: 50% (1874/3747)\\rUpdating files: 51% (1911/3747)\\rUpdating files: 52% (1949/3747)\\rUpdating files: 53% (1986/3747)\\rUpdating files: 53% (1997/3747)\\rUpdating files: 54% (2024/3747)\\rUpdating files: 55% (2061/3747)\\rUpdating files: 56% (2099/3747)\\rUpdating files: 57% (2136/3747)\\rUpdating files: 58% (2174/3747)\\rUpdating files: 58% (2185/3747)\\rUpdating files: 59% (2211/3747)\\rUpdating files: 60% (2249/3747)\\rUpdating files: 61% (2286/3747)\\rUpdating files: 62% (2324/3747)\\rUpdating files: 63% (2361/3747)\\rUpdating files: 64% (2399/3747)\\rUpdating files: 64% (2431/3747)\\rUpdating files: 65% (2436/3747)\\rUpdating files: 66% (2474/3747)\\rUpdating files: 67% (2511/3747)\\rUpdating files: 67% (2542/3747)\\rUpdating files: 68% (2548/3747)\\rUpdating files: 69% (2586/3747)\\rUpdating files: 70% (2623/3747)\\rUpdating files: 71% (2661/3747)\\rUpdating files: 72% (2698/3747)\\rUpdating files: 72% (2718/3747)\\rUpdating files: 73% (2736/3747)\\rUpdating files: 74% (2773/3747)\\rUpdating files: 75% (2811/3747)\\rUpdating files: 76% (2848/3747)\\rUpdating files: 77% (2886/3747)\\rUpdating files: 78% (2923/3747)\\rUpdating files: 79% (2961/3747)\\rUpdating files: 80% (2998/3747)\\rUpdating files: 81% (3036/3747)\\rUpdating files: 82% (3073/3747)\\rUpdating files: 83% (3111/3747)\\rUpdating files: 84% (3148/3747)\\rUpdating files: 84% (3179/3747)\\rUpdating files: 85% (3185/3747)\\rUpdating files: 86% (3223/3747)\\rUpdating files: 87% (3260/3747)\\rUpdating files: 87% (3273/3747)\\rUpdating files: 88% (3298/3747)\\rUpdating files: 89% (3335/3747)\\rUpdating files: 90% (3373/3747)\\rUpdating files: 90% (3391/3747)\\rUpdating files: 91% (3410/3747)\\rUpdating files: 92% (3448/3747)\\rUpdating files: 93% (3485/3747)\\rUpdating files: 94% (3523/3747)\\rUpdating files: 95% (3560/3747)\\rUpdating files: 96% (3598/3747)\\rUpdating files: 96% (3615/3747)\\rUpdating files: 97% (3635/3747)\\rUpdating files: 98% (3673/3747)\\rUpdating files: 99% (3710/3747)\\rUpdating files: 100% (3747/3747)\\rUpdating files: 100% (3747/3747), done.\\n'],\n",
" 'stdout': ['Requirement already satisfied: tqdm in /p/home/jusers/vandervlag1/jusuf/.local/lib/python3.9/site-packages (4.64.0)\\n',\n",
" 'Requirement already satisfied: tvb-data in /p/home/jusers/vandervlag1/jusuf/.local/lib/python3.9/site-packages (2.0)\\n']}"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# submit job\n",
"gc = site.new_job(job_description=setup)\n",
"\n",
"gc.poll()\n",
"print('Job finished!')\n",
"\n",
"result_job = {}\n",
"wd = gc.working_dir\n",
"result_job[\"stderr\"] = [x.decode('utf8') for x in wd.stat(\"/stderr\").raw().readlines()]\n",
"result_job[\"stdout\"] = [x.decode('utf8') for x in wd.stat(\"/stdout\").raw().readlines()]\n",
"result_job"
]
},
{
"cell_type": "markdown",
"id": "4d523799-4d50-4bbe-b559-dc186438bbae",
"metadata": {},
"source": [
"#### 8 Restart kernel\n",
"When all is done a kernel restart might be required if a notebook was already active. From dropdown menu: Kernel > Restart Kernel ..."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}