Weekly CI Python wheel build pushes to Test.PyPI.org (#1921)

* - Weekly CI Python wheel build pushes to Test.PyPI.org
    - Easy availability at https://test.pypi.org/project/arbor/
- Small change in documentation around VERSION
- Delete superfluous setup.py file.

.github/workflows/ciwheel.yml

@@ -24,11 +24,14 @@ jobs:
         with:
           fetch-depth: 0
           submodules: recursive
+      - name: Create unique VERSION
+        if: endsWith(github.ref,'ciwheel') || endsWith(github.ref,'rc') # https://docs.github.com/en/actions/learn-github-actions/expressions
+        run: python3 -c 'import time;f=open("VERSION","r+");content = f.readlines();f.seek(0);f.write(content[0].strip()+time.strftime("%Y%m%d%H%I%S"))'
       - name: Install cibuildwheel
         run: python3 -m pip install cibuildwheel
       - name: Build wheels
         run: python3 -m cibuildwheel --output-dir dist
-      - uses: actions/upload-artifact@v2
+      - uses: actions/upload-artifact@v3
         with:
           name: dist
           path: dist/*.whl
@@ -46,6 +49,9 @@ jobs:
         with:
           fetch-depth: 0
           submodules: recursive
+      - name: Create unique VERSION
+        if: endsWith(github.ref,'ciwheel') || endsWith(github.ref,'rc') # https://docs.github.com/en/actions/learn-github-actions/expressions
+        run: python3 -c 'import time;f=open("VERSION","r+");content = f.readlines();f.seek(0);f.write(content[0].strip()+time.strftime("%Y%m%d%H%I%S"))'
       - name: Make sdist
         run: python3 -m build -s
       - name: Install sdist
@@ -54,24 +60,23 @@ jobs:
         run: python3 -m unittest discover -v -s python
       - name: Run Python examples
         run: scripts/run_python_examples.sh
-      - uses: actions/upload-artifact@v2
+      - uses: actions/upload-artifact@v3
         with:
           name: dist
           path: dist/*.tar.gz
 
-# TODO
-  # upload_test_pypi:
-  #   name: upload to test pypi
-  #   runs-on: ubuntu-latest
-  #   needs: [build_binary_wheels, build_sdist]
-  #   steps:
-  #     - uses: actions/download-artifact@v2
-  #       with:
-  #         name: dist
-  #     - name: Publish distribution 📦 to Test PyPI
-  #       run: |
-  #         pip install twine
-  #         twine upload -r testpypi dist/*
-  #       env:
-  #         TWINE_USERNAME: __token__
-  #         TWINE_PASSWORD: ${{ secrets.ciwheeltest }}
+  upload_test_pypi:
+    name: upload to test pypi
+    runs-on: ubuntu-latest
+    needs: [build_binary_wheels, build_sdist]
+    steps:
+      - uses: actions/download-artifact@v3
+        with:
+          name: dist
+      - name: Publish distribution 📦 to Test PyPI
+        run: |
+          pip install twine
+          twine upload -r testpypi ./*
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.TESTPYPI_SECRET }}

doc/contrib/release.rst

@@ -38,98 +38,92 @@ Pre-release
 Update tags/versions and test
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-0. Check README.md, ATTRIBUTIONS.md, CONTRIBUTING.md, RELEASE_NOTES.md
-1. Create new temp-branch ending in ``-rc``. E.g. ``v0.6-rc``
-2. Bump the ``VERSION`` file:
-   https://github.com/arbor-sim/arbor/blob/master/VERSION
-   Don't append ``-rc`` here, but if you do, remove it before releasing.
-3. Run all tests.
-   - ``ciwheel.yml`` triggers when you push a branch called ``v*rc``, ON YOUR OWN REPO (so check ``github.com/$yourname/arbor/actions``). Make sure the tests pass.
-   - This should catch many problems. For a manual check:
+#. Check if some files are up to date
+    
+   - README.md, ATTRIBUTIONS.md, CONTRIBUTING.md, RELEASE_NOTES.md
    - Verify MANIFEST.in (required for PyPI sdist)
-   - Check Python/pip/PyPi metadata and scripts, e.g. ``setup.py``
    - Double check that all examples/tutorials/etc are covered by CI
+   - Check Python/pip/PyPi metadata and scripts, e.g. ``setup.py``, ``pyproject.toml``
+
+#. Create new temp-branch ending in ``-rc``. E.g. ``v0.6-rc``
+#. Bump the ``VERSION`` file:
+
+   - See also :ref:`dev-version`
+   - Append ``-rc``. (Make sure there's no ``-dev``)
+
+#. Run all tests.
 
-Test the RC
-~~~~~~~~~~~
+   - ``ciwheel.yml`` triggers when you push a branch called ``v*rc``, ON YOUR OWN REPO (so check ``github.com/$yourname/arbor/actions``). Make sure the tests pass.
+   - ``ciwheel.yml`` pushes automatically to `Test.PyPI.org <https://test.pypi.org/project/arbor/>`_. Test (consider asking other OS-users):
 
-4. Collect artifact from the above GA run.
-   In case you want to manually want to trigger ``ciwheel.yml`` GA, overwrite the ``ciwheel`` branch with the commit of your choosing and force push to Github.
-5. ``twine upload -r testpypi dist/*``
-6. Ask users to test the above, e.g.:
+     .. code-block:: bash
 
-.. code-block:: bash
+        python -m venv env && source env/bin/activate
+        pip install numpy
+        pip install -i https://test.pypi.org/simple/ arbor #should select the latest build, but doublecheck
+        python -c 'import arbor; print(arbor.__config__)'
 
-   python -m venv env && source env/bin/activate
-   pip install numpy
-   pip install -i https://test.pypi.org/simple/ arbor==0.6-rc
-   python -c 'import arbor; print(arbor.__config__)'
+   - Use build flags to test the source package: :ref:`in_python_adv`
+   - In case you want to manually want to trigger ``ciwheel.yml`` GA, overwrite the ``ciwheel`` branch with the commit of your choosing and force push to Github.
 
 Release
 -------
 
-0. Make sure ``ciwheel.yml`` passes tests, produced working wheels, and nobody reported problems testing the RC.
-   Make sure ``VERSION`` does not end with ``-rc`` or ``-dev``
+#. Make sure ``ciwheel.yml`` passed all tests, produced working wheels, were installable from `Test.PyPI.org <https://test.pypi.org/project/arbor/>`_ and that no problems were reported.
+   
+#. Change ``VERSION``. Make sure does not end with ``-rc`` or ``-dev``.
 
-1. Create tarball with
+#. Create tarball with
    ``scripts/create_tarball ~/loc/of/arbor tagname outputfile``
 
-   -  eg ``scripts/create_tarball /full/path/to/arbor v0.5.1 ~/arbor-v0.5.1-full.tar.gz``
+   - eg ``scripts/create_tarball /full/path/to/arbor v0.5.1 ~/arbor-v0.5.1-full.tar.gz``
+   
+#. Update ``spack/package.py``. The checksum of the targz is the sha256sum.
 
-2. Tag and release: https://github.com/arbor-sim/arbor/releases
+#. Start a new release on Zenodo, this allocated a DOI, but you don't have to finish it right away. Add new Zenodo badge/link to docs/README.
 
-   -  on cmdline: git tag -a TAGNAME
-   -  git push upstream TAGNAME
-   -  Go to `GH tags`_ and click “…” and “Create release”
-   -  Go through merged PRs to come up with a changelog
-      Or categorize/edit Github's autogenerated release notes.
-   - add tarball to release, created in previous step.
+#. Tag
 
-3. [`AUTOMATED`_] push to git@gitlab.ebrains.eu:arbor-sim/arbor.git
+   - commit the above changes.
+   - on cmdline: git tag -a TAGNAME
+   - git push upstream TAGNAME
 
-4. Download output of wheel action associated to this release commit and extract (verify the wheels and
+#. [`AUTOMATED`_] push to git@gitlab.ebrains.eu:arbor-sim/arbor.git
+
+#. Download output of wheel action associated to this release commit and extract (verify the wheels and
    source targz is in /dist)
 
    - Of course, the above action must have passed the tests successfully.
+   
+#. Create Github Release: https://github.com/arbor-sim/arbor/releases
 
-5. Upload to TEST pypi & verify
-
-.. code-block:: bash
-
-   twine upload -r arborpypi dist/*
-   python -m venv env && source env/bin/activate
-   pip install numpy
-   pip install -i https://test.pypi.org/simple/ arbor==0.6
-   python -c 'import arbor; print(arbor.__config__)'
-
-6. Upload to pypi & verify
-
-.. code-block:: bash
+   - Go to `GH tags`_ and click “…” and “Create release”
+   - Categorize/edit Github's autogenerated release notes (alternatively go through merged PRs to come up with a changelog).
+   - add tarball to release, created in previous step.
 
-   twine upload -r arborpypi dist/*
+#. Upload to pypi & verify
 
-   python -m venv env && source env/bin/activate
-   pip install arbor
-   python -c 'import arbor; print(arbor.__config__)'
+   .. code-block:: bash
 
-7. Update spack package / Ebrains Lab
+      twine upload -r arborpypi dist/*
 
-   -  first, update ``spack/package.py``. The checksum of the targz is the sha256sum.
-   -  Then, use the file to `make PR here <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/arbor/package.py>`_
-   -  Make an MR `here <https://gitlab.ebrains.eu/technical-coordination/project-internal/devops/platform/ebrains-spack-builds/>`_
+      python -m venv env && source env/bin/activate
+      pip install arbor
+      python -c 'import arbor; print(arbor.__config__)'
 
-8. In the same PR with the update to `spack/package.py`, bump `VERSION` file.
+#. Update spack package / Ebrains Lab / Opensourcebrain
 
-   - e.g. to 0.6.1-dev
+   - Spack upstream: `PR here <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/arbor/package.py>`_
+   - Ebrains Lab: `MR here <https://gitlab.ebrains.eu/technical-coordination/project-internal/devops/platform/ebrains-spack-builds/>`_
+   - OSB: update `dockerfile <https://github.com/OpenSourceBrain/OSBv2/blob/master/applications/jupyterlab/Dockerfile>`_ if needed.
 
-9. In the same PR with the update to `spack/package.py`, add new Zenodo badge/link.
+     - Make sure that `Notebooks <https://www.v2.opensourcebrain.org/repositories/38>`_ work on the version that their image is built with.
 
-   - Start a new release on Zenodo, this allocated a DOI, but you don't have to finish it right away.
-     OTOH, you can already make and upload the tarball in step 1.
 
 Post Release
 ------------
 
+#. Make a new PR setting ``VERSION`` to the next with a trailing ``-dev``. E.g. if you just release ``3.14``, change ``VERSION`` to ``3.15-dev``
 #. Update and submit Zenodo release if necessary.
 #. Announce on our website
 #. Announce on HBP newsletter newsletter@humanbrainproject.eu, HBP Twitter/socials evan.hancock@ebrains.eu

doc/cpp/index.rst

@@ -23,4 +23,3 @@ A :cpp:type:`arb::recipe` describes a model, and a :cpp:type:`arb::simulation` i
    simulation
    profiler
    cable_cell
-   version

doc/dev/index.rst

@@ -23,4 +23,5 @@ Here we document internal components of Arbor. These pages can be useful if you'
    extending_catalogues
    mechanism_abi
    util
+   version
 .. numerics

doc/cpp/version.rst → doc/dev/version.rst

-.. _cppversion:
+.. _dev-version:
 
 Version and build information
 =============================

python/setup.py

-import setuptools
-import os
-
-here = os.path.abspath(os.path.dirname(__file__))
-with open(os.path.join(here, "arbor/VERSION")) as version_file:
-    version_ = version_file.read().strip()
-
-setuptools.setup(
-    name="arbor",
-    packages=["arbor"],
-    version=version_,
-    author="CSCS and FZJ",
-    url="https://github.com/arbor-sim/arbor",
-    description="High performance simulation of networks of multicompartment neurons.",
-    long_description="",
-    classifiers=[
-        "Development Status :: 4 - Beta",  # Upgrade to "5 - Production/Stable" on release.
-        "Intended Audience :: Science/Research",
-        "Topic :: Scientific/Engineering :: Build Tools",
-        "License :: OSI Approved :: BSD License"
-        "Programming Language :: Python :: 3.6",
-        "Programming Language :: Python :: 3.7",
-        "Programming Language :: Python :: 3.8",
-        "Programming Language :: Python :: 3.9",
-    ],
-    project_urls={
-        "Source": "https://github.com/arbor-sim/arbor",
-        "Documentation": "https://docs.arbor-sim.org",
-        "Bug Reports": "https://github.com/arbor-sim/arbor/issues",
-    },
-    package_data={
-        "arbor": ["VERSION", "_arbor.*.so"],
-    },
-    python_requires=">=3.6",
-    install_requires=[],
-    setup_requires=[],
-    zip_safe=False,
-)