From 5b23e35ad4ccbaa9be1f1b616872c55913a882d3 Mon Sep 17 00:00:00 2001
From: Brent Huisman <brenthuisman@users.noreply.github.com>
Date: Mon, 18 Jul 2022 09:22:56 +0200
Subject: [PATCH] Weekly CI Python wheel build pushes to Test.PyPI.org (#1921)
* - Weekly CI Python wheel build pushes to Test.PyPI.org
- Easy availability at https://test.pypi.org/project/arbor/
- Small change in documentation around VERSION
- Delete superfluous setup.py file.
---
.github/workflows/ciwheel.yml | 41 +++++++------
doc/contrib/release.rst | 112 ++++++++++++++++------------------
doc/cpp/index.rst | 1 -
doc/dev/index.rst | 1 +
doc/{cpp => dev}/version.rst | 2 +-
python/setup.py | 38 ------------
6 files changed, 78 insertions(+), 117 deletions(-)
rename doc/{cpp => dev}/version.rst (99%)
delete mode 100644 python/setup.py
diff --git a/.github/workflows/ciwheel.yml b/.github/workflows/ciwheel.yml
index fc202758..51a3408b 100644
--- a/.github/workflows/ciwheel.yml
+++ b/.github/workflows/ciwheel.yml
@@ -24,11 +24,14 @@ jobs:
with:
fetch-depth: 0
submodules: recursive
+ - name: Create unique VERSION
+ if: endsWith(github.ref,'ciwheel') || endsWith(github.ref,'rc') # https://docs.github.com/en/actions/learn-github-actions/expressions
+ run: python3 -c 'import time;f=open("VERSION","r+");content = f.readlines();f.seek(0);f.write(content[0].strip()+time.strftime("%Y%m%d%H%I%S"))'
- name: Install cibuildwheel
run: python3 -m pip install cibuildwheel
- name: Build wheels
run: python3 -m cibuildwheel --output-dir dist
- - uses: actions/upload-artifact@v2
+ - uses: actions/upload-artifact@v3
with:
name: dist
path: dist/*.whl
@@ -46,6 +49,9 @@ jobs:
with:
fetch-depth: 0
submodules: recursive
+ - name: Create unique VERSION
+ if: endsWith(github.ref,'ciwheel') || endsWith(github.ref,'rc') # https://docs.github.com/en/actions/learn-github-actions/expressions
+ run: python3 -c 'import time;f=open("VERSION","r+");content = f.readlines();f.seek(0);f.write(content[0].strip()+time.strftime("%Y%m%d%H%I%S"))'
- name: Make sdist
run: python3 -m build -s
- name: Install sdist
@@ -54,24 +60,23 @@ jobs:
run: python3 -m unittest discover -v -s python
- name: Run Python examples
run: scripts/run_python_examples.sh
- - uses: actions/upload-artifact@v2
+ - uses: actions/upload-artifact@v3
with:
name: dist
path: dist/*.tar.gz
-# TODO
- # upload_test_pypi:
- # name: upload to test pypi
- # runs-on: ubuntu-latest
- # needs: [build_binary_wheels, build_sdist]
- # steps:
- # - uses: actions/download-artifact@v2
- # with:
- # name: dist
- # - name: Publish distribution 📦 to Test PyPI
- # run: |
- # pip install twine
- # twine upload -r testpypi dist/*
- # env:
- # TWINE_USERNAME: __token__
- # TWINE_PASSWORD: ${{ secrets.ciwheeltest }}
+ upload_test_pypi:
+ name: upload to test pypi
+ runs-on: ubuntu-latest
+ needs: [build_binary_wheels, build_sdist]
+ steps:
+ - uses: actions/download-artifact@v3
+ with:
+ name: dist
+ - name: Publish distribution 📦 to Test PyPI
+ run: |
+ pip install twine
+ twine upload -r testpypi ./*
+ env:
+ TWINE_USERNAME: __token__
+ TWINE_PASSWORD: ${{ secrets.TESTPYPI_SECRET }}
diff --git a/doc/contrib/release.rst b/doc/contrib/release.rst
index f961fcd2..4962193e 100644
--- a/doc/contrib/release.rst
+++ b/doc/contrib/release.rst
@@ -38,98 +38,92 @@ Pre-release
Update tags/versions and test
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-0. Check README.md, ATTRIBUTIONS.md, CONTRIBUTING.md, RELEASE_NOTES.md
-1. Create new temp-branch ending in ``-rc``. E.g. ``v0.6-rc``
-2. Bump the ``VERSION`` file:
- https://github.com/arbor-sim/arbor/blob/master/VERSION
- Don't append ``-rc`` here, but if you do, remove it before releasing.
-3. Run all tests.
- - ``ciwheel.yml`` triggers when you push a branch called ``v*rc``, ON YOUR OWN REPO (so check ``github.com/$yourname/arbor/actions``). Make sure the tests pass.
- - This should catch many problems. For a manual check:
+#. Check if some files are up to date
+
+ - README.md, ATTRIBUTIONS.md, CONTRIBUTING.md, RELEASE_NOTES.md
- Verify MANIFEST.in (required for PyPI sdist)
- - Check Python/pip/PyPi metadata and scripts, e.g. ``setup.py``
- Double check that all examples/tutorials/etc are covered by CI
+ - Check Python/pip/PyPi metadata and scripts, e.g. ``setup.py``, ``pyproject.toml``
+
+#. Create new temp-branch ending in ``-rc``. E.g. ``v0.6-rc``
+#. Bump the ``VERSION`` file:
+
+ - See also :ref:`dev-version`
+ - Append ``-rc``. (Make sure there's no ``-dev``)
+
+#. Run all tests.
-Test the RC
-~~~~~~~~~~~
+ - ``ciwheel.yml`` triggers when you push a branch called ``v*rc``, ON YOUR OWN REPO (so check ``github.com/$yourname/arbor/actions``). Make sure the tests pass.
+ - ``ciwheel.yml`` pushes automatically to `Test.PyPI.org <https://test.pypi.org/project/arbor/>`_. Test (consider asking other OS-users):
-4. Collect artifact from the above GA run.
- In case you want to manually want to trigger ``ciwheel.yml`` GA, overwrite the ``ciwheel`` branch with the commit of your choosing and force push to Github.
-5. ``twine upload -r testpypi dist/*``
-6. Ask users to test the above, e.g.:
+ .. code-block:: bash
-.. code-block:: bash
+ python -m venv env && source env/bin/activate
+ pip install numpy
+ pip install -i https://test.pypi.org/simple/ arbor #should select the latest build, but doublecheck
+ python -c 'import arbor; print(arbor.__config__)'
- python -m venv env && source env/bin/activate
- pip install numpy
- pip install -i https://test.pypi.org/simple/ arbor==0.6-rc
- python -c 'import arbor; print(arbor.__config__)'
+ - Use build flags to test the source package: :ref:`in_python_adv`
+ - In case you want to manually want to trigger ``ciwheel.yml`` GA, overwrite the ``ciwheel`` branch with the commit of your choosing and force push to Github.
Release
-------
-0. Make sure ``ciwheel.yml`` passes tests, produced working wheels, and nobody reported problems testing the RC.
- Make sure ``VERSION`` does not end with ``-rc`` or ``-dev``
+#. Make sure ``ciwheel.yml`` passed all tests, produced working wheels, were installable from `Test.PyPI.org <https://test.pypi.org/project/arbor/>`_ and that no problems were reported.
+
+#. Change ``VERSION``. Make sure does not end with ``-rc`` or ``-dev``.
-1. Create tarball with
+#. Create tarball with
``scripts/create_tarball ~/loc/of/arbor tagname outputfile``
- - eg ``scripts/create_tarball /full/path/to/arbor v0.5.1 ~/arbor-v0.5.1-full.tar.gz``
+ - eg ``scripts/create_tarball /full/path/to/arbor v0.5.1 ~/arbor-v0.5.1-full.tar.gz``
+
+#. Update ``spack/package.py``. The checksum of the targz is the sha256sum.
-2. Tag and release: https://github.com/arbor-sim/arbor/releases
+#. Start a new release on Zenodo, this allocated a DOI, but you don't have to finish it right away. Add new Zenodo badge/link to docs/README.
- - on cmdline: git tag -a TAGNAME
- - git push upstream TAGNAME
- - Go to `GH tags`_ and click “…†and “Create releaseâ€
- - Go through merged PRs to come up with a changelog
- Or categorize/edit Github's autogenerated release notes.
- - add tarball to release, created in previous step.
+#. Tag
-3. [`AUTOMATED`_] push to git@gitlab.ebrains.eu:arbor-sim/arbor.git
+ - commit the above changes.
+ - on cmdline: git tag -a TAGNAME
+ - git push upstream TAGNAME
-4. Download output of wheel action associated to this release commit and extract (verify the wheels and
+#. [`AUTOMATED`_] push to git@gitlab.ebrains.eu:arbor-sim/arbor.git
+
+#. Download output of wheel action associated to this release commit and extract (verify the wheels and
source targz is in /dist)
- Of course, the above action must have passed the tests successfully.
+
+#. Create Github Release: https://github.com/arbor-sim/arbor/releases
-5. Upload to TEST pypi & verify
-
-.. code-block:: bash
-
- twine upload -r arborpypi dist/*
- python -m venv env && source env/bin/activate
- pip install numpy
- pip install -i https://test.pypi.org/simple/ arbor==0.6
- python -c 'import arbor; print(arbor.__config__)'
-
-6. Upload to pypi & verify
-
-.. code-block:: bash
+ - Go to `GH tags`_ and click “…†and “Create releaseâ€
+ - Categorize/edit Github's autogenerated release notes (alternatively go through merged PRs to come up with a changelog).
+ - add tarball to release, created in previous step.
- twine upload -r arborpypi dist/*
+#. Upload to pypi & verify
- python -m venv env && source env/bin/activate
- pip install arbor
- python -c 'import arbor; print(arbor.__config__)'
+ .. code-block:: bash
-7. Update spack package / Ebrains Lab
+ twine upload -r arborpypi dist/*
- - first, update ``spack/package.py``. The checksum of the targz is the sha256sum.
- - Then, use the file to `make PR here <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/arbor/package.py>`_
- - Make an MR `here <https://gitlab.ebrains.eu/technical-coordination/project-internal/devops/platform/ebrains-spack-builds/>`_
+ python -m venv env && source env/bin/activate
+ pip install arbor
+ python -c 'import arbor; print(arbor.__config__)'
-8. In the same PR with the update to `spack/package.py`, bump `VERSION` file.
+#. Update spack package / Ebrains Lab / Opensourcebrain
- - e.g. to 0.6.1-dev
+ - Spack upstream: `PR here <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/arbor/package.py>`_
+ - Ebrains Lab: `MR here <https://gitlab.ebrains.eu/technical-coordination/project-internal/devops/platform/ebrains-spack-builds/>`_
+ - OSB: update `dockerfile <https://github.com/OpenSourceBrain/OSBv2/blob/master/applications/jupyterlab/Dockerfile>`_ if needed.
-9. In the same PR with the update to `spack/package.py`, add new Zenodo badge/link.
+ - Make sure that `Notebooks <https://www.v2.opensourcebrain.org/repositories/38>`_ work on the version that their image is built with.
- - Start a new release on Zenodo, this allocated a DOI, but you don't have to finish it right away.
- OTOH, you can already make and upload the tarball in step 1.
Post Release
------------
+#. Make a new PR setting ``VERSION`` to the next with a trailing ``-dev``. E.g. if you just release ``3.14``, change ``VERSION`` to ``3.15-dev``
#. Update and submit Zenodo release if necessary.
#. Announce on our website
#. Announce on HBP newsletter newsletter@humanbrainproject.eu, HBP Twitter/socials evan.hancock@ebrains.eu
diff --git a/doc/cpp/index.rst b/doc/cpp/index.rst
index f4f437ec..bd97b673 100644
--- a/doc/cpp/index.rst
+++ b/doc/cpp/index.rst
@@ -23,4 +23,3 @@ A :cpp:type:`arb::recipe` describes a model, and a :cpp:type:`arb::simulation` i
simulation
profiler
cable_cell
- version
diff --git a/doc/dev/index.rst b/doc/dev/index.rst
index 674559d5..b20ca5a6 100644
--- a/doc/dev/index.rst
+++ b/doc/dev/index.rst
@@ -23,4 +23,5 @@ Here we document internal components of Arbor. These pages can be useful if you'
extending_catalogues
mechanism_abi
util
+ version
.. numerics
diff --git a/doc/cpp/version.rst b/doc/dev/version.rst
similarity index 99%
rename from doc/cpp/version.rst
rename to doc/dev/version.rst
index 05d56e09..f32158d9 100644
--- a/doc/cpp/version.rst
+++ b/doc/dev/version.rst
@@ -1,4 +1,4 @@
-.. _cppversion:
+.. _dev-version:
Version and build information
=============================
diff --git a/python/setup.py b/python/setup.py
deleted file mode 100644
index 915f3ca1..00000000
--- a/python/setup.py
+++ /dev/null
@@ -1,38 +0,0 @@
-import setuptools
-import os
-
-here = os.path.abspath(os.path.dirname(__file__))
-with open(os.path.join(here, "arbor/VERSION")) as version_file:
- version_ = version_file.read().strip()
-
-setuptools.setup(
- name="arbor",
- packages=["arbor"],
- version=version_,
- author="CSCS and FZJ",
- url="https://github.com/arbor-sim/arbor",
- description="High performance simulation of networks of multicompartment neurons.",
- long_description="",
- classifiers=[
- "Development Status :: 4 - Beta", # Upgrade to "5 - Production/Stable" on release.
- "Intended Audience :: Science/Research",
- "Topic :: Scientific/Engineering :: Build Tools",
- "License :: OSI Approved :: BSD License"
- "Programming Language :: Python :: 3.6",
- "Programming Language :: Python :: 3.7",
- "Programming Language :: Python :: 3.8",
- "Programming Language :: Python :: 3.9",
- ],
- project_urls={
- "Source": "https://github.com/arbor-sim/arbor",
- "Documentation": "https://docs.arbor-sim.org",
- "Bug Reports": "https://github.com/arbor-sim/arbor/issues",
- },
- package_data={
- "arbor": ["VERSION", "_arbor.*.so"],
- },
- python_requires=">=3.6",
- install_requires=[],
- setup_requires=[],
- zip_safe=False,
-)
--
GitLab