Maintenance Prod Rancher @ JSC

All prod k8s based services will be down tonight, the 31st of March at 20:30 CEST for 30 minutes for a critical update.

changed the description

List of services that will be affected:

atlases.ebrains.eu
bluenaas-brain-area.apps.ebrains.eu
bluenaas-morph-viewer.apps.ebrains.eu
bluenaas-single-cell-svc.apps.ebrains.eu
bluenaas-single-cell.apps.ebrains.eu
bluenaas-small-area.apps.ebrains.eu
bluenaas-subcellular-rest.apps.ebrains.eu
bluenaas-subcellular.apps.ebrains.eu
collab-file-cloner.apps.ebrains.eu
core.kg-ppd.ebrains.eu
cron.kg-ppd.ebrains.eu
deepzoom.apps.ebrains.eu
dzip-svc.apps.ebrains.eu
dzseriesmaker.apps.ebrains.eu
dzseriesviewer.apps.ebrains.eu
editor.kg-ppd.ebrains.eu
grafana.ebrains.eu
handbook.ebrains.eu
lab.jsc.ebrains.eu
live-papers.apps.ebrains.eu
localizoom.apps.ebrains.eu
meshview.apps.ebrains.eu
metadata-wizard-dev.apps.ebrains.eu
metadata-wizard.apps.ebrains.eu
model-catalog.apps.ebrains.eu
model-validation-api.apps.ebrains.eu
neoviewer.apps.ebrains.eu
nest-desktop-v3.apps.ebrains.eu
nest-desktop.apps.ebrains.eu
neuromorphic-job-manager.apps.ebrains.eu
notes.ebrains.eu
pynutil.apps.ebrains.eu
query.kg-ppd.ebrains.eu
restrictedaccess.apps.ebrains.eu
rodentworkbench.apps.ebrains.eu
search-indexer.kg-ppd.ebrains.eu
search.kg-ppd.ebrains.eu
servicemeta.apps.ebrains.eu
siibra-api.apps.ebrains.eu
siibra-explorer.apps.ebrains.eu
stats.kg-ppd.ebrains.eu
tvb-bck.apps.ebrains.eu
tvb-web.apps.ebrains.eu
tvb.apps.ebrains.eu
voluba.apps.ebrains.eu
webalign.apps.ebrains.eu
webwarp.apps.ebrains.eu

changed title from Maintenance Rancher @ JSC to Maintenance Prod Rancher @ JSC

added Ops label

added Priority::High label

changed due date to March 31, 2025

maintenance is done / finished

an incident need to be created for TLS certificates updates (issue after? maintenance)

from https://chat.ebrains.eu/channel/tech-support comment https://chat.ebrains.eu/channel/tech-support?msg=LwCmDMutA6FgakySu

Adding point as-is in case rocketchat unavailable at any moment:

Nikos Pappas @npappas2:49 PM

Information about the tls issues on the rke2-1-jsccloud Kubernetes cluster:

• The ingress-controller was updated on 31st of March at 20:30 CEST due a CVE found on the ingress-controller.
• The previous version ran an RKE2 variant of ingress-nginx (something like a fork of the main helm chart) and in that version, and it "adopted" all the empty spec.ingressClassName.

In other words if you hadn't set spec.ingressClassName it defaulted to the nginx by default.

• The new patched version did not. That's why we had this problem. It wasn't a TLS or certificate problem. The services just didn't know what ingress to use.
• Measures taken: We have set the --watch-ingress-without-class=true. --watch-ingress-without-class=true on the controller tells the controller whether to process Ingress resources that don’t explicitly declare an ingressClassName or kubernetes.io/ingress.class annotation.

closed