...
9/05/23 - pgwrite bugfix rollout on external gateways | deemed irrelevant to the incident
Batch farm:
5th May: ` wn-2020-xma - wn-2022-lenovo` will be set to drain.
9th May :
Merge the sandbox http://aquilon.gridpp.rl.ac.uk/sandboxes/diff.php?sandbox=update_cvmfs_client into production.
Drained workers will have the package `egi-cvmfs` manually removed.
Manually remove the `xrootd-proxy.service` (/etc/systemd/system/xrootd-proxy.service)
Manage workers into
http://aquilon.gridpp.rl.ac.uk/sandboxes/diff.php?sandbox=xrootd-patch(will include cherry-picked commit for the Docker patch). http://aquilon.gridpp.rl.ac.uk/sandboxes/diff.php?sandbox=docker-cvmfs-xrootd-combo This sandbox combines all changes and gives a clear view of all changes (DO NOT DEPLOY).Confirm workers have successfully compiled:
Check: `healthcheck_wn_condor` outputs healthy status.
Check ` xrootd-gateway.service` outputs healthy Docker status.
Bring online tranche at a time confirming with data team xrootd is working as expected.
10-11th May: Let the updated workers run for a few days.
12th May: `wn-2017-dell (all 2017’s) - wn-2019-dell` will be set to drain.
15th May: Repeat above process for second half of workers.
16th May: Merge all required sandboxes into prod and manage farm back into `prod_batch` in A(9:30 am): Draining of first half of worker-nodes
11th May (9:30 am): Update drained worker-nodes
Bring back online updated tranches
12th May (16:00): Drain remaining half of worker-nodes
15th May (14:00): Update drained worker-nodes
Bring back online updated tranches
Health check entire farm
Plots and associated info
...