CMSD rollout

The rollout is planned in phases. When a gateway is added to the redirector alias (managed by echo manager), it is put in the cmsd sandbox.

Current status

Phase 1

Single active gateway in alias. This substitutes gw14 in the xrootd and webdav aliases and ensures correctness of the initial change. any protocol/cmsd interaction unexpected behaviour should be caught here. Load behaviour and overheads should also become apparent.

Phase 2

For small scale testing, add rdr alias to xrootd and webdav aliases this ensures production traffic using both protocols pass trough the redirector. Rdr redirects to 3 gateways, this is to test small scale load balancing. having 2 gateways might lead to instabilities (load ping-pong or fallback to single gw in case the other is loaded). The gateways are from different generations to evaluate impact on each. svc97 is purposefully kept within the xrootd/webdav aliases to evaluate the impact of cross-alias requests. This is useful to know for future projects (e.g. cross-site cmsd) but would not reflect the final state of this change.

Phase 3

When the previous phases pass, all other gws will be removed from the xrootd/webdav alias. All prod traffic will flow trough the redirector. This tests the capacity of the cmsd managers to cope with full prod traffic

Rollback

If any issues happens during the rollout, the rdr dns will be removed from xrootd and webdav aliases and the previous set of gateways will be reinstated.