Ideas on xrootd batch farm architecture

Ideas of interest:

Create 2 seperate xrootd gateway “pools” to handle the different traffic from FTS and the Batch Farm. The benefit of this is reducing impact of both services should one overload the gateways. Downside is potential further complication with DNS records. This could be avoided with spilt DNS, (internal vs external traffic?)
Have the Batch Farm use the local xrootd gateways for writes, pointing the rdr alias to the gateway container, bypassing the proxy. Reads would continue to use the proxy container. The downside to this is… it’s not elegant, Tom B and Tom B should discuss.
Create an internal CMSD clustered setup; mirroring the external CMSD configuration, but for internal traffic only. Have dedicated Xrootd servers behind the manager hosts. ‘DNS poisoning' on the WNs would point to the two (manager) floating IPs (e.g. even vs odd WN ips mapping to even vs odd floating IPs ?). This would primarily cover writes from WN to Echo.
- Disadvantage - isolation of gateways between internal and external. Only the specified alias(es) would be ‘redirected’ to the manager.
- Advantages: isolation of gateways between internal and external; no single point of failure as CMSD managers should provide the high available and load balancing across hosts. Easy to move servers across as needed.
- Does not address any change of architecture of reading data from Echo

“The wave” on the 15th

cms used the gateways for reads since the start of the gateway’s use

lhcb reads seem to be only from the 15th, timing spreads from 10:20 to 14:42, 14-14:40 seems to have most of them

A few other observations:

VOs may be unable - or want to be unable - to provide very fine grained control over reads and writes, and even if they can make certain settings, other mechanisms, e.g. failover from root to webdav, xrootd.echo to rdr.echo (as we’ve observed) can have unintended consequences if a very precisely configured set of routes / assumptions are made.
- i.e. a simple configuration (from the user facing side) is likely a better solution, even if there is complexity underneath.
While making it the responsibility of each WN to read the data from ECHO has a number of potential benefits, there is no real redundancy and resilience in place to deal with failures (aside from restarting the container, or draining the node). The daily restart of xrootd containers also can appear to induce unexpected consequences (such as triggering VO failover mechanisms). Additionally, as the number of slots on each WN increases with newer generations, the requirements on the xrootd servers on the WN also increases, which need to be understood better.
There is a general aim to attempt to remove the need for an Xcache on the WNs, however so far this has lead to unacceptable levels of small reads issued against Ceph.