12–16 May 2023 Echo instability following readV rollout

Other related links:

• https://stfc.atlassian.net/wiki/spaces/GRIDPP/pages/188153912/Echo+experiencing+critical+xrootd+failures?src=mail&src.mail.timestamp=1684246234856&src.mail.notification=com.atlassian.confluence.plugins.confluence-mentions-plugin%3Amention-created-notification&src.mail.recipient=6086b60f52a3d40069d40ae1&atlOrigin=eyJpIjoiMjY2NTIzMGE4MzE3NDAzYjhjNzc5NTc1ZjY0MmQzNmUiLCJwIjoiYyJ9&src.mail.action=view (Post mortem report)

• XRootD: Code review for ReadV implementation in XrdCeph (March 2023) (code review summary)

• https://stfc.atlassian.net/wiki/spaces/GRIDPP/pages/137265343/Non-striper+read+v+implementation+for+WN+s+xrootd+gateways (Change control document)

Useful pages for monitoring

Gateway status monitoring

XrootD report monitoring

Slow I/O monitoring

legacy network traffic

new network traffic

storage monitoring

Plots and associated info

Please at a screen shot (or more) of the plot, and the timestamp url from which it was obtained.
For URLs which are generally useful please also add to the section above, with some brief description.

The problem was initially found by gateway functional tests failing Friday evening. Restarting the gateways didn’t fix the issue. Memory spikes correlate to increased connections in the xrootd report monitoring shown below

which suggested some slowdown or issue in ceph. This was further supplemented by the slow operation monitoring