2023-07-27 Meeting Notes

XRootD Releases

Sam

xrootd releases 5.6.0 and 5.6.1 have changed behaviour in WebDAV with header handling which means that they fail the "GET" part of the EGI availability tests.

(This is covered in issue 2058 in the Xrootd github: [Regression] Chunked PUT creates empty files with 5.6.1 XrdHTTP · Issue #2058 · xrootd/xrootd · GitHub )

This will be fixed in 5.6.2, but if you were thinking of upgrading to 5.6.x for the fixes to SSL handshaking between RHEL7 and 9 releases, you should hold off for now if you care about EGI availability tests.

Prefetch studies

Ongoing

CMSD rollout

https://stfc.atlassian.net/browse/XRD-41

Status:

Next plans:

CMSD operations: observations

No observed problems on manager hosts.
Manual failover between managers also appears to work; need to measure the latency of failover (and tune parameters if needed).

…

CMSD outstanding items

Icinga / nags callout tests changes.

Improved load balancing / server failover triggering

better 'rolling server restart script'

Documentation; setup / configuration / operations / troubleshooting / testing

Review of Sandbox and deployment to prod.

Other recent observations on Gateways

Lots of reads on Gateways using parallel threads (e.g. AIO); most likely coming from offsite XCaches for (only?) atlas VP reads …

Occasional crashes in ‘thread’ calls

Memory usage increasing (possibly due to the AIO calls) - may want to raise the systemd hard memory limit

Gw6 (yesterday), “Forgot” how to do authentication - permission denied on many transfers; not caught by nagios checks

Tokens testing

dev-gw4:
- Managed to get ZTN root based tokens working

Simple ‘fix’: needs
sec.protocol ztn
but also
sec.protbind * only ztn gsi

Passing CMS Sam tests

AAA Gateways

Recent Sandbox updates with Katy:
- Add back memory cache
- ensure latest XrdCeph version included
- enable non-striper reads
- enable ztn token access (Passing CMS Sam tests)

Sandbox ready for review:

http://aquilon.gridpp.rl.ac.uk/sandboxes/diff.php?sandbox=jw-xrootd-aaa-5.5.4-3

Floating IPs for VM test cluster

Status:

• What certificates / SANs will be needed ?

SKA Gateway box

https://stfc.atlassian.net/wiki/spaces/UK/pages/215941180

Xrootd items:

• Testing strategy; including tests of different storage backends

• Confirm that network routing / Xrootd configuration are compatible …

• what Auth model?

• ‘high performance / long-distance’ tuning parameters ?

• Development of highly performant / parallel buffering (TBC)

• multi-stream transfer tests:

  • 1) against Posix with root / davs

  • 2) against XrdCeph (root / davs)

  • 3) try S3 (Davix) plugin.