2023-10-05 Meeting Notes

XRootD Releases

5.6.2-2 is out
- Plan for deployment

• Is passing on pre-prod

• upgrade of CMake version, exhibiting ‘stack smashing’ ?? (or different compile versions)

• Features for XrdCeph that need to be included ?

Aim for prod testing next week. (aim for 1 week of testing, then deploy if ok).

Sam notes:
if you're building xrootd 5.6.2 from source, the tagged 5.6.2 in the git repo does not have the bugfix for authdb parsing, as I just discovered to my cost (that's 2 commits later...)

Checksums fixes

Prefetch studies and WN changes

Alex

(temporarily rolled back, with the ongoing work in batch farm WNs)

• the LD_Preload for lockless reads were removed (compiled against ‘older’ version of ceph, and superseded by the recent readV work.

• WN containers need updated Ceph rpms (for 14.2.22)

• @Alexander Rogovskiy to present ‘final’ status of testing in next meeting on the prefetch work

• @James Walder to add work items to the queue in jira,

Deletion studies through RDR

Ian

CMSD rollout

https://stfc.atlassian.net/browse/XRD-41
New diagram required ?

svc01,02,17,18 stay as internal WN gateways for now.
The other svc hosts 03,05, 11,13-16 to added to CMSD production cluster.

svc19 (designated for Alice gateway)

Gateways on new network plan

ipv6 sorted, firewall rules change in progress

To consider the TPC instance port

Gateways: observations

Transfer efficiency between RAL and Glasgow (plot of GLA → RAL for ATLAS ) (Sam investigating)

Open

Monday - showed high system load, but had low throughput. Appears resolved as of yesterday.

Gateways updated to latest nautilus client version.

CMSD outstanding items

Icinga / nagios callout tests changes. - live and available

• ping test for the floating ips and getaway hosts needs some more refinement

Improved load balancing / server failover triggering -

better 'rolling server restart script'

Documentation; setup / configuration / operations / troubleshooting / testing

• blacklisting; post-5.6.2-2 to include the fix for a non-valid host in the blacklist file.

• [XrdCms] Try to load blacklist even if some entries are invalid · xrootd/xrootd@941025a

Review of Sandbox and deployment to prod:
- Awaiting time from Tom for load balancer test

Tokens testing

To Liaise with the TTT Taskforce (aka. @Matt Doidge )

AAA Gateways

Sandbox ready for review:

http://aquilon.gridpp.rl.ac.uk/sandboxes/diff.php?sandbox=jw-xrootd-aaa-5.5.4-3
Needs a bigger discussion regarding Tokens, and deployment in production hosts.

SKA Gateway box

https://stfc.atlassian.net/wiki/spaces/UK/pages/215941180

now working using ska pool on ceph dev

Initial Iperf3 tests: (see table and plots below).

• Actions

  • Ensure Xrootd01 is tuned correct, according to the Nvidia / mellanox instructions

  • Repeat the iperf tests

• Xrootd tests against:

  • dev-echo

  • cephfs (Deneb dev)

  • cephfs (openstack; permissions/routing issues)?

  • local disk / mem

• Frontend routing is also being worked on

Open

containerised gateways (kubernetes cluster)

identified an issue on workernode gateways where ceph nautilus 14.2.15 libraries were loaded (from a previous libradosstriper lockless read implementation) overriding the container installed ceph version