2024-02-29 Meeting Notes

Operational Issues
Gateways and WNs:
- Current status and upcoming changes

@Thomas, Jyothish (STFC,RAL,SC)

Further observations from Data Challenge

Future testing plans (RAL initiated / VO initiated) ?

All

DC24 observations

To improve still: LB, Checksums (inflight + metadata),

Open

RR with failsafe hotspotting

Rocky 8 migration planning

Deletion studies through RDR

@Ian Johnson

Previous deletion times reported were from the client side, end-to-end. Now analysing times for individual ceph_posix_unlink calls, and will look at deletion request times with XRootD itself. That is, looking at the amounts of time that a deletion request takes travelling though layers of XRootD code.

I’ve noted some strange outliers from an initial sample (y-axis times in ms, last tick should read 80000 ms.) Outlier for ceph_posix_unlink at 90s? stddev of 4476 for the plot below. 6808 samples taken for a single gateway from 22nd Feb:

Open

Deletions

https://stfc.atlassian.net/browse/XRD-83

Next steps:
Collate previous information to define the problem

Identify potential solutions.

Planning for ALICE CMSD redirection

@Thomas, Jyothish (STFC,RAL,SC)

INC-163994 - DNS ip additions

Checksums fixes

@Alexander Rogovskiy @Thomas, Jyothish (STFC,RAL,SC)

Noscript checksum by Jo-stfc · Pull Request #9 · stfc/xrootd

Prefetch studies and WN changes

@Alexander Rogovskiy

Tokens Status

@Thomas, Jyothish (STFC,RAL,SC) @Katy Ellis

CMSD Load balancing

@Thomas Byrne @Thomas, Jyothish (STFC,RAL,SC)

PR:
revised load balancing algorithm - weighed random selection by Jo-stfc · Pull Request #8 · stfc/xrootd

addresses the issue where the current selByLoad algorithm leads to load hotspotting and coarse load distribution

It replaces the current algorithm in xrootd. testing for simulated behaviour in progress.

SKA Gateway box

@James Walder

https://stfc.atlassian.net/wiki/spaces/UK/pages/215941180

5.6.x root TPC issue

root:// TPC transfer fail with xrootd 5.6.x · Issue #2202 · xrootd/xrootd

issue open, under investigation