2025-05-22 Meeting Notes

Operational Issues
Gateways and WNs:
- Current status and upcoming changes

error 500 ongoing, mostly on RAL-Glasgow link.

Compilation and rollout status of RAL XRootD versions

@Thomas, Jyothish (STFC,RAL,SC)

5.8.2 reverted to 5.7.3 to attempt mitigation

Shoveler

@Katy Ellis

High ‘ops’ time from Lancaster; timeouts?

Lancaster Shoveler needed ‘help’ in last few days.
Aiming for per-Gateway Shoveler instances

RAL: Single Collector instance

https://monit-grafana.cern.ch/d/000000444/xrootd-transfers?from=now-7d&orgId=20&to=now&var-bin=1h&var-dst_country=All&var-dst_exp_site=All&var-dst_site=RAL-LCG2&var-dst_site=UKI-SOUTHGRID-RALPP&var-dst_tier=All&var-group_by=src_site&var-ipver=All&var-remote=true&var-src_country=All&var-src_exp_site=All&var-src_site=All&var-src_tier=All&var-vo=All

On the fly Checksums
https://stfc.atlassian.net/browse/XRD-98

@Ian Johnson

Jyothish with a branch from Ian.
2 gateways still running the ‘old’ version (with issues). Once new version reviewed, planned to deploy again for testing.

cms-aaa naming convention

@Thomas, Jyothish (STFC,RAL,SC)

cms-aaa is the only remaining personality to use proxy/ceph as the xrootd service names

Separate naming convention would be more appropriate, to have main/supporting

(not so urgent).

CC created, and sandbox is prepared and has been tested on a test host

cms-aaa jemalloc use

@Thomas, Jyothish (STFC,RAL,SC)

cms-aaa overhaul

Deletions

https://stfc.atlassian.net/browse/XRD-83

On-hold

Plan: file query system to summarize XRootD Logs

100 GbE Gateway testing:
SKA / Tier-1

@James Walder @Thomas, Jyothish (STFC,RAL,SC)

UKSRC -

Test Campaign work ongoing.

Last week 1M 1MiB files from Algol replicated to other RSEs < 24 hours

(xrootd02)

Open

Tier-1
~8Gbps achieved with bash parallel transfers, FTS based testing today

Open

UKSRC Storage Architecture

Tokens Status

• Operational

• Technical

  • Token leak on redirect fixed by Brian’s PR converting to macaroons, tested on preprod but based on 5.8.1, which we’ll roll out first after the WLCG workshop

• Accounting

packet marking

test stress testing framework

Script ready, first test to be arranged
https://github.com/Jo-stfc/xrootd-utils/blob/main/ftsstresstest.sh

LHCb deletion delay resulting in missing files

WLCG Workshop

Tokens:

• Issuers causing confusion in accounting → possible further restrictions needed based on group, likely on the IAM side, unclear as of yet if this needs validation xrootd server side

• long lived token with narrow scopes → mainly for tape

• token DC 26/27 for testing token creation frequency can be handled

XRootD Monitoring

• Shoveler preferred, effort being added CERN side
  TBD: arrange a meeting with Katy and Borja to debug and feature request

AF/job scheduling interesting but out of scope for this meeting