\uD83D\uDDD3 Date
\uD83D\uDC65 Participants
Lancs: Matt, Steven, Gerard
Glasgow:
Apologies:
CC:
\uD83E\uDD45 Goals
List of Epics
New tickets
Consider new functionality / items
Detailed discussion of important topics
Site report activity
\uD83D\uDDE3 Discussion topics
Current status of Echo Gateways / WNs testing
Recent sandbox’s for review / deployments:
Item | Presenter | Notes | |
|---|---|---|---|
Operational Issues | |||
XRootD Managers De-VMWareification (Moving to physical hosts) | |||
XRootD collaboration Meeting | 1 - streamed checksum validation We're planning to implement a streamed checksum implementation in XrdCeph (computing checksum at the server and writing it into metadata directly instead of reading back) and give it a few months of validation testing comparing it to the read back checksum. Tests have already shown a good improvement in performance when done this way.
2 -xrdceph plans: We're planning to move away from libradosstriper into using rados directly for future proofing and performance improvements as well as merge our fork into core xrootd.
3- backward compatibility with 'broken' clients this was following an incident where ATLAS were trying to use 5.6.0 clients with their older analysis software, which had a TLS bug causing transfer failures against endpoints supporting tokens over IPv4. We mitigated it with the NOTLSOK environment variable, but it'd be good to have a consensus on how to deal with clients with known issues. | ||
cms-aaa naming convention | cms-aaa is the only remaining personality to use proxy/ceph as the xrootd service names Separate naming convention would be more appropriate, to have main/supporting (not so urgent). CC created, and sandbox is prepared and has been tested on a test host | ||
cms-aaa jemalloc use | testing on svc20, some memory leak still present | ||
Compilation and rollout status of RAL XRootD versions | 5.7.2 published. 5.7.2 skipped on farm due to pfc bug, 5.7.3 released | ||
Shoveler | |||
On the fly Checksums | |||
Deletions | XRD-83 - Getting issue details... STATUS | NTR | |
XRootD Writable Workernode Gateway Hackaton | The sandbox is deployed to the whole preprod farm. LHCb uploads look OK, Atlas and CMS have not tested the new setup yet.  | ||
Plan: file query system to summarize XRootD Logs | Plan to create a system to store info from across all gateways to search a filename and get creation time, last write time, last successful stat and deletion time in case of ‘lost’ files. Possible graduate sideproject. | ||
100 GbE Gateway testing: | UKSRC - Acting as source for SRCNet verification tests; not being stressed so far … Teir-1 . | ||
UKSRC Storage Architecture | |||
Tokens Status |
|
on GGUS:
Site reports
Lancaster:
All on 5.7.3 for the last week, no issues.
Following on from last week, Steven has been upping our number of shoveller instances, we’ll see how that goes.
As mentioned in storage we’re shopping for some new gateways - getting quotes for single socket Dell boxes with quad-port 25Gb network cards (and looking at 100Gb options). Our plan is to have the external port on a single 25Gb and internal on a pair of bonded 25Gb NICs. It would be nice to save money and not have to cram these with RAM.
Also mentioned was Gerard's “xrootd restarter” service, which will shepherd regular rolling restarts of our xroot services (as a means to deal with xroot’s poor connection handling). The aim is to cleanly restart the services every couple of days. Gerard’s been working on making it as “unhacky” as he possibly can.
Side topic, in an email thread with Dan T he mentioned tls hardware offloading which piqued my interest. Anyone looked into this recently? AIUI it’s not a feature on all cards. I see there’s also tcp checksum offload features in cards too.
(have we discussed this here before? It rings a bell…)
Glasgow -
✅ Action items
How to replace the original functionality of fstream monitoring, now opensearch has replaced existing solutions.