\uD83D\uDDD3 Date
\uD83D\uDC65 Participants
Lancs: Matt, Steven, Gerard
Glasgow:
Apologies:
CC:
\uD83E\uDD45 Goals
List of Epics
New tickets
Consider new functionality / items
Detailed discussion of important topics
Site report activity
\uD83D\uDDE3 Discussion topics
Current status of Echo Gateways / WNs testing
Recent sandbox’s for review / deployments:
Item | Presenter | Notes | |
|---|---|---|---|
Operational Issues | brief hiccup on icinga due to ipv6 issues | ||
XRootD Managers De-VMWareification (Moving to physical hosts) | /wiki/spaces/GRIDPP/pages/872644647 CC due today due to a better but slightly more complicated aquilon procedure being used for the first time | ||
Release of 5.7.3 (May expect an 5.8.X prior to 6.X?) | https://github.com/xrootd/xrootd/releases/tag/v5.7.3
| ||
Checksums issue with an ATLAS file | https://github.com/xrootd/xrootd/issues/2388 https://ggus.eu/index.php?mode=ticket_info&ticket_id=169360 Checksum requested before whole file is updated. No ability to do stale checksum check in ceph, so original checksum ‘sticks’ to the file. fix in place RAL side by clearing checksums after a write is complete | ||
cms-aaa naming convention | cms-aaa is the only remaining personality to use proxy/ceph as the xrootd service names Separate naming convention would be more appropriate, to have main/supporting (not so urgent). CC created, and sandbox is prepared and has been tested on a test host | ||
cms-aaa jemalloc use | testing on svc20, some memory leak still present | ||
Compilation and rollout status of RAL XRootD versions | 5.7.2 published. 5.7.2 skipped on farm due to pfc bug, 5.7.3 to be realed soon | ||
Shoveler | |||
On the fly Checksums | Integrated checksum attribute storage into PoC. Measured time to transfer 10x3GiB files in parallel to a dev gateway with xrdcp verifying the source checksum.  Next steps: add in optional CRC32C calculation. Conduct larger-scale performance tests, ideally against a gateway machine which is more representative of production GWs. | ||
Deletions | XRD-83 - Getting issue details... STATUS | NTR | |
XRootD Writable Workernode Gateway Hackaton | XRootD Writable Workernode Gateway Hackaton (XWWGH) sandbox with fixes present, tested on lhcb workernode, reading works fine as is, writes still need testing to let jobs only write on that WN GSTSM-284 - Getting issue details... STATUS first write completed!  | ||
Xrootd testing framework | Discussion in Storage Meeting in how to integrate the various testing structures within the UK. container with the testing framework TBD | ||
Plan: file query system to summarize XRootD Logs | Plan to create a system to store info from across all gateways to search a filename and get creation time, last write time, last successful stat and deletion time in case of ‘lost’ files. Possible graduate sideproject. | ||
100 GbE Gateway testing: | UKSRC - Acting as source for SRCNet verification tests; not being stressed so far … Teir-1 . | ||
UKSRC Storage Architecture | Through discussions, need to change the DNS entries for the data and mgmt interfaces, update netbox and reconfigure in AQ. Data network will be (exclusively) for the DTN / data traffic. mgmt for ancillary needs (icinga, AQ). Host will be known via its mgmt dns name (the canonical name). | ||
Tokens Status |
|
on GGUS:
Site reports
Lancaster: Thanks for Jyothish for sharing the load script, we’re looking at deploying it at Lancaster. To alleviate Matt’s worries about the cmsd load balancing not ignoring overloaded nodes he replaced one hosts script with one that just returnes “99” 5 times. Happily this node was ignored by the redirector, so that works (see green plot below between 10 and 10.30).
We have one host on 5.7.3, nothing exploded. Will roll out to the rest of our machines shortly.
Glasgow - Brief failures to authenticate internally - some of the lsc files for atlas iam were out of date despite using RPM. (possible issue on cron job), looking forward to the on the streamed checksums
✅ Action items
How to replace the original functionality of fstream monitoring, now opensearch has replaced existing solutions.