...
Item | Presenter | Notes | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
Operational Issues | mitigations have been communicated to ATLAS for the jobs using 5.6.0 clients reboot campaignbrief hiccup on icinga due to ipv6 issues | ||||||||||
XRootD Managers De-VMWareification (Moving to physical hosts) | /wiki/spaces/GRIDPP/pages/872644647
CC due today due to a better but slightly more complicated aquilon procedure being used for the first time | ||||||||||
Release of 5.7.3 (May expect an 5.8.X prior to 6.X?) | https://github.com/xrootd/xrootd/releases/tag/v5.7.3
| ||||||||||
Checksums issue with an ATLAS file | https://github.com/xrootd/xrootd/issues/2388 https://ggus.eu/index.php?mode=ticket_info&ticket_id=169360 Checksum requested before whole file is updated. No ability to do stale checksum check in ceph, so original checksum ‘sticks’ to the file. fix in place RAL side by clearing checksums after a write is complete | ||||||||||
cms-aaa naming convention | cms-aaa is the only remaining personality to use proxy/ceph as the xrootd service names Separate naming convention would be more appropriate, to have main/supporting (not so urgent). CC created, and sandbox is prepared and has been tested on a test host | ||||||||||
cms-aaa jemalloc use | testing on svc20, some memory leak still present | ||||||||||
Compilation and rollout status with XrdCeph and rocky 8: 5.7.x of RAL XRootD versions | 5.7.2 published. 5.7.2 skipped on farm due to pfc bug, possible RAL release 5.7.3 equivalent with a fix for that and 5.6.0 client compatibilityto be realed soon | ||||||||||
Shoveler | |||||||||||
On the fly Checksums
| |||||||||||
Deletions |
| NTR | |||||||||
XRootD Writable Workernode Gateway Hackaton | XRootD Writable Workernode Gateway Hackaton (XWWGH) sandbox with fixes present, tested on lhcb workernode, reading works fine as is, writes still need testing to let jobs only write on that WN
first write completed! | ||||||||||
Xrootd testing framework | Discussion in Storage Meeting in how to integrate the various testing structures within the UK. container with the testing framework TBD | ||||||||||
Plan: file query system to summarize XRootD Logs | Plan to create a system to store info from across all gateways to search a filename and get creation time, last write time, last successful stat and deletion time in case of ‘lost’ files. Possible graduate sideproject. | ||||||||||
100 GbE Gateway testing: | UKSRC - Acting as source for SRCNet verification tests; not being stressed so far … Teir-1 . | ||||||||||
UKSRC Storage Architecture | Through discussions, need to change the DNS entries for the data and mgmt interfaces, update netbox and reconfigure in AQ. Data network will be (exclusively) for the DTN / data traffic. mgmt for ancillary needs (icinga, AQ). Host will be known via its mgmt dns name (the canonical name). | ||||||||||
Tokens Status |
|
...
on GGUS:
Site reports
Lancaster: Following on from last week, we were looking at the load reported by the (default) cmsd load reporting scripts, and they didn’t seem to match up to any numbers we could pull from the servers. We got distracted by other things before we could dive deeper.
LSST planning to use small files for read/writes, planning to remove TLS on pure xrootd, these seems to be intermediate files, but might need to be made available for quality conrol? looking at object store route (s3 for internal use) maybe?
combination of uid/host based auth resulted in the following error on curl:
unknown.2:28@comp21-04.private.dns.zone Unable to open /cephfs/grid/dteam/curltest; permission deniedThanks for Jyothish for sharing the load script, we’re looking at deploying it at Lancaster. To alleviate Matt’s worries about the cmsd load balancing not ignoring overloaded nodes he replaced one hosts script with one that just returnes “99” 5 times. Happily this node was ignored by the redirector, so that works (see green plot below between 10 and 10.30).
We have one host on 5.7.3, nothing exploded. Will roll out to the rest of our machines shortly.
...
Glasgow - Brief failures to authenticate internally - some of the lsc files for atlas iam were out of date despite using RPM. (possible issue on cron job), looking forward to the on the streamed checksums
...