2022-08-17 Meeting notes

5.5.0-rcX status / problems in testing

@Thomas, Jyothish (STFC,RAL,SC) Sam

Sam - appears ok; (localgroupdisk permissions issues at Glasgow).
Jyothish 5.5.0 appears ok sofar.
XrdCeph bug discovered https://stfc.atlassian.net/wiki/spaces/CD/pages/30998699 while setting up unit tests:
errors in xrdceph related to the cluster object do not cause a clean restart, locking the service into an invisible crash - status still reports OK but all requests sent fail and cause the socket to go into close_wait

Centos 8: outstanding items?

Continuing to progess.

(See Lancs Site report for more details on vector size problems, but resolved in 5.5.0-rc2).

space info reporting:

https://stfc.atlassian.net/browse/XRD-21

Alice testing, dev-gw2 (Confirmed from (Alice) running tests on dev-GW2
JW review PR, and will confirm from Atlas once on a non-Slice node.
With Rob reviewed the aqualon config, with some updates.

Alice tests OK - sandbox ready and deployment in prep (aim Monday).

https://stfc.atlassian.net/browse/XRD-26

(Not running at the moment due to dev-gw2 testing).

https://stfc.atlassian.net/browse/XRD-27

James

Scripts created to extract problematic files

Deletions / current understanding / necessary changes

Options:

• Horizontal scaling, i.e. spread deletes across all available gateways

• Add (when ready) CMSD to ensure load balancing (at what extra cost),

• Offload to external tooling to manage the deletes asynchronously

  • Ideas below

  • (JW: currently ‘playing' with test implementation).

Open

“Unified” xrootd status / problems?

Avoid TPC for xroot transfers on the webdav gateways

xrdtpc.sh script without a proxy (is it possible?)
xrd.tpc redirect (to xrootd hosts, or some other proxy … )

Did not find a solution yet (see above).

What are we going to do with Vector Reads now?

All

Removal of locking did not appear to help significantly.

Work ongoing with EL8 and recent 5.X releases (both recommendations from Rob’s document).

Rob C’s script is currently the only metric for LHCb to compare / verify against.

Options:

• Write a proper striper-vector-read method

• Implement one of Andreas’ options

• Fall back on buffering / range coalesce in the short term, until correct fix available.

  • Collect data on what really is needed

• The socket timeouts (etc) should give and propagate up (sensible) error messages

  • i.e. concentrate on the cases that are ‘broken’

  • Jyothish’s work on 'close-wait + connection resilience '

• quantify the ‘slowness’ of the vector reads (and impact).