Page Comparison

5.4.3 releases in Centos 7

• Problems observed: (pgRead / pgWrites)

Thomas, Jyothish (STFC,RAL,SC)

5.5.0-rcX testing rc1 testing on dev VM, appears to be working ok.
and feedback to xrootd devs ?

Sam also testing building 5.5.0; finds python2 first?
- some updates to cmake required to make it find the relevant python3
- Sam will create an issue once changes finalised …

discovered /wiki/spaces/CD/pages/30998699 while setting up unit tests:
errors in xrdceph related to the cluster object do not cause a clean restart, locking the service into an invisible crash - status still reports OK but all requests sent fail and cause the socket to go into close_wait

Centos 8: outstanding items?

Workernodes with EL8 being prepared for deployment.
Some items like cephsum to be placed into EL8 rpms.

space info reporting:

functionality implemented in xrdceph, current usage info coming from ceph,
2 additional configurable parameters, ceph.quotapath and ceph.poolnames
quota info coming from local json file read from quotapath (with similar format as s3.echo.stfc.ac.uk/srr/storagesummary.json).
Default set to /etc/xrootd/storagesummary.json.
pool names defined as comma seperated string with trailing comma, read from ceph.poolnames.
Tested as working in dev setup, PR & RPM generated ,
some future proofing to be added (multiple VO pools, general pool name) - plan to roll this out into as v5.3.8 of xrootdcephfor generalization - are there any path request that would not contain ':' normally ?

• Perhaps less relevant for Glasgow.

5.5.0-rc1 is still not compliant with the WLCG testbed:
https://github.com/xrootd/xrootd/issues/1752

https://github.com/xrootd/xrootd/pull/1753/commits/0e5ef3b7bbf39c5fa4cb594617e2830c0ee72f07

Deletions update

Ian looking at 1-8 GiB deletions against gwX.
Single file writes and deletes

No deletion timeouts observed.
Previous tests (with more parallel deletions) do show longer deletion times

James

Occasional failed deletes for Atlas due to 0-byte files with partial striper metadata.
(Likely triggered by recent consistency check).
Manual

Future of Deletions?

Options:

• Horizontal scaling, i.e. spread deletes across all available gateways

• Add (when ready) CMSD to ensure load balancing (at what extra cost),

• Offload to external tooling to manage the deletes asynchronously

  • Ideas below

  • (JW: currently ‘playing' with test implementation).

What are we going to do with Vector Reads now?

All

Removal of locking did not appear to help significantly.

Rob C’s script is currently the only metric for LHCb to compare / verify against.

Options:

• Write a proper striper-vector-read method

• Implement one of Andreas’ options

• Fall back on buffering / range coalesce in the short term, until correct fix available.

  • Collect data on what really is needed