2024-02-01 Meeting Notes

 Date

Feb 1, 2024

 Participants

  • @Thomas, Jyothish (STFC,RAL,SC)

  • @Ian Johnson

  • @Alexander Rogovskiy

  • @Katy Ellis

  • @James Walder

  • @Thomas, Jyothish (STFC,RAL,SC)

  • Lancs: Gerard, Steven, @Matt Doidge

  • Glasgow:

Apologies:

CC:

 

 

 Goals

  • List of Epics

  • New tickets

  • Consider new functionality / items

  • Detailed discussion of important topics

  • Site report activity

 

 Discussion topics

Current status of Echo Gateways / WNs testing

Recent sandbox’s for review / deployments:

 

Item

Presenter

Notes

 

Item

Presenter

Notes

 

Operational Issues

@Thomas, Jyothish (STFC,RAL,SC)

 

 

 

Gateways and WNs:
- Current status and upcoming changes

@Thomas, Jyothish (STFC,RAL,SC)

  • Tokens

  • Checksum plugin (see below):

  • Rocky 8 (see next

Increase of memory to the proxy (XCache)to 64GB on the WN proxy due memory allocation errors.

 

Rocky 8 migration planning

@Thomas, Jyothish (STFC,RAL,SC)

  • Technical impediments ?

  • Performance comparisons:

To investigate missing rpms
** Checksum checksum servers / packages

  • Plans for migration

    • What to do about the redirectors ?

 

S&C ATLAS week

@James Walder @Thomas, Jyothish (STFC,RAL,SC)

Next week S&C; Anything to discuss with Andy H et al. ?

 

ECHO File transfer / throughput studies

@Katy Ellis

Updates?

 

Checksums fixes

@Alexander Rogovskiy

Status and plans for improving Checksumming work …

GitHub - alex-rg/xrd_ckslib
Ihttps://stfc.atlassian.net/browse/XRD-56

  • Code review complete; awaiting updates

  • Packaging; how to package the rpm (headers from XrdCks only from xrootd source files … ?)

    Some test results:

    patched.png
    unpatched.png

 

Prefetch studies and WN changes

@Alexander Rogovskiy

Sandbox ready and applied to 1 WN, pending envroinment variable for timeout increase

 

Deletion studies through RDR

@Ian Johnson

Latest deletion rates for 3 GiB files: 21 Hz

Desire deletion rate from ATLAS: ~ 4 Hz

Latest deletion rates for 1 KiB files: 59 Hz

Deletion rates measured using 10 threads (VM with 16 vCPUs). In conversation with CERN DDM, Rucio reaper is using 10 threads (for ATLAS?).

Next: Check timings for slow deletions. Measure deletion rate for 6 GiB files. Testing with 1 KiB files works up to 4000 files (deletion client on two VMs), but fails with 10000 files. Investigating.

 

Tokens testing

@Thomas, Jyothish (STFC,RAL,SC) @Katy Ellis

https://stfc.atlassian.net/browse/XRD-63
https://stfc.atlassian.net/browse/XRD-78
Attempt to use restricted paths even when scope is more generic by bbockelm · Pull Request #2152 · xrootd/xrootd

avoid duplicating basepath by Jo-stfc · Pull Request #2151 · xrootd/xrootd

 

Understanding CMSD Loadbalancing

@Thomas Byrne

 

 

SKA Gateway box

@James Walder

https://stfc.atlassian.net/wiki/spaces/UK/pages/215941180

 

TEST

SVC16(R8)MEAN

SVC26(SL7)MEAN

SVC16(R8)MEDIAN

SVC26(SL7)MEDIAN

TEST

SVC16(R8)MEAN

SVC26(SL7)MEAN

SVC16(R8)MEDIAN

SVC26(SL7)MEDIAN

davsdelete

0.47126

0.40214

0.464

0.381

davsread

13.7352

12.7532

13.677

12.727

davsstat

0.19786

0.18454

0.181

0.181

davswrite

28.9813

23.9641

28.992

23.798

rootdelete

0.27626

0.2847

0.258

0.253

rootread

13.0728

12.0312

13.041

11.86

rootstat

0.08458

0.07198

0.083

0.063

rootwrite

25.3381

23.7827

25.312

23.738

tpcdest

43.3166

40.3735

43.349

40.668

tpcsource

37.468

24.8932

34.181

31.581

 

on GGUS:

Site reports

Lancaster - Tokens still disabled, but we reckon there will be pressure to re-enable. Pondering how to implement a network QoS to help reduce cephfs drop outs by guaranteeing a slither of bandwidth on the gateway hosts for ceph traffic. Also pondering dedicated internal gateways.

Glasgow - relatively stable, few network issues. OS/Ceph version update to do.

 

 

 

 Action items

  • @James Walder to schedule a ‘hackathon’ within a F2F to have a session on architectural planning.

  • @James Walder to prepare an outline of the expected roadmap for XRootD developments in 2024.

  •  

 

 Decisions