• Rough draft
  • 2023-05-25 Meeting Notes

     Date

    May 25, 2023

     Participants

    • @James Walder

    • @Thomas, Jyothish (STFC,RAL,SC)

    • @Alexander Rogovskiy

    • @Ian Johnson

    • Lancaster: Gerard, Matt, Steven

    Apologies:
    @Alison Packer , @Alastair Dewhurst

     Goals

    • List of Epics

    • New tickets

    • Consider new functionality / items

    • Detailed discussion of important topics

    • Site report activity

     

     Discussion topics

    Current status of Echo Gateways / WNs testing

    Recent sandbox’s for review / deployments:

     

    Item

    Presenter

    Notes

     

    Item

    Presenter

    Notes

     

    Vector Read updates to Echo

     

    https://stfc.atlassian.net/wiki/spaces/GRIDPP/pages/188153912

    Post-mortem to be held today

     

     

    Next steps for WN deployment

     

    Move to 5.5.4-2 (core) + 5.5.4-3 (xroot-ceph-buffered):

    GitHub - stfc/xrootd at v5.5.4patched

    Release 5.5.4-3 non striper reads in buffered IO · stfc/xrootd-ceph

    Status:

    lcg2268 running with that config. n_connections from xrdceph file close metrics show similar numbers compared to other nodes in 5.3.3-2 without readv

    lcg2268 (with change)
    lcg2267 (without changes)

     

    No failures of lhcb jobs were observed on the test WN so far.

     

     

     

    EBUSY in readV requests

     

    https://stfc.atlassian.net/browse/XRD-64

     

    Discussion on merging bufferedIO into master.
    Also to discuss pushing changes to “upstream” (xrootd/xroot-ceph”

     

    Buffered io merge into master by Jo-stfc · Pull Request #44 · stfc/xrootd-ceph

    Needs testing for ‘correctness’

    Also some discussion ongoing on xrootd “issues” on the xrootd-ceph sub-module:
    Remove xrootd-ceph git submodule and bring the code back to the main repository by amadio · Pull Request #2008 · xrootd/xrootd

     

    SEGV investigations with -S multi-stream flags

     

    @Ian Johnson Preparing script for ‘git-bisect run <script>’ to run, automating the search of a range of commits. This should identify the commit that introduced the unexpected corrective action that now allows multiple streams in XrdCeph without the SEGV. Guilherme commented in segfault from XrdXrootdProtocol::do_OffloadIO · Issue #1821 · xrootd/xrootd about reading release notes for a clue; I tried and didn’t find anything relevant.

     

    CMSD status

     

    https://stfc.atlassian.net/browse/XRD-41

    @Thomas, Jyothish (STFC,RAL,SC) identified that xrdcp via the redirector using root protocol requires two “/” in the LFN to work correctly.

    • Although this is the nominally correct path construction for xroot-based paths; existing transfers work with just a single-/ and may break from some corner-case person/VO.

    Assuming we want a solution which maintains backwards compatibility, we may need to explicitly add:

    all.export / # already existing line # add in the lines all.export cms all.export store #special cms 'namespace' # and for other pools

    What’s the impact? Do we need “/” export now. Should we consider the authdb file logic too ?
    To understand exactly how the export / authdb / “// | /" finally interact ?

     

     

    Transfers of 0-byte files

     

    https://stfc.atlassian.net/browse/XRD-62Q: Could close?

    Observed Dune transfer failures using 0-byte files/

    @Thomas, Jyothish (STFC,RAL,SC) identified FTS not supporting 0-byte transfers, hence the cause of the reported failures.

    @Ian Johnson noted that the “classic” RADOS striper implementation doesn’t appear to create an object until data is written to it; is there a use case which might call for the creation of 0-byte files?

     

     

     

     

     

    on GGUS:

    Deletion problem at RAL

    Slow stat calls at RAL

    Problem accessing some LHCb files at RAL

    Site reports

     Action items

    •  

     

     Decisions

    1. Gateway configuration following Option 1 is preferred: “Move to 5.5.4-2 (core) + 5.5.4-3 (xroot-ceph-buffered)”