ChchACreateClustsuecheckAndCreateCluster issue
- Thomas, Jyothish (STFC,RAL,SC)
Previous behaviour
https://stfc.atlassian.net/wiki/spaces/CD/pages/30998699/CLOSE+WAIT+echo+gw+failure+mode
Current behaviour:
No CLOSE WAIT buildup, service restarts after ~2minutes of a new request coming in. Log:
230119 09:32:49 getRadosStriper : checkAndCreateStriper failed
230119 09:32:49 419109 ofs_stat: 45954061.26:35@lcgui05 Unable to locate /dteam:test450.txt; invalid argument
230119 09:32:49 419109 XrootdXeq: 45954061.26:35@lcgui05 disc 0:00:01 (send failure)
230119 09:32:57 418847 XrdVomsFun: proxy: /C=UK/O=eScience/OU=CLRC/L=RAL/CN=jyothish thomas/CN=1867364546
230119 09:32:57 418847 XrdVomsFun: adding cert: /C=UK/O=eScience/OU=CLRC/L=RAL/CN=jyothish thomas
230119 09:32:57 418847 XrdVomsFun: retrieval successful
230119 09:32:57 418847 XrdVomsFun: found VO: dteam
230119 09:32:57 418847 XrdVomsFun: ---> group: '/dteam', role: 'NULL', cap: 'NULL'
230119 09:32:57 418847 XrdVomsFun: ---> fqan: '/dteam/Role=NULL/Capability=NULL'
230119 09:32:57 418847 XrootdXeq: fur43467.3501210:32@lcgui05 pub IP64 login as 45954061.0
230119 09:32:57 418847 acc_Audit: fur43467.3501210:32@lcgui05 grant gsi 45954061.0@lcgui05.gridpp.rl.ac.uk stat dteam:test450.txt
230119 09:32:57 ceph_stat: dteam:test450.txt
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.15/rpm/el7/BUILD/ceph-14.2.15/src/osdc/Objecter.cc: In function 'void Objecter::_op_submit_with_budget(Objecter::Op*, Objecter::shunique_lock&, ceph_tid_t*, int*)' thread 7fe6fa4fa700 time 2023-01-19 09:32:57.854504
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.15/rpm/el7/BUILD/ceph-14.2.15/src/osdc/Objecter.cc: 2258: FAILED ceph_assert(initialized)
ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fe6eec56b27]
2: (()+0x25ccef) [0x7fe6eec56cef]
3: (Objecter::_op_submit_with_budget(Objecter::Op*, ceph::shunique_lockstd::shared_mutex&, unsigned long*, int*)+0x396) [0x7fe6f78a9506]
4: (Objecter::op_submit(Objecter::Op*, unsigned long*, int*)+0x90) [0x7fe6f78a95d0]
5: (librados::IoCtxImpl::aio_stat(object_t const&, librados::AioCompletionImpl*, unsigned long*, long*)+0x8f) [0x7fe6f786806f]
6: (int libradosstriper::RadosStriperImpl::aio_generic_stat<long>(std::string const&, librados::AioCompletionImpl*, unsigned long*, long*, libradosstriper::RadosStriperImpl::StatFunction<long>::Type)+0x22c) [0x7fe6f785ef0c]
7: (libradosstriper::RadosStriperImpl::aio_stat(std::string const&, librados::AioCompletionImpl*, unsigned long*, long*)+0x1a) [0x7fe6f785594a]
8: (libradosstriper::RadosStriperImpl::stat(std::string const&, unsigned long*, long*)+0x4d) [0x7fe6f785599d]
9: (ceph_posix_stat(XrdOucEnv*, char const*, stat*)+0xa5) [0x7fe6f7e61575]
10: (XrdCephOss::Stat(char const*, stat*, int, XrdOucEnv*)+0x38) [0x7fe6f807b188]
11: (XrdOfs::stat(char const*, stat*, XrdOucErrInfo&, XrdSecEntity const*, char const*)+0xf2) [0x7fe7009f8852]
12: (XrdXrootdProtocol::do_Stat()+0x3d6) [0x7fe7009ee376]
13: (XrdXrootdProtocol::Process2()+0x5af) [0x7fe7009dc0cf]
14: (XrdLinkXeq::DoIt()+0x46) [0x7fe70071f3e6]
15: (XrdLink::setProtocol(XrdProtocol*, bool, bool)+0x29) [0x7fe70071bbb9]
16: (XrdScheduler::Run()+0x13a) [0x7fe7007224ca]
17: (XrdStartWorking(void*)+0x9) [0x7fe7007225e9]
18: (XrdSysThread_Xeq()+0x37) [0x7fe7006b01e7]
19: (()+0x7ea5) [0x7fe6ff818ea5]
20: (clone()+0x6d) [0x7fe6ff54198d]
230119 09:33:08 419517 Starting on Linux 3.10.0-1160.80.1.el7.x86_64