Large-scale vector read patch: notes

Timeline

Patch was applied to the following generation on the 13th of July:
- 2017-xma
- 2018-xma
- 2021-xma
- 2022-lenovo
- +2 WNs from 2017 dell (lcg226[78])
Test jobs are used to asses the impact of the patch on vector read errors
From the very beginning failures of the test jobs were spotted on the 2017-xma generation.

Below are the plots with distribution of test job failures accross generations after ~1 week of testing.

Here are the reference plots (data obtained before patching):

Errors

The errors look the same as before – timeouts. Here is an example:

Error in <TNetXNGFile::ReadBuffers>: [ERROR] Socket timeout
Error in <TBranchElement::GetBasket>: File: root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/LHCb/Collision17/CHARMSPEC.MDST/00066595/0000/00066595_00008330_1.charmspec.mdst at byte:2639000, branch:_Event., entry:100, badread=1, nerrors=1, basketnumber=2
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-1154343242) ; trying to recover by setting it to zero

The error happened at 14:35:54 BST, at least not after that:

2023-06-14 13:35:54 UTC dirac-jobexec/LHCbScript ERROR: Non-zero status while executing /cvmfs/lhcb.cern.ch/lhcbdirac/versions/v11.0.11-1686558149/Linux-x86_64/bin/python3.9 jobScript/GaudiExec_arogovsk_Job_7.39_68c2254a-795b-4ad0-bf39-e3caa0193472_script.py exited with status 16
2023-06-14 13:35:54 UTC dirac-jobexec/LHCbScript INFO: Output written to Script1_Ganga_GaudiExec.log, execution complete.

Given the standard timeout value (60 seconds), original read(v) request was submitted not after 14:34:54 BST. Also note the offset where the failure happened – 2639000.

On the server side (lcg2294) the file was opened at 14:34:26, active reads were ongoing for ~2.5 minutes:

Jun 14 14:34:26 lcg2294 docker[2260133]: XrdCephOssBufferedFile::Open got fd: 64 lhcb:prod/lhcb/LHCb/Collision17/CHARMSPEC.MDST/00066595/0000/00066595_00008330_1.charmspec.mdst
[..]
Jun 14 14:36:55 lcg2294 docker[2260133]: XrdCephOssBufferedFile::Summary: {"fd":64, "Elapsed_time_ms":149284, "path":"lhcb:prod/lhcb/LHCb/Collision17/CHARMSPEC.MDST/00066595/0000/00066595_00008330_1.charmspec.mdst", read_B:1253870959, readV_B:0, readAIO_B:0, writeB:0, writeAIO_B:0, startTime:"2023-06-14 13:34:26", endTime:"2023-06-14 13:36:55", nBuffersRead:4}
Jun 14 14:36:55 lcg2294 docker[2260133]: 230614 13:36:55 ceph_close: closed fd 64 for file prod/lhcb/LHCb/Collision17/CHARMSPEC.MDST/00066595/0000/00066595_00008330_1.charmspec.mdst, read ops count 150, write ops count 0, async write ops 0/0, async pending write bytes 0, async read ops 0/0, bytes written/max offset 0/0, longest async write 0.000000, longest callback invocation 0.000000, last async op age 0.000000

There were no long ceph reads of the file – any read operation longer than 15 seconds should be logged, but there is nothing in the log:

Also we can see that very aggressive read-ahead strategy was used:

Here we can see that the block corresponding to the requested bytes (the first one, from 0 to 8KiB) should have been read by 14:34:46 (since all ceph reads were executed in less then 15 seconds and 31+15 = 46).
This means that xcache had plenty of time (before 14:35:54, when the error happened) to deliver these bytes to the client. However, for some reason this did not happen. Could it be that xcache was too busy with these read-aheads?

Access pattern

Here is the “access pattern” plot for the failed job (after the failure it was restarted with debug logging in isolated environment). It show the offsets of reads and readv chunks (identified from client-side xrootd debug log), as well as the offsets of xcache reads (identified by these Readthrough cache log messages) vs time. We can see that the file is fully read by xcache during the first 1 or 2 minutes of execution, way before the bytes were requested by the client. This might be too aggressive.

some 2017-xma nodes have seen occasional memory allocation errors:
Jun 15 04:59:40 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:40 3671 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/ed/9d/DAOD_JETM1.32139267._000037.pool.root.1

Prefetch test

On one empty WN the following test was done: 30 lhcb user jobs were submitted simultaneously. 8 jobs out of 30 failed. Then the same tests was repeated with prefetch off (pfc.prefetch 0), and all jobs finished successfully.

Error logs:

Gateways
Jun 13 23:39:30 lcg2296.gridpp.rl.ac.uk docker[318744]: 230613 22:39:30 Error while read
Jun 13 23:39:30 lcg2296.gridpp.rl.ac.uk docker[318744]: 230613 22:39:30 Error while read
Jun 13 23:39:30 lcg2296.gridpp.rl.ac.uk docker[318744]: 230613 22:39:30 Error while read
Jun 13 23:39:30 lcg2296.gridpp.rl.ac.uk docker[318744]: 230613 22:39:30 Error while read
Jun 13 09:25:06 lcg2307.gridpp.rl.ac.uk docker[2480083]: 230613 08:25:06 3902 XrootdXeq: u29.7:4821@localhost disc 0:02:22 (link read error)

Jun 14 10:32:09 lcg2298.gridpp.rl.ac.uk docker[388334]: 230614 09:32:09 1004 XrootdXeq: u37.7:3869@localhost disc 4:17:52 (link read error)

Jun 15 10:49:06 lcg2273.gridpp.rl.ac.uk docker[1727653]: 230615 09:49:06 011 XrootdXeq: u55.7:4774@localhost disc 0:04:39 (link read error)
Jun 15 09:50:36 lcg2280.gridpp.rl.ac.uk docker[1450658]: XrdCephOssBufferedFile::Summary: {"fd":1065, "Elapsed_time_ms":1265698, "path":"atlas:datadisk/rucio/mc16_13TeV/1a/cf/DAOD_TOPQ1.25523689._000190.pool.root.1", read_B:260046848, readV_B:198301958, readAIO_B:0, writeB:0, writeAIO_B:0, startTime:"2023-06-15 08:29:31", endTime:"2023-06-15 08:50:36", nBuffersRead:11}230615 08:50:36 636 XrootdXeq: u57.7:4270@localhost disc 1:15:04 (socket error)
Jun 15 05:39:10 lcg2297.gridpp.rl.ac.uk docker[3653708]: 230615 04:39:10 1017 XrootdXeq: u43.7:4457@localhost disc 0:02:04 (socket error)
Jun 15 05:39:10 lcg2297.gridpp.rl.ac.uk docker[3653708]: 230615 04:39:10 010 XrootdXeq: u41.7:4412@localhost disc 0:19:26 (socket error)
Jun 15 09:22:05 lcg2306.gridpp.rl.ac.uk docker[1624657]: 230615 08:22:05 5472 XrootdXeq: u49.7:5057@localhost disc 0:37:57 (link read error)
Jun 15 10:25:09 lcg2310.gridpp.rl.ac.uk docker[3390634]: XrdCephBufferAlgSimple::Destructor, fd=816, retrieved_bytes=2818572288, bypassed_bytes=2818572288, delivered_bytes=2818572288, cache_hit_frac=1230615 09:25:09 019 XrootdXeq: u67.7:4463@localhost disc 1:10:59 (link read error)

Proxy

Jun 13 17:24:48 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:24:48 1044 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:scratchdisk/rucio/user/weitao/5b/4a/user.weitao.31350496._000047.CxAOD.root
Jun 13 17:24:48 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:24:48 1044 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u45@127.0.0.1:1095/atlas:scratchdisk/rucio/user/weitao/5b/4a/user.weitao.31350496._000047.CxAOD.root
Jun 13 17:24:48 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:24:48 3596 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:scratchdisk/rucio/user/weitao/5b/4a/user.weitao.31350496._000047.CxAOD.root
Jun 13 17:24:48 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:24:48 3596 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 4 root://u45@127.0.0.1:1095/atlas:scratchdisk/rucio/user/weitao/5b/4a/user.weitao.31350496._000047.CxAOD.root
Jun 13 17:24:48 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:24:48 104 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:scratchdisk/rucio/user/weitao/5b/4a/user.weitao.31350496._000047.CxAOD.root
Jun 13 17:24:48 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:24:48 107 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:scratchdisk/rucio/user/weitao/5b/4a/user.weitao.31350496._000047.CxAOD.root
Jun 13 17:24:48 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:24:48 107 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: c root://u45@127.0.0.1:1095/atlas:scratchdisk/rucio/user/weitao/5b/4a/user.weitao.31350496._000047.CxAOD.root
Jun 13 17:24:48 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:24:48 104 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: b root://u45@127.0.0.1:1095/atlas:scratchdisk/rucio/user/weitao/5b/4a/user.weitao.31350496._000047.CxAOD.root
Jun 13 17:31:56 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:31:56 466 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/user/weitao/48/12/user.weitao.31350512._000032.CxAOD.root
Jun 13 17:31:56 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:31:56 466 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 4 root://u49@127.0.0.1:1095/atlas:datadisk/rucio/user/weitao/48/12/user.weitao.31350512._000032.CxAOD.root
Jun 13 17:31:56 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:31:56 1043 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/user/weitao/48/12/user.weitao.31350512._000032.CxAOD.root
Jun 13 17:31:56 lcg2272.gridpp.rl.ac.uk docker[1060291]: 230613 16:31:56 1043 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: c root://u49@127.0.0.1:1095/atlas:datadisk/rucio/user/weitao/48/12/user.weitao.31350512._000032.CxAOD.root
Jun 13 22:04:33 lcg2294.gridpp.rl.ac.uk docker[3809275]: 230613 21:04:33 266 XrootdXeq: tatls015.4685:49@htcjob3694441_0_slot1_47_pid270570.ralworker disc 0:38:24 (socket error)
Jun 13 23:32:24 lcg2296.gridpp.rl.ac.uk docker[318651]: CEPHSUM-2023-06-13 22:32:24,493-5397-ERROR-Mismatch in bytes read 0 and striped total size metadata 83886080
Jun 13 23:32:24 lcg2296.gridpp.rl.ac.uk docker[318651]: raise IOError(f"Mismatch in bytes read: {path}, {bytes_read}, {total_size}")
Jun 13 23:32:24 lcg2296.gridpp.rl.ac.uk docker[318651]: OSError: Mismatch in bytes read: prod/lhcb/MC/2011/ALLSTREAMS.DST/00188279/0000/00188279_00000118_5.AllStreams.dst, 0, 83886080
Jun 13 23:39:30 lcg2296.gridpp.rl.ac.uk docker[318651]: 230613 22:39:30 100 XrdPfc_File: error ProcessBlockResponse Wrong number of bytes received, assuming remote/local file size mismatch, unlinking local files and initiating shutdown of File object lhcb:prod/lhcb/MC/2011/ALLSTREAMS.DST/00188279/0000/00188279_00000118_5.AllStreams.dst
Jun 13 23:39:30 lcg2296.gridpp.rl.ac.uk docker[318651]: 230613 22:39:30 119 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-2, error=no such file or directory sid: 0 root://u59@127.0.0.1:1095/lhcb:prod/lhcb/MC/2011/ALLSTREAMS.DST/00188279/0000/00188279_00000118_5.AllStreams.dst
Jun 13 23:39:30 lcg2296.gridpp.rl.ac.uk docker[318651]: 230613 22:39:30 5221 XrdPfc_File: error ProcessBlockResponse Wrong number of bytes received, assuming remote/local file size mismatch, unlinking local files and initiating shutdown of File object lhcb:prod/lhcb/MC/2011/ALLSTREAMS.DST/00188279/0000/00188279_00000118_5.AllStreams.dst
Jun 13 23:39:30 lcg2296.gridpp.rl.ac.uk docker[318651]: 230613 22:39:30 010 XrdPfc_File: error ProcessBlockResponse Wrong number of bytes received, assuming remote/local file size mismatch, unlinking local files and initiating shutdown of File object lhcb:prod/lhcb/MC/2011/ALLSTREAMS.DST/00188279/0000/00188279_00000118_5.AllStreams.dst
Jun 13 23:39:30 lcg2296.gridpp.rl.ac.uk docker[318651]: 230613 22:39:30 100 XrdPfc_File: error ProcessBlockResponse Wrong number of bytes received, assuming remote/local file size mismatch, unlinking local files and initiating shutdown of File object lhcb:prod/lhcb/MC/2011/ALLSTREAMS.DST/00188279/0000/00188279_00000118_5.AllStreams.dst
Jun 13 22:44:19 lcg2298.gridpp.rl.ac.uk docker[388289]: 230613 21:44:19 2413 XrootdXeq: tatls015.4684:30@htcjob3047238_0_slot1_31_pid2294989.ralworker disc 0:41:46 (socket error)
Jun 13 22:58:32 lcg2298.gridpp.rl.ac.uk docker[388289]: 230613 21:58:32 449 XrootdXeq: tatls015.4684:37@htcjob3047238_0_slot1_31_pid2294989.ralworker disc 0:14:12 (socket error)
Jun 13 21:48:10 lcg2302.gridpp.rl.ac.uk docker[1559469]: 230613 20:48:10 1594 XrootdXeq: tatls015.4684:57@htcjob2969847_0_slot1_25_pid2072694.ralworker disc 0:22:38 (socket error)
Jun 13 18:45:30 lcg2303.gridpp.rl.ac.uk docker[3997244]: 230613 17:45:30 102 XrootdXeq: tatls015.2431:30@htcjob2965322_0_slot1_10_pid3905376.ralworker disc 4:58:59 (socket error)

Jun 14 05:00:59 lcg2273.gridpp.rl.ac.uk docker[2736605]: 230614 04:00:59 2797 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/51/89/DAOD_TOPQ1.25524671._000088.pool.root.1.cinfo Checksum Synced or AStats mismatch.
Jun 14 18:57:39 lcg2275.gridpp.rl.ac.uk docker[980457]: 230614 17:57:39 025 XrdPfc_CInfo: error Read() /atlas:datadisk/rucio/data15_13TeV/d1/83/DAOD_JETM1.32515590._000108.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 18:57:39 lcg2275.gridpp.rl.ac.uk docker[980457]: 230614 17:57:39 025 XrdPfc_Purge: warning FPurgeState::TraverseNamespace can't open or read /atlas:datadisk/rucio/data15_13TeV/d1/83/DAOD_JETM1.32515590._000108.pool.root.1.cinfo, err no error; purging.
Jun 14 10:21:40 lcg2311.gridpp.rl.ac.uk docker[2076779]: 230614 09:21:40 1570 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/df/fe/DAOD_TOPQ1.25524907._000016.pool.root.1.cinfo Checksum Synced or AStats mismatch.

Jun 15 05:31:51 lcg2271.gridpp.rl.ac.uk docker[1819819]: 230615 04:31:51 030 XrootdXeq: tatls015.4684:60@htcjob3427732_0_slot1_12_pid2605673.ralworker disc 0:16:19 (socket error)
Jun 15 09:04:17 lcg2271.gridpp.rl.ac.uk docker[1819819]: 230615 08:04:17 1171 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/data16_13TeV/81/96/DAOD_PHYS.30498447._000243.pool.root.1.cinfo Checksum Synced or AStats mismatch.
Jun 15 11:19:37 lcg2271.gridpp.rl.ac.uk docker[3450348]: 230615 10:19:37 025 XrdPfc_CInfo: error Read() /atlas:datadisk/rucio/data17_13TeV/cc/8a/AOD.12866835._000521.pool.root.1.cinfo Checksum Synced or AStats mismatch.
Jun 15 11:19:37 lcg2271.gridpp.rl.ac.uk docker[3450348]: 230615 10:19:37 025 XrdPfc_Purge: warning FPurgeState::TraverseNamespace can't open or read /atlas:datadisk/rucio/data17_13TeV/cc/8a/AOD.12866835._000521.pool.root.1.cinfo, err no error; purging.
Jun 15 09:02:32 lcg2276.gridpp.rl.ac.uk docker[1592705]: 230615 08:02:32 5313 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/data16_13TeV/97/f4/DAOD_PHYS.30498447._000007.pool.root.1.cinfo Checksum Synced or AStats mismatch.
Jun 15 10:55:25 lcg2280.gridpp.rl.ac.uk docker[1751745]: Error response from daemon: cannot stop container: xrootd-proxy: tried to kill container, but did not receive an exit event
Jun 15 04:29:13 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:13 3671 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/34/7e/DAOD_JETM1.32139267._000011.pool.root.1
Jun 15 04:29:13 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:13 3671 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/34/7e/DAOD_JETM1.32139267._000011.pool.root.1
Jun 15 04:29:13 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:13 032 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/34/7e/DAOD_JETM1.32139267._000011.pool.root.1
Jun 15 04:29:13 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:13 032 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/34/7e/DAOD_JETM1.32139267._000011.pool.root.1
Jun 15 04:29:14 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:14 13261 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/28/12/DAOD_JETM1.32139267._000012.pool.root.1
Jun 15 04:29:14 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:14 13261 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/28/12/DAOD_JETM1.32139267._000012.pool.root.1
Jun 15 04:29:14 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:14 11008 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/28/12/DAOD_JETM1.32139267._000012.pool.root.1
Jun 15 04:29:14 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:14 11008 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/28/12/DAOD_JETM1.32139267._000012.pool.root.1
Jun 15 04:29:15 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:15 032 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/6c/cf/DAOD_JETM1.32139267._000013.pool.root.1
Jun 15 04:29:15 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:15 032 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/6c/cf/DAOD_JETM1.32139267._000013.pool.root.1
Jun 15 04:29:15 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:15 12220 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/6c/cf/DAOD_JETM1.32139267._000013.pool.root.1
Jun 15 04:29:15 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:15 12220 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/6c/cf/DAOD_JETM1.32139267._000013.pool.root.1
Jun 15 04:29:16 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:16 12296 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/c2/38/DAOD_JETM1.32139267._000014.pool.root.1
Jun 15 04:29:16 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:16 12296 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/c2/38/DAOD_JETM1.32139267._000014.pool.root.1
Jun 15 04:29:17 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:17 3673 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/c2/38/DAOD_JETM1.32139267._000014.pool.root.1
Jun 15 04:29:17 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:17 3673 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/c2/38/DAOD_JETM1.32139267._000014.pool.root.1
Jun 15 04:29:17 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:17 032 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/b7/0c/DAOD_JETM1.32139267._000015.pool.root.1
Jun 15 04:29:17 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:17 032 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/b7/0c/DAOD_JETM1.32139267._000015.pool.root.1
Jun 15 04:29:17 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:17 3674 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/b7/0c/DAOD_JETM1.32139267._000015.pool.root.1
Jun 15 04:29:17 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:17 3674 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/b7/0c/DAOD_JETM1.32139267._000015.pool.root.1
Jun 15 04:29:18 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:18 12296 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/65/be/DAOD_JETM1.32139267._000016.pool.root.1
Jun 15 04:29:18 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:18 12296 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/65/be/DAOD_JETM1.32139267._000016.pool.root.1
Jun 15 04:29:18 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:18 12220 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/65/be/DAOD_JETM1.32139267._000016.pool.root.1
Jun 15 04:29:18 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:18 12220 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/65/be/DAOD_JETM1.32139267._000016.pool.root.1
Jun 15 04:29:18 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:18 12220 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/e7/b2/DAOD_JETM1.32139267._000017.pool.root.1
Jun 15 04:29:18 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:18 12220 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/e7/b2/DAOD_JETM1.32139267._000017.pool.root.1
Jun 15 04:29:19 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:19 12296 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/e7/b2/DAOD_JETM1.32139267._000017.pool.root.1
Jun 15 04:29:19 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:19 12296 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/e7/b2/DAOD_JETM1.32139267._000017.pool.root.1
Jun 15 04:29:19 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:19 12296 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/86/f8/DAOD_JETM1.32139267._000018.pool.root.1
Jun 15 04:29:19 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:19 12296 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/86/f8/DAOD_JETM1.32139267._000018.pool.root.1
Jun 15 04:29:19 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:19 3671 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/86/f8/DAOD_JETM1.32139267._000018.pool.root.1
Jun 15 04:29:19 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:19 3671 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/86/f8/DAOD_JETM1.32139267._000018.pool.root.1
Jun 15 04:29:20 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:20 3674 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/ff/d5/DAOD_JETM1.32139267._000019.pool.root.1
Jun 15 04:29:20 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:20 3674 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/ff/d5/DAOD_JETM1.32139267._000019.pool.root.1
Jun 15 04:29:20 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:20 13261 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/ff/d5/DAOD_JETM1.32139267._000019.pool.root.1
Jun 15 04:29:20 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:20 13261 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/ff/d5/DAOD_JETM1.32139267._000019.pool.root.1
Jun 15 04:29:21 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:21 3675 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/74/88/DAOD_JETM1.32139267._000020.pool.root.1
Jun 15 04:29:21 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:21 3675 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/74/88/DAOD_JETM1.32139267._000020.pool.root.1
Jun 15 04:29:21 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:21 3163 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/74/88/DAOD_JETM1.32139267._000020.pool.root.1
Jun 15 04:29:21 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:21 3163 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/74/88/DAOD_JETM1.32139267._000020.pool.root.1
Jun 15 04:29:22 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:22 032 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/67/c3/DAOD_JETM1.32139267._000021.pool.root.1
Jun 15 04:29:22 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:22 032 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/67/c3/DAOD_JETM1.32139267._000021.pool.root.1
Jun 15 04:29:22 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:22 3671 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/67/c3/DAOD_JETM1.32139267._000021.pool.root.1
Jun 15 04:29:22 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:22 3671 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/67/c3/DAOD_JETM1.32139267._000021.pool.root.1
Jun 15 04:29:22 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:22 3675 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/b8/63/DAOD_JETM1.32139267._000022.pool.root.1
Jun 15 04:29:22 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:22 3675 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/b8/63/DAOD_JETM1.32139267._000022.pool.root.1
Jun 15 04:29:23 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:23 8955 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/b8/63/DAOD_JETM1.32139267._000022.pool.root.1
Jun 15 04:29:23 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:23 8955 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/b8/63/DAOD_JETM1.32139267._000022.pool.root.1
Jun 15 04:29:23 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:23 032 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/2b/bf/DAOD_JETM1.32139267._000023.pool.root.1
Jun 15 04:29:23 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:23 032 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/2b/bf/DAOD_JETM1.32139267._000023.pool.root.1
Jun 15 04:29:23 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:23 3675 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/2b/bf/DAOD_JETM1.32139267._000023.pool.root.1
Jun 15 04:29:23 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:23 3675 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/2b/bf/DAOD_JETM1.32139267._000023.pool.root.1
Jun 15 04:29:24 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:24 12296 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/5c/91/DAOD_JETM1.32139267._000024.pool.root.1
Jun 15 04:29:24 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:24 12296 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/5c/91/DAOD_JETM1.32139267._000024.pool.root.1
Jun 15 04:29:24 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:24 3674 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/5c/91/DAOD_JETM1.32139267._000024.pool.root.1
Jun 15 04:29:24 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:24 3674 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/5c/91/DAOD_JETM1.32139267._000024.pool.root.1
Jun 15 04:29:25 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:25 3673 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/a6/b0/DAOD_JETM1.32139267._000025.pool.root.1
Jun 15 04:29:25 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:25 3673 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/a6/b0/DAOD_JETM1.32139267._000025.pool.root.1
Jun 15 04:29:25 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:25 13261 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/a6/b0/DAOD_JETM1.32139267._000025.pool.root.1
Jun 15 04:29:25 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:25 13261 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/a6/b0/DAOD_JETM1.32139267._000025.pool.root.1
Jun 15 04:29:26 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:26 3163 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/6b/f5/DAOD_JETM1.32139267._000026.pool.root.1
Jun 15 04:29:26 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:26 3163 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/6b/f5/DAOD_JETM1.32139267._000026.pool.root.1
Jun 15 04:29:27 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:27 3675 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/6b/f5/DAOD_JETM1.32139267._000026.pool.root.1
Jun 15 04:29:27 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:29:27 3675 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/6b/f5/DAOD_JETM1.32139267._000026.pool.root.1
Jun 15 04:59:31 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:31 3675 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/b8/2c/DAOD_JETM1.32139267._000030.pool.root.1
Jun 15 04:59:31 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:31 3675 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/b8/2c/DAOD_JETM1.32139267._000030.pool.root.1
Jun 15 04:59:32 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:32 8656 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/b8/2c/DAOD_JETM1.32139267._000030.pool.root.1
Jun 15 04:59:32 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:32 8656 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/b8/2c/DAOD_JETM1.32139267._000030.pool.root.1
Jun 15 04:59:32 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:32 032 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/0e/88/DAOD_JETM1.32139267._000031.pool.root.1
Jun 15 04:59:32 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:32 032 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/0e/88/DAOD_JETM1.32139267._000031.pool.root.1
Jun 15 04:59:32 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:32 10940 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/0e/88/DAOD_JETM1.32139267._000031.pool.root.1
Jun 15 04:59:32 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:32 10940 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/0e/88/DAOD_JETM1.32139267._000031.pool.root.1
Jun 15 04:59:33 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:33 8656 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/52/2b/DAOD_JETM1.32139267._000032.pool.root.1
Jun 15 04:59:33 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:33 8656 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/52/2b/DAOD_JETM1.32139267._000032.pool.root.1
Jun 15 04:59:33 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:33 3674 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/52/2b/DAOD_JETM1.32139267._000032.pool.root.1
Jun 15 04:59:33 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:33 3674 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/52/2b/DAOD_JETM1.32139267._000032.pool.root.1
Jun 15 04:59:34 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:34 3675 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/fb/76/DAOD_JETM1.32139267._000033.pool.root.1
Jun 15 04:59:34 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:34 3675 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/fb/76/DAOD_JETM1.32139267._000033.pool.root.1
Jun 15 04:59:34 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:34 8656 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/fb/76/DAOD_JETM1.32139267._000033.pool.root.1
Jun 15 04:59:34 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:34 8656 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/fb/76/DAOD_JETM1.32139267._000033.pool.root.1
Jun 15 04:59:34 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:34 13261 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/cf/d3/DAOD_JETM1.32139267._000034.pool.root.1
Jun 15 04:59:34 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:34 13261 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/cf/d3/DAOD_JETM1.32139267._000034.pool.root.1
Jun 15 04:59:35 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:35 12220 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/cf/d3/DAOD_JETM1.32139267._000034.pool.root.1
Jun 15 04:59:35 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:35 12220 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/cf/d3/DAOD_JETM1.32139267._000034.pool.root.1
Jun 15 04:59:36 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:36 032 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/d1/78/DAOD_JETM1.32139267._000035.pool.root.1
Jun 15 04:59:36 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:36 032 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/d1/78/DAOD_JETM1.32139267._000035.pool.root.1
Jun 15 04:59:36 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:36 3675 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/d1/78/DAOD_JETM1.32139267._000035.pool.root.1
Jun 15 04:59:36 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:36 3675 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/d1/78/DAOD_JETM1.32139267._000035.pool.root.1
Jun 15 04:59:37 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:37 3674 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/2f/dc/DAOD_JETM1.32139267._000036.pool.root.1
Jun 15 04:59:37 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:37 3674 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/2f/dc/DAOD_JETM1.32139267._000036.pool.root.1
Jun 15 04:59:40 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:40 12220 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/ed/9d/DAOD_JETM1.32139267._000037.pool.root.1
Jun 15 04:59:40 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:40 12220 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/ed/9d/DAOD_JETM1.32139267._000037.pool.root.1
Jun 15 04:59:40 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:40 3671 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/ed/9d/DAOD_JETM1.32139267._000037.pool.root.1
Jun 15 04:59:40 lcg2281.gridpp.rl.ac.uk docker[3817243]: 230615 03:59:40 3671 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u64@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/ed/9d/DAOD_JETM1.32139267._000037.pool.root.1
Jun 15 06:21:53 lcg2282.gridpp.rl.ac.uk docker[2612277]: 230615 05:21:53 9477 XrootdXeq: tatls015.4684:47@htcjob2986336_0_slot1_35_pid3353693.ralworker disc 0:21:32 (socket error)
Jun 15 06:23:53 lcg2282.gridpp.rl.ac.uk docker[2612277]: 230615 05:23:53 9477 XrootdXeq: tatls015.4684:32@htcjob756446_0_slot1_6_pid2507457.ralworker disc 0:19:57 (socket error)
Jun 15 02:24:00 lcg2302.gridpp.rl.ac.uk docker[3072431]: 230615 01:23:54 978 XrootdXeq: tatls015.4773:48@htcjob3708407_0_slot1_29_pid2004050.ralworker disc 0:33:54 (socket error)
Jun 15 02:27:32 lcg2308.gridpp.rl.ac.uk docker[43368]: 230615 01:27:32 030 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc20_13TeV/82/89/DAOD_JETM1.33054655._000327.pool.root.1
Jun 15 02:27:32 lcg2308.gridpp.rl.ac.uk docker[43368]: 230615 01:27:32 030 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u52@127.0.0.1:1095/atlas:datadisk/rucio/mc20_13TeV/82/89/DAOD_JETM1.33054655._000327.pool.root.1
Jun 15 02:27:32 lcg2308.gridpp.rl.ac.uk docker[43368]: 230615 01:27:32 011 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc20_13TeV/82/89/DAOD_JETM1.33054655._000327.pool.root.1
Jun 15 02:27:32 lcg2308.gridpp.rl.ac.uk docker[43368]: 230615 01:27:32 011 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u52@127.0.0.1:1095/atlas:datadisk/rucio/mc20_13TeV/82/89/DAOD_JETM1.33054655._000327.pool.root.1
Jun 15 02:27:33 lcg2308.gridpp.rl.ac.uk docker[43368]: 230615 01:27:33 2792 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc20_13TeV/ba/cf/DAOD_JETM1.33054655._000328.pool.root.1
Jun 15 02:27:33 lcg2308.gridpp.rl.ac.uk docker[43368]: 230615 01:27:33 2792 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u52@127.0.0.1:1095/atlas:datadisk/rucio/mc20_13TeV/ba/cf/DAOD_JETM1.33054655._000328.pool.root.1
Jun 15 02:27:33 lcg2308.gridpp.rl.ac.uk docker[43368]: 230615 01:27:33 011 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc20_13TeV/ba/cf/DAOD_JETM1.33054655._000328.pool.root.1
Jun 15 02:27:33 lcg2308.gridpp.rl.ac.uk docker[43368]: 230615 01:27:33 011 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u52@127.0.0.1:1095/atlas:datadisk/rucio/mc20_13TeV/ba/cf/DAOD_JETM1.33054655._000328.pool.root.1
Jun 15 07:46:34 lcg2309.gridpp.rl.ac.uk docker[3573813]: 230615 06:46:34 18154 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/data15_13TeV/84/b0/DAOD_PHYS.30497943._000444.pool.root.1.cinfo Checksum Synced or AStats mismatch.
Jun 15 10:05:37 lcg2316.gridpp.rl.ac.uk docker[2637037]: 230615 09:05:37 025 XrdPfc_CInfo: error Read() /atlas:datadisk/rucio/data17_13TeV/c3/cd/DAOD_JETM1.32422056._000269.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 10:05:37 lcg2316.gridpp.rl.ac.uk docker[2637037]: 230615 09:05:37 025 XrdPfc_Purge: warning FPurgeState::TraverseNamespace can't open or read /atlas:datadisk/rucio/data17_13TeV/c3/cd/DAOD_JETM1.32422056._000269.pool.root.1.cinfo, err no error; purging.
Jun 15 08:39:43 lcg2321.gridpp.rl.ac.uk docker[2107833]: 230615 07:39:43 010 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/data15_13TeV/08/23/DAOD_PHYS.30497984._000118.pool.root.1.cinfo Checksum Synced or AStats mismatch.
Jun 15 05:32:25 lcg2324.gridpp.rl.ac.uk docker[4049633]: 230615 04:32:25 025 XrdPfc_CInfo: error Read() /lhcb:prod/lhcb/LHCb/Collision12/EW.DST/00041834/0003/00041834_00030966_1.ew.dst.cinfo Checksum Store mismatch.
Jun 15 05:32:25 lcg2324.gridpp.rl.ac.uk docker[4049633]: 230615 04:32:25 025 XrdPfc_Purge: warning FPurgeState::TraverseNamespace can't open or read /lhcb:prod/lhcb/LHCb/Collision12/EW.DST/00041834/0003/00041834_00030966_1.ew.dst.cinfo, err no error; purging.
Jun 15 06:01:10 lcg2324.gridpp.rl.ac.uk docker[4049633]: 230615 05:01:10 10565 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/f7/27/DAOD_JETM1.32139156._000292.pool.root.1
Jun 15 06:01:10 lcg2324.gridpp.rl.ac.uk docker[4049633]: 230615 05:01:10 10565 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 3 root://u46@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/f7/27/DAOD_JETM1.32139156._000292.pool.root.1
Jun 15 06:01:11 lcg2324.gridpp.rl.ac.uk docker[4049633]: 230615 05:01:11 768 XrdPfc_File: error Read(), direct read finished with error 12 cannot allocate memory atlas:datadisk/rucio/mc16_13TeV/f7/27/DAOD_JETM1.32139156._000292.pool.root.1
Jun 15 06:01:11 lcg2324.gridpp.rl.ac.uk docker[4049633]: 230615 05:01:11 768 XrdPfc_IO: warning ReadEnd() error in File::Read(), exit status=-12, error=cannot allocate memory sid: 7 root://u46@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/f7/27/DAOD_JETM1.32139156._000292.pool.root.1

connection timeout errors present in unpatched 2017-dell are not seen in 2017-xma. Instead there’s a relatively high number of memory allocation errors troughout

Gateway

Jun 13 15:56:54 lcg2221.gridpp.rl.ac.uk docker[3565084]: 230613 14:56:54 1431 XrootdXeq: u42.7:2723@localhost disc 0:15:58 (socket error)
Jun 13 08:02:44 lcg2269.gridpp.rl.ac.uk docker[3207684]: 230613 07:02:44 3845 XrootdXeq: u44.7:2407@localhost disc 14:40:06 (socket error)

Jun 15 10:06:07 lcg2199.gridpp.rl.ac.uk docker[1717238]: 230615 09:06:07 6340 XrootdXeq: u43.7:5142@localhost disc 0:02:03 (socket error)
Jun 15 10:06:07 lcg2199.gridpp.rl.ac.uk docker[1717238]: 230615 09:06:07 6341 XrootdXeq: u41.7:5118@localhost disc 0:02:22 (socket error)
Jun 15 11:02:19 lcg2231.gridpp.rl.ac.uk docker[2711013]: 230615 10:02:19 040 XrootdXeq: u36.7:210@localhost disc 0:04:20 (socket error)
Jun 15 11:04:19 lcg2242.gridpp.rl.ac.uk docker[3644632]: 230615 10:04:19 062 XrootdXeq: u59.7:507@localhost disc 0:14:30 (socket error)

Proxy

Jun 13 13:12:13 lcg2201.gridpp.rl.ac.uk docker[874161]: 230613 12:12:13 3935 XrdPfc_File: error ProcessBlockResponse block 0x7f1216c1be70, idx=34, off=285212672 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0037/00187858_00373225_1.sim
Jun 13 13:12:13 lcg2201.gridpp.rl.ac.uk docker[874161]: 230613 12:12:13 102 XrdPfc_File: error Read() io 0x7f11fe841340, block 34 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187858/0037/00187858_00373225_1.sim
Jun 13 13:12:13 lcg2201.gridpp.rl.ac.uk docker[874161]: 230613 12:12:13 102 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u55@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187858/0037/00187858_00373225_1.sim?&pss.tid=tlhcb006.33:55@htcjob3686052_0_slot1_39_pid1508459.ralworker&oss.lcl=1
Jun 13 17:38:57 lcg2201.gridpp.rl.ac.uk docker[874161]: 230613 16:38:57 099 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc20_13TeV/b4/cd/DAOD_PHYS.33654469._000068.pool.root.1.cinfo Checksum Store mismatch.
Jun 13 14:41:12 lcg2205.gridpp.rl.ac.uk docker[2850385]: 230613 13:41:12 259 XrdPfc_File: error ProcessBlockResponse block 0x7fbaf494d5b0, idx=417, off=3498049536 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285312_1.sim
Jun 13 14:41:12 lcg2205.gridpp.rl.ac.uk docker[2850385]: 230613 13:41:12 3325 XrdPfc_File: error Read() io 0x7fbad3507100, block 417 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285312_1.sim
Jun 13 14:41:12 lcg2205.gridpp.rl.ac.uk docker[2850385]: 230613 13:41:12 3325 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u40@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285312_1.sim?&pss.tid=tlhcb006.33:40@htcjob3690999_0_slot1_32_pid3718783.ralworker&oss.lcl=1
Jun 13 14:41:12 lcg2205.gridpp.rl.ac.uk docker[2850385]: 230613 13:41:12 3340 XrdPfc_File: error ProcessBlockResponse block 0x7fbaf494d600, idx=418, off=3506438144 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285312_1.sim
Jun 13 14:41:12 lcg2205.gridpp.rl.ac.uk docker[2850385]: 230613 13:41:12 3336 XrdPfc_File: error Read() io 0x7fbad3507100, block 418 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285312_1.sim
Jun 13 14:41:12 lcg2205.gridpp.rl.ac.uk docker[2850385]: 230613 13:41:12 3336 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u40@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285312_1.sim?&pss.tid=tlhcb006.33:40@htcjob3690999_0_slot1_32_pid3718783.ralworker&oss.lcl=1
Jun 13 14:41:12 lcg2205.gridpp.rl.ac.uk docker[2850385]: 230613 13:41:12 2330 XrdPfc_File: error ProcessBlockResponse block 0x7fbaf494d650, idx=419, off=3514826752 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285312_1.sim
Jun 13 14:41:12 lcg2205.gridpp.rl.ac.uk docker[2850385]: 230613 13:41:12 3346 XrdPfc_File: error Read() io 0x7fbad3507100, block 419 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285312_1.sim
Jun 13 14:41:12 lcg2205.gridpp.rl.ac.uk docker[2850385]: 230613 13:41:12 3346 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u40@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285312_1.sim?&pss.tid=tlhcb006.33:40@htcjob3690999_0_slot1_32_pid3718783.ralworker&oss.lcl=1
Jun 13 15:56:12 lcg2205.gridpp.rl.ac.uk docker[2850385]: 230613 14:56:12 2330 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/99/ae/DAOD_TOPQ1.29045750._000401.pool.root.1.cinfo Checksum Store mismatch.
Jun 13 11:45:20 lcg2215.gridpp.rl.ac.uk docker[2954171]: 230613 10:45:20 102 XrdPfc_File: error ProcessBlockResponse block 0x7f3e93896470, idx=116, off=973078528 error=-110 atlas:datadisk/rucio/mc20_13TeV/3a/fd/DAOD_EGAM1.27824509._000169.pool.root.1
Jun 13 11:45:20 lcg2215.gridpp.rl.ac.uk docker[2954171]: 230613 10:45:20 315 XrdPfc_File: error ProcessBlockResponse block 0x7f3e938963d0, idx=115, off=964689920 error=-110 atlas:datadisk/rucio/mc20_13TeV/3a/fd/DAOD_EGAM1.27824509._000169.pool.root.1
Jun 13 13:37:08 lcg2215.gridpp.rl.ac.uk docker[2954171]: 230613 12:37:08 3474 XrootdXeq: tlhcb006.945:27@htcjob2957446_0_slot1_18_pid1646467.ralworker disc 0:04:09 (socket error)
Jun 13 02:33:41 lcg2216.gridpp.rl.ac.uk docker[1178371]: 230613 01:33:41 025 XrdPfc_CInfo: error Read() /atlas:datadisk/rucio/mc16_13TeV/ef/02/DAOD_JETM3.33553797._000050.pool.root.1.cinfo Checksum Store mismatch.
Jun 13 02:33:41 lcg2216.gridpp.rl.ac.uk docker[1178371]: 230613 01:33:41 025 XrdPfc_Purge: warning FPurgeState::TraverseNamespace can't open or read /atlas:datadisk/rucio/mc16_13TeV/ef/02/DAOD_JETM3.33553797._000050.pool.root.1.cinfo, err no error; purging.
Jun 13 15:49:41 lcg2223.gridpp.rl.ac.uk docker[1518971]: 230613 14:49:41 103 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/dc/28/DAOD_TOPQ1.29045750._000059.pool.root.1.cinfo Checksum Store mismatch.
Jun 13 16:30:32 lcg2226.gridpp.rl.ac.uk docker[804269]: 230613 15:30:32 388 XrdPfc_File: error ProcessBlockResponse block 0x7f78e847f9c0, idx=30, off=251658240 error=-110 lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 16:30:32 lcg2226.gridpp.rl.ac.uk docker[804269]: 230613 15:30:32 2729 XrdPfc_File: error Read() io 0x7f78c8c126d0, block 30 finished with error 110 connection timed out lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 16:30:32 lcg2226.gridpp.rl.ac.uk docker[804269]: 230613 15:30:32 2729 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u38@127.0.0.1:1095/lhcb:user/lhcb/user/t/tevans/Allen.tar.gz?&pss.tid=tlhcb006.1215:38@htcjob3399652_0_slot1_19_pid3751410.ralworker&oss.lcl=1
Jun 13 17:41:00 lcg2228.gridpp.rl.ac.uk docker[2909566]: 230613 16:41:00 323 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc20_13TeV/f4/b1/DAOD_PHYS.33654487._000043.pool.root.1.cinfo Checksum Store mismatch.
Jun 13 18:24:19 lcg2231.gridpp.rl.ac.uk docker[180996]: 230613 17:24:19 7515 XrdPfc_File: error ProcessBlockResponse block 0x7f4c7087f100, idx=358, off=3003121664 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379515_1.sim
Jun 13 18:24:19 lcg2231.gridpp.rl.ac.uk docker[180996]: 230613 17:24:19 7333 XrdPfc_File: error Read() io 0x7f4c4d810280, block 358 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379515_1.sim
Jun 13 18:24:19 lcg2231.gridpp.rl.ac.uk docker[180996]: 230613 17:24:19 7333 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u53@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379515_1.sim?&pss.tid=tlhcb006.2103:53@htcjob3685011_0_slot1_5_pid2331735.ralworker&oss.lcl=1
Jun 13 18:24:19 lcg2231.gridpp.rl.ac.uk docker[180996]: 230613 17:24:19 011 XrdPfc_File: error ProcessBlockResponse block 0x7f4c7087f150, idx=359, off=3011510272 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379515_1.sim
Jun 13 18:24:19 lcg2231.gridpp.rl.ac.uk docker[180996]: 230613 17:24:19 5521 XrdPfc_File: error Read() io 0x7f4c4d810280, block 359 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379515_1.sim
Jun 13 18:24:19 lcg2231.gridpp.rl.ac.uk docker[180996]: 230613 17:24:19 5521 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u53@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379515_1.sim?&pss.tid=tlhcb006.2103:53@htcjob3685011_0_slot1_5_pid2331735.ralworker&oss.lcl=1
Jun 13 18:24:19 lcg2231.gridpp.rl.ac.uk docker[180996]: 230613 17:24:19 7516 XrdPfc_File: error ProcessBlockResponse block 0x7f4c7087f1a0, idx=360, off=3019898880 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379515_1.sim
Jun 13 18:24:19 lcg2231.gridpp.rl.ac.uk docker[180996]: 230613 17:24:19 5523 XrdPfc_File: error Read() io 0x7f4c4d810280, block 360 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379515_1.sim
Jun 13 18:24:19 lcg2231.gridpp.rl.ac.uk docker[180996]: 230613 17:24:19 5523 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u53@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379515_1.sim?&pss.tid=tlhcb006.2103:53@htcjob3685011_0_slot1_5_pid2331735.ralworker&oss.lcl=1
Jun 13 12:36:28 lcg2234.gridpp.rl.ac.uk docker[27808]: 230613 11:36:28 023 XrdPfc_File: error ProcessBlockResponse block 0x7f1dcd00e240, idx=342, off=2868903936 error=-42 lhcb:prod/lhcb/LHCb/Collision18/DIMUON.DST/00146832/0000/00146832_00003247_1.dimuon.dst
Jun 13 12:25:38 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 11:25:38 191 XrdPfc_File: error ProcessBlockResponse block 0x7f28c4c0ece0, idx=323, off=2709520384 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0037/00187858_00372817_1.sim
Jun 13 12:25:38 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 11:25:38 102 XrdPfc_File: error Read() io 0x7f28c4d7ddf0, block 323 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187858/0037/00187858_00372817_1.sim
Jun 13 12:25:38 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 11:25:38 031 XrdPfc_File: error ProcessBlockResponse block 0x7f28c4c0ef60, idx=325, off=2726297600 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0037/00187858_00372817_1.sim
Jun 13 12:25:38 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 11:25:38 102 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u52@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187858/0037/00187858_00372817_1.sim?&pss.tid=tlhcb006.918:52@htcjob3681118_0_slot1_11_pid1150825.ralworker&oss.lcl=1
Jun 13 12:25:38 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 11:25:38 1651 XrdPfc_File: error ProcessBlockResponse block 0x7f28c4c0ed30, idx=324, off=2717908992 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0037/00187858_00372817_1.sim
Jun 13 13:03:36 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 12:03:36 232 XrdPfc_File: error ProcessBlockResponse block 0x7f28c4cde7e0, idx=306, off=2566914048 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00372553_1.sim
Jun 13 13:03:36 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 12:03:36 233 XrdPfc_File: error Read() io 0x7f28c0419f70, block 306 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00372553_1.sim
Jun 13 13:03:36 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 12:03:36 233 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u51@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00372553_1.sim?&pss.tid=tlhcb006.4356:51@htcjob2950901_0_slot1_18_pid2394890.ralworker&oss.lcl=1
Jun 13 13:03:36 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 12:03:36 234 XrdPfc_File: error ProcessBlockResponse block 0x7f28c4cde8d0, idx=307, off=2575302656 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00372553_1.sim
Jun 13 13:03:36 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 12:03:36 1919 XrdPfc_File: error Read() io 0x7f28c0419f70, block 307 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00372553_1.sim
Jun 13 13:03:36 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 12:03:36 031 XrdPfc_File: error ProcessBlockResponse block 0x7f28c4cde920, idx=308, off=2583691264 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00372553_1.sim
Jun 13 13:03:36 lcg2236.gridpp.rl.ac.uk docker[825094]: 230613 12:03:36 1919 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u51@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00372553_1.sim?&pss.tid=tlhcb006.4356:51@htcjob2950901_0_slot1_18_pid2394890.ralworker&oss.lcl=1
Jun 13 09:20:18 lcg2238.gridpp.rl.ac.uk docker[1652523]: Error response from daemon: cannot stop container: xrootd-proxy: tried to kill container, but did not receive an exit event
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 428 XrdPfc_File: error ProcessBlockResponse block 0x7fa876b5f740, idx=73, off=612368384 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 032 XrdPfc_File: error Read() io 0x7fa88fd3c820, block 73 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 032 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u43@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim?&pss.tid=tlhcb006.9800:43@htcjob3027570_0_slot1_4_pid3833291.ralworker
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 930 XrdPfc_File: error ProcessBlockResponse block 0x7fa876b5f790, idx=74, off=620756992 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 429 XrdPfc_File: error Read() io 0x7fa88fd3c820, block 74 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 429 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u43@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim?&pss.tid=tlhcb006.9800:43@htcjob3027570_0_slot1_4_pid3833291.ralworker
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 119 XrdPfc_File: error ProcessBlockResponse block 0x7fa876b5f880, idx=75, off=629145600 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 772 XrdPfc_File: error Read() io 0x7fa88fd3c820, block 75 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 772 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u43@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim?&pss.tid=tlhcb006.9800:43@htcjob3027570_0_slot1_4_pid3833291.ralworker
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 889 XrdPfc_File: error ProcessBlockResponse block 0x7fa876b5f8d0, idx=76, off=637534208 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 031 XrdPfc_File: error Read() io 0x7fa88fd3c820, block 76 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim
Jun 13 13:40:01 lcg2239.gridpp.rl.ac.uk docker[3222875]: 230613 12:40:01 031 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u43@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00285685_1.sim?&pss.tid=tlhcb006.9800:43@htcjob3027570_0_slot1_4_pid3833291.ralworker
Jun 13 10:02:19 lcg2245.gridpp.rl.ac.uk docker[3395744]: Error response from daemon: cannot stop container: xrootd-proxy: tried to kill container, but did not receive an exit event
Jun 13 20:44:24 lcg2245.gridpp.rl.ac.uk docker[3399413]: 230613 19:44:24 3539 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/c3/81/DAOD_TOPQ1.25526058._001371.pool.root.1.cinfo Checksum Store mismatch.
Jun 13 12:25:15 lcg2248.gridpp.rl.ac.uk docker[1652556]: 230613 11:25:15 103 XrdPfc_File: error ProcessBlockResponse block 0x7fa438159650, idx=381, off=3196059648 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00284765_1.sim
Jun 13 12:25:15 lcg2248.gridpp.rl.ac.uk docker[1652556]: 230613 11:25:15 102 XrdPfc_File: error ProcessBlockResponse block 0x7fa4381596a0, idx=155, off=1300234240 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00360688_1.sim
Jun 13 12:25:15 lcg2248.gridpp.rl.ac.uk docker[1652556]: 230613 11:25:15 030 XrdPfc_File: error Read() io 0x7fa41c022af0, block 381 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00284765_1.sim
Jun 13 12:25:15 lcg2248.gridpp.rl.ac.uk docker[1652556]: 230613 11:25:15 158 XrdPfc_File: error Read() io 0x7fa42414d430, block 155 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00360688_1.sim
Jun 13 12:25:15 lcg2248.gridpp.rl.ac.uk docker[1652556]: 230613 11:25:15 030 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u54@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187864/0028/00187864_00284765_1.sim?&pss.tid=tlhcb006.937:54@htcjob3402249_0_slot1_18_pid1928899.ralworker&oss.lcl=1
Jun 13 12:25:15 lcg2248.gridpp.rl.ac.uk docker[1652556]: 230613 11:25:15 158 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u49@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00360688_1.sim?&pss.tid=tlhcb006.5576:49@htcjob3399158_0_slot1_6_pid3114930.ralworker&oss.lcl=1
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 2755 XrdPfc_File: error ProcessBlockResponse block 0x7f41febfbce0, idx=258, off=2164260864 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 102 XrdPfc_File: error Read() io 0x7f423d1b0820, block 258 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 102 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u27@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim?&pss.tid=tlhcb006.2488:27@htcjob2959894_0_slot1_37_pid895859.ralworker&oss.lcl=1
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 278 XrdPfc_File: error ProcessBlockResponse block 0x7f41febfbd80, idx=259, off=2172649472 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 102 XrdPfc_File: error Read() io 0x7f423d1b0820, block 259 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 102 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u27@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim?&pss.tid=tlhcb006.2488:27@htcjob2959894_0_slot1_37_pid895859.ralworker&oss.lcl=1
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 101 XrdPfc_File: error ProcessBlockResponse block 0x7f41febfbdd0, idx=260, off=2181038080 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 011 XrdPfc_File: error Read() io 0x7f423d1b0820, block 260 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 011 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u27@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim?&pss.tid=tlhcb006.2488:27@htcjob2959894_0_slot1_37_pid895859.ralworker&oss.lcl=1
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 277 XrdPfc_File: error ProcessBlockResponse block 0x7f41febfbe20, idx=261, off=2189426688 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 2756 XrdPfc_File: error Read() io 0x7f423d1b0820, block 261 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim
Jun 13 12:33:25 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:33:25 2756 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u27@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363436_1.sim?&pss.tid=tlhcb006.2488:27@htcjob2959894_0_slot1_37_pid895859.ralworker&oss.lcl=1
Jun 13 12:37:40 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:37:40 277 XrdPfc_File: error ProcessBlockResponse block 0x7f421ac1c1f0, idx=391, off=3279945728 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363831_1.sim
Jun 13 12:37:40 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:37:40 101 XrdPfc_File: error ProcessBlockResponse block 0x7f421ac1c380, idx=392, off=3288334336 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363831_1.sim
Jun 13 12:37:40 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:37:40 102 XrdPfc_File: error ProcessBlockResponse block 0x7f421ac1c4c0, idx=393, off=3296722944 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363831_1.sim
Jun 13 12:37:40 lcg2251.gridpp.rl.ac.uk docker[2780765]: 230613 11:37:40 100 XrdPfc_File: error ProcessBlockResponse block 0x7f421ac1cb00, idx=394, off=3305111552 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187858/0036/00187858_00363831_1.sim
Jun 13 16:06:39 lcg2258.gridpp.rl.ac.uk docker[2103584]: 230613 15:06:39 5850 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/08/d6/DAOD_TOPQ1.25522933._000254.pool.root.1.cinfo Checksum Store mismatch.
Jun 13 23:47:00 lcg2263.gridpp.rl.ac.uk docker[3792453]: 230613 22:47:00 12538 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/91/48/DAOD_TOPQ1.25523157._000011.pool.root.1.cinfo Checksum Store mismatch.
Jun 13 17:40:43 lcg2269.gridpp.rl.ac.uk docker[690798]: 230613 16:40:43 2668 XrdPfc_File: error ProcessBlockResponse block 0x7f8091966e20, idx=33, off=276824064 error=-110 lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 17:40:43 lcg2269.gridpp.rl.ac.uk docker[690798]: 230613 16:40:43 917 XrdPfc_File: error Read() io 0x7f806181b4f0, block 33 finished with error 110 connection timed out lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 17:40:43 lcg2269.gridpp.rl.ac.uk docker[690798]: 230613 16:40:43 917 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u51@127.0.0.1:1095/lhcb:user/lhcb/user/t/tevans/Allen.tar.gz?&pss.tid=tlhcb006.209:51@htcjob3691276_0_slot1_25_pid2671403.ralworker&oss.lcl=1
Jun 13 21:42:31 lcg2269.gridpp.rl.ac.uk docker[690798]: 230613 20:42:31 7732 XrootdXeq: tatls015.4684:51@htcjob3695014_0_slot1_31_pid1565744.ralworker disc 0:16:29 (socket error)

Jun 14 02:38:01 lcg2204.gridpp.rl.ac.uk docker[300212]: 230614 01:38:01 13735 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/00/88/DAOD_TOPQ1.31482162._000190.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 18:02:24 lcg2205.gridpp.rl.ac.uk docker[1925005]: 230614 17:02:24 103 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/6d/82/DAOD_TOPQ1.25522636._000037.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 20:54:12 lcg2212.gridpp.rl.ac.uk docker[3788043]: 230614 19:54:12 5859 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/4c/02/DAOD_TOPQ1.25524095._000029.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 08:43:08 lcg2214.gridpp.rl.ac.uk docker[1374875]: 230614 07:43:08 13229 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/b4/8c/DAOD_TOPQ1.25524270._000045.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 102 XrdPfc_File: error ProcessBlockResponse block 0x7f139ec4cb50, idx=56, off=469762048 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 031 XrdPfc_File: error Read() io 0x7f1370418760, block 56 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 031 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u70@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim?&pss.tid=tlhcb006.33:70@htcjob3715186_0_slot1_26_pid517875.ralworker&oss.lcl=1
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 1456 XrdPfc_File: error ProcessBlockResponse block 0x7f139ec4cc40, idx=57, off=478150656 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 1175 XrdPfc_File: error Read() io 0x7f1370418760, block 57 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 1175 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u70@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim?&pss.tid=tlhcb006.33:70@htcjob3715186_0_slot1_26_pid517875.ralworker&oss.lcl=1
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 1479 XrdPfc_File: error ProcessBlockResponse block 0x7f139ec4cd30, idx=58, off=486539264 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 1457 XrdPfc_File: error Read() io 0x7f1370418760, block 58 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 1457 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u70@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim?&pss.tid=tlhcb006.33:70@htcjob3715186_0_slot1_26_pid517875.ralworker&oss.lcl=1
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 030 XrdPfc_File: error ProcessBlockResponse block 0x7f139ec4ce20, idx=59, off=494927872 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 1176 XrdPfc_File: error Read() io 0x7f1370418760, block 59 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim
Jun 14 21:07:09 lcg2226.gridpp.rl.ac.uk docker[2030663]: 230614 20:07:09 1176 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u70@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00185198/0017/00185198_00175749_1.sim?&pss.tid=tlhcb006.33:70@htcjob3715186_0_slot1_26_pid517875.ralworker&oss.lcl=1
Jun 14 02:55:50 lcg2227.gridpp.rl.ac.uk docker[3264646]: 230614 01:55:50 11867 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/66/ec/DAOD_TOPQ1.25521602._000309.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 20:20:12 lcg2231.gridpp.rl.ac.uk docker[3681552]: 230614 19:20:12 4283 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/dd/00/DAOD_TOPQ1.25522310._000025.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 20:04:55 lcg2233.gridpp.rl.ac.uk docker[3837853]: 230614 19:04:55 010 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/03/64/DAOD_TOPQ1.25521556._000013.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 02:40:46 lcg2242.gridpp.rl.ac.uk docker[1274405]: 230614 01:40:46 2380 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/46/0a/DAOD_TOPQ1.25521602._000573.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 00:38:33 lcg2258.gridpp.rl.ac.uk docker[2103584]: 230613 23:38:33 10523 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/51/5a/DAOD_TOPQ1.25523157._000598.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 00:56:41 lcg2258.gridpp.rl.ac.uk docker[2103584]: 230613 23:56:41 10525 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/3e/e1/DAOD_TOPQ1.25523157._000045.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 17:17:03 lcg2259.gridpp.rl.ac.uk docker[585727]: 230614 16:17:02 025 XrdPfc_CInfo: error Read() /atlas:datadisk/rucio/data16_13TeV/4e/6e/DAOD_PHYS.33082380._000033.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 17:17:03 lcg2259.gridpp.rl.ac.uk docker[585727]: 230614 16:17:02 025 XrdPfc_Purge: warning FPurgeState::TraverseNamespace can't open or read /atlas:datadisk/rucio/data16_13TeV/4e/6e/DAOD_PHYS.33082380._000033.pool.root.1.cinfo, err no error; purging.
Jun 14 20:22:00 lcg2259.gridpp.rl.ac.uk docker[585727]: 230614 19:22:00 5330 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/3d/58/DAOD_TOPQ1.25522310._000021.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 09:45:18 lcg2262.gridpp.rl.ac.uk docker[2280622]: Error response from daemon: cannot stop container: xrootd-proxy: tried to kill container, but did not receive an exit event
Jun 14 02:11:40 lcg2264.gridpp.rl.ac.uk docker[681927]: 230614 01:11:40 752 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/a0/ab/DAOD_TOPQ1.25521602._000512.pool.root.1.cinfo Checksum Store mismatch.

Jun 15 03:55:49 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:55:49 023 XrdPfc_File: error ProcessBlockResponse block 0x7fb7e500e830, idx=1120, off=9395240960 error=-42 atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_File: error ProcessBlockResponse block 0x7fb7e0c30060, idx=1120, off=9395240960 error=-42 atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_File: error Read() io 0x7fb7e0c106d0, block 1120 finished with error 42 no message of desired type atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_IO: warning Read() error in File::Read(), exit status=-42, error=no message of desired type root://u50@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1?&pss.tid=tatls015.4684:50@htcjob3721391_0_slot1_45_pid3207759.ralworker
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_File: error ProcessBlockResponse block 0x7fb7e0c30060, idx=1121, off=9403629568 error=-42 atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_File: error Read() io 0x7fb7e0c106d0, block 1121 finished with error 42 no message of desired type atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_IO: warning Read() error in File::Read(), exit status=-42, error=no message of desired type root://u50@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1?&pss.tid=tatls015.4684:50@htcjob3721391_0_slot1_45_pid3207759.ralworker
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_File: error ProcessBlockResponse block 0x7fb7e0c30060, idx=1122, off=9412018176 error=-42 atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_File: error Read() io 0x7fb7e0c106d0, block 1122 finished with error 42 no message of desired type atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_IO: warning Read() error in File::Read(), exit status=-42, error=no message of desired type root://u50@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1?&pss.tid=tatls015.4684:50@htcjob3721391_0_slot1_45_pid3207759.ralworker
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_File: error ProcessBlockResponse block 0x7fb7e0c30060, idx=1123, off=9420406784 error=-42 atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_File: error Read() io 0x7fb7e0c106d0, block 1123 finished with error 42 no message of desired type atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 03:58:19 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:19 100 XrdPfc_IO: warning Read() error in File::Read(), exit status=-42, error=no message of desired type root://u50@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1?&pss.tid=tatls015.4684:50@htcjob3721391_0_slot1_45_pid3207759.ralworker
Jun 15 03:58:21 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 02:58:21 023 XrdPfc_File: error ProcessBlockResponse block 0x7fb7e50efa60, idx=1120, off=9395240960 error=-42 atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 789 XrdPfc_File: error ProcessBlockResponse block 0x7fb7b804c290, idx=1120, off=9395240960 error=-42 atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 789 XrdPfc_File: error Read() io 0x7fb7e0c10430, block 1120 finished with error 42 no message of desired type atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 789 XrdPfc_IO: warning Read() error in File::Read(), exit status=-42, error=no message of desired type root://u50@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1?&pss.tid=tatls015.4684:50@htcjob3721391_0_slot1_45_pid3207759.ralworker
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 11382 XrdPfc_File: error ProcessBlockResponse block 0x7fb7c3d50ab0, idx=1121, off=9403629568 error=-42 atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 11382 XrdPfc_File: error Read() io 0x7fb7e0c10430, block 1121 finished with error 42 no message of desired type atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 11382 XrdPfc_IO: warning Read() error in File::Read(), exit status=-42, error=no message of desired type root://u50@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1?&pss.tid=tatls015.4684:50@htcjob3721391_0_slot1_45_pid3207759.ralworker
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 4632 XrdPfc_File: error ProcessBlockResponse block 0x7fb7ca8d5dd0, idx=1122, off=9412018176 error=-42 atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 4632 XrdPfc_File: error Read() io 0x7fb7e0c10430, block 1122 finished with error 42 no message of desired type atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 4632 XrdPfc_IO: warning Read() error in File::Read(), exit status=-42, error=no message of desired type root://u50@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1?&pss.tid=tatls015.4684:50@htcjob3721391_0_slot1_45_pid3207759.ralworker
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 100 XrdPfc_File: error ProcessBlockResponse block 0x7fb7e0c30240, idx=1123, off=9420406784 error=-42 atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 100 XrdPfc_File: error Read() io 0x7fb7e0c10430, block 1123 finished with error 42 no message of desired type atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1
Jun 15 04:02:31 lcg2214.gridpp.rl.ac.uk docker[458871]: 230615 03:02:31 100 XrdPfc_IO: warning Read() error in File::Read(), exit status=-42, error=no message of desired type root://u50@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/bc/f9/DAOD_RPVLL.22357302._000007.pool.root.1?&pss.tid=tatls015.4684:50@htcjob3721391_0_slot1_45_pid3207759.ralworker
Jun 15 07:51:58 lcg2222.gridpp.rl.ac.uk docker[2973486]: 230615 06:51:58 13856 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/de/ce/DAOD_TOPQ1.25522134._000015.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 09:00:09 lcg2223.gridpp.rl.ac.uk docker[2380847]: 230615 08:00:09 5841 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/bb/98/DAOD_TOPQ1.25524327._000082.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 10:40:35 lcg2227.gridpp.rl.ac.uk docker[2646334]: 230615 09:40:35 149 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/data16_13TeV/4c/dd/DAOD_PHYS.30497998._000043.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 08:28:25 lcg2231.gridpp.rl.ac.uk docker[3681552]: 230615 07:28:25 4258 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/data16_13TeV/96/17/DAOD_PHYS.30498355._000339.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 10:00:17 lcg2234.gridpp.rl.ac.uk docker[274973]: Error response from daemon: cannot stop container: xrootd-proxy: tried to kill container, but did not receive an exit event
Jun 15 06:56:55 lcg2240.gridpp.rl.ac.uk docker[2486007]: 230615 05:56:55 177 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/1e/9e/DAOD_TOPQ1.25522586._000083.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 10:03:52 lcg2242.gridpp.rl.ac.uk docker[3661892]: 230615 09:03:52 025 XrdPfc_CInfo: error Read() /atlas:datadisk/rucio/data17_13TeV/29/fd/DAOD_JETM1.32421862._000015.pool.root.1.cinfo Checksum Synced or AStats mismatch.
Jun 15 10:03:52 lcg2242.gridpp.rl.ac.uk docker[3661892]: 230615 09:03:52 025 XrdPfc_Purge: warning FPurgeState::TraverseNamespace can't open or read /atlas:datadisk/rucio/data17_13TeV/29/fd/DAOD_JETM1.32421862._000015.pool.root.1.cinfo, err no error; purging.
Jun 15 09:26:31 lcg2248.gridpp.rl.ac.uk docker[2340499]: 230615 08:26:31 18454 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/data18_13TeV/a3/89/DAOD_PHYS.30499503._001145.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 08:05:04 lcg2250.gridpp.rl.ac.uk docker[2859743]: 230615 07:05:04 6411 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/88/f1/DAOD_TOPQ1.25523709._000061.pool.root.1.cinfo Checksum Synced or AStats mismatch.
Jun 15 08:42:07 lcg2257.gridpp.rl.ac.uk docker[3835425]: 230615 07:42:07 12572 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/data15_13TeV/a6/7d/DAOD_PHYS.30498026._000116.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 09:38:17 lcg2268.gridpp.rl.ac.uk docker[988358]: Error response from daemon: cannot stop container: xrootd-proxy: tried to kill container, but did not receive an exit event

unpatched - lots of read timeout errors

Gateway

none

Proxy

Jun 15 05:09:06 lcg2368.gridpp.rl.ac.uk docker[2613536]: 230615 04:09:06 025 XrdPfc_CInfo: error Read() /atlas:datadisk/rucio/mc16_13TeV/04/f8/DAOD_JETM1.32139156._000048.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 05:09:06 lcg2368.gridpp.rl.ac.uk docker[2613536]: 230615 04:09:06 025 XrdPfc_Purge: warning FPurgeState::TraverseNamespace can't open or read /atlas:datadisk/rucio/mc16_13TeV/04/f8/DAOD_JETM1.32139156._000048.pool.root.1.cinfo, err no error; purging.

Jun 14 17:16:50 lcg2372.gridpp.rl.ac.uk docker[3451515]: 230614 16:16:50 2568 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc15_13TeV/2a/94/EVNT.26402741._011434.pool.root.1.cinfo Checksum Store mismatch.

only 2 errors, likely from corrupted files

Gateway

Jun 13 10:50:03 lcg2399.gridpp.rl.ac.uk docker[4049939]: 230613 09:50:03 6042 XrootdXeq: u35.7:5322@localhost disc 0:01:43 (link read error)

Jun 15 10:48:03 lcg2391.gridpp.rl.ac.uk docker[2623992]: 230615 09:48:03 6350 XrootdXeq: u65.6:5316@localhost disc 4:25:33 (link read error)
Jun 15 10:58:03 lcg2431.gridpp.rl.ac.uk docker[3548186]: 230615 09:58:03 5938 XrootdXeq: u47.7:5345@localhost disc 0:01:45 (link read error)

Proxy

Jun 13 17:40:40 lcg2393.gridpp.rl.ac.uk docker[3169872]: 230613 16:40:40 011 XrdPfc_File: error ProcessBlockResponse block 0x7f4edf887470, idx=20, off=167772160 error=-110 lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 17:40:40 lcg2393.gridpp.rl.ac.uk docker[3169872]: 230613 16:40:40 7531 XrdPfc_File: error Read() io 0x7f4ee54405b0, block 20 finished with error 110 connection timed out lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 17:40:40 lcg2393.gridpp.rl.ac.uk docker[3169872]: 230613 16:40:40 7531 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u103@127.0.0.1:1095/lhcb:user/lhcb/user/t/tevans/Allen.tar.gz?&pss.tid=tlhcb006.34:103@htcjob3697936_0_slot1_128_pid19448.ralworker&oss.lcl=1
Jun 13 12:14:20 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:20 010 XrdPfc_File: error ProcessBlockResponse block 0x7f495100ef60, idx=604, off=5066719232 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim
Jun 13 12:14:20 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:20 1064 XrdPfc_File: error Read() io 0x7f4933410190, block 604 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim
Jun 13 12:14:20 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:20 1064 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u65@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim?&pss.tid=tlhcb006.3956:65@htcjob3027585_0_slot1_19_pid2419695.ralworker
Jun 13 12:14:21 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:20 372 XrdPfc_File: error ProcessBlockResponse block 0x7f4951032600, idx=605, off=5075107840 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim
Jun 13 12:14:21 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:21 1064 XrdPfc_File: error Read() io 0x7f4933410190, block 605 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim
Jun 13 12:14:21 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:21 1064 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u65@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim?&pss.tid=tlhcb006.3956:65@htcjob3027585_0_slot1_19_pid2419695.ralworker
Jun 13 12:14:21 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:21 031 XrdPfc_File: error ProcessBlockResponse block 0x7f4951032bf0, idx=606, off=5083496448 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim
Jun 13 12:14:21 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:21 1064 XrdPfc_File: error Read() io 0x7f4933410190, block 606 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim
Jun 13 12:14:21 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:21 1064 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u65@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim?&pss.tid=tlhcb006.3956:65@htcjob3027585_0_slot1_19_pid2419695.ralworker
Jun 13 12:14:21 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:21 382 XrdPfc_File: error ProcessBlockResponse block 0x7f4951032d80, idx=607, off=5091885056 error=-110 lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim
Jun 13 12:14:21 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:21 1064 XrdPfc_File: error Read() io 0x7f4933410190, block 607 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim
Jun 13 12:14:21 lcg2423.gridpp.rl.ac.uk docker[1583828]: 230613 11:14:21 1064 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u65@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2017/SIM/00187867/0028/00187867_00280330_1.sim?&pss.tid=tlhcb006.3956:65@htcjob3027585_0_slot1_19_pid2419695.ralworker
Jun 13 16:24:45 lcg2426.gridpp.rl.ac.uk docker[3383439]: 230613 15:24:45 011 XrdPfc_File: error ProcessBlockResponse block 0x7f1c32ce0740, idx=26, off=218103808 error=-110 lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 16:24:45 lcg2426.gridpp.rl.ac.uk docker[3383439]: 230613 15:24:45 101 XrdPfc_File: error Read() io 0x7f1c2e418a90, block 26 finished with error 110 connection timed out lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 16:24:45 lcg2426.gridpp.rl.ac.uk docker[3383439]: 230613 15:24:45 101 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u32@127.0.0.1:1095/lhcb:user/lhcb/user/t/tevans/Allen.tar.gz?&pss.tid=tlhcb006.33:32@htcjob3687683_0_slot1_52_pid3772916.ralworker&oss.lcl=1
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 1141 XrdPfc_File: error ProcessBlockResponse block 0x7fe2ae545e70, idx=697, off=5846859776 error=-110 atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 186 XrdPfc_File: error Read() io 0x7fe29a8b0370, block 697 finished with error 110 connection timed out atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 186 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u40@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1?&pss.tid=patls002.4684:40@htcjob3401454_0_slot1_6_pid607623.ralworker
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 010 XrdPfc_File: error ProcessBlockResponse block 0x7fe29c1f59c0, idx=698, off=5855248384 error=-110 atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 029 XrdPfc_File: error Read() io 0x7fe29a8b0370, block 698 finished with error 110 connection timed out atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 029 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u40@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1?&pss.tid=patls002.4684:40@htcjob3401454_0_slot1_6_pid607623.ralworker
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 2792 XrdPfc_File: error ProcessBlockResponse block 0x7fe2b1c0e100, idx=699, off=5863636992 error=-110 atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 187 XrdPfc_File: error Read() io 0x7fe29a8b0370, block 699 finished with error 110 connection timed out atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 187 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u40@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1?&pss.tid=patls002.4684:40@htcjob3401454_0_slot1_6_pid607623.ralworker
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 186 XrdPfc_File: error ProcessBlockResponse block 0x7fe2b1c0e510, idx=700, off=5872025600 error=-110 atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 029 XrdPfc_File: error Read() io 0x7fe29a8b0370, block 700 finished with error 110 connection timed out atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1
Jun 13 02:23:03 lcg2446.gridpp.rl.ac.uk docker[519919]: 230613 01:23:03 029 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u40@127.0.0.1:1095/atlas:datadisk/rucio/mc16_13TeV/aa/f9/AOD.18035485._004568.pool.root.1?&pss.tid=patls002.4684:40@htcjob3401454_0_slot1_6_pid607623.ralworker
Jun 13 20:57:24 lcg2450.gridpp.rl.ac.uk docker[3106856]: 230613 19:57:24 239 XrootdXeq: ttcms032.296:43@htcjob3019574_0_slot1_7_pid2268606.ralworker disc 0:00:16 (link read error)
Jun 13 16:30:25 lcg2458.gridpp.rl.ac.uk docker[2434101]: 230613 15:30:25 716 XrootdXeq: ttcms032.297:48@htcjob2941747_0_slot1_11_pid2670887.ralworker disc 0:00:20 (link read error)
Jun 13 16:17:32 lcg2469.gridpp.rl.ac.uk docker[1110102]: 230613 15:17:32 118 XrootdXeq: ttcms032.296:44@htcjob3665660_0_slot1_6_pid1017402.ralworker disc 0:00:02 (link read error)
Jun 13 17:45:35 lcg2471.gridpp.rl.ac.uk docker[3678244]: 230613 16:45:35 1684 XrootdXeq: ttcms032.297:49@htcjob2954459_0_slot1_11_pid737460.ralworker disc 0:00:21 (link read error)
Jun 13 20:57:52 lcg2475.gridpp.rl.ac.uk docker[1240046]: 230613 19:57:52 101 XrootdXeq: ttcms032.311:38@htcjob3664445_0_slot1_6_pid3090819.ralworker disc 0:00:35 (link read error)

Jun 14 02:52:00 lcg2391.gridpp.rl.ac.uk docker[387622]: 230614 01:52:00 2050 XrdPfc_CInfo: warning Read() atlas:datadisk/rucio/mc16_13TeV/bb/54/DAOD_TOPQ1.27222063._000009.pool.root.1.cinfo Oss Read failed at off=196 size=4 ret=0 error=<no error>
Jun 14 15:55:09 lcg2408.gridpp.rl.ac.uk docker[1486208]: 230614 14:55:09 024 XrdPfc_CInfo: error Read() /lhcb:prod/lhcb/MC/2015/ALLSTREAMS.DST/00082730/0000/00082730_00000625_6.AllStreams.dst.cinfo Checksum Synced or AStats mismatch.
Jun 14 15:55:09 lcg2408.gridpp.rl.ac.uk docker[1486208]: 230614 14:55:09 024 XrdPfc_Purge: warning FPurgeState::TraverseNamespace can't open or read /lhcb:prod/lhcb/MC/2015/ALLSTREAMS.DST/00082730/0000/00082730_00000625_6.AllStreams.dst.cinfo, err no error; purging.
Jun 14 16:09:56 lcg2419.gridpp.rl.ac.uk docker[2780806]: 230614 15:09:56 4547 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc15_13TeV/df/e7/EVNT.26402741._003752.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 15:23:16 lcg2424.gridpp.rl.ac.uk docker[2851174]: 230614 14:23:16 010 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc15_13TeV/14/9b/EVNT.26402741._008450.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 01:47:14 lcg2428.gridpp.rl.ac.uk docker[1785412]: 230614 00:47:14 8355 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/f1/f7/DAOD_TOPQ1.27222897._000029.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 12:59:12 lcg2438.gridpp.rl.ac.uk docker[3933091]: 230614 11:59:12 527 XrootdXeq: ttcms032.297:49@htcjob2960940_0_slot1_8_pid1837245.ralworker disc 0:00:02 (link read error)
Jun 14 14:43:37 lcg2439.gridpp.rl.ac.uk docker[1218421]: 230614 13:43:37 239 XrootdXeq: ttcms032.297:38@htcjob2954442_0_slot1_13_pid3392209.ralworker disc 0:00:13 (link read error)
Jun 14 23:53:56 lcg2448.gridpp.rl.ac.uk docker[3267198]: 230614 22:53:56 3750 XrootdXeq: ttcms032.297:43@htcjob2950428_0_slot1_10_pid3658779.ralworker disc 0:00:20 (link read error)
Jun 14 18:12:32 lcg2450.gridpp.rl.ac.uk docker[712975]: 230614 17:12:32 369 XrootdXeq: ttcms032.297:43@htcjob3024355_0_slot1_12_pid3886962.ralworker disc 0:00:12 (link read error)
Jun 14 07:42:07 lcg2451.gridpp.rl.ac.uk docker[326669]: 230614 06:42:07 256 XrootdXeq: ttcms032.297:45@htcjob3662681_0_slot1_8_pid3339327.ralworker disc 0:00:20 (link read error)
Jun 14 14:23:58 lcg2451.gridpp.rl.ac.uk docker[525710]: 230614 13:23:58 101 XrootdXeq: ttcms032.311:43@htcjob2942150_0_slot1_1_pid3773400.ralworker disc 0:00:20 (link read error)
Jun 14 22:02:55 lcg2453.gridpp.rl.ac.uk docker[245827]: 230614 21:02:55 3549 XrootdXeq: ttcms032.297:57@htcjob3042584_0_slot1_15_pid2560325.ralworker disc 0:00:21 (link read error)
Jun 14 14:23:19 lcg2460.gridpp.rl.ac.uk docker[3392146]: 230614 13:23:19 032 XrootdXeq: ttcms032.297:41@htcjob3384099_0_slot1_9_pid939401.ralworker disc 0:00:29 (link read error)
Jun 14 21:50:17 lcg2461.gridpp.rl.ac.uk docker[2941468]: 230614 20:50:17 1942 XrootdXeq: ttcms032.297:28@htcjob3041909_0_slot1_16_pid2654460.ralworker disc 0:00:04 (link read error)
Jun 14 16:20:04 lcg2469.gridpp.rl.ac.uk docker[2856678]: 230614 15:20:04 1575 XrootdXeq: ttcms032.304:55@htcjob3665660_0_slot1_6_pid1017402.ralworker disc 0:00:21 (link read error)
Jun 14 14:43:28 lcg2470.gridpp.rl.ac.uk docker[556673]: 230614 13:43:28 794 XrootdXeq: ttcms032.299:43@htcjob2939034_0_slot1_16_pid763066.ralworker disc 0:00:05 (link read error)
Jun 14 15:25:43 lcg2474.gridpp.rl.ac.uk docker[3244662]: 230614 14:25:43 692 XrootdXeq: ttcms032.297:41@htcjob748973_0_slot1_10_pid550312.ralworker disc 0:00:22 (link read error)
Jun 14 09:20:44 lcg2475.gridpp.rl.ac.uk docker[1240046]: 230614 08:20:44 4067 XrootdXeq: ttcms032.297:42@htcjob3677768_0_slot1_12_pid1647995.ralworker disc 0:00:20 (link read error)
Jun 14 15:25:59 lcg2475.gridpp.rl.ac.uk docker[1967782]: 230614 14:25:59 100 XrootdXeq: ttcms032.305:46@htcjob3677768_0_slot1_12_pid1647995.ralworker disc 0:00:20 (link read error)

Jun 15 04:19:24 lcg2389.gridpp.rl.ac.uk docker[372560]: 230615 03:19:24 16630 XrdPfc_File: error ProcessBlockResponse block 0x7f0197c0ea10, idx=27, off=226492416 error=-110 lhcb:prod/lhcb/LHCb/Collision17/DIMUON.DST/00071499/0003/00071499_00030682_1.dimuon.dst
Jun 15 05:03:15 lcg2396.gridpp.rl.ac.uk docker[2555442]: 230615 04:03:15 30217 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/80/cd/DAOD_TOPQ1.25526058._001550.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 04:19:29 lcg2419.gridpp.rl.ac.uk docker[2780806]: 230615 03:19:29 23974 XrdPfc_File: error ProcessBlockResponse block 0x7f9cc8389e20, idx=312, off=2617245696 error=-110 lhcb:prod/lhcb/LHCb/Collision17/DIMUON.DST/00071571/0000/00071571_00003547_1.dimuon.dst
Jun 15 04:19:29 lcg2419.gridpp.rl.ac.uk docker[2780806]: 230615 03:19:29 7420 XrdPfc_File: error ProcessBlockResponse block 0x7f9cc8389ec0, idx=313, off=2625634304 error=-110 lhcb:prod/lhcb/LHCb/Collision17/DIMUON.DST/00071571/0000/00071571_00003547_1.dimuon.dst
Jun 15 04:19:29 lcg2419.gridpp.rl.ac.uk docker[2780806]: 230615 03:19:29 11620 XrdPfc_File: error ProcessBlockResponse block 0x7f9cc8389f60, idx=314, off=2634022912 error=-110 lhcb:prod/lhcb/LHCb/Collision17/DIMUON.DST/00071571/0000/00071571_00003547_1.dimuon.dst
Jun 15 04:19:29 lcg2419.gridpp.rl.ac.uk docker[2780806]: 230615 03:19:29 17238 XrdPfc_File: error ProcessBlockResponse block 0x7f9cc838a380, idx=315, off=2642411520 error=-110 lhcb:prod/lhcb/LHCb/Collision17/DIMUON.DST/00071571/0000/00071571_00003547_1.dimuon.dst
Jun 15 10:27:50 lcg2425.gridpp.rl.ac.uk docker[3619258]: 230615 09:27:50 110 XrdPfc_File: error ProcessBlockResponse block 0x7faa9e962920, idx=0, off=0 error=-110 atlas:datadisk/rucio/data17_13TeV/9b/21/DAOD_JETM1.32822204._000472.pool.root.1
Jun 15 10:27:50 lcg2425.gridpp.rl.ac.uk docker[3619258]: 230615 09:27:50 114 XrdPfc_File: error Read() io 0x7faa9e9ae880, block 0 finished with error 110 connection timed out atlas:datadisk/rucio/data17_13TeV/9b/21/DAOD_JETM1.32822204._000472.pool.root.1
Jun 15 10:27:50 lcg2425.gridpp.rl.ac.uk docker[3619258]: 230615 09:27:50 114 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u67@127.0.0.1:1095/atlas:datadisk/rucio/data17_13TeV/9b/21/DAOD_JETM1.32822204._000472.pool.root.1
Jun 15 10:27:50 lcg2425.gridpp.rl.ac.uk docker[3619258]: 230615 09:27:50 113 XrdPfc_File: error ProcessBlockResponse block 0x7faa69817fb0, idx=1, off=8388608 error=-110 atlas:datadisk/rucio/data17_13TeV/9b/21/DAOD_JETM1.32822204._000472.pool.root.1
Jun 15 10:27:50 lcg2425.gridpp.rl.ac.uk docker[3619258]: 230615 09:27:50 1327 XrdPfc_File: error ProcessBlockResponse block 0x7faa698c72e0, idx=2, off=16777216 error=-110 atlas:datadisk/rucio/data17_13TeV/9b/21/DAOD_JETM1.32822204._000472.pool.root.1
Jun 15 10:27:50 lcg2425.gridpp.rl.ac.uk docker[3619258]: 230615 09:27:50 009 XrdPfc_File: error ProcessBlockResponse block 0x7faa698c7380, idx=3, off=25165824 error=-110 atlas:datadisk/rucio/data17_13TeV/9b/21/DAOD_JETM1.32822204._000472.pool.root.1
Jun 15 00:27:28 lcg2438.gridpp.rl.ac.uk docker[3933091]: 230614 23:27:28 2906 XrootdXeq: ttcms032.298:41@htcjob2960940_0_slot1_8_pid1837245.ralworker disc 0:00:02 (link read error)
Jun 15 01:59:27 lcg2439.gridpp.rl.ac.uk docker[1218421]: 230615 00:59:27 5100 XrootdXeq: ttcms032.297:51@htcjob3041328_0_slot1_6_pid3210648.ralworker disc 0:00:02 (link read error)
Jun 15 08:05:16 lcg2449.gridpp.rl.ac.uk docker[1877198]: 230615 07:05:16 611 XrootdXeq: ttcms032.296:44@htcjob3045566_0_slot1_16_pid1583315.ralworker disc 0:00:05 (link read error)
Jun 15 01:21:31 lcg2471.gridpp.rl.ac.uk docker[1623134]: 230615 00:21:31 596 XrootdXeq: ttcms032.297:43@htcjob2954459_0_slot1_11_pid737460.ralworker disc 0:00:04 (link read error)
Jun 15 02:27:48 lcg2478.gridpp.rl.ac.uk docker[3809362]: 230615 01:27:48 5228 XrootdXeq: ttcms032.297:58@htcjob3705246_0_slot1_16_pid277700.ralworker disc 0:00:11 (link read error)

unpatched - timeout errors

Gateway

none

Proxy

Jun 13 15:45:10 lcg2487.gridpp.rl.ac.uk docker[1366666]: 230613 14:45:10 025 XrdPfc_CInfo: error Read() /lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00373135_1.sim.cinfo Checksum Store mismatch.
Jun 13 15:45:10 lcg2487.gridpp.rl.ac.uk docker[1366666]: 230613 14:45:10 025 XrdPfc_Purge: warning FPurgeState::TraverseNamespace can't open or read /lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00373135_1.sim.cinfo, err no error; purging.
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 17235 XrdPfc_File: error ProcessBlockResponse block 0x7f4dd88de060, idx=440, off=3690987520 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 17592 XrdPfc_File: error Read() io 0x7f4db63ed100, block 440 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 17592 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u55@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim?&pss.tid=tlhcb006.33:55@htcjob3703522_0_slot1_40_pid692204.ralworker&oss.lcl=1
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 7339 XrdPfc_File: error ProcessBlockResponse block 0x7f4dd88de0b0, idx=441, off=3699376128 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 17592 XrdPfc_File: error Read() io 0x7f4db63ed100, block 441 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 17592 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u55@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim?&pss.tid=tlhcb006.33:55@htcjob3703522_0_slot1_40_pid692204.ralworker&oss.lcl=1
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 104 XrdPfc_File: error ProcessBlockResponse block 0x7f4dd88de100, idx=442, off=3707764736 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 17592 XrdPfc_File: error Read() io 0x7f4db63ed100, block 442 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 17592 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u55@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim?&pss.tid=tlhcb006.33:55@htcjob3703522_0_slot1_40_pid692204.ralworker&oss.lcl=1
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 13284 XrdPfc_File: error ProcessBlockResponse block 0x7f4dd88de2e0, idx=443, off=3716153344 error=-110 lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 17592 XrdPfc_File: error Read() io 0x7f4db63ed100, block 443 finished with error 110 connection timed out lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim
Jun 13 20:03:00 lcg2489.gridpp.rl.ac.uk docker[2428229]: 230613 19:03:00 17592 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u55@127.0.0.1:1095/lhcb:buffer/lhcb/MC/2018/SIM/00187861/0037/00187861_00379954_1.sim?&pss.tid=tlhcb006.33:55@htcjob3703522_0_slot1_40_pid692204.ralworker&oss.lcl=1
Jun 13 17:40:40 lcg2494.gridpp.rl.ac.uk docker[2618962]: 230613 16:40:40 9936 XrdPfc_File: error ProcessBlockResponse block 0x7fac81c4d790, idx=6, off=50331648 error=-110 lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 17:40:40 lcg2494.gridpp.rl.ac.uk docker[2618962]: 230613 16:40:40 17430 XrdPfc_File: error Read() io 0x7fac79cb97c0, block 6 finished with error 110 connection timed out lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 17:40:40 lcg2494.gridpp.rl.ac.uk docker[2618962]: 230613 16:40:40 17430 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u46@127.0.0.1:1095/lhcb:user/lhcb/user/t/tevans/Allen.tar.gz?&pss.tid=tlhcb006.209:46@htcjob744331_0_slot1_34_pid2282395.ralworker&oss.lcl=1
Jun 13 12:14:20 lcg2505.gridpp.rl.ac.uk docker[1350974]: 230613 11:14:20 032 XrdPfc_File: error ProcessBlockResponse block 0x7fd42cca4150, idx=667, off=5595201536 error=-110 atlas:datadisk/rucio/data15_13TeV/48/ee/DAOD_RPVLL.22357430._000043.pool.root.1
Jun 13 12:14:20 lcg2505.gridpp.rl.ac.uk docker[1350974]: 230613 11:14:20 011 XrdPfc_File: error Read() io 0x7fd4110465b0, block 667 finished with error 110 connection timed out atlas:datadisk/rucio/data15_13TeV/48/ee/DAOD_RPVLL.22357430._000043.pool.root.1
Jun 13 12:14:20 lcg2505.gridpp.rl.ac.uk docker[1350974]: 230613 11:14:20 011 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u44@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/48/ee/DAOD_RPVLL.22357430._000043.pool.root.1?&pss.tid=tatls015.4684:44@htcjob741803_0_slot1_19_pid1988549.ralworker
Jun 13 12:14:20 lcg2505.gridpp.rl.ac.uk docker[1350974]: 230613 11:14:20 030 XrdPfc_File: error ProcessBlockResponse block 0x7fd42cca4330, idx=668, off=5603590144 error=-110 atlas:datadisk/rucio/data15_13TeV/48/ee/DAOD_RPVLL.22357430._000043.pool.root.1
Jun 13 12:14:20 lcg2505.gridpp.rl.ac.uk docker[1350974]: 230613 11:14:20 011 XrdPfc_File: error Read() io 0x7fd4110465b0, block 668 finished with error 110 connection timed out atlas:datadisk/rucio/data15_13TeV/48/ee/DAOD_RPVLL.22357430._000043.pool.root.1
Jun 13 12:14:20 lcg2505.gridpp.rl.ac.uk docker[1350974]: 230613 11:14:20 011 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u44@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/48/ee/DAOD_RPVLL.22357430._000043.pool.root.1?&pss.tid=tatls015.4684:44@htcjob741803_0_slot1_19_pid1988549.ralworker
Jun 13 12:14:20 lcg2505.gridpp.rl.ac.uk docker[1350974]: 230613 11:14:20 141 XrdPfc_File: error ProcessBlockResponse block 0x7fd42cca45b0, idx=669, off=5611978752 error=-110 atlas:datadisk/rucio/data15_13TeV/48/ee/DAOD_RPVLL.22357430._000043.pool.root.1
Jun 13 12:14:20 lcg2505.gridpp.rl.ac.uk docker[1350974]: 230613 11:14:20 011 XrdPfc_File: error Read() io 0x7fd4110465b0, block 669 finished with error 110 connection timed out atlas:datadisk/rucio/data15_13TeV/48/ee/DAOD_RPVLL.22357430._000043.pool.root.1
Jun 13 12:14:20 lcg2505.gridpp.rl.ac.uk docker[1350974]: 230613 11:14:20 011 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u44@127.0.0.1:1095/atlas:datadisk/rucio/data15_13TeV/48/ee/DAOD_RPVLL.22357430._000043.pool.root.1?&pss.tid=tatls015.4684:44@htcjob741803_0_slot1_19_pid1988549.ralworker
Jun 13 22:53:31 lcg2505.gridpp.rl.ac.uk docker[1350974]: 230613 21:53:31 17701 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/0f/9a/DAOD_TOPQ1.31482170._000022.pool.root.1.cinfo Checksum Store mismatch.
Jun 13 15:12:39 lcg2514.gridpp.rl.ac.uk docker[1177872]: 230613 14:12:39 157 XrdPfc_CInfo: warning Read() atlas:datadisk/rucio/mc16_13TeV/74/9a/DAOD_TAUP5.33605140._000007.pool.root.1.cinfo Oss Read failed at off=57 size=168 ret=116 error=<no error>
Jun 13 16:27:15 lcg2520.gridpp.rl.ac.uk docker[3555648]: 230613 15:27:15 5647 XrdPfc_File: error ProcessBlockResponse block 0x7f2b4e232100, idx=25, off=209715200 error=-110 lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 16:27:15 lcg2520.gridpp.rl.ac.uk docker[3555648]: 230613 15:27:15 382 XrdPfc_File: error Read() io 0x7f2b6a415850, block 25 finished with error 110 connection timed out lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 16:27:15 lcg2520.gridpp.rl.ac.uk docker[3555648]: 230613 15:27:15 382 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u73@127.0.0.1:1095/lhcb:user/lhcb/user/t/tevans/Allen.tar.gz?&pss.tid=tlhcb006.3180:73@htcjob3403685_0_slot1_95_pid677988.ralworker&oss.lcl=1
Jun 13 16:30:32 lcg2531.gridpp.rl.ac.uk docker[1626649]: 230613 15:30:32 9795 XrdPfc_File: error ProcessBlockResponse block 0x7efc63d28420, idx=26, off=218103808 error=-110 lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 16:30:32 lcg2531.gridpp.rl.ac.uk docker[1626649]: 230613 15:30:32 10999 XrdPfc_File: error Read() io 0x7efc3e4c4a30, block 26 finished with error 110 connection timed out lhcb:user/lhcb/user/t/tevans/Allen.tar.gz
Jun 13 16:30:32 lcg2531.gridpp.rl.ac.uk docker[1626649]: 230613 15:30:32 10999 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u66@127.0.0.1:1095/lhcb:user/lhcb/user/t/tevans/Allen.tar.gz?&pss.tid=tlhcb006.772:66@htcjob3034977_0_slot1_28_pid3922008.ralworker&oss.lcl=1
Jun 13 06:46:55 lcg2536.gridpp.rl.ac.uk docker[3748565]: 230613 05:46:55 5316 XrdPfc_File: error ProcessBlockResponse block 0x7f749100e5b0, idx=320, off=2684354560 error=-110 atlas:datadisk/rucio/mc20_13TeV/ff/27/AOD.27231629._000101.pool.root.1
Jun 13 06:46:55 lcg2536.gridpp.rl.ac.uk docker[3748565]: 230613 05:46:55 030 XrdPfc_File: error Read() io 0x7f748f80e7f0, block 320 finished with error 110 connection timed out atlas:datadisk/rucio/mc20_13TeV/ff/27/AOD.27231629._000101.pool.root.1
Jun 13 06:46:55 lcg2536.gridpp.rl.ac.uk docker[3748565]: 230613 05:46:55 030 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u48@127.0.0.1:1095/atlas:datadisk/rucio/mc20_13TeV/ff/27/AOD.27231629._000101.pool.root.1?&pss.tid=tatls015.4683:48@htcjob3405478_0_slot1_1_pid1846878.ralworker
Jun 13 06:46:55 lcg2536.gridpp.rl.ac.uk docker[3748565]: 230613 05:46:55 25425 XrdPfc_File: error ProcessBlockResponse block 0x7f749100e650, idx=321, off=2692743168 error=-110 atlas:datadisk/rucio/mc20_13TeV/ff/27/AOD.27231629._000101.pool.root.1
Jun 13 06:46:55 lcg2536.gridpp.rl.ac.uk docker[3748565]: 230613 05:46:55 1798 XrdPfc_File: error Read() io 0x7f748f80e7f0, block 321 finished with error 110 connection timed out atlas:datadisk/rucio/mc20_13TeV/ff/27/AOD.27231629._000101.pool.root.1
Jun 13 06:46:55 lcg2536.gridpp.rl.ac.uk docker[3748565]: 230613 05:46:55 1798 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u48@127.0.0.1:1095/atlas:datadisk/rucio/mc20_13TeV/ff/27/AOD.27231629._000101.pool.root.1?&pss.tid=tatls015.4683:48@htcjob3405478_0_slot1_1_pid1846878.ralworker
Jun 13 06:46:55 lcg2536.gridpp.rl.ac.uk docker[3748565]: 230613 05:46:55 16558 XrdPfc_File: error ProcessBlockResponse block 0x7f749100e7e0, idx=322, off=2701131776 error=-110 atlas:datadisk/rucio/mc20_13TeV/ff/27/AOD.27231629._000101.pool.root.1
Jun 13 06:46:55 lcg2536.gridpp.rl.ac.uk docker[3748565]: 230613 05:46:55 1798 XrdPfc_File: error Read() io 0x7f748f80e7f0, block 322 finished with error 110 connection timed out atlas:datadisk/rucio/mc20_13TeV/ff/27/AOD.27231629._000101.pool.root.1
Jun 13 06:46:55 lcg2536.gridpp.rl.ac.uk docker[3748565]: 230613 05:46:55 1798 XrdPfc_IO: warning Read() error in File::Read(), exit status=-110, error=connection timed out root://u48@127.0.0.1:1095/atlas:datadisk/rucio/mc20_13TeV/ff/27/AOD.27231629._000101.pool.root.1?&pss.tid=tatls015.4683:48@htcjob3405478_0_slot1_1_pid1846878.ralworker
Jun 13 16:51:58 lcg2542.gridpp.rl.ac.uk docker[513699]: 230613 15:51:58 292 XrootdXeq: ttcms032.297:26@htcjob3011703_0_slot1_9_pid333113.ralworker disc 0:00:21 (link read error)
Jun 13 16:01:15 lcg2548.gridpp.rl.ac.uk docker[2558012]: 230613 15:01:15 221 XrootdXeq: ttcms032.297:48@htcjob2943148_0_slot1_18_pid501076.ralworker disc 0:00:15 (link read error)
Jun 13 19:41:59 lcg2553.gridpp.rl.ac.uk docker[3111883]: 230613 18:41:59 677 XrootdXeq: ttcms032.299:43@htcjob3677759_0_slot1_9_pid3406423.ralworker disc 0:00:21 (link read error)
Jun 13 03:35:40 lcg2555.gridpp.rl.ac.uk docker[3836129]: 230613 02:35:40 1903 XrdPfc_File: error ProcessBlockResponse block 0x7f2a4240e7e0, idx=1395, off=11702108160 error=-110 cms:/store/mc/Run3Summer21PrePremix/Neutrino_E-10_gun/PREMIX/Summer22_124X_mcRun3_2022_realistic_v11-v2/40011/807c15b4-4705-43bc-975d-e9fc638a79c6.root
Jun 13 03:02:28 lcg2556.gridpp.rl.ac.uk docker[2805874]: 230613 02:02:28 2689 XrootdXeq: ttcms032.298:28@htcjob3384215_0_slot1_3_pid2095229.ralworker disc 0:00:04 (link read error)
Jun 13 17:03:41 lcg2556.gridpp.rl.ac.uk docker[530599]: 230613 16:03:41 1423 XrootdXeq: ttcms032.297:28@htcjob3677989_0_slot1_2_pid3612397.ralworker disc 0:00:21 (link read error)
Jun 13 17:04:02 lcg2556.gridpp.rl.ac.uk docker[530599]: 230613 16:04:02 1422 XrootdXeq: ttcms032.304:47@htcjob3677989_0_slot1_2_pid3612397.ralworker disc 0:00:21 (link read error)
Jun 13 10:31:50 lcg2559.gridpp.rl.ac.uk docker[1371305]: 230613 09:31:50 011 XrootdXeq: ttcms032.297:31@htcjob3390935_0_slot1_5_pid111107.ralworker disc 0:00:11 (link read error)
Jun 13 10:49:01 lcg2579.gridpp.rl.ac.uk docker[3024745]: 230613 09:49:01 117 XrootdXeq: ttcms032.337:40@htcjob3674599_0_slot1_11_pid3337317.ralworker disc 0:00:11 (link read error)
Jun 13 20:06:56 lcg2593.gridpp.rl.ac.uk docker[963626]: 230613 19:06:56 029 XrootdXeq: ttcms032.297:28@htcjob2952195_0_slot1_3_pid4032605.ralworker disc 0:00:13 (link read error)

Jun 14 13:45:19 lcg2484.gridpp.rl.ac.uk docker[1990329]: 230614 12:45:19 011 XrdPfc_File: error ProcessBlockResponse block 0x7f19ad091880, idx=1, off=8388608 error=-110 atlas:datadisk/rucio/mc20_13TeV/25/40/DAOD_JETM1.32904303._000141.pool.root.1
Jun 14 13:45:19 lcg2484.gridpp.rl.ac.uk docker[1990329]: 230614 12:45:19 3413 XrdPfc_File: error ProcessBlockResponse block 0x7f19ad0918d0, idx=2, off=16777216 error=-110 atlas:datadisk/rucio/mc20_13TeV/25/40/DAOD_JETM1.32904303._000141.pool.root.1
Jun 14 13:45:19 lcg2484.gridpp.rl.ac.uk docker[1990329]: 230614 12:45:19 031 XrdPfc_File: error ProcessBlockResponse block 0x7f19ad091a10, idx=3, off=25165824 error=-110 atlas:datadisk/rucio/mc20_13TeV/25/40/DAOD_JETM1.32904303._000141.pool.root.1
Jun 14 00:50:05 lcg2494.gridpp.rl.ac.uk docker[2618962]: 230613 23:50:05 27608 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/e1/35/DAOD_TOPQ1.27222598._000047.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 16:16:36 lcg2500.gridpp.rl.ac.uk docker[2599277]: 230614 15:16:36 3632 XrootdXeq: tlhcb006.2620:67@htcjob3696452_0_slot1_86_pid247751.ralworker disc 0:58:47 (link read error)
Jun 14 17:10:53 lcg2502.gridpp.rl.ac.uk docker[2498796]: 230614 16:10:53 123 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc15_13TeV/95/ac/EVNT.26402741._007737.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 17:25:30 lcg2502.gridpp.rl.ac.uk docker[2498796]: 230614 16:25:30 10054 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc15_13TeV/d3/3a/EVNT.26402741._007767.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 16:39:47 lcg2508.gridpp.rl.ac.uk docker[2837806]: 230614 15:39:47 123 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc15_13TeV/99/47/EVNT.26402741._000921.pool.root.1.cinfo Checksum Synced or AStats mismatch.
Jun 14 02:46:40 lcg2519.gridpp.rl.ac.uk docker[1235219]: 230614 01:46:40 16808 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/91/4a/DAOD_TOPQ1.25521602._000197.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 15:33:34 lcg2521.gridpp.rl.ac.uk docker[2543550]: 230614 14:33:34 904 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc15_13TeV/5e/fe/EVNT.26402741._006606.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 15:55:34 lcg2535.gridpp.rl.ac.uk docker[404892]: 230614 14:55:34 611 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc15_13TeV/40/25/EVNT.26402741._003962.pool.root.1.cinfo Checksum Store mismatch.
Jun 14 14:43:48 lcg2548.gridpp.rl.ac.uk docker[1588383]: 230614 13:43:48 032 XrootdXeq: ttcms032.304:40@htcjob2943148_0_slot1_18_pid501076.ralworker disc 0:00:08 (link read error)
Jun 14 15:08:26 lcg2548.gridpp.rl.ac.uk docker[1588383]: 230614 14:08:26 031 XrootdXeq: ttcms032.297:26@htcjob2943148_0_slot1_18_pid501076.ralworker disc 0:00:03 (link read error)
Jun 14 06:07:26 lcg2553.gridpp.rl.ac.uk docker[3111883]: 230614 05:07:26 011 XrootdXeq: ttcms032.300:44@htcjob3037903_0_slot1_2_pid1017422.ralworker disc 0:00:06 (link read error)
Jun 14 14:44:10 lcg2555.gridpp.rl.ac.uk docker[2963981]: 230614 13:44:10 032 XrootdXeq: ttcms032.311:49@htcjob3665357_0_slot1_2_pid1650275.ralworker disc 0:00:06 (link read error)
Jun 14 15:09:04 lcg2555.gridpp.rl.ac.uk docker[2963981]: 230614 14:09:04 254 XrootdXeq: ttcms032.298:28@htcjob3665357_0_slot1_2_pid1650275.ralworker disc 0:00:13 (link read error)
Jun 14 14:23:47 lcg2576.gridpp.rl.ac.uk docker[957617]: 230614 13:23:47 445 XrootdXeq: ttcms032.310:40@htcjob3391274_0_slot1_5_pid3201325.ralworker disc 0:00:16 (link read error)
Jun 14 16:20:30 lcg2576.gridpp.rl.ac.uk docker[957617]: 230614 15:20:30 029 XrootdXeq: ttcms032.310:44@htcjob3391274_0_slot1_5_pid3201325.ralworker disc 0:00:12 (link read error)
Jun 14 14:23:09 lcg2579.gridpp.rl.ac.uk docker[3897240]: 230614 13:23:09 426 XrootdXeq: ttcms032.297:26@htcjob3021867_0_slot1_17_pid2824586.ralworker disc 0:00:21 (link read error)
Jun 14 20:22:15 lcg2582.gridpp.rl.ac.uk docker[2744168]: 230614 19:22:15 3347 XrootdXeq: ttcms032.336:65@htcjob3713843_0_slot1_16_pid2379190.ralworker disc 0:00:51 (link read error)
Jun 14 14:43:40 lcg2583.gridpp.rl.ac.uk docker[2182683]: 230614 13:43:40 010 XrootdXeq: ttcms032.299:40@htcjob3024326_0_slot1_14_pid937184.ralworker disc 0:00:22 (link read error)
Jun 14 18:24:34 lcg2588.gridpp.rl.ac.uk docker[3767612]: 230614 17:24:34 010 XrootdXeq: ttcms032.297:43@htcjob3022283_0_slot1_8_pid1229561.ralworker disc 0:00:09 (link read error)
Jun 14 17:44:31 lcg2592.gridpp.rl.ac.uk docker[2871981]: 230614 16:44:31 119 XrootdXeq: ttcms032.297:44@htcjob3667727_0_slot1_5_pid3069494.ralworker disc 0:00:10 (link read error)

Jun 15 10:27:46 lcg2485.gridpp.rl.ac.uk docker[1251348]: 230615 09:27:46 127 XrdPfc_File: error ProcessBlockResponse block 0x7f602040e650, idx=78, off=654311424 error=-110 atlas:datadisk/rucio/data18_13TeV/f1/77/DAOD_STDM13.33523620._000031.pool.root.1
Jun 15 06:00:19 lcg2498.gridpp.rl.ac.uk docker[821729]: 230615 05:00:19 20944 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/96/1b/DAOD_TOPQ1.25524579._000049.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 05:20:31 lcg2517.gridpp.rl.ac.uk docker[1655524]: 230615 04:20:31 9126 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/1d/ac/DAOD_TOPQ1.25523893._000022.pool.root.1.cinfo Checksum Store mismatch.
Jun 15 07:52:34 lcg2517.gridpp.rl.ac.uk docker[1655524]: 230615 06:52:34 25169 XrdPfc_CInfo: error Read() atlas:datadisk/rucio/mc16_13TeV/7b/77/DAOD_TOPQ1.25524515._000001.pool.root.1.cinfo Checksum Synced or AStats mismatch.
Jun 15 09:36:46 lcg2523.gridpp.rl.ac.uk docker[1169358]: 230615 08:36:46 109 XrdPfc_File: error ProcessBlockResponse block 0x7f06f1553d30, idx=192, off=1610612736 error=-110 atlas:datadisk/rucio/mc16_13TeV/47/50/DAOD_TOPQ1.25522576._000011.pool.root.1
Jun 15 05:56:44 lcg2551.gridpp.rl.ac.uk docker[69869]: 230615 04:56:44 3616 XrootdXeq: ttcms032.297:28@htcjob2949963_0_slot1_16_pid1766569.ralworker disc 0:00:20 (link read error)
Jun 15 09:06:27 lcg2553.gridpp.rl.ac.uk docker[2213419]: 230615 08:06:27 5280 XrootdXeq: ttcms032.311:26@htcjob3034114_0_slot1_5_pid3424234.ralworker disc 0:00:14 (link read error)
Jun 15 07:05:02 lcg2554.gridpp.rl.ac.uk docker[2160741]: 230615 06:05:02 6862 XrootdXeq: ttcms032.297:26@htcjob3676105_0_slot1_13_pid1108897.ralworker disc 0:00:09 (link read error)
Jun 15 00:43:44 lcg2556.gridpp.rl.ac.uk docker[2020667]: 230614 23:43:44 2329 XrootdXeq: ttcms032.299:48@htcjob3030986_0_slot1_7_pid1430866.ralworker disc 0:00:22 (link read error)
Jun 15 03:30:21 lcg2557.gridpp.rl.ac.uk docker[911539]: 230615 02:30:21 4728 XrootdXeq: ttcms032.297:45@htcjob3408451_0_slot1_17_pid1896865.ralworker disc 0:00:16 (link read error)
Jun 15 04:03:45 lcg2559.gridpp.rl.ac.uk docker[3650729]: 230615 03:03:45 6217 XrootdXeq: ttcms032.297:44@htcjob2960695_0_slot1_10_pid2451851.ralworker disc 0:00:06 (link read error)
Jun 15 09:24:30 lcg2559.gridpp.rl.ac.uk docker[3650729]: 230615 08:24:30 6690 XrootdXeq: ttcms032.298:45@htcjob3408708_0_slot1_3_pid574373.ralworker disc 0:00:04 (link read error)
Jun 15 00:26:55 lcg2563.gridpp.rl.ac.uk docker[1016418]: 230614 23:26:55 2637 XrootdXeq: ttcms032.297:45@htcjob3399280_0_slot1_8_pid589118.ralworker disc 0:00:11 (link read error)
Jun 15 08:38:54 lcg2565.gridpp.rl.ac.uk docker[1868316]: 230615 07:38:54 4788 XrootdXeq: ttcms032.298:44@htcjob3402075_0_slot1_5_pid188658.ralworker disc 0:00:10 (link read error)

unpatched - timeouts on proxy

memory allocation errors here too - along with a disconnection from cluster on the 15th around 1136. cluster disconnect matches a network change that was rolled back