TPC (Third Party Copy) using xrootd

Progress so far in testing this service:

• Set up test gateway (ceph-test-gw683.gridpp.rl.ac.uk)

• Added recent/latest version of xrootd ( currently v 4.8.6) with rpms made by Andy Hanushevsky. These were needed to get around the 'leading slash' issue with echo.

• Following instructions from : https://twiki.cern.ch/twiki/bin/view/LCG/XrootdTpc
  Installing the vomsxrd package was not easy and Adrian couldn't help. We did a manual install of this rpm instead.

• Edited /etc/xrootd/xrootd-proxy.cfg :

  # Configure TPC ofs.tpc oids autorm pgm /etc/xrootd/xrdcp-tpc.sh xrootd.chksum adler32 /etc/xrootd/xrdadler32-tpc.sh

• For testing, added: xrd.trace debug

• Can also remove old Castor stuff:

  # Redirection to CASTOR # xrootd.redirect ccmsdlf.ads.rl.ac.uk:1094 ? /store # The following line is for lhcb testing and should not be uncommented in prod (until we are doing lhcb in prod) #xrootd.redirect clhcbdlf.ads.rl.ac.uk:1094 ? /castor/ads.rl.ac.uk/prod/lhcb

• I then do 'systemctl restart xrootd@proxy' and 'systemctl status xrootd@proxy'

• To test, need xrootd client 4.8.4 or above. Can use lxplus, which has this version, or use root@ceph-gw5.gridpp.rl.ac.uk as client:

  # Don't need this on lxplus. ceph-gw5 doesn't have voms-proxy-init, so copy grid proxy and set to use it like this: export X509_USER_PROXY=/tmp/x509up_u24287

Previously Katy tried the following, but we now don't think this is necessary:

• Specific configuration used (in file /etc/xrootd/xrootd-clustered.cfg):
  This is what I am currently trying:
  
  

• If I then do 'systemctl start xrootd@clustered' and 'systemctl status xrootd@clustered'...

• I see the service is : active (running) - good!

Katy, 13/12/18:

Got the service running with the new rpms with the 'leading slash' issue fixed. Using LXPLUS to issue commands to transfer over our test gateway.

Succesful:

[kellis@lxplus021]~% xrdcp root://ceph-test-gw683.gridpp.rl.ac.uk:1094/dteam:test1/testKaty1 root://ceph-test-gw683.gridpp.rl.ac.uk:1094/dteam:test1/testTim023

[140B/140B][100%][==================================================][140B/s]

Not successful (using tpc):

[kellis@lxplus021]~% xrdcp --tpc only root://ceph-test-gw683.gridpp.rl.ac.uk:1094/dteam:test1/testKaty1 root://ceph-test-gw683.gridpp.rl.ac.uk:1094/dteam:test1/testTim022

[0B/140B][ 0%][> ][0B/s]

Run: [ERROR] Server responded with an error: [3005] [FATAL] Auth failed

Tim + Katy 08/04/2019

Ian changed the authdb and the blargh to fix the // problem.

We updated from 4.9.0-0.rc3 to 4.9.1-0.1.rc1 with:

Now it accepts //, but still no TPC

Tim + Katy 26/04/2019

Changed in /etc/xrootd/xrootd-proxy.cfg:

pss.origin ceph-test-gw683.gridpp.rl.ac.uk:1095

to

pss.origin ceph-test-gw683.gridpp.rl.ac.uk:1094

Tim + Katy 08/05/2019

Wei's email from 26/04 provided a new config file, /etc/xrootd/xrootd-proxy.cfg (original in xrootd-proxy.cfg.bak1). We also had to install:

yum install http://linuxsoft.cern.ch/wlcg/centos7/x86_64/vomsxrd-0.3.0-1.el7.cern.x86_64.rpm

Now it works!!!!

xrdcp --tpc delegate only root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test1/testKaty1 root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test1/testTim7e
[140B/140B][100%][==================================================][140B/s]

(Also worked without // in path.) Did not work without "delegate":

xrdcp --tpc only root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test1/testKaty1 root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test1/testTim7f
[0B/140B][ 0%][> ][0B/s]
Run: [ERROR] Server responded with an error: [3005] Copy failed with return code 52

Can copy to/from SLAC:

xrdcp --tpc delegate only -f root://ceph-test-gw683.gridpp.rl.ac.uk:1094/dteam:test1/testKaty1 root://griddev03.slac.stanford.edu:2094//xrootd/atlas/tpctest/testRAL3a
[140B/140B][100%][==================================================][35B/s]
xrdcp --tpc delegate only root://griddev03.slac.stanford.edu:2094//xrootd/atlas/tpctest/testRAL3a root://ceph-test-gw683.gridpp.rl.ac.uk:1094/dteam:test1/testSLAC3a
[140B/140B][100%][==================================================][35B/s]

but only out of Brunel DPM:

xrdcp -f --tpc delegate only root://ceph-test-gw683.gridpp.rl.ac.uk:1094/dteam:test1/testKaty1 root://dc2-grid-64.brunel.ac.uk:1094//dpm/brunel.ac.uk/home/dteam/wlcg-tpc/testRAL4b
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Server responded with an error: [3011] Unable to create /dpm/brunel.ac.uk/home/dteam/wlcg-tpc/testRAL4b; Server error: No such file or directory
xrdcp --tpc delegate only root://dc2-grid-64.brunel.ac.uk:1094//dpm/brunel.ac.uk/home/dteam/wlcg-tpc/testRAL4a root://ceph-test-gw683.gridpp.rl.ac.uk:1094/dteam:test1/testTim5r
[140B/140B][100%][==================================================][70B/s]

Tim + Katy 22/05/2019

Updated to XRootD 4.9.1 (release version):

Test from/to Brunel (from lxplus) still OK. The heatmap shows all OK (that should work), except RAL->Glasgow. Tried:

lxplus738> xrdcp --tpc delegate only root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test1/testKaty1 root://svr018.gla.scotgrid.ac.uk:1094//dpm/gla.scotgrid.ac.uk/home/dteam/testRAL1
190522 16:53:06 21979 secgsi_ClientDoCert: no signed DH parameters from svr018.gla.scotgrid.ac.uk. Will not delegate x509 proxy to it
190522 16:53:08 21979 secgsi_ClientDoCert: no signed DH parameters from disk034.gla.scotgrid.ac.uk. Will not delegate x509 proxy to it
[0B/140B][ 0%][> ][0B/s]
Run: [ERROR] Server responded with an error: [3005] [FATAL] Auth failed

which created an empty file, as did:

lxplus738> xrdcp --tpc only root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test1/testKaty1 root://svr018.gla.scotgrid.ac.uk:1094//dpm/gla.scotgrid.ac.uk/home/dteam/testRAL2
[0B/140B][ 0%][> ][0B/s]
Run: [ERROR] Server responded with an error: [3005] [FATAL] Auth failed

without TPC, it works:

lxplus738> xrdcp root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test1/testKaty1 root://svr018.gla.scotgrid.ac.uk:1094//dpm/gla.scotgrid.ac.uk/home/dteam/testRAL3
[140B/140B][100%][==================================================][70B/s]