CMSD configuration for Echo external gateways

Owned by James Walder
Last updated: May 21, 2024 by Thomas, Jyothish (STFC,RAL,SC)
3 min read

Current Sandbox:

http://aquilon.gridpp.rl.ac.uk/sandboxes/diff.php?sandbox=jw-gateway-xrootd-cmsd

Fabric requirements

named:
echo-internal-manager01.gridpp.rl.ac.uk
echo internal-manager02.gridpp.rl.ac.uk

with associated x509 certificates with the following SANs:
*.echo.stfc.ac.uk,
xrootd.echo.stfc.ac.uk, 
webdav.echo.stfc.ac.uk
internal.echo.stfc.ac.uk

with external firewall holes for port 1094 (xrootd traffic)

they should should be able to contact echo gateways on port 1094,1095 and 1213

with the following specs
4 CPUs
8GB RAM
60GB disk

with IP addresses changed so that they are in the OPN subnet
Ideally they should be in the lower part of 130.246.176.0/24 https://netbox.esc.rl.ac.uk/ipam/prefixes/323/ip-addresses/  (James A's words.) (v4 and v6)

with AAAA DNS records added once set

Operational items

Know issues / limitations

N/A

Manager hosts

Frontend:

https://rdr.echo.stfc.ac.uk:1094 root://rdr.echo.stfc.ac.uk:1094 echo-manager01.gridpp.rl.ac.uk echo-manager02.gridpp.rl.ac.uk

Restarting services

systemctl restart xrootd@{unified,tpc} systemctl restart cmsd@unified

Blacklisting of server (gateway) hosts

On each of the manager hosts the following file should be used, and the relevant gateway host included:

/etc/xrootd/cms.blacklist

add the given host on a single line (wildcards are in principle also ok).
This file is re-read on a per-minute basis, and requires no restart of services

if a host in the blacklist does not exist, the blacklist will fail to parse and will be ignored after a service restart

ensure the xrootd:xrootd ownership is set for it

 

Adding a new Server (Gateway host) to the cluster

When a new Gateway needs to be added to a cluster, the following steps (in addition to the usual set of checks for ensuring a fully functional gateway) are required.

  • Ensure the host has the correct personality (i.e. ceph-unified-gw-echo)

  • In Aquilon the manager hosts must be recompiled, in order to find the new host, and update the managers that the new host is available.

    • As we have a pair of managers, it is preferable to (using keepavlived) remove one manager, compile it, check it restarts services correctly, and (using keepalived) add it back.
      (this may require some quattor commands on the host to force the compilation to be deployed immediately).

    • Then, repeat this step for the second manager.

  • Finally, check the cms.blacklist blocklist files on each manager to ensure that the new Server (aka Gateway) is not explicitly excluded from the cluster here.

Development items

Services

A new service has been created to hold the list of manager hosts for each ceph instance (e.g. echo)

xrootd-clustered

For Echo, the specific instance of this service is called xrootd-clustered-echo

These are added with

A host may need to be reconfigured in order to get the new service included in it, and a couple might fail unless this is done; e.g.

 

Xrootd and CMSD configuration

The configuration for xrootd and csmd is stored in the xrootd-unified.cfg configuration file (and the additional xrootd-tpc.cfg - for root TPC transfers).

 

keepalived

The keepalived configuration for the manager CMSD hosts is here:

A summary of the main files:

  • vrrp-instance : the label of the script xrootd that will determine the state of the hosts

  • global: the content of the check script xrootd and notification email

  • config.pan: the floating IP addresses, and specifics of the failover priorities

 

 

Manager cluster setup

 

New cluster

 

Fabric


Could you please create 2 new rocky8 VMware hosts which should act similar roles as echo-manager01.gridpp.rl.ac.uk,

named:
echo-alice-manager01.gridpp.rl.ac.uk
echo-alice-manager02.gridpp.rl.ac.uk

with associated x509 certificates with the following SANs:
echo.stfc.ac.uk
alice.echo.stfc.ac.uk
*.echo.stfc.ac.uk 
*.s3.echo.stfc.ac.uk

with external firewall holes for port 1094 (xrootd traffic)

they should should be able to contact echo gateways on port 1094,1095 and 1213

with the following specs
4 CPUs
8GB RAM
60GB disk

with IP addresses changed so that they are in the OPN subnet
Ideally they should be in the lower part of 130.246.176.0/24 https://netbox.esc.rl.ac.uk/ipam/prefixes/323/ip-addresses/  (James A's words.) (v4 and v6)

with AAAA DNS records added once set,

along with a pair of floating IPs (like 130.246.176.2 and 130.246.176.3 and the associated v6 2001:630:58:1820::82f6:b002 and 2001:630:58:1820::82f6:b003) to be assigned to keepalived for load balancing

 

 

Aquilon

aq add service --service xrootd-clustered --instance xrootd-clustered-echo-internal
aq bind_server --service xrootd-clustered --instance xrootd-clustered-echo-internal --hostname echo-internal-manager01.gridpp.rl.ac.uk

aq map_service --service xrootd-clustered --instance xrootd-clustered-echo-internal --archetype ral-tier1 --personality ceph-gw-echo-internal --campus Harwell --justification tcm=000

copy /shared/service/xrootd-clustered/xrootd-clustered-echo into /shared/service/xrootd-clustered/xrootd-clustered-echo-internal and replace naming in configs appropriately

copy ral-tier1/features/keepalived/echo-managers to ral-tier1/features/keepalived/echo-managers-internal
in ral-tier1/features/keepalived/echo-managers-internal/config.pan, replace the ip addresses with the new floating ips and replace vrid[N] with a different number (not included in other keepalived configs)