Compiling Alternate Version of RADOS Striper

This requirement arises out of the development work to improve XRootD 's read access to ECHO by providing lock-less reads.

We are broadly following the direction suggested by Andreas-Joachim Peters by investigating removing read locks, at least

for the case of Vector Reads.

Compiling enough of Ceph to build the RADOS Striper
- Instructions for Ubuntu
- Instructions for Scientific Linux

Instructions for Ubuntu

I created a VM in SCD OpenStack from the image ubuntu-focal-20.04-nogui. I specified a c3.large instance (8 CPUs, 16GB RAM, 80GB disk).

In the following instructions, commands run as an ordinary user are prefixed with '$', and commands which need root are prefixed with '#'.

Before starting run an update:

# apt-get update

I cloned the Ceph source from Git:

# apt-get install -y cmake git
$ git clone https://github.com/ceph/ceph.git

I then checked out the revision we are currently using (as of March 2022, this is v14.2.15):

$ cd ceph && git checkout v14.2.15

Next, I installed the suggested dependencies:

# ./install-deps.sh

The install-deps.sh script didn't quite install everything needed...

The packages and options I supplied to satisfy the requirements were:

# apt-get install -y -o APT::Install-Recommends=true python3-pip python3-virtualenv doxygen ditaa libxml2-dev libxslt1-dev graphviz ant cython3
# apt --fix-broken install

# apt-get install -y libsystemd-dev libudev-dev libblkid-dev libkeyutils-dev libaio-dev
# apt-get install -y libnl-genl-3-dev libleveldb-dev pkg-config libsnappy-dev liblz4-dev libgoogle-perftools-dev
# apt-get install -y libcurl4-openssl-dev libnss3-dev libssl-dev liboath-dev python3.8-dev libncurses-dev libcap-ng-dev gperf

Edit the file do_cmake.sh to set "PYBUILD=3" just before the check for the PYBUILD version (line 39):

else
echo Unknown release
exit 1
fi
PYBUILD="3"
if [ "$PYBUILD" = "3" ] ; then
ARGS+=" -DWITH_PYTHON2=OFF -DWITH_PYTHON3=ON -DMGR_PYTHON_VERSION=3"
fi

Try the configuration builder:

./do_cmake.sh -DWITH_MANPAGE=OFF -DWITH_RDMA=OFF -DWITH_OPENLDAP=OFF -DWITH_FUSE=OFF -DWITH_XFS=OFF -DWITH_RADOSGW=OFF -DWITH_LTTNG=OFF -DWITH_BABELTRACE=OFF -DMGR_PYTHON_VERSION="3.8"

If there are no errors, compile the required library:

cd build && make radosstriper

Instructions for Scientific Linux

One method of building Ceph under Scientific Linux requires the configuration scripts to believe that the operating system is "centos", and fixing some Python package names.

Prerequisites for this approach are:

An installation of Python 3.8 (e.g. building it from source), and
Providing a recent version of the GCC/C++ tools, e.g. version 8 or or newer. On ceph-test-gw691, we have DevToolSet 9 from the Software Collections Library; installation instructions are here.

Install git if not already available:

yum install -y git

Clone the Ceph source code from GitHub:

git clone GitHub - ceph/ceph: Ceph is a distributed object, block, and file storage platform

I then checked out the revision we are currently using (as of October 2021, this is v14.2.15):

cd ceph && git checkout v14.2.15

Provide access to a recent version of the GCC/c++ development tools:

source /opt/rh/devtoolset-9/enable
gcc --version
gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2)
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Enable Python 3.8 as the current version. You might need to build this from source; see the instructions for building Python and installing with "make altinstall" at https://computingforgeeks.com/how-to-install-python-3-on-centos/ ), and set the install directory as /usr/local/bin. Install libffi-devel before compiling python or you might run into a missing ctypes module error.

yum install libffi-devel

export PATH=/usr/local/bin:$PATH
python3.8 --version
Python 3.8.12

Edit the script that installs dependencies to set the OS ID as "centos" and prevent attempts to install the Centos version of the Software Collections Library (as we already have the DevToolSet from the Software Collections Library available):

git diff install-deps.sh
diff --git a/install-deps.sh b/install-deps.sh
index 0fccd5b..0b61f82 100755
--- a/install-deps.sh
+++ b/install-deps.sh
@@ -275,6 +275,7 @@ else
for_make_check=false
fi
source /etc/os-release
+ ID=centos
case "$ID" in
debian|ubuntu|devuan)
echo "Using apt-get to install dependencies"
@@ -349,7 +350,7 @@ else
$SUDO $yumdnf install -y python36-devel
case "$ARCH" in
x86_64)

$SUDO $yumdnf -y install centos-release-scl

+ # $SUDO $yumdnf -y install centos-release-scl
dts_ver=8
;;
aarch64)

Fix the name of the Python scipy package in the Ceph spec file:

git diff ceph.spec.in
diff --git a/ceph.spec.in b/ceph.spec.in
index 8211ae7..7ddfa8a 100644
--- a/ceph.spec.in
+++ b/ceph.spec.in
@@ -296,7 +296,7 @@ BuildRequires: python%{_python_buildid}-pyOpenSSL
BuildRequires: python%{_python_buildid}-cherrypy
BuildRequires: python%{_python_buildid}-jwt
BuildRequires: python%{_python_buildid}-routes
-BuildRequires: python%{_python_buildid}-scipy
+BuildRequires: scipy
BuildRequires: python%{_python_buildid}-werkzeug
%endif
%if 0%{?suse_version}

Convince the CMake invocation script to use Python 3:

git diff do_cmake.sh
diff --git a/do_cmake.sh b/do_cmake.sh
index ab8f5c8..28e260c 100755
--- a/do_cmake.sh
+++ b/do_cmake.sh
@@ -35,7 +35,7 @@ else
echo Unknown release
exit 1
fi
-
+PYBUILD=3
if [ "$PYBUILD" = "3" ] ; then
ARGS+=" -DWITH_PYTHON2=OFF -DWITH_PYTHON3=ON -DMGR_PYTHON_VERSION=3"
fi

Install the dependencies:

./install_deps.sh

You might have to install cython trough pip if it's not detected.

Run CMake, turning off the options we don't need for a Ceph gateway client:

./do_cmake.sh -DWITH_MANPAGE=OFF -DWITH_RDMA=OFF -DWITH_OPENLDAP=OFF -DWITH_FUSE=OFF   -DWITH_XFS=OFF -DWITH_RADOSGW=OFF -DWITH_LTTNG=OFF -DWITH_BABELTRACE=OFF -DMGR_PYTHON_VERSION="3.8" -DWITH_RDB=OFF -DWITH_BLUESTORE=OFF -DWITH_ZFS=OFF -DWITH_LIBURING=OFF   -DWITH_SYSTEM_LIBURING=OFF -DWITH_WITH_BLUESTORE_PMEM=OFF -DWITH_SYSTEM_PMDK=OFF -DWITH_PMDK=OFF   -DWITH_RBD_MIGRATION_FORMAT_QCOW_V1=OFF -DWITH_SYSTEM_ROCKSDB=OFF

Compile the RADOS Striper library:

cd build && make radosstriper -j4 # Set -jN to make best use of CPUs

This will create the following files in the lib directory:

[root@host-172-16-103-243 build]# ls -lt lib
total 586440
-rwxr-xr-x 1 root root 37682480 Mar 21 15:25 libradosstriper.so.1.0.0
lrwxrwxrwx 1 root root 20 Mar 21 15:25 libradosstriper.so -> libradosstriper.so.1
lrwxrwxrwx 1 root root 24 Mar 21 15:25 libradosstriper.so.1 -> libradosstriper.so.1.0.0
-rwxr-xr-x 1 root root 44711768 Mar 21 15:24 librados.so.2.0.0
lrwxrwxrwx 1 root root 13 Mar 21 15:24 librados.so -> librados.so.2
lrwxrwxrwx 1 root root 17 Mar 21 15:24 librados.so.2 -> librados.so.2.0.0
-rwxr-xr-x 1 root root 353313248 Mar 21 15:24 libceph-common.so.0
lrwxrwxrwx 1 root root 19 Mar 21 15:24 libceph-common.so -> libceph-common.so.0
-rw-r--r-- 1 root root 70213016 Mar 21 15:23 libosdc.a
-rw-r--r-- 1 root root 44841044 Mar 21 15:22 liblibrados_impl.a
-rw-r--r-- 1 root root 80786 Mar 21 15:20 libcrc32.a
-rw-r--r-- 1 root root 5741374 Mar 21 15:15 libcls_lock_client.a
-rw-r--r-- 1 root root 42969264 Mar 21 15:12 libjson_spirit.a
-rw-r--r-- 1 root root 911104 Mar 21 15:10 liberasure_code.a
-rw-r--r-- 1 root root 16944 Mar 21 15:10 libarch.a
-rw-r--r-- 1 root root 13272 Mar 21 15:00 libcommon_utf8.a
[root@host-172-16-103-243 build]

Of these, we only need the shared object files; we don't need the static library files, those ending in ".a".

Notes:

symlinking python3.8 to python3 in /usr/bin or usr/local/bin makes it easier to debug when unexpected things happen
edit [python -version] - Cython to python3-Cython on ceph.spec.in if not detected

Compiling Alternate Version of RADOS Striper

Instructions for Ubuntu

Instructions for Scientific Linux

Related content