5 file servers, 1 name-space

Scalla/XRootD  EPEL

Professionally, I recently had to set up some improvised storage, making use of 6 machines, each with a 1 TB disk that was not used for anything else. Preferably with a common name space. Pooling such 6 disks into a common storage solution may sound like a common, mundane task, but there does not appear to be any open-source solutions that have reached anywhere near the status of a standard solution. Of stuff that I’ve heard of, I’ll mention GlusterFS, and, for the more academically inclined, a research project like Chelonia. Eventually, I went with XRootD – mostly because it is created for what my users are doing: high-energy-physics analysis of Root files, but also because it was co-developed by some of my old colleagues from the ARDA group at CERN.

I spent a good deal of time googling around, but the available documentation did not really seem up to date or particularly user-friendly: the task of tying together a small number of machines with large disks, to a one-name-space storage element, should, I believe, be a rather common one. So, being used to the open-source ecosystem, I was looking for a short howto, including step-by-step instructions, a couple of rpms or debs or an apt or yum repository and a sample configuration file. I didn’t find that. Perhaps I didn’t look hard enough, but the available docs did not really match the current version (which uses cmsd by default and apparently has the functionality of libXrdOfs.so compiled into the binaries) and were either lengthy or, well, not really what I needed.

So, here you have my short writeup – for Redhat/Fedora systems (thanks to Matthias Ellert for creating the rpms):

  1. Get and install xrootd, xrootd-libs, xrootd-server, xrootd-client and dependencies from the EPEL repository. Do this on the manager and data servers.
  2. Open ports in your firewalls: 1094 on the manager and data servers, 1213 on the manager (see below for some iptables lines.
  3. Create the directory /SE/xrootd on the data server and /var/cache/xrootd on the manager and make them owned by xrootd.
  4. Edit the two config files in /etc/xrootd/ – start with the templates below. Put both files in /etc/xrootd on both the manager and data servers.

Now start xrootd and cmsd on manager and data servers with the standard startup scripts and voila! you have an xrootd cluster. Notice that my config files serve data only from the data servers – the hard disk on the manager is not used for data.

Comments: XRootD could in principle provide a nice solution for those of us that need a place to just store files that don’t need to be accessed with low latency or too often. For this to be true, I would still like to see:

  • built-in http/https file serving – or at least simple instructions for serving with apache (without fuse)
  • some level of guarantee that if a file server goes down, my files will still be there – i.e. a possibility to specify number of redundant copies of each file


* xrootd.cf

* cmsd.cf

Firewall configuration:

A RH-Firewall-1-INPUT -m state –state NEW -m tcp -p tcp –dport 1094 -j ACCEPT
-A RH-Firewall-1-INPUT -m state –state NEW -m tcp -p tcp –dport 1213 -j ACCEPT

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>