Introduction Introduction

2014-11-25 -About a year ago we introduced the "CloudCase" an cluster in a briefcase intended to demonstrate and teach distributed computing, anything from Cloud computing to supercomputing including, of course, Crowd computing. During the past year we took it "On the Road" through all of Europe. Mostly by plane, but also by train. The CloudCase proved to be a useful tool for the intended purpose.

We also made several enhancements, both in hardware and software. In the hardware we changed the cabling. We now use cables which are as short as possible – 25 centimetre Ethernet cables for instance. This gives a much cleaner, uncluttered look. We also added a USB-WiFI and a USB Bluetooth connector to one of the Raspberry Pis. This allows to connect to an external WiFi network and to connect a bluetooth keyboard and mouse. One of the Raspbery Pis got a very tiny screen, useful for demonstrating what is happening inside.


It's a CloudCase now - a portable demo of the XtreemFS Cloud file system It's a CloudCase now - a portable demo of the XtreemFS Cloud file system

It's a CloudCase now - a portable demo of the XtreemFS Cloud file system

Demonstrating how the Cloud works is difficult, isn't it? It's all somewhere out there, isn't it? Well, it was. Thanks to the new Raspberry Pi credit size computer we can now build a fully functional Cloud system in a briefcase. We can carry it around to workshops and meetings just to tell about the workings of the XtreemFS Cloud file system, only by means of this CloudCase.

XtreemFS is a Cloud file system, designed to be distributed over many computers. The description of a file is separated from the file content itself, and XtreemFS can maintain several copies of the file data. What if the computer fails or is unreachable? No worries, XtreemFS just gets the data from another copy. Sounds complicated? Well, we can show it if we take our CloudCase.


This is the CloudCase closed.Just a briefcase we do not use anymore these days. Documents are on a tablet. So the briefcase was just gaining dust. But that changed. When we open the briefcase, now we see the actual miniature Cloud:

There are eight credit card sized Raspberry Pi computers. Despite their small size they are fully functional computers that run a standard Linux operating system. Three of them have a USB drive and are acting as storage for the files.

We included a network switch in the CloudCase that connects the Raspberries into one cluster of computers. Just like in any Cloud.

There are also some small USB Hubs that actually only serve to provide the Raspberry computers and the network with power.

And there are cables. Nineteen in total. Makes it all look pretty cool. There are small lights on the computers, the switch and the power supplies. Looks nice in the dark as you can see.

The only thing to get the micro XtreemFS Cloud started is hook it up, start the system, and your Cloud is running.

One of the Raspberry Pi computers acts as a network server, the router for the complete system.


Of the other 7 Raspberries six computers are configured to run XtreemFS.

First, there are three file data servers, or OSDs as the XtreemFS guys call them. These hold the data and we have three so there can be three copies to create a very resilient file system. Our favorite demo is the one where you play a video from the XtreemFS file system and unplug the file data server that is currently active. XtreemFS notices that, and redirects the video stream to another file data server, and the video continues.

Then there is one file catalogue server or MRC. This is where the file seems to be for the user, but actually this file catalogue server only knows where the content actually is. If you request a file from the file catalogue server it looks up were the data is, sends you a pointer to a file data server and off you go.

The fifth Raspberry Pi knows where the catalogue is and where the file data servers are. It is the directory server.

So these five servers are the core of the XtreemFS Cloud file system.

To access the file system we put an XtreemFS client on the sixth Raspberry Pi. From this machine you can as a user, access the file system, get a file - a document, video, anything - and use it. This is the boring part: files on XtreemFS look like any other file, and you access an XtreemFS Cloud file system as any other file system on your computer.

So we can also connect other computers, PCs, laptops to the XtreemFS in the CloudCase, and they can access the files in there.

XtreemFS has a lot of other nice features. Which ones? Well, we can come to you and show it in the CloudCase.

How did we build it?

We were looking for a reasonable number or computers to put in the CloudCase. Eight is a nice number. It allows for enough complexity to create a small Cloud, Grid or parallel computer. Four would be too small to run the complete distributed XtreemFS for instance.

Eight is also a typical computer number. So there are 8 port network switches that can connect eight computers. We opted for a fast 100 Mbit/s Ethernet switch, fast enough for nice demos.

Outside computers can also connect to the switch (but then you can only use 7 Raspberry Pis) or you can connect outside computers to the "network" computer: A Raspberry Pi that acts as a router and as a DHCP server giving networking addresses to the other computers.

The whole system consumes about 25W.

The software took a bit longer to put together.

Each Raspberry Pi runs a full Debian Linux operating system. The latest version has good support for Java, something you need for XtreemFS.

The XtreemFS server software (DIR, MRC, OSD) is available for Debian at the XtreemFS website. Because it is implemented in Java, it runs on any hardware that supports Debian.

Just follow the instructions and it is up and running in no time.

Some additional tweaks were needed to take care the USB storage is automatically attached to the system. Also some tweaks to get XtreemFS to automatically start when the Raspberries power up.

Installing the XtreemFS client takes a little more work. It is not built in Java, so you need a version that is specific to the ARM processor of the Raspberry Pi. The XtreemFS website has clients for Windows, MacOSX, and even Debian Linux. But unfortunately not one suited for the ARM processor of the Raspberry Pi. So we had to build that ourselves. The computer source code is available on the XtreemFS web site. So you download that computer code, download a number of libraries and some tools and you start compiling the software and building the client. We did all that on the Raspberry Pi itself. That takes patience, because the Rapberry Pi is not the fastest computer in the world. But after many hours, the XtreemFS client was successfully built.


After that we had the CloudCase XtreemFS Cloud file system fully functional: we can store files from the Rasperry Pi client computer into the XtreemFS file system and retrieve them. When we connect an outside computer, we can also access the files on XtreemFS.

It is a little bit like building your own DropBox or OneDrive, although these work slightly different. And as demos with the CloudCase may show, it can be as secure and user friendly as those commercial Cloud file systems, but more resilient and better controllable.

CloudCase - XtreemFS Cloud file system demonstration.


XtreemFS has a website at

More info on the Raspberry Pi:

(This work was done as part of the Contrail project.)

Some technical notes on running XtreemFS on Raspberry Pi Some technical notes on running XtreemFS on Raspberry Pi

Setting up the XtreemFS server and services on Raspberry Pi's

We use Raspian as operating system. The new versions include hardware supported Java. Because the XtreemFS software is in Java, the installation is straight forward.
We first installed XtreemFS on one Raspberry and then cloned the SD card for the other machines. However, XtreemFS generates a unique id for the services (UUID) at installation time.  This is cloned too so it is not unique anymore. What we needed to do, was to remove the UUID from the configuration file, and generate a new UUID.

We used five Raspberry Pis for the XtreemFS services: one for the MRC, one for the DIR and three for three OSDs.

We want to make them automatically boot the service when the system is powered on.

Two things that needed attention:

The services are automatically installed with runtime level 3. So we need to move them to level 2 to make the automatic start on Raspian:

pi@raspberrypi ~ $ sudo mv /etc/rc3.d/S02xtreemfs-* /etc/rc2.d/
pi@raspberrypi ~ $ sudo update-rc.d xtreemfs-osd defaults
update-rc.d: using dependency based boot sequencing
update-rc.d: warning: default start runlevel arguments (2 3 4 5) do not match xtreemfs-osd Default-Start values (3 5)
update-rc.d: warning: default stop runlevel arguments (0 1 6) do not match xtreemfs-osd Default-Stop values (0 1 2 6)
insserv: warning: current start runlevel(s) (2 5) of script `xtreemfs-osd' overrides LSB defaults (3 5).
insserv: warning: script 'mathkernel' missing LSB tags and overrides
pi@raspberrypi ~ $

"S02xtreemfs" can be different on your system. The warnings at the end do not really matter.

Second, the USB storage needs to be automatically mounted. We choose to do that by looking for the UUID for the storage device and include that in fstab:

pi@raspberrypi ~ $ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 15 Jan  1  1970 056E-4E83 -> ../../mmcblk0p1
lrwxrwxrwx 1 root root 10 Jan  1  1970 5aafc1ad-xxxx-yyyy-a247-2c5ef91f7929 -> ../../sda1
lrwxrwxrwx 1 root root 15 Jan 24 11:17 af599925-1134-4b6e-8883-fb6a99cd58f1 -> ../../mmcblk0p2
pi@raspberrypi ~ $ sudo more  /etc/fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that
pi@raspberrypi ~ $ sudo pico /etc/fstab
pi@raspberrypi ~ $ sudo more  /etc/fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that
UUID=5aafc1ad-xxxx-yyyy-a247-2c5ef91f7929 /mnt/usb ext4 defaults 0 2



XtreemFS client on Raspberry Pi

There is no standard executable of the XtreemFS client available for the ARM processor of the Raspberry Pi.

So you need to download the code and build the client yourself. This is straight forward. Take care you download the necessary libraries. But this is not difficult, following the build messages.

Compiling and building on the Raspberry Pi itself takes a long time, many hours, but you do not have to sit and look at it.

We needed:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install cmake
sudo apt-get install libfuse-dev
sudo apt-get install libssl-dev
sudo apt-get install libattr1-dev
sudo apt-get install boost
sudo apt-get install libboost-system-dev
sudo apt-get install libboost-thread-dev
sudo apt-get install libboost-program-options-dev
sudo apt-get install libboost-regex-dev

before we successfully could run:

make client

Video example

Setting up the example for the video test is taken from the presentation at the Cloud Summerschool Almere. We have three OSD data servers available. so let us use them.

The idea is to replicate a video file on all three. Pull the plug out of the main replica while the video is running and see how XtreemFS recovers from that.

Set replication to use 3 replicas on three OSD data servers.

ad$ xtfsutil --set-drp --replication-policy WqRq --replication-factor 3 /xtreemfs
Updated default replication policy to: WQRQ with 3 replicas

Next shows we have three OSD data servers available:

ad$ xtfsutil /xtreemfs
Path (on volume)     /
XtreemFS file Id     f3e6ff5c-xxxx-yyyy-8b64-39f802ce0d26:1
XtreemFS URL         pbrpc://
Owner                ad
Group                ad
ACL                  {}
Type                 volume
Free/Used Space      40 GB / 380 MB
Num. Files/Dirs      7 / 2
Access Control p.    POSIX (permissions & ACLs)
OSD Selection p.     1000,3002
Replica Selection p. default
Default Striping p.  STRIPING_POLICY_RAID0 / 1 / 128kB
Default Repl. p.     WqRq with 3 replicas
Selectable OSDs      01cb30af-b087-xxxx-yyyy-a0b1fa055064 (
                     ad4a3df9-5b88-xxxx-yyyy-2bcfe66dee73 (
                     ec4fc377-e8ef-xxxx-yyyy-b87f3323ae55 (
A file is replicated over these three data servers:

ad$ xtfsutil BigBuckBunny_640x360\ 2.m4v
Path (on volume)     /BigBuckBunny_640x360 2.m4v
XtreemFS file Id     f3e6ff5c-xxxx-yyyy-8b64-39f802ce0d26:3
XtreemFS URL         pbrpc:// 2.m4v
Owner                ad
Group                ad
ACL                  {}
Type                 file
Replication policy   ronly
XLoc version         7
  Replica 1
     Striping policy     STRIPING_POLICY_RAID0 / 1 / 128kB
     Replication Flags   full + complete
     OSD 1               01cb30af-xxxx-yyyy-a6c6-a0b1fa055064 (
  Replica 2
     Striping policy     STRIPING_POLICY_RAID0 / 1 / 128kB
     Replication Flags   full
     OSD 1               ec4fc377-xxxx-yyyy-9339-b87f3323ae55 (
  Replica 3
     Striping policy     STRIPING_POLICY_RAID0 / 1 / 128kB
     Replication Flags   full
     OSD 1               ad4a3df9-xxxx-yyyy-8551-2bcfe66dee73 (