Top Nav

GlusterFS Cluster With Ubuntu On RackSpace Cloud

GlusterFS Storage Cluster - New PageWe want to build a high-availability storage cluster based on GlusterFS using RackSpace Cloud Servers on Ubuntu. Additionally we want to use RackSpace Cloud Networks to provide private networks for replication and cluster access.

There’s a great doc by Falko Timme on Howto Forge titled:

High-Availability Storage With GlusterFS 3.2.x On Ubuntu 11.10 – Automatic File Replication Across Two Storage Servers

We’re going to follow this doc with a few modifications for Cloud Networks and using a newer version of Gluster.

As shown in the diagram, we’ll have two Cloud Networks, one for replication between the servers and one for access by clients to the storage.

Part 1 – Server Setup

1. Create servers and networks – Login to your RackSpace Cloud account and create two server following these guidelines:

  • Make sure you choose a Next Gen region for the servers.
  • For the Image choose “Ubuntu 13.04 (Raring Ringtail)”
  • Select a size based on your requirements. We’re using 1GB for this article.
  • In the Networks section, when creating the first server, click the “Create Network” button and add a network named “StorageInternal”. Then create a second network named “StorageAccess” When creating additional servers, make sure that you select the “StorageInternal” and “StorageAccess” networks.
  • If you need more storage then is included in the base image or if you want higher performance the consider using RackSpace Cloud Block Storage. This is not covered in this article.

Here are the addresses assigned to the servers that we created but yours my be different:


2. Open SSH shell to servers – Open an SSH shell to each of the servers. Unless noted otherwise the following steps should be repeated on both servers.

3. Upgrade packages – Update repository listings and then upgrade all packages to latest versions:

4. Setup /etc/hosts – Add hostnames on the “StorageInternal” network for each server.

5. Install glusterfs PPA repository – The default apt repositories for Ubuntu only offer version 3.2.7 of GlusterFS. We wanted a newer version so we’ll be modifying the install process to use the PPA distribution.  Execute the following commands

6. Install glusterfs server – Use this command to install:

7. Confirm gluster version – Make sure you have the correct version:

If you get an older version then you probably forgot to run “apt-get update”. Just remove the older package, update and reinstall:

8. Firewall configuration – Next we need to get a basic firewall configured to protect the servers. Ubuntu includes UFW (Uncomplicated Firewall) so we’ll go with that:

9. Add nodes to cluster – On fileserver1, run this command to register fileserver2 to the trusted storage pool:

The results should look like this:

10. Check cluster status – On each node run “gluster peer status” and confirm the peer addresses. Here’s fileserver1:

and here’s fileserver2:

11. Create volume – Next we need to create the actual GlusterFS volume:

You should get back a response like:

12. Start the volume – Now start the volume to make it available to clients with:

You should get a response like:

The howto by Falko Timme mentioned at the start of this article gives some useful advice and tips on how to troubleshoot if the volume fails to create or start.

13. Verify volume status – Check the status of the volume with:

You should get a response similar to:

14. Allow client access – We want to allow all clients on the “StorageAccess” network access to the cluster:

Now show volume info again:

And you should get:

Setup on the server is complete. Now it’s time to add clients.

Part 2 – Client Setup

1. Install glusterfs PPA repository – Execute the following commands

2. Install glusterfs client – Execute the following command:

3. Confirm version – Verify that you have the correct version installed:

root@db1:~# glusterfs –version
glusterfs 3.3.2 built on Jul 21 2013 16:38:01

4. Create mount point – Create a directory that will be the mount point for the gluster partition:

5. Setup /etc/hosts – Add the following entries to /etc/hosts:

Notice that we’re mapping both the StorageInternal and StorageAccess network names to the same IP addreses on the StorageAccess network.

6. Mount volume – Execute the command to mount the gluster volume:

7. Edit /etc/fstab – Add this line to /etc/fstab to make the mount start automatically on boot:

And that completes the client setup procedure. You now have a working GlusterFS storage cluster and a connected client.

Comments and suggestions are welcomed.

  • Pingback: GlusterFS Cluster with CentOS on RackSpace CloudReliable Penguin - Blog()

  • Gopu


    I followed you link to configure gluster across my two rackspace cloud server. I am not configuring client as I only need to replicate it across these servers and use it under Loadbalancer. Setup is completed successfully but the file I add on /data in first server is not replicating in second server and vice versa. Following are the output :
    root@gluster1:/data# gluster volume info

    Volume Name: testvol
    Type: Replicate
    Volume ID: ceb5e415-8194-4e30-85b6-d92492e86a11
    Status: Started
    Number of Bricks: 1 x 2 = 2
    Transport-type: tcp
    Brick1: gluster1:/data
    Brick2: gluster2:/data

    root@gluster1:~# gluster volume status
    Status of volume: testvol
    Gluster process Port Online Pid
    Brick gluster1:/data 24009 Y 794
    Brick gluster2:/data 24009 Y 784
    NFS Server on localhost 38467 Y 800
    Self-heal Daemon on localhost N/A Y 806
    NFS Server on gluster2 38467 Y 790
    Self-heal Daemon on gluster2 N/A Y 796

    root@gluster1:/data# gluster peer status
    Number of Peers: 1

    Hostname: gluster2
    Uuid: 5fead283-8f03-4495-86e4-2d80d2fefb1b
    State: Peer in Cluster (Connected)

    Did I miss anything ?

    • admin

      Files uploaded or copied directly to /data will not be replicated. You must up upload the files via NFS or a gluster client in order for replication to function. On way to accomplish this is to configure each server as a client and mount the gluster partition at something like /data-gl.

      Thanks for commenting — Lee

    • Gopu,
      Files uploaded or copied directly to /data will not be replicated. You must up upload the files via NFS or a gluster client in order for replication to function. On way to accomplish this is to configure each server as a client and mount the gluster partition at something like /data-gl.

      Thanks for commenting — Lee

      • Gopu

        Thanks a lot Lee !! I have configured gluster client in the same server and successfully configured.