Top Nav

HA Linux Cluster On RackSpace Cloud Servers

Our goal is to setup a pair of RackSpace Cloud Servers in a redundant cluster using a shared IP address. We’ll use the “heartbeat” package from Linux-HA (http://www.linux-ha.org) for the cluster messaging layer and “pacemaker” package from ClusterLabs (http://clusterlabs.org) for the cluster resource manager.

Before starting this procedure you’ll need to:

a. Create the two cloud servers. These instructions are specific to CentOS for the operating system.

b. Open a ticket with RackSpace Cloud support and request a public IP address to be shared between the servers.

You can use the instructions for other situations but you’ll need to make the appropriate adjustments.

1. Setup hosts file entries. On each server, edit /etc/hosts and add entries for each servers public and private interfaces. You’ll also find it convenient to setup ssh keys between the servers for easy access.

2. Now use yum to install some prerequisite packages:

Repeat this step on the second server.

Note: Several of these packages are not available on the standard RHEL yum channels. If you’re working on something other then a RackSpace Cloud server then you might need to install the EPEL channel. Just go to:

http://fedoraproject.org/wiki/EPEL

Then download and install the appropriate package to add EPEL.

3. The version of heartbeat available in the standard yum repositories is outdated. So we’ll install a more recent version of heartbeat, pacemaker and supporting components from:

http://www.clusterlabs.org/rpm

Start by creating a working folder:

Then use wget to download the latest version of each of the following package:

Finally install the packages:

Repeat this step on the second server.

5. Next step is to configure heartbeat.

a. Setup keys for authentication between the instances.

Edit /etc/ha.d/authkeys and add:

Replace [PASSWORD] with a long random string.

b. Set permissions on the authkeys file:

c. Next edit /etc/ha.d/ha.cf and add the following:

Set [HOST1] and [HOST2] to the hostnames of the servers.

Set [INTERNAL IP OF HOST2] to the private IP address of the second server.

Repeat these steps on the second server. When you create the ha.cf file for the second server, you’ll use the internal IP of the first server in the ucast line.

d. Setup logd for automatic startup:

Now repeat this procedure on the second server but make sure you set the internal IP of the first server in the ha.cf file.

6. Finally start the heartbeat and logd service on both servers:

7. The next step is to configure pacemaker.

Run the pacemaker configuration tool. It is called “crm”. You’ll use it to configure “resources” which in this case is a shared IP.

If you get an error like “cibadmin not available, check your installation” when trying to run crm, then make sure that the “which” package is installed and that /usr/sbin is in your path.

Now enter the following into the pacemaker shell:

Where [SHARED_IP] is the IP address to be shared between the servers and [HOST1] is the hostname of the primary server.

Once this is done then you should be able to monitor the status of the cluster from either node using the crm_mon command. You’ll get output like this:

8. Next step is to test failover on the servers.

a. Run crm_mon on the second server.

b. Reboot the first server:

c. Monitor the second server and notice that when the first goes offline, the “shared_one_ip” is switched to the second server. After the first server finishes rebooting then you should see it come back online and “shared_one_ip” return to it’s original location on the first server.

d. Repeat this test but reboot the second server and monitor the first.

And that completes the setup process. You now have an HA Linux cluster on the cloud!

  • Pingback: MySQL Load Balancing With HAProxy - Reliable Penguin Blog()

  • codex73

    Won’t rackspace try to sell the load balancing service instead? You suggest opening a ticket for ‘public IP address to be shared between the servers” . Would they know what to do?

    • Anonymous

      RackSpace just recently released the new Load Balancer service. We’ve been using it for several months in private beta
      for some of our clients. It’s a great service and can be used in place of building your own load balancer. For most
      simple cases the load balancer service will be a perfect solution. But for cases with special requirements a custom
      built load balancer may still be appropriate.

      RackSpace Cloud support definitely knows how to setup a shared IP address. We’ve made the request dozens of times with
      out a problem.

  • codex73
    • Anonymous

      Yes, RackSpace can allocate extra public IP addresses that can be shared between multiple servers.

  • codex73

    Isn’t a load balancer different in nature than a HA? Won’t we need both to maintain proper load distribution (dns) and/or HA? I’m studying how to build a high traffic architecture (initial) with low small effort. So far some kind of Load Balancer(distributes requests), HA Web Servers (file system) and MYSQL cluster. What order goes to what it’s still the challenge. One more thing I’ve researched is using Nginx which caches better than others aparently.

    • http://reliablepenguin.com Lee Blakely

      Yes load balancer and high availability are different goals. In a cluster of webheads, the load balancer monitors the webheads and stops sending request to servers that are down or overloaded. So in this case the LB and HA are handled by the same device. In a fileserver or mysql cluster LB and HA are separate functions. This set of articles is focused on HA/LB configurations for MySQL. The HA is provided by HA Linux. The LB is provided by HA Proxy.

      We use both HA Proxy and Nginx. Nginx is a great reverse proxy and cache. The ability to define custom rules is really powerful. We use HA Proxy for cases where we don’t need a cache or we’re proxying non-HTTP protocols like MySQL. In a couple of cases we’ve used both – HA Proxy for load balancing and Nginx for content caching.

  • Seveman

    In my experience, you should NOT use rackspacecloud for anything more than small workhorse-type servers that will require little setup work and keep no valuable information. I was running a server and somehow they lost my server images and were not able to recover it from the backup file in their own cloud files storage after 5 days. No way to upload a backup copy of your server images and techically unknowledgeable tech support so in case of what they called a “bug” you’re out of anything you had in your servers

  • Tarik Jabri

    Interesting article… is there any way to do the same except share a private IP address? I’d like to setup HA servers for private infrastructure that will not be accessible via the Internet (only used be other nodes within the cloud). Is this possible?

    • http://reliablepenguin.com Lee Blakely

      As of a few weeks ago, RS was not willing to provision private shared addresses. This will probably change in the near future. Best you can do right now I think is just configure IPTable to only accept traffic from known addresses.

Email
Print