Archive | Cloud Servers

Scalr 4.5 Install Notes For Cento 6.4

By admin on February 1, 2014 in Cloud Servers, Scalr

Last year we published an article on installing the open-source version of Scalr:

http://blogs.reliablepenguin.com/2013/08/29/scalr-install-notes

Now there’s a new 4.5 release of Scalr available so it’s time for an update.

We’re installing on Cento 6.4 hosted on a RackSpace Performance 1 Cloud Server.

The installation instructions have improved since the last time around but Scalr is till a complex install:

https://scalr-wiki.atlassian.net/wiki/display/docs/Installing+Scalr+4.5

Follow these instructions one section at a time and make sure you’ve got the section complete and working before moving to the next section. We’ve provided notes below about each section.

Before you start, select a hostname for the server and add a DNS A record. For this article we’ll use:

scalr.domain.com

1	scalr.domain.com

You’ll have problems later on if you don’t get the hostname setup in advance.

Also I like to add some swap to the server:

dd if=/dev/zero of=/1GB.swap bs=1M count=1024
mkswap /1GB.swap
swapon /1GB.swap
echo "/1GB.swap  none  swap  sw 0  0" >>/etc/fstab

dd if=/dev/zero of=/1GB.swap bs=1M count=1024

mkswap /1GB.swap

swapon /1GB.swap

echo "/1GB.swap none swap sw 0 0" >>/etc/fstab

Create UNIX Users and Group for Scalr

I setup a script at /root/env.sh with the environment variable as follows:

#!/bin/bash

export SERVICE_USER=root
export WEB_USER=apache
export SCALR_GROUP=scalr
export SCALR_INSTALL=/opt/scalr

#!/bin/bash

export SERVICE_USER=root

export WEB_USER=apache

export SCALR_GROUP=scalr

export SCALR_INSTALL=/opt/scalr

This makes it easy to get the environment right.

/usr/sbin/useradd  www-data
/usr/sbin/groupadd scalr
/usr/sbin/groupmems -g scalr -a root -a apache

/usr/sbin/useradd www-data

/usr/sbin/groupadd scalr

/usr/sbin/groupmems -g scalr -a root -a apache

Notice that we’re using user “apache” instead of “www-data” since this is CentOS instead of Ubuntu.

Configure your firewall

Edit /etc/sysconfig/iptables and add lines shown:

-A INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 8080 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited

-A INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 22 -j ACCEPT

-A INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 80 -j ACCEPT

-A INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 443 -j ACCEPT

-A INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 8080 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

And restart the service:

/sbin/service iptables restart

1	/sbin/service iptables restart

Install Scalr’s PHP Dependencies

The PHP dependencies are tricky. Tried using the IUS repository but kept having problems with errors like this:

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/http.so' - /usr/lib64/php/modules/http.so: undefined symbol: php_iconv_string in Unknown on line 0

1	PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/http.so' - /usr/lib64/php/modules/http.so: undefined symbol: php_iconv_string in Unknown on line 0

So I started over with the Remi repository:

wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
yum install remi-release-6.rpm

1 2	wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm yum install remi-release-6.rpm

Now edit /etc/yum.repos.d/remi.repo and enable the “remi” and “remi-php55” repositories.

yum install php php-mcrypt php-soap php-snmp php-mysqlnd php-ldap php-pecl-http1 php-pecl-yaml php-pecl-ssh2

1	yum install php php-mcrypt php-soap php-snmp php-mysqlnd php-ldap php-pecl-http1 php-pecl-yaml php-pecl-ssh2

The instructions say to install the php-pecl-rrd extension now but it’s better to wait due to dependency issues that we’ll see later.

Also notice that we’re installing php-pecl-http1 instead of php-pecl-http. This is critical.

Install Scalr’s Python Dependencies

Python is installed by default so just a couple of additional packages to install:

yum install python-devel libevent-devel python-crypto m2crypto net-snmp-python

1	yum install python-devel libevent-devel python-crypto m2crypto net-snmp-python

As with PHP, we’ll defer installing the python-rrdtool package until a later step.

Configure PHP for Scalr

Edit /etc/php.ini and (a) enable “short_open_tags” and (b) set the “date.timezone” setting.

Update System SNMP MIBs

For CentOS there is no “snmp-mibs-downloader”. I ended up doing nothing for this step.

Download and Install Scalr 4.5

I choose to install Scalr in /opt/scalr with the following steps:

cd /opt
wget https://github.com/Scalr/scalr/archive/v4.5.1.tar.gz
tar -xvzf v4.5.1
mv scalr-4.5.1 scalr

cd /opt

wget https://github.com/Scalr/scalr/archive/v4.5.1.tar.gz

tar -xvzf v4.5.1

mv scalr-4.5.1 scalr

Notice that back in the first step, I set the SCALR_INSTALL environment variable. No run the installer:

cd /opt/scalr/app/python
python setup.py install

1 2	cd /opt/scalr/app/python python setup.py install

Install and Configure MySQL

Install MySQL server and set to start on boot:

yum install mysql-server
/sbin/chkconfig --level 345 mysqld on
/sbin/service mysqld start

yum install mysql-server

/sbin/chkconfig --level 345 mysqld on

/sbin/service mysqld start

Set the mysql root login and secure:

http://blogs.reliablepenguin.com/2012/10/09/secure-mysql-installation

I like to add a .my.cnf file:

http://blogs.reliablepenguin.com/2012/10/09/create-my-cnf-file-for-mysql-authentication

Edit the /etc/my.cnf file and add this line to the “[mysqld]” section:

max_connections = 300

1	max_connections = 300

Now create the database and user for Scalr:

mysql
mysql> create database scalr_db;
mysql> grant all privileges on scalr_db.* to 'scalr_user'@'localhost' identified by 'xxxxxxxxxx';
mysql>flush privileges;

mysql

mysql> create database scalr_db;

mysql> grant all privileges on scalr_db.* to 'scalr_user'@'localhost' identified by 'xxxxxxxxxx';

mysql>flush privileges;

And load the database structure and data:

mysql scalr_db < /opt/scalr/sql/structure.sql
mysql scalr_db < /opt/scalr/sql/data.sql

1 2	mysql scalr_db < /opt/scalr/sql/structure.sql mysql scalr_db < /opt/scalr/sql/data.sql

Create the Scalr Cache folder

cd /opt/scalr
mkdir app/cache
chown root.apache app/cache
chmod 770 app/cache/

cd /opt/scalr

mkdir app/cache

chown root.apache app/cache

chmod 770 app/cache/

Install and Configure rrdtool and rrdcached

The default version of rrdtool is too old …. we need at least 1.4:

cd /root
yum erase rrdtool rrdtool-devel
yum install gettext ruby perl-Time-HiRes
wget http://packages.express.org/rrdtool/rrdtool-perl-1.4.7-1.el6.wrl.x86_64.rpm
wget http://packages.express.org/rrdtool/rrdtool-1.4.7-1.el6.wrl.x86_64.rpm
rpm -i rrdtool-1.4.7-1.el6.wrl.x86_64.rpm rrdtool-perl-1.4.7-1.el6.wrl.x86_64.rpm

cd /root

yum erase rrdtool rrdtool-devel

yum install gettext ruby perl-Time-HiRes

wget http://packages.express.org/rrdtool/rrdtool-perl-1.4.7-1.el6.wrl.x86_64.rpm

wget http://packages.express.org/rrdtool/rrdtool-1.4.7-1.el6.wrl.x86_64.rpm

rpm -i rrdtool-1.4.7-1.el6.wrl.x86_64.rpm rrdtool-perl-1.4.7-1.el6.wrl.x86_64.rpm

And now we can get the PHP and Python dependencies that we skipped earlier:

yum install php-pecl-rrd
wget http://packages.express.org/rrdtool/rrdtool-python-1.4.7-1.el6.wrl.x86_64.rpm
rpm -i rrdtool-python-1.4.7-1.el6.wrl.x86_64.rpm

yum install php-pecl-rrd

wget http://packages.express.org/rrdtool/rrdtool-python-1.4.7-1.el6.wrl.x86_64.rpm

rpm -i rrdtool-python-1.4.7-1.el6.wrl.x86_64.rpm

Set rrdcached to start on boot:

/sbin/chkconfig --level 345  rrdcached on

1	/sbin/chkconfig --level 345 rrdcached on

Edit /etc/sysconfig/rrdcached and change the “RRDCACHED_USER” to “root” and add the following line:

OPTIONS="$OPTIONS -s scalr -j /var/lib/rrdcached/journal/ -F -b /var/lib/rrdcached/db/ -B -l unix:/var/run/rrdcached.sock ";

1	OPTIONS="$OPTIONS -s scalr -j /var/lib/rrdcached/journal/ -F -b /var/lib/rrdcached/db/ -B -l unix:/var/run/rrdcached.sock ";

Now create the graphics and data directories:

mkdir -p /opt/scalr/app/www/graphics
chmod 755 /opt/scalr/app/www/graphics
chown root.www-data /opt/scalr/app/www/graphics
mkdir /var/lib/rrdcached/journal/
mkdir /var/lib/rrdcached/db/
mkdir --mode=775 /var/lib/rrdcached/db/{x1x6,x2x7,x3x8,x4x9,x5x0}
chown root:apache /var/lib/rrdcached/db/{x1x6,x2x7,x3x8,x4x9,x5x0}

mkdir -p /opt/scalr/app/www/graphics

chmod 755 /opt/scalr/app/www/graphics

chown root.www-data /opt/scalr/app/www/graphics

mkdir /var/lib/rrdcached/journal/

mkdir /var/lib/rrdcached/db/

mkdir --mode=775 /var/lib/rrdcached/db/{x1x6,x2x7,x3x8,x4x9,x5x0}

chown root:apache /var/lib/rrdcached/db/{x1x6,x2x7,x3x8,x4x9,x5x0}

And start the service:

/sbin/service rrdcached start

1	/sbin/service rrdcached start

Install and Configure Apache

Install Apache “httpd” package and SSL support:

yum install httpd mod_ssl

1	yum install httpd mod_ssl

Edit /etc/httpd/conf.d/vhosts.conf and add:

<VirtualHost *:80>
    ServerAdmin webmaster@reliablepenguin.com
    ServerName scalr.domain.com
    DocumentRoot /opt/scalr/app/www
    ErrorLog logs/scalr.domain.com-error_log
    CustomLog logs/scalr.domain.com-access_log combined
    <Directory /opt/scalr/app/www>
        Options -Indexes +FollowSymLinks +MultiViews
        AllowOverride All
        Order allow,deny
        allow from all
    </Directory>
</VirtualHost>

ServerAdmin webmaster@reliablepenguin.com

ServerName scalr.domain.com

DocumentRoot /opt/scalr/app/www

ErrorLog logs/scalr.domain.com-error_log

CustomLog logs/scalr.domain.com-access_log combined

Options -Indexes +FollowSymLinks +MultiViews

AllowOverride All

Order allow,deny

allow from all

</Directory>

</VirtualHost>

Set Apache to start on boot and start it now:

/sbin/chkconfig --level 345 httpd on
/sbin/service httpd start

1 2	/sbin/chkconfig --level 345 httpd on /sbin/service httpd start

Configure Scalr

Copy the sample config file:

cp /opt/scalr/app/etc/config.yml-sample /opt/scalr/app/etc/config.yml

1	cp /opt/scalr/app/etc/config.yml-sample /opt/scalr/app/etc/config.yml

Now edit the config file at /opt/scalr/app/etc/config.yml and set the following parameters:

scalr:
  connections:
    mysql: &connections_mysql
      host: 'localhost'
      port: ~
      name: scalr_db
      user: scalr_user
      pass: 'XXXXXXXXXXXXX'
  email:
    address: 'scalr@domain.com'
    name: 'RP Scalr'
  endpoint:
    host: 'scalr2.domain.com'
  load_statistics:
    connections:
      plotter:
        host: http://scalr2.domain.com
        port: 8080
    rrd_dir: /var/lib/rrdcached/db
    img_dir: /opt/scalr/app/www/graphics
    img_url: /graphics

scalr:

connections:

mysql: &connections_mysql

host: 'localhost'

port: ~

name: scalr_db

user: scalr_user

pass: 'XXXXXXXXXXXXX'

email:

address: 'scalr@domain.com'

name: 'RP Scalr'

endpoint:

host: 'scalr2.domain.com'

load_statistics:

connections:

plotter:

host: http://scalr2.domain.com

port: 8080

rrd_dir: /var/lib/rrdcached/db

img_dir: /opt/scalr/app/www/graphics

img_url: /graphics

Leave the other parameters at default settings.

Configure the Scalr Cronjobs

Edit “apache” cronjobs:

crontab -e -u apache

1	crontab -e -u apache

and add the following:

* * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --Scheduler
*/5 * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --UsageStatsPoller
*/2 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --Scaling
*/2 * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --SzrMessaging
*/2 * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --BundleTasksManager
*/15 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --MetricCheck
*/2 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --Poller
* * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --DNSManagerPoll
17 5 * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --RotateLogs
*/2 * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --EBSManager
*/20 * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --RolesQueue
*/5 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --DbMsrMaintenance
*/20 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --LeaseManager
*/1 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --ServerTerminate

* * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --Scheduler

*/5 * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --UsageStatsPoller

*/2 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --Scaling

*/2 * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --SzrMessaging

*/2 * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --BundleTasksManager

*/15 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --MetricCheck

*/2 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --Poller

* * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --DNSManagerPoll

17 5 * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --RotateLogs

*/2 * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --EBSManager

*/20 * * * * /usr/bin/php -q /opt/scalr/app/cron/cron.php --RolesQueue

*/5 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --DbMsrMaintenance

*/20 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --LeaseManager

*/1 * * * * /usr/bin/php -q /opt/scalr/app/cron-ng/cron.php --ServerTerminate

Configure the Scalr Daemons

Edit /etc/init.d/scalr and copy/paste the following contents:

#!/bin/bash
# scalr daemon
# chkconfig: 345 20 80
# description: scalr daemons
# processname: scalr

PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

case "$1" in
start)
        printf "%-50s" "Starting message sender..."
        PID=`python -m scalrpy.msg_sender -p /var/run/msg_sender.pid -l /var/log/scalr.msg_sender.log -c /opt/scalr/app/etc/config.yml --start > /dev/null 2>&1 & echo $!`
        if [ -z $PID ]; then
            printf "%s\n" "Fail"
        else
            echo $PID > $PIDFILE
            printf "%s\n" "Ok"
        fi

        printf "%-50s" "Starting queue processor..."
        PID=`python -m scalrpy.dbqueue_event -p /var/run/dbqueue_event.pid -l /var/log/scalr.dbqueue_event.log -c /opt/scalr/app/etc/config.yml --start > /dev/null 2>&1 & echo $!`
        if [ -z $PID ]; then
            printf "%s\n" "Fail"
        else
            echo $PID > $PIDFILE
            printf "%s\n" "Ok"
        fi

        printf "%-50s" "Starting stats plotter..."
        PID=`python -m scalrpy.load_statistics -p /var/run/load_statistics.plotter.pid -l /var/log/scalr.load_statistics.plotter.log -c /opt/scalr/app/etc/config.yml --plotter --start > /dev/null 2>&1 & echo $!`
        if [ -z $PID ]; then
            printf "%s\n" "Fail"
        else
            echo $PID > $PIDFILE
            printf "%s\n" "Ok"
        fi

        printf "%-50s" "Starting stats poller..."
        PID=`python -m scalrpy.load_statistics -p /var/run/load_statistics.poller.pid -l /var/log/scalr.load_statistics.poller.log -c /opt/scalr/app/etc/config.yml --poller --start  > /dev/null 2>&1 & echo $!`
        if [ -z $PID ]; then
            printf "%s\n" "Fail"
        else
            echo $PID > $PIDFILE
            printf "%s\n" "Ok"
        fi

;;
status)
        printf "%-50s" "Checking message sender..."
        PIDFILE=/var/run/msg_sender.pid
        if [ -f $PIDFILE ]; then
            PID=`cat $PIDFILE`
            if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
                printf "%s\n" "Process dead but pidfile exists"
            else
                echo "Running"
            fi
        else
            printf "%s\n" "Service not running"
        fi

        printf "%-50s" "Checking queue processor..."
        PIDFILE=/var/run/dbqueue_event.pid
        if [ -f $PIDFILE ]; then
            PID=`cat $PIDFILE`
            if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
                printf "%s\n" "Process dead but pidfile exists"
            else
                echo "Running"
            fi
        else
            printf "%s\n" "Service not running"
        fi

        printf "%-50s" "Checking stats plotter..."
        PIDFILE=/var/run/load_statistics.plotter.pid
        if [ -f $PIDFILE ]; then
            PID=`cat $PIDFILE`
            if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
                printf "%s\n" "Process dead but pidfile exists"
            else
                echo "Running"
            fi
        else
            printf "%s\n" "Service not running"
        fi

        printf "%-50s" "Checking stats poller..."
        PIDFILE=/var/run/load_statistics.poller.pid
        if [ -f $PIDFILE ]; then
            PID=`cat $PIDFILE`
            if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
                printf "%s\n" "Process dead but pidfile exists"
            else
                echo "Running"
            fi
        else
            printf "%s\n" "Service not running"
        fi

;;
stop)
        printf "%-50s" "Stopping message sender..."
        PIDFILE=/var/run/msg_sender.pid
            PID=`cat $PIDFILE`
            cd $DAEMON_PATH
        if [ -f $PIDFILE ]; then
            kill -HUP $PID
            printf "%s\n" "Ok"
            rm -f $PIDFILE
        else
            printf "%s\n" "pidfile not found"
        fi

        printf "%-50s" "Stopping queue processor..."
        PIDFILE=/var/run/dbqueue_event.pid
            PID=`cat $PIDFILE`
            cd $DAEMON_PATH
        if [ -f $PIDFILE ]; then
            kill -HUP $PID
            printf "%s\n" "Ok"
            rm -f $PIDFILE
        else
            printf "%s\n" "pidfile not found"
        fi

        printf "%-50s" "Stopping stats plotter..."
        PIDFILE=/var/run/load_statistics.plotter.pid
            PID=`cat $PIDFILE`
            cd $DAEMON_PATH
        if [ -f $PIDFILE ]; then
            kill -HUP $PID
            printf "%s\n" "Ok"
            rm -f $PIDFILE
        else
            printf "%s\n" "pidfile not found"
        fi

        printf "%-50s" "Stopping stats poller..."
        PIDFILE=/var/run/load_statistics.poller.pid
            PID=`cat $PIDFILE`
            cd $DAEMON_PATH
        if [ -f $PIDFILE ]; then
            kill -HUP $PID
            printf "%s\n" "Ok"
            rm -f $PIDFILE
        else
            printf "%s\n" "pidfile not found"
        fi

;;

restart)
        $0 stop
        $0 start
;;

*)
        echo "Usage: $0 {status|start|stop|restart}"
        exit 1
esac

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

#!/bin/bash

# scalr daemon

# chkconfig: 345 20 80

# description: scalr daemons

# processname: scalr

PIDFILE=/var/run/$NAME.pid

SCRIPTNAME=/etc/init.d/$NAME

case "$1" in

start)

printf "%-50s" "Starting message sender..."

PID=`python -m scalrpy.msg_sender -p /var/run/msg_sender.pid -l /var/log/scalr.msg_sender.log -c /opt/scalr/app/etc/config.yml --start > /dev/null 2>&1 & echo $!`

if [ -z $PID ]; then

printf "%s\n" "Fail"

else

echo $PID > $PIDFILE

printf "%s\n" "Ok"

printf "%-50s" "Starting queue processor..."

PID=`python -m scalrpy.dbqueue_event -p /var/run/dbqueue_event.pid -l /var/log/scalr.dbqueue_event.log -c /opt/scalr/app/etc/config.yml --start > /dev/null 2>&1 & echo $!`

if [ -z $PID ]; then

printf "%s\n" "Fail"

else

echo $PID > $PIDFILE

printf "%s\n" "Ok"

printf "%-50s" "Starting stats plotter..."

PID=`python -m scalrpy.load_statistics -p /var/run/load_statistics.plotter.pid -l /var/log/scalr.load_statistics.plotter.log -c /opt/scalr/app/etc/config.yml --plotter --start > /dev/null 2>&1 & echo $!`

if [ -z $PID ]; then

printf "%s\n" "Fail"

else

echo $PID > $PIDFILE

printf "%s\n" "Ok"

printf "%-50s" "Starting stats poller..."

PID=`python -m scalrpy.load_statistics -p /var/run/load_statistics.poller.pid -l /var/log/scalr.load_statistics.poller.log -c /opt/scalr/app/etc/config.yml --poller --start > /dev/null 2>&1 & echo $!`

if [ -z $PID ]; then

printf "%s\n" "Fail"

else

echo $PID > $PIDFILE

printf "%s\n" "Ok"

;;

status)

printf "%-50s" "Checking message sender..."

PIDFILE=/var/run/msg_sender.pid

if [ -f $PIDFILE ]; then

PID=`cat $PIDFILE`

if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then

printf "%s\n" "Process dead but pidfile exists"

else

echo "Running"

else

printf "%s\n" "Service not running"

printf "%-50s" "Checking queue processor..."

PIDFILE=/var/run/dbqueue_event.pid

if [ -f $PIDFILE ]; then

PID=`cat $PIDFILE`

if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then

printf "%s\n" "Process dead but pidfile exists"

else

echo "Running"

else

printf "%s\n" "Service not running"

printf "%-50s" "Checking stats plotter..."

PIDFILE=/var/run/load_statistics.plotter.pid

if [ -f $PIDFILE ]; then

PID=`cat $PIDFILE`

if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then

printf "%s\n" "Process dead but pidfile exists"

else

echo "Running"

else

printf "%s\n" "Service not running"

printf "%-50s" "Checking stats poller..."

PIDFILE=/var/run/load_statistics.poller.pid

if [ -f $PIDFILE ]; then

PID=`cat $PIDFILE`

if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then

printf "%s\n" "Process dead but pidfile exists"

else

echo "Running"

else

printf "%s\n" "Service not running"

;;

stop)

printf "%-50s" "Stopping message sender..."

PIDFILE=/var/run/msg_sender.pid

PID=`cat $PIDFILE`

cd $DAEMON_PATH

if [ -f $PIDFILE ]; then

kill -HUP $PID

printf "%s\n" "Ok"

rm -f $PIDFILE

else

printf "%s\n" "pidfile not found"

printf "%-50s" "Stopping queue processor..."

PIDFILE=/var/run/dbqueue_event.pid

PID=`cat $PIDFILE`

cd $DAEMON_PATH

if [ -f $PIDFILE ]; then

kill -HUP $PID

printf "%s\n" "Ok"

rm -f $PIDFILE

else

printf "%s\n" "pidfile not found"

printf "%-50s" "Stopping stats plotter..."

PIDFILE=/var/run/load_statistics.plotter.pid

PID=`cat $PIDFILE`

cd $DAEMON_PATH

if [ -f $PIDFILE ]; then

kill -HUP $PID

printf "%s\n" "Ok"

rm -f $PIDFILE

else

printf "%s\n" "pidfile not found"

printf "%-50s" "Stopping stats poller..."

PIDFILE=/var/run/load_statistics.poller.pid

PID=`cat $PIDFILE`

cd $DAEMON_PATH

if [ -f $PIDFILE ]; then

kill -HUP $PID

printf "%s\n" "Ok"

rm -f $PIDFILE

else

printf "%s\n" "pidfile not found"

;;

restart)

$0 stop

$0 start

;;

echo "Usage: $0 {status|start|stop|restart}"

exit 1

esac

Now set the service to start on boot and start it for the first time:

/sbin/chkconfig --level 345 scalr on
/sbin/service scalr start

1 2	/sbin/chkconfig --level 345 scalr on /sbin/service scalr start

Validate your Scalr installation

Now run the validation script:

cd /opt/scalr/app/www/
php testenvironment.php

1 2	cd /opt/scalr/app/www/ php testenvironment.php

Log in to Scalr

Open a browser and go to:

http://scalr.domain.com

Go to the Admin -> admin -> edit and change the admin password.

All Done!

Scalr install is now complete. You can get started using Scalr by adding a user and building an environment.

Scale Out For SharkTank

By leerb on December 7, 2013 in Cloud Load Balancers, Cloud Servers, Varnish

Recently one of our clients was featured on SharkTank, the critically-acclaimed business-themed show, featuring the Sharks and their continuing the search to invest in the best businesses and products that America has to offer.

Fohawx is a line of cool accessories that can easily attach to any kind of safety helmet – instantly transforming annoying headgear into a fashion statement. The appearance on SharkTank was a great opportunity for Fohawx to showcase their product.

Based on past experience we know that their web site could get 5000 or more simultaneous users. The small RackSpace Cloud Server on which they were hosted would be vaporized by this traffic. Due to the proprietary nature of their application server (ColdFusion) we could not add additional web servers. So a couple of days before the event we outlined measures to temporarily scale out their hosting capacity:

1. Re-sized the web server to 16GB of RAM. We wanted to go to 30GB but there was not sufficient space available on the “huddle” so we could only go to 16GB. We could move to a diffierent huddle but this would require and IP address change.

2. Added a separate database server. We spun up a 60GB server in RackSpace’s new Performance flavor and moved the application database from the web server to this new database server. The goal was to offload the web server and allow for faster database operations.

3. Added a pair of Varnish cache servers in front of the web server. The Varnish servers were built on small RackSpace Cloud Servers. As a caching proxy, Varnish can be used to offload static content from the web server. We used two Varnish servers to provide redundancy and increase throughput.

4. Added a RackSpace Cloud Load Balancer in front of the Varnish servers. The Cloud Load Balancer was configured to distribute traffic evenly between the Varnish servers.

In total this configuration cost about $3 per hour to operate and it ran for 4 days so the total cost was less then $220 hour.

The Sharks were not big fans of Fohawx but viewers seemed to disagree. In the hour after the initial airing the site served more the 1 million hits with a peak of over 4100 simultaneous users. Amazingly the Varnish cache servers handled 99% of all request and only passed 1% through to the web server.

This case study demonstrates how Reliable Penguin can combine RackSpace Cloud services with open-source components to rapidly meet emerging hosting challenges.

Watch the SharkTank episode here:

http://watchabc.go.com/shark-tank/SH559076/VDKA0_pbz8umsy/week-11

Fohawx starts around 00:23.

And don’t forget to get your Fohawx at:

http://fohawx.com

Percona XtraDB Cluster On CentOS

By admin on November 14, 2013 in Cloud Servers, Percona, Uncategorized

This article is built off of a similar article we published last month:

Percona XtraDB Cluster on Ubuntu

The primary difference is this time we’re going to use CentOS instead of Ubuntu.

In this article we’re going to build a Percona XtraDB Cluster using a pair of RackSpace Cloud Servers. Percona XtraDB Cluster is a MySQL compatible replacement supporting multi-master replication. For this project we’ll use the latest CentOS release and we’re going to use RackSpace Cloud Networks to setup an isolated segment for the replication between the cluster nodes. We’ll do two nodes in the cluster but you can add additional nodes as desired. Finally we’ll use a RackSpace Cloud Load Balancer to distribute traffic between the nodes.

To get started, create the cloud servers from the RackSpace control panel using the following guidelines:

Make sure you choose a Next Gen region for the servers.
For the Image choose “CentOS 6.4”
Select a size based on your requirements. We’re using 1GB for this article.
In the Networks section, when creating the first server, click the “Create Network” button and add a network named “DatabaseInternal”. When creating additional servers, make sure that you select the “DatabaseInternal” network.
If you need more storage then is included in the base image or if you want higher performance the consider using RackSpace Cloud Block Storage. This is not covered in this article.

For this article we’ll assume that the “DatabaseInternal” cloud network is:

192.168.3.0/24

1	192.168.3.0/24

and our servers are:

db1.acme.com
public: 1.1.1.1
private: 10.10.10.1
DatabaseInternal: 192.168.5.2

db1.acme.com

public: 1.1.1.1

private: 10.10.10.1

DatabaseInternal: 192.168.5.2

and:

db2.acme.com
public: 1.1.1.2
private: 10.10.10.2
DatabaseInternal: 192.168.5.3

db2.acme.com

public: 1.1.1.2

private: 10.10.10.2

DatabaseInternal: 192.168.5.3

Here are the basic instructions from Percona that we’ll be following:

http://www.percona.com/doc/percona-xtradb-cluster/installation.html

To get started open SSH terminal sessions to each server and complete the following steps on each server unless noted otherwise:

1. Disable SELinux – the Percona docs state that the cluster will not work with SELinux and it must be disabled. CentOS on RackSpace Cloud install with SELinux disabled by default. You can confirm this with:

sestatus

sestatus

The response should be:

SELinux status:                 disabled

1	SELinux status: disabled

If SELinux is enabled then follow these instructions to disable:

http://www.centos.org/docs/5/html/5.2/Deployment_Guide/sec-sel-enable-disable.html

Now reboot the server to get a clean system without SELinux.

2. Configure firewall – Next we need to get a basic firewall configured to protect the servers. CentOS uses iptables with no frontend so we’ll edit:

/etc/sysconfig/iptables

Add the two lines highlighted in bold:

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth1 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 3306 -i eth2 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

# Firewall configuration written by system-config-firewall

# Manual customization of this file is not recommended.

*filter

:INPUT ACCEPT [0:0]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [0:0]

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

-A INPUT -p icmp -j ACCEPT

-A INPUT -i lo -j ACCEPT

-A INPUT -i eth1 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 3306 -i eth2 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

-A FORWARD -j REJECT --reject-with icmp-host-prohibited

COMMIT

The first new rule for eth1 allows all traffic between the database servers on the “DatabaseInternal” network. The second rule allows connections to TCP 3306 (mysql) on the eth0 network which is the public interface. You might want to change this rule and instead limit access to specific IP addresses like this:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 3306 -s x.x.x.x -j ACCEPT

1	-A INPUT -m state --state NEW -m tcp -p tcp --dport 3306 -s x.x.x.x -j ACCEPT

Replace x.x.x.x with the IP address of the client (web) server.

Now restart the service to apply changes:

/sbin/service iptables restart

1	/sbin/service iptables restart

3. /etc/hosts – Let’s add some entries to the /etc/hosts file:

192.168.5.2 db1-dbint
192.168.5.3 db2-dbint

1 2	192.168.5.2 db1-dbint 192.168.5.3 db2-dbint

4. Add Percona yum Repository – Just follow the instructions here:

http://www.percona.com/doc/percona-xtradb-cluster/installation/yum_repo.html

Basically just execute:

rpm -Uhv http://www.percona.com/redir/downloads/percona-release/percona-release-0.0-1.x86_64.rpm

1	rpm -Uhv http://www.percona.com/redir/downloads/percona-release/percona-release-0.0-1.x86_64.rpm

5. Install packages – Run the following command to install the cluster packages:

yum install Percona-XtraDB-Cluster-server Percona-XtraDB-Cluster-client

1	yum install Percona-XtraDB-Cluster-server Percona-XtraDB-Cluster-client

6. /etc/my.cnf – Setup configuration files on each server. The Percona distribution does not include a my.cnf file to you need to role your own. The minimal configuration would be something like this:

[mysqld]
wsrep_node_address=192.168.5.2
wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_cluster_address=gcomm://db02-dbint
wsrep_slave_threads=8
wsrep_sst_method=rsync
wsrep_replicate_myisam=1
binlog_format=ROW
default_storage_engine=InnoDB
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
max_connect_errors=10000000

[mysqld]

wsrep_node_address=192.168.5.2

wsrep_provider=/usr/lib64/libgalera_smm.so

wsrep_cluster_address=gcomm://db02-dbint

wsrep_slave_threads=8

wsrep_sst_method=rsync

wsrep_replicate_myisam=1

binlog_format=ROW

default_storage_engine=InnoDB

innodb_locks_unsafe_for_binlog=1

innodb_autoinc_lock_mode=2

max_connect_errors=10000000

Most of this is straight from the Percona documentation. Key lines are:

wsrep_node_address=192.168.5.3

1	wsrep_node_address=192.168.5.3

This tells Percona what address to advertise to other nodes in the cluster. We need this set to the server “DatabaseInternal” interface. Without this setting SST will fail when it uses the wrong interface and is blocked by the firewall.

wsrep_cluster_address=gcomm://db01-dbint

1	wsrep_cluster_address=gcomm://db01-dbint

This line identifies at least one other member in the cluster. Notice that we’re using the names we set in the /etc/hosts file.

The above example for my.cnf is very minimal. It does not address any database memory or performance tuning issues so you’ll likely want to expand upon the example.

7. Bootstrap Cluster – The cluster needs to be bootstrapped on the first server when it’s started for the first time. This can be accomplished with:

/sbin/service mysql bootstrap-pxc

1	/sbin/service mysql bootstrap-pxc

The subject of bootstrapping is covered in more detail here:

http://www.percona.com/doc/percona-xtradb-cluster/manual/bootstrap.html

The first node startup will look something like this:

root@db1:~# /etc/init.d/mysql bootstrap-pxc
Bootstrapping PXC (Percona XtraDB Cluster)Starting MySQL (Percona XtraDB Cluster).... SUCCESS!

1 2	root@db1:~# /etc/init.d/mysql bootstrap-pxc Bootstrapping PXC (Percona XtraDB Cluster)Starting MySQL (Percona XtraDB Cluster).... SUCCESS!

8. Secure mysql installation

mysql_secure_installation

1	mysql_secure_installation

9. /root/.my.cnf – Add a .my.cnf for MySQL root authentication as described here:

http://blogs.reliablepenguin.com/2012/10/09/create-my-cnf-file-for-mysql-authentication

This step is optional. The .my.cnf is convenient but should not be used in high security environments.

10. Start additional nodes

Now with the first node started, open a mysql command shell and view the wsrep_cluster_% status variables:

mysql> show status like "wsrep_cluster_%";
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id    | 1                                    |
| wsrep_cluster_size       | 1                                    |
| wsrep_cluster_state_uuid | 67d8795f-1339-11e3-94f1-9b7bbcee0e35 |
| wsrep_cluster_status     | Primary                              |
+--------------------------+--------------------------------------+
4 rows in set (0.01 sec)

mysql> show status like "wsrep_cluster_%";

+--------------------------+--------------------------------------+

| Variable_name | Value |

+--------------------------+--------------------------------------+

| wsrep_cluster_conf_id | 1 |

| wsrep_cluster_size | 1 |

| wsrep_cluster_state_uuid | 67d8795f-1339-11e3-94f1-9b7bbcee0e35 |

| wsrep_cluster_status | Primary |

+--------------------------+--------------------------------------+

4 rows in set (0.01 sec)

Notice that the wsrep_cluster_size is 1 and the wsrep_cluster_status is “Primary”. This is normal for the first node in a newly bootstrapped cluster.

Now we’ll start Percona normally on each additional server. So go to db2 and run:

/sbin/service mysql start

1	/sbin/service mysql start

This time the start should look something like this:

root@db2:~# /sbin/service mysql start
 Starting MySQL (Percona XtraDB Cluster)......SST in progress, setting sleep higher. SUCCESS!

1 2	root@db2:~# /sbin/service mysql start Starting MySQL (Percona XtraDB Cluster)......SST in progress, setting sleep higher. SUCCESS!

Startup for the second node adds the SST or “State Snapshot Transfer”. In this step the servers will use rsync and SSH keys to transfer a copy of the the database from the first server to the second server.

Back on the first server in our mysql command shell we can check the status again:

mysql&gt; show status like "wsrep_cluster_%";
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id    | 2                                    |
| wsrep_cluster_size       | 2                                    |
| wsrep_cluster_state_uuid | 613f8f9a-133a-11e3-b4e0-9b8eeba56f39 |
| wsrep_cluster_status     | Primary                              |
+--------------------------+--------------------------------------+
4 rows in set (0.00 sec)

mysql> show status like "wsrep_cluster_%";

+--------------------------+--------------------------------------+

| Variable_name | Value |

+--------------------------+--------------------------------------+

| wsrep_cluster_conf_id | 2 |

| wsrep_cluster_size | 2 |

| wsrep_cluster_state_uuid | 613f8f9a-133a-11e3-b4e0-9b8eeba56f39 |

| wsrep_cluster_status | Primary |

+--------------------------+--------------------------------------+

4 rows in set (0.00 sec)

Notice now according to the “wsrep_cluster_size” there are 2 nodes in the cluster.

At this point we have a functional cluster up and running.

If the second node fails to start then check the log file at:

/var/lib/mysql/db2.err

The most likely cause is a problem resolving or connecting to the db1-int server for SST.

9. Add extra functions – There are a couple of Percona specific functions that can be added to support monitoring:

mysql -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'"
mysql -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"
mysql -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"

mysql -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'"

mysql -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"

mysql -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"

10. Add load balancer – The next step is to add the RackSpace Cloud Load Balancer. The load balancer will provide a single IP address for clients to connect to. It will then distribute these connections to the nodes in the cluster.

a. Login to your RackSpace Cloud Control Panel.
b. Go to Hosting section and the Load Balancers tab.
c. Click the “Create Load Balancer” button.
d. In the “Identification” section, enter a name for the load balancer like “lb-db-01” and select the Region. Use the same Region that the cluster nodes are located in.
e. In the “Configuration” section, select “On the Private RackSpace Network” for the “Virtual IP”. Set the “Protocol” to “MySQL” and the port to “3306”. Set the “Algorithm” to “Least Connections”.
f. In the “Add Nodes” section, click the “Add Cloud Servers” button and select each of the servers in the cluster.
g. Click the “Create Load Balancer” button to save the new load balancer.

It may take a couple of minutes for the load balancer to be created. When complete the IP address assigned to the load balancer will be visible. We’ll assume for this article that the address is:

10.10.10.10

1	10.10.10.10

Notice that this is a private, unroutable address on the RackSpace Service Network. This address is not accessible from the public Internet but it is visible to other cloud servers and devices in the same region on the RackSpace Service network. This is the address that your web or application servers will use to connect to the database.

11. Load balancer access controls – To minimize exposure of the database servers we need to add access controls on the load balancer that will limit the range of addresses that are allowed to connect. Generally you’ll only want connections from your web or application servers. In the RackSpace Cloud control panel, drill down to your load balance, find “Access Control” rules at the bottom and add a rule or rules to allow your client servers. Of course if you’re dynamically adding and removing servers then it might not be possible to use these access controls. Or you might need to use the load balancer API to dynamically change access controls.

12. Allow load balancer on firewalls – Next we need to adjust the firewall on each node to allow MySQL connections from the load balancer. The connections from the load balancer to the cluster nodes do not come from the load balancer address. Instead they can come from a range of addresses. The exact range depends on what region the load balancer was created in. At the time of writing this article the ranges were:

For the DFW region, use:
10.183.248.0/22
10.189.254.0/23

For the IAD region, use:
10.189.254.0/23

For the ORD region, use:
10.183.250.0/23
10.183.252.0/23
10.183.254.0/23
10.189.246.0/23

For the LON region, use:
10.189.246.0/23
10.190.254.0/23

For the SYD region, use:
10.189.254.0/23

For the HKG region, use:
10.189.254.0/23

This list may change over time. The latest ranges should be available here:

http://www.rackspace.com/knowledge_center/article/using-cloud-load-balancers-with-rackconnect

For this article we used the DFW region so the iptables rules would be:

-A INPUT -i eth1 -p tcp -s 10.183.248.0/22 -j ACCEPT
-A INPUT -i eth1 -p tcp -s 10.189.254.0/23 -j ACCEPT

1 2	-A INPUT -i eth1 -p tcp -s 10.183.248.0/22 -j ACCEPT -A INPUT -i eth1 -p tcp -s 10.189.254.0/23 -j ACCEPT

13. Create database & users – Now we’re ready to create a database and user for our web application. For any of the cluster nodes, open a MySQL command shell, create the database and add a database user:

CREATE DATABASE mywebapp;
GRANT ALL PRIVILEGES ON mywebapp.* TO 'mywebapp_user'@'%' IDENTIFIED BY 'some pass';

1 2	CREATE DATABASE mywebapp; GRANT ALL PRIVILEGES ON mywebapp.* TO 'mywebapp_user'@'%' IDENTIFIED BY 'some pass';

Now from your web application should be able to connect to the database cluster with the user that you just create. The database host should be the address of the load balancer.

14. Raise timeout – By default Cloud Load Balancers will timeout any idle connection after 30 seconds. This article shows how to raise the timeout:

https://community.rackspace.com/products/f/25/t/89

15. Add monitoring tools – You’ll probably want to add a few tools to monitor the database. Here’s what we normally install:

myq_gadets is a great set of monitoring utilites:

http://blogs.reliablepenguin.com/wp-admin/post.php?post=818&action=edit&message=6

mysqltuner helps adjust memory allocations based on actual performance:

wget mysqltuner.pl

1	wget mysqltuner.pl

Percona Toolkit which can be downloaded from here:

http://www.percona.com/software/percona-toolkit

At this point you’re done! You have a working database cluster connected to your web application. Questions and comments are welcomed.

RackSpace Cloud Networks Bug and Interesting Notes

By leerb on October 6, 2013 in Cloud Networks, Cloud Servers, RackSpace

There’s a bug in the automation for provisioning RackSpace Cloud Servers with Cloud Networks from the control panel. If you setup a server with more then one Cloud Network, the networks get assigned to the wrong interfaces. This is not a problem if all the servers you’re connecting to the Cloud Network are created from the control panel because they will all be consistently wrong. But if some of your server are created from the API then you’ll have a problem because the API automation assigns the networks correctly.

The fix, until RackSpace corrects the bug, is to manually correct the address assignments on the control panel generated cloud servers.

While troubleshooting this issue, RackSpace support pointed out a couple of interesting commands:

To see what your networking should be, you can run this:

xenstore-ls vm-data/networking

1	xenstore-ls vm-data/networking

And to view a specific interface run:

xenstore-read vm-data/networking/BC764E051479 | python -mjson.tool

1	xenstore-read vm-data/networking/BC764E051479 \| python -mjson.tool

This will show the config that should be assigned to a particular MAC address.

We love Cloud Networks but hate bugs in automation as we wasted an hour tracking this down!

Need help immediately?

$100/hour
Satisfaction Guaranteed

Top Nav

Navigation

Archive | Cloud Servers

Scalr 4.5 Install Notes For Cento 6.4

Create UNIX Users and Group for Scalr

Configure your firewall

Install Scalr’s PHP Dependencies

Install Scalr’s Python Dependencies

Configure PHP for Scalr

Update System SNMP MIBs

Download and Install Scalr 4.5

Install and Configure MySQL

Create the Scalr Cache folder

Install and Configure rrdtool and rrdcached

Install and Configure Apache

Configure Scalr

Configure the Scalr Cronjobs

Configure the Scalr Daemons

Validate your Scalr installation

Log in to Scalr

All Done!

Scale Out For SharkTank

Percona XtraDB Cluster On CentOS

RackSpace Cloud Networks Bug and Interesting Notes