Top Nav

Archive | Amazon AWS

Networking Broken On EC2 Instance Created From Snapshot

Situations have been observed where the network interface will fail to start on an AWS EC2 instance created from a snapshot of another instance. Since AWS EC2 lacks a facility to access the console, it’s not possible to login and fix this condition. One way to solve the problem is to:

  1. Stop the new instance.
  2. Detach root volume and attach to alternate server.
  3. From the alternate server, mount the root volume and edit the network config.
  4. Unmount the root volume.
  5. Make a snapshot of the root volume.
  6. Make an AMI image from the snapshot.
  7. Create a new EC2 instance from the AMI.

This will work but it takes significant effort and you have to have an available alternate server to mount and fix the root volume.

An alternate approach is to use the “User data” feature when creating the EC2 instance to inject a script that fixes the network config. For example let’s assume that we have a CentOS based AMI image named “host1-backup” which we wish to use create a new EC2 instance. The network config in the image has the MAC address explicitly specified in “/etc/sysconfig/network-scripts/ifcfg-ens5”. When the AMI boots, the specified MAC address does not match the interface and the new instance fails to start networking. This can be resolved as follows:

a. Launch a new EC2 using the “host1-backup” AMI from the AWS console. On “Step 3: Configure instance details”, in the “Advanced details” section enter the following:

b. When the instance starts the user data script will run and remove the HWADDR line from the interface control file.

c. Reboot the instance to allow the modified network config to activate.

There are other cases where this approach might be useful. for example a broken iptables / firewall config could be disabled from user data script.

Note that the “user data” script runs late in the boot process so it can not be used to fix problem like a missing volume from /etc/fstab or a corrupt boot loader.


WordFence / CloudFront – Automatically Update Trusted Proxies

If you are using WordPress with CloudFront and WordFence then some extra configuration is required. WordFence does blocking based on IP address but it will fail to determine the correct IP address when you have CloudFront and an Elastic Load Balancer in front of the site. The work around is to setup a cronjob that updates the list of trusted proxies in WordFence.

  1. Login to WordPress admin and to to WordFence -> All Options. Under “How does Wordfence get IPs” select “Use the X-Forwarded-For HTTP header”.  Click “Save Changes”
  2. Add a cronjob using the script shown below to update the list of trusted proxies.

Here’s a simple script for the cronjob:





AWS Resize Partition

After expanding the size of an EBS volume on an AWS EC2 instance, you’ll need to expand the partition and the filesystem. Here are the steps:

  1. Confirm available storage with “lsblk” command.
  2. Expand the partition with “growpart” command like “growpart /dev/xvdi 1”. Of course you’ll need to change the device name to match your system. The partition number on the end will be “1” if there is only one partition on the device or can be changed to select a different partition. “growpart” is in the “cloud-guest-utils” package if it’s now already installed. Note that if you have a large  (> 2TB) partition) created with parted then “growpart” may not work. Instead use the “resize” command in “parted”.
  3. Confirm new partition size with “lsblk” command.
  4. If using LVM then expand the physical and logical volumes:
    1. Reread partitions with “partprobe”
    2. Resize physical volume with “pvresize /dev/xvdi1”
    3. Expand logical volume with “lvextend -l +100%FREE /dev/vg_data2/lv_data2”
  5. Resize the filesystem to fill the expanded partition. Command will depend on the filesystem type:
    • ext2/3/4 – “resize2fs /dev/xvdi1”
    • xfs – “xfs_growfs /dev/vg_data2/lv_data2”

Now you should have the expanded storage available for use.

Here’s the man pages for these commands:



AWS ELB Subnet Selection

If you have an AWS VPC with public and private subnets, it’s important to remember to select the public subnets when creating an Elastic Load Balancer. When the public subnets are chosen, replies from instances behind the load balancer are returned through the load balancer. If you instead select the private subnets then reply traffic is routed via the routing table for the private subnet. This results in asymmetric routing which can create a range of problems.


Sync MySQL Slave From Another Slave On Amazon AWS

I was looking for a procedure to setup a new MySQL replication slave on Amazon AWS without having to do a MySQL dump of the master. Ended up using a couple of AWS features to make this an easy process. Let’s assume that we have a master server (master01) and a slave (slave01). We want to create a new slave (slave02). We’ll do this without taking the master offline and without doing a MySQL dump. This is especially important with large databases.

A couple of notes before we start:

  • On AWS we put MySQL storage on it’s own EBS storage volume. We create a new empty volume, partition, add LVM and then format with xfs. This makes it easy to add additional storage in the future.
  • We prefer to use the innodb-file-per-table setting in my.cnf to force MySQL to use a separate file for each InnoDB table. This keeps all the files that belong to a database in a common folder instead of mixing multiple databases in a single file. Also in may help avoid having very large database files.
  • We’re assuming that you’ve already setup the slave02 EC2 instance. You might do this by cloning the server01 instance.

Now here’s my procedure:

1. Login to AWS and go to EC2 -> Volumes. Find the volume holding the MySQL data. If you’ve not already done so we would recommend adding a Name tag to the volume so that you can easily see which server/function it is associated with. For example you might name it “slave01-mysql”. Right click on the volume and select “Create Snapshot”. A dialog will open where you can name the snapshot … something like “slave01-mysql-snap-1” would be appropriate. Depending on the size of the volume, this snapshot will take some time to complete.

One of the greatest features of EBS snapshots is their incremental nature. Here’s how it’s described by Amazon:

Amazon EBS snapshots are incremental backups, meaning that only the blocks on the device that have changed since your last snapshot will be saved. If you have a device with 100 GBs of data, but only 5 GBs of data has changed since your last snapshot, only the 5 additional GBs of snapshot data will be stored back to Amazon S3.

So the first snapshot we take of a volume has to copy the entire volume but additional snapshots only copy changed blocks. We’re going to use this is next step. At this point we have a snapshot of the server01 slave but it is inconsistent and we don’t have the point-in-time log coordinates that we need to initialize a new slave.

2. Login to slave01, stop replication and apply a read lock as follows:

Make sure that the “flush tables …” completes before proceeding. It may take a few seconds or more if there are locked tables that have to be released before the lock can be completed. At this point the slave server is not fully functional. You’ll want to proceed quickly through the next steps.  Until the next step (3) is complete, do not close the terminal with the “flush tables …” as we need to keep the read lock active.

3. Open a second terminal session to slave01, record log coordinates and shutdown the mysql process as follows:

The slave status will look something like:


The columns that we need to record are:



Take note of these values for later use.

The slave01 database is now stopped and we have the log coordinates from right before the shutdown.

4. Next in AWS, start a new snapshot of the slave01-mysql volume. Name it slave01-mysql-snap-2 or something similar. This snapshot will run much faster since it is only copying blocks that changed since the first snapshot.

5. When the second snapshot is complete, start the mysql process on the slave01 server. Check replication and confirm that it comes back in sync with the master. The slave01 is now back to a functional status.

6. In AWS, right click on the slave01-mysql-snap-2 snapshot and select “Create volume from snapshot”. Name the new volume something like slave02-mysql.

Another great feature of EBS snapshots is “lazy loading” as described here:

New volumes created from existing Amazon S3 snapshots load lazily in the background. This means that once a volume is created from a snapshot, there is no need to wait for all of the data to transfer from Amazon S3 to your Amazon EBS volume before your attached instance can start accessing the volume and all of its data. If your instance accesses a piece of data which hasn’t yet been loaded, the volume will immediately download the requested data from Amazon S3, and then will continue loading the rest of the volume’s data in the background.

This means that we can proceed with the new slave setup without waiting for the new volume to be loaded … instead it will load as needed.

7. In AWS, right click on the newly created volume and attach it to the new slave02 instance.

8. Login to the slave02 instance, confirm that mysql is stopped and mount the new volume.

9. Edit /etc/my.conf and add “skip-slave-start” to the “[mysqld]” section.

10. In the mysql data directory (/var/lib/mysql by default), remove the following files:

  • relay logs  (mysqld-relay-bin.*)

11. Start mysql server

12. Set replication config:

13. Now start replication:

It may take a few minutes or more for replication on the new slave to catch up with the master. This is due to the “lazy load” on the volume created from the snapshot and the need to process the logs from the master.

Once the new slave catches up with the master then the task is complete and you have a new slave. This procedure can work with very large databases. There is no downtime on the master for this process. The existing slave is down for only a short period.