Top Nav

Archive | Cloud

Networking Broken On EC2 Instance Created From Snapshot

Situations have been observed where the network interface will fail to start on an AWS EC2 instance created from a snapshot of another instance. Since AWS EC2 lacks a facility to access the console, it’s not possible to login and fix this condition. One way to solve the problem is to:

  1. Stop the new instance.
  2. Detach root volume and attach to alternate server.
  3. From the alternate server, mount the root volume and edit the network config.
  4. Unmount the root volume.
  5. Make a snapshot of the root volume.
  6. Make an AMI image from the snapshot.
  7. Create a new EC2 instance from the AMI.

This will work but it takes significant effort and you have to have an available alternate server to mount and fix the root volume.

An alternate approach is to use the “User data” feature when creating the EC2 instance to inject a script that fixes the network config. For example let’s assume that we have a CentOS based AMI image named “host1-backup” which we wish to use create a new EC2 instance. The network config in the image has the MAC address explicitly specified in “/etc/sysconfig/network-scripts/ifcfg-ens5”. When the AMI boots, the specified MAC address does not match the interface and the new instance fails to start networking. This can be resolved as follows:

a. Launch a new EC2 using the “host1-backup” AMI from the AWS console. On “Step 3: Configure instance details”, in the “Advanced details” section enter the following:

b. When the instance starts the user data script will run and remove the HWADDR line from the interface control file.

c. Reboot the instance to allow the modified network config to activate.

There are other cases where this approach might be useful. for example a broken iptables / firewall config could be disabled from user data script.

Note that the “user data” script runs late in the boot process so it can not be used to fix problem like a missing volume from /etc/fstab or a corrupt boot loader.


WordFence / CloudFront – Automatically Update Trusted Proxies

If you are using WordPress with CloudFront and WordFence then some extra configuration is required. WordFence does blocking based on IP address but it will fail to determine the correct IP address when you have CloudFront and an Elastic Load Balancer in front of the site. The work around is to setup a cronjob that updates the list of trusted proxies in WordFence.

  1. Login to WordPress admin and to to WordFence -> All Options. Under “How does Wordfence get IPs” select “Use the X-Forwarded-For HTTP header”.  Click “Save Changes”
  2. Add a cronjob using the script shown below to update the list of trusted proxies.

Here’s a simple script for the cronjob:





Downloading RackSpace CloudFiles Container with Swiftly

Swiftly is a handy utility for managed RackSpace CloudFiles containers from the Linux command line. You can easily download an entire container with a simple command line. Here’s the project page on Github:

And here’s the documentation:

Installation through a package manager is as follows:


  1. Update the apt-get database.
  2. Install the Python installer, pip, using apt-get.
  3. Install Swiftly using pip.


  1. Install the Python installer, pip, using yum.
  2. Install swiftly using pip.

After installation you can start transferring files. Here’s an example:

The entire [container] will be downloaded to [destfolder].

Check the Swiftly documentation for a wide range of options. You can select with precision which files to download.


AWS Resize Partition

After expanding the size of an EBS volume on an AWS EC2 instance, you’ll need to expand the partition and the filesystem. Here are the steps:

  1. Confirm available storage with “lsblk” command.
  2. Expand the partition with “growpart” command like “growpart /dev/xvdi 1”. Of course you’ll need to change the device name to match your system. The partition number on the end will be “1” if there is only one partition on the device or can be changed to select a different partition. “growpart” is in the “cloud-guest-utils” package if it’s now already installed. Note that if you have a large  (> 2TB) partition) created with parted then “growpart” may not work. Instead use the “resize” command in “parted”.
  3. Confirm new partition size with “lsblk” command.
  4. If using LVM then expand the physical and logical volumes:
    1. Reread partitions with “partprobe”
    2. Resize physical volume with “pvresize /dev/xvdi1”
    3. Expand logical volume with “lvextend -l +100%FREE /dev/vg_data2/lv_data2”
  5. Resize the filesystem to fill the expanded partition. Command will depend on the filesystem type:
    • ext2/3/4 – “resize2fs /dev/xvdi1”
    • xfs – “xfs_growfs /dev/vg_data2/lv_data2”

Now you should have the expanded storage available for use.

Here’s the man pages for these commands:



AWS ELB Subnet Selection

If you have an AWS VPC with public and private subnets, it’s important to remember to select the public subnets when creating an Elastic Load Balancer. When the public subnets are chosen, replies from instances behind the load balancer are returned through the load balancer. If you instead select the private subnets then reply traffic is routed via the routing table for the private subnet. This results in asymmetric routing which can create a range of problems.