How To Automate EC2 Instance Backup Step

How To Automate EC2 Instance Backup Step-By-Step

Different Ways to Take EC2 Instance Backup

There are two ways to implement backup of your EC2 instances on AWS:

·         If your instance is EBS backed, you can create the snapshots of the EBS volume.

·         You can create an AMI of your instances as a backup solution.

Both the EBS snapshot and the AMI are automatically stored on Amazon S3 which is known for being highly durable and reliable. 

Manual EC2 Instance Backup by an EBS Snapshot

Before showing how to automate your instance backup, let's first look at the different manual steps we should perform.

Follow the steps below to implement a backup for an EC2 EBS volume:

·         OpenAWS Console.

·         Click the "Instances" section in AWS console under the EC2 dashboard.

·         Select the instance for which you want to create a backup.

·         Under the description tab for that instance you can see the details of that instance which will also show you the block devices. Clicking on a block device will show

·         the volume ID.

·         Click on EBS ID (volume ID), which will take you to the volume section under the EC2 dashboard. To create the snapshot click on the action button and select "Create Snapshot" option.

·         Enter a name and description for the snapshot in the snapshot dialog as below. After creating a snapshot you can see your snapshot in the snapshot section under the EC2 dashboard.

·         To restore the volume from a snapshot

·         Search for the snapshot under the snapshot section using its description.

·         Right click and select the "Create Volume" option.

·         Fill in the required details in the Create Volume dialog box and click the "Create" option.

·         A volume with the same snapshot will be created and then you can attach the new volume to an EC2 Instance for further use.

Manual Backup Using AMI

This solution can be used if your instance is not EBS backed up. To create the EC2 Instance backup, you can create the AMI of an Instance. Furthermore, by using the same AMI you can launch an instance in the same state.

·         Search for the Instance for which you want to create the AMI. Select the "Create Image" option underImage after clicking on the action button.

·         Follow the steps below once you have selected the "Create Image" option:

o    Enter the AMI details in the "Create Image" dialog box as below.

o    Enter the AMI Name and Description.

o    If you want your instance to stay in a running state without restarting while creating the AMI, then check the "No Reboot" option.

o    You can also customize the size and other options like "delete on termination". The "delete on termination" option will delete the volume if the instance is terminated, otherwise it will still be in an available state if the EC2 instance is terminated.

Automating the EC2 Backup

To automate the EC2 Backup, you will need to write a script to automate the above steps by using AWS' API.

Below is the step by step process which should be followed in the script:

·         Get the list of instances.

·         Connect to AWS through API to list the Amazon EBS volumes that are attached locally to the instance.

·         List the snapshots of each volume.

·         Assign a retention period to the snapshot.

·         Create snapshot of each volume.

·         Delete the snapshot if it is older than the retention period.

By using AWS Command Line Interface (AWS CLI) you can write a shell script which will be used for automating the EBS volume backup. It's recommended to install the AWS CLI if it has not already been installed. You can refer to this resource for details:AWS CLI Installation.

Commands to Install AWS CLI

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
./awscli-bundle/install -b ~/bin/aws

After installing AWS CLI, configure it using the aws configure command

aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: ENTER

Creating Shell Script

Use the script below to copy code to snapshot.sh and set it as cron for automatic timely backup. You can find an explanation of script in comments.

#!/bin/bash

# Volume list file will have volume-id:Volume-name format

VOLUMES_LIST = /var/log/volumes-list
SNAPSHOT_INFO = /var/log/snapshot_info
DATE = `date +%Y-%m-%d`
REGION = "eu-west-1"

# Snapshots Retention Period for each volume snapshot
RETENTION=6

SNAP_CREATION = /var/log/snap_creation
SNAP_DELETION = /var/log/snap_deletion

EMAIL_LIST = abc@domain.com

echo "List of Snapshots Creation Status" > $SNAP_CREATION
echo "List of Snapshots Deletion Status" > $SNAP_DELETION

# Check whether the volumes list file is available or not?

if [ -f $VOLUMES_LIST ]; then

# Creating Snapshot for each volume using for loop

for VOL_INFO in `cat $VOLUMES_LIST`
do
# Getting the Volume ID and Volume Name into the Separate Variables.

VOL_ID = `echo $VOL_INFO | awk -F":" '{print $1}'`
VOL_NAME = `echo $VOL_INFO | awk -F":" '{print $2}'`

# Creating the Snapshot of the Volumes with Proper Description.

DESCRIPTION = "${VOL_NAME}_${DATE}"

/usr/local/bin/aws ec2 create-snapshot --volume-id $VOL_ID --description "$DESCRIPTION" --region $REGION &>> $SNAP_CREATION
done
else
echo "Volumes list file is not available : $VOLUMES_LIST Exiting." | mail -s "Snapshots Creation Status" $EMAIL_LIST
exit 1
fi

echo >> $SNAP_CREATION
echo >> $SNAP_CREATION

# Deleting the Snapshots which are 10 days old.

for VOL_INFO in `cat $VOLUMES_LIST`
do

# Getting the Volume ID and Volume Name into the Separate Variables.

VOL_ID = `echo $VOL_INFO | awk -F":" '{print $1}'`
VOL_NAME = `echo $VOL_INFO | awk -F":" '{print $2}'`

# Getting the Snapshot details of each volume.

/usr/local/bin/aws ec2 describe-snapshots --query Snapshots[*].[SnapshotId,VolumeId,Description,StartTime] --output text --filters "Name=status,Values=completed" "Name=volume-id,Values=$VOL_ID" | grep -v "CreateImage" > $SNAPSHOT_INFO

# Snapshots Retention Period Checking and if it crosses delete them.

while read SNAP_INFO
do
SNAP_ID=`echo $SNAP_INFO | awk '{print $1}'`
echo $SNAP_ID
SNAP_DATE=`echo $SNAP_INFO | awk '{print $4}' | awk -F"T" '{print $1}'`
echo $SNAP_DATE

# Getting the no.of days difference between a snapshot and present day.

RETENTION_DIFF = `echo $(($(($(date -d "$DATE" "+%s") - $(date -d "$SNAP_DATE" "+%s"))) / 86400))`
echo $RETENTION_DIFF

# Deleting the Snapshots which are older than the Retention Period

if [ $RETENTION -lt $RETENTION_DIFF ];
then
/usr/local/bin/aws ec2 delete-snapshot --snapshot-id $SNAP_ID --region $REGION --output text> /tmp/snap_del
echo DELETING $SNAP_INFO >> $SNAP_DELETION
fi
done < $SNAPSHOT_INFO
done

echo >> $SNAP_DELETION

# Merging the Snap Creation and Deletion Data

cat $SNAP_CREATION $SNAP_DELETION > /var/log/mail_report

# Sending the mail Update

cat /var/log/mail_report | mail -s "Volume Snapshots Status" $EMAIL_LIST

Follow the steps below for creating and running shell script:

·         Create a script by the name of snapshot.sh using command below.

·         Set it as a cron in crontab.

# Edit Cron File
crontab -e

Hope you liked the article. Taking backup of your infrastructure resources frequently is very important in order to be able to recover from a disaster. It's important to schedule AWS backups on a timely basis, such as taking backup weekly or monthly on different availability zones. It's one of the best practices that is followed by devops teams all over the world.

Automating Instance Backup Using CPM

While using in-house scripts can provide for a basic backup solution, it doesn't make business sense for organizations to invest in a fully-featured in-house backup solution rather than focusing on their business critical tasks.
Cloud Protection Manager (CPM) is an enterprise-class backup-recovery and disaster recovery solution designed for AWS EC2 covering all the essential backup and recovery features to ensure robustness of the backup and DR solution as well as simplifying processes and saving precious devops time. 
CPM is available as a service model that allows users to manage multiple AWS accounts and configure policies and schedules to take automated snapshot backups. It also has a Windows agent to consistently back up Windows applications. CPM allows you to recover a volume from a snapshot, increase its size and switch it with an existing attached volume in a single step.

Furthermore, in a dynamic cloud environment you need to be able to keep consistent backup policy across all your instances at any point in time. To be most effective, your solution needs to be dynamic and automated when a server is terminated and a new instance needs to be launched. Using EC2 tags, CPM can automatically assign each one of these new instances the appropriate backup policy based on their purpose and your initial configuration. For additional information, see our previous article about tag-based continuous AWS cloud backup.