Recently, one of my websites went down. After noticing, I checked my EC2 dashboard and saw the instance stopped. AWS had emailed me to say that due to a physical hardware issue, it was terminated. When an instance is terminated, all of its data is lost. Luckily, all of my data is backed up automatically every night.
Since I don’t use RDS, I have to manually manage data redundancy. After a few disasters, I came up with a solution to handle it. I trigger a nightly cron-job to run a shell script. That script takes a MySQL dump and uploads it to S3.
As long as I have the user generated data, everything else is replaceable. The website that went down is a fitness tracking app. Every day users record their martial arts progress. Below are the ten steps taken to bring everything back up.
- Launch a new EC2 instance
- Configure the security group for that instance – I can just use the existing one
- Install Apache and MariaDB
- Secure the database
- Install PhpMyAdmin – I use this tool to import the .sql file in the next step
- Import the database backup – I downloaded the nightly .sql file dump from my S3 repo
- Setup automatic backups, again
- Install WordPress and restore the site’s blog
- Configure Route 53 (domain name) and SSL (https) – make the website live again
- Quality Assurance – “smoke test” everything to make sure it all looks correct
Use this as a checklist to make sure you don’t forget any steps. Read through the blog posts that I hyperlinked to get more details.