disaster recovery – Ant Pace Blog

The Challenge

When the client reached out, he was understandably frustrated. His website, which was supposed to be active and processing sign-ups, was inactive, displaying a message typically associated with unconfigured or newly registered domains. This was particularly perplexing as the site had been running smoothly just days before.

My first step was to verify the DNS settings. A parked domain page usually indicates issues with domain pointing, possibly due to changes in DNS or hosting services. The client told me he had tried to create a subdomain through his domain registrar, GoDaddy. It turned out that his domain’s name servers pointed to his web hosting company, DreamHost. Making DNS changes directly in GoDaddy led to a change in the domain’s nameserver records so that it became managed by GoDaddy directly, removing any reference to the web hosting package where the site actually lived.

The Solution

After identifying the mismatch, the next step was clear but not simple—access the DreamHost account to find the correct nameserver value to point the domain to. However, the client didn’t have the DreamHost login credentials readily available, which added another layer to our challenge.

Using SecurityTrails, I conducted a DNS history check. This not only confirmed that the name servers were indeed pointing to DreamHost but also provided a clear historical view of the DNS changes over time. I was able to copy and paste the name server values from the history log. Subsequently, we were able to recover access to the DreamHost account by going through account recovery processes, which involved verifying the client’s identity and ownership of the domain.

Once we regained access, I immediately updated the DNS records to ensure they correctly pointed to the client’s active web server. This change effectively removed the parked page error, restoring the site’s functionality.

Conclusion

This scenario underscored the critical importance of keeping domain and hosting details organized and accessible. It also highlighted how easy it is for essential information to become overlooked, especially when multiple service providers are involved over the years. Effective disaster recovery strategies are crucial for quickly restoring operations in the event of a service disruption, whether caused by human error, technical failures, or external threats.

With the website back up and running, and the domain correctly configured, the client was relieved and could focus on his business again. For me, it reinforced a key aspect of my work: solving complex tech puzzles not only requires technical know-how but also an investigative approach to untangle the often convoluted web of digital assets. My role transcends building websites—it’s about ensuring they continue to serve their purpose, even when unexpected disruptions occur.

While my tagline says, “I can build your website,” days like these remind me it should also include, “and I can rescue it too.” This experience serves as a testament to the value of professional web management, especially in a world where digital presence is synonymous with business viability.

A crashed database is a problem I’ve encountered across multiple WordPress websites. When trying to load the site you’re faced with a dreaded “Error establishing a database connection” message. Restarting the DB service usually clears things up. But, sometimes it won’t restart at all – which is why I started automating nightly data dumps to an S3 bucket.

Recently, one particular site kept going down unusually often. I assumed it was happening due to low computing resources on the EC2 t3.micro instance. I decide to spin up a a new box with more RAM (t3.small) and migrate the entire WordPress setup.

Since I couldn’t be sure of what was causing the issue, I needed a way to monitor the health of my WordPress websites. I decided to write code that would periodically ping the site, and if it is down send an email alert and attempt to restart the database.

The first challenge was determining the status of the database. Even if it crashed, my site would still return a 200 OK response. I figured I could use cURL to get the homepage content, and then strip out any HTML tags to check the text output. If the text did match the error message, I could take further action.

Next, I needed to programmatically restart MySql. This is the command I run to do it manually: sudo service mariadb restart

After doing some research, I found that I could use shell_exec() to run it from my PHP code. Unfortunately, Apache wouldn’t let the (non-password using) web server user execute that without special authorization. I moved that command to its own restart-db.sh file, and allowed my code to run it by adding this to the visudo file: apache ALL=NOPASSWD: /var/www/html/restart-db.sh

My visudo file was located at /usr/sbin/visudo. It is a tool found on most Linux systems to safely update the /etc/sudoers file, which is the configuration file for the sudo command. To edit this file, I don’t open it in vim like I would with other editable files. Instead, I run the file as its own command: sudo visudo. Once it is open, you can press the i key to enter “insert” mode. It is considered “safe” because it edits the sudoers file following a strict procedure.

I also needed to make the file executable by adjusting permissions: sudo chmod +x /var/www/html/restart-db.sh

Once those pieces were configured, my code would work:

<?php

$url = "https://www.antpace.com/blog/";
$curl_connection = curl_init();

curl_setopt($curl_connection, CURLOPT_URL, $url);

curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
$curl_response = curl_exec($curl_connection);
$plain_text = strip_tags($curl_response);

if(strpos($plain_text, "Error establishing a database connection") !== false){
	echo "The DB is down.";
        
        //restart the database
        shell_exec('sudo /var/www/html/restart-db.sh');
        
        //send notification email
        include 'send-email.php';
        send_email();
}else{
	echo "The DB is healthy.";
}

?>

The contents of restart-db.sh looks like this:

#!/bin/bash

sudo service mariadb restart

Create the cron job

A cron job is a scheduled task in Linux that runs at set times. For my PHP code to effectively monitor the health of the database, it needs to run often. I decided to execute it every five minutes. Below are three shell commands to create a cron job.

The first creates the cron file for the root user:

sudo touch /var/spool/cron/root

The next appends my cron command to that file:

echo "*/5 * * * * sudo wget -q https://www.antpace.com/check-db-health.php" | sudo tee -a /var/spool/cron/root

And, the last sets the cron software to listen for that file:

sudo crontab /var/spool/cron/root

Alternatively, you can create, edit, and set the cron file directly by running sudo crontab -e . The contents of the cron file can be confirmed by running sudo crontab -l .

Pro-tip: If your WordPress site does continually crash, you probably do need to upgrade to an instance with more RAM. Also, consider using RDS for the database.

Update

I previously used the localhost loop back address in my cron file: */5 * * * * sudo wget -q 127.0.0.1/check-db-health.php. After setting up 301 redirects to prevent traffic from hitting my public DNS, that stopped working. It is more reliable to use an explicit domain name URL: */5 * * * * sudo wget -q https://www.antpace.com/check-db-health.php

Tag: disaster recovery

How to Fix a GoDaddy Parked Page and Revive Your Website

The Challenge

The Solution

Conclusion

Error establishing connection to database – WordPress solution

Create the cron job

Update