Over the past few days since the power failure in the hosting comapnies data centre in Leeds I have been monitoring all of my clients web sites especially those that rely on MySQL databases as well as my own and everything appears to be stable although some sites may be slow to load, due to the massive amount of traffic flowing in and out of the data centre in the aftermath of this event.

From the information provided by the hosting company responsible for the data centre the power failure was caused by a fault in a voltage sensing device which triggered an alarm and the power system then shut down power in order to protect the system and then the UPS power supply kicked in 9 seconds later and power was restored.

The problem is that computers don’t like it when you shut off the power supply, and when you switch the power back on they don’t just carry on as if nothing happened, someone has to push the power button and hard reboot the machine, in this case the system admins had to hard reboot 15.000 servers, which has to be done in a certain order and is not a trivial task by any means.

It is now 5 days since the power failure and there are still many servers still down and customers sites offline and the system admins are still working to get them online, which clearly demonstrates the gravity of this event and its effect, fortunately all my sites and my clients were back online within hours, so its impact was minimal in that respect, I can only apologise for any inconvenience this caused. 

Yesterday I got a few calls from clients reporting that they could not send or recieve emails and that their web sites were down, when I checked I found this site was offline as was my hosting store Source Host and the system status page was also offline along with all of my clients web sites every site I tried to browse returned a 404 error.

When I checked the hosting company’s own system status page it stated that they had suffered a power failure mid afternoon within the data centre which had the effect of shutting down all the machines in one of the halls within the data centre, the system administrators were quickly on hand to start working restarting all the servers resolving any issues.

By late afternoon the admins had got the mail servers up and running and email traffic was begining to flow, I started to recieve emails sent 2 to 3 hours earlier, fortunately no email data had been lost during the time the mail servers went dark, but the webmail interface has been disabled while they rebuild the servers from backups.

I spent the rest of the evening monitoring the situation keeping an eye on the system status page and the companies FaceBook page for updates and slowly my clients sites started to come online as the admins continued working to restart all the servers and get everything back online, and get the system stable again as quickly as possible.

After a while all my clients sites that do not rely on a MySQL database started to come back online slowly mine included, but all the sites that use MySQL databases were still returning 404 or 503 errors, but after a couple more hours the Cube Cart stores came online and then all the WordPress sites, it was 10:30pm before all of my clients sites and my own were back online much to everyones relief.

Hopefully things will become more stable over the next few hours as the system admins get everything under control again.

I apologise for any inconvenience that this may have caused.

Over the past few weeks clients had had issues sending emails to Sky, Yahoo, BT and AOL email addresses, when an email was sent from a domain email address/account to any of these free service email accounts it would result in a bounce message being returned to the sender.

Over the past couple of weeks the system administrators within the datacentre have been working to resolve the issue, it seems that there was a problem with how the email traffic was being routed within the datacentres network to and from the mail servers.

I am glad to anounce that the system administrators in the datacentre have now resolved this issue.

Over the past few weeks a few clients have reported problems when sending emails to Sky, Yahoo, BT and or AOL email addresses.

Sending an email from a domain email account to one of these free email accounts usually results in a bounce message being returned to the sender of the message.

It appears that this issue started a couple of weeks ago shortly after the datacentre suffered a massive DDOS (Distributed Denial Of Service) attack, at first the problem was intermittent and had little effect, but the problem slowly got worse.

The system administrators in the datacentre are now working to resolve the issue.