|T O P I C R E V I E W
||Posted - Oct 24 2006 : 8:13:37 PM
It was ugly: NordicSkiRacer.com was down from Wednesday Oct 18 around 9:00pm (+/- 30 minutes) and didn't come back up until Saturday, Oct 21 at around 2:30pm.
This was obviously a really severe problem at my web hosting company, 1and1.com. I'm still trying to get an explanation of what happened. For a hosting firm that is supposed to have tons of redundancy, something went wrong.
Fortunately, when they restored NordicSkiRacer.com back from tape, everything was there - I didn't need to repair any pages.
Apologies to all...
|2 L A T E S T R E P L I E S (Newest First)
||Posted - Oct 24 2006 : 8:36:05 PM
Another message from 1and1: Some technical details!
Dear Michael Muha,
As you are probably aware, 1&1 experienced a major problem with the server your website is hosted on in the last few days.
I am writing you today to personally apologize for the prolonged downtime which you experienced. Please allow me to explain what happened and what measures we have already added to prevent something like this from happening again in the future.
- On Thursday 10/18/06 the RAID system of the server had a major filesystem problem. Unfortunately, all attempts to fix the filesystem failed thus requiring a complete reinstall of the operating system. At no time was any customer data lost or jeopardized.
- Reinstalling the machine and restoring the configuration from the systemstate-backup went very quickly and without any problems. However, when attempting to restore the Internet Information Server (IIS) configuration file (MetaBase), our engineers encountered a major problem.
- This was raised to a Priority A Case to Microsoft who agreed with our engineers' assessment that the only remedy would be reinstalling IIS and then re-creating each configuration for every site. This, unfortunately, is a very slow process which ended up taking until early Monday morning.
We have now analyzed the issue and devised two steps which will enable us to restore the MetaBase within a very short period of time if the need ever arises again. We will be running a full trial of this new system next week to verify the concept and integrate it in our recovery plan.
Additionally, we made changes to our communication and reaction policies to make sure you will be better informed in the future.
We thank you for your understanding and your patience. If you have any questions, please feel free to contact us anytime.
For those who don't know, RAID is a technology where data is spread across multiple hard drive in a way such that if one hard drive fails, the data is still protected. It's unusual for a RAID system to fail.
||Posted - Oct 24 2006 : 8:15:12 PM
E-mail from 1and1:
We are currently aware of the issues that you've been experiencing. I can only apologize for the inconvenience that you've suffered thus far.
The server that you are underneath has been experiencing problems that are both known and evident for a select portion of our Microsoft shared hosting customers, including yourself. Our developers and administrators have been working extremely hard to resolve this matter.
In addition, we recognize that further action would need to be taken in a situation such as this one. The heads of our customer service and technical support teams are working on a response and course of action that will be sent to all customers affected by this burden. The email that you'll receive in a short period of time will apologize further, along with the chosen path to compensate customers who have been feeling the frustration caused by this technical concern. Please understand that we know the severity of the dilemma. It is our number one priority to resolve this matter in an amicable and expeditious way. We're approaching this from both a technical and customer service stand point.
Your patience will be greatly appreciated in this matter. We will be contacting you soon with details of what has happened and what is happening. Again, we apologize for the inconvenience and any possible disruption of service. Thank you and you will be hearing from us soon.
If you have any further questions please do not hesitate to contact us.