Following the two incidents on the 9th of November 2017
Last Thursday, the 9th of November two of our sites were impacted by two simultaneous but unrelated incidents: a power failure in Strasbourg (SBG) and a software bug on our optical equipment in Roubaix (RBX). This had some direct consequences on other services. For example, it was especially difficult for our customers to receive emails. All our teams were summoned immediately. They worked non-stop to resolve the situation as quickly as possible, while also communicating transparently, as soon as the incident started, via the voice of Founder and CEO, Octave Klaba. Customers were informed of the development of the situation in real time, via Twitter and then via status updates (travaux.ovh.net). Thanks to our teams’ intervention we could get the Roubaix site up and running again in under two and a half hours and the Strasbourg site in under four hours. Apart from some customers hosted in Strasbourg, whose services are still being rebooted, our services have now returned to normal. We have now activated the compensation package. OVH’s technical and commercial teams are still fully mobilized to support our customers, to facilitate their understanding, and to answer their questions. We have of course learnt some lessons from this exceptional situation. We will be making some changes as a consequence. An action plan has been decided upon, as follows:
- In Roubaix: we have fast tracked the project to redistribute the optical multiplexing system on two separate systems, to limit the scope of a failure. At the same time, we are also conducting investigations with the equipment manufacturer to fix the bug by upgrading the incriminating software.
- In Strasbourg: we have reconsidered the electrical design of our site and decided to once and for all stop using shipping containers, a technology used in datacentres SGB 1 and 4. Instead, we will make way for “fixed”, large capacity datacentres, designed for large scale and maximum reliability. The servers in SBG 1 and 4 will be migrated to the SBG3 datacentre, which is under construction, to ensure that every aspect of the site meets the same standards as all the company’s other sites.
- On all our 12 sites: a preventive audit of the electrical systems is underway, to ensure that they fully comply with internal quality standards.