Statement following the two incidents on the 9th of November 2017

Last Thursday, the 9th of November two of our sites were impacted by two simultaneous but unrelated incidents: a power failure in Strasbourg (SBG) and a software bug on our optical equipment in Roubaix (RBX).

This had some direct consequences on other services. For example, it was especially difficult for our customers to receive emails.

All our teams were summoned immediately. They worked non-stop to resolve the situation as quickly as possible, while also communicating transparently, as soon as the incident started, via the voice of Founder and CEO, Octave Klaba. Customers were informed of the development of the situation in real time, via Twitter and then via status updates (travaux.ovh.net).

Thanks to our teams’ intervention we could get the Roubaix site up and running again in under two and a half hours and the Strasbourg site in under four hours. Today, our services have fully returned to normal. We have now activated the compensation package.

OVH’s technical and commercial teams are still fully mobilised to support our customers, to facilitate their understanding, and to answer their questions.

We have of course learnt some lessons from this exceptional situation. We will be making some changes as a consequence. An action plan has been decided upon, as follows:

* In Roubaix: we have fast tracked the project to redistribute the optical multiplexing system on two separate systems, to limit the scope of a failure. At the same time, we are also conducting investigations with the equipment manufacturer to fix the bug by upgrading the incriminating software.

* In Strasbourg: we have reconsidered the electrical design of our site and decided to once and for all stop using shipping containers, a technology used in datacentres SGB 1 and 4. Instead, we will make way for “fixed”, large capacity datacentres, designed for large scale and maximum reliability. The servers in SBG 1 and 4 will be migrated to the SBG3 datacentre, which is under construction, to ensure that every aspect of the site meets the same standards as all the company’s other sites.

* On all our 12 sites: a preventive audit of the electrical systems is underway, to ensure that they fully comply with internal quality standards.

Our objective is to deliver the technical standards and excellent service level that our customers expect, each day, anywhere in the world. We are all aware of how important our services are to our customers. We would once again like to offer our most sincere apologies for these incidents and the damage they may have caused.

Our job is to ensure that the services you trust us to host are protected in all circumstances. Even the most unlikely. It is also our job to learn the necessary lessons when the measures we have put into place do not work as they should. The 9th of November incidents have caused us to further perfect our infrastructure, to take care of each detail and to push our standards above and beyond those of the industry. Every day we must earn your trust. And even more so in these types of situations.