As the OVHCloud fire demonstrates, never assume your business data is safe, and always back it up offsite.
On March 10th, 2021, OVHcloud, a colocation, hosting, and data center provider in France, experienced a devastating fire that destroyed one of its facilities and damaged another. Physical data center buildings often have multiple data centers within them, with separate power, HVAC, and environmental capabilities for each. OVHcloud’s Strasbourg SBG2 burned “to the ground,” and the fire subsequently damaged neighboring SBG1 as well. Ironically, the OVHcloud facilities in Strasbourg sit on a small peninsula on the Rhine River, with water roughly 30 meters away.
With no official cause yet released, officials speculate the fire started with an uninterruptible power supply (UPS) that serviced SBG2. UPSes are large banks of batteries that store power in case of a power outage. Batteries are not exempt from flaws and can overheat, short, and cause fires. Workers performed maintenance to the UPS in question the day before the fire, which is where the investigation began.
Typically, data centers have extensive fire suppression and ventilation systems, so it is unclear why SBG2 received such extensive damage. The shocking part is the lack of redundancy, backups, or continuity plans for customers, and OVHcloud allegedly charges customers a small monthly fee for providing certain types of backups. Many of these backups were still on-premises and destroyed in the fire, along with many cloud application providers’ and cloud companies’ production systems. OVH recommended customers activate their disaster recovery plans, as SBG1-4 services all went offline.
-Photo tweeted by SDIS_67, Bas-Rhin fire service
Ten days later, the majority, if not all, of OVHcloud’s Strasbourg resources were still offline and OVHcloud founder Octave Klaba tweeted that they were not attempting to restart SBG1 and were relocating resources. At that time, SBG3 and 4 restarts were ongoing.
On April 21, Klaba tweeted again: (quote markup in Divi)
VPS SBG2 > SBG3
– 19% had a paid backup, are UP now
– 51% didn’t have paid backup, are UP now
Work in Progress (WIP):
– 29% no paid backup : need additional server to deliver them.
– paid backup: 0.5% restoration failed, we fix them + 0.5% backup with 6%
To understand how this relates to businesses, if you assume your cloud provider is protecting your data by default, you introduce risk. For example, Microsoft provides a shared responsibility model that states you, the customer, are responsible to back up your Office 365 data, while Microsoft takes care of the underlying infrastructure. In the case of OVH, they could not protect the infrastructure. If your company does not have a method to get data offsite, your business is at risk no matter where the data lies.
In whatever location your production workloads run, it is critical to take regular backups of your applications and data and send them to another location. In fact, for rock-solid data protection and disaster recovery, go back to the basics with the 3-2-1 Backup Rule.
How Can You Protect Your Data?
Disaster can strike at any time, whether natural, artificial, or other. Businesses should back up data and send the backups offsite as frequently as possible to reduce risk and lower recovery time- and recovery point objectives. If you run production workloads in a cloud environment and keep backups in the same environment, you are not effectively getting that data “offsite.” Additionally, by replicating workloads to a disaster recovery environment, you can power them on if something happens to the production workloads. You should also test backups and DR sites regularly to ensure recoverability.
Global Data Vault provides offsite data protection, backup as a service (BaaS), disaster recovery as a service (DRaaS), and backup for Microsoft Office 365. We specialize in helping customers with disaster recovery needs. The integrity of your data, and the service we provide to you, come first.