Hypervisor Failure: Restart Your VMs in 5 Minutes, Not 4 Hours

1. What happens when a hypervisor dies

A power supply gives up, a memory module throws an uncorrectable error, a kernel freezes. The physical machine stops, and with it every virtual machine it was hosting. What happens over the following minutes does not depend on the failure — it is identical in every case — but only on what was put in place beforehand.

And the decisive factor is not the one people expect. It is neither storage speed nor bandwidth: it is how long it takes for something, or someone, to notice the problem and decide to act. Everything else follows from that.

The scenario we see most often

The hypervisor dies at 2:40 in the morning. Monitoring raises an alert. With no on-call rota, nobody reads that alert before 8:30. The restore starts at 9:00 and finishes at 11:00. The hardware failure lasted a few seconds; the outage lasted eight hours and twenty minutes, nearly six of which were spent doing nothing at all.

2. Without HA: someone has to notice

Detection time is where recovery time is actually decided

In an infrastructure with no failover mechanism, service restoration begins with a human decision. The delay before that decision is almost always longer than the technical operation itself — and that is where most of the recovery time is spent. A 24/7 on-call rota brings that delay down to tens of minutes; its absence can stretch it to an entire night.

The prerequisite: a backup that is restorable elsewhere

To restart a virtual machine on another node, its backup has to be restorable elsewhere — not only on the original hardware. That is what makes the Proxmox ecosystem interesting here: the backup format is native and neutral, and the very same backup that protects you from a mistake also lets you start again on another server. One mechanism, two uses, and you do not pay twice. All the depth of the subject — retention, immutability, isolation — belongs to backup itself: see NimbusBackup and disaster recovery with Proxmox.

The real duration: one to four hours

Once the decision is made, three factors set the duration: the volume to transfer, the usable restore throughput — rarely the link's, often the storage's — and whether someone is available to watch the operation and start the application behind it.

For an order of magnitude: 500 GB at 200 MB/s usable is around forty minutes of pure transfer. Add preparing the target node, booting the system and starting the application, and you rarely come in under an hour — easily over three for a set of several machines.

What a resolution time SLA changes here

A resolution time commitment puts a contractual bound on that delay, but it does not remove it: it commits a human to intervene within a given window. That is exactly the difference with what follows — high availability does not shorten the intervention, it makes it unnecessary.

3. With HA: nobody needs to notice

When several hypervisors form a cluster, the survivors observe that one of them has gone silent and restart its virtual machines on themselves. The operation takes a few minutes and requires no human decision. Three prerequisites make it possible — and the absence of any one of them makes it dangerous.

1. At least three nodes

The third one does not necessarily host workload: it breaks ties. With three, a majority always exists, so the group that holds it knows it is legitimate. With two, neither can tell a dead neighbour from an unplugged cable. A lightweight witness can stand in for a full third hypervisor.

2. Reachable storage

A virtual machine can only restart elsewhere if its disk is reachable there. That means storage shared between nodes, or replication that keeps a recent copy on them. Without one of the two, there is nothing to restart.

3. A fencing mechanism

Before restarting a machine elsewhere, you need certainty that it is no longer running anywhere. Without that guarantee, two instances can write to the same disk — and there, corruption is real. That is what fencing is for, and a cluster without it is not a cluster, it is a gamble.

The real duration: two to five minutes

That delay breaks down into three parts, none of them human: detecting the absence, which takes tens of seconds so as not to react to a passing network hiccup; fencing and the decision to restart; then booting the system and the application, which is often the longest part. A large database replaying its journal can consume several minutes on its own.

Those 2 to 5 minutes assume the VM boots cleanly

That is an assumption, not a guarantee, and two cases blow it away. A filesystem check triggered by the abrupt shutdown can take long minutes on a large volume — the time depends on size, not on your SLA. And above all, an encrypted disk waiting for a passphrase will never boot on its own: the machine sits blocked until a human types it in, which cancels exactly the benefit you were after at three in the morning.

Both are handled at design time — automated unlocking through a key server, journalling filesystems, data volumes kept separate from the system — not on the day of the failure. It is the kind of check we run during a failover test, because a deliberate restart reveals them immediately.

What it does not do — and it matters

The virtual machine restarts, it does not fail over. That distinction is not cosmetic:

• In-flight sessions are lost. Application connections, user sessions, uncommitted transactions: everything is cut, exactly as in a power outage.
• A dead application inside a running VM is never detected. The cluster watches nodes, not what runs inside them. A machine answering with a crashed service is, as far as it is concerned, perfectly healthy.
• A datacenter outage is not covered if every node sits in the same place. Redundancy stops at the boundary of the site.

Those three limits are exactly what the higher levels cover — see the 4 architectures.

4. The case without shared storage

Storage shared between several nodes assumes a fast internal network and the budget that goes with it. Plenty of infrastructures do without, replacing that sharing with scheduled replication: the virtual machine's disk is copied at regular intervals to another hypervisor, which then becomes able to restart it.

The restart mechanism is the same, but the consequence differs: the machine comes back from the latest copy, not from its state at the moment of failure. Writes that happened in between are lost — typically one to fifteen minutes depending on the configured interval.

It is often an excellent trade-off for a web front end, a stateless application server or a staging environment. It is far less so for a transactional database, where losing ten minutes of writes means losing orders. The choice between replication modes and their implications is covered in detail on our Proxmox site.

5. The three situations compared

	Nothing in place	Organised restore	HA cluster
Who detects	A user calling in	Monitoring, then on-call	The cluster itself
Recovery time	A day, sometimes more	1 to 4 h	2 to 5 min
Data lost	Back to the last backup	Back to the last backup	None with shared storage
Prerequisites	—	Backup restorable elsewhere, on-call	3 nodes, reachable storage, fencing
Extra cost	None	Low	2 extra nodes + networking

The middle column is where most of the infrastructures we take over currently sit. It is not a bad place to be — it is the serious minimum — but it plateaus at a few hours, and no amount of organisational effort will bring it below that.

6. What we operate afterwards

A correctly installed cluster degrades quietly if it is not watched — and not watched in the usual sense of the word. The service answers, the machines run, the graphs are green: nothing indicates that redundancy has gone.

• Regular failover tests. A failover you have never deliberately triggered is a hypothesis, not a guarantee. We schedule them and measure the actual time achieved, which is the only figure that counts.
• Fencing verification. It is the component that breaks most discreetly, often after an update or a credential change on the server's management card.
• Quorum and spare capacity tracking. A cluster running for weeks with one node missing is no longer redundant: it is simply waiting for the second failure. And a cluster whose surviving nodes could not absorb the missing one's load is not redundant either — it is optimistic.

All of this is routine operation, included in our managed services plans, with 24/7 on-call for whatever automation cannot handle.

7. Who this is enough for

It is plenty if…

• A few minutes of interruption are tolerable
• Users can reconnect without consequence
• Activity is concentrated in business hours
• A single site hosts the whole infrastructure

It is not enough if…

• A dropped session costs a lost order
• The application can crash without the VM stopping
• The service must stay reachable during failover
• Losing an entire site is one of your scenarios

In the right-hand column, the answer is not "more hypervisors": it is a change of granularity. You have to watch the service rather than the machine, which means an application cluster — possibly sitting on top of the hypervisor cluster described here. That is the two-layer architecture, and its four traps.

Frequently asked questions

Do you really need three hypervisors for high availability?

For the cluster to decide on its own, yes. The third node is not there to host extra machines, it is there to break ties: with three, a majority always exists, and the group holding it knows it is legitimate while the isolated one knows it is not. With two, neither can tell a dead neighbour from a cut network link. There is an alternative: a lightweight witness, a small machine that only arbitrates without hosting any workload. It costs far less than a third full hypervisor and it is enough to unblock the decision.

Can you build high availability without shared storage?

Yes, using scheduled replication between nodes, but the outcome is not the same. The virtual machine's disk is copied at regular intervals to another hypervisor; on failure it restarts from the latest copy. You therefore lose the writes that happened since, typically one to fifteen minutes of data depending on the interval you chose. That is acceptable for an application server or a web front end, much less so for a transactional database.

How long does restoring a 500 GB VM take?

The basic arithmetic is simple: 500 GB at 200 MB/s of usable throughput is roughly forty minutes of transfer. But usable throughput is rarely the link's: it depends on source storage, target storage and deduplication. More importantly, that transfer is only part of the total delay, on top of which come detection time, the decision, preparing the target node and starting the application. In practice we observe one to four hours depending on the time of day, with nights and weekends being the worst cases.

What happens to a database when the VM is forcibly restarted?

It is exactly equivalent to a power cut. A modern transactional engine survives that treatment: on restart it replays its journal, rolls back uncommitted transactions and comes back consistent. What is lost are the transactions in flight at the moment of the cut, and the application connections, which will have to be re-established. The real risk is therefore not corruption but the duration of the replay: on a very busy database it can add several minutes to the restart.

Does hypervisor high availability detect a crashed application?

No, and that is its fundamental limit. The cluster watches nodes, not what runs inside the virtual machines. A running VM whose application service is dead is, from its point of view, a perfectly healthy VM: nothing will be triggered. Covering that scenario requires monitoring at the service level itself, either through supervision and on-call, or through an application cluster that fails over to another instance.

Is high availability included in managed services?

Operating an existing cluster is included in the managed services plan, at the per-server rate, with a reduced rate from the second node of the same cluster. Initial design and deployment are billed separately, on quote, based on €150 excl. VAT per hour: the scope depends on the number of nodes, the hosting provider and the application that has to stay available. The hardware itself is bought directly by the client and remains theirs.

Would your infrastructure survive losing a hypervisor?

We audit what you have, we quote what is missing, and we tell you whether the third node is worth its cost in your case.

Get a quote

Hypervisor failure:
restart your VMs in 5 minutes, not 4 hours

Table of contents