Two-layer HA:
2 clustered VMs on 2 hypervisors, and the 4 traps
It is the best high-availability architecture on a single site. It is also the easiest to get wrong — because the two layers do not talk to each other, and wherever they ignore each other, they contradict each other.
Table of contents
1. Why stack: each layer covers the other's blind spot
The architecture consists of installing a service cluster inside two virtual machines, themselves hosted on hypervisors in high availability. You do not adopt it to add two safety nets together: you adopt it because each one covers precisely what the other lets through.
The service cluster alone
It fails over within seconds, but after the failure only one node is left standing. The service works, with no redundancy at all, until a human rebuilds the lost member. You survived the first failure and you are defenceless against the second.
The hypervisor layer alone
It rebuilds on its own, but its granularity is too coarse: it takes minutes, sessions are cut, and above all it sees nothing of a dead application inside a running machine. Covered in detail at level 1.
The two stacked
The service fails over within seconds — users see a reconnection at worst — and then the layer below restarts the lost machine on another hypervisor, and the cluster gets both its members back without anyone intervening.
That two-step sequence is where all the value of the design lies: immediate failover, then automatic rebuilding of redundancy. Neither layer can do both, and that is why you do not choose between them.
2. Trap 1 — Anti-affinity
Two virtual machines forming a cluster provide no redundancy whatsoever if they run on the same hypervisor. Put like that, it is obvious. It is far less obvious six months into production.
By default, nothing stops the layer below from placing them side by side. A maintenance migration, a node reboot, an automatic restart after an incident: on each of those occasions, both members can end up in the same place. So you have to declare a rule forbidding that cohabitation — and above all verify that it is still being honoured.
The silent scenario
The rule exists. A maintenance window required draining a node, the operator temporarily worked around it, and nobody restored it. The service answers perfectly, monitoring is green, the application cluster sees both members healthy. Nothing, absolutely nothing, indicates that redundancy is gone — until the day the hypervisor hosting both of them fails, and the architecture that cost three servers performs worse than a single machine would have.
That is why we verify the actual placement of members as part of routine operation, not merely the existence of the rule. A declared but circumvented constraint is more dangerous than no constraint at all, because it creates the illusion of being covered.
3. Trap 2 — The two quorums
There are two stacked decision mechanisms, and they do not talk to each other. The hypervisor cluster counts its physical nodes; the service cluster counts its application members. They share neither thresholds, nor timings, nor the same view of the network.
As long as everything is fine, that independence is invisible. It becomes visible during a network partition — the case where the network splits in two without any machine dying — and that is where the two layers can draw different conclusions from the same event.
The scenario, step by step
Three hypervisors, two clustered VMs spread across two of them. A network link isolates the third node, which hosts one of the two members.
- • Hypervisor layer: two nodes out of three can still see each other, so they hold the majority. They decide the third is lost and that its VM should be restarted on them.
- • Service layer: with two members, neither holds a majority on its own. Without an arbiter, each may consider itself legitimate — or, more cautiously, both stand down and the service stops although no machine has failed.
- • The isolated node keeps running its VM: from where it stands, it is the rest of the world that has disappeared.
The practical conclusion is clear: a two-member service cluster needs its own arbiter, independent from the hypervisors'. A lightweight witness, a third member, or a fencing mechanism that settles matters by force — but something. Relying on the lower layer's quorum to protect the upper one is a common design mistake, and it only reveals itself the day the network splits.
4. Trap 3 — The timeout war
Both layers detect the same failure, at different speeds. If the order in which they react has not been decided, chance decides for you — and chance produces situations nobody planned for.
The service cluster detects within seconds: that is its whole purpose. The hypervisor layer detects within tens of seconds, deliberately, so as not to restart machines on every network hiccup. When those two scales overlap badly, two scenarios appear.
The double failover
The service cluster has already failed over to the surviving member. The slower hypervisor layer then restarts the lost VM. It boots, rejoins the cluster, and depending on configuration may take the active role back — causing a second interruption, this time with no failure to justify it.
The zombie node
Worse: the VM excluded from the cluster is still running somewhere, because the hypervisor hosting it was not dead but merely unreachable. It comes back onto the network still believing it holds its role, and two instances contend for the same address or the same disk.
The rule we apply
The application detection window must be clearly shorter than the hypervisor's — a factor of three to five, not twenty percent. The fast layer fails over and settles the situation; the slow layer then observes and merely rebuilds. Each in its role, in that order, and both scenarios above disappear.
5. Trap 4 — Cascading fencing
Each layer has its own way of isolating a doubtful member: the service cluster can cut a VM off or shut it down, the hypervisor layer can cut power to an entire server. When both decide to act at the same moment, the question becomes: which one has the final say?
The problematic case is easy to describe. The service cluster wants to shut down the VM it considers faulty. At the same moment, the hypervisor layer wants to reboot the physical server hosting it, because it no longer answers its own probes. Depending on the order, you get either a cleanly fenced VM restarted elsewhere — the intended outcome — or a server rebooted in the middle of a fencing operation, with a service cluster that never receives the confirmation it was waiting for and therefore refuses to fail over at all.
The design principle
Application-level fencing must be faster and finer-grained than the hypervisor's, and must never depend on a component the layer below can cut at the same instant. In practice that means a fencing mechanism acting on the VM itself rather than on its host, and a hypervisor-side delay long enough to let the application layer reach a conclusion. It is not a question of tooling, it is a question of ordering.
6. The reference architecture we deploy
Three hypervisors, two clustered virtual machines, one witness. It is the configuration that resolves all four traps at once, and the one we propose by default.
Lower layer — 3 hypervisors
- • Three nodes: a majority always exists
- • An anti-affinity rule on the two clustered VMs
- • Storage reachable from all three
- • Deliberately slow detection, reacting only to real failures
Upper layer — 2 VMs + 1 witness
- • Two active members, on two distinct hypervisors
- • A lightweight witness on the third, arbitrating only
- • Fast detection, three to five times faster than the layer below
- • Fencing acting on the VM, not on its host
The witness is the most cost-effective component of the whole design: it carries no load, costs almost nothing, and single-handedly removes the ambiguity of trap 2. It is also the one most often missing from the architectures we take over.
A common variant when three application nodes are available: instead of a single shared address, you declare two or three, spread across the members and published together in DNS. You get approximate load distribution on top of failover, without adding any central component at all.
At what downtime cost does this design pay for itself?
Quantify one hour of outage and the rest of the calculation follows.
7. Real cost and break-even
Three lines to separate, because they are not billed the same way and comparisons routinely conflate them.
| Item | Who pays | How |
|---|---|---|
| Hardware | The client, directly from the provider | Three servers instead of one, plus internal networking |
| Design and deployment | RDEM engagement | On quote, based on €150 excl. VAT/hour |
| Operation | Managed services plan | From €150 excl. VAT/month/server, reduced from the 2nd node |
Break-even is never calculated in the abstract: it is calculated against the cost of an hour of downtime. A company losing a few hundred euros per hour has no reason to pay for three servers; one losing several thousand has already recouped the difference on the first incident avoided. That comparison, and only that one, should decide. See the managed services plans.
8. When it is overkill
We regularly advise against this architecture, and it is better said here than discovered mid-project. Three signals indicate it is one step too far.
- • A five-minute outage costs nothing. If the business tolerates a restart, level 1 is enough, costs two servers less and is far easier to maintain.
- • The application cannot run as a pair. Many line-of-business applications assume a single instance. Clustering them then requires adaptation work that far exceeds the cost of the infrastructure — when it is possible at all.
- • Nobody will ever test the failover. An untested two-layer architecture is more fragile than a simple one, because it has four mechanisms that can degrade silently instead of one.
In those cases, stepping back down to level 1 is the right decision. Conversely, if the scenario that worries you is losing an entire site, what you need is not another layer but distribution across several sites — see the four architectures.
Frequently asked questions
Do you need three hypervisors for two clustered VMs?
Yes, for two distinct reasons that are often conflated. The first belongs to the hypervisor layer: with three nodes the cluster can decide on its own what restarts where, which it cannot do with two. The second is about capacity: with two hypervisors and two VMs spread across them, losing one node puts both VMs on the same machine, and you no longer have real redundancy even though the service keeps answering. The third node provides a destination that preserves the separation.
What happens if both VMs land on the same hypervisor?
The service keeps working perfectly, and that is exactly what makes the situation dangerous. The application cluster sees two healthy members, monitoring is green, nobody is alerted. But redundancy has become fictional: the failure of the single hypervisor hosting both takes down both members at once, and the two-layer architecture then behaves worse than a single server would, having cost three times as much for the same result. That is what the anti-affinity rule is for, and why it has to be checked regularly.
Should you disable hypervisor high availability when you have a service cluster?
No, but you have to subordinate it. The two layers must react in the right order: the service cluster, which detects within seconds, fails over first; the hypervisor, which detects within tens of seconds, then merely rebuilds redundancy by restarting the lost VM elsewhere. Disabling the hypervisor layer would deprive you of that automatic rebuild and leave you durably without redundancy after the first incident. What needs tuning is the timing, not the existence of the mechanisms.
How much does a two-layer high-availability architecture cost?
There are three lines to separate. Hardware, bought directly by the client: three hypervisors instead of one, plus the internal network linking them. Design and deployment, billed on quote at €150 excl. VAT per hour, because the scope depends entirely on what exists and on the application. And operation, from €150 excl. VAT per month per server, with a reduced rate from the second node of the same cluster. The break-even point is calculated against the cost of an hour of downtime, never in the abstract.
Does this architecture protect against a datacenter outage?
No, if all three hypervisors sit in the same room. A two-layer architecture covers the loss of a machine and the loss of a service remarkably well, but its redundancy stops at the boundary of the site: a general power cut, a cooling incident or a disaster takes all three nodes together. Covering that scenario belongs to a different logic, that of distribution across several sites, with the latency and cost constraints that come with it.
Is your cluster still redundant today?
We audit existing architectures: actual member placement, arbitration, detection timings, fencing. The surprise is rarely in the design — it is in what the design has become.
Request an audit