The Inactivity Leak

Gracefully handling offline Validators by forfeiting their stake

Inactivity leak — continuously punish non-cooperative Validators until >2/3+ of the remaining staked ETH can finalise a new checkpoint.

Nakamoto Consensus led to a fundamental shift in consensus protocol research to prioritise liveness over safety — allowing the block proposers to continue to make progress by proposing new blocks and eventually agreeing what should be one true blockchain.

In the case of proof of stake Ethereum, both liveness and safety are tackled independently of each other with their own consensus protocol:

• Liveness. The LMD-GHOST voting protocol allows online Validators to continuously propose and vote for new blocks, even if the vast majority of Validators are offline.

• Safety. The Casper FFG voting protocol captures safety as a supermajority of Validators must reach agreement on an irreversible checkpoint.

As long as >2/3 of Validators are active and they converge on a single decision, then the proof of stake protocol can achieve both liveness and safety.

On the flip-slide, if >1/3 of Validators are offline (or failing to cooperate), then the safety property is violated and a new checkpoint cannot be finalised due to a lack of votes. Proof of stake Ethereum will prioritise availability (liveness) to ensure all actions by Validators are widely propagated across the network with the hope that they will eventually converge and achieve consistency again.

As a side note, this is why Casper FFG is an overlay on top of LMD-Ghost and lives on the blockchain as all votes/actions must be recorded on-chain before it is counted — essentially just like a smart contract.

State of Emergency

The failure for Validators to reach agreement implies that >1/3 of Validators cannot agree on a single blockchain fork.

There are several reasons why this situation may arise:

• >1/3 of Validators are offline,

• Validators are applying different and conflicting fork-choice rules,

• Validators may be victim of an eclipse/sway attack and lack access to the same data as the other Validators,

In the latter two cases, it implies there are two or more competing blockchain forks. This is problematic as network participants cannot have confidence which blockchain fork will be considered the canonical chain, especially if the competing forks have conflicting transactions.

If the safety property continues to be violated for at least 4 epochs in a row, then the system is in a state of emergency and the inactivity leak is triggered.

A state of emergency changes the reward and penalty policy:

• No attestation reward. Validators are no longer rewarded for casting their vote, but the block proposer for a slot is still rewarded for including attestations in their block.

• Quadratic punishment. It increases the penalty for all offline Validators over time until the influence of non-cooperating Validators is reduced and the cooperating Validators represent 2/3+ of the staked ETH. issued to offline Validators are increased over time.

Attestation rewards were removed during a state of emergency to prevent potential discouragement attacks. One example is when an attacker purposely triggers an inactivity leak to forcefully penalise honest Validators while still earning rewards in the process. It is implemented as a precautionary measure. To the best of my knowledge, there is no known or concrete attack strategy that can be used to disadvantage or discourage honest Validators.

On the other hand, quadratic punishment is at the heart of an inactivity leak and the focus of this article. The quadratic nature provides time for Validators to resolve the fork out-of-band via software upgrades, but continues to penalise them the longer this takes. If the network recovers, then non-cooperative Validators will not be shielded as voting for the wrong blockchain fork is treated the same as not voting at all.

Before diving into the details on how to compute the inactivity leak penalty, we need to understand what it means for a Validator to be inactive and how this is measured by the proof of stake protocol.

Subjective chain-splits

Subjective views — blue Validators think the yellow Validators are wrong (and vice versa). The longer they don’t cooperate, the more penalties that will be issued against the “wrong validators” on the respective fork.

The inactivity leak progressively reduces the staked ETH and influence of inconsistent Validators who are perceived to be making the wrong decision until >2/3 of the staked ETH are in agreement.

We need to consider the world from a Validator’s view and this allows us to define two types of Validators:

• Subjective Validator. In their eyes, only their chosen blockchain fork should be considered the canonical chain.

• Non-cooperative Validator. Another Validator who is not voting for the “correct blockchain fork” according to the Subjective Validator.

A Subjective Validator considers the non-cooperative Validators as inactive and will punish them. All punishments from the inactivity leak are applied on the subjective Validator’s chosen blockchain fork. If the dispute is not resolved, then the chain split will become permanent without harming the cooperative Validators on the respective forks.

Let’s consider a concrete example:

• Validator A believes that blockchain fork A is the canonical chain.

• Validator B believes that blockchain fork B is the canonical chain.

Validator A believes that Validator B’s preferred blockchain fork is wrong. On blockchain fork A, Validator B will progressive be penalised and their staked ETH is reduced over time. The flip-side is also true, Validator B will believe that Validator A’s preferred blockchain fork is wrong and they will be progressively penalised on blockchain fork B.

If the Validators refuse to change their mind and start voting for the other blockchain fork, then the cooperative Validators will eventually represent >2/3 of the staked ETH on their chosen blockchain fork. The non-cooperative Validators will eventually be ejected as they cannot cooperate on all potential blockchain forks without running foul of the voting protocol’s commandments.

As such — there is a permanent, yet peaceful, chain split as the subjective blockchain forks continue into perpetuity. We can call it a peaceful form of mutually assured destruction as the conflicting Validators can place their bets and go there separate ways.

Inactivity scores

Activity scores are updated on a per-epoch basis. Image taken from the ETH2 book.

A Validator is considered active if they cast an attestation with the same source and target vote.

It is up to the proof of stake protocol, from the perspective of a Subjective Validator, to detect and measure the inactivity of other Validators before a penalty can be issued. All attestations must be recorded in the beacon blockchain to allow the proof of stake protocol to keep track of Validator activity.

It is called an inactivity score and it is updated on a per-epoch basis:

• The inactivity leak IS active.

  • Correct target vote in prior epoch?

    • Yes — Reduce score by 1

  • No vote in prior epoch?

    • No — Increase score by 4

• The inactivity leak IS NOT active.

  • Reduce score by 16

Generally speaking, the rate of increase is linear and it will quickly distinguish Validators who are not co-operating. If a Validator returns online ('“cooperates”), then the penalty will reduce over time, albeit more slowly than its increase. It is designed to help alleviate the penalty for Validators who are trying to cooperate, but they are struggling to keep up with all the competing blockchain forks.

When the inactivity leak is over and cooperative Validators represent more than >2/3 of the staked ETH, then the score for all Validators decreases until it reaches 0. Penalties will continue to be issued for a short period of time after the inactivity leak and this helps build a smaller buffer for the cooperative Validators. Otherwise, it may result in oscillation if the >2/3 of staked ETH buffer is too tight.

Impact on a cooperative Validator that may temporarily be offline during an inactivity leak. Image and analysis taken from the ETH2 book.

In the ETH2 book, there is a nice analysis on how the inactivity score is applied to Validators with various degrees of participation:

• Always offline. A non-cooperative Validator that has not voted for the chosen blockchain fork. The inactivity score ramps up linearly.

• Temporarily offline. A Validator that suffers an outrage, lets say for 25 epochs between epoch 50 and 75, will continue to suffer penalties as the score slowly decreases.

• 70% online. The score will slowly increase over time, but it is relatively less than a Validator who is offline.

• 90% online. The score will hover around 5 or less. Penalties will be issued, but they will not be impactful.

• Always online. No penalty is issued to the Validator.

As we can see in the chart, it is problematic to be offline for a continuous period of time. The inactivity score will increase relatively fast. There is no quick fix for a Validator who returns online as the penalty will continuously be applied as the inactivity score slowly decreases.

Thankfully — it does achieve its goal of supporting Validators who are online most of the time as >90% uptime results in minimal penalties.

Computing the quadratic punishment

Validator validator; // Validator to be punished

 // Total epochs of inactivity leak
uint total_epochs = getInactivityLeakLength();

// Validator's current effective balance
uint effectiveBalance = getEffectiveBalance(validator); 

// Hard-coded value
uint INACTIVITY_PENALTY_QUOTIENT = 16,777,216; 

// Compute inactivity score (per epoch)
uint score = getInactivityScore(validator); 

// Total penalty (gwei)
uint penalty = (total_epochs*B)/(score * INACTIVITY_PENALTY_QUOTIENT);

All penalties are denominated in gwei and the calculation for computing the penalty has four components:

• Total epochs. The number of epochs since the last finalised checkpoint.

  • It is denoted as total_epochs.

• Effective balance. The effective balance of the Validator for the current epoch and it may be reducing over time due to issued penalties.

  • It is denoted as effectiveBalance.

• Inactivity score. The total score computed by this epoch to measure a Validator’s inactivity.

  • It is denoted score.

• Inactivity Penalty Quotient. A hard-coded value called set as 16,777,216.

  • It as INACTIVITY_PENALTY_QUOTIENT.

One way to interpret the equation is that the top half (the numerator) is to compute a base line penalty that should be applied to a Validator and the bottom half (the denominator) is the scale in which the base penalty should be applied.

Base line penalty. All Validators follow the same penalty path/curve, but base line penalty is dependent on their current effective balance. As the calculation relies on the effective balance and not the actual balance, it acts as a step-wise like function as the effective balance decreases in steps (and not gradually). The total length of the inactivity leak acts as a multiplier and increases the base line penalty over time.

Scaling penalty. The inactivity score is initially set to 0 and no penalty will be applied to the Validator. If a Validator is non-cooperative (or offline) during this time period, then the score increases by 4 every epoch. It acts as a linear increase for the base line penalty. The hard-coded value for the inactivity penalty is simply used to help establish the rate of increase.

What makes it quadratic? When the inactivity score is greater than 0 for any epoch, then the penalty is applied to the Validator’s effective balance. The nature of updating the effective balance in real time as the penalty is applied is what makes it quadratic. In practice, it is a step-wise quadratic penalty as the effective balance falls in a step-wise manner. If you want to jump more into the math, then check out the ETH2 book.

Severity of penalties. The penalty is continuously applied and it is increased over time as the Validator fails to cooperate.

To put the penalty into perspective:

A Validator will lose ~60% of their staked ETH after 18 days of inactivity (4096) epochs). It takes around three weeks (~4686 epochs) for the effective balance to fall to ~16 ETH and for the Validator to be ejected from the proof of stake protocol.

The above is assuming the inactivity leak is in full-force and the cooperative Validators do not yet represent >2/3 of all staked ETH. If the inactivity leak finishes earlier, then the inactivity score will fall to 0 over time. Non-cooperative Validators will suffer the expected penalty which is the inverse of rewards until they are ejected, but this is less severe and it may take a long time.

Necessity of multi-clients

Dankrad published a blog post with the provocative title Ethereum Merge: Run the majority client at your own peril! to warn Validators to diversify away from the majority client (Pyrsm) for the safety of the network and the Validator’s own staked funds.

Why is this important?

• Consensus bug induced chain split. If the majority client (>33% of Validators) has a consensus bug in the implementation of the fork-choice rule, then it can trigger a chain split.

It is a real and unpredictable issue. Consensus bugs have occurred in both Bitcoin and Ethereum over the years. The only way to fix the bug is intervention by client developers to quickly patch it — one of the reasons why the inactivity leak slowly ramps up as this provides time to rectify the issue.

A consensus bug can lead to three cases depending on Validator adoption of client software:

• No leak. The proof of stake protocol can handle the situation if <1/3 of Validators are impacted by the consensus bug. All impacted validators will only suffer the standard penalty for missing votes (inverse of rewards).

The next two cases result in a chain split with two blockchain forks, one will be the canonical chain (bug-free) and the other will be an invalid chain (consensus bug):

• Temporary chain split. The finality gadget for checkpointing does not work if >1/3 and <2/3 Validators cannot agree upon a single blockchain fork. The inactivity leak is triggered on the bug-free chain and all Validators voting for the wrong canonical chain will suffer the inactivity leak while developers attempt to fix the bug. The chain split can be resolved when the bug is fixed and the buggy Validators upgrade to the new client, but they will not be shielded from the penalties already issued.

• Permanent chain split. The worst-case scenario is when >2/3 Validators are impacted by the consensus bug. They will collectively agree and finalise the canonical chain with the consensus bug. If allowed, the bugger Validators cannot return to the bug-free canonical chain without violating the voting commandments. The bug-free canonical chain will activate the inactivity leak and eventually eject the buggy Validators.

The temporary chain split case is bad as Validators can forfeit some staked ETH, but the permanent chain split is the worst-case scenario as the buggy Validators can never return to the canonical chain. If a permanent chain split occurred, there is no agreement amongst the community whether a hard-fork will be deployed to resolve the situation and that may take a long time to implement across all consensus clients.

The multi-client approach is recommended to avoid the above outcomes. It is a form of N-level programming and it is hoped independent implementations should not lead to the same consensus bug across multiple clients. We will find out whether that approach truly works someday :)

If there a single takeaway — DO NOT RUN A MAJORITY CLIENT and look into fun consensus/execution client combos — this is the final line of defence for protecting your staked funds against unforeseeable events!

To the best of my knowledge, a mechanism like the inactivity leak, that reduces the influence and eventually eject non-cooperating Validators is unique to Ethereum’s proof of stake protocol.