Thanks to Danny and Joe for review.

As the launch of the beacon chain grows nearer and eth2 becomes ever more final, the time has come to fast-sync the community with the latest on the inner workings of eth2 and on the concrete requirements, incentives and experience of being a validator. This article will provide a high-level overview of eth2 which will form the basis for a series on all aspects of eth2 relevant to validators.
eth2 has been in the works for a long time now and has improved dramatically over the years. What were initially separate sharding and Proof of Stake (PoS) efforts managed via smart contracts has transmogrified into a highly interconnected design which yields dramatic improvements regarding efficiency, scalability and security.

The phases

As parts of eth2 have become more interconnected, other pieces have been separated out into phases to allow for better pipelining of the different aspects of eth2. At the time of writing, Phase 0 is nearing launch as developers put the finishing touches on the client software. Meanwhile, the specification for Phase 1 is being completed, and Phase 2 is under active R&D.

• Phase 0 is concerned with the beacon chain, the core of eth2, which manages validators and the coordination of shards. The beacon chain is the source of ground truth from which all other aspects of eth2 are bootstrapped.
• Phase 1 builds upon this by allowing data to be put into shards. The implementation complexity of this component is much smaller than the others as phase 0 lays most of the ground work for the shards.
• Phase 2 adds execution to eth2 basically upgrading eth2 from a robust database to a fully decentralised computing platform.

What exactly is Phase 0?

As mentioned previously, the beacon chain tracks the state of both the set of validators and the shards. In practice this means that if you (periodically) follow what is happening on the beacon chain, you will know enough to verify anything said to be happening within eth2. Trust, but verify.

In order for a PoS system to function, there needs to be consensus on who the validators are, and on what each of their stakes are in order to know how much their votes are worth, and to appropriately reward and/or punish them for their behaviour. The beacon chain also manages the sharding aspects of eth2 by assigning validator duties in the shards as well as tracking the current state of each shard.

Part of what differentiates eth2 from other PoS systems is the sheer number of validators that can participate in the protocol. In contrast to the 10s, 100s, and 1000s of participants that are possible in other systems, eth2 scales to hundreds of thousands or even millions of validators. This level of decentralisation is only possible due to the intermediate levels of consensus achieved by groups of validators called committees. The beacon chain uses the eponymous random beacon at its core to assign validators to committees which are tasked with evaluating what is and isn’t a part of the beacon and shard chains. A committee’s votes are then cryptographically aggregated into an attestation meaning that verifying an entire committee’s votes is only marginally more effort than checking a single vote. Therefore, to check the validity of the beacon chain, only a few aggregated signatures need to be considered to evaluate the votes of many validators.

The beacon chain also tracks the eth1 chain and the deposits thereupon so that new validators can join eth2 by sending 32 Ether to the deposit contract on eth1. As a result of the beacon chain voting on the eth1 chain, eth2 will, at some point in the future, enhance the security of eth1 by providing an economic guarantee that blocks that are a part of the canonical eth1 chain.

Nodes vs. Clients

eth2 makes the distinction between beacon nodes and validator clients, and validators will need both in order to perform their duties. A beacon node (or just node) concerns itself with maintaining a view of the beacon chain as well as whichever shards may be needed by a user or validator.

As their name suggests, validator clients (or just clients) handle the logic of a single validator. This is achieved by communicating with the beacon node to understand the current state of the chain, by attesting to and proposing blocks as well when appropriate, and finally by asking the beacon node to send this information on to its peers.

If you are not running a validator, a beacon node contains all of the information you need to trustlessly interact with eth2, much like a full node in eth1.

Below are some of the many arguments for this separation:

• Each validator needs to be initiated with a deposit of exactly 32 Ether and therefore people who wish to stake more ETH will need to run multiple validator instances. The node-client separation allows such users to only run a single beacon node with multiple validators connected to it thereby reducing computation, memory, and storage requirements.
• By having validator nodes be separate modules, they will likely be more secure as it is easier to write, reason about, and audit smaller code modules.
• For users particularly worried about redundancy, multiple nodes can be run in parallel, thus reducing the chance of a validator going offline.
• Because validator clients can only interact with the rest of the eth2 network via a beacon node, and even then via a restricted API, the attack surface of a validator node is greatly reduced.
• For users who wish to interact with eth2, but don’t want to be a validator, they need only operate a beacon node which will grant them access to the beacon chain and all the shards they require.

Design Philosophy

The design philosophy of eth2 provides useful context for all the decisions made within eth2 and in many instances encapsulate the differences between eth2 and other protocols.

• Protocol über alles: With the acknowledgement that everything is a trade off, the protocol’s safety and liveness take precedence over other design desiderata.
• Hope for the best, but expect the worst: eth2 assumes validators will be lazy, take bribes, and that they will try to attack the system unless they are otherwise incentivised not to. Furthermore, the network is assumed not to be entirely reliable and that catastrophic events could force large numbers of validators to go offline. For these reasons, eth2 should be capable of surviving World War 3.
• Minimally viable complexity: Wherever possible, eth2 has been simplified as this makes it easier to reason about, explain to others, audit, write bug free clients, and generally avoid edge cases.
• Maximally decentralised: Proof of stake protocols commonly compromise on the number of validators that can participate, eth2 is designed to scale to millions of validators while encouraging these validators to work independently of one another.
• Expect the unexpected: All components of eth2 are resistant to quantum computers or can be swapped out for those that are in the event of a quantum apocalypse.
• By the people for the people: eth2 must be able to run on a consumer laptop. The lower the barrier to entry, the more people who can participate which translates into a higher degree of decentralisation.

Wrapping up

Now that you have the basics of eth2 under your belt, the next posts in this series will tackle the juicy details of what makes eth2 tick.

Special thanks to Sacha Yves Saint-Leger & Danny Ryan for review.

In this installment, we’ll discuss the consensus mechanisms behind eth2. Eth2 has a novel approach to deciding which block is the head of the chain, along with which blocks are and are not a part of the chain.

By using a hybrid between the two mechanisms, eth2 aims to have a consensus which, in addition to being rapid and safe when the network is behaving normally, remains safe even when it’s being attacked.

A Trilemma

FLP impossibility is a core result in the field of distributed computation which states that in a distributed system it is not possible to simultaneously have safety, liveness, and full asynchrony unless some unreasonable assumptions can be made about your system.

Safety is the idea that decisions cannot be unmade whereas liveness captures the notion that new things can be decided. A protocol is asynchronus if there is no bound on how long a message may take to get delivered.

If nodes could communicate reliably, always follow the protocol honestly and never crash, then consensus would be easy, but that is not how the world works. When these assumption don’t hold, FLP Impossibility is the proof that at least one of: safety, liveness, or full asynchrony must be compromised.

GHOSTs and their opinions on forks

Eth2 uses Greedy Heaviest Observed Subtree (GHOST) as its fork-choice rule. GHOST selects the head of the chain by choosing the fork which has the most votes (it does this by considering all of the votes for each fork block and their respective child blocks).

Put another way, each time there is a fork, GHOST chooses the side where more of the latest messages support that block’s subtree (i.e. more of the latest messages support either that block or one of its descendants). The algorithm does this until it reaches a block with no children.

GHOST has the benefit of reducing the efficacy of attacks during times of high network latency as well as minimizing the depth of chain reorgs when compared to the longest-chain rule. This is because while an attacker can keep building blocks efficiently on their own chain thereby making it the longest, GHOST would choose the other fork as there are more votes for it in total.

In particular, eth2 uses a variation of GHOST which has been adapted to a PoS context called Latest Message Driven GHOST (LMD-GHOST). The idea behind LMD-GHOST is that when calculating the head of the chain, one only considers the latest vote made by each validator, and not any of the votes made in the past. This dramatically decreases the computation required when running GHOST, since the number of forks that need to be considered to execute the fork choice cannot be greater than the number of validators ($O(v)$ in Big O notation).

Under the rules of GHOST, validators/miners can always try to add a new block to the blockchain (liveness), and they can do this at any point in the chain’s history (asynchronous). Since it is live and fully asynchronous, thanks to our friend FLP, we know it can’t be safe.

The lack of safety presents itself in the form of reorgs where a chain can suddenly switch between forks of arbitrary depth. Obviously this is undesirable and eth1 deals with this by having users make assumptions about how long miners’ blocks will take to be communicated with the rest of the network, this takes the form of waiting for $x$ confirmations. Eth2, by contrast, makes no such assumptions.

The friendly finality gadget

A blockchain without any notion of safety is useless because no decisions could be reached and users could not agree on the state of the chain. Enter Casper the Friendly Finality Gadget (Casper FFG). Casper FFG is a mechanism which favours safety over liveness when making decisions. This means that while the decisions it makes are final, under poor network conditions, it may not be able to decide on anything.

FFG is a crypto-economic adaption of the classic Practical Byzantine Fault Tolerent (PBFT) which has phases where nodes first indicate that they’d like to agree on something (justification) and then agree that they’ve seen each other agreeing (finalisation).

Eth2 does not try to justify and finalise every slot (the time when a block is expected to be produced), but instead only every 32 slots. Collectively, 32 slots is called an epoch. First, validators sign that they agree with all 32 blocks in an epoch. Then, if $\geq \frac{2}{3}$ do so, the block is justified. In a later epoch, validators get another chance to vote to indicate that they have seen the earlier justified epoch and if $\geq \frac{2}{3}$ do this, the epoch is finalised and is forever a part of the eth2 chain.

FFG employs a clever trick. Votes actually consist of two sub-votes, one for the epoch that is attempting to be justified and another for an earlier epoch that is to become finalised. This saves a lot of extra communication between nodes and helps to achieve the goal of scaling to millions of validators.

Two ghosts in a trench coat

Consensus within eth2 relies on both LMD-GHOST – which adds new blocks and decides what the head of the chain is – and Casper FFG which makes the final decision on which blocks are and are not a part of the chain. GHOST’s favourable liveness properties allow new blocks to quickly and efficiently be added to the chain, while FFG follows behind to provide safety by finalising epochs.

The two protocols are merged by running GHOST from the last finalised block as decided upon by FFG. By construction, the last finalised block is always a part of the chain which means GHOST doesn’t need to consider earlier blocks.

In the normal case when blocks are being produced and $\geq \frac{2}{3}$ validators are voting on them, these blocks are added to the head of the chain by GHOST, and not long after justified and finalised by FFG (which considers the last few epochs).

If there is an attack on the network and/or a large proportion of validators go offline, then GHOST continues adding new blocks. However, since GHOST is live, but not safe, it may change its mind about the head of the chain – this is because new blocks are continually added to the chain, which means nodes keep learning new information. FFG on the other hand, favours safety over liveness meaning that it stops finalising blocks until the network is stable enough for validators to vote consistently again.

Special thanks to Sacha Yves Saint-Leger & Joseph Schweitzer for review.

Sharding is one of the many improvements that eth2 has over eth1. The term was borrowed from database research where a shard means a piece of a larger whole. In the context of databases and eth2, sharding means breaking up the storage and computation of the whole system into shards, processing the shards separately, and combining the results as needed. Specifically, eth2 implements many shard chains, where each shard has similar capabilities to the eth1 chain. This results in massive scaling improvements.

However, there’s a less-well-known type of sharding in eth2. One which is arguably more exciting from a protocol design point of view. Enter sharded consensus.

Sharding Consensus

In much the same way that the processing power of the slowest node limits the throughput of the network, the computing resources of a single validator limit the total number of validators that can participate in consensus. Since each additional validator introduces extra work for every other validator in the system, there’ll come a point where the validator with the least resources can no longer participate (because it can no longer keep track of the votes of all of the other validators). The solution eth2 employs to this is sharding consensus.

Breaking it down

Eth2 breaks time down into two durations, slots and epochs.

A slot is the 12 second time-frame in which a new block is expected to be added to the chain. Blocks are the mechanism by which votes cast by validators are included on the chain in addition to the transactions that actually make the chain useful.

An epoch is comprised of 32 slots (6.4 minutes) during which the beacon chain performs all of the calculations associated with the upkeep of the chain, including: justifying and finalising new blocks, and issuing rewards and penalties to validators.

As we touched upon in the first post of this series, validators are organised into committees to do their work. At any one time, each validator is a member of exactly one beacon chain and one shard chain committee, and is called on to make an attestation exactly once per epoch – where an attestation is a vote for a beacon chain block that has been proposed for a slot.

The security model of eth2’s sharded consensus rests upon the idea that committees are more or less an accurate statistical representation of the overall validator set.

For example, if we have a situation in which 33% of validators in the overall set are malicious, there is a chance that they could end up in the same committee. This would be a disaster for our security model.

So we need a way to ensure that this can’t happen. In other words, we need a way to ensure that if 33% of validators are malicious, only about ~33% of validators in a committee will be malicious.

It turns out we can achieve this by doing two things:

1. Ensuring committee assignments are random
2. Requiring a minimum number of validators in each committee

For example, with 128 randomly sampled validators per committee, the chance of an attacker with 1/3 of the validators gaining control of > 2/3 committee is vanishingly small (probability less than 2^-40).

Building it up

Votes cast by validators are called attestations. An attestation is comprised of many elements, specifically:

• a vote for the current beacon chain head
• a vote on which beacon block should be justified/finalised
• a vote on the current state of the shard chain
• the signatures of all of the validators who agree with that vote

By combining as many components as possible into an attestation, the overall efficiency of the system is increased. This is possible since, instead of having to check votes and signatures for beacon blocks and shard blocks separately, nodes need only do the math on attestations to be informed about the state of the beacon chain and of every shard chain.

If every validator produced their own attestation and every attestation needed to be verified by all other nodes, then being an eth2 node would be prohibitively expensive. Enter aggregation.

Attestations are designed to be easily combined such that if two or more validators have attestations with the same votes, they can be combined by adding the signatures fields together in one attestation. This is what we mean by aggregation.

Committees, by their construction, will have votes that are easy to aggregate because they are assigned to the same shard, and therefore should have the same votes for both the shard state and beacon chain. This is the mechanism by which eth2 scales the number of validators. By breaking the validators up into committees, validators need only to care about their fellow committee members and only have to check very few aggregated attestations from each of the other committees.

Signature aggregation

Eth2 makes use of the BLS signatures – a signature scheme defined over several elliptic curves that is friendly to aggregation. On the specific curve chosen, signatures are 96 bytes each.

If 10% of all ETH ends up staked, then there will be ~350,000 validators on eth2. This means that an epoch’s worth of signatures would be 33.6 megabytes which comes to ~7.6 gigabytes per day. In this case, all of the false claims about the eth1 state-size reaching 1TB back in 2018 would be true in eth2’s case in fewer than 133 days (based on signatures alone).

The trick here is that BLS signatures can be aggregated: If Alice produces signature A, and Bob’s signature is B on the same data, then both Alice’s and Bob’s signatures can be stored and checked together by only storing C = A + B. By using signature aggregation, only 1 signature needs to be stored and checked for the entire committee. This reduces the storage requirements to less than 2 megabytes per day.

In summary,

By separating validators out into committees, the effort required to verify eth2 is reduced by orders of magnitude.

For a node to validate the beacon chain and all of the shard chains, it only needs to look at the aggregated attestations from each of the committees. In this way it can know the state of every shard, and every validator’s opinions on which blocks are and aren’t a part of the chain.

The committee mechanism therefore helps eth2 achieve two of the design goals established in the first article: namely that participating in the eth2 network must be possible on a consumer-grade laptop, and that it must strive to be maximally decentralised by supporting as many validators as possible.

To put numbers to it, while most Byzantine Fault Tolerant Proof of Stake protocols scale to tens (and in extreme cases, hundreds of validators), eth2 is capable of having hundreds of thousands of validators all contributing to security without compromising on latency or throughput.

*Disclaimer: None of this is meant as a slight against any client in particular. There is a high likelihood that each client and possibly even the specification has its own oversights and bugs. Eth2 is a complicated protocol, and the people implementing it are only human. The point of this article is to highlight how and why the risks are mitigated.*

With the launch of the Medalla testnet, people were encouraged to experiment with different clients. And right from genesis, we saw why: Nimbus and Lodestar nodes were unable to cope with the workload of a full testnet and got stuck. [0][1] As a result, Medalla failed to finalise for the first half hour of its existence.

On the 14th of August, Prysm nodes lost track of time when one of the time servers they were using as a reference suddenly jumped one day into the future. These nodes then started making blocks and attestations as though they were also in the future. When the clocks on these nodes were corrected (either by updating the client, or because the timeserver returned to the correct time), those that had disabled the default slashing protection found their stakes slashed.

Exactly what happened is a bit more subtle, I highly recommend reading Raul Jordan’s write-up of the incident.

Clock Failure – The enworsening

The moment when Prysm nodes started time traveling, they made up ~62% of the network. This meant that the threshold for finalising blocks (>2/3 on one chain) could not be met. Worse still, these nodes couldn’t find the chain that they were expecting (there was a 4 hour “gap” in the history and they all jumped ahead to slightly different times) and so they flooded the network with short forks as they guessed at the “missing” data.

Prysm currently makes up 82% of Medalla nodes ! [ethernodes.org]

At this point, the network was flooded with thousands of different guesses at what the head of the chain was and all the clients started to buckle under the increased workload of figuring out which chain was the right one. This led to nodes falling behind, needing to sync, running out of memory, and other forms of chaos, all of which worsened the problem.

Ultimately this was a good thing, as it allowed us to not only fix the root problem relating to clocks, but to stress test the clients under condition of mass node failure and network load. That said, this failure need not have been so extreme, and the culprit in this case was Prysm’s dominance.

Shilling Decentralisation – Part I, it’s good for eth2

As I’ve discussed previously, 1/3 is the magic number when it comes to safe, asynchronous BFT algorithms. If more than 1/3 of validators are offline, epochs can no longer be finalised. So while the chain still grows, it is no longer possible to point to a block and guarantee that it will remain a part of the canonical chain.

Shilling Decentralisation – Part II, it’s good for you

To the maximum possible extent, validators are incentived to do what is good for the network and not simply trusted to do something because it is the right thing to do.

If more than 1/3 of nodes are offline, then penalties for the offline nodes start ramping up. This is called the inactivity penalty.

This means that, as a validator, you want to try to ensure that if something is going to take your node offline, it is unlikely to take many other nodes offline at the same time.

The same goes for being slashed. While, there’s always a chance that your validators are slashed due to a spec or software mistake/bug, the penalties for single slashings are “only” 1 ETH.

However, if many validators are slashed at the same time as you, then penalties go up to as high as 32 ETH. The point at which this happens is again the magic 1/3 threshold. [An explanation of why this is the case can be found here].

These incentives are called liveness anti-correlation and safety anti-correlation respectively, and are very intentional aspects of eth2’s design. Anti-correlation mechanisms incentivise validators to make decisions that are in the best interest of the network, by tying individual penalties to how much each validator is impacting the network.

Shilling Decentralisation – Part III, the numbers

Eth2 is being implemented by many independent teams, each developing independent clients according to the specification written primarily by the eth2 research team. This ensures that there are multiple beacon node & validator client implementations, each making different decisions about the technology, languages, optimisations, trade-offs etc required to build an eth2 client. This way, a bug in any layer of the system will only impact those running a specific client, and not the whole network.

If, in the example of the Prysm Medalla time-bug, only 20% of eth2 nodes were running Prysm and 85% of people were online, then the inactivity penalty wouldn’t have kicked in for Prysm nodes and the problem could have been fixed with only minor penalties and some sleepless nights for the devs.

In contrast, because so many people were running the same client (many of whom had disabled slashing protection), somewhere between 3500 and 5000 validators were slashed in a short period of time.* The high degree of correlation means that slashings were ~16 ETH for these validators because they were using a popular client.

* At the time of writing, slashings are still pouring in, so there is no final number yet.

Try something new

Now is the time to experiment with different clients. Find a client that a minority of validators are using, (you can look at the distribution here). Lighthouse, Teku, Nimbus, and Prysm are all reasonably stable at the moment while Lodestar is catching up fast.

Most importantly, TRY A NEW CLIENT! We have an opportunity to create a more healthy distribution on Medalla in preparation for a decentralised mainnet.

T'was the day before genesis, when all was prepared,
geth was in sync, my beacon node paired.
Firewalls configured, VLANs galore,
hours of preparation meant nothing ignored.

Then all at once everything went awry,
the SSD in my system decided to die.
My configs were gone, chain data was history,
nothing to do but trust in next day delivery.

I found myself designing backups and redundancies.
Complicated systems consumed my fantasies.
Thinking further I came to realise:
worrying about these kinds of failures was quite unwise.

Events

The beacon chain has several mechanisms to incentivise validator behaviour, all of which are dependant on the current status of the network, so it is vital to consider these failure cases in the greater context of how other validators might fail when deciding what are, and what aren’t, worthwhile ways of securing your node(s).

As an active validator, your balance either increases or decreases, it never goes sideways*. Therefore a pretty reasonable way of maximising your profits, is to minimise your downsides. There are 3 ways your balance can be reduced by the beacon chain:

• Penalties are issued when your validator misses one of their duties (e.g. because they are offline)
• Inactivity Leaks are handed out to validators that miss their duties while the network is failing to finalise (i.e. when your validator being offline is highly correlated with other validators being offline)
• Slashings are given to validators who produce blocks or attestations that are contradictory and therefore could be used in an attack

* On average, a validator’s balance may stay the same, but for any given duty, they are either rewarded or punished.

Correlation

The effect of a single validator being offline or performing slashable behaviour is small in terms of the overall health of the beacon chain. It is therefore not punished heavily. In contrast, if many validators are offline, the balance of offline validators can decrease much more rapidly.

Similarly, if many validators perform slashable actions at the same time, from the beacon chain’s perspective, this is indistinguishable from an attack. It is therefore treated as such, and 100% of the offending validators’ stake is burned.

Because of these “anti-correlation” incentives, validators should worry more about failures that might affect others at the same time rather than isolated, individual issues.

Causes and their probability.

So let’s think through some failure cases and examine them through the lens of how many others would be affected at the same time, and how badly your validators would be punished.

I disagree with @econoar here that these are worst case issues. These are more moderate level issues. Home UPS and Dual WAN address failures aren’t correlated with other users and so should be far down your list of concerns.

Internet/power failure

If you are validating from home, then it’s highly likely you’ll encounter one of these failures at some point in the future. Residential internet and power connections do not have guaranteed uptime. However, when the internet does go down, or your power is out, the outage is usually limited to your area and even then only for a few hours.

Unless you have very spotty internet/power, it might not be worthwhile paying for fall-over connections. You’ll receive a few hours of penalties, but as the rest of the network is running normally, your penalties will be roughly equal to what your rewards would have been over the same period. In other words, a k hour-long failure sets your validator’s balance back to roughly where it was k hours before the failure, and in k additional hours your validator’s balance will be back to its pre-failure amount.

[Validator #12661 regaining ETH as quickly as it was lost – Beaconcha.in

Hardware failure

Like internet failure, hardware failure strikes randomly, and when it does, your node might be down for a few days. It is valuable to consider the expected rewards over the lifetime of the validator versus the cost of redundant hardware. Is the expected value of the failure (the offline penalties times the chance of it happening) greater than the cost of the redundant hardware?

Personally, the chance of failure is low enough and the cost of fully redundant hardware high enough, that it almost certainly isn’t worth it. But then again, I am not a whale ; as with any failure scenario, you need to evaluate how this applies to your particular situation.

Cloud services failure

Maybe, to avoid the risks of hardware or internet failure altogether, you decide to go with a cloud provider. With a cloud provider, you have introduced the risk of correlated failures. The question that matters is, how many other validators are using the same cloud provider as you?

A week before genesis, Amazon AWS had a prolonged outage which affected a large portion of the web. If something similar were to happen now, enough validators would go offline at the same time that the inactivity penalties would kick in.

Even worse, if a cloud provider were to duplicate the VM running your node and accidentally leave the old and the new node running at the same time, you could be slashed (the penalties incurred would be especially bad if this accidental duplication affected many other nodes too).

If you are insistent on relying on a cloud provider, consider switching to a smaller provider. It may end up saving you a lot of ETH.

Staking Services

There are several staking services on mainnet today with varying degrees of decentralisation, but they all contain an increased risk of correlated failures if you trust them with your ETH. These services are necessary components of the eth2 ecosystem, especially for those with less than 32 ETH or without the technical know-how to stake, but they are architected by humans and therefore imperfect.

If staking pools eventually grow to be as large as eth1 mining pools, then it is conceivable that a bug could cause mass slashings or inactivity penalties for their members.

Infura Failure

Last month Infura went down for 6 hours causing outages across the Ethereum ecosystem; it is easy to see how this is likely to result in correlated failures for eth2 validators.

In addition, 3rd party eth1 API providers necessarily rate-limit calls to their service: In the past this has caused validators to be unable to produce valid blocks (on the Medalla testnet).

The best solution is to run your own eth1 node: you won’t encounter rate-limiting, it will reduce the likelihood of your failures being correlated, and it will improve the decentralisation of the network as a whole.

Eth2 clients have also started adding the possibility of specifying multiple eth1 nodes. This makes it easy to switch to a backup endpoint, in the event your primary endpoint fails (Lighthouse: –eth1-endpoints, Prysm: PR#8062, Nimbus & Teku will likely add support somewhere in the future).

I highly recommend adding backup API options as cheap/free insurance (EthereumNodes.com shows the free and paid API endpoints and their current status). This is useful whether you are running your own eth1 node or not.

Failure of a particular eth2 client

Despite all the code review, audits, and rockstar work, all of the eth2 clients have bugs hiding somewhere. Most of them are minor and will be caught before they present a major problem in production, but there is always the chance that the client you choose will go offline or cause you to be slashed. If this were to happen, you would not want to be running a client with > 1/3 of the nodes on the network.

You must strike a tradeoff between what you deem to be the best client vs how popular that client is. Consider reading through the documentation of another client so that if something happens to your node, you know what to expect in terms of installing and configuring a different client.

If you have lots of ETH at stake, it is probably worth running multiple clients each with some of your ETH to avoid putting all your eggs in one basket. Otherwise, Vouch is an interesting offering for multi-node staking infrastructure, and Secret Shared Validators are seeing rapid development.

Black swans

There are of course many unlikely, unpredictable, yet dangerous scenarios that will always present a risk. Scenarios that lie outside the obvious decisions about your staking set-up. Examples such as Spectre and Meltdown at the hardware level, or kernel bugs such as BleedingTooth hint at some of the hazards that exist across the entire hardware stack. By definition, it is not possible to entirely predict and avoid these problems, instead you generally must react after the fact.

What to worry about

Ultimately this comes down to calculating the expected value E(X) of a given failure: how likely an event is to happen, and what the penalties would be if it did. It is vital to consider these failures in the context of the rest of the eth2 network since the correlation greatly affects the penalties at hand. Comparing the expected cost of a failure to the cost of mitigating it will give you the rational answer as to whether it is worth getting in front of.

No one knows all the ways a node can fail, nor how likely each failure is, but by making individual estimates of the chances of each failure type and mitigating the biggest risks, the “wisdom of the crowd” will prevail and on average the network as a whole will make a good estimate. Furthermore, because of the different risks each validator faces, and the differing estimates of those risks, the failures you did not account for will be caught by others and therefore the degree of correlation will be reduced. Yay decentralisation!

DON’T PANIC

Finally, if something does happen to your node, don’t panic! Even during inactivity leaks, penalties are small on short time scales. Take a few moments to think through what happened and why. Then make a plan of action to fix the problem. Then take a deep breath before you proceed. An extra 5 minutes of penalties is preferable to being slashed because you did something ill-advised in a rush.

Most of all: Do not run 2 nodes with the same validator keys!

Thanks Danny Ryan, Joseph Schweitzer, and Sacha Yves Saint-Leger for review

[Slashings because validators ran >1 node – Beaconcha.in]