Last week, seven of the eight Eth2 clients under active development succeeded in marking the major milestone of moving from single-client to multi-client testnets at the “Interop Lock-in”. With this exciting success in Eth2 development, we wanted to reflect on how this point was reached and on what it means to the Ethereum network and ecosystem.

Anyone following Ethereum over the past couple of years has likely become familiar with terms such as “Ethereum 2.0”, “Eth2”, or “Serenity”. Each of these refer to substantial upgrades slated for the Ethereum protocol that have been envisioned in some form since before the network went live in 2015.

In the early years of Ethereum, groundbreaking research was accomplished in parallel to the original chain (Eth1) launching, while the massive growth of the Ethereum community that followed aided the initial adoption of decentralized applications. Still, the road from these early breakthroughs to a highly decentralized, yet scalable proof-of-stake blockchain has been long. Over the past 18 months though, research has finally stablized into a cohesive and complete vision for the coming major upgrades known as Eth2.

As research moved into specifications toward the end of 2018, many teams (client teams) from across the community stepped up to build out core implementations of the protocol (clients). Since then, there has been a dynamic play between specification and implementation. Fort-nightly calls and a common spec repository organize communication and the sharing of ideas, but client teams have primarily worked in relative isolation, building and testing their implementations of the protocol.

While the spec was a moving target, clients could only dig so deep into interoperability and optimizations, but once the Phase 0 specification of Eth2 was deemed “frozen” on July 1, 2019, clients made tremendous progress and began to take concrete steps toward production.

Interop

Joseph Delong from Pegasys had the crazy idea of gathering members from each of the client engineering teams in a remote location for a week of interoperability work. The event was deemed the “Interop Lock-in” or as it was generally referred — “Interop”. With the spec freeze in sight and Devcon on the horizon, Interop in September was an opportunity have all of these stakeholders work through initial interop-issues in person.

The primary purpose of the event was to have each participating client to achieve pair-wise interoperability with each other client in small test networks — Lighthouse <-> Artemis, Lodestar <-> Lighthouse, Lodestar <-> Artemis, etc.

Participating client teams included:

Additional goals involved testing (1) larger networks in both node count and (2) validator count, (3) networks with 3+ clients, (4) enhancing tooling for monitoring and debugging Eth2 networks, and (5) other fun things like getting raspberry pis running and building fork visualizers.

Leading up to the event, some goals seemed like a stretch, but teams worked diligently until the deadline and achieved amazing progress. By the end of the week, client teams far exceeded original expectations of having a few pair-wise networks, instead completing the entire pair-wise tests, building a small network of all 7 participating clients, and more.

The following represent a glimpse into the highlights of the client successes, but is certainly not exhaustive:

Multi-client testnets

• All 7 participating clients achieved pair-wise interoperability, and although an 8th, Shasper, was not able to be in attendence, they have begun to work through this milestone as well.
• Many larger testnets were formed between 3+ clients, 3+ nodes, and higher than minimal validator counts.
• All 7 clients in attendence were successfully run on a single network.
• All participating languages’ libp2p implementations are now interoperable after debugging some minor issues.

Network debugging and tools

• Some consensus errors between clients were identified, debugged, and recorded as portions of the state transition that require increased test coverage.
• Command line tools were built to better debug ssz objects and state transitions (zcli, pycli, and similar tools embedded within clients).
• Progress made on metrics dashboards, a fork visualizer, and other tools to better understand clients and networks
• Clients were packaged up into containers to perform large-scale network tests within the Whiteblock genesis platform.

And then some

• Client teams served as eachother’s first alpha users resulting in extensive build/run scripts and related documentation.
• Isolated load tests with Nimbus and Lighthouse handled 2000+ validators on a single machine paired with similarly full nodes over LAN.
• Multiple clients were built and tested on a small raspberry pi network.

And beyond

Interop marked a major inflection point for Eth2. There is still much work to accomplish before launch, but engineering efforts will increasingly be geared toward testnets, optimizations, and usability — work that begins to shift this software into the hands of users.

So what’s up next for client teams and eth2 development?

• Benchmarks and optimizations
• Test sync, stress test networks, etc
• Public and incentivized testnets
• Third party audits
• Polishing the validator user experience

Finally, we owe a special thank you to the ConsenSys team for helping to organize, host and provide resources that made Interop possible.

With The Merge now firmly behind us, protocol developers have been making progress across a (record?) number of areas over the past few months. Withdrawals, danksharding, EOF, verkle tries, history expiry, SSZ and more have all seen significant progress recently!

In order to help move each of these threads forward, and to run through one more series of Shapella stress tests, client team members gathered in person for a week-long interop event in Austria: Edelweiss

Unlike Amphora, which had a singular focus on The Merge, this event had two major tracks, focused on the Shapella and ProtoDanksharding network upgrades respectively. Several breakout sessions were also held to dive into other open problems. Here is a brief overview of what was accomplished, as well as links to artifacts from the workshops & ongoing discussion threads.

Shapella

The week began with a Shanghai/Capella mainnet shadow fork. Flooding the network with withdrawal credential update messages revealed performance issues on the network, and led to a different consensus-layer queueing design to process these messages.

Throughout the week, additional devnets were launched and stress tested with large amounts of credential updates, withdrawals, and even bad blocks. Client implementations ended the week hardened and ready for the fork on the newly-launched Zhejiang testnet.

Assuming the Shapella upgrade happens without issue on Zhejiang, the Sepolia and Goerli testnets will be upgraded next!

(Proto)Danksharding

The main EIP-4844 interop goal was the launch of an all-client EIP-4844 devnet. By Friday, all but one client were syncing on the network!

Several design discussions also took place during the week, stemming from a transaction pool design proposal. Questions around allowing “blobless” 4844 transactions, if and how blocks & blobs should be coupled for gossip and how to encode these transactions were discussed extensively and surfaced on last week’s AllCoreDevs Execution Layer call.

Over the next few weeks, teams hope to finalize all spec changes resulting from these discussions and launch a new devnet.

EVM Object Format (EOF)

After having been conditionally accepted and then removed from Shanghai, EOF was one of the topics where opinions about the best path forward diverged the most.

Whether EOF should ban code introspection, aim for a minimal deployment ASAP, or even only ever go live on L2s were all discussed during the week.

No concrete specification came out of the workshop, but teams now have a shared understanding of the design space and potential paths forward. The EOF breakout rooms resumes next week to continue this conversation!

Everything Else

Aside from these three topics, teams discussed the future of light clients on the network, how the EL & CL specs processes could converge (and potentially carve out ERCs from other EIPs), launched a new Verkle Trie testnet, put forward a proposal to SSZ encode EL transactions, discussed changing the validator EL->CL deposit mechanics, and even started a Capella annotated spec!

Next Steps

Less than a week after the event, client teams have begun discussing Shapella timelines for testnets. Keep an eye out on this blog, as well as on clients’ repositories, for announcements in the coming weeks!

For other efforts, such as EIP-4844, EOF, SSZ, expect to see active design discussions in the coming weeks, leading to prototype implementations afterwards.

Shapella is almost here, and Dencun is clear on the horizon

Last week, Ethereum client, testing and research teams from around the world gathered together for a week of intense work on the Pectra network upgrade, PeerDAS and the Verge.

Similarly to previous events like Edelweiss or Amphora , the focus of the Nyota interop was achieving multi-client interoperability across the different technical tracks.

In addition to this, Nyota was preceded with a first of its kind public event: Frontiers Africa, where core devs & researchers spent the day meeting & exchanging with builders from across the region.

Frontiers Africa

Inspired by the impactful trips that borderless.africa organized in Africa last year, the continent quickly became a top contender to host interop. To maximize the impact of having over 100 core Ethereum contributors in one of the places that can most benefit from it, we organized a one day community event right before interop: Frontiers Africa.

The purpose of the event was to allow L1 maintainers and local builders to learn from each other. After a few morning panels, the agenda was filled with breakout sessions for attendees to take part in two-way conversations on topics ranging from the Ethereum roadmap, to the onboarding journey for core contributors, how to build applications for mass adoption on the African continent and how to bridge the African and global communities.

Frontiers Africa had over 350 local attendees along with interop participants. The event hopefully laid the groundwork for more, larger, Ethereum-focused events taking place across Africa!

Nyota

This year’s interop event focused on three main technical tracks: Pectra, PeerDAS and Verkle. In addition to these, the teams also made progress on EOF, SSZ, History Expiry and more.

Pectra Devnet 0

The main focus of Nyota was cross-client interoperability on devnet-0. Teams were expected to come to interop with full implementations and spend the week gradually building towards multi-client devnets running smoothly.

On the first day, we saw many 1:1 devnets, where specific EL:CL pairs could run together locally. Quickly, more complex devnets started appearing, with a 5×5 one launching mid-week! While in past interop events this would have represented all client combinations, Nyota had two new client teams join for the first time: Grandine and Reth.

By the end of the week, all but one client was running on devnet-0, and external dependencies such as relays were getting tested. As a final test on the last evening of the event, an execution layer-triggered withdrawal was sent over the network, with the validator successfully exiting and the network finalizing smoothly!

Verkle

Prior to interop, Verkle contributors were expected to join the Kaustinen testnet. While this wasn’t achieved by all teams ahead of the event, by the end of the week all but two teams had at least partial implementations, with many also supporting post-transition networks from genesis.

In addition to implementation work, one key goal for the Verkle track was to try and resolve many outstanding spec issues. The various sessions on the topic resulted in a draft PR to the EIP for broader discussion before finalizing the changes.

PeerDAS

Last but not least, the PeerDAS track is likely the one where the most progress was made throughout the interop. At the start of the week, only two of the consensus layer clients had partial implementations.

By Friday, all six teams could distribute and receive data to subnets, all but one could serve the data over RPC, multi-client devnets had been stood up, and some teams had started working on sync integrations!

Everything Else

In addition to the core technical tracks, teams made progress on a range of other initiatives, such as:

…and even an entire SSZ Technical Track!

Next Steps

Expect the notes and draft PRs coming out of Nyota to be refined into formal specs & proposals over the next few weeks. While many ideas were discussed throughout the week, any changes to the Ethereum protocol will be brought up, as always, on AllCoreDevs calls.

Over the next few weeks, teams expect to finalize the scope of the Pectra network upgrade and continue testing their implementations. Once this process has completed, an announcement will be made for the upgrade deployment on existing testnets.

Thank you to everyone who participated in interop and made it both an incredibly productive and fun week. Special shout out to the EthPandaOps, specs and testing teams whose work enabled everyone else to hit the ground running. For Amphora , our interop target was to stand up a single devnet by the end of the week. This time, we launched over 100!