Digest – Earlybirds Invest

January 14th tl;dc (too long, didn’t call)

Disclaimer: This is a digest of the topics discussed in the recurring Eth1.x research call, and doesn’t represent finalized plans or commitments to network upgrades.

The main topics of this call were

Rough data quantifying advantages of switching to a binary trie structure
Transition strategies and potential challenges for a switch to binary tries
“Merklizing” contract code for witnesses, and implications for gas scheduling/metering
Chain pruning and historical chain/state data — network implications and approaches to distribution.

Logistics

The weekend following EthCC (March 7-8), there will be a small 1.x research summit, with the intent of having a few days of solid discussion and work on the topics at hand. The session will be capped (by venue constraints) at 40 attendees, which should be more than enough for the participants expected.

There will also likely be some informal, ad-hoc gathering around Stanford Blockchain week and ETHDenver, but nothing explicitly planned.

The next call is tentatively scheduled for the first or second week in February — half-way between now and the summit in Paris.

Technical discussion

EIP #2465

Although not directly related to stateless ethereum, this EIP improves the network protocol for transaction propagation, and is thus a pretty straightforward improvement that moves things in the right direction for what research is working on. Support!

Binary Trie size savings

Transitioning to a binary trie structure (instead of the current hexary trie structure) should in theory reduce the size of witnesses by something like 3.75x, but in practice that reduction might only be about half, depending on how you look at it..

Witnesses are about 30% code and 70% hashes. Hashes within the trie are reduced by 3x, but code is not improved with a binary trie, since it always needs to be included in the witness. So switching to a binary trie format will bring witness sizes to ~300-1400kB, down from ~800-3,400kB in the hexary trie.

Making the switch

Enacting the actual transition to a binary trie is another matter, with a few questions that need to be fleshed out. There are essentially two different possible strategies that could be followed:

progressive transition — This is a ‘ship of Theseus’ model of transition whereby the entire state trie is migrated to a binary format account-by-account and storageSlot-by-storageSlot, as each part of state is touched by EVM execution. This implies that, forevermore, Ethereum’s state would be a hexary/binary hybrid, and accounts would need to be “poked” in order to be updated to the new trie format (maybe with a POKE opcode ;). The advantages are that this does not interrupt the normal functioning of the chain, and does not require large-scale coordination for upgrading. The disadvantage is complexity: both hexary and binary trie formats need to be accounted for in clients, and the process would never actually “finish”, because some parts of the state cannot be accessed externally, and would need to be explicitly poked by their owners which probably wont happen for the entire state. The progressive strategy would also require clients to modify their database to be a kind of ‘virtualized’ binary trie inside of a hexary database layout, to avoid a sudden dramatic increase in storage requirements for all clients (note: this database improvement can happen independent of the full ‘progressive’ transition, and would still be beneficial alone).

compute and clean-cut — This would be an ‘at once’ transition accomplished over one or more hard-forks, whereby a date in the future would be chosen for the switch, and then all participants in the network would need to recompute the state as a binary trie, and then switch to the new format together. This strategy would be in some sense ‘simpler’ to implement because it’s straightforward on the engineering side. But it’s more complex from a coordination perspective: The new binary trie state needs to be pre-computed before the fork which could take an hour (or thereabouts) — during that window, its not clear how transactions and new blocks would be handled (because they would need to be included in the yet-un-computed binary state trie, and/or the legacy trie). This process would be made harder by the fact that many miners and exchanges prefer to upgrade clients at the last moment. Alternatively we could imagine halting the entire chain for a short time to re-compute the new state — a process which might be even trickier, and potentially controversial, to coordinate.

Both options are still ‘on the table’, and require further consideration and discussion before any decisions are made with regards to next steps. In particular weighing the trade-offs between implementation complexity on one hand and coordination challenges on the other.

Code “chunking”

Addressing the code portion of witnesses, there has been some prototyping work done on code ‘merklization’, which essentially allows contract code to be split up into chunks before being put into a witness. The basic idea being that, if a method in a smart contract is called, the witness should only need to include the parts of the contract code that were actually called, rather than the entire contract. This is still very early research, but it suggests an additional ~50% reduction in the code portion of a witness. More ambitiously, the practice of code chunking could be extended to create a single global ‘code trie’, but this is not a well developed idea and likely has challenges of its own that warrant further investigation.

There are different methods by which code can be broken up into chunks, and then be used to generate witnesses. The first is ‘dynamic’, in that it relies on finding JUMPDEST instructions, and cleaving near those points, which results in variable chunk sizes depending on the code being broken up. The second is ‘static’, which would break up code into fixed sizes, and add some necessary metadata specifying where correct jump destinations are within the chunk. It seems like either of these two approaches would be valid, and both might be compatible and could be left up to users to decide which to employ. Either way, chunking enables a further shrinking of witness sizes.

(un)gas

One open question is what changes would be necessary or desirable in gas scheduling with the introduction of block witnesses. Witness generation needs to be paid for in gas. If the code is chunked, within a block there would be some overlap where multiple transactions cover the same code, and thus parts of a block witness would be paid for more than once by all the included transactions in the block. It seems like a safe idea (and one that would be good for miners) would be to leave it to the poster of a transaction to pay the full cost of their own transaction’s witness, and then let the miner keep the overpayment. This minimizes the need for changes in gas costs and incentivizes miners to produce witnesses, but unfortunately breaks the current security model of only trusting sub-calls (in a transaction) with a portion of the total committed gas. How that change to the security model is handled is something that needs to be considered fully and thoroughly. At the end of the day, the goal is to charge each transaction the cost of producing its own witness, proportional to the code it touches.

Wei Tang’s UNGAS proposal might make any changes to the EVM easier to accomplish. It’s not strictly necessary for stateless Ethereum, but it is an idea for how to make future breaking changes to gas schedules easier. The question to ask is “What do the changes look like both without and with UNGAS — and those things considered, does UNGAS actually make this stuff significantly easier to implement?”. To answer this, we need experiments that run things with merklized code and new gas rules appled, and then see what should change with regard to cost and execution in the EVM.

Pruning and data delivery

In a stateless model, nodes that do not have some or all of the state need a way to signal to the rest of the network what data they have and what data they lack. This has implications for network topology — stateless clients that lack data need to be able to reliably and quickly find the data they need somewhere on the network, as well as broadcast up-front what data they don’t have (and might need). Adding such a feature to one of the chain-pruning EIPs is a networking (but not consensus) protocol change, and its something that also can be done now.

The second side of this problem is where to store the historical data, and the best solution so far proposed is an Eth-specific distributed storage network, that can serve requested data. This could come in many flavors; the complete state might be amenable to ‘chunking’, similar to contract code; partial-state nodes could watch over (randomly assigned) chunks of state, and serve them by request on the edges of the network; clients might employ additional data routing mechanism so that a stateless node can still get missing data through an intermediary (which doesn’t have the data it needs, but is connected to another node that does). However it’s implemented, the general goal is that clients should be able to join the network and be able to get all the data they need, reliably, and without jockying for position connecting to a full-state node, which is effectively what happens with LES nodes now. Work surrounding these ideas is still in early stages, but the geth team has some promising results experimenting with ‘state tiling’ (chunking), and turbo-geth is working on data routing for gossiping parts of state.

As always, if you have questions about Eth1x efforts, requests for topics, or want to contribute, attend an event, come introduce yourself on ethresear.ch or reach out to @gichiba and/or @JHancock on twitter.

February 26th tl;dc (too long, didn’t call)

Disclaimer: This is a digest of the topics discussed in the recurring Eth1.x research call, and doesn’t represent finalized plans or commitments to network upgrades.

The main topics of this call were:

The rough plan for the 1.x research summit in Paris following EthCC
The Witness Format
The ‘data retrieval problem’

Logistics

The summit to discuss and collaborate on Stateless Ethereum is planned for the weekend following EthCC, which will be an indispensable time for working on the most important and unsolved problems for this effort.

The schedule is not fixed yet, but a rough outline is coming together:

Saturday – After an hour of breakfast and free discussion, we’ll come together to agree on goals and scope for the summit. Then there is about 4 hours reserved for organized presentations and ‘deep dives’ on particular topics of importance. In the later afternoon/evening there will be another hour+ of free time and informal discussion.

Sunday – The same as before, but with only 2 hours of structured presentations, to encourage attendees to break out into groups and work on the various research or implementation topics for the rest of the Summit. Finally, there will be a concluding discussion to map out next steps and revise the tech tree.

It should be stated that this research summit is not focused on public or general engagement, in favor of making meaningful progress on the work ahead. This is not meant to be a spectator’s event, and indeed there is some expectation that attendees will have ‘done their homework’ so that the short amount of time for discussion is efficiently spent.

Technical discussion

Witness Format

The first topic of technical discussion was centered around the recently submitted draft witness specification, which will help to define implementation for all client teams.

The witness specification is really comprised of two parts: Semantics and Format. This organization has the desirable property of cleanly separating two aspects of the witness that might have different goals.

Semantics are a bit harder to get to grips with, and are concerned merely with the abstract methods of taking one group of objects and transforming them into other objects. The witness semantics are in simple formal language describing how to get from inputs to outputs, leaving all implementation details abstracted away. For example, questions about data serialization or parsing are not relevant to the witness semantics, as they are more of an implementation detail. The high-level goal of defining the semantics of witnesses in a formal way is to have a completely un-ambiguous reference for client teams to implement without a lot of back-and-forth. Admittedly, starting with formal semantics and working towards implementation (rather than say, coding out a reference implementation) is experimental, but it’s hoped that it will save effort in the long run and lead to much more robust and diverse Stateless Ethereum implementations. Format is much more concrete, and specifies real details that affect interoperability between different implementations.

The witness format is where things like the size of code chunks will be defined, and a good witness format will help different implementations stay inter-operable, and in general terms describes encoding and decoding of data. The format is not specifically geared at reducing witness size, rather at keeping the client implementations memory-efficient, and maximizing the efficiency of generation and transmission. For example, the current format can be computed in real time while walking through the state trie without having to buffer or process whole chunks, allowing the witness to be split into small chunks and streamed.

As a first draft, there is expected to be some refactoring before and after Paris as other researchers give feedback, and already there is a request for a bit more content on design motivations and high-level explanation concerning the above content. It was also suggested in the call that the witness format be written in about in an upcoming “The 1x Files” post, which seems like a great idea (stay tuned for that in the coming weeks).

Transaction validation, an interlude

Moving towards less concrete topics of discussion, one fundamental issue was brought up in the chat that warrants discussion: A potential problem with validating transactions in a stateless paradigm.

Currently, a node performs two checks on all transactions it sees on the network. First, the transaction nonce is checked to be consistent with all transactions from that account, and discarded if it is not valid. Second the account balance is checked to ensure that the account has enough gas money. In a stateless paradigm, these checks cannot be performed by anyone who does not have the state, which opens up a potential vector for attack. It’s eminently possible that the format of witnesses could be made to include the minimum amount of state data required to validate transactions from witnesses only, but this needs to be looked into further.

The transaction validation problem is actually related to a more general problem that Stateless Ethereum must solve, which is tentatively being called “The data retrieval problem”. The solution for data retrieval will also solve the transaction validation problem, so we’ll turn to that now.

Data retrieval in Stateless Ethereum

The full scope of this challenge is outlined in an ethresearch forum post, but the idea relatively straightforward and built from a few assumptions:

It’s possible to, within the current eth protocol, build a stateless client using existing network primitives. This is sort of what beam sync is, with the important distinction that beam sync is meant to keep state data and ‘backfill’ it to eventually become a full node. A stateless client, by contrast, throws away state data and relies entirely on witnesses to participate in the network.

The current protocol and network primitives assume that there is a high probability that connected peers keep valid state, i.e. that connected peers are full nodes. This assumption holds now because most nodes are indeed full nodes with valid state. But this assumption cannot be relied upon if a high proportion of the network is stateless. The current protocol also does not specify a way for a new connected node to see if a connected peer has or does not have a needed piece of state data.

Stateless clients have better UX than full nodes. They will sync faster, and allow for near instantaneous connection to the network. It’s therefore reasonable to assume that over time more and more nodes will move towards the stateless end of the spectrum. If this is the case, then the assumption of data availability will become less and less sound with a higher proportion of stateless nodes on the network. There is a theoretical ‘tipping point’ where stateless nodes outnumber stateful nodes by far, and a random assortment of peers has a sufficiently low probability of at least one holding the desired piece of state. At that (theoretical) point, the network breaks.

The kicker here is that if the network allows state to be gotten on demand (as it does now), a stateless client can (and will) be made on the same protocol. Extending this reasoning to be more dramatic: Stateless clients are inevitable, and the data retrieval problem will come along with them. It follows then, that significant changes to the eth network protocol will need to be made in order to categorically prevent the network from reaching that tipping point, or at least push it further away through client optimizations.

There are a lot of open-ended topics to discuss here, and importantly there is disagreement amongst the 1x researchers about exactly how far the network is from that theoretical breaking point, or if the breaking point exists at all. This highlights the need for more sophisticated approaches to network simulation, as well as the need for defining the problem clearly at the research summit before working towards a solution.

À tout à l’heure !

Exciting things will undoubtedly be unfolding as a result of the in-person research to be conducted in Paris in the coming fortnight, and the next few installments of “The 1.x Files” will be devoted to documenting and clearly laying out that work.

The summit in Paris is very nearly at full capacity, so if you have not filled out the RSVP form to attend please get in touch with Piper to see if there is space.

As always, if you’re interested in participating in the Stateless Ethereum research effort, come join us on ethresear.ch, get invited to the telegram group, and reach out to @gichiba and/or @JHancock on twitter.

XRP goes big in New York

Star investor Ray Dalio reveals why crypto is real alternative to dollar

Dogecoin rockets 617% in liquidation imbalance as DOGE bulls take $7 million hit

Shiba Inu forms first 2025 golden cross

Bitcoin faces double-top risk after whale sell-off

Dogecoin whale pulls $12M from Binance

SHIB whale activity surges

ETH holders in full profit

Coinbase XRP reserves down 57.4% since June

January 14th tl;dc (too long, didn’t call)

Logistics

Technical discussion

EIP #2465

Binary Trie size savings

Making the switch

Code “chunking”

(un)gas

Pruning and data delivery

February 26th tl;dc (too long, didn’t call)

Logistics

Technical discussion

Witness Format

Transaction validation, an interlude

Data retrieval in Stateless Ethereum

À tout à l’heure !

Top Stories of The Week

SEC Chair Aktins calls to ‘reshore crypto’ as companies move back to the US

US SEC rolls out ‘Project Crypto’ to rewrite rules for digital assets

99% of CFOs plan to use crypto long term, 23% within two years: Deloitte

UK regulator lifts ban on crypto ETNs for retail investors

CoinDCX employee arrested in connection with $44M crypto hack: Report

Winners and Losers

Memorable Quotations

Top Prediction of The Week

XRP’s ‘bullish divergence’ raises 20% price rally potential in August

Top FUD of The Week

Crypto hacks top $142M in July, with CoinDCX leading losses

Tornado Cash co-founder faces jury after closing arguments wrap

Indonesia raises taxes on crypto exchange sales and miners

Top Magazine Stories of The Week

Ethereum’s roadmap to 10,000 TPS using ZK tech: Dummies’ guide

China mocks US crypto policies, Telegram’s new dark markets: Asia Express

Training AI to secretly love owls… or Hitler. Meta + AI porn? AI Eye

Top Stories of The Week

Australian fintech Finder wins court battle over crypto yield product

BitMine gobbles over $2B in ETH in 16 days amid treasury arms race

XRP price drops 19% but analysts say it’s a ‘healthy correction’

Hulk Hogan, Ozzy memecoins soar as tributes roll in over icons’ deaths

Ether will ‘knock on $4,000’ and soon outperform Bitcoin: Novogratz

Most Memorable Quotations

Winners and Losers

Top Prediction of The Week

Eric Trump ‘agrees’ Ether should be over $8K as Global M2 money soars

Top FUD of The Week

Arizona woman sentenced for helping North Korea coders get US crypto jobs

Bitcoin tumbles below $116K in bloodbath for crypto longs

New York crypto torture case suspects out on $1M bail each

Top Magazine Stories of The Week

Robinhood’s tokenized stocks have stirred up a legal hornet’s nest

Hong Kong hoses down stablecoin frenzy, Pokémon on Solana: Asia Express

Bitcoin inheritances: A guide for heirs and the not-yet-dead

Crypto market is red today

Aave DAO Integrates Chainlink SVR on Ethereum Mainnet

The Sandbox Alpha Season 5 Starts Monday, Includes Attack on Titan, Jurassic World, Teletubbies, Atari, Hellboy, Terminator, and More

Decentralized Storage Platform Walrus Launches Mainnet, Welcomes Claynosaurz

__________

À tout à l’heure !