After waaay too much time under development, we’re proud to finally announce version 1.8.0 of the Go Ethereum client: Iceberg! The release fixes a lot of pain points felt by the community and ships a few notable new features, tallying up to ~170 modifications!

Please note, this release introduces a few breaking changes that may affect certain power users! If you are running a production setup, make sure you read the “Breaking changes” section at the end of this blog post!

Client synchronization

A huge amount of work went into this release that isn’t immediately visible, rather they’re under the hood changes to make everybody’s life just a little bit more pleasant. We’ve tried to address many of the issues our users were reporting around syncing and block processing. We’re not quite where we’d like to be, but the experience with v1.8.0 should blow all previous releases out of the water.

Reliable light client

Geth v1.7.3 – released shortly after Devcon3 – was the first release to ship version 2 of the light client protocol. It was meant to be a huge improvement over version 1, finally enabling log filtering from Ethereum contracts. It broke the light client.

The breakage was massive, with multiple experimental protocols (discovery v5, light client v2) playing badly with each other. Geth v1.7.3 tried to advertise both les/1 and les/2, which conflicted in the discovery, breaking both; les/2 servers would crash serving some light client requests; and discovery v5, running behind an undocumented port, didn’t help either.

Geth v1.8.0 tries to pick up all the pieces and make les/2 what it was supposed to be in v1.7.3. We’ve dropped support for les/1 in the discovery, so there should be no more problems finding peers while we iron out the kinks. Light servers have been polished up to be more robust with existing connections, as well as extended to cleanly separate eth and les peers, preventing server side starvation. Version 4 and 5 of the discovery protocols are also running on the same port, and will from now on better avoid issues with firewalls or NAT traversals.

With all of the above changes, the light client in v1.8.0 should find servers within a few seconds from startup, and synchronizing the mainnet should finish within a minute. Since light clients rely on charitable nodes serving them, we ask anyone running non-sensitive full nodes with spare capacity to consider enabling the light server to help people with less capable hardware.

Reliable fast sync

For a long time now we’ve been receiving reports from users experiencing fast sync hangs with a “stalling peer” error message, or that trying to synchronize on an average machine often crashes with an “out of memory” error. These issues have become more and more prevalent as the Ethereum mainnet grew, yet they have been elusive to us due to their rare occurrence.

The heavy internal rewrites allowed us to reliably reproduce and fix these issues. The hang was a very rare race that occurred when state sync restarted; the fix for which is amusing given that it took us a year to catch. The memory issue was also fixed by aggressively capping the amount of memory that sync may consume.

The final result of these optimizations is that fast sync became stable again. From one perspective there are no more hangs, so you don’t have to constantly monitor the sync progress. From the other perspective memory usage is constant, so there’s no need for machines with insane RAM.

The above chart plots the memory usage during mainnet fast sync of two m4.2xlarge Amazon instance types (purple = Geth 1.8, blue = Geth 1.7). At the time of writing, fast sync completes in around 3 hours on these instance types. The exponential growth of Ethereum however resulted in a state trie of around 85 million nodes, the import of which can take even half a day on end-user laptops (with an SSD). Hopefully 1.9 will tackle this issue.

Initial state pruning

Ethereum organizes its state into a gigantic trie data structure. At the bottom – in the leaves we have the accounts – and on top of the accounts we have an 16th order Merkle trie cryptographically guaranteeing forgery resistance. We have one of these giant tries for each and every block, the latest of which weighing at around 85 million nodes. Most of these nodes are common between subsequent blocks, but every new block does add a few thousand new nodes into the trie.

If we would like to know what our balance was years ago, we’d have to maintain every single version of this Merkle trie since the genesis block, which could total to almost 1TB of data currently. In reality almost nobody cares about historical data – as long as it can be recomputed – rather only about the recent state of the network. Fast sync gets you “quickly” to the recent state, but blindly piling blocks on top will forever use more and more disk space.

The important property of the Merkle tries to be aware of is that whilst every new block adds thousands of new nodes, thousands of old ones become obsolete at the same time. If we could easily delete these obsolete ones, disk growth would be significantly capped. However, once the data is on disk, it’s extremely expensive to get rid of them.

Geth v1.8.0 takes an initial stab at the problem by introducing an in-memory cache in which to store the recent trie nodes. As long as the nodes are in memory, they are cheap to reference count and garbage collect. Instead of writing each trie node to disk, we keep it around as long as possible, hoping that a future block will make it obsolete and save us a database write.

Geth v1.8.0 by default will use 25% of the user’s cache allowance (–cache) for trie caching and will flush to disk either if the memory allowance is exceeded, or if block processing time since the last flush exceeds 5 minutes. This doesn’t completely solve database growth just yet, but looking at the disk stats between v1.8 (purple) and v1.7 (blue) in the course of a single week, pruning makes a huge difference.

Transaction tracing

Pretty much since forever, Geth supported tracing transactions by dumping the executed opcodes. These dumps can be invaluable for finding consensus issues among clients, but they aren’t the nicest to look at. Although post-processing these traces is possible, it’s a waste of resources to collect so much data just to throw most of it away.

Custom tracing scripts

The v1.5 release family of Geth introduced a new way to trace transactions by allowing users to write custom JavaScript scripts that run within the node while tracing. Instead of producing pre-defined traces, users could gather whatever data they deemed useful without having to export everything else. Although we did use it internally, the feature never really graduated to a useful and robust enough state for wide spread use.

Geth v1.8.0 however completely revamps the custom tracing support. For starters, we’ve replaced the ottovm we used previously to run the tracers, to duktape, resulting in a 5x speed increase. We no longer require the state upon which a transaction relies to be present to trace it, rather the tracer can reconstruct anything missing from historical states (bearing the cost of re-executing the blocks in memory). Furthermore, when tracing multiple transactions at once (i.e. an entire block), those are executed concurrently, slashing tracing time by the number of available CPU cores.

All said and done, writing a custom tracer is complicated, taking up a significant time even for veteran Ethereum developers. As such, we’ve made the decision to provide a few tracers out of the box for users to use, and potentially improve. We eagerly await any community improvements to these, or even the addition of brand new ones!

• The callTracer is a full blown transaction tracer that extracts and reports all the internal calls made by a transaction, along with any information deemed useful.
• The prestateTracer outputs sufficient information to create a local execution of the transaction from a custom assembled genesis block.
• The 4byteTracer searches for 4byte-identifiers, and collects them for post-processing. It collects the methods identifiers along with the size of the supplied data, so a reversed signature can be matched against the size of the data.

E.g. executing the callTracer against the same transaction linked above gets us a much much friendlier output debug.traceTransaction(“0xhash”, {tracer: “callTracer”}).

Streaming chain tracers

Tracing an entire block of transactions is a lot more optimal than tracing transactions one-by-one, because we don’t need to generate the pre-state for each one individually. This holds true even more strongly if generating the starting state entails re-executing multiple past blocks (pruned state). The same issue however arises when tracing multiple blocks too: if the pre-state was pruned, it’s a waste to throw away regenerated state just to do it all over for the next block.

To cater for tracing multiple subsequent blocks with minimal overhead, Geth v1.8.0 introduces a new API endpoint that can trace chain segments. This endpoint can reuse the computed states in between blocks without rerunning transactions over and over again. What’s more, individual blocks are traced concurrently, so total tracing time gets proportionally lower the more CPU cores you throw at it.

Tracing a transaction or a block takes a relatively short amount of time. Tracing a chain segment however can take arbitrarily long, depending on how long the chain is and what transactions are included in it. It would be very impractical to wait for all the transactions to be traced before starting to return the ones already done. This rules out chain tracing as a simple RPC method. Instead, Geth v1.8.0 implements chain tracing via a subscription (IPC/WebSocket), where the user starts a background tracing process and Geth will stream the results until all transactions are traced:

$ nc -U /work/temp/rinkeby/geth.ipc
{"id": 1, "method": "debug_subscribe", "params": ["traceChain", "0x0", "0xfff", {"tracer": "callTracer"}]}

{"jsonrpc":"2.0","id":1,"result":"0xe1deecc4b399e5fd2b2a8abbbc4624e2"}
{"jsonrpc":"2.0","method":"debug_subscription","params":{"subscription":"0xe1deecc4b399e5fd2b2a8abbbc4624e2","result":{"block":"0x37","hash":"0xdb16f0d4465f2fd79f10ba539b169404a3e026db1be082e7fd6071b4c5f37db7","traces":[{"from":"0x31b98d14007bdee637298086988a0bbd31184523","gas":"0x0","gasUsed":"0x0","input":"0x","output":"0x","time":"1.077µs","to":"0x2ed530faddb7349c1efdbf4410db2de835a004e4","type":"CALL","value":"0xde0b6b3a7640000"}]}}}
{"jsonrpc":"2.0","method":"debug_subscription","params":{"subscription":"0xe1deecc4b399e5fd2b2a8abbbc4624e2","result":{"block":"0xf43","hash":"0xacb74aa08838896ad60319bce6e07c92edb2f5253080eb3883549ed8f57ea679","traces":[{"from":"0x31b98d14007bdee637298086988a0bbd31184523","gas":"0x0","gasUsed":"0x0","input":"0x","output":"0x","time":"1.568µs","to":"0xbedcf417ff2752d996d2ade98b97a6f0bef4beb9","type":"CALL","value":"0xde0b6b3a7640000"}]}}}
{"jsonrpc":"2.0","method":"debug_subscription","params":{"subscription":"0xe1deecc4b399e5fd2b2a8abbbc4624e2","result":{"block":"0xf47","hash":"0xea841221179e37ca9cc23424b64201d8805df327c3296a513e9f1fe6faa5ffb3","traces":[{"from":"0xbedcf417ff2752d996d2ade98b97a6f0bef4beb9","gas":"0x4687a0","gasUsed":"0x12e0d","input":"0x6060604052341561000c57fe5b5b6101828061001c6000396000f30060606040526000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff168063230925601461003b575bfe5b341561004357fe5b61008360048080356000191690602001909190803560ff1690602001909190803560001916906020019091908035600019169060200190919050506100c5565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b6000600185858585604051806000526020016040526000604051602001526040518085600019166000191681526020018460ff1660ff1681526020018360001916600019168152602001826000191660001916815260200194505050505060206040516020810390808403906000866161da5a03f1151561014257fe5b50506020604051035190505b9493505050505600a165627a7a7230582054abc8e7b2d8ea0972823aa9f0df23ecb80ca0b58be9f31b7348d411aaf585be0029","output":"0x60606040526000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff168063230925601461003b575bfe5b341561004357fe5b61008360048080356000191690602001909190803560ff1690602001909190803560001916906020019091908035600019169060200190919050506100c5565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b6000600185858585604051806000526020016040526000604051602001526040518085600019166000191681526020018460ff1660ff1681526020018360001916600019168152602001826000191660001916815260200194505050505060206040516020810390808403906000866161da5a03f1151561014257fe5b50506020604051035190505b9493505050505600a165627a7a7230582054abc8e7b2d8ea0972823aa9f0df23ecb80ca0b58be9f31b7348d411aaf585be0029","time":"658.529µs","to":"0x5481c0fe170641bd2e0ff7f04161871829c1902d","type":"CREATE","value":"0x0"}]}}}
{"jsonrpc":"2.0","method":"debug_subscription","params":{"subscription":"0xe1deecc4b399e5fd2b2a8abbbc4624e2","result":{"block":"0xfff","hash":"0x254ccbc40eeeb183d8da11cf4908529f45d813ef8eefd0fbf8a024317561ac6b"}}}

Native events

For about one and a half years now we’ve supported generating Go wrappers for Ethereum contracts. These are extremely useful as they allow calling and transacting with contracts directly using Go. The main benefit is that our abigen tool generates static types for just about everything, ensuring that code interacting with contracts is compile-time type safe. It’s very useful during development too, as any contract ABI change immediately produces compilation errors, eliminating most runtime failures.

That being said, abigen was always lacking support for Ethereum contract log filtering: you couldn’t filter past events, and you couldn’t subscribe to future events. Geth v1.8.0 finally lands event filtering for native dapps! Go wrappers generated by abigen from now on will contain two extra methods for each event, FilterMyEvent and WatchMyEvent. Adhering to abigen‘s strict type safety, both event filters and returned logs are strongly and statically typed. Developers only need to work with Go types, and everything else gets taken care of under the hood.

A nice example is filtering for Akasha posts on the Rinkeby test network. The publishing event is defined as event Publish(address indexed author, bytes32 indexed entryId). Filtering for posts created by addresses 0xAlice or 0xBob would look like:

contract.FilterPublish(nil, []common.Address{"0xAlice", "0xBob"}, nil)

Devcon3 puppeth

As many of you probably know, the Rinkeby test network is almost fully managed via puppeth. For those who don’t, puppeth is “a tool to aid you in creating a new Ethereum network down to the genesis block, bootnodes, signers, ethstats server, crypto faucet, wallet browsers, block explorer, dashboard and more; without the hassle that it would normally entail to manually configure all these services one by one”.

Puppeth was an invaluable tool for us in maintaining the Rinkeby network since its creation 10 months ago. It was fit for its purpose – as an internal tool – alas it had a lot of rough edges. We wanted to make this tool useful not just for Rinkeby, rather for all other developer networks out there too, so for Devcon3 we’ve heavily polished it. It became user friendly(-er), it gained support for configuring Parity, C++ Ethereum, pyethapp and Harmony (on ethash consensus) and it could deploy online wallets and basic block explorers too.

It seems to have been ages since Devcon3 and Puppeth being merged on master, but v1.8.0 finally ships the next incarnation of puppeth for those who have been holding out. Go on and deploy your own Ethereum network!

Breaking changes

• Discovery v4 and v5 have been merged to use the same UDP port (30303 by default). If you are doing manual peer management and using the light client, you may need to ensure your v1.8.0 clients are pointed to port 30303 and not 30304 as previously.
• Trie pruning is enabled on all –syncmode variations (including –syncmode=full). If you are running an archive node where you would like to retain all historical data, you should disable pruning via –gcmode=archive.
• Only the latest 128 tries are kept in memory, most tries are garbage collected. If you are running a block explorer or other service relying on transaction tracing without an archive node (–gcmode=archive), you need to trace within this window! Alternatively, specify the reexec: 12345 tracer option to allow regenerating historical state; and ideally switch to chain tracing which amortizes overhead across all traced blocks.
• Native events rely on modifications to internal go-ethereum types within generated code. If you are using wrappers generated prior to v1.8.0, you will need to regenerate them to be compatible with the new code base.
• The HTTP/WS RPC endpoint was extended with DNS rebind protection. If you are running an RPC endpoint addressed by name rather than IP, run with –rpcvhosts=your.domain to continue accepting remote requests.

Although we consider Geth 1.8.0 our best release yet, we urge everyone to exercise caution with the upgrade and monitor it closely afterwards as it does contain non-trivial changes. We would also like to emphasize that Geth 1.8.0 introduces state pruning, which is backward incompatible with previous versions of Geth (old versions reject the pruned database).

As with previous large releases, our recommendation for production users it to sync from scratch, and leave the old database backed up until you confirm that the new release works correctly for all your use cases.

For a full rundown of the changes please consult the Geth 1.8.0 release milestone.

Binaries and mobile libraries are available on our download page.

Acknowledgement

As a final note for this release, we’d like to shout out to Ming Chan for all of her insanely hard work as the previous EF Executive Director! Among her multitude of tasks, she always found the time to proof-read our release posts, correcting any lost-in-translation errors; whilst also ensuring clarity for our less technical readers. Thank you for everything you did for the Foundation and the community!

¹ “Because the previous version was un-sync-able” ~Nick Johnson

After many months of silence, we’re proud to announce the v1.9.0 release of Go Ethereum! Although this release has been in the making for a lot longer than we anticipated, we’re confident there will be some juicy feature for everyone to enjoy!

Warning: We’ve tried our best to squash all the bugs, but as with all major releases, we advise everyone to take extra care when upgrading. The v1.9.0 release contains database schema changes, meaning it’s not possible to downgrade once updated. We also recommend a fresh fast sync as it can drastically reduce the database size.

Some of the features mentioned here have been silently shipped over the course of the 1.8.x release family, but we’ve deemed them important enough to explicitly highlight.

Performance

It’s interesting to realize that the “Performance” section was somewhere at the end of previous announcements, but over the years it became one of the most sought after improvement.

Over the past 6 months, we’ve tried to dissect the different components that are on the critical path of block processing, in an attempt to identify and optimize some of bottlenecks. Among the many improvements, the highest impact ones were:

• The discovery and optimization of a quadratic CPU and disk IO complexity, originating from the Go implementation of LevelDB. This caused Geth to be starved and stalled, exponentially getting worse as the database grew. Huge shoutout to Gary Rong for his relentless efforts, especially as his work is beneficial to the entire Go community.
• The analysis and optimization of the account and storage trie access patterns across blocks. This resulted in stabilizing Geth’s memory usage even during the import of the Shanghai DoS blocks and speeding up overall block processing by concurrent heuristic state prefetching. This work was mostly done by Péter Szilágyi.
• The analysis and optimization of various EVM opcodes, aiming to find outliers both in Geth’s EVM implementation as well as Ethereum’s protocol design in general. This led to both fixes in Geth as well as infos funneled into the Eth 1.x scaling discussions. Shoutout goes to Martin Holst Swende for pioneering this effort.
• The analysis and optimization of our database schemas, trying to both remove any redundant data as well as redesign indexes for lower disk use (sometimes at the cost of a slight CPU hit). Props for these efforts (spanning 6-9 months) go to Alexey Akhunov, Gary Rong, Péter Szilágyi and Matthew Halpern.
• The discovery of a LevelDB compaction overhead during the state sync phase of fast sync. By temporarily allocating pruning caches to fast sync blooms, we’ve been able to short circuit most data accesses in-memory. This work was mostly done by Péter Szilágyi.

[TL;DR] Fast sync

We’ve run a fast sync benchmark on two i3.2xlarge AWS EC2 instances (8 core, 61 GiB RAM, 1.9 TiB NVMe SSD) with –cache=4096 –maxpeers=50 (defaults on v1.9.0) on the 25th of April.

[TL;DR] Full sync

We’ve run a full sync benchmark on two i3.2xlarge AWS EC2 instances (8 core, 61 GiB RAM, 1.9 TiB NVMe SSD) with –cache=4096 –maxpeers=50 –syncmode=full.

*Whilst the performance is similar, we’ve achieved that while reducing the memory use by about 1/3rd and completely removing spurious memory peaks (Shanghai DoS). The reason for the higher disk IO is due to using less memory for caching, having to push more aggressively to disk.

[TL;DR] Archive sync

We’ve run an archive sync benchmark on two m5.2xlarge AWS EC2 instances (8 core, 32 GiB RAM, 3TiB EBS SSD) with –cache=4096 –syncmode=full –gcmode=archive.

* EBS volumes are significantly slower than physical SSDs attached to the VM. Better performance can be achieved on VMs with real SSDs or actual physical hardware.

Freezer

Wouldn’t it be amazing if we didn’t have to waste so much precious space on our expensive and sensitive SSDs to run an Ethereum node, and could rather move at least some of the data onto a cheap and durable HDD?

With the v1.9.0 release, Geth separated its database into two parts (done by Péter Szilágyi, Martin Holst Swende and Gary Rong):

• Recent blocks, all state and accelerations structures are kept in a fast key-value store (LevelDB) as until now. This is meant to be run on top of an SSD as both disk IO performance is crucial.
• Blocks and receipts that are older than a cutoff threshold (3 epochs) are moved out of LevelDB into a custom freezer database, that is backed by a handful of append-only flat files. Since the node rarely needs to read these data, and only ever appends to them, an HDD should be more than suitable to cover it.

A fresh fast sync at block 7.77M placed 79GB of data into the freezer and 60GB of data into LevelDB.

Freezer basics

By default Geth will place your freezer inside your chaindata folder, into the ancient subfolder. The reason for using a sub-folder was to avoid breaking any automated tooling that might be moving the database around or across instances. You can explicitly place the freezer in a different location via the –datadir.ancient CLI flag.

When you update to v1.9.0 from an older version, Geth will automatically being migrating blocks and receipts from the LevelDB database into the freezer. If you haven’t specified –datadir.ancient at that time, but would like to move it later, you will need to copy the existing ancient folder manually and then start Geth with –datadir.ancient set to the correct path.

Freezer tricks

Since the freezer (cold data) is stored separately from the state (hot data), an interesting question is what happens if one of the two databases goes missing?

• If the freezer is deleted (or a wrong path specified), you essentially pull the rug from underneath Geth. The node would become unusable, so it explicitly forbids doing this on startup.
• If, however, the state database is the one delete, Geth will reconstruct all its indices based on the frozen data; and then do a fast sync on top to back-fill the missing state.

Essentially, the freezer can be used as a guerrilla state pruner to periodically get rid of accumulated junk. By removing the state database, but not the freezer, the node will do a fast sync to fetch the latest state, but will reuse all the existing block and receipt data already downloaded previously.

You can trigger this via geth removedb (plus the –datadir and –datadir.ancient flags if you used custom ones); asking it to only remove the state database, but not the ancient database.

Be advised, that reindexing all the transactions from the ancient database can take over an hour, and fast sync will only commence afterwards. This will probably be changed into a background process in the near future.

GraphQL

Who doesn’t just love JSON-RPC? Me!

As its name suggests, JSON-RPC is a *Remote Procedure Call* protocol. Its design goal is to permit calling functions, that do some arbitrary computation on the remote side, after which they return the result of said computation. Of course – the protocol being generic – you can run data queries on top, but there’s no standardized query semantic, so people tend to roll their own.

Without support for flexible queries however, we end up wasting both computational and data transfer resources:

• RPC calls that return a lot of data (e.g. eth_getBlock) waste bandwidth if the user is only interested in a handful of fields (e.g. only the header, or even less, only the miner’s address).
• RPC calls that return only a bit of data (e.g. eth_getTransactionReceipt) waste CPU capacity if the user is forced to repeat the call multiple times (e.g. retrieving all receipts one-by-one results in loading all of them from disk for each call).

In the case of Ethereum’s JSON-RPC API, the above issues get exacerbated by the mini-reorg nature of the blockchain, as doing multiple queries (e.g. eth_getBalance) need to actually ensure that they execute against the same state and even against the same node (e.g. load balanced backends might have slight sync delays, so can serve different content).

Yes, we could invent a new, super optimal query mechanism that would permit us to retrieve only the data we need, whilst minimizing computational and data transfer overhead… or we could also not-reinvent the wheel (again) and rather use one that’s been proven already: GraphQL.

Querying with GraphQL

First thing’s first, a huge shoutout goes to Raúl Kripalani, Kris Shinn, Nick Johnson, Infura and Pegasys, for pioneering both the GraphQL spec and its implementation, as well as to Guillaume Ballet for doing the final integrations!

Geth v1.9.0 introduces native GraphQL query capabilities via the –graphql CLI flag. GraphQL itself being a protocol on top of HTTP, the same suite of sub-flags (restrictions, CORS and virtual hosts rules) are available as for HTTP RPC. But enough of this yada-yada, let’s see it!

For a quick spin, lets try and find all the ENS domain registrations on the Görli testnet! Start Geth v1.9.0 on Görli with GraphQL enabled (geth –goerli –graphql), wait until it syncs (should be 1-2 minutes tops) and point your browser to the… gasp… built in GraphQL explorer at http://localhost:8547!

To keep things simple, here’s a quick sample query that finds the ENS HashRegistered events and returns the address of the user doing the registration along with the block number and timestamp it was included in:

Albeit the example is a bit contrived and simplistic, it does highlight that GraphQL allows us to answer complex “join queries” that previously required many RPC calls and returned a lot more data than actually needed.

As with all epic developer tools, Geth v1.9.0’s GraphQL explorer has integrated code completion, field documentation and live query execution! Go and query something awesome!

Hardware wallets

Geth already supported certain hardware wallets in the past, but with the v1.9.0 release, we’ve upped our game and made that list a lot more extensive!

Ledger wallets

We’ve already supported the Ledger Nano S for a couple years now, but Geth v1.9.0 also introduces native support for the Ledger Nano X (via USB)!

Additionally, v1.9.0 replaces the default HD derivation path from the legacy one, that Ledger originally advertised, to the canonical one, used by all Ethereum wallets (and presently by Ledger too). Don’t worry, Geth will find all your old accounts too, just will use the canonical path for new accounts! This work was done by Péter Szilágyi.

If you haven’t used a Ledger through Geth until now, the workflow is:

• Plug in your Ledger Nano S or Ledger Nano X and unlock via your PIN code.
• Start the Ethereum app on your Ledger (Geth will log Ethereum app offline).
• You can list all your accounts via personal.listWallets from the Geth console.
  
  • This will auto-derive any accounts that you’ve used before + 1 empty new one.
  • Alternatively you can do the same thing through RPC via personal_listWallets.

• Transact via your preferred means and Geth will forward the signing request to the Ledger.

Linux users be aware, you need to explicitly permit your user to access your Ledger wallet via udev rules!

Trezor wallets

For almost two years now we’ve supported the Trezor One. Unfortunately a firmware update (v1.7.0+) changed the USB protocol in a backwards incompatible way. Although we recommend everyone use the latest software when it comes to security, we also acknowledge the reluctance of regularly updating firmware on a cold storage device.

As such, Geth v1.9.0 implements the new WebUSB protocol supporting updated Trezor One models, but at the same time keeps support for the old USB HID protocol too for non-updated devices. This support was added by Guillaume Ballet and Péter Szilágyi (we’ve even published a new usb library for Go to support it).

The Trezor One workflow is a bit more complex due to the unique PIN entry:

• Plug in your Trezor One, Geth will detect it but will prompt you to open it.
• Call personal.openWallet(‘trezor://…’) with the URL of the device.
  
  • If you don’t know the URL, you can check via personal_listWallets.
  • The console will keep prompting for PIN entry and password as needed.
  • Calling via RPC, openWallet returns a detailed error if it needs another call.

• You can list all your accounts via personal.listWallets from the Geth console.
  
  • This will auto-derive any accounts that you’ve used before + 1 empty new one.
  • Alternatively you can do the same thing through RPC via personal_listWallets.

• Transact via your preferred means and Geth will forward the signing request to the Trezor.

In addition to extended support for the Trezor One, Geth v1.9.0 also introduces native support for the Trezor Model T. The Model T’s workflow is a bit simpler as the PIN entry is done on device:

• Plug in your Trezor Model T and unlock via your PIN code, Geth should detect it.
• You can list all your accounts via personal.listWallets from the Geth console.
  
  • This will auto-derive any accounts that you’ve used before + 1 empty new one.
  • Alternatively you can do the same thing through RPC via personal_listWallets.

• Transact via your preferred means and Geth will forward the signing request to the Trezor.

Linux users be aware, you need to explicitly permit your user to access your Trezor wallet via udev rules!

Status keycards

Prototyped more than a year ago, Geth v1.9.0 finally ships support for the Status keycard, a full HD hardware wallet based on Java SmartCards. The Status keycard can be used via Geth only through the PC/SC daemon for now (you need to install it) and via USB (the +iD is a good USB smartcard reader). This work was heavy lifted by Nick Johnson, initially integrated by Péter Szilágyi and finalized by Guillaume Ballet (and of course Andrea Franz and the rest of the Status team).

If you already have an initialized Status keycard, the Geth workflow is:

• Plug in your Status keycard via a USB card reader.
• Check the status of your card via personal_listWallets.
• Permit Geth to use the card via personal.openWallet(‘keycard://…’).
  
  • The very first time Geth will ask you to pair your card via the passphrase.
  • In normal operation, Geth will ask you to unlock your card via your PIN code.
  • On too many wrong PINs, Geth will ask you to reset your card via your PUK code.
  • On too many wrong PUKs, your card will be bricked and you’ll need to reinstall it.
  • Alternatively you can do the same thing through RPC via multiple personal_openWallet().

• Transact via your preferred means and Geth will forward the signing request to the Status keycard.

If you don’t have a pre-initialized Status keycard; are using a developer card; or managed to brick your existing card (hey, we’re developers, we must know what happens then), you can follow our technical guide on how to wipe your keycard and reinitialize it. Note, you will lose your private key on a wipe.

Clef

Wallets, wallets everywhere!

When Ethereum launched in 2015, there was no third party tooling whatsoever, so client implementations needed to be these all-encompassing Swiss army knives. Ranging from peer-to-peer networking, through account management, to contract and user interactions, everything was done by the client. This was necessary, but seriously sub-optimal: accounts and networking don’t go well together security wise, and everything done by a single binary doesn’t permit a composable ecosystem.

We’ve been wanting to do this for at least 2 years now, and Geth v1.9.0 finally ships the work of Martin Holst Swende (with the help of many others): a standalone signer for the entire Ethereum ecosystem called Clef. As simple as a “standalone signer” might sound, Clef is the result of an insane amount of architectural work to make it secure, flexible and composable.

A small release blog post simply cannot do this project justice, but we’ll try nonetheless to at least mention the major features of Clef, the design decisions behind them and how they can enable a whole set of new use cases.

Ecosystem composability

The main reason for creating Clef was to remove account management from Geth (don’t worry, the old way will still work for the foreseeable future). This permits Geth to be an “insecure” network gateway into Ethereum, which should solve many many issues with regard to accidentally exposing accounts via RPC (and unlocking them, the deadly combo).

But hogging all this work for Geth wouldn’t be nice of us. Instead, we designed Clef to be usable by arbitrary programs, so that you can have a single signer securely managing your keys, to which arbitrary applications (e.g. Geth, Parity, Trinity, Metamask, MyCrypto, Augur) can send signing requests to!

To achieve this, Clef exposes a tiny external API (changelog) either via IPC (default) or HTTP. Any program that can access these endpoints (e.g. Geth via IPC, Metamask via HTTP) can send signing requests to Clef, which will prompt the user for manual confirmation. The API is deliberately tiny and uses JSON-RPC, so it should be trivial to support in any project.

Our goal with Clef is not to be “The Geth Signer”, rather we’d like it to become a standalone entity that can be used by any other project, be it different client implementations (Trinity), browser integrations (Metamask), service components (Raiden) or decentralized applications (Augur). If you’d like to integrate Clef, reach out and we’ll HEPL!

Pluggable interface

What is the perfect user interface?

If you ask me, I’d say command line: simple, works over SSH, and I can code it :D. But I’m a minority here and even I often prefer a proper UI. So, Electron? Some think it’s the best thing since sliced bread and many developers can code it; but it’s large and slow and JavaScript :P. How about Qt? It’s cross platform, tiny and fast, but not many developers are familiar with it and it has a funky license. Android, GTK, iThingy?… Win32 ?

The answer is all of them! The perfect UI depends on what you want to use it for, and we don’t want to make that choice for you, rather allow you to run Clef the way it fits best into your life:

• If you are on the move all the time, you may prefer an Android or iOS interface.
• If you have a locked down remote server, you may prefer CLI on top of SSH.
• If you have a powerful laptop, the beauty of Electron might be just the thing.
• If you have an offline signer machine, a Qt UI might be simple, but enough.
• If you are a bank, you might want a custom integration into your own infra.

We can’t implement all this. But you can! We’ve designed Clef with enough flexibility to allow anyone to implement a custom UI on top, without needing to touch Clef itself, or know any Go at all. The goal is to provide a building block to the community so that designers and UI developers can do what they’re good at, without having to worry about cryptography and stuff.

To achieve this, Clef exposes an extended internal API (changelog), solely via standard input/output. Any user interface is meant to start itself up and internally start an instance of Clef, binding to it’s IO streams. The IO streams speak JSON-RPC, so the UI can send arbitrary trusted requests to Clef, and Clef will send notifications and confirmation prompts to the UI.

Clef itself ships with a built in CLI interface (otherwise it’s not much useful) and we’ve prepared an entire Quickstart Guide to familiarize yourself with the general features and concepts. There are also various proof-of-concept UIs that we’ve used to validate architectural decisions, but to get a solid UI, we need the community, as we don’t have the knowledge ourselves!

Integrated 4bytes

You probably figured out the general direction by now. We want Clef to be a reusable piece of puzzle. The trick is to make it the right size! Too much functionality baked in (e.g. fixed UI), and possible uses get limited. Too few (e.g. no hardware wallet) and UI builders reinvent the wheel. It’s a delicate balance of maximizing utility and security without compromising flexibility.

So, we agree that “fixed UI bad, pluggable UI good”, “no hardware wallet bad, Ledger + Trezor + Keycard good”. What else do wallet implementations reinvent all the time? 4bytes!

In Ethereum, whenever a user interacts with a contract, they send a giant blob of binary data, encoded in a very specific ABI format. This is needed so that the EVM can make heads or tails of it, and of course this is generated by some program (e.g. Augur). Problem is, the user is then prompted to confirm a transaction that looks like this:

The solution of the Ethereum community was to assemble a 4byte database, so that by looking at the first 4 bytes of the above data, you can guess what the rest of the data is meant to represent, and can thus show the user a meaningful dump of what they are about to confirm (images above and below courtesy of Etherscan).

Currently all Ethereum wallet UIs reinvent the wheel when it comes to integrating 4bytes! The database is public, but the integrations are custom. Clef ships the entire 4byte database embedded in itself, and whenever a transaction is made, it decodes the calldata internally. Not only does Clef send the decoded call to the UI, but also adds warning messages if the data does not match the method signature! Clef will handle Ethereum, you can focus on the UI!

Programmatic rules

Clef seems awesome, what more could we ask for? Well… based on the above sections, we can build the perfect signer to confirm any and all of our transactions… manually. What happens, however, if we want to automate some of that (e.g. Clique signer, Raiden relay, Swarm exchange, etc). We could just not care and let the UI sort it out… but then we’re back in square one, as all wrapping UIs need to reinvent the same mechanisms, and most will probably do it insecurely.

Clef solves this via an encrypted key-value store and an ingenious rule engine! Instead of prompting the user to confirm each and every request via a passphrase entry, we can permit Clef to sign on our behalf by storing our passphrase in its encrypted database. This will only allow passwordless signing, but still needs manual confirmation!

As a second step, however, we can also provide Clef with a JavaScript rule file, that will run whenever a request arrives and can decide to auto-confirm, auto-reject, or forward the request for manual confirmation. The JavaScript rules have access to the entire request and can also store arbitrary data in a key-value store for persistence. E.g. An academic demo rule file:

function ApproveSignData(req) {
  if (req.address.toLowerCase() == '0xd9c9cd5f6779558b6e0ed4e6acf6b1947e7fa1f3') {
    if (req.messages[0].value.indexOf('bazonk') >= 0) {
      return 'Approve';
    }
    return 'Reject';
  }
  // Otherwise goes to manual processing
}

The goal of these rules is to allow you to configure arbitrary approval logic for whatever your use case might be, whether that’s automatic server-side transactions (Clique, Raiden, Swarm, Faucet) or low-value client-side automation (approve X Wei / 24h to Augur). The programmable rules ensure that Clef remains true to its composability promise, permitting anyone to build their dream integration on top.

For a full demo on how to set up automatic rules, please check the Clef Quickstart Guide.

Light clients

Light clients are tricky and they make everything more complicated than it should be. The root cause is more philosophical than technical: the best things in life are free, and the second best are cheap. In Ethereum client terms, the “best” clients are those that work with 0 overhead (think Metamask, Infura), the second best are the light clients.

Problem is, trusted servers go against the ethos of the project, but light clients are often too heavy for resource constrained devices (ethash murders your phone battery). Geth v1.9.0 ships a new mode for light clients, called an ultra light client. This mode aims to position itself midway on the security spectrum between a trusted server and a light server, replacing PoW verification with digital signatures from a majority of trusted servers.

With enough signatures from independent entities, you could achieve more than enough security for non-critical DApps. That said, ultra light client mode is not really meant for your average node, rather for projects wishing to ship Geth embedded into their own process. This work was spearheaded by Boris Petrov and Status.

Checkpoint oracle

Light clients are dirty little cheats! Instead of downloading and verifying each header from the genesis to chain head, they use a hard coded checkpoint (shipped within Geth) as a starting point. Of course, this checkpoint contains all the necessary infos to cryptographically verify even past headers, so security wise nothing is lost.

Still, as useful as the embedded checkpoints are, they do have their shortcomings:

• As the checkpoints are hard coded into our release binaries, older releases will always start syncing from an older block. This is fine for a few months, but eventually it gets annoying. You can, of course, update Geth to fetch a new checkpoint, but that also pulls in all our behavioral changes, which you may not want to do for whatever reason.
• Since these checkpoints are embedded into the code, you’re out of luck if you want to support them in your own private network. You’d need to either ship a modified Geth, or configure the checkpoints via a config file, distributing a new one whenever you update the checkpoint. Doable, but not really practical long term.

This is where Gary Rong’s and Zsolt Felföldi’s work comes in to play. Geth v1.9.0 ships support for an on-chain checkpoint oracle. Instead of relying on hard-coded checkpoints, light clients can reach out to untrusted remote light servers (peer-to-peer, no centralized bs) and ask them to return an updated checkpoint stored within an on-chain smart contract. The best part, light clients can cryptographically prove that the returned data was signed by a required number of approved signers!

Wait, how does a light client know who’s authorized to sign an on-chain checkpoint? For networks supported out of the box, Geth ships with hard coded checkpoint oracle addresses and lists of authorized signers (so you’re trusting the same devs who ship Geth itself). For private networks, the oracle details can be specified via a config file.

Although the old and new checkpoint mechanisms look similar (both require hard-coded data in Geth or a config file), the new checkpoint oracle needs to be configured only once and afterwards can be used arbitrarily long to publish new checkpoints.

checkpoint-admin

Ethereum contracts are powerful, but interacting with them is not for the faint of heart. Our checkpoint oracle contract is an especially nasty beast, because a) it goes out of its way to retain security even in the face of chain reorgs; and b) it needs to support sharing and proving checkpoints to not-yet-synced clients.

As we don’t expect anyone (not even ourselves) to manually interact with the checkpoint oracle, Geth v1.9.0 also ships an admin tool specifically for this contract, checkpoint-admin. Note, you’ll only ever need to care about this if you want to run your own checkpoint oracle for your own private network.

The checkpoint-admin can be used to query the status of an already deployed contract (–rpc needs to point to either a light node, or a full node with –lightserv enabled, both with the les RCP API namespace exposed):

$ checkpoint-admin --rpc ~/.ethereum/rinkeby/geth.ipc status
Oracle => 0xebe8eFA441B9302A0d7eaECc277c09d20D684540

Admin 1 => 0xD9C9Cd5f6779558b6e0eD4e6Acf6b1947E7fA1F3
Admin 2 => 0x78d1aD571A1A09D60D9BBf25894b44e4C8859595
Admin 3 => 0x286834935f4A8Cfb4FF4C77D5770C2775aE2b0E7
Admin 4 => 0xb86e2B0Ab5A4B1373e40c51A7C712c70Ba2f9f8E

Checkpoint (published at #4638418) 140 => 0x488c2eba92d31baeccfb6968fad5c21a3df93181b43b4cf253b4d572b64172ef

The admin command can also be used to deploy a new oracle, sign an updated checkpoint and publish it into the network. Furthermore, checkpoint-admin also works in offline mode (without a live chain to provide data) and can also be backed by clef for signing instead of using key files, but describing all these is for another day.

Monitoring

This is perhaps something that not many knew about, but since pretty much forever, Geth had built in support for monitoring different subsystems and events. Naturally, the original version was quite crude (text UI, RPC reporting), but it provided the ground work. We can do better than this!

Metrics collection

First thing’s first, metrics need to be gathered before they can be exported and visualized. Geth can be instructed to collect all its known metrics via the –metrics CLI flag. To expose these measurements to the outside world, Geth v1.9.0 features 3 independent mechanisms: ExpVars, InfluxDB and Prometheus.

ExpVars are a somewhat custom means in the Go ecosystem to expose public variables on an HTTP interface. Geth uses its debug pprof endpoint to expose these on. Running Geth with –metrics –pprof will expose the metrics in expvar format at http://127.0.0.1:6060/debug/metrics. Please note, you should never expose the pprof HTTP endpoint to the public internet as it can be used to trigger resource intensive operations!

ExpVars are well-ish supported within the Go ecosystem, but are not the industry standard. A similar mechanism, but with a more standardized format, is the Prometheus endpoint. Running Geth with –metrics –pprof will also expose this format at http://127.0.0.1:6060/debug/metrics/prometheus. Again, please never expose the pprof HTTP endpoint to the public internet! Shoutout to Maxim Krasilnikov for contributing this feature.

Whereas ExpVars and Prometheus are pull based monitoring mechanisms (remote servers pull the data from Geth), we also support push based monitoring via InfluxDB (Geth pushes the data to remote servers). This feature requires a number of CLI flags to be set to configure the database connection (server, database, username, password and Geth instance tag). Please see the METRICS AND STATS OPTIONS section of geth help for details (–metrics.influxdb and subflags). This work was done by Anton Evangelatov.

Metrics visualization

Visualizing metrics can be a little daunting since you need to pick a charting program/service and put a whole lot of work into it to configure all the hosts, charts and dashboards.

We ourselves are using Datadog internally and have been contiguously tweaking our monitoring dashboards ever since we created them 1.5 years ago. If you are already using Datadog or are considering to do so, here’s a teaser of what you can assemble based on the metrics exposed by Geth (this is the dashboard through which we compare PRs against master):

Unfortunately Datadog does not support sharing dashboards with external entities (since they depend on how the monitored machines have been configured). As such, we can’t easily share the above work with you, but we did export a JSON dump of it in case anyone’s willing to follow in our footsteps!

Of course, we also understand that a paid service such as Datadog is not always ideal, especially if you are just starting out and don’t have money to burn on monitoring. A brilliant free monitoring tool is Grafana!

Maxim Krasilnikov made a Grafana dashboard a while ago against an older development version of Geth. We took his awesome work and merged into it the stats that we ourselves grew fond of in Datadog, resulting in quite a bit of extra work on Geth. The end result, however, is stunning (expect further updates over the next releases):

You can quickly reproduce the above charts via my clone of Maxim Krasilnikov’s project by running docker-compose up in the repo root and accessing http://localhost:3000 with the admin/admin credentials. Alternatively, you can view my testing snapshot on Raintank, or import this dashboard into your own Grafana instance

Puppeth explorer

A long time ago in a far away land, Puppeth saw the first light of day (specifically, in Mexico, just shy of two years ago). If you haven’t head about it, “Puppeth is a tool to aid you in creating a new Ethereum network down to the genesis block, bootnodes, signers, ethstats, faucet, wallet, explorer and dashboard”. Originally it was created to support deploying and maintaining the Rinkeby testnet, but has since been used by various groups for other private networks too.

Puppeth is not a tool for maintaining a high value production network, but it has, nonetheless, proven itself robust enough to keep Rinkeby relevant for over two years now! If you’d like a deeper dive into Puppeth, here’s my reveal talk from way back. In this post however lets focus on what’s new!

Puppeth is awesome! It allowed you since day 1 to deploy a full, operational Ethereum network across multiple machines, extended with a stats page to aid maintenance, along with a faucet and a dashboard to help onboard users easily. Puppeth, however, lacked a robust block explorer, since the only contenders back then were Etherscan and Etherchain, both closed source. We did hack something in, but it was kind of meh…

With the announcement of Blockscout late last year, everything changed! The Ethereum community finally got a real, open source block explorer, courtesy of the POA Network team. Compared to the established players, Blockscout of course has some catching up to do, but that does not stop us from realizing that it is already an insanely valuable asset. As such, Geth v1.9.0 ships a preliminary integration of Blockscout into Puppeth, filling a huge hole in our private network deployment tool!

This effort was pioneered by Gary Rong, but a huge shoutout goes to Ayrat Badykov too for his help in sorting out issues, questions and whatnot.

Please note, that we expect the initial integration to be rough (e.g. due to a “bug” in Blockscout, the Puppeth explorer will need to fully sync a Geth archive node before it can boot up the explorer web interface). By all means test it, run it, report any issues, but don’t be surprised if it goes down at 3AM!

Discovery protocol

Now here’s another piece of legacy infrastructure! Apart from a teeny-tiny modification, Ethereum’s discovery protocol has been specced, implemented and set in stone since pretty much forever. For those wondering what the discovery protocol is all about, it’s the mechanism through which a new node can find other Ethereum nodes on the internet and join them into a global peer-to-peer network.

So… what’s wrong with it then? Didn’t it work well enough until now? If it ain’t broken, don’t fix it and all that?

Well, Ethereum’s original discovery protocol was made for a different time, a time when there was only one chain, when there weren’t private networks, when all nodes in the network were archive nodes. We outgrew these simplistic assumptions, which although is a success story, it also brings new challenges:

• The Ethereum ecosystem nowadays has many public, private and test networks. Although Ethereum mainnet consists of a large number of machines, other networks are generally a lot smaller (e.g. Görli testnet). The discovery protocol doesn’t differentiate between these networks, so connecting to a smaller one is a never ending trial and error of finding unknown peers, connecting to them, then realizing they are on a different network.
• The same original Ethereum network can end up partitioning itself into multiple disjoint pieces, where participants might want to join one piece or the other. Ethereum Classic is one of the main examples here, but a similar issue arises every time a network upgrade (hard fork) passes and some nodes upgrade late. Without information concerning the rules of the network, we again fall back to trial and error connectivity, which is computationally extremely expensive.
• Even if all nodes belong to the same network and all nodes adhere to the same fork rules, there still exists a possibility that peering is hard: if there is connectivity asymmetry, where some nodes depend on services offered by a limited subset of machines (i.e. light clients vs. light servers).

Long term we’re working towards a brand new version of the discovery protocol. Geth’s light clients have been since forever using a PoC version of this, but rolling out such a major change for the entire Ethereum network requires a lot of time and a lot of care. This effort it being piloted primarily by Felix Lange and Frank Szendzielarz in collaboration with Andrei Maiboroda from Aleth/C++, Antoine Toulme with Java, Age Manning from Lighthouse/Rust and Tomasz Stańczak from Nethermind/C#.

Ethereum Node Records

The above was a whole lot of text about something we didn’t ship! What we did ship however, is the Ethereum Node Record (ENR) extension of the new discovery protocol, which can actually run on top of the old protocol too! An ENR is a tiny, 300 byte, arbitrary key-value data set, that nodes can advertise and query via discovery. Although the new discovery protocol will provide fancy ways of sharing these in the network, the old protocol too is capable of directly querying them.

The immediate benefit is that nodes can advertise a lot of metadata about themselves without an expensive TCP + crypto handshake, thus allowing potential peers to filter out unwanted connections without ever making them in the first place! All credits go to Felix Lange for his unwavering efforts on this front!

Ok, ok, we get it, it’s fancy. But what is it actually, you know, useful for, in human-speak?

Geth v1.9.0 ships two extensions to the discovery protocol via ENRs:

• The current discovery protocol is only capable of handling one type of IP address (IPv4 or IPv6). Since most of the internet still operates on IPv4, that’s what peers advertise and share with each other. Even though IPv6 is workable, in practice you cannot find such peers. Felix Lange’s work on advertising both IPv4 and IPv6 addresses via ENRs allows peers to discover and maintain Kademlia routing tables for both IP types. There’s still integration work to be done, but we’re hoping to elevate IPv6 to a first class citizen of Ethereum.
• Finding a Rinkeby node nowadays works analogously to connecting to random websites and checking if they are Google or not. The discovery protocol maintains a soup of internet addresses that speak the Ethereum protocol, but otherwise has no idea which chain or which forks they are on. The only way to figure out, is to connect and see, which is a very expensive shooting-in-the-dark. Péter Szilágyi proposed an extension to ENR which permits nodes to advertise their chain configuration via the discovery protocol, resulting in a 0-RTT mechanism for rejecting surely bad peers.

The most amazing thing however with ENR – and its already implemented extras – is that anyone can write a UDP crawler to index Ethereum nodes, without having to connect to them (most nodes won’t have free slots; and crawlers that do connect via TCP waste costly resources). Having simple access to all the nodes, their IPs/ports, capabilities and chain configurations permits the creation of a brand new discovery protocol based on DNS, allowing nodes with blocked UPD ports (e.g. via Tor) to join the network too!

Bootnodes

We’ve had a varying number of bootnodes of varying quality, managed by varying people since the Frontier launch. Although it worked well-ish, from a devops perspective it left a lot to desire, especially when it came to monitoring and maintenance. To go along our Geth v1.9.0 release, we’ve decided to launch a new set of bootnodes that is managed via Terraform and Ansible; and monitored via Datadog and Papertrail. We’ve also enabled them to serve light clients, hopefully bumping the reliability of the light protocol along the way. Huge shoutout to Rafael Matias for his work on this!

Our new list of bootnodes is:

• enode://d860a01f9722d78051619d1e2351aba3f43f943f6f00718d1b9baa4101932a1f5011f16bb2b1bb35db20d6fe28fa0bf09636d26a87d31de9ec6203eeedb1f666@18.138.108.67:30303 (Singapore, AWS)
• enode://22a8232c3abc76a16ae9d6c3b164f98775fe226f0917b0ca871128a74a8e9630b458460865bab457221f1d448dd9791d24c4e5d88786180ac185df813a68d4de@3.209.45.79:30303 (Virginia, AWS)
• enode://ca6de62fce278f96aea6ec5a2daadb877e51651247cb96ee310a318def462913b653963c155a0ef6c7d50048bba6e6cea881130857413d9f50a621546b590758@34.255.23.113:30303 (Ireland, AWS)
• enode://279944d8dcd428dffaa7436f25ca0ca43ae19e7bcf94a8fb7d1641651f92d121e972ac2e8f381414b80cc8e5555811c2ec6e1a99bb009b3f53c4c69923e11bd8@35.158.244.151:30303 (Frankfurt, AWS)
• enode://8499da03c47d637b20eee24eec3c356c9a2e6148d6fe25ca195c7949ab8ec2c03e3556126b0d7ed644675e78c4318b08691b7b57de10e5f0d40d05b09238fa0a@52.187.207.27:30303 Australia, Azure)
• enode://103858bdb88756c71f15e9b5e09b56dc1be52f0a5021d46301dbbfb7e130029cc9d0d6f73f693bc29b665770fff7da4d34f3c6379fe12721b5d7a0bcb5ca1fc1@191.234.162.198:30303 (Brazil, Azure)
• enode://715171f50508aba88aecd1250af392a45a330af91d7b90701c436b618c86aaa1589c9184561907bebbb56439b8f8787bc01f49a7c77276c58c1b09822d75e8e8@52.231.165.108:30303 (South Korea, Azure)
• enode://5d6d7cd20d6da4bb83a1d28cadb5d409b64edf314c0335df658c1a54e32c7c4a7ab7823d57c39b6a757556e68ff1df17c748b698544a55cb488b52479a92b60f@104.42.217.25:30303 (West US, Azure)

Our legacy bootnodes will continue to function for the time being, but will be gradually sunset in the following months.

Other changes

Beside all the awesome features enumerated above, there are a few other notable changes that are not large enough to warrant their own section, but nonetheless important enough to explicitly mention.

RPC APIs:

• The origin check on WebSocket connections (–wsorigins) is enforced only when the Origin header is present. This makes it easier to connect to Geth from non-browser environments such as Node.js, while preventing use of the RPC endpoint from arbitrary websites.
• You can set the maximum gas for eth_call using the –rpc.gascap command line option. This is useful if exposing the JSON-RPC endpoint to the Internet.
• All RPC method invocations are now logged at debug level. Failing methods log as warning so you can always see when something isn’t right.
• Geth v1.9.0 supports the eth_chainId RPC method defined in EIP 695.

Networking:

• The default peer count is now 50 instead of 25. This change improves sync performance.
• A new CLI tool (cmd/devp2p) was added to the source tree for for debugging P2P networking issues. While we don’t distribute this tool in the alltools archive yet, it’s already very useful to check issues with peer discovery.
• The P2P server now rejects connections from IPs that attempt to connect too frequently.

Miscellaneous:

• A lot of work has gone into improving the abigen tool. Go bindings now support Solidity struct and function pointer arguments. The Java generator is improved as well. The mobile framework can create deploy transactions.
• Significant parts of the go-ethereum repo now build without CGO. Big thanks to Jeremy Schlatter for this work.

Compatibility

Although Go Ethereum v1.9.0 brings an impressive number of improvements, there are a few backwards incompatible changes too. This section is a rundown of all the things that got changed or sunset in the release:

• Account unlocking with open HTTP, WebSocket or GraphQL ports have been disallowed due to security reasons. Power users can restore the old behavior with the –allow-insecure-unlock CLI flag at their own risk.
• The old Ubuntu docker images and the old (monolithic) Alpine docker images have been removed as deprecated over a year ago. Unless you configured your cluster in 2016, you most probably used the slim Alpine images and are safe.
• The original geth monitor CLI command was removed along with its supporting debug_metrics RPC API endpoint. Anyone relying on monitoring should use the ExpVar, InfuxDB or Prometheus metrics reporting along with Datadog or Grafana.
• The geth bug CLI command has been removed, being an unnecessary nicety. If you encounter a bug, you can simply open an issue on our GitHub tracker and fill out the template manually.
• The les/1 and eth/62 protocols were removed. les/1 was only supported by Geth and everyone on Constantinople runs les/2 already. eth/62 was deprecated even before Frontier, but was left in for cpp-ethereum.
• Google+ authentication has been removed from the Puppeth faucet since Google sunset its social network in the beginning of April, 2019.
• The Ledger HD wallet derivation path was updated from the orignal legacy path to the canonical ecosystem one. Accounts from old paths will still be discovered.
• The default cache allowance is chosen dynamically based on the network and sync modes. Mainnet full nodes default to 4GB, testnet and private networks to 1GB. Light clients default to 128MB. Explicit –cache is of course honored.
• The PoW calculation in Whisper v6 was incompatible with Parity due to not fully adhering to the spec. This was fixed, but it also means that Whisped v6 shipped with Geth v1.9.0 is incompatible with previous versions.
• The –lightserv and –lightpeers flags were renamed to –light.serve and –light.maxpeers respectively. The old versions are deprecated, but will continue to work for the next year or so.
• The default datadir on Windows is now derived from the LocalAppData environment variable. The old location in $HOME/AppData/Roaming is still recognized. The location change works better with Cygwin and setups using remote user accounts.
• The JSON-RPC server has been rewritten and now supports bi-directional communication. You can expose method handlers on the client side using the rpc.Client.RegisterName method. While we did test this extensively, there may be compatibility issues with the new server. Please report any RPC issues you find.

Epilogue

We’re really proud of this release! It took a lot longer than expected, but we wanted to ship all the breaking changes in one go to minimize potential surprises (upgrade issues); and to finalize the APIs of new features, to avoid breaking them later. Hope you too will find a gem for yourself among our shipped ~370 changes.

As with all our previous releases, you can find the:

And as a last word before signing off (better twice than none):

Warning: We’ve tried our best to squash all the bugs, but as with all major releases, we advise everyone to take extra care when upgrading. The v1.9.0 release contains database schema changes, meaning it’s not possible to downgrade once updated. We also recommend a fresh fast sync as it can drastically reduce the database size.

*This is part #1 of a series where anyone can ask questions about Geth and I’ll attempt to answer the highest voted one each week with a mini writeup. This week’s highest voted question was: Could you share how the flat db structure is different from the legacy structure?*

State in Ethereum

Before diving into an acceleration structure, let’s recap a bit what Ethereum calls state and how it is stored currently at its various levels of abstraction.

Ethereum maintains two different types of state: the set of accounts; and a set of storage slots for each contract account. From a purely abstract perspective, both of these are simple key/value mappings. The set of accounts maps addresses to their nonce, balance, etc. A storage area of a single contract maps arbitrary keys – defined and used by the contract – to arbitrary values.

Unfortunately, whilst storing these key-value pairs as flat data would be very efficient, verifying their correctness becomes computationally intractable. Every time a modification would be made, we’d need to hash all that data from scratch.

Instead of hashing the entire dataset all the time, we could split it up into small contiguous chunks and build a tree on top! The original useful data would be in the leaves, and each internal node would be a hash of everything below it. This would allow us to only recalculate a logarithmic number of hashes when something is modified. This data structure actually has a name, it’s the famous Merkle tree.

Unfortunately, we still fall a bit short on the computational complexity. The above Merkle tree layout is very efficient at incorporating modifications to existing data, but insertions and deletions shift the chunk boundaries and invalidate all the calculated hashes.

Instead of blindly chunking up the dataset, we could use the keys themselves to organize the data into a tree format based on common prefixes! This way an insertion or deletion wouldn’t shift all nodes, rather will change just the logarithmic path from root to leaf. This data structure is called a Patricia tree.

Combine the two ideas – the tree layout of a Patricia tree and the hashing algorithm of a Merkle tree – and you end up with a Merkle Patricia tree, the actual data structure used to represent state in Ethereum. Guaranteed logarithmic complexity for modifications, insertions, deletions and verification! A tiny extra is that keys are hashed before insertion to balance the tries.

State storage in Ethereum

The above description explains why Ethereum stores its state in a Merkle Patricia tree. Alas, as fast as the desired operations got, every choice is a trade-off. The cost of logarithmic updates and logarithmic verification is logarithmic reads and logarithmic storage for every individual key. This is because every internal trie node needs to be saved to disk individually.

I do not have an accurate number for the depth of the account trie at the moment, but about a year ago we were saturating the depth of 7. This means, that every trie operation (e.g. read balance, write nonce) touches at least 7-8 internal nodes, thus will do at least 7-8 persistent database accesses. LevelDB also organizes its data into a maximum of 7 levels, so there’s an extra multiplier from there. The net result is that a single state access is expected to amplify into 25-50 random disk accesses. Multiply this with all the state reads and writes that all the transactions in a block touch and you get to a scary number.

[Of course all client implementations try their best to minimize this overhead. Geth uses large memory areas for caching trie nodes; and also uses in-memory pruning to avoid writing to disk nodes that get deleted anyway after a few blocks. That’s for a different blog post however.]

As horrible as these numbers are, these are the costs of operating an Ethereum node and having the capability of cryptograhically verifying all state at all times. But can we do better?

Not all accesses are created equal

Ethereum relies on cryptographic proofs for its state. There is no way around the disk amplifications if we want to retain our capability to verify all the data. That said, we can – and do – trust the data we’ve already verified.

There is no point to verify and re-verify every state item, every time we pull it up from disk. The Merkle Patricia tree is essential for writes, but it’s an overhead for reads. We cannot get rid of it, and we cannot slim it down; but that doesn’t mean we must necessarily use it everywhere.

An Ethereum node accesses state in a few different places:

• When importing a new block, EVM code execution does a more-or-less balanced number of state reads and writes. A denial-of-service block might however do significantly more reads than writes.
• When a node operator retrieves state (e.g. eth_call and family), EVM code execution only does reads (it can write too, but those get discarded at the end and are not persisted).
• When a node is synchronizing, it is requesting state from remote nodes that need to dig it up and serve it over the network.

Based on the above access patterns, if we can short circuit reads not to hit the state trie, a slew of node operations will become significantly faster. It might even enable some novel access patterns (like state iteration) which was prohibitively expensive before.

Of course, there’s always a trade-off. Without getting rid of the trie, any new acceleration structure is extra overhead. The question is whether the additional overhead provides enough value to warrant it?

Back to the roots

We’ve built this magical Merkle Patricia tree to solve all our problems, and now we want to get around it for reads. What acceleration structure should we use to make reads fast again? Well, if we don’t need the trie, we don’t need any of the complexity introduced. We can go all the way back to the origins.

As mentioned in the beginning of this post, the theoretical ideal data storage for Ethereum’s state is a simple key-value store (separate for accounts and each contract). Without the constraints of the Merkle Patricia tree however, there’s “nothing” stopping us from actually implementing the ideal solution!

A while back Geth introduced its snapshot acceleration structure (not enabled by default). A snapshot is a complete view of the Ethereum state at a given block. Abstract implementation wise, it is a dump of all accounts and storage slots, represented by a flat key-value store.

Whenever we wish to access an account or storage slot, we only pay 1 LevelDB lookup instead of 7-8 as per the trie. Updating the snapshot is also simple in theory, after processing a block we do 1 extra LevelDB write per updated slot.

The snapshot essentially reduces reads from O(log n) to O(1) (times LevelDB overhead) at the cost of increasing writes from O(log n) to O(1 + log n) (times LevelDB overhead) and increasing disk storage from O(n log n) to O(n + n log n).

Devil’s in the details

Maintaining a usable snapshot of the Ethereum state has its complexity. As long as blocks are coming one after the other, always building on top of the last, the naive approach of merging changes into the snapshot works. If there’s a mini reorg however (even a single block), we’re in trouble, because there’s no undo. Persistent writes are one-way operation for a flat data representation. To make matters worse, accessing older state (e.g. 3 blocks old for some DApp or 64+ for fast/snap sync) is impossible.

To overcome this limitation, Geth’s snapshot is composed of two entities: a persistent disk layer that is a complete snapshot of an older block (e.g. HEAD-128); and a tree of in-memory diff layers that gather the writes on top.

Whenever a new block is processed, we do not merge the writes directly into the disk layer, rather just create a new in-memory diff layer with the changes. If enough in-memory diff layers are piled on top, the bottom ones start getting merged together and eventually pushed to disk. Whenever a state item is to be read, we start at the topmost diff layer and keep going backwards until we find it or reach the disk.

This data representation is very powerful as it solves a lot of issues. Since the in-memory diff layers are assembled into a tree, reorgs shallower than 128 blocks can simply pick the diff layer belonging to the parent block and build forward from there. DApps and remote syncers needing older state have access to 128 recent ones. The cost does increase by 128 map lookups, but 128 in-memory lookups is orders of magnitude faster than 8 disk reads amplified 4x-5x by LevelDB.

Of course, there are lots and lots of gotchas and caveats. Without going into too much details, a quick listing of the finer points are:

• Self-destructs (and deletions) are special beasts as they need to short circuit diff layer descent.
• If there is a reorg deeper than the persistent disk layer, the snapshot needs to be completely discarded and regenerated. This is very expensive.
• On shutdown, the in-memory diff layers need to be persisted into a journal and loaded back up, otherwise the snapshot will become useless on restart.
• Use the bottom-most diff layer as an accumulator and only flush to disk when it exceeds some memory usage. This allows deduping writes for the same slots across blocks.
• Allocate a read cache for the disk layer so that contracts accessing the same ancient slot over and over don’t cause disk hits.
• Use cumulative bloom filters in the in-memory diff layers to quickly detect whether there’s a chance for an item to be in the diffs, or if we can go to disk immediately.
• The keys are not the raw data (account address, storage key), rather the hashes of these, ensuring the snapshot has the same iteration order as the Merkle Patricia tree.
• Generating the persistent disk layer takes significantly more time than the pruning window for the state tries, so even the generator needs to dynamically follow the chain.

The good, the bad, the ugly

Geth’s snapshot acceleration structure reduces state read complexity by about an order of magnitude. This means read-based DoS gets an order of magnitude harder to pull off; and eth_call invocations get an order of magnitude faster (if not CPU bound).

The snapshot also enables blazing fast state iteration of the most recent blocks. This was actually the main reason for building snapshots, as it permitted the creation of the new snap sync algorithm. Describing that is an entirely new blog post, but the latest benchmarks on Rinkeby speak volumes:

Of course, the trade-offs are always present. After initial sync is complete, it takes about 9-10h on mainnet to construct the initial snapshot (it’s maintained live afterwards) and it takes about 15+GB of additional disk space.

As for the ugly part? Well, it took us over 6 months to feel confident enough about the snapshot to ship it, but even now it’s behind the –snapshot flag and there’s still tuning and polishing to be done around memory usage and crash recovery.

All in all, we’re very proud of this improvement. It was an insane amount of work and it was a huge shot in the dark implementing everything and hoping it will work out. Just as a fun fact, the first version of snap sync (leaf sync) was written 2.5 years ago and was blocked ever since because we lacked the necessary acceleration to saturate it.

Epilogue

Hope you enjoyed this first post of Ask about Geth. It took me about twice as much to finish it than I aimed for, but I felt the topic deserves the extra time. See you next week.

[PS: I deliberately didn’t link the asking/voting website into this post as I’m sure it’s a temporary thing and I don’t want to leave broken links for posterity; nor have someone buy the name and host something malicious in the future. You can find it among my Twitter posts.]

Oh wow, it’s been a while… over 1.5 years since we’ve released Geth v1.9.0. We did do 26 point releases in that time frame (about one per three weeks), but pushing out a major release is always a bit more special. The adrenaline rush of shipping new features, coupled with the fear of something going horribly wrong. Still unsure if I like it or hate it. Either way, Ethereum is evolving and we need to push the envelope to keep up with it.

Without further ado, please welcome Geth v1.10.0 to the Ethereum family.

Here be dragons

Before diving into the details of our newest release, it’s essential to emphasize that with any new feature, come new risks. To cater for users and projects with differing risk profiles, many of our heavy hitter features can be (for now) toggled on and off individually. Whether you read the entire content of this blog post – or only skim parts interesting to you – please read the ‘Compatibility’ section at the end of this document!

With that out of the way, let’s dive in and see what Geth v1.10.0 is all about!

Berlin hard-fork

Let’s get the elephant out of the room first. Geth v1.10.0 does not ship the Berlin hard-fork yet, as there was some 11th hour concerns from the Solidity team about EIP-2315. Since v1.10.0 is a major release, we don’t want to publish it too close to the fork. We will follow up with v1.10.1 soon with the final list of EIPs and block numbers baked in.

Snapshots

We’ve been talking about snapshots for such a long time now, it feels strange to finally see them in a release. Without going into too many details (see linked post), snapshots are an acceleration data structure on top of the Ethereum state, that allows reading accounts and contract storage significantly faster.

To put a number on it, the snapshot feature reduces the cost of accessing an account from O(logN) to O(1). This might not seem like much at a first glance, but translated to practical terms, on mainnet with 140 million accounts, snapshots can save about 8 database lookups per account read. That’s almost an order of magnitude less disk lookups, guaranteed constant independent of state size.

Whoa, does this mean we can 10x the gas limit? No, unfortunately. Whilst snapshots do grant us a 10x read performance, EVM execution also writes data, and these writes need to be Merkle proven. The Merkle proof requirement retains the necessity for O(logN) disk access on writes.

So, what’s the point then?! Whilst fast read access to accounts and contract storage isn’t enough to bump the gas limit, it does solve a few particularly thorny issues:

• DoS. In 2016, Ethereum sustained its worse DoS attack ever – The Shanghai Attacks – that lasted about 2-3 months. The attack revolved around bloating Ethereum’s state and abusing various underpriced opcodes to grind the network to a halt. After numerous client optimizations and repricing hard forks, the attack was repelled. The root cause still lingers: state access opcodes have a fixed EVM gas cost O(1), but an ever slowly increasing execution cost O(logN). We’ve bumped the gas costs in Tangerine Whistle, Istanbul and now Berlin to bring the EVM costs back in line with the runtime costs, but these are stopgap measures. Snapshots on the other hand reduce execution cost of state reads to O(1) – in line with EVM costs – thus solves the read-based DoS issues long term (don’t quote me on that).
• Call. Checking a smart contract’s state in Ethereum entails a mini EVM execution. Part of that is running bytecode and part of it is reading state slots from disk. If you have your personal Ethereum node that you only use for your own personal needs, there’s a high chance that the current state access speed is more than adequate. If you’re operating a node for the consumption of multiple users however, the 10x performance improvement granted by snapshots means that you can serve 10x as many queries at +- the same cost to you.
• Sync. There are two major ways you can synchronize an Ethereum node. You can download the blocks and execute all the transactions within; or you can download the blocks, verify the PoWs and download the state associated a recent block. The latter is much faster, but it relies on benefactors serving you a copy of the recent state. With the current Merkle-Patricia state model, these benefactors read 16TB of data off disk to serve a syncing node. Snapshots enable serving nodes to read only 96GB of data off disk to get a new node joined into the network. More on this in the Snap sync section.

As with all features, it’s a game of tradeoffs. Whilst snapshots have enormous benefits, that we believe in strongly enough to enable for everyone, there are certain costs to them:

• A snapshot is a redundant copy of the raw Ethereum state already contained in the leaves of the Merkle Patricia trie. As such, snapshots entail an additional disk overhead of about 20-25GB on mainnet currently. Hopefully snapshots will allow us to do some further state optimizations and potentially remove some of the disk overhead of Merkle tries as they are currently.
• Since nobody has snapshots constructed in the network yet, nodes will initially need to bear the cost of iterating the state trie and creating the initial snapshot themselves. Depending on the load to your node, this might take anywhere between a day to a week, but you only need to do it once in the lifetime of your node (if things work as intended). The snapshot generation runs in the background, concurrently with all other node operations. We have plans to not require this once snapshots are generally available in the network. More on this in the Snap sync section.

If you are not confident about the snapshot feature, you can disable it in Geth 1.10.0 via –snapshot=false, but be advised that we will make it mandatory long term to guarantee a baseline network health.

Snap sync

If you thought snapshots took a long time to ship, wait till you hear about snap sync! We’ve implemented the initial prototype of a new synchronization algorithm way back in October, 2017… then sat on the idea for over 3 years?! Before diving in, a bit of history.

When Ethereum launched, you could choose from two different ways to synchronize the network: full sync and fast sync (omitting light clients from this discussion). Full sync operated by downloading the entire chain and executing all transactions; vs. fast sync placed an initial trust in a recent-ish block, and directly downloaded the state associated with it (after which it switched to block execution like full sync). Although both modes of operation resulted in the same final dataset, they preferred different tradeoffs:

• Full sync minimized trust, choosing to execute all transactions from genesis to head. Whilst it might be the most secure option, Ethereum mainnet currently contains over 1.03 billion transactions, growing at a rate of 1.25 million / day. Chosing to execute everything from genesis means full sync has a forever increasing cost. Currently it takes 8-10 days to process all those transactions on a fairly powerful machine.
• Fast sync chose to rely on the security of the PoWs. Instead of executing all transactions, it assumed that a block with 64 valid PoWs on top would be prohibitively expensive for someone to construct, as such it’s ok to download the state associated with HEAD-64. Fast sync trusting the state root from a recent block, it could download the state trie directly. This replaced the need of CPU & disk IO with a need for network bandwidth and latency. Specifically, Ethereum mainnet currently contains about 675 million state trie nodes, taking about 8-10 hours to download on a fairly well connected machine.

Full sync remained available for anyone who wanted to expend the resources to verify Ethereum’s entire history, but for most people, fast sync was more than adequate. There’s a computer science paradox, that once a system reaches 50x the usage it was designed at, it will break down. The logic is, that irrelevant how something works, push it hard enough and an unforeseen bottleneck will appear.

In the case of fast sync, the unforeseen bottleneck was latency, caused by Ethereum’s data model. Ethereum’s state trie is a Merkle tree, where the leaves contain the useful data and each node above is the hash of 16 children. Syncing from the root of the tree (the hash embedded in a block header), the only way to download everything is to request each node one-by-one. With 675 million nodes to download, even by batching 384 requests together, it ends up needing 1.75 million round-trips. Assuming an overly generous 50ms RTT to 10 serving peers, fast sync is essentially waiting for over 150 minutes for data to arrive. But network latency is only 1/3rd of the problem.

When a serving peer receives a request for trie nodes, it needs to retrieve them from disk. Ethereum’s Merkle trie doesn’t help here either. Since trie nodes are keyed by hash, there’s no meaningful way to store/retrieve them batched, each requiring it’s own database read. To make matters worse, LevelDB (used by Geth) stores data in 7 levels, so a random read will generally touch as many files. Multiplying it all up, a single network request of 384 nodes – at 7 reads a pop – amounts to 2.7 thousand disk reads. With the fastest SATA SSDs’ speed of 100.000 IOPS, that’s 37ms extra latency. With the same 10 serving peer assumption as above, fast sync just added an extra 108 minutes waiting time. But serving latency is only 1/3 of the problem.

Requesting that many trie nodes individually means actually uploading that many hashes to remote peers to serve. With 675 million nodes to download, that’s 675 million hashes to upload, or 675 * 32 bytes = 21GB. At a global average of 51Mbps upload speed (X Doubt), fast sync just added an extra 56 minutes waiting time. Downloads are a bit more than twice as large, so with global averages of 97Mbps, *fast sync* popped on a further 63 minutes. Bandwidth delays are the last 1/3 of the problem.

Sum it all up, and fast sync spends a whopping 6.3 hours doing nothing, just waiting for data:

• If you have an above average network link
• If you have a good number of serving peers
• If your peers don’t serve anyone else but you

Snap sync was designed to solve all three of the enumerated problems. The core idea is fairly simple: instead of downloading the trie node-by-node, snap sync downloads the contiguous chunks of useful state data, and reconstructs the Merkle trie locally:

• Without downloading intermediate Merkle trie nodes, state data can be fetched in large batches, removing the delay caused by network latency.
• Without downloading Merkle nodes, downstream data drops to half; and without addressing each piece of data individually, upstream data gets insignificant, removing the delay caused by bandwidth.
• Without requesting randomly keyed data, peers do only a couple contiguous disk reads to serve the responses, removing the delay of disk IO (iff the peers already have the data stored in an appropriate flat format).

Whilst snap sync is eerily similar to Parity’s warp sync – and indeed took many design ideas from it – there are significant improvements over the latter:

• Warp sync relies on static snapshots created every 30000 blocks. This means serving nodes need to regenerate the snapshots every 5 days or so, but iterating the entire state trie can actually take more time than that. This means warp sync is not sustainable long term. Opposed to this, snap sync is based on dynamic snapshots, which are generated only once, no matter how slowly, and then are kept up to date as the chain progresses.
• Warp sync‘s snapshot format does not follow the Merkle trie layout, and as such chunks of warp-data cannot be individually proven. Syncing nodes need to download the entire 20+GB dataset before they can verify it. This means warp syncing nodes could be theoretically grieved. Opposed to this, snap sync‘s snapshot format is just the sequential Merkle leaves, which allows any range to be proven, thus bad data is detected immediately.

To put a number on snap sync vs fast sync, synchronizing the mainnet state (ignoring blocks and receipts, as those are the same) against 3 serving peers, at block ~#11,177,000 produced the following results:

Do note, that snap sync is shipped, but not yet enabled, in Geth v1.10.0. The reason is that serving snap sync requires nodes to have the snapshot acceleration structure already generated, which nobody has yet, as it is also shipped in v1.10.0. You can manually enable snap sync via –syncmode snap, but be advised that we expect it not to find suitable peers until a few weeks after Berlin. We’ll enable it by default when we feel there are enough peers to rely on it.

Offline pruning

We’re really proud of what we’ve achieved with Geth over the past years. Yet, there’s always that one topic, which makes you flinch when asked about. For Geth, that topic is state pruning. But what is pruning and why is it needed?

When processing a new block, a node takes the current state of the network as input data and mutates it according to the transactions in the block, generating a new, output data. The output state is mostly the same as the input, only a few thousand items modified. Since we can’t just overwrite the old state (otherwise we couldn’t handle block reorgs), both old and new end up on disk. (Ok, we’re a bit smarter and only push new diffs to disk if they stick around and don’t get deleted in the next few blocks, but let’s ignore that part for now).

Pushing these new pieces of state data, block-by-block, to the database is a problem. They keep accumulating. In theory we could “just delete” state data that’s old enough to not run the risk of a reorg, but as it turns out, that’s quite a hard problem. Since state in Ethereum is stored in a tree data structure – and since most blocks only change a small fraction of the state – these trees share huge portions of the data with one another. We can easily decide if the root of an old trie is stale and can be deleted, but it’s exceedingly costly to figure out if a node deep within an old state is still referenced by anything newer or not.

Throughout the years, we’ve implemented a range of pruning algorithms to delete leftovers (lost count, around 10), yet we’ve never found a solution that doesn’t break down if enough data is thrown at it. As such, people grew accustomed that Geth’s database starts slim after a fast sync, and keeps growing until you get fed up and resync. This is frustrating to say the least, as re-downloading everything just wastes bandwidth and adds meaningless downtime to the node.

Geth v1.10.0 doesn’t quite solve the problem, but it takes a big step towards a better user experience. If you have snapshots enabled and fully generated, Geth can use those as an acceleration structure to relatively quickly determine which trie nodes should be kept and which should be deleted. Pruning trie nodes based on snapshots does have the drawback that the chain may not progress during pruning. This means, that you need to stop Geth, prune its database and then restart it.

Execution time wise, pruning takes a few hours (greatly depends on your disk speed and accumulated junk), one third of which is indexing recent trie node from snapshots, one third deleting stale trie nodes and the last third compacting the database to reclaim freed up space. At the end of the process, your disk usage should approximately be the same as if you did a fresh sync. To prune your database, please run geth snapshot prune-state.

Be advised, that pruning is a new and dangerous feature, a failure of which can cause bad blocks. We’re confident that it’s reliable, but if something goes wrong, there’s likely no way to salvage the database. Our recommendation – at least until the feature gets battle tested – is to back up your database prior to pruning, and try with testnet nodes first before going all in on mainnet.

Transaction unindexing

Ethereum has been around for a while now, and in its almost 6 years’ of existence, Ethereum’s users issued over 1 billion transactions. That’s a big number.

Node operators always took it for granted that they can look up an arbitrary transaction from the past, given only its hash. Truth be told, it seems like a no-brainer thing to do. Running the numbers though, we end up in a surprising place. To make transactions searchable, we need to – at minimum – map the entire range of transaction hashes to the blocks they are in. With all tradeoffs made towards minimizing storage, we still need to store 1 block number (4 bytes) associated with 1 hash (32 bytes).

36 bytes / transaction doesn’t seem much, but multiplying with 1 billion transactions ends up at an impressive 36GB of storage, needed to be able to say transaction 0xdeadbeef is in block N. It’s a lot of data and a lot of database entries to shuffle around. Storing 36GB is an acceptable price if you want to look up transactions 6 years back, but in practice, most users don’t want to. For them, the extra disk usage and IO overhead is wasted resources. It’s also important to note that transaction indices are not part of consensus and are not part of the network protocol. They are purely a locally generated acceleration structure.

Can we shave some – for us – useless data off of our nodes? Yes! Geth v1.10.0 switches on transaction unindexing by default and sets it to 2,350,000 blocks (about 1 year). The transaction unindexer will linger in the background, and every time a new block arrives, it ensures that only transactions from the most recent N blocks are indexed, deleting older ones. If a user decides they want access to older transactions, they can restart Geth with a higher –txlookuplimit value, and any blocks missing from the updated range will be reindexed (note, the trigger is still block import, you have to wait for 1 new block).

Since about 1/3rd of Ethereum’s transaction load happened in 2020, keeping an entire year’s worth of transaction index will still have a noticeable weight on the database. The goal of transaction unindexing is not to remove an existing feature in the name of saving space. The goal is to move towards a mode of operation where space does not grow indefinitely with chain history.

If you wish to disable transaction unindexing altogether, you can run Geth with –txlookuplimit=0, which reverts to the old behavior of retaining the lookup map for every transaction since genesis.

Preimage discarding

Ethereum stores all its data in a Merkle Patricia trie. The values in the leaves are the raw data being stored (e.g. storage slot content, account content), and the path to the leaf is the key at which the data is stored. The keys however are not the account addresses or storage addresses, rather the Keccak256 hashes of those. This helps balance the branch depths of the state tries. Using hashes for keys is fine as users of Ethereum only ever reference the original addresses, which can be hashed on the fly.

There is one use case, however, where someone has a hash stored in the state trie and wants to recover it’s preimage: debugging. When stepping over an EVM bytecode, a developer might want to glipmse over all the variables in the smart contract. The data is there, but without the preimages, its hard to say which data corresponds to which Solidity variable.

Originally Geth had a half-baked solution. We stored in the database all preimages that originated from user calls (e.g. sending a transaction), but not those originating from EVM calls (e.g. accessing a slot). This was not enough for Remix, so we extended our tracing API calls to support saving the preimages for all SHA3 (Keccak256) operations. Although this solved the debugging issue for Remix, it raised the question about all that data unused by non-debugging nodes.

The preimages aren’t particularly heavy. If you do a full sync from genesis – reexecuting all the transactions – you’ll only end up with 5GB extra load. Still, there is no reason to keep that data around for users not using it, as it only increases the load on LevelDB compactions. As such, Geth v1.10.0 disables preimage collection by default, but there’s no mechanism to actively delete already stored preimages.

If you are using your Geth instance to debug transactions, you can retain the original behavior via –cache.preimages. Please note, it is not possible to regenerate preimages after the fact. If you run Geth with preimage collection disabled and change your mind, you’ll need to reimport the blocks.

ETH/66 protocol

The eth/66 protocol is a fairly small change, yet has quite a number of beneficial implications. In short, the protocol introduces request and reply IDs for all bidirectional packets. The goal behind these IDs is to more easily match up responses to requests, specifically, to more easily deliver a response to a subsystem that made the original request.

These IDs are not essential, and indeed we’ve been happily working around the lack of them these past 6 years. Unfortunately, all code that needs to request anything from the network becomes overly complicated, if multiple subsystems can request the same type of data concurrently. E.g. block headers can be requested by the downloader syncing the chain; it can be requested by the fetcher fulfilling block announcements; and it can be requested by fork challenges. Furthermore, timeouts can cause late/unexpected deliveries or re-requests. In all these cases, when a header packet arrives, every subsystem peeks at the data and tries to figure out if it was meant for itself or someone else. Consuming a reply not meant for a particular subsystem will cause a failure elsewhere, which needs graceful handling. It just gets messy. Doable, but messy.

The importance of eth/66 in the scope of this blog post is not that it solves a particular problem, rather that it is introduced prior to the Berlin hard-fork. As all nodes are expected to upgrade by the fork time, this means Geth can start deprecating the old protocols after the fork. Only after discontinuing all older protocols can we rewrite Geth’s internals to take advantage of request ids. Following our protocol deprecation schedule, we’ll be dropping eth/64 shortly and eth65 by the end of summer.

Some people might consider Geth using its weight to force protocol updates on other clients. We’d like to emphasize that the typed transactions feature from the Berlin hard-fork originally called for a new protocol version. As only Geth implemented the full suite of eth/xy protocols, other clients requested “hacking” it into old protocol versions to avoid having to focus on networking at this time. The agreement was that Geth backports typed transaction support into all its old protocol code to buy other devs time, but in exchange will phase out the old versions in 6 months to avoid stagnation.

ChainID enforcement

Way back in 2016, when TheDAO hard-fork passed, Ethereum introduced the notion of the chain id. The goal was to modify the digital signatures on transactions with a unique identifier to differentiate between what’s valid on Ethereum and what’s valid on Ethereum Classic (and what’s valid on testnets). Making a transaction valid on one network but invalid on another ensures they cannot be replayed without the owner’s knowledge.

In order to minimize issues around the transition, both new/protected and old/unprotected transactions remained valid. Fast forward 5 years, and about 15% of transaction on Ethereum are still not replay-protected. This doesn’t mean there’s an inherent vulnerability, unless you reuse the same keys across multiple networks. Top tip: Don’t! Still, accidents happen, and certain Ethereum based networks have been known to go offline due to replay issues.

As much as we don’t want to play big brother, we’ve decided to try and nudge people and tooling to abandon the old, unprotected signatures and use chain ids everywhere. The easy way would be to just make unprotected transactions invalid at the consensus level, but that would leave 15% of people stranded and scattering for hotfixes. To gradually move people towards safer alternatives without pulling the rug from underneath their feet, Geth v1.10.0 will reject transactions on the RPC that are not replay protected. Propagation through the P2P protocols remains unchanged for now, but we will be pushing for rejection there too long term.

If you are using code generated by abigen, we’ve included in the go-ethereum libraries additional signer constructors to allow easily creating chain-id-bound transactors. The legacy signers included out of the box were written before EIP155 and until now you needed to construct the protected signer yourself. As this was error prone and some people assumed we guessed the chain ID internally, we decided to introduce direct APIs ourselves. We will deprecate and remove the legacy signers in the long term.

Since we realize people/tooling issuing unprotected transactions can’t change overnight, Geth v1.10.0 supports reverting to the old behavior and accepting non-EIP155 transactions via –rpc.allow-unprotected-txs. Be advised that this is a temporary mechanism that will be removed long term.

Database introspection

Every now and again we receive an issue report about a corrupted database, with no real way to debug it. Shipping a 300GB data directory to us is not feasible, and sending custom dissection tools to users is cumbersome. Also since a corrupted database often manifests itself in an inability to start up Geth, even using debugging RPC APIs are useless.

Geth v1.10.0 ships a built-in database introspection tool to try and alleviate the situation a bit. It is a very low level accessor to LevelDB, but it allows arbitrary data retrievals, insertions and deletions. We are unsure how useful these will turn out to be, but they at least give a fighting chance to restore a broken node without having to resync.

The supported commands are:

• geth db inspect – Inspect the storage size for each type of data in the database
• geth db stats – Print various database usage and compaction statistics
• geth db compact – Compact the database, optimizing read access (super slow)
• geth db get – Retrieve and print the value of a database key
• geth db delete – Delete a database key (super dangerous)
• geth db put – Set the value of a database key (super dangerous)

Flag deprecations

Throughout the v1.9.x release family we’ve marked a number of CLI flags deprecated. Some of them were renamed to better follow our naming conventions, others were removed due to dropped features (notably Whisper). Throughout the previous release family, we’ve kept the old deprecated flags functional too, only printing a warning when used instead of the recommended versions.

Geth v1.10.0 takes the opportunity to completely remove support for the old CLI flags. Below is a list to help you fix your commands if you by any chance haven’t yet upgraded to the new versions the past year:

• –rpc -> –http – Enable the HTTP-RPC server
• –rpcaddr -> –http.addr – HTTP-RPC server listening interface
• –rpcport -> –http.port – HTTP-RPC server listening port
• –rpccorsdomain -> –http.corsdomain – Domain from which to accept requests
• –rpcvhosts -> –http.vhosts – Virtual hostnames from which to accept requests
• –rpcapi -> –http.api – API’s offered over the HTTP-RPC interface
• –wsaddr -> –ws.addr – WS-RPC server listening interface
• –wsport -> –ws.port – WS-RPC server listening port
• –wsorigins -> –ws.origins – Origins from which to accept websockets requests
• –wsapi -> –ws.api – API’s offered over the WS-RPC interface
• –gpoblocks -> –gpo.blocks – Number of blocks to check for gas prices
• –gpopercentile -> –gpo.percentile – Percentile of recent txs to use as gas suggestion
• –graphql.addr -> –graphql – Enable GraphQL on the HTTP-RPC server
• –graphql.port -> –graphql – Enable GraphQL on the HTTP-RPC server
• –pprofport -> –pprof.port – Profiler HTTP server listening port
• –pprofaddr -> –pprof.addr – Profiler HTTP server listening interface
• –memprofilerate -> –pprof.memprofilerate – Turn on memory profiling with the given rate
• –blockprofilerate -> –pprof.blockprofilerate – Turn on block profiling with the given rate
• –cpuprofile -> –pprof.cpuprofile – Write CPU profile to the given file

A handful of the above listed legacy flags may still work for a few releases, but you should not rely on them remaining available.

Since most people running full nodes do not use USB wallets through Geth – and since USB handling is a bit quirky on different platforms – a lot of node operators just had to explicitly turn off USB via –nosub. To cater the defaults to the requirements of the many, Geth v1.10.0 disabled USB wallet support by default and deprecated the –nousb flag. You can still use USB wallets, just need to explicitly request it from now on via –usb.

Unclean shutdown tracking

Fairly often we receive bug reports that Geth started importing old blocks on startup. This phenomenon is generally caused by the node operator terminating Geth abruptly (power outage, OOM killer, too short shutdown timeout). Since Geth keeps a lot of dirty state in memory – to avoid writing to disk things that get stale a few blocks later – an abrupt shutdown can cause these to not be flushed. With recent state missing on startup, Geth has no choice but to rewind it’s local chain to the point where it last saved the progress.

To avoid debating whether an operator did or did not shut down their node cleanly, and to avoid having a clean cycle after a crash hide the fact that data was lost, Geth v1.10.0 will start tracking and reporting node crashes. We’re hopeful that this will allow operatos to detect that their infra is misconfigured or has issue before those turn into irreversible data loss.

WARN [03-03|06:36:38.734] Unclean shutdown detected        booted=2021-02-03T06:47:28+0000 age=3w6d23h

Compatibility

Doing a major release so close to a hard fork is less than desired, to say the least. Unfortunately, shipping all the large features for the next generation Geth took 2 months longer than we’ve anticipated. To try and mitigate production problems that might occur from the upgrade, almost all new features can be toggled off via CLI flags. There is still 6 weeks left until the currently planned mainnet block, to ensure you have a smooth experience. Nonetheless, we apologize for any inconveniences in advance.

To revert as much functionality as possible to the v1.9.x feature-set, please run Geth with:

• –snapshot=false to disable the snapshot acceleration structure and snap sync
• –txlookuplimit=0 to keep indexing all transactions, not just the last year
• –cache.preimages tp keep generating and persisting account preimages
• –rpc.allow-unprotected-txs – to allow non-replay-protected signatures
• –usb – to reenable the USB wallet support

Note, the eth_protocolVersion API call is gone as it made no sense. If you have a very good reason as to why it’s needed, please reach out to discuss it.

Epilogue

As with previous major releases, we’re really proud of this one too. We’ve delayed it quite a lot, but we did it in the name of stability to ensure that all the sensitive features are tested as well as we could. We’re hopeful this new release family will open the doors to a bit more transaction throughput and a bit lower fees.

As with all our previous releases, you can find the:

]]> https://earlybirdsinvest.com/geth-v1-10-0/feed/ 0 45442 https://earlybirdsinvest.com/geth-v1-13-0/ https://earlybirdsinvest.com/geth-v1-13-0/#respond Sat, 22 Mar 2025 00:09:04 +0000 https://earlybirdsinvest.com/geth-v1-13-0/

Geth v1.13 comes fairly close on the heels of the 1.12 release family, which is funky, considering it’s main feature has been in development for a cool 6 years now.

This post will go into a number of technical and historical details, but if you just want the gist of it, Geth v1.13.0 ships a new database model for storing the Ethereum state, which is both faster than the previous scheme, and also has proper pruning implemented. No more junk accumulating on disk and no more guerilla (offline) pruning!

• ¹Excluding ~589GB ancient data, the same across all configurations.
• ²Hash scheme full sync exceeded our 1.8TB SSD at block ~15.43M.
• ³Size difference vs snap sync attributed to compaction overhead.

Before going ahead though, a shoutout goes to Gary Rong who has been working on the crux of this rework for the better part of 2 years now! Amazing work and amazing endurance to get this huge chunk of work in!

Gory tech details

Ok, so what’s up with this new data model and why was it needed in the first place?

In short, our old way of storing the Ethereum state did not allow us to efficiently prune it. We had a variety of hacks and tricks to accumulate junk slower in the database, but we nonetheless kept accumulating it indefinitely. Users could stop their node and prune it offline; or resync the state to get rid of the junk. But it was a very non-ideal solution.

In order to implement and ship real pruning; one that does not leave any junk behind, we needed to break a lot of eggs within Geth’s codebase. Effort wise, we’d compare it to the Merge, only restricted to Geth’s internal level:

• Storing state trie nodes by hashes introduces an implicit deduplication (i.e. if two branches of the trie share the same content (more probable for contract storages), they get stored only once). This implicit deduplication means that we can never know how many parent’s (i.e. different trie paths, different contracts) reference some node; and as such, we can never know what is safe and what is unsafe to delete from disk.
  
  • Any form of deduplication across different paths in the trie had to go before pruning could be implemented. Our new data model stores state trie nodes keyed by their path, not their hash. This slight change means that if previously two branches has the same hash and were stored only once; now they will have different paths leading to them, so even though they have the same content, they will be stored separately, twice.

• Storing multiple state tries in the database introduces a different form of deduplication. For our old data model, where we stored trie nodes keyed by hash, the vast majority of trie nodes stay the same between consecutive blocks. This results in the same issue, that we have no idea how many blocks reference the same state, preventing a pruner from operating effectively. Changing the data model to path based keys makes storing multiple tries impossible altogether: the same path-key (e.g. empty path for the root node) will need to store different things for each block.
  
  • The second invariant we needed to break was the capability to store arbitrarily many states on disk. The only way to have effective pruning, as well as the only way to represent trie nodes keyed by path, was to restrict the database to contain exactly 1 state trie at any point in time. Originally this trie is the genesis state, after which it needs to follow the chain state as the head is progressing.

• The simplest solution with storing 1 state trie on disk is to make it that of the head block. Unfortunately, that is overly simplistic and introduces two issues. Mutating the trie on disk block-by-block entails a lot of writes. Whilst in sync it may not be that noticeable, but importing many blocks (e.g. full sync or catchup) it becomes unwieldy. The second issue is that before finality, the chain head might wiggle a bit across mini-reorgs. They are not common, but since they can happen, Geth needs to handle them gracefully. Having the persistent state locked to the head makes it very hard to switch to a different side-chain.
  
  • The solution is analogous to how Geth’s snapshots work. The persistent state does not track the chain head, rather it is a number of blocks behind. Geth will always maintain the trie changes done in the last 128 blocks in memory. If there are multiple competing branches, all of them are tracked in memory in a tree shape. As the chain moves forward, the oldets (HEAD-128) diff layer is flattened down. This permits Geth to do blazing fast reorgs within the top 128 blocks, side-chain switches essentially being free.
  • The diff layers however do not solve the issue that the persistent state needs to move forward on every block (it would just be delayed). To avoid disk writes block-by-block, Geth also has a dirty cache in between the persistent state and the diff layers, which accumulates writes. The advantage is that since consecutive blocks tend to change the same storage slots a lot, and the top of the trie is overwritten all the time; the dirty buffer short circuits these writes, which will never need to hit disk. When the buffer gets full however, everything is flushed to disk.

• With the diff layers in place, Geth can do 128 block-deep reorgs instantly. Sometimes however, it can be desirable to do a deeper reorg. Perhaps the beacon chain is not finalizing; or perhaps there was a consensus bug in Geth and an upgrade needs to “undo” a larger portion of the chain. Previously Geth could just roll back to an old state it had on disk and reprocess blocks on top. With the new model of having only ever 1 state on disk, there’s nothing to roll back to.
  
  • Our solution to this issue is the introduction of a notion called reverse diffs. Every time a new block is imported, a diff is created which can be used to convert the post-state of the block back to it’s pre-state. The last 90K of these reverse diffs are stored on disk. Whenever a very deep reorg is requested, Geth can take the persistent state on disk and start applying diffs on top until the state is mutated back to some very old version. Then is can switch to a different side-chain and process blocks on top of that.

The above is a condensed summary of what we needed to modify in Geth’s internals to introduce our new pruner. As you can see, many invariants changed, so much so, that Geth essentially operates in a completely different way compared to how the old Geth worked. There is no way to simply switch from one model to the other.

We of course recognize that we can’t just “stop working” because Geth has a new data model, so Geth v1.13.0 has two modes of operation (talk about OSS maintanance burden). Geth will keep supporting the old data model (furthermore it will stay the default for now), so your node will not do anything “funny” just because you updated Geth. You can even force Geth to stick to the old mode of operation longer term via –state.scheme=hash.

If you wish to switch to our new mode of operation however, you will need to resync the state (you can keep the ancients FWIW). You can do it manually or via geth removedb (when asked, delete the state database, but keep the ancient database). Afterwards, start Geth with –state.scheme=path. For now, the path-model is not the default one, but if a previous database already exist, and no state scheme is explicitly requested on the CLI, Geth will use whatever is inside the database. Our suggestion is to always specify –state.scheme=path just to be on the safe side. If no serious issues are surfaced in our path scheme implementation, Geth v1.14.x will probably switch over to it as the default format.

A couple notes to keep in mind:

• If you are running private Geth networks using geth init, you will need to specify –state.scheme for the init step too, otherwise you will end up with an old style database.
• For archive node operators, the new data model will be compatible with archive nodes (and will bring the same amazing database sizes as Erigon or Reth), but needs a bit more work before it can be enabled.

Also, a word of warning: Geth’s new path-based storage is considered stable and production ready, but was obviously not battle tested yet outside of the team. Everyone is welcome to use it, but if you have significant risks if your node crashes or goes out of consensus, you might want to wait a bit to see if anyone with a lower risk profile hits any issues.

Now onto some side-effect surprises…

Semi-instant shutdowns

Head state missing, repairing chain…

…the startup log message we’re all dreading, knowing our node will be offline for hours… is going away!!! But before saying goodbye to it, lets quickly recap what it was, why it happened, and why it’s becoming irrelevant.

Prior to Geth v1.13.0, the Merkle Patricia trie of the Ethereum state was stored on disk as a hash-to-node mapping. Meaning, each node in the trie was hashed, and the value of the node (whether leaf or internal node) was inserted in a key-value store, keyed by the computed hash. This was both very elegant from a mathematical perspective, and had a cute optimization that if different parts of the state had the same subtrie, those would get deduplicated on disk. Cute… and fatal.

When Ethereum launched, there was only archive mode. Every state trie of every block was persisted to disk. Simple and elegant. Of course, it soon became clear that the storage requirement of having all the historical state saved forever is prohibitive. Fast sync did help. By periodically resyncing, you could get a node with only the latest state persisted and then pile only subsequent tries on top. Still, the growth rate required more frequent resyncs than tolerable in production.

What we needed, was a way to prune historical state that is not relevant anymore for operating a full node. There were a number of proposals, even 3-5 implementations in Geth, but each had such a huge overhead, that we’ve discarded them.

Geth ended up having a very complex ref-counting in-memory pruner. Instead of writing new states to disk immediately, we kept them in memory. As the blocks progressed, we piled new trie nodes on top and deleted old ones that weren’t referenced by the last 128 blocks. As this memory area got full, we dripped the oldest, still-referenced nodes to disk. Whilst far from perfect, this solution was an enormous gain: disk growth got drastically cut, and the more memory given, the better the pruning performance.

The in-memory pruner however had a caveat: it only ever persisted very old, still live nodes; keeping anything remotely recent in RAM. When the user wanted to shut Geth down, the recent tries – all kept in memory – needed to be flushed to disk. But due to the data layout of the state (hash-to-node mapping), inserting hundreds of thousands of trie nodes into the database took many many minutes (random insertion order due to hash keying). If Geth was killed faster by the user or a service monitor (systemd, docker, etc), the state stored in memory was lost.

On the next startup, Geth would detect that the state associated with the latest block never got persisted. The only resolution is to start rewinding the chain, until a block is found with the entire state available. Since the pruner only ever drips nodes to disk, this rewind would usually undo everything until the last successful shutdown. Geth did occasionally flush an entire dirty trie to disk to dampen this rewind, but that still required hours of processing after a crash.

We dug ourselves a very deep hole:

• The pruner needed as much memory as it could to be effective. But the more memory it had, the higher probability of a timeout on shutdown, resulting in data loss and chain rewind. Giving it less memory causes more junk to end up on disk.
• State was stored on disk keyed by hash, so it implicitly deduplicated trie nodes. But deduplication makes it impossible to prune from disk, being prohibitively expensive to ensure nothing references a node anymore across all tries.
• Reduplicating trie nodes could be done by using a different database layout. But changing the database layout would have made fast sync inoperable, as the protocol was designed specifically to be served by this data model.
• Fast sync could be replaced by a different sync algorithm that does not rely on the hash mapping. But dropping fast sync in favor of another algorithm requires all clients to implement it first, otherwise the network splinters.
• A new sync algorithm, one based on state snapshots, instead of tries is very effective, but it requires someone maintaining and serving the snapshots. It is essentially a second consensus critical version of the state.

It took us quite a while to get out of the above hole (yes, these were the laid out steps all along):

• 2018: Snap sync’s initial designs are made, the necessary supporting data structures are devised.
• 2019: Geth starts generating and maintaining the snapshot acceleration structures.
• 2020: Geth prototypes snap sync and defines the final protocol specification.
• 2021: Geth ships snap sync and switches over to it from fast sync.
• 2022: Other clients implement consuming snap sync.
• 2023: Geth switches from hash to path keying.
  
  • Geth becomes incapable of serving the old fast sync.
  • Geth reduplicates persisted trie nodes to permit disk pruning.
  • Geth drops in-memory pruning in favor of proper persistent disk pruning.

One request to other clients at this point is to please implement serving snap sync, not just consuming it. Currently Geth is the only participant of the network that maintains the snapshot acceleration structure that all other clients use to sync.

Where does this very long detour land us? With Geth’s very core data representation swapped out from hash-keys to path-keys, we could finally drop our beloved in-memory pruner in exchange for a shiny new, on-disk pruner, which always keeps the state on disk fresh/recent. Of course, our new pruner also uses an in-memory component to make it a bit more optimal, but it primarilly operates on disk, and it’s effectiveness is 100%, independent of how much memory it has to operate in.

With the new disk data model and reimplemented pruning mechanism, the data kept in memory is small enough to be flushed to disk in a few seconds on shutdown. But even so, in case of a crash or user/process-manager insta-kill, Geth will only ever need to rewind and reexecute a couple hundred blocks to catch up with its prior state.

Say goodbye to the long startup times, Geth v1.13.0 opens brave new world (with –state.scheme=path, mind you).

Drop the –cache flag

No, we didn’t drop the –cache flag, but chances are, you should!

Geth’s –cache flag has a bit of a murky past, going from a simple (and ineffective) parameter to a very complex beast, where it’s behavior is fairly hard to convey and also to properly account.

Back in the Frontier days, Geth didn’t have many parameters to tweak to try and make it go faster. The only optimization we had was a memory allowance for LevelDB to keep more of the recently touched data in RAM. Interestingly, allocating RAM to LevelDB vs. letting the OS cache disk pages in RAM is not that different. The only time when explicitly assigning memory to the database is beneficial, is if you have multiple OS processes shuffling lots of data, thrashing each other’s OS caches.

Back then, letting users allocate memory for the database seemed like a good shoot-in-the-dark attempt to make things go a bit faster. Turned out it was also a good shoot-yourself-in-the-foot mechanism, as it turned out Go’s garbage collector really really dislikes large idle memory chunks: the GC runs when it piles up as much junk, as it had useful data left after the previous run (i.e. it will double the RAM requirement). Thus began the saga of Killed and OOM crashes…

Fast-forward half a decade and the –cache flag, for better or worse, evolved:

• Depending whether you’re on mainnet or testnet, –cache defaults to 4GB or 512MB.
• 50% of the cache allowance is allocated to the database to use as dumb disk cache.
• 25% of the cache allowance is allocated to in-memory pruning, 0% for archive nodes.
• 10% of the cache allowance is allocated to snapshot caching, 20% for archive nodes.
• 15% of the cache allowance is allocated to trie node caching, 30% for archive nodes.

The overall size and each percentage could be individually configured via flags, but let’s be honest, nobody understands how to do that or what the effect will be. Most users bumped the –cache up because it lead to less junk accumulating over time (that 25% part), but it also lead to potential OOM issues.

Over the past two years we’ve been working on a variety of changes, to soften the insanity:

• Geth’s default database was switched to Pebble, which uses caching layers outide of the Go runtime.
• Geth’s snapshot and trie node cache started using fastcache, also allocating outside of the Go runtime.
• The new path schema prunes state on the fly, so the old pruning allowance was reassigned to the trie cache.

The net effect of all these changes are, that using Geth’s new path database scheme should result in 100% of the cache being allocated outside of Go’s GC arena. As such, users raising or lowering it should not have any adverse effects on how the GC works or how much memory is used by the rest of Geth.

That said, the –cache flag also has no influece whatsoever any more on pruning or database size, so users who previously tweaked it for this purpose, can drop the flag. Users who just set it high because they had the available RAM should also consider dropping the flag and seeing how Geth behaves without it. The OS will still use any free memory for disk caching, so leaving it unset (i.e. lower) will possibly result in a more robust system.

Epilogue

As with all our previous releases, you can find the:

]]> https://earlybirdsinvest.com/geth-v1-13-0/feed/ 0 26492