Ethereum Mainnet Shadow Forking: An Overview

It refers to using data from a testnet or the mainnet to test sync assumptions for a network upgrade, so that developers can test features before deploying the actual upgrade to the mainnet.

Ethereum Mainnet Shadow Forking: An Overview

The transition of Ethereum from Proof-of-Work (PoW) to Proof-of-Stake (PoS) consensus is getting closer, with developers launching the first mainnet shadow fork on April 11th. The goal is to put existing assumptions about testnets and the mainnet to the test. This was one of the most important historical events in the Merge upgrade. According to Marius Van Der Wijden, a developer at Go-ethereum Client, this mainnet shadow fork was a huge success.

An Update (April 23, 2022)

Some Ethereum client developers are meeting in Amsterdam for Devconnect, a week-long in-person gathering featuring independent Ethereum events.

#TestingTheMerge team Shadow forked the Ethereum Mainnet for the second time on April 23, 2022. Terminal total difficulty (TTD) was hit and the shadow fork of Mainnet is running on Proof of Stake. Except Erigon, other consensus & execution layer client combinations are synced.

According to Danny Ryan "it seems to be gone well", live tweeted Ben Edgington from Amsterdam.

Progress of Shadow Fork 2 can be followed on the EthStats. Green is good!

sf

Table of Content

Merge

What is Shadow Forking?

It refers to using data from a testnet or the mainnet to test sync assumptions for a network upgrade, so that developers can test features before deploying the actual upgrade to the mainnet.

A shadow fork is a test limited to a few weeks in which developers fork an existing testnet, take its config and add merge related fields such as Total Terminal Difficulty (TTD) and Merge Fork Block (for peering, forkID changes). They have to spin up a new beacon chain for testing purposes. The new fork essentially inherits the state/txs of the canonical testnet. Inheriting the state of existing testnets allows to stress test sync assumptions as well as assumptions around how long it takes to build a block/timeouts. The interesting part of staying connected to the peers on the canonical chain allows the team to import some of their transaction gossip as well.

“A shadow fork does not affect the canonical chain in any meaningful way. Since we reuse the chainID and the gossip channels are still connected, transactions submitted to the shadow fork could be included in the main chain as well. Proceed with extreme caution!” said Parithosh Jayanthi.

A shadow fork can transition to proof of stake after reaching TTD while still having mainnet (testnet or Ethereum Mainnet) transactions and other history/state information. After a successful merge on the shadow fork, it then continues on the proof of stake chain, ignoring any blocks that are added using proof of work on the mainnet.

Timeline

January:

  • Goerli Testnet was shadow forked 1st time on January 21st, 2022. It was shadow-forked as part of a test to experiment different merge scenarios in preparation for The Merge upgrade to the actual testnet.

March:

  • Later, Goerli Testnet was shadow forked two more times.

April:

  • The first mainnet shadow fork was done on April 11 and second was done 23rd April.

Why did Ethereum Developers do Shadow Forking?

Ethereum is a multi-billion dollar chain. The Merge is a big upgrade for changing the consensus mechanism. Ethereum developers are exploring innovative ways for testing before getting to real testnet such as Goerli, Ropstan etc.

  • The shadow fork allows developers to stress test.
  • Possibly expose bugs in clients and would suggest optimizations needed.
  • Helping test the realistic scenario and for Testing Optimistic Sync on Consensus Layer and Snap Sync on the Execution Layer.

The Kiln Merge testnet aimed to allow the community to practice running their nodes, deploying contracts, testing infrastructure, etc. However, since it was a fresh testnet with a little activity, the team needed a way to stress test their assumptions around syncing and state growth.

The team additionally needed a way to check if their assumptions work on existing testnets and mainnet. So that's why they have shadow forked an existing testnet and added Merge related fields such as Total Terminal Difficulty (TTD) and Merge Fork Block.

Testnets don't have much state as they run for weeks or months. Also, no one uses them, but the mainnet is used quite a lot, so it’s state is huge, i.e. around 50 gigabytes. So, developers needed to synchronize all of the states and shadow forking gave them an excellent way to test the synchronization. Also, the synchronization takes way longer on the mainnet, as the possibilities of bugs are higher.

Goerli Testnet Shadow Forking

In this section, we will understand how developers did the shadow forking of Goerli Testnet.

So, firstly developers deployed a contract on Goerli with all the validators. Once the TTD, i.e. Total Terminal Difficulty was hit, those nodes that developers configured to follow the proof of stake chain after a specific block or after a particular total difficulty went on and merged, i.e. switched to Proof of Stake.

Basically they tested the merge on a real testnet by creating a fork. Developers deployed a genesis contract on Goerli and deposited enough validators to start a chain. It went through all the stages of the merge. Transactions from Goerli were also executed on the testnet, and the network was finalizing.

So this makes shadow forking a good technique as it gives us a way to test whether the fork works and the merging mechanism works without disturbing the public testnet. Here is a link to the Goerli Shadow Fork Testing Plan. The aim of this document is to list various test cases that can be performed. The goal is to repeat this shadow fork process relatively often, allowing us to test the merge transition in various scenarios.

After the 1st shadow fork, two more shadow forks of Goerli were done to find more bugs and better preparation before The Merge.

Bugs

During the Goerli Shadow forking, some bugs were encountered when the testnet was approaching The Merge. So, developers had to reconfig the nodes at some point. Despite that, some issues persisted and developers were able to find loads of pitfalls and issues during deployment. This was a huge leap towards the merge testing. During the last two shadow forks of Goreli, various bugs varying from sync code to request timeouts were found.

Mainnet Shadow Forking 1

After the first mainnet shadow fork goes live, the blocks are produced and finalized with no issues.

The node split on this shadow-fork attempts to mimic the mainnet. Here's a grafana dashboard where Green is Good, and Red is Bad.

The participation rate did drop, but the it is still well above the minimum required for finality.

This was the first attempt toward a mainnet-shadow-fork, the team is expecting to learn a lot from this transition.

The following week would be spent with sync tests against this fork and trying to trigger more edge cases. The team is also planning to repeat it next week for more advanced users.

The shadow fork has already processed 6,345,216 transactions with an average block time of 14.3 seconds as of now. We can find all the latest details on the Transaction Explorer.

We can find more details like Network Uptime, Block Propagation, Average Network Hashrate etc. on Ethstats.

Many thanks to Parithosh Jayanthi, a developer at Ethereum Foundation, for all this information and images.

Bugs

According to Marius Van Der Wijden, Nethermind and Besu stopped at the transition, but a fix is being deployed for Nethermind that allows them to sync up. Geth and Erigon are progressing. All beacon chain clients are now in agreement.

According to Marek Moraczyński, a Software Engineer working at Nethermind, most of the Nethermind nodes were working fine on mainnet shadow fork but due to the wrong conditions in the beacon headers sync, their nodes were stuck with syncing.

Mainnet Shadow Forking 2

Mainnet Shadow Fork 2 was also successful. Developers have seen a really high participation for attestations and sync committees. There are also a few nodes still syncing the Execution Layer(EL) which has given developers a good test of optimistic sync. In particular some nodes had the Consensus Layer(CL) in sync prior to the merge but the EL not yet sync'd.

Here is a quick screenshot of teku and geth successfully validating.

Here is a quick screenshot of teku and besu successfully validating.

Credits to Adrian Sutton, a Lead Blockchain Protocol Engineer at ConsenSys, for all this available information and images.

Difference between Mainnet Shadow Forking 1 & 2

This was the first shadow fork where every client combination survived the transition and managed to stay in sync afterwards.Another small difference is that this shadow fork used the develop/unstable branch of every client, so developers aren't using merge branches anymore.

Developers have reused the mainnet deposit contract with a new fork ID. This means that every mainnet deposit needs to be processed and listed as invalid on the shadow fork. This huge computation triggered some edge cases in some clients, the good news is that the network still chugged along. Developers are gonna continue monitoring the network over the week to look for issues, perform sync tests and stress the nodes.

Credits to Parithosh Jayanthi, a developer at Ethereum Foundation, for all this available information.

Bugs

Prior to the merge, both Prysm and Nimbus had some issues handling the huge stream of deposits that were being processed. Prysm was generating blocks very late causing a lot of missed head votes and low sync committee participation. Nimbus was producing invalid blocks. Neither issue was related to the merge but both have been resolved. The actual merge went through fine despite those issues and having so many late blocks.

MetaMask

Related Videos


Disclaimer: The information contained on this web page is for education purpose only. Readers are suggested to conduct their own research, review, analyze and verify the content before relying on them.

To publish press releases, project updates and guest posts with us, please email at contact@etherworld.co.

Subscribe to EtherWorld YouTube channel for easy digestable content.

Support us at Gitcoin

You've something to share with the blockchain community, join us on Discord!

Follow us at Twitter, Facebook, LinkedIn, and Instagram.


Share Tweet Send
0 Comments
Loading...
You've successfully subscribed to EtherWorld.co
Great! Next, complete checkout for full access to EtherWorld.co
Welcome back! You've successfully signed in
Success! Your account is fully activated, you now have access to all content.