Shapella Testing

Shapella Testing Timeline, Shandong Testnet, Withdrawals Devnets, Mainnet Shadow Forks, Zhejiang Testnet & Withdrawal Testing Techniques

Shapella Testing

As the Shapella network upgrade is approaching in April 2023, rigorous testing is being conducted to ensure a smooth transition and minimize any potential risks or issues during the upgrade. The Shapella upgrade consists of two parts, the Shanghai Execution Layer upgrade, and the Capella Consensus Layer upgrade. In this article, we will discuss all the testing done in the Shapella upgrade and the different testing techniques used.

Shapella Testing Timeline

  • 13th Oct 2022: Shandong Testnet
  • 9th Dec 2022: Withdrawal Devnet 0
  • 22nd Dec 2022: Withdrawal Devnet 1
  • 10th Jan 2023: Withdrawal Devnet 2
  • 17th Jan 2023: Withdrawal Devnet 3
  • 24th Jan 2023: Withdrawal Mainnet Shadow Fork 1
  • 24th Jan 2023: Withdrawal Devnet 4
  • 26th Jan 2023: Withdrawal Devnet 5
  • 27th Jan 2023: Withdrawal Devnet 6
  • 1st Feb 2023: Zhejiang Testnet
  • 16th Feb 2023: Withdrawal Devnet 7
  • 28th Feb 2023: Sepolia Testnet
  • 3rd Mar 2023: Withdrawal Mainnet Shadow Fork 2
  • 14th Mar 2023: Goerli Testnet
  • 6th Apr 2023: Withdrawal Mainnet Shadow Fork 3

Shandong Testnet

It was an experimental testnet that ran in association between EF JavaScript Team and EF DevOps Team. A set of Shanghai-considered EIPs was selected for early client testing on this Testnet.

It was a Pure JS-based testnet, i.e., running on a set of Lodestar (CL) and EthereumJS (EL) client boot nodes.

To learn more, readers can follow our earlier article on Shandong Testnet.

Withdrawals Devnets

Testnets are the testing environment where community members can participate and test. On the other hand, devnets are only for developers and testers.

  • Withdrawal Devnet 0:

It was mainly used to test all the essential testing functionalities after the Capella side of the fork.

Bugs: Core Developers found a bug in Lodestar, where the pool content of the BLS changes was not distributed to lodestar nodes.

It was again relaunched with 28000 validators with updated specs.

Bugs: Core Developers again found a bug in Lodestar. It was unable to parse the payload fetched from Nethermind. Also, Shanghai Timestamp was no longer printed on Geth Nodes.

TX-Fuzz was also used in the Devnet 0. However, core Developers decided to make it standard on every devnet now.

TX-Fuzz is a package containing helpful functions to create random transactions.

  • Withdrawal Devnet 1:

It was a semi-public testnet with some external entities running validators to test features. It was successfully running and finalized.

Bugs: Lighthouse called V2 methods, whereas Teku, Prysm, and Lodestar call V1 methods. This was a minor issue. However, some bad blocks were seen in clients.

  • Withdrawal Devnet 2:

This devnet followed CL spec v1.3.0-rc.0. It also tested EIP-6122: Forkid checks based on timestamps in addition to Shanghai EIPs.

Bugs: There was an issue in EthereumJS where it sometimes produced bad blocks. Also, an issue in Geth was found where the transaction pool would accept bad transactions. Both these issues were found by TX-Fuzz.

  • Withdrawal Devnet 3:

After the discussions on Consensus-Layer Call 101, Core Developers wanted to explore harmonization in the withdrawals format from CL to EL. As a result, a minor change was introduced in the withdrawal format, where the field amount in the withdrawal structure was changed from wei to gwei.

This devnet followed CL spec v1.3.0-rc.1 and Engine API spec with respect to commit 939255f . In addition, it also tested EIP-6122: Forkid checks based on timestamps in addition to Shanghai EIPs.

Bugs: Besu nodes saw a bad block.

  • Withdrawal Devnet 4:

There were a total of 605k validators, where 565k validators were run by EF and 40k validators by other client teams. Spec & features were similar to Devnet 3. Developers wanted to mimic the mainnet environment as much as possible. They have also increased the gas limit to 25M from 4M to include larger contracts. Its main goal was to test BLS submission to the local pool before Capella.

  • Withdrawal Devnet 5:

Here, developers were running 4 bootnodes instead of 1 for easier peering and less load on a single bootnode.

  • Withdrawal Devnet 6:

Some Besu Nodes were producing bad blocks.

  • Withdrawal Devnet 7:

Withdrawal Devnet 7 was launched on February 14th. The Capella fork took place 48 hours later on February 16th, bringing together 600K validators, 300K BLS changes, and 100K exits, all in an effort to simulate worst-case scenarios.

Bugs: The Besu Client recently encountered a problem where it was creating bad blocks due to a misreporting of proposed bad blocks. The developers quickly identified the problem and deployed a fix to address the issue.

Withdrawal Mainnet Shadow Forks

Do you know there were 19 Shadow Forks done during The Merge Upgrade (13 Mainnet Shadow Forks + 6 Goerli Testnet Shadow Forks)?

Shadow Forking refers to using data from a testnet or the mainnet to test sync assumptions for a network upgrade so that developers can test features before deploying the actual upgrade to the mainnet.

  • It possibly exposes bugs in clients and would suggest optimizations needed.
  • It helps developers test the realistic scenario.
  • It helps developers increase their confidence that implementations work as expected.

Withdrawal Mainnet Shadow Fork 1: It started with a few issues because the configuration wasn't correctly applied on Geth. Core Developers have also used some evil nodes that send bad blocks, bad network messages, etc., and trick other nodes into the bad chain. These nodes try to spam invalid blocks/messages Both on the execution layer and the consensus layer.

Withdrawal Mainnet Shadow Fork 2: Shapella Fork on Withdrawal Mainnet Shadow Fork 2 was successfully launched on March 3, 2023, and performed well with all client combinations. To ensure the network's stability, developers also ran several rigorous scenario tests, such as taking the mock relay offline and triggering circuit breaker conditions of missing blocks in a row/epoch. These tests proved successful, as the circuit breaker was triggered as expected in clients with it implemented, and the network continued to finalize.

Withdrawal Mainnet Shadow Fork 3: Developers prepared for the MSF-3 by implementing the latest recommended release of clients and lining up BLS changes. They also configured MEV-Boost at a reasonable level and had two builders present on the network. In addition, they set up validator operations, depositing a few validators, and exiting a few without BLS changes applied. This thorough setup allowed the developers to test the Capella and Shanghai upgrades effectively.

Zhejiang Testnet

Zhejiang was the first Shanghai-Capella (Shapella) public testnet. It meant to invite community to test how Staked ETH Withdrawal process will look like after the mainnet upgrade.

To learn more, readers can follow our earlier article on Zhejiang Testnet.

Sepolia Testnet

The activation of the Shapella fork on the Sepolia testnet at epoch 56832 was successful, occurring at 4:04:48 AM UTC on February 28, 2023. Sepolia marks the second public testnet to test the Shapella Fork after Zhejiang, and we're thrilled to report that it's performing exceptionally well with all client combinations.

To learn more, readers can follow our earlier article on Sepolia Testnet.

Goerli Testnet

The Shapella fork has successfully been activated on the Goerli testnet at epoch 162304, occurring at 10:25:36 PM UTC on March 14, 2023. This is the third public testing of the Shapella fork, following its earlier tests on the Zhejiang and Sepolia testnets.

To learn more, readers can follow our earlier article on Goerli Testnet.

Withdrawal Testing Techniques

Shapella network upgrade is being extensively tested using various testing techniques to ensure a smooth and secure transition. The testing has covered all the different aspects of the upgrade, interoperability, and all the expected EIPs. In this section, we will discuss various techniques used in Shapella Testing.

  • Local and Interop Devnets: Testing the compatibility and interoperability of different clients is crucial in ensuring that the upgrade works seamlessly and as intended. The first step is to verify that the clients follow the specs and have implemented all the functions correctly. This is essential to avoid any potential compatibility issues during the upgrade process.
  • Spec Tests: This acts as a sanity check to ensure that clients aren't implementing a spec that won't pass tests. It is of 3 types; Consensus spec tests, State tests, and Execution spec tests.
  • Hive Tests: It is an interoperability tool that is used in testing between multiple clients at the same time. It is very short-lived and deterministic. It is highly useful in happy path tests,i.e., a well-defined test case using known input, which executes without exception, produces an expected output, and also, in bad scenarios. It runs using a simulator that starts up the clients and runs the tests against a predefined interface. It acts as an integration and regression check to ensure clients aren't failing defined edge cases. Here are examples of various test scenarios used in Withdrawals Testing.
  • Kurtosis Tests: It spins up a local testnet with the required EL/CL combinations and then allows them to transition. It then asserts some happy case conditions. Finally, it acts as an integration test to make sure clients are compatible. It is helpful in rapidly iterating ideas. It is also available as a Go library, and developers can write Go tests with kurtosis.

Related Videos


Disclaimer: The information contained on this web page is for education purposes only. Readers are suggested to conduct their own research, review, analyze and verify the content before relying on them.

To publish press releases, project updates and guest posts with us, please email at

Subscribe to EtherWorld YouTube channel for ELI5 content.

Support us at Gitcoin

You've something to share with the blockchain community, join us on Discord!

Follow us at Twitter, Facebook, LinkedIn, and Instagram.

Share Tweet Send
You've successfully subscribed to
Great! Next, complete checkout for full access to
Welcome back! You've successfully signed in
Success! Your account is fully activated, you now have access to all content.