How Holesky Finally Reached Stability

The Ethereum testnet, Holesky, has finally reached finalization after nearly two weeks of non-finality, marking a significant milestone for validators, node operators, and the wider Ethereum staking community.

The past couple of weeks have been a rollercoaster of troubleshooting, patching, and coordination as contributors across the ecosystem worked tirelessly to restore stability. Here’s a deep dive into what happened, how it was resolved, and the key takeaways for the future.

What Happened to Holesky Testnet?

Holesky, a crucial Ethereum testnet, encountered an extended period of non-finality that lasted almost two weeks. This issue meant that validators and nodes were burdened with excessive state storage, as finalization was not occurring as expected.

Over time, this accumulation of state data led to increased memory and computational demands on nodes, causing instability and downtime for many participants. The prolonged non-finality in Holesky led to several significant challenges.

One of the primary issues was state overload, as validators were forced to maintain a vast amount of historical state data due to the lack of finalization. This dramatically increased resource consumption and placed additional strain on node operators.

In an effort to address the problem, some modifications were made to experimental branches, but instead of alleviating the situation, these changes unintentionally took several clients offline. Once finality was finally restored, nodes had to prune outdated state data efficiently to regain stability.

This required operators to perform checkpoint syncs from the last finalized state to ensure smooth recovery. Additionally, the entire resolution process demanded extensive coordination among multiple teams, client developers, and node operators, highlighting the complexity of managing such an unprecedented network challenge.

How Was It Resolved?

On Epoch 119090, Holesky successfully finalized again. The resolution process involved several key steps. First, operators were advised to clear their consensus layer (CL) data directories and perform a checkpoint sync using the Checkpoint Sync for Holesky.

This approach allowed nodes to quickly resync from the last finalized state without having to process the overwhelming backlog of non-finalized blocks. Additionally, backup nodes played a vital role in the recovery, with the Holesky-Rescue service providing an alternative recovery route for those experiencing difficulties.

Once finalization was restored, nodes had to undergo a significant pruning process to clear outdated state data and return to stable operation. Finally, various patch deployments and client-side adjustments were implemented to address the root causes of the outage, ensuring that similar disruptions could be mitigated in the future.

These collective efforts played a pivotal role in stabilizing Holesky and bringing it back online.

The successful restoration of Holesky’s stability was made possible through the dedication and relentless efforts of numerous individuals who put in long hours to resolve the crisis. Huge appreciation goes to @skylenet, @pk910, whose tools proved invaluable even in extreme edge cases, @BarnabasBusa, and @samcmAU for their significant contributions.

Special recognition is also due to Paul Harris and @r_krasiuk, who played a crucial role in coordinating efforts during the initial stages of the crisis. Additionally, @ChodoKamil was instrumental in ensuring effective communication and verification among all involved parties, facilitating a smooth resolution process.

Their collective commitment and expertise were essential in bringing Holesky back online. The Ethereum community's ability to quickly collaborate, coordinate, and implement solutions remains a fundamental strength in maintaining network health.

With Holesky back to finalization, the focus now shifts to detailed retrospectives over the coming weeks. These reviews will provide deeper insights into the incident, allowing the Ethereum ecosystem to learn from this challenge and for protection against future disruptions.

Related Articles

1. Ethereum’s Institutional & Government Adoption
2. Solving the Puzzle of Duplicate Blocks in Ethereum
3. Ethereum Developers are Rethinking Transaction Signatures & Authority
4. The Debate Over Freezing Ethereum's Core for Good
5. Fixing Ethereum’s Message Signing Chaos