The merger is imminent: a detailed explanation of Ethereum’s latest technical route

If "The Merge" goes smoothly, sharding will become the main development axis of Ethereum in 2023 and beyond, and its meaning has changed a lot since it was proposed in 2015.

After Vitalik proposed the "Rollup-centric Ethereum roadmap" and Ethereum's "Endgame", Ethereum's general direction has actually changed - "stepping back to the background" as the security guarantee and data availability layer of Rollup.

Danksharding and Proto-Danksharding are a series of technical combinations, which are manifested in a combination of "finding problems" and introducing or proposing new technologies to "solve problems".

If the timeline is extended to the next few years, the overall value of Rollup will become greater: Ethereum will present a multi-Rollup development pattern, cross-Rollup infrastructure will be highly improved, and the Rollup ecosystem will be highly prosperous - even surpassing Ethereum itself.

Introduction

Image source: https://consensys.net/blog/blockchain-explained/the-roadmap-to-serenity-2/

In the blink of an eye, 2022 is already halfway through. Looking back at the Serenity Roadmap proposed by Vitalik in his 2018 Devcon speech, it is easy to find that Ethereum's development path has changed several times - compared with the current roadmap, sharding has been given a new meaning, and eWASM is rarely mentioned.

In order to avoid potential fraud and user misleading, at the end of January this year, the Ethereum Foundation announced that it would abandon the term "ETH2" and instead change the current Ethereum mainnet to the "execution layer" for processing transactions and execution, and change the original ETH2 term to the "consensus layer" for coordinating and processing PoS.

Currently, the official Ethereum roadmap covers three parts: beacon chain, merger and sharding.

Among them, the Beacon Chain, as the preparatory work for Ethereum's migration to PoS and the coordination network of the consensus layer, was launched on December 1, 2020 and has been running for nearly 20 months.

The Merge refers to the final merger of the current Ethereum mainnet and the Beacon Chain, that is, the unification of the execution layer and the consensus layer, marking the official migration of Ethereum to PoS. In IOSG's article "Dawn is Coming: Ethereum Merger is Near", we introduced the important progress related to the merger: the current Ethereum Ropsten and Sepolia test networks have successfully completed the merger, followed by the Goerli merger; if everything goes well, it means that we are not far from the mainnet merger.

Image source: https://medium.com/decipher-media/blockchain-scaling-solutions-4-1-ethereum-sharding-e88e8cacdfc

In this article, we will focus on sharding. The reason is:

First, assuming that the mainnet merger can be successfully achieved within the year, sharding will follow closely and become the main development axis of Ethereum in 2023.

Secondly, the concept of Ethereum sharding was first proposed by Vitalik in Devcon 1 in 2015. Since then, GitHub's Sharding FAQ has proposed six stages of sharding development (as shown above). However, with the update of the Ethereum roadmap and the promotion of related EIPs, the meaning and priority of sharding have changed a lot. When we discuss sharding, we need to ensure that we have a consistent understanding of its meaning.

In summary, it is important to sort out the ins and outs of sharding. This article will focus on the origin, progress and future direction of Ethereum original sharding, Danksharding and Proto-Danksharding, rather than going into each technical detail. For details about Danksharding and Proto-Danksharding, please refer to IOSG's previous articles: "Will Danksharding, the killer of expansion, be the future of Ethereum sharding?", "EIP4844: The foreseeable depression effect of L2 transaction fee reduction is about to start."

Quick Review

Rollup, data availability and sharding will be mentioned many times in this article.

Here we quickly review the basic concepts of the three.

The current mainstream Rollup is divided into zkRollup and Optimistic Rollup. The former is based on proof of validity, that is, batch execution of transactions, relying on cryptographic proof SNARK to ensure the correctness of state transitions; the latter "optimistically" assumes that all state transitions are correct unless they are falsified; that is, a time window is required to ensure that incorrect state transitions can be discovered.

Data availability is very important for both zkRollup and Optimistic Rollup. For the former, users can reconstruct all transactions on the second layer based on data availability to ensure anti-censorship; for the latter, all data on the second layer needs to be published, and no transaction is hidden. As for the current bottlenecks facing data availability and corresponding solutions, they will be mentioned below.

Image source: https://www.web3.university/article/ethereum-sharding-an-introduction-to-blockchain-sharding

Ethereum full nodes store the complete state of EVM and participate in all transaction verifications, which ensures decentralization and security, but with it comes the problem of scalability: transactions are executed linearly and require each node to confirm one by one, which is undoubtedly inefficient.

In addition, as time goes by, the Ethereum network data continues to accumulate (currently up to 786GB), and the hardware requirements for running full nodes have risen accordingly. A decrease in the number of full nodes will lead to potential single points of failure and weaken the degree of decentralization.

Intuitively, sharding is equivalent to division of labor, that is, all nodes are grouped, each transaction only needs to be verified by a single group of nodes, and transaction records are regularly submitted to the main chain, so as to achieve parallel processing of transactions (for example, there are 1,000 nodes, and each transaction must be verified by each node; if they are divided into 10 groups, each with 100 nodes to verify the transaction, the efficiency is obviously greatly improved). Sharding improves scalability while reducing the hardware requirements of a single group of nodes, thus solving the above two problems.

Original Sharding

Image source: https://vitalik.ca/general/2021/04/07/sharding.html

There are 64 shards in the original Ethereum plan, each with an independent proposer and committee. The proposer is a randomly selected validator who collects and sorts transactions; the committee is a set of validators (consisting of at least 128 validators) who are randomly assigned to each shard at regular intervals and verify the validity of the transaction. If 2/3 of the committee votes in favor, the validator management contract (VMC) is called to submit transaction records to the beacon chain. Different from the "data sharding" described below, this sharding is also called "execution sharding".

Background

Before talking about Danksharding, let's take some time to understand its background. Personally, I guess that the community atmosphere launched by Danksharding mainly comes from two articles by Vitalik. These two articles set the tone for the future development direction of Ethereum.

First, Vitalik published the "Rollup-centric Ethereum Roadmap" in October 2020, proposing that Ethereum needs to centrally support Rollup in the short and medium term. First, the expansion of the Ethereum base layer will focus on expanding the data capacity of the block, rather than improving the efficiency of on-chain computing or IO operations. That is: Ethereum sharding is designed to provide more space for data blobs (not transactions), and Ethereum does not need to interpret this data, only to ensure that the data is available. Second, Ethereum's infrastructure is adjusted to support Rollup (such as L2 support for ENS, L2 integration of wallets, and cross-L2 asset transfers). In the long run, the future of Ethereum should be a single execution shard with high security that can be processed by everyone, as well as a scalable data availability layer.

Image source: https://vitalik.ca/general/2021/12/06/endgame.html

After that, Vitalik described the final picture of Ethereum in "Endgame" published in December 2021: block output is centralized, but block verification is trustless and highly decentralized, while ensuring anti-censorship. The underlying chain provides guarantees for the data availability of the block, while Rollup provides guarantees for the validity of the block (in zkRollup, this is achieved through SNARK; in Optimistic Rollup, only one honest participant needs to run a fraud proof node). Similar to the multi-chain ecosystem of Cosmos, the future of Ethereum will be the coexistence of multiple Rollups - they are all based on the data availability and shared security provided by Ethereum. Users rely on bridges to move between different Rollups without having to pay the high fees of the main chain.

The above two articles basically determine the development direction of Ethereum: optimize the construction of Ethereum's basic layer to serve Rollup. The above argument may be based on the view that since Rollup has been proven to be effective and well adopted, "instead of spending several years waiting for an uncertain and complex expansion plan (note: referring to the original sharding), it is better to focus on the Rollup-based solution."

After this, Dankrad proposed a new sharding solution Danksharding. Below we will break down the specific technical components of Danksharding to understand.

Proto-Danksharding

Image source: https://l2fees.info/

The background of Proto-Danksharding is that although the Rollup solution significantly reduces transaction fees compared to the Ethereum main chain, it is not yet low enough to be ideal. This is because CALLDATA, which provides data availability on the Ethereum main chain, still occupies a large fee (16gas/byte). In the original concept, Ethereum proposed to provide 16MB of dedicated data space for Rollup in each block in the data sharding, but the actual implementation of data sharding is still a long way off.

On February 25 this year, Vitalik, DankRad and others proposed the EIP-4844 (Shard Blob Transactions) proposal, also known as Proto-Danksharding, which aims to expand Ethereum's data availability in a simple, forward-compatible way so that it is still available after the launch of Danksharding. The changes in this proposal only occur at the consensus layer, and no additional adaptation work is required for clients, users and Rollup developers at the execution layer.

Proto-Danksharding does not actually perform sharding, but introduces a transaction format called "Blob-carrying Transactions" for future sharding. This transaction format is different from ordinary transactions in that it carries an additional data block called a blob (about 125kB), which actually makes the block larger, thereby providing cheaper data availability than CALLDATA (about 10kB).

However, the common problem of "big blocks" is the increasing demand for disk space. The use of Proto-Danksharding will increase Ethereum's storage by an additional 2.5TB per year (the current total network data is only 986GB). Therefore, Proto-Danksharding sets a time window (for example, 30 days) after which the blob is deleted. Users or protocols can back up the blob data during this period.

That is, Ethereum's consensus layer only serves as a highly secure "real-time bulletin board" to ensure that the data is available for a long enough time and that other users or protocols have enough time to back up the data, rather than Ethereum permanently retaining all blob historical data.

The reason for this is that the annual increase of 2.5TB is not a problem for storage, but it brings a considerable burden to Ethereum nodes. As for the trust assumption problem that may result, in fact, only one data storage party is honest (1 of N) for the system to operate normally, and there is no need for a set of validator nodes (N/2 of N) that participate in real-time verification and consensus execution to store this part of historical data.

So, is there any incentive to encourage third parties to store this data? The author has not found the launch of the incentive plan for the time being, but Vitalik himself proposed several possible data storage parties:

1. Protocols for applications (such as Rollup). They can require nodes to store historical data related to the application. If the historical data is lost, it will pose a risk to this part of the application, so they are motivated to do storage;

2. BitTorrent;

3. Ethereum's Portal Network, which is a platform that provides lightweight access to the protocol;

4. Blockchain browsers, API providers or other data service providers;

5. Personal enthusiasts or scholars engaged in data analysis;

6. Third-party indexing protocols such as The Graph.

Danksharding Data Availability Sampling (DAS)

Image source: https://notes.ethereum.org/@hww/workshop_feb_2022

In Proto-Danksharding, we mentioned that the new transaction format actually makes the block larger, and Rollup also accumulates a lot of data, which nodes need to download to ensure data availability.

The idea of DAS is: if the data can be divided into N blocks, each node randomly downloads K of them, it can verify whether all the data is available without downloading all the data, which can greatly reduce the burden on the node. But what if a data block is lost? It is difficult to find that a block is lost just by randomly downloading K blocks.

In order to implement DAS, the erasure coding technology is introduced. Erasure coding is a coding fault-tolerant technology. The basic principle is to segment the data, add certain checks and associate the data segments. Even if some data segments are lost, the complete data can still be calculated through the algorithm.

If the redundancy ratio of the erasure code is set to 50%, it means that only 50% of the block data is available, and anyone in the network can reconstruct all the block data and broadcast it. If an attacker wants to deceive the node, he must hide more than 50% of the blocks, but as long as multiple random sampling is performed, this situation is almost impossible. (For example, assuming that 30 random samplings are performed on the blocks, the probability that all of these blocks are hidden by the attacker is )

Since the node does not download all the data, but relies on erasure codes to reconstruct the data, it is necessary to ensure that the erasure codes are correctly encoded first, otherwise the data cannot be reconstructed with incorrectly encoded erasure codes.

In this way, KZG Polynomial Commitments were further introduced. Polynomial commitments are a simplified form of "representing" polynomials, which are used to prove that the value of a polynomial at a specific position is consistent with the specified value without including all the data of the polynomial. Danksharding uses KZG commitments to verify erasure codes.

It would be very convenient if we could put all the data in a KZG commitment, but the resource requirements for building this KZG commitment or rebuilding this data once some data is unavailable are huge. (In fact, the data of a single block requires multiple KZG commitments to guarantee) In order to reduce the burden on nodes and avoid centralization, Danksharding further splits the KZG commitment and proposes a two-dimensional KZG commitment framework.

After we solve the above problems one by one, relying on DAS, nodes or light clients only need to randomly download K data blocks to verify that all data is available; in this way, even after the introduction of "big blocks", the burden on nodes will not be increased too much.

(Note: In particular, the erasure code algorithm used in Danksharding is Reed-Solomon coding; KZG commitment is a polynomial commitment scheme published by Kate, Zaverucha and Goldberg. It will not be expanded here for the time being, and readers who are interested in the principles of the algorithm can expand on it by themselves. In addition, the scheme to ensure the correctness of erasure codes also includes fraud proofs used in Celestia)

Block Proposer and Builder Separation (PBS)

In the current situation, PoW miners and PoS validators are both block builders (Builder) and block proposers (Proposer) - in PoS, validators can use MEV profits to obtain more new validator seats, thereby having more opportunities to achieve MEV; in addition, large validation pools obviously have more powerful MEV than ordinary validators Capture ability, which leads to serious centralization problems. Therefore, PBS proposes to separate Builder and Proposer.

The idea of PBS is as follows: Builders build a sorted transaction list and submit bids to Proposer. Proposer only needs to accept the transaction list with the highest bid, and no one can know the specific content of the transaction list before the winner of the auction is selected.

This separation and auction mechanism introduces "involution" between the game and the Builder: after all, each Builder has different abilities to capture MEV, and the Builder needs to weigh the potential MEV profits and the relationship between the auction bids, which actually reduces the net income of MEV; and regardless of whether the block submitted by the Builder can be successfully produced, it needs to pay the Proposer for the bidding fee. In this way, the Proposer (in a broad sense, all validators, randomly re-elected within a certain period of time) is equivalent to sharing part of the MEV's income, which weakens the centralization of MEV.

The above introduces the advantages of PBS in solving MEV, and there is another reason for introducing PBS. In Danksharding, the requirements for the Builder are: to calculate the KZG proof of 32MB of data in about 1 second, which requires a 32-64 core CPU; and to broadcast 64MB of data in a P2P manner within a certain period of time, which requires a bandwidth of 2.5Gbit/s. Obviously, the Verifier cannot meet such requirements.

So PBS separates the two. The Proposer is still a general verifier node, responsible for selecting the transaction list and broadcasting the block header; while the Builder is a special role, responsible for the above work and building the transaction list.

Image source: https://ethresear.ch/t/two-slot-proposer-builder-separation/10980

Last October, Vitalik proposed a dual-slot PBS solution (Note: Each Slot is 12 seconds, which is the time unit of the beacon chain), but the specific PBS solution is still under discussion.

Anti-censorship list (crList)

Image source: https://notes.ethereum.org/@hww/workshop_feb_2022

But PBS also brings a problem. If a Builder always bids the highest price (even willing to bear financial losses) to win the auction, then he actually has the ability to censor transactions and can selectively not include certain transactions in the block.

To this end, Danksharding further introduced the Censorship Resistance List (crList), that is, the Proposer has the right to specify a transaction list that must be included by the Builder; after winning the auction, the Builder needs to prove that all transactions in the crList have been included (or the block is full), otherwise the block will be considered invalid.

Summary

Image source: https://notes.ethereum.org/@hww/workshop_feb_2022

Combining the above-mentioned data availability sampling (DAS), block builder and proposer separation (PBS) and anti-censorship list (crList), we get the complete Danksharding. We found that the concept of "sharding" has actually been diluted. Although the name of Sharding is retained, the focus is actually on supporting data availability.

So what are the advantages of Danksharding compared to the original sharding?

(Dankrad himself listed 10 advantages of Danksharding here, we will select two to explain in detail)

In the original sharding, each individual shard has its proposer and committee, which vote on the transaction verification within the shard, and the proposer of the beacon chain collects all the voting results. This work is difficult to complete in a single slot. In Danksharding, there is only a committee on the beacon chain (a generalized validator set, which is randomly re-elected within a certain period of time), and this committee verifies the beacon chain blocks and shard data. This is equivalent to simplifying the original 64 groups of proposers and committees into 1 group, which greatly reduces the complexity of both theoretical and engineering implementation.

Another advantage of Danksharding is that it is possible to achieve synchronous calls between the Ethereum main chain and zkRollup. As mentioned above, in the original sharding, the beacon chain needs to collect the voting results of all shards, which will cause confirmation delays. In Danksharding, the blocks and shard data of the beacon chain are uniformly authenticated by the beacon chain committee, which means that transactions in the same beacon block can instantly access the shard data. This inspires more imagination of composability: for example, the distributed AMM (dAMM) proposed by StarkWare can swap or share liquidity across L1/L2, thereby solving the problem of liquidity fragmentation.

After Danksharding is implemented, Ethereum will become the unified settlement layer and data availability layer of Rollup.

Closing Thoughts

In the above figure, we summarize Danksharding.

In summary, we can roughly see that in the next 2 to 3 years, the direction of the Ethereum roadmap is very obvious - it revolves around the service Rollup. Although whether the roadmap will be changed in this process is still unknown: Danksharding is expected to be implemented in the next 18-24 months, and Proto-Danksharding will be implemented in 6-9 months. But at least we have made it clear that Rollup, as the basis for Ethereum's expansion, occupies a certain dominant position.

Based on the outlook proposed by Vitalik, we also put forward some predictive thinking and conjectures here:

First, a multi-chain ecosystem similar to Cosmos, there will be a competitive landscape of multiple Rollups on Ethereum in the future, and Ethereum will provide them with security and data availability guarantees.

Second, cross-L1/Rollup infrastructure will become a rigid demand. Cross-domain MEV will bring more complex arbitrage combinations, similar to the dAMM mentioned above, which brings richer composability.

Third, the ecological application of multiple Rollups will surpass Ethereum itself. Since Ethereum's positioning is secondary, as the data availability layer of Rollup, we speculate that more applications will migrate to Rollup (if the second point is true); or at least apply applications on Ethereum and Rollup at the same time.

References:

https://consensys.net/blog/blockchain-explained/the-roadmap-to-serenity-2/

https://www.web3.university/article/ethereum-sharding-an-introduction-to-blockchain-sharding

https://ethereum-magicians.org/t/a-rollup-centric-ethereum-roadmap/4 698

https://vitalik.ca/general/2021/12/06/endgame.html

https://notes.ethereum.org/@vbuterin/proto_danksharding_faq

https://twitter.com/pseudotheos/status/1504457560396468231

https://ethos.dev/beacon-chain/

https://notes.ether eum.org/@vbuterin/pbs_censorship_resistance#How-does-proposerbuilder-separation-PBS-work

https://notes.ethereum.org/@fradamt/H1ZqdtrBF

https://cloud.tencent.com/developer/article/1829995

https://medium.com/coinmonks/builder-proposer-separation-for-ethereum-explained-884c8 f45f8dd

https://dankradfeist.de/ethereum/2021/10/13/kate-polynomial-commitments-mandarin.html

https://members.delphidigital.io/reports/the-hitchhikers-guide-to-ethereum

https://vitalik.ca/general/2019/09/22/plonk.html

Original link