Merger is imminent: Detailed explanation of the latest technical route of Ethereum

If "The "Merge" is progressing smoothly, and sharding will become the main development axis of Ethereum in 2023 and beyond. Since sharding was proposed in 2015, its meaning has changed a lot.

Proposed "Rollup-centered Ethereum Roadmap" and Ethereum's "Endgame" in Vitalik Afterwards, the general direction of Ethereum took a de facto shift — “retiring to the background” as Rollup’s security guarantee and data availability layer.

Danksharding and Proto-Danksharding are a series of technical combinations, which are expressed in the form of "discovering problems", introducing or proposing new technologies to "solve A set of combo punches.

The timeline is extended to the next few years, and the overall value of Rollup will become larger: Ethereum presents The multi-Rollup development pattern, the highly perfect cross-Rollup infrastructure, and the highly prosperous Rollup ecology—even beyond Ethereum itself.

Introduction

Image source: https://consensys.net/blog/blockchain-explained/the-roadmap-to-serenity-2/

In the blink of an eye, 2022 is halfway through. Looking back at the Serenity Roadmap proposed by Vitalik in his Devcon speech in 2018, it is easy to find that the development path of Ethereum has changed several times. Compared with the current roadmap, sharding has been given a new meaning, and eWASM is rarely mentioned.

In order to avoid potential fraud and user misleading, at the end of January this year, the Ethereum Foundation announced that it would abandon the term "ETH2", and It is to rename the current Ethereum mainnet as the "execution layer" for processing transactions and executions, and to rename the original ETH2 as the "consensus layer" for coordinating and processing PoS.

Currently, the official route of Ethereum The diagram covers three parts: the beacon chain, merging and sharding.

Among them, the Beacon Chain (Beacon Chain), as the pre-work for the migration of Ethereum to PoS, and the coordination network of the consensus layer, It was launched on December 1, 2020 and has been in operation for nearly 20 months so far.

The Merge refers to the final merger of the current Ethereum mainnet and the beacon chain, that is, the unification of the execution layer and the consensus layer. With Ethereum officially migrating to PoS. In the IOSG article "Dawn is approaching: The merger of Ethereum is close at hand", we introduced important progress related to the merger: the current Ropsten and Sepolia testnets of Ethereum have successfully merged, followed by the merger of Goerli; if all goes well , which means that we are not far from the mainnet merger.

Image source: https:// medium.com/decipher-media/blockchain-scaling-solutions-4-1-ethereum-sharding-e88e8cacdfc

In this article we will focus on Sharding. The reasons are:

First, assuming that the merger of the main network can be successfully realized within the year, then the sharding will follow, as 2023 The main axis of development for Ethereum.

Secondly, the concept of Ethereum sharding was first proposed by Vitalik in Devcon 1 in 2015, and then proposed in GitHub's Sharding FAQ The six development stages of sharding are shown (as shown in the figure above). However, with the update of the Ethereum roadmap and the promotion of related EIPs, the meaning and priority of sharding have changed a lot. When we discuss sharding, we need to first make sure we agree on what it means.

To sum up the above two points, it is very important to sort out the ins and outs of fragmentation. This article will focus on discussing the origin, progress and future route of Ethereum original sharding, Danksharding and Proto-Danksharding, rather than going into every technical detail. For details on Danksharding and Proto-Danksharding, please refer to IOSG's previous articles: "Is Danksharding the future of Ethereum sharding", "EIP4844: The predictable depression effect of L2 transaction fee reduction is about to start".

Quick Review

This article will mention several times to Rollup, data availability and sharding.

Here we quickly go over the basic concepts of the three.

The current mainstream Rollup is divided into zkRollup and Optimistic Rollup. The former is based on validity proof, that is, batch execution of transactions, relying on cryptographic proof SNARK to ensure the correctness of state transitions; the latter "optimistically" assumes that all state transitions are correct unless falsified; that is, it takes a period of time to ensure Wrong state transitions can be detected.

Data availability for zkRollup and Optimistic Rollup are very important. For the former, users can reconstruct all transactions on the second layer based on data availability to ensure anti-censorship; for the latter, all data on the second layer needs to be published without any transaction being hidden. As for the bottlenecks and corresponding solutions currently facing data availability, they will be mentioned below.

Image source: https://www.web3.university /article/ethereum-sharding-an-introduction-to-blockchain-sharding

Ethereum full node stores the complete state of EVM, And participate in all transaction verification, which ensures decentralization and security, but then comes the problem of scalability: transactions are executed linearly, and each node needs to be confirmed one by one, which is undoubtedly inefficient.

In addition, as time goes by, the Ethereum network data continues to accumulate (currently up to 786GB), and the hardware requirements for running a full node will increase accordingly . A decrease in the number of full nodes will create a potential single point of failure and reduce the degree of decentralization.

Intuitively, sharding is equivalent to division of labor and cooperation, that is, all nodes are grouped, and each transaction only needs to be performed by a single group of nodes Verification, and regularly submit transaction records to the main chain to achieve parallel processing of transactions (for example, if there are 1000 nodes, each transaction must be verified by each node; if they are divided into 10 groups, each group has 100 nodes to verify transactions, the efficiency is obviously greatly improved). The use of fragmentation makes it possible to improve scalability while reducing the hardware requirements of a single group of nodes, thereby solving the above two problems.

Original Fragment

Image source: https://vitalik.ca/general/2021/04/07/sharding.html

There are 64 shards in the original Ethereum plan, each shard has an independent proposer and committee, the proposer is a randomly selected verifier, and collects transactions And sort; the committee is a set of verifiers (composed of at least 128 verifiers), which are randomly assigned to each shard at regular intervals, and verify the validity of the transaction. If 2/3 of the committee votes, Then call the verifier management contract (VMC) to submit transaction records to the beacon chain. Different from the "data sharding" described below, this kind of sharding is also called "execution sharding".

Background

Before talking about Danksharding, let's take a moment to understand its background. Personally, the basis of the community atmosphere launched by Danksharding mainly comes from two articles by Vitalik. These two articles set the tone for the future direction of Ethereum.

First, Vitalik In October 2020, the "Ethereum Roadmap Centered on Rollup" was published, proposing that Ethereum needs to provide centralized support for Rollup in the short to medium term. First, the expansion of the Ethereum base layer will focus on expanding the data capacity of blocks, rather than improving the efficiency of on-chain calculations or IO operations. That is: Ethereum sharding is designed to provide more space for data blobs (rather than transactions), Ethereum does not need to interpret these data, only to ensure that the data is available. Second, Ethereum's infrastructure is adjusted to support Rollup (such as L2 support for ENS, L2 integration for wallets, and cross-L2 asset transfers). In the long run, the future of Ethereum should be as a highly secure, single execution shard everyone can process, and a scalable data availability layer.

Image source: https://vitalik.ca/general/2021/12/06/endgame.html

Thereafter, Vitalik described the final picture of Ethereum in "Endgame" published in December 2021: block output is Centralized, but block validation is trustless and highly decentralized while ensuring censorship resistance. The underlying chain provides guarantees for the data availability of blocks, while Rollups provide guarantees for the validity of blocks (in zkRollup, this is achieved through SNARKs; in Optimistic Rollup, only one honest participant needs to run a fraud proof node). Similar to the multi-chain ecology of Cosmos, the future of Ethereum will be multi-Rollup coexistence-they are all based on the data availability and shared security provided by Ethereum. Users rely on the bridge to move between different Rollups without paying the high fees of the main chain.

The above two articles have basically determined the development direction of Ethereum: to optimize the construction of the basic layer of Ethereum, for Rollup Serve. The above argument may be based on such a view: since Rollup has been proven effective and well adopted, then "instead of spending a few years waiting for an uncertain and complicated expansion plan (note: refers to the original sharding), it is better to pay attention to Focus on the Rollup-based solution."

After that, Dankrad proposed a new sharding scheme Danksharding. Let's break down the specific technical components of Danksharding to understand.

Proto-Danksharding

< img _width="512px" src="https://image.blockbeats.cn/upload/2022-07-19/4cbacc44eb39c838dcc42049cc85e041262afe8c.png?x-oss-process=image/quality,q_50/format,webp" alt=" Picture">

Image source: https://l2fees.info/

The background of Proto-Danksharding is that although the Rollup scheme significantly reduces transaction costs compared to the Ethereum main chain, it is not yet low enough to be ideal. This is because CALLDATA, which provides data availability on the Ethereum main chain, still occupies a relatively large cost (16 gas / byte). In the original vision, Ethereum proposed to provide 16MB of dedicated data space for each block in data sharding for Rollup, but the actual implementation of data sharding is still far away.

On February 25 this year, Vitalik and DankRad proposed the EIP-4844 (Shard Blob Transactions) proposal, which also That is, Proto-Danksharding, which aims to expand the data availability of Ethereum in a simple and forward-compatible way, making it still available after the launch of Danksharding. The changes in this proposal only occur on the consensus layer, and do not require additional adaptation work by clients, users, and Rollup developers of the execution layer.

Proto-Danksharding does not actually perform sharding, but introduces a method called It is the transaction format of "Blob-carrying Transactions". This transaction format is different from ordinary transactions in that it carries an additional data block called blob (about 125kB), which makes the block actually larger, thus providing cheaper data availability than CALLDATA (about 10kB).

However, the general problem with "big blocks" is that the requirements for disk space continue to accumulate, using Proto-Danksharding It will increase the storage capacity of Ethereum by an additional 2.5TB per year (the current network-wide data is only 986GB). Therefore, Proto-Danksharding sets a period of time window (for example, 30 days), after which the blob is deleted, and the user or protocol can back up the blob data within this period.

That is, the consensus layer of Ethereum is only used as a highly secure "real-time bulletin board" to ensure that these data are in It is available long enough to allow other users or protocols enough time to back up the data, rather than Ethereum keeping all blob history data forever.

The reason for this is that for storage, the addition of 2.5TB per year is not a problem, but for Ethereum nodes bring a lot of burden. As for the trust assumption that may be caused, in fact, only one data storage party is honest (1 of N), the system can operate normally, and there is no need for a validator node set that participates in verification and executes consensus in real time (N/2 of N) to store this part of historical data.

So, is there any incentive to push third parties to store these data? The author has not found the launch of the incentive plan for the time being, but Vitalik himself proposed several possible data storage methods:

1 , Application-specific protocols (such as Rollup). They can require nodes to store historical data related to the application. If the historical data is lost, it will cause risks to this part of the application, so they have the motivation to do storage;

2. BitTorrent;< /p>

3. Ethereum’s Portal Network, which is a platform that provides lightweight access to protocols;

4. Blockchain Browsers, API providers or other data service providers;

5. Individual enthusiasts or scholars engaged in data analysis;

6. Third-party indexing protocols such as The Graph.

Danksharding Data Availability Sampling (DAS)

Image source: https://notes .ethereum.org/@hww/workshop_feb_2022

In Proto-Danksharding we mentioned the new The transaction format of the block actually becomes larger, and Rollup also accumulates a large amount of data, which nodes need to download to ensure data availability.

The idea of DAS is: if the data can be divided into N blocks, each node randomly downloads the K blocks can verify whether all data is available without downloading all data, which can greatly reduce the burden on nodes. But what if a block of data is lost? It is difficult to find that a block is missing just by randomly downloading K blocks.

In order to realize DAS, erasure coding (Erasure Coding) technology is introduced. Erasure code is a coding error-tolerant technology. The basic principle is to segment data, add certain checksums and make associations between each data segment. Even if some data segments are lost, the complete data can still be calculated through the algorithm. .

If the redundancy ratio of the erasure code is set to 50%, it means that only 50% of the Once the block data is available, anyone in the network can reconstruct all block data and broadcast it. An attacker would have to hide more than 50% of the blocks if they wanted to trick nodes, but this rarely happens as long as multiple random samples are taken. (For example, assuming 30 random samples of blocks, the probability that these blocks are all hidden by the attacker is )

Since nodes do not download all data, but rely on erasure codes to reconstruct data, Then you first need to ensure that the erasure code is correctly encoded, otherwise the data cannot be reconstructed with an incorrectly encoded erasure code.

In this way, KZG Polynomial Commitments (KZG Polynomial Commitments) are further introduced, which is a "representative" polynomial A simplified form used to prove that the value of a polynomial at a particular location agrees with a specified value without including all of the data for the polynomial. In Danksharding, the verification of the erasure code is realized by using the KZG commitment.

It would be easy if we could put all the data in one KZG commitment, but building this KZG commitment, Or once part of the data is not available, rebuilding this data - both resource requirements are huge. (Actually, the data of a single block needs multiple KZG commitments to guarantee) Also in order to reduce the burden on nodes and avoid centralization, Danksharding further splits KZG commitments and proposes a two-dimensional KZG commitment framework.

When we solve the above problems in turn, relying on DAS, nodes or light clients only need to randomly download K pieces of data block, it is possible to verify that all data is available; in this way, even after the introduction of "big blocks", the burden on nodes will not be increased too much.

(Note: In particular, the erasure coding algorithm used in Danksharding is Reed-Solomon coding; the KZG commitment is The polynomial commitment scheme published by Kate, Zaverucha and Goldberg. I will not expand it here, and readers who are interested in the algorithm principle can expand it by themselves. In addition, the scheme to ensure the correctness of the erasure code is the fraud proof adopted in Celestia)

Separation of block proposers and builders (PBS)

In the current situation, PoW miners and PoS verifiers are both block builders (Builder) and block proposers (Proposer)—— In PoS, verifiers can use MEV profits to obtain more new verifier seats, thus having more opportunities to realize MEV; in addition, large verification pools obviously have stronger MEV capture capabilities than ordinary verifiers, which leads to Serious centralization problem. Therefore, PBS proposed to separate Builder and Proposer.

The idea of PBS is as follows: Builders build a sorted list of transactions and submit bids to Proposers. The proposer only needs to accept the transaction list with the highest bid, and no one can know the specific content of the transaction list until the winner of the auction is selected.

This separation and auction mechanism introduces an "involution" between the game and the Builder: After all, each Builders have different abilities to capture MEV. Builders need to weigh the relationship between potential MEV profits and auction bids, which actually reduces the net income of MEV; regardless of whether the final block submitted by Builder can be successfully produced, it will The bid fee needs to be paid to the Proposer. In this way, Proposer (in a broad sense, it is the set of all verifiers, randomly reselected within a certain period of time) is equivalent to sharing a part of the income of MEV, which weakens the degree of centralization of MEV.

The above describes the advantages of PBS in solving MEV, but there is another reason for introducing PBS. In Danksharding, the requirements for Builder are: to calculate the KZG proof of 32MB data in about 1 second, which requires a CPU with 32-64 cores; and to broadcast 64MB data in a P2P manner within a certain period of time, which requires 2.5Gbit/ s bandwidth. Obviously the verifier cannot meet such requirements.

So PBS separated the two, and Proposer is still a general validator node, responsible for selecting the transaction list and broadcasting Block header; Builder, as a special role, is responsible for the above work and building the transaction list.

Image source : https://ethresear.ch/t/two-slot-proposer-builder-separation/10980

In October last year, Vitalik proposed a dual-slot PBS scheme (note: each slot is 12 seconds, which is the time unit of the beacon chain), but the specific PBS scheme is still under discussion.

Censorship Resistant List (crList)

Image source: https://notes.ethereum .org/@hww/workshop_feb_2022

But PBS also poses a problem, if a Builder always bids the highest price (even willing to bear economic losses) to win the auction, then he actually has the ability to review transactions and can selectively not include certain transactions in the block.

For this reason, Danksharding further introduces the anti-censorship list crList (Censorship Resistance List), that is, the Proposer has the right to specify A transaction list, which must be included by the Builder; after winning the auction, the Builder needs to prove that all transactions in the crList have been included (or the block is full), otherwise the block will be considered invalid.

Summary

Image source: https://notes.ethereum.org/@hww/workshop_feb_2022

Combines the aforementioned data availability sampling (DAS), separation of block builders and proposers (PBS), and censorship resistant list (crList) , you get a complete Danksharding. We found that the concept of "sharding" has actually been downplayed. Although the name Sharding is retained, the actual focus has been on supporting data availability.

So what are the advantages of Danksharding compared to the original sharding?

(Dankrad himself listed 10 advantages of Danksharding here, we choose two to explain in detail)

p>

In the original shard, each individual shard has its proposer and committee, which Transaction verification votes, and all voting results are collected by the proposer of the beacon chain, which is difficult to complete in a single Slot. In Danksharding, there is only a committee on the beacon chain (a generalized validator set, randomly re-selected within a certain period of time), and this committee verifies the beacon chain blocks and shard data. This is equivalent to simplifying the original 64 groups of proposers and committees to 1 group, which greatly reduces the complexity of both theory and engineering implementation.

Another advantage of Danksharding is the possibility of synchronous calls between the Ethereum main chain and zkRollup. As we mentioned above, in the original shard, the beacon chain needs to collect the voting results of all shards, which will cause a delay in confirmation. In Danksharding, the blocks and shard data of the beacon chain are uniformly certified by the committee of the beacon chain, that is, transactions in the same beacon block can instantly access the shard data. This stimulates the imagination of more composability: for example, the distributed AMM (dAMM) proposed by StarkWare can perform Swap or share liquidity across L1/L2, so as to solve the problem of liquidity fragmentation.

After Danksharding is implemented, Ethereum will become the unified settlement layer and data availability layer for Rollup.

Closing Thoughts

In the image above , we make a summary of Danksharding.

To sum up, we can roughly see the directionality of the Ethereum roadmap in the next 2 to 3 years It is very obvious - it revolves around the service Rollup. Although it is still unknown whether the roadmap will change or not in the process: Danksharding is expected to be implemented in the next 18-24 months, and Proto-Danksharding will be implemented in 6-9 months. But at least we have made it clear that Rollup, as the expansion basis of Ethereum, occupies a certain dominant position.

According to Vitalik’s outlook, here we also propose some predictive thinking and conjectures:

One is a multi-chain ecology similar to Cosmos. In the future, there will be a multi-Rollup competition pattern on Ethereum, and Ethereum will provide them with security Guarantee of security and data availability.

Second, cross-L1/Rollup infrastructure will become a rigid need. Cross-domain MEV will bring more complex arbitrage combinations, similar to the dAMM mentioned above, which will bring richer composability.

Third, the multi-Rollup ecological application will surpass Ethereum itself. Since the positioning of Ethereum takes a backseat, as the data availability layer of Rollup, we guess that more applications will be migrated to Rollup (if the second point is true); or at least apply on Ethereum and Rollup at the same time.

Reference material:

https://consensys.net/blog/blockchain-explained/the-roadmap- to-serenity-2/

https://www.web3.university/article/ethereum-sharding-an-introduction-to-blockchain-sharding

https://ethereum -magicians.org/t/a-rollup-centric-ethereum-roadmap/4698

https://vitalik.ca/general/2021/12/06/endgame.html

https://notes.ethereum.org/@vbuterin/proto_danksharding_faq

https://twitter.com/pseudotheos/status/1504457560396468231

https://ethos.dev/ beacon-chain/

https://notes.ethereum.org/@vbuterin/pbs_censorship_resistance#How-does-proposerbuilder-separation-PBS-work

https://notes. ethereum.org/@fradamt/H1ZqdtrBF

https://cloud.tencent.com/developer/article/1829995

https://medium.com/coinmonks/builder-proposer -separation-for-ethereum-explained-884c8f45f8dd

https://dankradfeist.de/ethereum/2021/10/13/kate-polynomial-commitments-mandarin.html

https ://members.delphidigital.io/reports/the-hitchhikers-guide-to-ethereum

https://vitalik.ca/general/2019/09/22/plonk.html

Original link