2022 End of summary: The status and future of decentralized storage

2022 End of the Year: The State and Future of decentralized Storage

Original article by Jason, Puzzle Ventures

TL; DR

Decentralizing non-core data from the main chain to the Decentralized Storage Network (DSN) has become the mainstream solution to capacity expansion, interoperability and privacy protection.

Filecoin has a significant advantage in miner networks and storage costs, but Arweave's permanent storage provides a more reliable solution for NFT or social applications.

L1 extended storage networks can offer new solutions with greater interoperability than DSNS, but it still takes time to validate the schema.

Future decentralized storage scenarios include complete interoperability, data availability, security, and user/developer-friendly off-chain computing through middleware and apis.

The evolution of decentralized storage

Data storage has always been the cornerstone of the development of science and technology industry and the focus of the field. Since Amazon, Microsoft and Alibaba have provided centralized cloud storage services, the cloud storage market has basically formed a market scale of more than 100 billion US dollars (2021) and the head effect is obvious (Amazon 40%, Microsoft 22%, Ali Cloud 10%, data source Gartner).

With the explosion of Web3, a number of decentralized storage providers began to meet the needs that the traditional cloud storage market could not meet. While the majority of Dapps are still deployed on AWS, and more than 50% of Ethereum nodes ran on AWS prior to the merge [1], the Web3 project has gradually discovered the importance and inevitability of connecting the back end and metadata to decentralized storage.

According to the definition of Ethereum.org, decentralized storage is a data sharing system composed of p2p networks, where each operator owns a part of the overall data and can recover it through algorithms [2]. Although decentralized storage can solve many problems, such as data privacy protection, anti-censorship, etc., the development of the current decentralized system is not mature and there are many problems to be solved. After continuous attempts of on-chain Storage solutions, the Decentralized Storage Network (DSN) is the mainstream solution at present. This solution can extend storage capacity to infinity to the maximum extent and guarantee data security and privacy.

Table 1 Comparison of blockchain data storage schemes

Concrete expansion in the chain of plan can refer to the article: https://zhuanlan.zhihu.com/p/48078642

Source :  Puzzle Ventures

The author believes that the storage layer under the chain will eventually become an important infrastructure in the Web3 ecosystem, and solve the data storage problems of any protocol layer and operation layer through hot storage and cold storage.

Source: Puzzle Ventures

In the future, the capacity expansion problem in the "Impossible Triangle" can be solved by completely stripping the storage layer but maintaining interoperability. Meanwhile, middleware, API and other services built on the cold storage layer can improve the efficiency of data calculation, call, and push.

State of the art for decentralized storage: Classification and scale

Decentralized storage essentially serves the application layer of the Web3 ecosystem, so the solution is more likely to meet the needs of end users, that is, to perform data storage, computation, and invocation in a more efficient and cost-effective manner. In terms of classification, Arweave, Filecoin, and Storj have formed three separate header decentralized storage networks, with Filecoin and Storj being more decentralized P2P storage networks and Arweave being a more tightly organized network of storage nodes. On the other hand, different types of storage aggregator networks are becoming more attractive gateways for user cost optimization, storage convenience, etc., but are still built on top of Arweave or Filecoin. In addition, the L1 extended network represented by EthStorage focuses on the design that can directly call the DA layer of the main chain and provide additional computing power, which is deeply linked with L1.

(Note: The projects listed in the figure are mainly existing projects and new projects, excluding past projects)

Source :  Puzzle Ventures

In addition, there are also independent computing networks and coordinated middleware built on the underlying storage network, which are also complementary parts of the decentralized storage ecosystem. However, ecological dApp is the main part and will not be discussed in this paper.

In terms of scale, Filecoin is far and away the leader in terms of revenue, FDV, and market share, and the amount of storage used by users continues to increase. However, more than $3 million in quarterly revenue isn't much in either Web2 or Web3, and that's in a near-monopoly situation. So the whole decentralized storage circuit still has a lot of room to grow.

Table 2 Decentralized Storage Q3 2022 Revenue and size

Source :  Messari, Web3 Index, CoinGecko

Arweave and Storj, on the other hand, are still making $100,000 a quarter, or more than $1,000 a day, not nearly as big a market as they should be. The reason is partly related to the overall market malaise in the bear market, and partly because the current DSN storage solution has not yet reached the inflection point that perfectly fits Web3. In the future, on the one hand, the whole DSN market needs more middleware and aggregators to provide more user traffic, on the other hand, it also needs to constantly adjust and optimize its own cost and value.

Mainstream solution: Filecoin vs. Arweave

1 Functionality

At this stage, Filecoin and Arweave use different technology architectures that also create functional differences. Based on the research report of Fundamental Labs[3], we define the functional evaluation framework as: flexibility of storage scope, permanence of storage, ability to avoid redundancy, incentive of data storage, universality of stored data, and availability of data. In some ways, Arweave and Filecoin are complementary solutions: Arweave is more focused on permanence and stability of data storage, and is better suited for things like metadata and historical data; Filecoin, on the other hand, provides a more flexible storage solution (storage duration, type) that is more suitable for personal and non-critical data storage.

Source: Fundamental Labs

According to the latest development process, Filecoin version 17 "Shark" mainly establishes the programmability of storage protocol, that is, adds the interaction layer of smart contract on the storage layer, and plans to launch FEVM system in February next year. Data availability and activity on Filecoin will increase and allow for more diverse interoperability, which gives Filecoin a flexibility advantage over Arweave. Arweave, on the other hand, continues to build on data security, node incentive stability (endowment) and storage entry, and continues to "top up" its belief in the idea of permanent storage. Therefore, although it is difficult to say which solution is more advantageous, however, these two decentralized storage protocols have developed relatively complete functions to meet the basic needs of decentralized storage.

2 Cost

Similarly, it is difficult to compare Filecoin and Arweave directly in terms of cost, so this article aims to get a sense of the two costs in as reasonable a way as possible. For data comparability, this article uses prices for 200 years of storage for the same storage size (Arweave is generally believed to support 200 years of storage) as a comparison object.

Table 3  Cloud Storage cost Comparison (Data captured in December 2022)

Sources: Amazon.com, Arweavefees.com, File.app, Filfox.info

As you can see from Table 3, the most basic AWS S3 Standard plan costs about $0.023 per GB per month and $55.2 per GB for 200 years of storage. The Glacier Deep Archive option, which is better for long-term cold storage, can drive the price down to $2.38 per GB, but there is a limit to the number of archiving times per year. Arweave, on the other hand, comes in at a different price depending on the price of AR tokens. In December 2022, Arweave will cost $1.42 per GB, which means that after five years of storage, it will cost the same as S3 standard storage. On the other hand, the price of Filecoin is complicated to calculate because different miners pay different prices and the price is affected by the price of FIL token, storage time, and frequency of data calls. However, Filecoin's storage costs are several orders of magnitude cheaper than Arweave and S3, and can be considered almost cost-free storage.

But is price the main factor? Arweave's permanent storage is even more significant for Web3. For example, if the metadata and picture data of NFT are lost, they will instantly lose their value and become a closed connection. In Filecoin storage protocol, the risk of Garbage Collection will be caused due to the decentralized management mode. That is, there is a risk that some documents will be deleted by mistake if they are not star-marked on the node where they are stored. There are also risks associated with the need to update the Filecoin store when it expires. Arweave, on the other hand, uses AR incentives to encourage nodes to store data for 200 years, and compensates for future AR depreciation, while periodically checking the storage process to ensure that nodes are still fully stored. These measures are to ensure that every node can obtain incentives through the complete storage of data, but also to ensure the security and reliability of the stored data. In the already hack-prone world of Web3, the value of data storage reliability is not simply measured in terms of the price of storage.

3 Ecology

The ecology of decentralized storage includes miners/nodes and eco-Dapps.

Table 4  Comparison of ecological data between Filecoin and Arweave

Source: Filscan.io, Messari

Filecoin currently has around 18.9 exabytes of storage, 170,000 times that of Arweave, thanks to early community building and high awareness. But Arweave has grown 2.5 times faster than Filecoin in just over three years, and Filecoin's growth rate has actually stabilized in a straight line. In terms of market cap, Arweave's FDV is now about 40% of Filecoin's, so we can see that Arweave will catch up to Filecoin's storage ecosystem very quickly in the future.

According to Filecoin's future plan released in November [4], Filecoin will focus on computation and broader data onboarding requirements to truly become the Layer 0 of the Web3 world. From a storage expansion perspective, Filecoin is encouraging more miners to participate in node construction through incentive programs such as FIL+ and ESPA Bootcamps. In terms of data onboarding and computation, Filecoin focuses on specific market segments, such as NFT metadata and Web2 data sources, and builds computation on the chain through the Filecoin Virtual Machine. Make it easier to use the data. The latest Filecoin proposals (FIP-0044, FIP-0045) also show that data onboarding is an ongoing focus.

In contrast, Arweave, the latest version 2.6 [5], focuses on cost reduction and efficiency, which means reducing the overall storage cost and storage price by iterating hardware, etc., while improving the efficiency of data storage and retrieval. So Arweave is trying to attract more developers and users by building a better base environment, while meeting further use cases such as gateways and computation through eco-projects. So Filecoin's logic is to provide a unified, storage-focused, decentralized storage network that relies on the ecosystem of miners; Arweave, on the other hand, is more like a standard Web3 ecosystem, enabling organic growth in the number of users and Dapps by providing a sound underlying architecture and incentives to create a developer-friendly and user-friendly environment.

In terms of ecological projects, Arweave has already seen a number of excellent projects including Bundlr, Kyve, ArDrive, and KwilDB that focus on technology optimization, storage entry, and application expansion, and most of the project leaders are young entrepreneurs with high potential for growth. The Filecoin ecosystem has the advantage of a large number of projects. The Filecoin ecosystem is more of an open source environment that can meet a broader storage-related needs but doesn't have a focused direction.

New scheme: aggregation network and L1 extension network

1 Aggregation Network

The concept of aggregating networks has actually been around since Filecoin was launched, with early projects such as Coldstack. Coldstack hasn't been a huge success, however, mainly because pure, Uber-style aggregators don't really improve storage efficiency and don't offer any further value beyond that. Aggregating networks of storage protocols can solve the pain points of storage efficiency and cost for users, but there are many ways to do this. First, these aggregators will have to do better than Filecoin and Arweave's own gateway to gain market acceptance.

As Filecoin moves towards FVM chain computing, it is likely to crowd out aggregator networks that are building on top of Filecoin to provide extra computation capacity. Second, different aggregation protocols address the problem that Filecoin or Arweave are difficult to implement on their own, rather than creating a single aggregate traffic entry point (at least not in the early entry points), which will ultimately be controlled by Filecoin and Arweave themselves.

Jackal is an aggregator solution that builds on Filecoin and focuses on improving data privacy and security. It provides a set of schemes for further AES-256 encryption of existing files, managed and charged through the Cosmos PoS consensus L1 chain, creating a private cloud storage protocol on top of Filecoin. Jackal will also use its Proof-of-Persistence algorithm to provide 3x file copies and optimize for storage price. Redundancy and cost are not the core issues of decentralized storage, but data privacy and security improvements are difficult for Filecoin itself to achieve in a short period of time. 4EVERLAND is another cloud computing platform that combines Filecoin, Arweave, and Dfinity as the underlying storage layer, focusing on providing Dapps with the core capabilities of decentralized storage, computing, and networking to create a connectivity layer between the underlying decentralized storage layer and the application layer.

At present, Web hosting and Bucket are its two main products. hosting can complete front-end hosting with the acceleration of more than 200 gateways of 4EVERLAND worldwide, while Bucket is an aggregation gateway. Both products greatly improve the practical use of decentralized storage in terms of user friendliness, and solve optimization problems at the level of details such as multi-currency payments, data panels, and so on.

2 L1 Storage extension network

L1 storage extension Network is essentially the same as the existing Layer 2, which is the general term for the solution to improve L1 performance and scalability. EthStorage is the first Layer2 solution with storage extension as the core, which mainly includes three categories of functions: Proof of Publication, External Data Retention and Access Protocol. Proof of Publication is CRUD for storing data through its special KZG Commitment and Reed-Solomon Code, whereas Filecoin and Arweave are currently CRD only. After adding the Update function, combined with the Access Protocol Web3 URL function that can directly render resources hosted on the Ethereum contract, you can easily implement on-chain activities such as Web3 Email, Web3 Blog, Web3 Drive, etc. All of these attempts will help in the future of Web3 mass adoption.

On the other hand, due to EVM compatibility and on-chain storage, the interaction path is shorter than Filecoin and Arweave, so it is more composible. For example, EthStorage can store the metadata of NFT in EVM environment on the chain, and can complete the NFT combination and other new ways that need to be programmable through smart contracts to improve user experience. The programmability of on-chain data (on a large scale) will spawn new Defi, Gamefi, Socialfi applications in the future. Currently, L1 storage extension networks and DSN networks are complementary and competitive, and it will take time to prove that they can survive in the EVM ecosystem and evolve their storage paradigm into an official standard.

The possibility of future development

Taken together, Filecoin and Arweave have become the poster children for decentralized storage and represent the 0-to-1 process of decentralized storage. In terms of development concept, the separation of non-on-chain core data into the storage tier has become the main means of expansion of Ethereum. At the same time, storage aggregation network and L1 extended network, as a supplement and extension, can meet the specific needs of users more efficiently and at low cost, while the basic DSN network still has many problems to be solved: A lack of ecological applications, separation of storage and computing, and difficult Web2 onboarding all hinder DSNS from achieving the next wave of growth. In the future, this paper identifies several types of projects that are likely to be the next engines of growth:

DSN Aggregator: Aggregator portals built on different/same DSN networks, and provide cost optimization, unified Token payment, data availability/invocation services and other user-friendly tools. Similar projects: Lighthouse, 4EVERLAND 

Off-chain computing: While Filecoin already has a decentralized computing layout on the chain, pure on-chain computing does not meet more complex computing requirements. A comprehensive solution that combines on-chain data and off-chain computation can increase the value of data usage and expand use cases by performing more complex computations. Similar items: KwilDB, Tableland

L1 storage extension network: The advantage of L1 storage and computing extension is that it can be deeply connected directly to the L1 ecosystem, even for projects with high data storage requirements such as social and gaming, and can be quickly integrated on a large scale after official approval. Similar projects: EthStorage 

Third-party data security network: security protection of data stored in existing DSNS, review, backup and block hacker attacks on storage nodes, and relevant compensation insurance schemes are provided. Similar projects: Jackal   

Original link

Welcome to join the official BlockBeats community:

Telegram Subscription Group: https://t.me/theblockbeats

Telegram Discussion Group: https://t.me/BlockBeats_App

Official Twitter Account: https://twitter.com/BlockBeatsAsia