Erasure Coding Based Optimization in Decentralized Distributed Storage Systems

EasyChair Preprint 15199, version 2

Abstract

Node failures in decentralized distributed storage systems are common. To ensure data availability, these systems employ data redundancy mechanisms, typically relying on replicas. This paper proposes a decentralized data redundancy scheme based on erasure coding, with Reed-Solomon and IPFS as examples. Compared with replication, the proposed scheme reduces storage space and enhances fault tolerance. Files are sharded and encoded across multiple nodes, avoiding the high redundancy of replicas. Users can retrieve any K shards from N nodes to reconstruct the original files. This erasure coding optimization combines efficient data exchange among decentralized nodes with erasure coding technology, significantly reducing storage space compared with the replica mechanism. The implementation involves truncating and sharding the blocks within the Merkle DAG generated by files, enabling flexible adjustments to the code rate of erasure codes and the allocation of storage nodes based on user needs and available resources. This method achieves a balance between storage efficiency and data availability.

Keyphrases: Decentralized Distributed Storage System, Erasure Coding, IPFS, Merkle DAG, ReedSolomon

@booklet{EasyChair:15199,
  author    = {Yiwei Gan and Zhijie Huang and Yulong Shi and Xiao Zhang and Nannan Zhao},
  title     = {Erasure Coding Based Optimization in Decentralized Distributed Storage Systems},
  howpublished = {EasyChair Preprint 15199},
  year      = {EasyChair, 2024}}