Erasure Coding Based Optimization in Decentralized Distributed Storage Systems

EasyChair Preprint 15199, version 1

Abstract

Nodes failure in decentralized distributed storage systems is common. To ensure high data availability, these systems employ data redundancy mechanisms, typically relying on replicas. This paper proposes a decentralized data redundancy scheme based on erasure coding, with Reed-Solomon and IPFS as examples. The scheme reduces storage space and enhances fault tolerance. Files are sharded and encoded across multiple nodes, avoiding the redundancy of replicas. Users can retrieve any K shards from N nodes to reconstruct complete files. This erasure coding optimization combines efficient data exchange among decentralized nodes with erasure coding technology, significantly reducing storage space compared to the replica mechanism. The implementation involves truncating and sharding the blocks within the Merkle DAG generated by files, enabling flexible adjustments to the code rate of erasure codes and the allocation of storage nodes based on user needs and available resources. This method achieves a balance between storage efficiency and availability.

Keyphrases: Decentralized Distributed Storage System, Erasure Coding, IPFS, Merkle DAG, ReedSolomon

@booklet{EasyChair:15199,
  author    = {Yiwei Gan and Zhijie Huang and Yulong Shi and Xiao Zhang and Nannan Zhao},
  title     = {Erasure Coding Based Optimization in Decentralized Distributed Storage Systems},
  howpublished = {EasyChair Preprint 15199},
  year      = {EasyChair, 2024}}