English
MinIO Erasure Coding Deep Dive: From Mathematical Principles to Production Practice
1. What is erasure coding?
Erasure coding (Erasure Code, EC) is a kind of Data protection algorithms, it splits the original data into several data blocks and computes additional parity blocks through encoding. When some data blocks are lost or corrupted, the remaining data blocks and parity blocks can be used Reconstruct the complete data.
In distributed storage systems, the core problem solved by erasure coding is:How can storage space utilization be maximized while ensuring data reliability?
1.1 Why is erasure coding needed?
Reviewing the evolution of data protection in the storage field:
| Plan | Storage efficiency | Reliability | Typical scenario |
|---|---|---|---|
| No redundancy | 100% | ❌ A single point of failure means data loss | Temporary cache |
| 2 copies | 50% | Tolerate 1 device failure | Early distributed storage |
| 3 copies | 33.3% | Tolerate 2 failures | HDFS, Ceph (old version) |
| Erasure coding 8+4 | 66.7% | Tolerate 4 machine failures | MinIO default |
Conclusion: Under the same reliability, erasure coding has much higher storage efficiency than multiple replicas. MinIO adopts 8+4 Erasure coding, use 1.5 times storage overhead achieved compared to 3 replicas (3x overhead) Higher fault tolerance.
2. Mathematical Principles of Erasure Coding
The erasure coding implementation used by MinIO is based on Reed-Solomon (RS) code, This is a classic erasure coding algorithm, widely used in RAID 6, CDs, QR codes, and various distributed storage systems.
2.1 k+m model
The core of Reed-Solomon codes is k+m model:
k（Data Shards）: Number of data shards; the original data is split into k parts.
m（Parity Shards）: Validate the number of parity shards and generate m pieces of parity data through encoding computation.
Total number of shards: n = k + m
Fault tolerance: as long as any k If a shard survives, the complete data can be reconstructed. That is, it can tolerate at most m a shard is lost.
2.2 Encoding Process (Illustrated)
Assume k=4, m=2(4 data blocks + 2 parity blocks):
when B and D When lost:
Core Principles: RS code by constructing a the generator matrix of k × (k+m) mapping k data blocks to k+m encoded blocks. During decoding, extract a submatrix from the surviving blocks and solve the system of linear equations to recover the original data.
2.3 Mathematical Formulas (for technical depth reference)
Let the original data vector be D(k elements), the generated matrix is G(k × (k+m)), encoding result E = D×G.
When some shards are lost, from G extract from ... the rows corresponding to the surviving shards, forming G', surviving data E'=D×G'. Due to G' Reversible,D=E'×G'⁻¹, the data was reconstructed.
If you are interested in specific Galois Field operations, I recommend reading Backblaze's Reed-Solomon Implementation Explained in Detail.
3. Erasure Coding Implementation in MinIO
3.1 Default Policy
MinIO's erasure coding configuration and Number of drives Directly related:
| Number of drives | Default erasure coding configuration | Tolerable number of faults | Storage utilization rate |
|---|---|---|---|
| 1 | No erasure coding (single-disk mode) | 0 | 100% |
| 2 | 1+1 | 1 | 50% |
| 3 | 2+1 | 1 | 66.7% |
| 4 | 2+2 | 2 | 50% |
| 5 | 3+2 | 2 | 60% |
| 6 | 4+2 | 2 | 66.7% |
| 7 | 4+3 | 3 | 57.1% |
| 8 | 4+4(Default) | 4 | 50% |
| 8+ | Automatically select the optimal ratio | about n/2 | About 50% |
📌Attention: MinIO Official Recommend at least 8 drives to obtain a better erasure code configuration. In actual production,8+4(12 plates) or 8+8(16 drives) is a common configuration.
3.2 Storage Space Utilization Calculation
with 3 copies and Erasure coding 8+4 Taking the comparison of ... as an example:
| Plan | Physical space required to store 1 TiB of valid data | Number of simultaneous disk failures allowed | utilization rate |
|---|---|---|---|
| 3 copies | 3 TiB | 2 | 33.3% |
| Erasure coding 8+4 | 1.5 TiB | 4 | 66.7% |
With the same physical storage space, erasure coding can store 2 times valid data with 3 replicas, while fault tolerance is doubled.
3.3 Impact of Erasure Set (Erasure Set)
MinIO divides drives into several Erasure Set, each Set is independently erasure-coded.
Example: 32 drives,8+4 erasure code
Erasure Set 1: Drives 1-12
Erasure Set 2: Drives 13-24
Eraser Set 3: drives 25-32 (only 8 disks, use 4+4)Performance impact:
The larger the Erasure Set: A single read/write involves more disks,Higher throughput, but a single disk failure has a larger impact scope.
The smaller the Erasure Set: Better fault isolation, but slightly lower concurrent read/write performance.
MinIO official recommendation: Each Erasure Set contains 8 to 16 drives, achieving a balance between performance and failure domains.
4. Hands-on: MinIO Erasure Coding Deployment and Testing
4.1 Multi-disk deployment: Docker local simulation
Use Docker to simulate 12 drives, deploy 8+4 Erasure coding mode:
bash
# Create 12 empty directories to simulate disks.
mkdir -p /tmp/minio/{1..12}
# Start MinIO erasure coding mode (12 drives, automatically uses 8+4)
docker run -d \
--name minio-erasure \
-p 9000:9000 -p 9001:9001 \
-e "MINIO_ROOT_USER=minioadmin" \
-e "MINIO_ROOT_PASSWORD=minioadmin" \
-v /tmp/minio/1:/data1 \
-v /tmp/minio/2:/data2 \
-v /tmp/minio/3:/data3 \
-v /tmp/minio/4:/data4 \
-v /tmp/minio/5:/data5 \
-v /tmp/minio/6:/data6 \
-v /tmp/minio/7:/data7 \
-v /tmp/minio/8:/data8 \
-v /tmp/minio/9:/data9 \
-v /tmp/minio/10:/data10 \
-v /tmp/minio/11:/data11 \
-v /tmp/minio/12:/data12 \
minio/minio server /data{1..12} --console-address ":9001"Check the startup log to confirm the erasure code configuration:
log
Formatting 1 pool, 1 set(s), 12 drives per set.
├─ 12 drives, 8 data, 4 parity4.2 Simulating Disk Failure and Automatic Recovery
Step 1: Upload the test file
go
// Upload a 100MB test file.
func uploadTestFile(client *minio.Client) error {
ctx := context.Background()
bucketName := "test-bucket"
objectName := "test-file-100m.dat"
// Generate 100MB of random data
data := make([]byte, 100*1024*1024)
rand.Read(data)
reader := bytes.NewReader(data)
_, err := client.PutObject(ctx, bucketName, objectName, reader, int64(len(data)), minio.PutObjectOptions{})
return err
}Step 2: View the distribution of objects on disk
MinIO will distribute the object's 8 data blocks and 4 parity blocks across 12 drives.
bash
# View the distribution of objects across data directories
ls -la /tmp/minio/*/test-bucket/test-file-100m.dat/You will see something like:
/tmp/minio/1/test-bucket/test-file-100m.dat/part.1 # data block
/tmp/minio/2/test-bucket/test-file-100m.dat/part.2 # data block
...
/tmp/minio/9/test-bucket/test-file-100m.dat/part.9 # Check block
/tmp/minio/10/test-bucket/test-file-100m.dat/part.10 # Check block
...Step 3: Simulate a disk failure
bash
# Stop the container and delete 4 data directories (simulating the simultaneous failure of 4 disks).
docker stop minio-erasure
rm -rf /tmp/minio/{1,2,3,4}
docker start minio-erasureStep 4: Verify data integrity
go
// Attempt to download the file and verify whether the data is complete.
func verifyDataIntegrity(client *minio.Client) error {
ctx := context.Background()
bucketName := "test-bucket"
objectName := "test-file-100m.dat"
object, err := client.GetObject(ctx, bucketName, objectName, minio.GetObjectOptions{})
if err != nil {
return err
}
defer object.Close()
// Read and verify data (hash verification can be performed here)
data, err := io.ReadAll(object)
if err != nil {
return err
}
log.Printf("Successfully restored data, size: %d bytes\n", len(data))
return nil
}Expected results: Complete data recovery with no impact on the business. This is because MinIO automatically uses the surviving 8 shards (4 data blocks + 4 parity blocks) during reads to reconstruct the complete data.
5. Erasure Coding Trade-offs and Best Practices
5.1 The Triangular Trade-off Between Performance, Space, and Reliability
| Configuration | Storage utilization rate | Reliability | Read/write performance | Applicable scenarios |
|---|---|---|---|---|
| 4+2 | 66.7% | Tolerate 2 disk failures | relatively high | Small-scale deployment, cost-sensitive |
| 8+4 | 66.7% | Tolerate 4 disk failures | Medium | MinIO official recommendation, Balancing plan |
| 8+8 | 50% | Tolerates 8 disk failures | lower | Extremely high reliability requirements |
| 16+4 | 80% | Tolerate 4 disk failures | High throughput | Big data analytics, cold storage |
Core trade-off principle:
the larger k is: The higher the storage utilization, the more disks are involved in each read/write, latency increases slightly, and more data must be read when recovering from a single disk failure.
the larger m is: The higher the reliability, but the verification computation overhead increases, and storage costs rise.
k+m total: The total number of drives in the Set affects parallelism. The larger the total, the higher the throughput, but the larger the failure blast radius.
5.2 MinIO Officially Recommended Configuration
| Deployment scale | Total number of drives | Recommended erasure code configuration | Explanation |
|---|---|---|---|
| Development/Testing | 4 | 2+2 | Minimum erasure code configuration |
| Small-scale production | 8 | 4+4 | Balance reliability and performance |
| Medium-scale production | 12 | 8+4 | Best value for money, the official first choice. |
| Large-scale production | 16 | 8+8 | High reliability, suitable for mission-critical business |
| Hyperscale | 32+ | 8+4 or 16+4 | Multiple Erasure Sets in Parallel |
5.3 Common Misconceptions
| Misconception | Correct understanding |
|---|---|
| ❌ Erasure coding can replace backups | ✅ What erasure coding protects is disk failure, not protected Data corruption, accidental deletion, ransomware Backups are still essential. |
| ❌ The more drives, the safer erasure coding. | ✅ Fault tolerance is determined by m (number of check blocks) determined, not the total number of drives. 16+4 and 8+4 have the same fault tolerance (both 4 drives), but different utilization. |
| ❌ Erasure code reconstruction is slower than replica reconstruction | ✅ Replica reconstruction only requires copying data, while erasure code reconstruction requires decoding computation. However, on modern CPUs, the overhead of RS encoding is extremely low, and the actual difference is acceptable. |
| ❌ Erasure coding is only suitable for cold storage | ✅ MinIO’s Go implementation is highly optimized, and erasure coding also performs well in small-object scenarios (especially after inline storage optimization), making it equally suitable for hot storage. |
5.4 Monitoring and O&M Recommendations
bash
# View the cluster erasure coding health status
mc admin health myminio
# View the status of each Erasure Set
mc admin info myminio --json | jq '.info.sets'
# Monitor erasure code rebuild progress (if there is a failed disk)
mc admin heal myminio6. Summary
| dimension | Key points |
|---|---|
| Why erasure coding? | Higher storage efficiency than multiple replicas (66.7% vs 33.3%), stronger fault tolerance (tolerates 4 disk failures vs 2 disks) |
| Mathematical Principles | Reed-Solomon encoding, k+m model, any k shards are sufficient to reconstruct the complete data |
| MinIO default configuration | 8+4 (12 drives), storage utilization 66.7%, tolerates 4 simultaneous drive failures |
| Performance trade-off | The larger k is, the higher the utilization; the larger m is, the higher the reliability; choose based on the actual scenario. |
| Best practices | For production environments, it is recommended to start with 12 disks (8+4), with both monitoring and backup in place. |
MinIO's erasure coding implementation is what enables it to High performance and High Availability the fundamental reason for achieving a perfect balance between them. For storage engineers, a deep understanding of erasure coding not only enables better configuration and tuning of MinIO clusters, but also allows them to draw inferences and apply its design ideas to architectural decisions in other distributed systems.
Erasure coding is not a silver bullet; it cannot replace backups, nor can it eliminate hardware failures. But under limited hardware resources, it is Best engineering practices for maximizing data security and storage efficiency.
📌扩展阅读: