Deeply Understanding Distributed Storage: Evolution from Inode to Metadata Service

If we compare the file system to a vast library, then an inode is the "call number" of each book. How does this call number work in a distributed environment? This article takes you to find out.

• Preface
• I. Core Concepts of Distributed Storage
  • 1. Metadata vs. Data
  • 2. Data block
  • 3. Core Components
  • 4. Distributed
• II. Inodes in the Local File System
  • What is an inode?
  • What is inside an inode?
  • The Clever Design of Data Block Pointers
  • Key Insights
  • Workflow Example
• III. Evolution of Inodes in Distributed Storage
  • Local vs. Distributed: Comparison at a Glance
  • Taking HDFS as an example: classic design
  • HDFS read and write process
  • Ceph's revolution: decentralization
• IV. Conclusion: The Evolution Path of inode
  • Key points review

Preface

In distributed storage systems, metadata management is the core of architectural design. And inode As a cornerstone concept of file systems, its meaning has undergone profound changes from local file systems to distributed storage. Understanding this evolution allows one to grasp the design essence of many distributed storage systems.

I. Core Concepts of Distributed Storage

in the discussion that unfolds inode Before that, let us first go over a few basic concepts in distributed storage.

1. Metadata vs. Data

type	Definition	Example
metadata	Data that describes data	File name, creation time, size, storage location, permissions
data	The actual content of the file	The pixels of photos, the text of documents, the frames of videos

Simply put, metadata is the "household registration booklet" or "index card" of a file.

2. Data block

File contents are usually very large, making it inconvenient to store and transmit as a whole. The system will split the file into fixed-size Data block(e.g., 64MB or 128MB). This is the basic unit of operation for distributed storage.

3. Core Components

• Data node: The servers that actually store data blocks usually form a cluster composed of multiple machines.

• Metadata node: The server that manages metadata, maintains the "directory tree" of the entire file system and the inodes of all files, and is the "brain" of the entire cluster

4. Distributed

Metadata nodes and data nodes run on multiple commodity servers and collaborate over the network. Data blocks are also replicated across multiple nodes to prevent data loss.

II. Inodes in the Local File System

Before understanding distributed versions, let's first lay a solid foundation.inode The concept originates from UNIX/Linux local file systems (such as ext4, xfs).

What is an inode?

• 全称: Index Node (index node)

• essence: one data structure, assign one to each file Unique, fixed-size identifier——You can think of it as the file's "ID number."

What is inside an inode?

A typical inode contains the following metadata:

struct inode {
    // Basic Information
    uint64_t i_size;        // File size
    uint16_t i_uid;         // Owner ID
    uint16_t i_gid;         // Group ID
    uint16_t i_mode;        // Permission mode (rwxr-xr-x)

    // timestamp
    time_t i_atime;         // Access time
    time_t i_mtime;         // Modification time
    time_t i_ctime;         // State change time

    // Link information
    uint16_t i_nlink;       // hard link count

    // 🔥 The most essential part: data block pointer
    uint32_t i_block[15];   // direct/indirect pointer array
};

The Clever Design of Data Block Pointers

i_block The 15 pointers in the array address data blocks through different levels of indirection:

pointer type	Location	Jump level	Description
direct pointer	i_block[0]~i_block[11]	1 time	Directly points to the data block, suitable for small files.
single indirect pointer	i_block[12]	2 times	Read the pointer block first, then read the data block.
double indirect pointer	i_block[13]	3 times	two-level pointer block jump
triple indirect pointer	i_block[14]	4 times	Triple-level pointer block jump, supports very large files.

This design makes small-file access efficient (using only direct pointers), while also supporting large files (extended through indirect pointers).

Key Insights

The file name is not in the inode!

The file name is stored in Directory file In. The content of the directory file is simply a mapping table:

filename → inode number
a.txt  →  12345
b.pdf  →  67890

Workflow Example

Open /home/user/a.txt the process of:

III. Evolution of Inodes in Distributed Storage

When the model is extended to a distributed environment, the concept of inode is retained, but data block pointer of which the meaning has undergone a fundamental change.

Local vs. Distributed: Comparison at a Glance

Characteristics	local file system	Distributed file system
Storage location	Fixed disk area	Metadata node memory + disk
Data block pointer	pointing to the local disk Block number	point to Data Block ID(including data node IP and local path)
Main structure	Direct/indirect pointer array	Usually Block list
File size	Limited by pointer hierarchy	Theoretically almost unlimited

Taking HDFS as an example: classic design

HDFS's inode (INodeFile category) includes:

• Traditional metadata such as file names, permissions, timestamps, etc.

• Block list: Contains BlockInfo A list of objects, each recording the ID of a data block (default 128MB)

Key changes: Block IDs do not directly point to physical locations. The metadata node maintains another mapping table:

Block ID → [Data Node 1, Data Node 2, Data Node 3] (replica locations)

HDFS read and write process

This kind of Separation of control flow and data flow The design is very clever:

• Metadata nodes only handle metadata requests and do not touch file data.

• Clients interact directly with data nodes to transmit data.

• Avoids the metadata node becoming a bottleneck

Ceph's revolution: decentralization

In pursuit of extreme performance, Ceph adopted a more aggressive design:

• ❌There is no separate metadata node.

• ✅ Metadata, like data, is distributed across the entire cluster

• ✅ Passed CRUSH algorithm Dynamically calculate the storage location of the inode

This completely eliminates the single-point performance bottleneck and achieves true linear scaling.

IV. Conclusion: The Evolution Path of inode

Stage	Core features	Representative system
The Single-Player Era	inode stores local disk block pointers	ext4, xfs
Distributed Classics	Separation of metadata and data, centralized metadata node	HDFS
Distributed Modern	Decentralized, with metadata also stored in a distributed manner	Ceph

Key points review

• What is an inode?: Describe the core data structures of the "ID card" and "address book" files, storing almost all metadata except the file name.

• Evolution in a Distributed Context: The data block pointer no longer points to a local disk address, but instead points to a globally unique, cross-server data block ID.

• Architectural Essence: Based on inodes, the distributed system separates metadata nodes from data nodes—this is the foundation for achieving massive scalability and fault tolerance.

Understand the similarities and differences of inodes in standalone and distributed environments, and you’ve grasped half of the design of distributed storage systems.

If you find this helpful, you are welcome to share it with more friends who are interested in storage technology.