Getting Started with Protocol Buffers: Setting Up a gRPC Development Environment and Buf Dependency Management

Preface

In today's increasingly widespread microservices architecture, Protocol Buffers (Protobuf for short), as an efficient structured data serialization protocol, has become the de facto standard for gRPC service definition. This article will guide you step by step through setting up a complete Protobuf development environment, from IDE plugin configuration to dependency management and then to code generation, enabling you to quickly get started with Protobuf project development.

IDE editor supports Protocol Buffers

1. Install the plugin

In IntelliJ IDEA, you first need to install the Protobuf support plugin:

File → Settings → Plugins → Search for "Protocol Buffers" and install

After installation is complete, restart the IDE for it to take effect.

2. Configure import path

In order for the IDE to correctly recognize the ... in Protobuf files import statement, you need to configure the import path:

File → Settings → Languages & Frameworks → Protocol Buffers

• Check Configure automatically, let the IDE automatically recognize it

• or in Import Paths Manually add the Protobuf definition directory in the project.

Tips

After correctly configuring the import path, the IDE will be able to intelligently recognize import "google/protobuf/any.proto" Such import statements, and provide code completion and navigation features. ::

Managing Protocol Buffers

In the Protobuf ecosystem,Buf It has gradually become an industry-standard tool. It not only provides efficient dependency management capabilities but also integrates code quality checks (Lint), version compatibility checks (Breaking Change Detection), and other features, greatly improving the Protobuf development experience.

Buf CLI Installation

Buf CLI is a command-line tool provided by Buf and can be installed in multiple ways:

macOS (Homebrew):

brew install bufbuild/buf/buf

Other platforms:

For reference Official installation documentation, use the installation script or download the binary file for the corresponding operating system.

Use Buf to manage dependencies

1. Initialize the project root directory

Run the following command in the project root directory to initialize the Buf configuration:

buf config init

After execution, a file named buf.yaml configuration file:

# For details on buf.yaml configuration, visit https://buf.build/docs/configuration/v2/buf-yaml
version: v2
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

• lint: Configure code style inspection rules,STANDARD Indicates the use of Buf's recommended standard rule set

• breaking: Configure compatibility check rules,FILE Indicates whether to check for breaking changes at the file level.

2. Add Protocol Buffers dependency

Taking the addition of the officially maintained Google Protobuf definition library as an example, in buf.yaml increase in deps Field:

# For details on buf.yaml configuration, visit https://buf.build/docs/configuration/v2/buf-yaml
version: v2
deps: 
  - buf.build/googleapis/googleapis
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

Note

buf.build/googleapis/googleapis is Google's officially maintained public Protobuf definition library, containing google/type/decimal.proto,google/protobuf/timestamp.proto and other common type definitions.

After adding dependencies, run the following command to download and lock dependency versions:

buf dep update

This command will generate a file named buf.lock dependency version lock file:

# Generated by buf. DO NOT EDIT.
version: v2
deps:
  - name: buf.build/googleapis/googleapis
    commit: 72c8614f3bd0466ea67931ef2c43d608
    digest: b5:13efeea24e633fd45327390bdee941207a8727e96cf01affb84c1e4100fd8f48a42bbd508df11930cd2884629bafad685df1ac3111bc78cdaefcd38c9371c6b1

Note

• First execution buf dep update will generate buf.lock file; this file will be automatically updated when new dependencies are added later.

• buf.lock Records the specific version (commit hash) and checksum (digest) of the dependency,Please do not modify manually., to ensure consistency in team building

• This dependency management approach is similar in design philosophy to tools like Go Modules and npm, and is very intuitive to get started with.

3. Add the Buf dependency directory to the IDE import path

In order for the IDE to recognize the downloaded third-party dependencies (such as google/type/decimal.proto), we need to add the actual storage path of the dependency to the IDE's Protocol Buffers import path.

First, run the following command to view the storage information of the dependencies:

buf dep prune --debug

In the output log, you can find the following JSON information:

{"moduleFullName":"buf.build/googleapis/googleapis","commitID":"72c8614f3bd0466ea67931ef2c43d608","dirPath":"b5/buf.build/googleapis/googleapis/72c8614f3bd0466ea67931ef2c43d608"}

Analysis

• moduleFullName: The complete name of the module

• commitID: Version identifier currently in use

• dirPath: relative path of the module in the cache

On macOS, Buf's global cache directory is ~/.cache/buf/v3/modules, therefore the module's Full path For:

~/.cache/buf/v3/modules/b5/buf.build/googleapis/googleapis/72c8614f3bd0466ea67931ef2c43d608

Key configuration

We need to [verb] the ... under the above path.files subdirectory Add to the IDE's import path, i.e.:

~/.cache/buf/v3/modules/b5/buf.build/googleapis/googleapis/72c8614f3bd0466ea67931ef2c43d608/files

Because this directory happens to contain google directory, so that in other Proto files it can be accessed via import "google/type/decimal.proto" The correct citation is defined.

4. Create a Protocol Buffers service definition file

Now, let's create an actual Protobuf service definition file, demonstrating how to import and use third-party dependencies:

// proto/product/product.proto
syntax = "proto3";

package product;
option go_package = "github.com/kittycoffee/kittycoffee-protos/product";

import "google/protobuf/any.proto";
import "google/type/decimal.proto";
import "proto/user/user.proto";

service ProductService {
  rpc GetProduct(GetProductRequest) returns (ProductResponse);
  rpc CreateProduct(CreateProductRequest) returns (ProductResponse);
}

message GetProductRequest {
  string id = 1;
}

message CreateProductRequest {
  string name = 1;
  string description = 2;
  google.type.Decimal price = 3;
  string image_url = 4;
  string created_user_id = 5;
}

message ProductResponse {
  string id = 1;
  string name = 2;
  string description = 3;
  string price = 4;
  string image_url = 5;
  string created_user_id = 6;
  user.UserMinimalResponse created_user = 7;
  string created_at = 8;
  string updated_at = 9;
}

Note

The reason why the IDE can correctly recognize import "google/type/decimal.proto", precisely because we correctly configured the import path earlier—using Buf’s cached files The directory has been added to the IDE's Protocol Buffers import path.

5. Configure the buf.gen.yaml file

Code generation is a key step in the Protobuf development workflow. Create buf.gen.yaml file to configure code generation rules:

# https://buf.build/docs/configuration/v2/buf-gen-yaml/
version: v2
managed:
  enabled: true
  disable:
    # Don't modify any files in buf.build/googleapis/googleapis
    - module: buf.build/googleapis/googleapis
  override:
    - file_option: go_package_prefix
      value: github.com/kittycoffee/kittycoffee-protos/gen
plugins:
  - protoc_builtin: go
    protoc_path: /usr/local/bin/protoc
    out: gen
    opt:
      - paths=source_relative
  - protoc_builtin: go-grpc
    protoc_path: /usr/local/bin/protoc
    out: gen
    opt:
      - paths=source_relative
inputs:
  - directory: proto

Configuration Item Details

managed.disable

The configuration requires disabling the automatically managed dependency module. With buf.build/googleapis/googleapis for example:

• If Do not close, Buf will use the current project's go_package_prefix Rewrite the prefix of the module's Go import path, causing the generated code to reference local paths, such as github.com/kittycoffee/kittycoffee-protos/gen/type/decimal

• If Close(i.e., configuration disable), then retain the original import paths of third-party libraries and correctly reference officially maintained Go language libraries, such as google.golang.org/genproto/googleapis/type/decimal

✅Of course we need to close it.—Simply use the official remote dependency library directly; there is no need to maintain a local Go implementation of third-party libraries.

plugins

Declare the code generation plugins to be used:

• go Generate standard Go message type code (.pb.go)

• go-grpc: Generate gRPC server and client code (_grpc.pb.go)

inputs

Specify the Protobuf source file directory as proto, Buf will only process those under that directory .proto File.

6. Generate Protocol Buffers code

Once all configurations are ready, run the generation command:

buf generate

After execution is complete, it will be generated under the project directory.gen Directory containing all generated Go code:

gen
├── product
│   ├── product.pb.go
│   └── product_grpc.pb.go
└── user
    ├── user.pb.go
    └── user_grpc.pb.go

• *.pb.go: Go struct definitions containing all message types and serialization/deserialization methods

• *_grpc.pb.go: Includes gRPC service interface definitions, client stub, and server registration function

Summary

At this point, we have completed the full Protocol Buffers development environment setup, including:

• ✅ IDE plugin installation and import path configuration

• ✅ Buf CLI Tool Installation

• ✅ Use Buf to manage third-party Protobuf dependencies

• ✅ Configure the IDE to recognize dependency files in the Buf cache

• ✅ Write a custom Protobuf service definition

• ✅ Configured the code generation rules and successfully generated Go code

After completing the above steps,github.com/kittycoffee/kittycoffee-protos The defined service can then be referenced as a dependency by other projects. In the server project, you can implement the generated interface and start the gRPC service; in the client project, you can use the generated client stub to easily call the remote service.

Reference

• Buf CLI Quick Start

• Buf Official Documentation

• Protocol Buffers official documentation

The code examples in this document are based on the macOS environment. Linux/Windows users may need to make corresponding adjustments to paths and commands.