gRPC Keepalive Testing Tool keepalivectl from Zero to One: Best Practices for Verifying Server-Side Connection Keepalive Mechanisms

keepalivectl gRPC Keepalive testing tool

Introduction

Today, as microservice architectures become increasingly widespread, gRPC has become one of the mainstream protocols for inter-service communication. However, while enjoying the high performance and strong typing advantages brought by gRPC, we are also facing some thorny operational issues.

One typical scenario is:How to verify whether a gRPC server correctly supports the Keepalive mechanism?

Imagine that your service connection pool frequently disconnects, connection status is unstable, and intermittent timeouts occur in the production environment... These problems are very likely related to improper Keepalive configuration. But when you try to use grpcurl However, during verification, we obtained:

bash

$ grpcurl -plaintext localhost:8500 list
Failed to list services: server does not support the reflection API

The reflection API is not enabled on the server, so it cannot be used.grpcurl Perform verification.

This is exactly what I developed.keepalivectl The original intention — a command-line tool specifically designed for testing gRPC server Keepalive and connection pool support.

This article will give you an in-depth look at:

Principles of the gRPC Keepalive mechanism
Design and Implementation of the keepalivectl Tool
Practical verification of server-side Keepalive support
The complete development journey from zero to one

What is gRPC Keepalive?

HTTP/2 and Keepalive

gRPC is based on the HTTP/2 protocol, and HTTP/2 introduces a multiplexing mechanism that allows multiple requests to share the same TCP connection. While this long connection brings performance improvements, it also introduces a problem:How can I check whether the connection is still valid?

The keepalive mechanism detects the health status of the connection by periodically sending HTTP/2 PING frames:

Client                         Server
  |                              |
  |── HTTP/2 PING (keepalive) ──>|
  |                              |
  |<── HTTP/2 PONG (response) ───|
  |                              |

Why is Keepalive so important?

Question	No Keepalive	Has Keepalive
connection pool	The connection may be unexpectedly closed by middleware.	Keep connections alive to avoid reconnection overhead.
load balancer	Idle connections are disconnected by the LB.	Regularly ping to keep the connection alive
Cloud environment	NAT timeout causes the connection to disconnect	Maintain connection status
Fault detection	Unable to detect network problems in a timely manner	Actively probe connection status

Key parameters

Parameter	Client	Server side	Description
`Time`	✅	✅	PING frame transmission interval
`Timeout`	✅	✅	PING response timeout
`MinTime`	❌	✅	The minimum PING interval allowed by the server
`MaxConnectionIdle`	❌	✅	Maximum idle connection time
`PermitWithoutStream`	✅	✅	Is PING allowed when there is no active stream?

Key pitfalls: If the client's Time less than the server-side one MinTime, the server will return ENHANCE_YOUR_CALM Error code, prompt too_many_pings.

Tool Design

Design goals

Simple and easy to use: One command to verify server-side Keepalive
Real-world scenarios: Simulate actual connection pool behavior
Real-time feedback: Visualize connection status changes
Detailed report: Compile and summarize test results

Technology selection

component	Select	Reason
CLI framework	Cobra	The most popular Go CLI frameworks
gRPC	google.golang.org/grpc	Official gRPC client
Log	gologger	Structured logging with level support
color	aurora	Chained API, elegant colored output
Connection pool	grpc-go-pool	Connection pool management

Architecture design

┌─────────────────────────────────────────────────────────────┐
│                      keepalivectl                          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    │
│  │   Config    │    │   Monitor   │    │   Reporter  │    │
│  │  (Configuration Management)  │    │  (Connection Monitoring)  │    │  (Report Generation)  │    │
│  └─────────────┘    └─────────────┘    └─────────────┘    │
│         │                  │                  │             │
│         ▼                  ▼                  ▼             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │                    gRPC Connection Pool                       │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Connection state machine

gRPC connections have 5 states, and keepalivectl monitors these state changes in real time:

       ┌─────────┐
       │  IDLE   │ ← Connection created, underlying connection not yet established
       └────┬────┘
            │
            ▼
    ┌─────────────┐
    │ CONNECTING  │ ← Establishing connection
    └──────┬──────┘
           │
           ▼
    ┌─────────────┐
    │   READY     │ ← Connection available, data can be transmitted
    └──────┬──────┘
           │
     ┌─────┴─────┐
     ▼           ▼
┌──────────┐ ┌───────────────┐
│   IDLE   │ │ TRANSIENT_    │ ← Connection temporarily unavailable
│          │ │  FAILURE      │   (may automatically recover)
└──────────┘ └───────────────┘
     │
     ▼
┌──────────┐
│SHUTDOWN  │ ← Connection closed
└──────────┘

Practical verification

Scenario 1: Basic Test

bash

$ keepalivectl --endpoint localhost:8500 --duration 2m

[INF] 2026-09-03T16:30:50+08:00 [Connection 0] connected successfully, monitoring (keepalive: 10s, timeout: 5s)
[INF] 2026-09-03T16:31:05+08:00 [Connection 0] state changed: READY (change #1)
[INF] 2026-09-03T16:31:20+08:00 [Connection 0] state: READY
[INF] 2026-09-03T16:31:35+08:00 [Connection 0] state: READY
[INF] 2026-09-03T16:31:50+08:00 [Connection 0] state: READY
...
[INF] 2026-09-03T16:33:50+08:00 Test completed, duration: 3m0s

Scenario 2: Concurrency Testing

bash

$ keepalivectl --concurrency 10 --duration 3m

All connections are established simultaneously and remain in READY state, verifying the concurrency stability of the connection pool.

Scenario 3: Capturing Configuration Issues

When the client Keepalive interval is shorter than the server's MinTime Time:

ERROR: Client received GoAway with error code ENHANCE_YOUR_CALM
and debug data equal to ASCII "too_many_pings".

Solution:

bash

keepalivectl --keepalive-time 30s

Complete test result example

============================================================
📊 Test Summary
============================================================
Endpoint:           127.0.0.1:8500
Keepalive Interval: 10s
Keepalive Timeout:  5s
Test Duration:      3m0s
Concurrency:        2
============================================================
Total Checks:       25
Ready State Count:  23
State Changes:      2
✅ All connections healthy, server Keepalive support is good
============================================================

Core Implementation Analysis

1. Keepalive client configuration

kacp := keepalive.ClientParameters{
    Time:                10 * time.Second, // PING interval
    Timeout:             5 * time.Second,  // PING timeout
    PermitWithoutStream: true,             // Send PING even with no active stream
}

conn, err := grpc.DialContext(ctx, endpoint,
    grpc.WithInsecure(),
    grpc.WithKeepaliveParams(kacp),
)

2. Connection Status Monitoring

ticker := time.NewTicker(checkInterval)
defer ticker.Stop()

for {
    select {
    case <-ctx.Done():
        return
    case <-ticker.C:
        state := conn.GetState()
        // Record status changes
        if state != lastState {
            log.Printf("State changed: %s", state)
            lastState = state
        }
    }
}

3. Group help information

Using Cobra's SetAnnotation Implement flags grouping:

rootCmd.Flags().StringVarP(&cfg.Endpoint, "endpoint", "e", "localhost:8500", "gRPC server endpoint")
rootCmd.Flags().SetAnnotation("endpoint", "group", []string{"Target"})

Custom help template:

Target:
  -e, --endpoint          gRPC server endpoint address (default: localhost:8500)

Test Behavior:
  -d, --duration          Test duration (default: 2m0s)
  -i, --check-interval    Connection state check interval (default: 3s)
  -c, --concurrency       Number of concurrent connections (default: 1)

Keepalive Settings:
  -t, --keepalive-time    Keepalive ping interval (default: 10s)
  -T, --keepalive-timeout  Keepalive ping timeout (default: 5s)

4. Colored log output

import "github.com/logrusorgru/aurora"

msg := fmt.Sprintf("%s %s %s",
    aurora.Cyan(timestamp),
    aurora.Green("READY"),
    aurora.BrightBlue(message),
)

gologger.Info().Msg(msg)

Open Source Contributions

keepalivectl Open-sourced, welcome to use and contribute:

GitHub:
License: MIT

Quick Installation

bash

go install github.com/fishfinal/keepalivectl/cmd/keepalivectl@latest

Stars, Issues, and PRs are welcome.

Summary

keepalivectl The development process validated several key points:

gRPC Keepalive validation is necessary.
Improper server configuration can cause connection problems.
Production environments require proactive verification.
The Value of Tool-Based Testing
Automated verification, reducing manual troubleshooting
Visualized output for quick problem identification
The maturity of the Go ecosystem
Cobra + gRPC + Logging Library = Efficient Development
Cross-platform compilation, build once and run everywhere.
The root cause of connection pool problems
Connection instability may be related to the Keepalive configuration.
Tools can help quickly verify and adjust.

Quick Reference

bash

# Quick verification
keepalivectl --endpoint localhost:8500 --duration 1m

# Concurrency testing
keepalivectl --concurrency 50 --duration 5m

# Custom Keepalive
keepalivectl --keepalive-time 30s --keepalive-timeout 10s

# Complete parameters
keepalivectl --endpoint grpc-server:8500 \
  --concurrency 10 \
  --duration 5m \
  --keepalive-time 30s \
  --keepalive-timeout 10s \
  --check-interval 5s

Reference materials

Related article recommendations:

Closing Remarks:keepalivectl It originates from a real debugging scenario—when a gRPC service does not have reflection enabled, Keepalive support needs to be verified. If you have encountered similar issues, you are welcome to use it and provide feedback!

Appendix: Complete Command Reference

bash

$ keepalivectl --help

keepalivectl is a command-line tool for testing gRPC server Keepalive and connection pooling features.

It validates server-side keepalive handling by establishing long-lived connections and sending Keepalive pings,
with support for concurrent connection testing, real-time connection state monitoring, and detailed statistics reporting.

Core Features:
  • Establish gRPC long-lived connections and send Keepalive pings
  • Monitor connection state changes (Idle, Ready, TransientFailure)
  • Support concurrent connection testing
  • Customizable Keepalive parameters (interval, timeout)
  • Real-time reporting and final statistics summary

Usage:
  keepalivectl [flags]


Target:
  -e, --endpoint          gRPC server endpoint address (default: localhost:8500)

Test Behavior:
  -d, --duration          Test duration (default: 2m0s)
  -i, --check-interval    Connection state check interval (default: 3s)
  -c, --concurrency       Number of concurrent connections (default: 1)

Keepalive Settings:
  -t, --keepalive-time    Keepalive ping interval (default: 10s)
  -T, --keepalive-timeout  Keepalive ping timeout (default: 5s)

Global Flags:
  -h, --help   help for keepalivectl

gRPC Keepalive Testing Tool keepalivectl from Zero to One: Best Practices for Verifying Server-Side Connection Keepalive Mechanisms ​

Introduction ​

What is gRPC Keepalive? ​

HTTP/2 and Keepalive ​

Why is Keepalive so important? ​

Key parameters ​

Tool Design ​

Design goals ​

Technology selection ​

Architecture design ​

Connection state machine ​

Practical verification ​

Scenario 1: Basic Test ​

Scenario 2: Concurrency Testing ​

Scenario 3: Capturing Configuration Issues ​

Complete test result example ​

Core Implementation Analysis ​

1. Keepalive client configuration ​

2. Connection Status Monitoring ​

3. Group help information ​

4. Colored log output ​

Open Source Contributions ​

Quick Installation ​

Stars, Issues, and PRs are welcome. ​

Summary ​

Quick Reference ​

Reference materials ​

Appendix: Complete Command Reference ​

gRPC Keepalive Testing Tool keepalivectl from Zero to One: Best Practices for Verifying Server-Side Connection Keepalive Mechanisms

Introduction

What is gRPC Keepalive?

HTTP/2 and Keepalive

Why is Keepalive so important?

Key parameters

Tool Design

Design goals

Technology selection

Architecture design

Connection state machine

Practical verification

Scenario 1: Basic Test

Scenario 2: Concurrency Testing

Scenario 3: Capturing Configuration Issues

Complete test result example

Core Implementation Analysis

1. Keepalive client configuration

2. Connection Status Monitoring

3. Group help information

4. Colored log output

Open Source Contributions

Quick Installation

Stars, Issues, and PRs are welcome.

Summary

Quick Reference

Reference materials

Appendix: Complete Command Reference