crictl Debugging Tool Usage Guide — The Swiss Army Knife for Kubernetes Node Troubleshooting

Overview

When troubleshooting Pod startup failures, image pull anomalies, and container status issues on Kubernetes nodes, choosing a handy debugging tool is crucial.

crictl is the CRI (Container Runtime Interface)-compatible container runtime debugging tool officially recommended by Kubernetes, designed specifically for Kubernetes node operations and maintenance scenarios. This document records my from ctr from confusion to use crictl A complete practice for efficient troubleshooting, hope it helps you.

1. Why use crictl?

1.1 My Predicament

When debugging the source of a container image, I originally intended to use ctr When viewing image metadata, I ran into these issues:

# Trying to use ctr to view the image list, but getting a syntax error.
ctr -n=k8s.io image ls docker.m.daocloud.io/library/nginx:1.31.3
# Error: filters: parse error: expected an operator

# Want to use ctr to view image details, but the "info" subcommand simply doesn't exist.
ctr -n=k8s.io image info docker.m.daocloud.io/library/nginx:1.31.3
# Error: No help topic for 'info'

Issue Summary:

• ctr is containerd's native debugging tool,very low-level, parameter design is obscure

• Its design goal is to debug containerd itself, not to make things convenient for Kubernetes users.

• The command syntax is counterintuitive, and the learning curve is steep.

1.2 Positioning of crictl

crictl Yes Kubernetes Official Provided for CRI-compatible container runtimes (containerd, CRI-O, etc.)Unified debugging tool.

Characteristics	Explanation
Unified interface	Regardless of whether the underlying runtime is containerd or CRI-O, the command is exactly the same.
Kubernetes semantics	Command structure (pods,ps,inspecti) and kubectl Consistent style
Designed specifically for troubleshooting	Directly supports K8s operations scenarios such as viewing Pod Sandbox, container logs, and image metadata.

One-sentence positioning:crictl Yes The "Swiss Army Knife" for Kubernetes node troubleshooting.

II. crictl vs Other Tools

tool	Positioning	Applicable scenarios
crictl	Kubernetes Node Debugging Tool (CRI Standard Interface)	The first choice for K8s node troubleshooting, view Pod/container/image status
ctr	containerd native client (very low-level)	Debugging containerd itself, ordinary users should use with caution
docker	Docker Engine client (not CRI-compatible)	Building images in the development environment is no longer recommended for use on K8s nodes.

III. Installation and Configuration

3.1 Installation

CentOS 7 / RHEL series:

# Add the official Kubernetes repository
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
EOF

# Install crictl
yum install -y cri-tools

Ubuntu / Debian series:

# Download the binary (please replace the version number with the latest version)
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.36.0/crictl-v1.36.0-linux-amd64.tar.gz
tar -zxvf crictl-v1.36.0-linux-amd64.tar.gz -C /usr/local/bin/
chmod +x /usr/local/bin/crictl

Verify installation:

crictl version

3.2 Configure Runtime Endpoints

crictl You need to know the socket address of the container runtime in order to communicate.

Method 1: Configuration file (Recommended)

cat > /etc/crictl.yaml << EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
pull-image-on-create: false
EOF

Method 2: Environment variables

export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
export IMAGE_SERVICE_ENDPOINT=unix:///run/containerd/containerd.sock

3.3 Verification Configuration

crictl info

Expected output: runtime information in JSON format; success is indicated by no errors.

Click to view example runtime information

{
  "cniconfig": {
    "Networks": [
      {
        "Config": {
          "CNIVersion": "0.3.1",
          "Name": "cni-loopback",
          "Plugins": [
            {
              "Network": {
                "dns": {},
                "ipam": {},
                "type": "loopback"
              },
              "Source": "{\"type\":\"loopback\"}"
            }
          ],
          "Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n  \"type\": \"loopback\"\n}]\n}"
        },
        "IFName": "lo"
      },
      {
        "Config": {
          "CNIVersion": "0.3.1",
          "Name": "k8s-pod-network",
          "Plugins": [
            {
              "Network": {
                "dns": {},
                "ipam": {
                  "type": "calico-ipam"
                },
                "type": "calico"
              },
              "Source": "{\"datastore_type\":\"kubernetes\",\"ipam\":{\"type\":\"calico-ipam\"},\"kubernetes\":{\"kubeconfig\":\"/etc/cni/net.d/calico-kubeconfig\"},\"log_file_path\":\"/var/log/calico/cni/cni.log\",\"log_level\":\"info\",\"mtu\":0,\"nodename\":\"k8s-master-239\",\"policy\":{\"type\":\"k8s\"},\"type\":\"calico\"}"
            },
            {
              "Network": {
                "capabilities": {
                  "portMappings": true
                },
                "dns": {},
                "ipam": {},
                "type": "portmap"
              },
              "Source": "{\"capabilities\":{\"portMappings\":true},\"snat\":true,\"type\":\"portmap\"}"
            },
            {
              "Network": {
                "capabilities": {
                  "bandwidth": true
                },
                "dns": {},
                "ipam": {},
                "type": "bandwidth"
              },
              "Source": "{\"capabilities\":{\"bandwidth\":true},\"type\":\"bandwidth\"}"
            }
          ],
          "Source": "{\n  \"name\": \"k8s-pod-network\",\n  \"cniVersion\": \"0.3.1\",\n  \"plugins\": [\n    {\n      \"type\": \"calico\",\n      \"log_level\": \"info\",\n      \"log_file_path\": \"/var/log/calico/cni/cni.log\",\n      \"datastore_type\": \"kubernetes\",\n      \"nodename\": \"k8s-master-239\",\n      \"mtu\": 0,\n      \"ipam\": {\n          \"type\": \"calico-ipam\"\n      },\n      \"policy\": {\n          \"type\": \"k8s\"\n      },\n      \"kubernetes\": {\n          \"kubeconfig\": \"/etc/cni/net.d/calico-kubeconfig\"\n      }\n    },\n    {\n      \"type\": \"portmap\",\n      \"snat\": true,\n      \"capabilities\": {\"portMappings\": true}\n    },\n    {\n      \"type\": \"bandwidth\",\n      \"capabilities\": {\"bandwidth\": true}\n    }\n  ]\n}"
        },
        "IFName": "eth0"
      }
    ],
    "PluginConfDir": "/etc/cni/net.d",
    "PluginDirs": [
      "/opt/cni/bin"
    ],
    "PluginMaxConfNum": 1,
    "Prefix": "eth"
  },
  "config": {
    "cni": {
      "binDir": "/opt/cni/bin",
      "confDir": "/etc/cni/net.d",
      "confTemplate": "",
      "ipPref": "",
      "maxConfNum": 1
    },
    "containerd": {
      "defaultRuntime": {
        "ContainerAnnotations": null,
        "PodAnnotations": null,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0,
        "options": null,
        "privileged_without_host_devices": false,
        "runtimeEngine": "",
        "runtimePath": "",
        "runtimeRoot": "",
        "runtimeType": ""
      },
      "defaultRuntimeName": "runc",
      "disableSnapshotAnnotations": true,
      "discardUnpackedLayers": false,
      "ignoreRdtNotEnabledErrors": false,
      "noPivot": false,
      "runtimes": {
        "runc": {
          "ContainerAnnotations": null,
          "PodAnnotations": null,
          "baseRuntimeSpec": "",
          "cniConfDir": "",
          "cniMaxConfNum": 0,
          "options": {
            "SystemdCgroup": true
          },
          "privileged_without_host_devices": false,
          "runtimeEngine": "",
          "runtimePath": "",
          "runtimeRoot": "",
          "runtimeType": "io.containerd.runc.v2"
        }
      },
      "snapshotter": "overlayfs",
      "untrustedWorkloadRuntime": {
        "ContainerAnnotations": null,
        "PodAnnotations": null,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0,
        "options": null,
        "privileged_without_host_devices": false,
        "runtimeEngine": "",
        "runtimePath": "",
        "runtimeRoot": "",
        "runtimeType": ""
      }
    },
    "containerdEndpoint": "/run/containerd/containerd.sock",
    "containerdRootDir": "/var/lib/containerd",
    "device_ownership_from_security_context": false,
    "disableApparmor": true,
    "disableCgroup": false,
    "disableHugetlbController": true,
    "disableProcMount": false,
    "disableTCPService": true,
    "enableSelinux": false,
    "enableTLSStreaming": false,
    "enableUnprivilegedICMP": false,
    "enableUnprivilegedPorts": false,
    "ignoreImageDefinedVolumes": false,
    "imageDecryption": {
      "keyModel": "node"
    },
    "maxConcurrentDownloads": 3,
    "maxContainerLogSize": 16384,
    "netnsMountsUnderStateDir": false,
    "registry": {
      "auths": null,
      "configPath": "",
      "configs": null,
      "headers": null,
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://registry-1.docker.io"
          ]
        }
      }
    },
    "restrictOOMScoreAdj": false,
    "rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
    "sandboxImage": "registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.7",
    "selinuxCategoryRange": 1024,
    "stateDir": "/run/containerd/io.containerd.grpc.v1.cri",
    "statsCollectPeriod": 10,
    "streamIdleTimeout": "4h0m0s",
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "systemdCgroup": false,
    "tolerateMissingHugetlbController": true,
    "unsetSeccompProfile": "",
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    }
  },
  "golang": "go1.17.9",
  "lastCNILoadStatus": "OK",
  "lastCNILoadStatus.default": "OK",
  "status": {
    "conditions": [
      {
        "message": "",
        "reason": "",
        "status": true,
        "type": "RuntimeReady"
      },
      {
        "message": "",
        "reason": "",
        "status": true,
        "type": "NetworkReady"
      }
    ]
  }
}

IV. Core Command Practice

4.1 Scenario 1: View Pods and Containers

View all Pod Sandboxes:

crictl pods

View all containers (including stopped ones):

crictl ps -a

View running containers:

crictl ps

Filter by namespace:

crictl pods --namespace <namespace>

💡technique:crictl the command style and kubectl Very similar, and quick to get the hang of.

4.2 Scenario 2: Troubleshooting Image Issues (Core Hands-on Practice)

View all downloaded images:

crictl images

View image details (JSON format):

crictl inspecti <image-name:tag>

Hands-on: View nginx Detailed information about the image

Assume that it has already been pulled on the cluster nodes.nginx:1.31.3 For an image, you can use the following command to view its metadata:

crictl inspecti docker.m.daocloud.io/library/nginx:1.31.3

Observe the output JSON information, especially info.imageSpec.config.Labels Part:

"Labels": {
  "maintainer": "NGINX Docker Maintainers <[email protected]>"
}

Conclusion:maintainer The tag clearly indicates that the maintainer of the image is the official NGINX team. This is crictl The value lies in the fact that it can directly tell us the "origin" of the image, and it is especially useful when you need to confirm the image source.

4.3 Scenario 3: Debugging Containers

Step 1: List all containers in the Pod

First, you need to find the Sandbox ID of the target Pod.

# View all Pod Sandboxes
crictl pods --namespace <namespace>

Output example:

POD ID         CREATED      STATE   NAME                  NAMESPACE   ATTEMPT   RUNTIME
8c2937299e17b  7 days ago   Ready   csi-nodeplogin-dv5l2  finalfs     0         (default)

Then, list the IDs of all containers under that Pod.

# Use the Pod Sandbox ID obtained in the previous step
crictl ps -a --pod 8c2937299e17b

Output example:

CONTAINER      IMAGE          CREATED      STATE            NAME                ATTEMPT   POD ID        POD
86a10f7dda626  edf68cd56ffcc  Running     cluster           o                   8c2937299e17b  csi-nodeplogin-dv5l2
f2d53f329ffc5  720dcb196378   Running     driver-registrar  o                   8c2937299e17b  csi-nodeplogin-dv5l2
ffe48bd9be3e9  4947e46903b36  Running     liveness-probe    o                   8c2937299e17b  csi-nodeplogin-dv5l2

in the output CONTAINER The column is the container ID (abbreviation), for example 86a10f7dda626.

Step 2: View the logs of the business container

Use the container ID obtained in the previous step to view its standard output logs:

crictl logs 86a10f7dda626

If you need to track log output in real time:

crictl logs -f 86a10f7dda626

To view the last 100 lines of logs:

crictl logs --tail=100 86a10f7dda626

Step 3: Enter the container and execute commands (interactive debugging)

When you need to troubleshoot the configuration, network connectivity, or file system status inside a container, entering the container directly is the most effective debugging method.

Method 1: Enter the container and run an interactive shell

Use the container ID obtained in the previous step to enter the container:

# Enter the container (using sh)
crictl exec -it 86a10f7dda626 /bin/sh

# If there is no sh in the container, try bash
crictl exec -it 86a10f7dda626 /bin/bash

# If the container uses a minimal system such as alpine, try ash
crictl exec -it 86a10f7dda626 /bin/ash

After entering the container, you can execute arbitrary commands for debugging:

# View process list
ps aux

# Check network connectivity
curl -I http://localhost:8080/health

# View the configuration file
cat /etc/app/config.yaml

# View environment variables
env | grep APP_

# Test DNS resolution
nslookup kubernetes.default.svc.cluster.local

Method 2: Execute a single command directly inside the container

If you only need to execute a single command temporarily, there is no need to enter an interactive shell:

# List the /app directory.
crictl exec 86a10f7dda626 ls -la /app

# View environment variables inside the container
crictl exec 86a10f7dda626 env

# Check DNS resolution inside the container.
crictl exec 86a10f7dda626 nslookup kubernetes.default.svc.cluster.local

# View the network connections inside the container
crictl exec 86a10f7dda626 netstat -tlnp

Method 3: Use the Pod name to automatically locate and enter the container.

If you are more accustomed to using Pod names, you can combine commands to achieve one-click entry:

# Method 1: Obtain the container ID by Pod name and enter it
POD_NAME="csi-nodeplogin-dv5l2"
CONTAINER_ID=$(crictl ps --pod $(crictl pods --name $POD_NAME -q) -q | head -1)
crictl exec -it $CONTAINER_ID /bin/sh

# Method 2: A more concise approach (if there is only one business container under the Pod)
crictl exec -it $(crictl ps --pod $(crictl pods --name $POD_NAME -q) -q | head -1) /bin/sh

Precautions:

• -it In the parameters,-i Keep STDIN open,-t Allocate a pseudo-terminal; only by combining the two can you get a complete interactive experience.

• If the container is based on alpine such as those built from slim images, there may only be /bin/sh but not /bin/bash

• After entering the container, use exit or press Ctrl + D Exit

• For production environments, operate with caution and avoid executing destructive commands inside containers.

• Some containers may not have curl,nslookup and other tools; at this point, you can use apt update && apt install -y curl or apk add curl Temporary installation

4.4 Scenario 4: Clearing Space

Delete image:

crictl rmi <image-id>

Delete container:

crictl rm <container-id>

Clean up all stopped containers:

crictl rm $(crictl ps -a -q --state=Exited)

Clean up all unused images (use with caution):

crictl rmi --prune

V. Quick Reference Table of Common Commands

Command	Usage
crictl pods	List all Pod Sandboxes
crictl ps -a	List all containers (including stopped)
crictl images	List all images
crictl inspecti <镜像>	View image JSON metadata
crictl inspect <容器>	View container JSON metadata
crictl logs <容器>	View container logs
crictl logs -f <容器>	Track container logs in real time
crictl exec -it <容器> sh	Enter the container to execute interactive commands.
crictl exec <容器> <命令>	Execute a single command in the container
crictl rmi <镜像>	Delete image
crictl rm <容器>	Delete container
crictl rmi --prune	Clean up all unused images
crictl info	View runtime information and configuration
crictl version	View client and runtime versions

VI. Conclusion

Key conclusions	Description
crictl is the best tool for K8s node troubleshooting	The command semantics are clear, and kubectl Consistent style
Specifically handles CRI-compatible runtimes	containerd, CRI-O unified interface
inspecti Commands are the most practical.	You can directly check the image's Labels to verify the source.
exec Commands are a powerful tool for interactive debugging.	You can enter the container to troubleshoot configuration, network, and other issues.
Do not use ctr Perform routine troubleshooting	It is too low-level and only suitable for debugging containerd itself.

Through crictl I can efficiently verify image sources and troubleshoot container issues. I hope this document also helps you troubleshoot Kubernetes node issues more efficiently.