Detailed Guide to Offline Installation of a Three-Node Kubernetes v1.33.0 Cluster in an Ubuntu 22.04 Environment

This document applies to the Ubuntu 22.04 + containerd environment. The network plugin uses Flannel as an example; other plugins such as Calico can be handled similarly.

Preparatory work

1. Three servers

Suggested hostnames: master, node1, node2

Click to view more related information

Use the hostnamectl tool to configure the hostname, for example, set the master node to master.

shell

hostnamectl set-hostname master

Configure static IP

Click to view more related information

Through editing /etc/netplan/00-installer-config.yaml Modify the network card configuration file, for example:

shell

network:
  version: 2
  ethernets:
    ens160:
      addresses:
        - 192.168.0.160/24
      routes:
        - to: default
          via: 192.168.0.1
      nameservers:
        addresses:
          - 114.114.114.114
          - 8.8.8.8

Execute after modification netplan apply Takes effect immediately!

⚠️ If connecting via ssh, execute netplan apply After that, the connection will be disconnected and may freeze; just close the terminal and open a new one!

Configure static IP, hostname, and mutual /etc/hosts
Disable the firewall

Click to view more related information

shell

sudo systemctl disable --now ufw

This will turn off the firewall and disable automatic startup at boot.

Permanently disable the swap partition

Click to view more related instructions

shell

sudo sed -ri 's/.*swap.*/#&/' /etc/fstab

After restarting, check whether the output is empty using the following command.

shell

swapon --show

If there is no output, it means no Swap is currently enabled.
If there is output, it means there are still Swap devices or files in use.

Modify kernel parameters to allow bridge firewall and IP forwarding

Click to view more related information

Edit the system configuration file

write /etc/sysctl.conf Configuration file, Cancel net.ipv4.ip_forward=1 Just save the previous comments.

txt

net.ipv4.ip_forward=1

Reload configuration (no restart required)

shell

sudo sysctl -p

Enable cgroups v2

Click to view more related information

Step 1: Enable cgroups v2 and disable v1

Modify the /etc/default/grub file, modify the GRUB_CMDLINE_LINUX line, and add the following parameters

toml

GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all"

Update GRUB and restart

shell

sudo update-grub && sudo reboot

Step 2: Verify the switching result

After reboot, verify that cgroups v2 is enabled.

shell

mount | grep cgroup

The expected output should contain only the cgroup2 mount point:

txt

cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

Write DNS records

Click to view more related information

Write DNS records on the servers pending Kubernetes installation; note: write records for every server—however many there are!

shell

echo '192.168.0.160 master' >> /etc/hosts
echo '192.168.0.161 node1' >> /etc/hosts
echo '192.168.0.162 node2' >> /etc/hosts

[!tip] Reminder The Kubernetes cluster to be installed has several servers; just perform the operations on these servers one by one!

Configure passwordless transfer

Click to view more related information

Generate SSH public-private key pair

shell

ssh-keygen  #Just keep pressing Enter and don't enter a password.

Write DNS records on the servers awaiting Kubernetes installation; copy them to as many servers as there are!

shell

ssh-copy-id master
ssh-copy-id node1
ssh-copy-id node2

[!important] ssh-copy-id ssh-copy-id is used to copy the local user's public key (by default~/.ssh/id_rsa.pub) copy to the remote directory host's~The commands in the /.ssh/authorized_keys file.

thereby enabling passwordless SSH login, subsequently through ssh node1 When logging in, the system will directly use key authentication, with no need to manually enter a password.

[!tip] Reminder The Kubernetes cluster to be installed has several servers; just operate on these servers one by one! Theoretically, the current host does not need to copy the current host.

3. Modify kernel parameters

Allow bridged firewall and IP forwarding; otherwise, the Kubernetes cluster may experience network issues.

Click to view more related information

Manually load the br_netfilter module

shell

sudo modprobe br_netfilter

Verify whether loading is successful:

shell

lsmod | grep br_netfilter  # Should return lines related to br_netfilter

Modify /etc/sysctl.conf

shell

sudo tee /etc/sysctl.d/99-kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

Application configuration

shell

sudo sysctl --system  # Load all sysctl configurations (including /etc/sysctl.d/)

Verify whether it takes effect:

shell

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

The output should all be 1

4. Configure time synchronization

Click to view more related information

If you are using VMware ESXi virtualization, you can configure time synchronization through the web management interface: Host -> Manage -> System -> Time and Date, click 编辑设置 Select in the pop-up dialog box 使用网络时间协议（启用 NTP 客户端） at NTP 服务器 Input pool.ntp.org Just click Save!

shell

timedatectl

Will output time information similar to the following

text

               Local time: Mon 2025-06-23 13:38:18 UTC
           Universal time: Mon 2025-06-23 13:38:18 UTC
                 RTC time: Mon 2025-06-23 13:38:18
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

You can see that the local time is incorrect, and the time zone needs to be changed.

shell

timedatectl set-timezone Asia/Shanghai

Enter again timedatectl Verification is sufficient.

txt

               Local time: Mon 2025-06-23 21:42:25 CST
           Universal time: Mon 2025-06-23 13:42:25 UTC
                 RTC time: Mon 2025-06-23 13:42:26
                Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Now the local time matches too!

5. Install IPVS

Click to view more related instructions

Install the IPVS kernel module

shell

apt install -y ipset ipvsadm

Load kernel module

shell

sudo modprobe ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh

Check kernel loading ip_vs

shell

lsmod | grep ip_vs

Check IPVS rules

shell

sudo ipvsadm -L

[!tip] reminder After Kubernetes installation is complete, kube-proxy can use IPVS.

shell

kubectl edit cm -n kube-system kube-proxy

Change proxy-mode to ipvs:

yaml

mode: "ipvs"

6. Configure an apt-get proxy (if network access is restricted)

Click to view more related information

In the upcoming offline package production process, we need to download a large amount from official sources..deb Package. Limited by the domestic network environment or the company's intranet firewall,apt-get download will often encounter Could not handshake and other connection failure issues.

If your environment has network restrictions, please follow the steps below to apt-get Configure proxy.

6.1. Check whether the proxy port is available

Assume your proxy server address is http://192.168.0.23:7890, please first test connectivity (modify the IP and port according to the actual situation):

shell

curl -I -x http://192.168.0.23:7890 https://www.google.com

If it returns HTTP/1.1 200 OK, indicating that the proxy is available and you can proceed to the next step; if it prompts a connection timeout or Connection refused Please resolve the proxy server network issue first.

6.2. Configure one-time proxy download parameters

If you do not want to modify system files, you can execute it later.apt-get download When doing so, simply add the proxy parameter before the command. The recommended format is as follows:

shell

# Example (please replace with your actual proxy IP and port)
sudo apt-get download -o Acquire::http::proxy="http://192.168.0.23:7890" -o Acquire::https::proxy="http://192.168.0.23:7890" <package name>

For example, download containerd.io can be used when:

shell

sudo apt-get download -o Acquire::http::proxy="http://192.168.0.23:7890" -o Acquire::https::proxy="http://192.168.0.23:7890" containerd.io=1.7.25-1

6.3. Persistent Configuration (Long-Term Effect)

If you want all subsequent items on the current machine apt All commands automatically go through the proxy; the configuration can be written into apt in the configuration file, once and for all:

shell

sudo tee /etc/apt/apt.conf.d/99proxy.conf <<EOF
Acquire::http::proxy "http://192.168.0.23:7890";
Acquire::https::proxy "http://192.168.0.23:7890";
EOF

After writing to the file, no operation is required; all subsequent apt-get update and apt-get download will all automatically use this proxy.

[!tip] Reminder After configuring the proxy, if you have never run it before sudo apt-get update, please be sure to perform a cache update first:

shell

sudo apt-get update

Phase 1: Prepare the offline installation package in a networked environment

1. Set up the download environment

Switch to a normal user

shell

su testify

Reminder

When installing Ubuntu, there is always a default user, that is, the user created when installing the system; other users are also fine, as long as they are non-root users! If using root Downloading software packages will cause permission warning issues!W: Download is performed unsandboxed as root as file

Create working directory

shell

mkdir -p ~/k8s-offline/v1.33.0

Reminder

Can always remain in the home directory ($HOME) Complete the relevant operations for preparing the offline installation package!

Install the necessary tools

shell

sudo apt-get update && sudo apt-get install -y apt-transport-https curl gnupg

2. Add the Kubernetes apt repository

1. Create the keyrings directory (if it does not exist)

shell

sudo mkdir -p /etc/apt/keyrings

2. Download and add the GPG key

shell

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

3. Add the Kubernetes apt repository

shell

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list

4. Update the apt cache

shell

sudo apt-get update

3. Download Kubernetes components

Create a download directory

shell

DOWNLOAD_DIR="$HOME/k8s-offline/v1.33.0/deb-packages/kubernetes"
mkdir -p $DOWNLOAD_DIR

Download the specified version of kubeadm, kubelet, and kubectl

shell

apt-get download \
  kubelet=1.33.0-1.1 \
  kubeadm=1.33.0-1.1 \
  kubectl=1.33.0-1.1

Download dependency packages

shell

apt-get download \
  conntrack \
  ebtables \
  socat \
  ethtool \
  ipset \
  nfs-common \
  libseccomp2 \
  libltdl7 \
  libnfsidmap1 \
  libipset13 \
  libtirpc3 \
  libtirpc-common \
  rpcbind \
  keyutils \
  cri-tools \
  kubernetes-cni

Move all .deb files to the Downloads directory.

shell

mv *.deb $DOWNLOAD_DIR

4. Download container runtime components

Create download directory

shell

DOWNLOAD_DIR="$HOME/k8s-offline/v1.33.0/deb-packages/cri"
mkdir -p $DOWNLOAD_DIR

Install containerd Download the necessary tools required

Kubernetes v1.33.0 requirements containerd Version >= 1.7.25, while the official Ubuntu 22.04 repository by default only provides 1.7.12. Therefore, we need to use the latest version provided by the official Docker repository containerd.io.

shell

# 1. Add Docker official GPG key and apt source (compatible with Ubuntu 22.04)
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
sudo echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu jammy stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# 2. Update cache
sudo apt-get update

Download the specified version of containerd

shell

# Use the package from Docker's official repository; note that the package name is containerd.io, not containerd.
# Here, 1.7.25 is used as an example, which meets the K8s 1.33 requirements.
apt-get download containerd.io=1.7.25-1

Download dependency packages

shell

apt-get download runc

Move all .deb files to the Downloads directory

shell

mv *.deb $DOWNLOAD_DIR

5. Download the network plugin (Calico) configuration file

Specify the download directory

shell

DOWNLOAD_DIR="$HOME/k8s-offline/v1.33.0/manifests"
mkdir -p $DOWNLOAD_DIR

shell

wget -c https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/calico.yaml \
 -O $DOWNLOAD_DIR/calico.yaml

6. Download the container image

First install kubeadm (online environment)

shell

sudo apt-get install -y kubelet=1.33.0-1.1 kubeadm=1.33.0-1.1 kubectl=1.33.0-1.1
sudo apt-mark hold kubelet kubeadm kubectl  # Lock version

Install Docker

shell

sudo apt-get install -y docker.io
sudo systemctl start docker

Configure Docker domestic mirror source

Click to view more related information

Create or modify Docker configuration file If it does not exist /etc/docker/daemon.json, vim automatically creates it; if it already exists, edit it directly:

bash

sudo vim /etc/docker/daemon.json

Add mirror source configuration Write the following content into the file (using the Alibaba Cloud mirror as an example; for other sources, see the appendix):

json

{
  "registry-mirrors": ["https://<your-mirror-url>.mirror.aliyuncs.com"]
}

Restart the Docker service

bash

sudo systemctl daemon-reload
sudo systemctl restart docker

Verify configuration

bash

sudo docker info | grep "Registry Mirrors" -A 1

If you see the configured mirror address, it means it was successful.

You can try mirror sources to see if they are available.

Configure Docker to use a proxy

Click to view more related instructions

If it still doesn't work after configuring the domestic mirror source above, just configure a proxy directly.

Create or edit Docker service configuration

Create a configuration directory

shell

sudo mkdir -p /etc/systemd/system/docker.service.d

Add configuration file

shell

sudo tee /etc/systemd/system/docker.service.d/http-proxy.conf <<EOF
[Service]
Environment="HTTP_PROXY=http://192.168.0.23:7890"
Environment="HTTPS_PROXY=http://192.168.0.23:7890"
Environment="NO_PROXY=localhost,127.0.0.1,.aliyuncs.com"
EOF

Reload and restart Docker

shell

sudo systemctl daemon-reload && sudo systemctl restart docker

Test download mirror

shell

sudo docker pull nginx:latest

List the images required for Kubernetes v1.33.0

shell

sudo kubeadm config images list --kubernetes-version=v1.33.0

This will output all image versions required by Kubernetes v1.33.0.

txt

registry.k8s.io/kube-apiserver:v1.33.0
registry.k8s.io/kube-controller-manager:v1.33.0
registry.k8s.io/kube-scheduler:v1.33.0
registry.k8s.io/kube-proxy:v1.33.0
registry.k8s.io/coredns/coredns:v1.12.0
registry.k8s.io/pause:3.10
registry.k8s.io/etcd:3.5.21-0

Pull the Kubernetes v1.33.0 image

shell

images=(
  registry.k8s.io/kube-apiserver:v1.33.0
  registry.k8s.io/kube-controller-manager:v1.33.0
  registry.k8s.io/kube-scheduler:v1.33.0
  registry.k8s.io/kube-proxy:v1.33.0
  registry.k8s.io/coredns/coredns:v1.12.0
  registry.k8s.io/pause:3.10
  registry.k8s.io/etcd:3.5.21-0
)

for image in "${images[@]}"; do
  sudo docker pull $image
done

Save the Kubernetes v1.33.0 image as a tar file

Create an image storage directory

shell

mkdir -p $HOME/k8s-offline/v1.33.0/images

Export image to archive file

shell

sudo docker save -o $HOME/k8s-offline/v1.33.0/images/k8s-images.tar \
  registry.k8s.io/kube-apiserver:v1.33.0 \
  registry.k8s.io/kube-controller-manager:v1.33.0 \
  registry.k8s.io/kube-scheduler:v1.33.0 \
  registry.k8s.io/kube-proxy:v1.33.0 \
  registry.k8s.io/coredns/coredns:v1.12.0 \
  registry.k8s.io/pause:3.10 \
  registry.k8s.io/etcd:3.5.21-0

Download the Calico image for the network plugin.

shell

images=(
  calico/node:v3.26.0
  calico/cni:v3.26.0
  calico/kube-controllers:v3.26.0
)

for image in "${images[@]}"; do
  sudo docker pull $image
done

Export Calico network plugin images as an archive.

shell

sudo docker save -o $HOME/k8s-offline/v1.33.0/images/calico-images.tar \
  calico/node:v3.26.0 \
  calico/cni:v3.26.0 \
  calico/kube-controllers:v3.26.0

7. Prepare the installation script

Create installation script

shell

cat > $HOME/k8s-offline/v1.33.0/install.sh << 'EOF'
#!/bin/bash

# Install Kubernetes components
sudo dpkg -i ./deb-packages/kubernetes/*.deb

# Install the container runtime CRI component.
sudo dpkg -i ./deb-packages/cri/*.deb
systemctl start containerd
# sudo systemctl start docker

# Load Kubernetes image into Docker
# docker load -i ./images/k8s-images.tar
# Load Kubernetes image to Contained
sudo ctr -n=k8s.io images import ./images/k8s-images.tar

# Load Calico Docker image
# docker load -i ./images/calico-images.tar
# Load Calico image Contained
sudo ctr -n=k8s.io images import ./images/calico-images.tar

# Pin Kubernetes version
sudo apt-mark hold kubelet kubeadm kubectl

# Copy the network plugin configuration file to Kubernetes' default /etc/kubernetes/manifests/
cp ./manifests/calico.yaml /etc/kubernetes/manifests/
EOF

Add executable permission

shell

chmod +x $HOME/k8s-offline/v1.33.0/install.sh

8. Package all files

shell

sudo tar -czvf k8s-offline-v1.33.0.tar.gz -C $HOME/k8s-offline/v1.33.0 .

Phase 2: Install the cluster in an offline environment

1. Transfer files to all nodes

will k8s-offline-v1.33.0.tar.gz Transfer to all three node servers.

Tip

can pass through scp Use commands to transfer files conveniently and quickly.

2. Perform the basic installation on all nodes.

Extract the offline installation package resource archive file

Create the extraction target directory

shell

mkdir -p /opt/k8s-install

Extract to the extraction target directory

shell

tar -xzvf k8s-offline-v1.33.0.tar.gz -C /opt/k8s-install

Enter the extraction target directory

shell

cd /opt/k8s-install

Run the installation script

shell

sudo ./install.sh

Verify installation

View kubeadm version information

shell

kubeadm version

View kubectl client version information

shell

kubectl version --client

View kubelet version information

shell

kubelet --version

View container images

shell

# View contained View container image
# sudo docker images
# View contained View container image
sudo ctr -n=k8s.io images ls -q

Check the rpcbind service status

shell

systemctl status rpcbind.service

Check the containerd service status

shell

systemctl status containerd.service

Check kubelet service status

shell

systemctl status kubelet

Warning

Currently, kubelet should be in a non-running state, even if using systemctl restart kubelet will not start successfully either, only Initialize the control plane Only then will it be pulled up.kubelet Service!

Of course, if after initializing the control platform, kubelet should start normally, and can be accessed through http://127.0.0.1:10248/healthz This endpoint checks the service status.

3. Initialize the cluster on the master node

3.1. Initialize containerd configuration

Detect using the file command /etc/containerd/config.toml Does the file exist?

shell

file /etc/containerd/config.toml

If the containerd configuration file has not been initialized, under normal circumstances it should output

txt

/etc/containerd/config.toml: cannot open `/etc/containerd/config.toml' (No such file or directory)

Initialize the default containerd configuration file

shell

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

Reminder

This operation will write the containerd default configuration information to /etc/containerd/config.toml File!

🍔 Check whether the CRI sandbox image version matches the containerd image version

View CRI sandbox image version information

toml

// other config ......
[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]
    device_ownership_from_security_context = false
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    drain_exec_sync_io_timeout = "0s"
    enable_cdi = false
    enable_selinux = false
    enable_tls_streaming = false
    enable_unprivileged_icmp = false
    enable_unprivileged_ports = false
    ignore_deprecation_warnings = []
    ignore_image_defined_volumes = false
    image_pull_progress_timeout = "5m0s"
    image_pull_with_sync_fs = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "registry.k8s.io/pause:3.8"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""
// other config ......

Run the following command to list the containerd images.

shell

ctr -n=k8s.io images ls -q

Will output containerd image list data similar to the following.

txt

docker.io/calico/cni:v3.26.0
docker.io/calico/kube-controllers:v3.26.0
docker.io/calico/node:v3.26.0
registry.k8s.io/coredns/coredns:v1.12.0
registry.k8s.io/etcd:3.5.21-0
registry.k8s.io/kube-apiserver:v1.33.0
registry.k8s.io/kube-controller-manager:v1.33.0
registry.k8s.io/kube-proxy:v1.33.0
registry.k8s.io/kube-scheduler:v1.33.0
registry.k8s.io/pause:3.10
sha256:1cf5f116067c67da67f97bff78c4bbc76913f59057c18627b96facaced73ea0b
sha256:1d579cb6d696709ea7c8613023cbc1204ac2af295477fe577c8fa741a76efa02
sha256:44f52c09dececf0d842450cfbdcf6f1ce1e6eaf2d7183d643b9fbf77dde03a38
sha256:45ae357729e3a6db7de47d4efb04453ac384d5cfec2f062a86523f3482cb1cdb
sha256:499038711c0816eda03a1ad96a8eb0440c005baa6949698223c6176b7f5077e1
sha256:5d6f5c26c655486ee59f7c517dbd383336f4ce2c0db77f7d5ffd015395deee6f
sha256:6ba9545b2183ef722d7e8a7f9e9c2abfaf483cd980bc378480631699413d9cf4
sha256:873ed75102791e5b0b8a7fcd41606c92fcec98d56d05ead4ac5131650004c136
sha256:8d72586a76469984dc4c5c7c36b24fbe4baed63056998c682f07b591d5e0aba4
sha256:f1184a0bd7fe53a4c7098147f250b1f8b287a0e4f8a4e1509ef1d06893267c68

You can see that the image version specified by the sandbox image and the one in the containerd image list pause The version doesn't match!

Need to take the containerd configuration file /etc/containerd/config.toml The sandbox image in is corrected to registry.k8s.io/pause:3.10, that is, the value must match an entry in the containerd image list pause Keep the versions consistent!

⚠️ Remember to restart containerd after making changes

shell

systemctl restart containerd

Warning

This should be a BUG in Kubernetes v1.33.0; during the process of downloading offline resources, by executing sudo kubeadm config images list --kubernetes-version=v1.33.0 obtained by the command pause The version is indeed 3.10, and now containerd The default configuration is 3.8 This is obviously problematic. If the sandbox image version is not confirmed, problems will arise during the control plane initialization operation!

3.2. Create the crictl.yaml configuration file

shell

cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: false
EOF

If this configuration file has not been created, after initializing the control plane, by executing crictl ps -a A warning message will be received when running the command.

txt

WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/usr/bin/crictl.yaml"
WARN[0000] runtime connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
WARN[0000] Image connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID              POD                               NAMESPACE
be6c9ad389276       f1184a0bd7fe5       38 seconds ago       Running             kube-proxy                1                   165cdc961d1bf       kube-proxy-4hg8c                  kube-system
5fc516e787ee8       f1184a0bd7fe5       39 seconds ago       Exited              kube-proxy                0                   08c97bca72201       kube-proxy-4hg8c                  kube-system
c988838d71433       1d579cb6d6967       About a minute ago   Running             kube-controller-manager   1                   324e2bf77651d       kube-controller-manager-testify   kube-system
975a48e4e3ef2       499038711c081       About a minute ago   Running             etcd                      1                   c9c4da99fc922       etcd-testify                      kube-system
143f7444e674c       6ba9545b2183e       About a minute ago   Running             kube-apiserver            1                   d001ac7718052       kube-apiserver-testify            kube-system
88dafa6cc3b63       8d72586a76469       About a minute ago   Running             kube-scheduler            1                   d16427a51cb45       kube-scheduler-testify            kube-system
8d4b1eb629996       6ba9545b2183e       About a minute ago   Exited              kube-apiserver            0                   4ae4675ab1514       kube-apiserver-testify            kube-system
687077efcc7bc       8d72586a76469       About a minute ago   Exited              kube-scheduler            0                   da34d839bc256       kube-scheduler-testify            kube-system
cf06d312b77dc       1d579cb6d6967       About a minute ago   Exited              kube-controller-manager   0                   67ccddd02f0a9       kube-controller-manager-testify   kube-system
ae9f3da58d14d       499038711c081       About a minute ago   Exited              etcd                      0                   c7afacdc52760       etcd-testify                      kube-system

3.3. Initialize the control plane

Set a temporary node IP variable to make it easier to specify the initialization parameters below.

shell

node_address=<master node IP>

Initialize the control plane using kubeadm

shell

sudo kubeadm init \
  --kubernetes-version v1.33.0 \
  --pod-network-cidr=192.168.0.0/16 \
  --apiserver-advertise-address=$node_address \
  --control-plane-endpoint=$node_address

Click to view more related information

For example, your control plane primary node IP is 192.168.0.160, then after replacing the above command, it should be

shell

node_address=192.168.0.160

The kubeadm command to initialize the control plane is

shell

sudo kubeadm init \
  --kubernetes-version v1.33.0 \
  --pod-network-cidr=192.168.0.0/16 \
  --apiserver-advertise-address=$node_address \
  --control-plane-endpoint=$node_address

There may be an issue where the sandbox image used by the container runtime is inconsistent with the sandbox image used by kubeadm, specifically as follows:

txt

[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
W0617 16:13:21.203208   47633 checks.go:846] detected that the sandbox image "registry.k8s.io/pause:3.8" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.

If you encounter this issue, you can refer to Resolve the inconsistency between the sandbox image used by the container runtime and the sandbox image used by kubeadm. Refer to the specific instructions in the relevant documentation. After resolving the issue, run the above initialization command again to verify the operation.

You may also encounter the issue "cgroups v1 support is in maintenance mode, please migrate to cgroups v2"; you can refer to Resolve the issue where kubeadm initialization prompts cgroups v1 support is in maintenance mode, please migrate to cgroups v2. Specific instructions in the relevant documents!

You may also encounter A control plane component may have crashed or exited when started by the container runtime. The issue of: To troubleshoot, list all containers using your preferred container runtimes CLI.

The following is an example of using crictl to list all running Kubernetes containers:

shell

crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause

Analyze and troubleshoot issues based on the logs. The following is a possible issue output.

txt

WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/usr/bin/crictl.yaml"

Set up kubectl configuration

shell

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Install the network plugin (Calico) on the master node.

The Kubernetes V1.33.0 offline installation resource package provides an offline Calico configuration file.

shell

kubectl apply -f /opt/k8s-install/calico.yaml --validate=false

Reminder

The above operations can only succeed after the kube-apiserver Pod has started! You can via crictl ps View kube-apiserver Is the status whether Running, for Running Execute the above command when!

If the node can connect to the external network, online internet resources can be used

shell

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/calico.yaml

View node status

shell

kubectl get nodes

Will output node status data information similar to the following.

txt

NAME      STATUS     ROLES           AGE     VERSION
testify   NotReady   control-plane   5m10s   v1.33.0

Fully reset the cluster. Click to view more related instructions.

Reset cluster

shell

sudo kubeadm reset -f
sudo rm -rf /etc/kubernetes/ /var/lib/etcd/ /etc/cni/net.d/ ~/.kube/

Reinitialize Execute again Initialize the cluster on the primary node. Relevant operations to fix the issue

Okay, I'll supplement that for you.Join worker node and Verify cluster status The complete content of the two chapters:

markdown

### 4. Join Worker Nodes

After the control plane is successfully initialized on the master node, kubeadm will output a join command similar to the following:

```txt
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join 192.168.0.160:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:7e5a3f9c2b8d1a6e4f7c3b9a2d5e8f1c4a7b2d9e6f3c8a1b5e7d9f2c4a6b8e0f \
    --control-plane

Then you can join any number of worker nodes by running the following on each as root:

  kubeadm join 192.168.0.160:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:7e5a3f9c2b8d1a6e4f7c3b9a2d5e8f1c4a7b2d9e6f3c8a1b5e7d9f2c4a6b8e0f

4.1. Run the join command on the worker node

bash

sudo kubeadm join 192.168.0.160:6443 --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:7e5a3f9c2b8d1a6e4f7c3b9a2d5e8f1c4a7b2d9e6f3c8a1b5e7d9f2c4a6b8e0f

[!tip] Tip If you forget or lose the join command output during initialization, you can regenerate it on the master node:

bash

# View the existing token list
sudo kubeadm token list

# If the token has expired or does not exist, create a new token
sudo kubeadm token create

# Get the hash value of the CA certificate
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

# Complete join command (can be used in combination)
kubeadm token create --print-join-command

4.2. Verify that the worker node joins successfully

After executing the join command on the worker node, the expected output is as follows:

txt

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[preflight] Activating the kubelet service
[post-start] The kubelet is now running

4.3. Confirm node join on the master node

bash

kubectl get nodes

Expected output:

txt

NAME     STATUS     ROLES           AGE     VERSION
master   NotReady   control-plane   10m     v1.33.0
node1    NotReady   <none>          30s     v1.33.0
node2    NotReady   <none>          30s     v1.33.0

[!warning] Caution At this point, the node's STATUS is still NotReady, this is normal, because the network plugin (Calico) is configuring the Pod network. Wait 1-2 minutes and check again, STATUS should become Ready.

5. Verify cluster status

5.1. View all node statuses

bash

kubectl get nodes -o wide

Expected output (all nodes are Ready Status):

txt

NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP    OS-IMAGE             KERNEL-VERSION
master   Ready    control-plane   15m   v1.33.0   192.168.0.160  Ubuntu 22.04.3 LTS   5.15.0-91-generic
node1    Ready    <none>          5m    v1.33.0   192.168.0.161  Ubuntu 22.04.3 LTS   5.15.0-91-generic
node2    Ready    <none>          5m    v1.33.0   192.168.0.162  Ubuntu 22.04.3 LTS   5.15.0-91-generic

5.2. View the status of all Pods (especially in the kube-system namespace)

bash

kubectl get pods -n kube-system

Expected output (all Pod statuses should be Running):

txt

NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-7d9b5b5b5f-xyz12   1/1     Running   0          12m
calico-node-abc12                          1/1     Running   0          12m
calico-node-def34                          1/1     Running   0          10m
calico-node-ghi56                          1/1     Running   0          10m
coredns-5d4d8b9c9f-abc12                   1/1     Running   0          15m
coredns-5d4d8b9c9f-def34                   1/1     Running   0          15m
etcd-master                                1/1     Running   0          15m
kube-apiserver-master                      1/1     Running   0          15m
kube-controller-manager-master             1/1     Running   0          15m
kube-proxy-abc12                           1/1     Running   0          15m
kube-proxy-def34                           1/1     Running   0          5m
kube-proxy-ghi56                           1/1     Running   0          5m
kube-scheduler-master                      1/1     Running   0          15m

5.3. View Pods in all namespaces

bash

kubectl get pods --all-namespaces

5.4. Check cluster health status

bash

kubectl get cs

Expected output:

txt

NAME                 STATUS    MESSAGE   ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   ok

[!tip] Hint In newer versions of Kubernetes kubectl get cs May return unhealthy, this is a known issue and does not affect normal cluster operation. It is recommended to use kubectl get componentstatuses Or directly check the core Pod status.

5.5. Check kubelet service status (on all nodes)

bash

systemctl status kubelet

The expected output should include active (running).

5.6. Verify that DNS resolution is working properly

Create a temporary Pod for testing DNS:

bash

kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup kubernetes.default

Expected output:

txt

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1

5.7. Verify Pod network communication

Create a test Nginx Pod:

bash

kubectl create deployment nginx --image=nginx:alpine
kubectl expose deployment nginx --port=80 --type=ClusterIP
kubectl get svc nginx

Expected output:

txt

NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
nginx   ClusterIP   10.96.123.45    <none>        80/TCP    10s

5.8. View cluster resource usage (if metrics-server is installed)

bash

kubectl top nodes
kubectl top pods --all-namespaces

[!note] Description If metrics-server is not installed,kubectl top The command will report an error. This is normal; metrics-server needs to be installed separately.

6. Installation Verification Checklist

Check item	Verification command	Expected results
Node status	`kubectl get nodes`	All nodes `Ready`
Core Pods	`kubectl get pods -n kube-system`	All Pods `Running`
DNS resolution	`nslookup kubernetes.default`	Return the correct ClusterIP
Pod network	Create test Nginx Pod	The Pod can start and be accessed normally.
kubelet service	`systemctl status kubelet`	`active (running)`

7. Troubleshooting Common Issues

If the node status remains NotReady, troubleshoot in the following order:

Check whether the network plugin is installed properly.

bash

kubectl get pods -n kube-system | grep calico

View node details

bash

kubectl describe node <node-name>

View kubelet logs

bash

journalctl -u kubelet -f --lines=50

Check the CNI configuration

bash

ls /etc/cni/net.d/

Check container runtime status

bash

systemctl status containerd
ctr -n=k8s.io containers ls

At this point, the Kubernetes v1.33.0 offline three-node cluster has been installed successfully! 🎉

Detailed Guide to Offline Installation of a Three-Node Kubernetes v1.33.0 Cluster in an Ubuntu 22.04 Environment ​

Preparatory work ​

1. Three servers ​

2. SSH passwordless login ​

3. Modify kernel parameters ​

4. Configure time synchronization ​

5. Install IPVS ​

6. Configure an apt-get proxy (if network access is restricted) ​

Phase 1: Prepare the offline installation package in a networked environment ​

1. Set up the download environment ​

2. Add the Kubernetes apt repository ​

3. Download Kubernetes components ​

4. Download container runtime components ​

5. Download the network plugin (Calico) configuration file ​

6. Download the container image ​

7. Prepare the installation script ​

8. Package all files ​

Phase 2: Install the cluster in an offline environment ​

1. Transfer files to all nodes ​

2. Perform the basic installation on all nodes. ​

3. Initialize the cluster on the master node ​

3.1. Initialize containerd configuration ​

3.2. Create the crictl.yaml configuration file ​

3.3. Initialize the control plane ​

4.1. Run the join command on the worker node ​

4.2. Verify that the worker node joins successfully ​

4.3. Confirm node join on the master node ​

5. Verify cluster status ​

5.1. View all node statuses ​

5.2. View the status of all Pods (especially in the kube-system namespace) ​

5.3. View Pods in all namespaces ​

5.4. Check cluster health status ​

5.5. Check kubelet service status (on all nodes) ​

5.6. Verify that DNS resolution is working properly ​

5.7. Verify Pod network communication ​

5.8. View cluster resource usage (if metrics-server is installed) ​

6. Installation Verification Checklist ​

7. Troubleshooting Common Issues ​

Detailed Guide to Offline Installation of a Three-Node Kubernetes v1.33.0 Cluster in an Ubuntu 22.04 Environment

Preparatory work

1. Three servers

2. SSH passwordless login

3. Modify kernel parameters

4. Configure time synchronization

5. Install IPVS

6. Configure an apt-get proxy (if network access is restricted)

Phase 1: Prepare the offline installation package in a networked environment

1. Set up the download environment

2. Add the Kubernetes apt repository

3. Download Kubernetes components

4. Download container runtime components

5. Download the network plugin (Calico) configuration file

6. Download the container image

7. Prepare the installation script

8. Package all files

Phase 2: Install the cluster in an offline environment

1. Transfer files to all nodes

2. Perform the basic installation on all nodes.

3. Initialize the cluster on the master node

3.1. Initialize containerd configuration

3.2. Create the crictl.yaml configuration file

3.3. Initialize the control plane

4.1. Run the join command on the worker node

4.2. Verify that the worker node joins successfully

4.3. Confirm node join on the master node

5. Verify cluster status

5.1. View all node statuses

5.2. View the status of all Pods (especially in the kube-system namespace)

5.3. View Pods in all namespaces

5.4. Check cluster health status

5.5. Check kubelet service status (on all nodes)

5.6. Verify that DNS resolution is working properly

5.7. Verify Pod network communication

5.8. View cluster resource usage (if metrics-server is installed)

6. Installation Verification Checklist

7. Troubleshooting Common Issues