Kubernetes kubeadm 生产部署实战¶

作者：JiangChong | 撰写时间：2026年08月

系列定位：Kubernetes 二进制部署实战让你「懂 K8s」（每个组件手工起、证书手签）；本文用 kubeadm 让你「会生产」——官方推荐的生产部署方式，同一套 6 台阿里云环境，全程对照 Kubernetes 二进制部署实战的坑。

0 本文概览¶

《Kubernetes 二进制部署实战》二进制部署踩平的坑，在 kubeadm 下大部分自动解决：

Kubernetes 二进制部署实战手动做的事	kubeadm 自动完成
cfssl 手签 7 套证书	`kubeadm init` 自动生成 + `certs renew` 自动续期
手写 6 个 systemd service	控制面组件以静态 Pod 运行（kubelet 管理）
手动补 RBAC 绑定（v1.36 最小权限）	自动创建
手写 5 个 kubeconfig	自动生成到 `/etc/kubernetes/`
手动改 kubeconfig 指向 LB 入口	`--control-plane-endpoint` 一步完成

但《Kubernetes 二进制部署实战》的环境层经验全部复用：containerd 配置（SystemdCgroup / cni bin_dir / sandbox_image）、内核模块（br_netfilter / ip_vs）、SLB 健康检查。

前置阅读：Kubernetes 二进制部署实战（理解每个组件的含义后再看 kubeadm 的黑盒才有意义）、Kubernetes 核心架构。

1. 环境规划（3 control-plane + 3 worker + SLB）¶

1.1 机器规划¶

6 台阿里云 ECS（Debian 13.6，2核4G，40GiB 系统盘），全部同一 VPC：

序号	角色	私网IP	主机名	组件
1	control-plane-1	172.20.65.80	k8s-cp1 `sudo hostnamectl set-hostname k8s-cp1`	kube-apiserver / kube-controller-manager / kube-scheduler / etcd（stacked）
2	control-plane-2	172.20.65.79	k8s-cp2 `sudo hostnamectl set-hostname k8s-cp2`	同上
3	control-plane-3	172.20.65.82	k8s-cp3 `sudo hostnamectl set-hostname k8s-cp3`	同上
4	worker-1	172.20.65.81	k8s-w1 `sudo hostnamectl set-hostname k8s-w1`	kubelet / kube-proxy / containerd
5	worker-2	172.20.65.78	k8s-w2 `sudo hostnamectl set-hostname k8s-w2`	同上
6	worker-3	172.20.65.83	k8s-w3 `sudo hostnamectl set-hostname k8s-w3`	同上

为什么 3 个 control-plane：kubeadm 的 etcd 与控制面同机部署（stacked etcd）——3 个控制面 = etcd 3 节点 = quorum 2/3，挂 1 台集群照常（这是生产底线，2 个控制面挂 1 台就完）。 3 个 worker 是为后续 Vertica 部署预留容量（本系列 Vertica on Kubernetes 部署指南 / 基于 MinIO 对象存储的 Vertica Eon on K8s 部署实战会基于此环境）。

1.2 网络规划¶

网段	值	说明
Pod 网段	`10.244.0.0/16`	与《Kubernetes 二进制部署实战》一致（Flannel）
Service 网段	`10.0.0.0/24`	与《Kubernetes 二进制部署实战》一致
集群入口	SLB 内网 VIP（如 172.20.65.75）	TCP 6443，后端挂 3 台 control-plane

为什么用 SLB 不用 nginx/keepalived：《Kubernetes 二进制部署实战》 9.x 已验证——阿里云 VPC 不支持 keepalived（VRRP 隔离）。SLB 托管、后端自动摘除，高可用完全交给云平台。

注意两点：

① 服务器组关闭「客户端地址保持」（开启时后端访问自己 VIP 会连接超时，实测 curl 69 秒）；

② 关闭后仍可能出现 TLS 奇异性问题（NLB TCP 转发改动数据流导致 RSA 签名校验失败），控制面节点本机 kubectl 建议直连节点 IP（见 3.3 节），其他节点经 VIP 访问不受影响。

1.3 操作系统初始化（6 台全部执行）¶

# ① 主机名（⚠️ cloud-init 会覆盖，先改配置文件）
sudo sed -i '/^preserve_hostname:/d' /etc/cloud/cloud.cfg
echo "preserve_hostname: true" >> /etc/cloud/cloud.cfg
sudo hostnamectl set-hostname k8s-cp1    # 各台换成自己的

# ② 关闭 swap
swapoff -a
sed -i 's/^\(.*[[:space:]]swap[[:space:]].*\)/#\1/' /etc/fstab

# ③ hosts（6 台一致）
cat >> /etc/hosts <<EOF
172.20.65.80 k8s-cp1
172.20.65.79 k8s-cp2
172.20.65.82 k8s-cp3
172.20.65.81 k8s-w1
172.20.65.78 k8s-w2
172.20.65.83 k8s-w3
EOF

# ④ 内核模块 + 持久化（《Kubernetes 二进制部署实战》 4.2.6 的完整版，含 ipvs）
# 网络模块先行（br_netfilter 依赖 bridge，Flannel vxlan 必需，失败则后续全挂）
modprobe overlay br_netfilter nf_conntrack
lsmod | grep br_netfilter    # ⚠️ 必须确认已加载，空输出 = 失败，先单独 modprobe br_netfilter
# IPVS 模块后加载（内核可能未编译 ip_vs_sh，失败可忽略）
modprobe ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh 2>/dev/null || true
cat > /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
nf_conntrack
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
EOF

# ⑤ sysctl
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system

# ⑥ 时间同步
timedatectl set-ntp true

2. 前置准备：containerd + kubeadm 工具¶

2.1 安装 containerd（6 台）¶

apt update && apt install -y containerd iptables ipset

# 生成默认配置
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

# SystemdCgroup（与 kubelet 一致）
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

# ⚠️ Debian 坑：cni bin_dir 改为 /opt/cni/bin（《Kubernetes 二进制部署实战》 7.1 实测，否则 Pod 网络全挂）
sed -i 's|bin_dir = "/usr/lib/cni"|bin_dir = "/opt/cni/bin"|' /etc/containerd/config.toml

# 国内镜像加速（Docker Hub 用 DaoCloud）
cat >> /etc/containerd/config.toml <<EOF

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
  endpoint = ["https://docker.m.daocloud.io"]
EOF

# ⚠️ pause 沙箱镜像改阿里云（containerd 默认 registry.k8s.io/pause:3.8 国内超时）
# kubeadm init --image-repository 只覆盖控制面组件（apiserver/etcd），管不到 pause
# 用正则匹配任意 pause 版本（containerd 1.x 默认 3.8，2.x 默认 3.10，均可命中）
sed -i 's|sandbox_image = "registry.k8s.io/pause:[^"]*"|sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.10.2"|' /etc/containerd/config.toml

systemctl restart containerd && systemctl enable containerd

# 验证（没有报错即正常）
containerd --version && grep sandbox_image /etc/containerd/config.toml

pause 镜像为什么单独处理：kubeadm init --image-repository 管的是 apiserver/etcd 等控制面组件，pause 沙箱镜像由 containerd 的 sandbox_image 参数控制，默认指向 registry.k8s.io/pause:3.8。每个 Pod 启动前必须先创建 pause 沙箱，pause 拉不下来 = 所有 Pod 卡死 = apiserver 永远起不来 = NLB 建联失败 + context deadline exceeded。这里把 sandbox_image 改为与 3.2 节 --image-repository 同一个阿里云源，保证可达。

2.2 安装 kubeadm / kubelet / kubectl（6 台，v1.36.2）¶

# 清华 TUNA 镜像（pkgs.k8s.io 官方源国内直连不稳定；阿里云 kubernetes-xenial 旧源停更于 v1.28，无 v1.36）
apt install -y gnupg                        # gpg 命令（Debian 最小化安装不自带）
mkdir -p /etc/apt/keyrings                  # 密钥目录（默认不存在）
curl -fsSL https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.36/deb/Release.key \
  | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.36/deb/ /" \
  | tee /etc/apt/sources.list.d/kubernetes.list

apt update
apt install -y "kubelet=1.36.2-*" "kubeadm=1.36.2-*" "kubectl=1.36.2-*" cri-tools
apt-mark hold kubelet kubeadm kubectl   # 防止意外升级

systemctl enable --now kubelet          # kubelet 先注册为服务（此时未 init，处于 CrashLoop 属正常）

版本钉死：apt-mark hold 三个包——kubeadm 的升级是显式演练（第 8 节），不允许 apt 自动升。

2.3 验证¶

kubeadm version      # v1.36.2
kubectl version --client

# 验证 containerd 运行时连通（crictl 输出 runtime 信息无报错即正常）
crictl info 2>&1 | head -5

3. kubeadm init 首个控制面（k8s-cp1）¶

3.1 先建 SLB¶

阿里云控制台：创建内网 SLB（NLB） → 监听 TCP 6443 → 后端挂 k8s-cp1/2/3（172.20.65.80/79/82:6443）→ 健康检查 TCP 6443（间隔 5s）。服务器组必须关闭「客户端地址保持」（开启时后端访问自己 VIP 会环路丢包，实测 curl 超时 69 秒）。记下 SLB 内网 VIP（示例 172.20.65.75）。

⚠️ 健康检查源 IP 放行：SLB 健康检查源是 VPC 内特殊网段（如 100.64.0.0/10），3 台 control-plane 的安全组入方向需放行该网段访问 6443，否则后端永远"异常"（《Kubernetes 二进制部署实战》实测）。

3.2 执行 init¶

# k8s-cp1 上
kubeadm init \
  --control-plane-endpoint=172.20.65.75:6443 \
  --image-repository=registry.aliyuncs.com/google_containers \
  --kubernetes-version=v1.36.2 \
  --pod-network-cidr=10.244.0.0/16 \
  --service-cidr=10.0.0.0/24 \
  --apiserver-cert-extra-sans=172.20.65.75 \
  --upload-certs

关键参数含义：

参数	作用
`--control-plane-endpoint`	集群统一入口（SLB VIP）——所有 kubeconfig 的 server 自动指向它，worker 和控制面 join 都用它
`--image-repository`	镜像源指向阿里云（绕开 registry.k8s.io 直连超时）
`--apiserver-cert-extra-sans`	证书 SAN 自动补 SLB VIP（《Kubernetes 二进制部署实战》手动改 csr 的坑，这里一个参数）
`--pod-network-cidr` / `--service-cidr`	与《Kubernetes 二进制部署实战》一致的网段规划
`--upload-certs`	把控制面证书加密上传，后续 join 控制面时自动分发（无需手动 scp）

典型输出：

root@k8s-cp1:~# kubeadm init \
  --control-plane-endpoint=172.20.65.75:6443 \
  --image-repository=registry.aliyuncs.com/google_containers \
  --kubernetes-version=v1.36.2 \
  --pod-network-cidr=10.244.0.0/16 \
  --service-cidr=10.0.0.0/24 \
  --apiserver-cert-extra-sans=172.20.65.75 \
  --upload-certs

[init] Using Kubernetes version: v1.36.2
[preflight] Running pre-flight checks
E0810 14:58:08.993645    2499 remote_runtime.go:1177] "RuntimeConfig from runtime service failed" err="rpc error: code = Unimplemented desc = method RuntimeConfig not implemented"
        [WARNING ContainerRuntimeVersion]: You must update your container runtime to a version that supports the CRI method RuntimeConfig. Falling back to using cgroupDriver from kubelet config will be removed in 1.37. For more information, see https://git.k8s.io/enhancements/keps/sig-node/4033-group-driver-detection-over-cri
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-cp1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.0.0.1 172.20.65.80 172.20.65.75]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-cp1 localhost] and IPs [172.20.65.80 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-cp1 localhost] and IPs [172.20.65.80 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.537517ms
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://172.20.65.80:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-controller-manager is healthy after 21.585387ms
[control-plane-check] kube-scheduler is healthy after 30.380315ms
[control-plane-check] kube-apiserver is healthy after 2.501879146s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
188f74f3502cbe19769469f4d0ab955ccc61cf13acee7d64f122e2e3c99ec9cc
[mark-control-plane] Marking the node k8s-cp1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-cp1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: ychkln.lh1qk3x7efxnzcf5
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Configured RBAC rules to allow the API server kubelet client certificate to access the kubelet API
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes running the following command on each as root:

  kubeadm join 172.20.65.75:6443 --token ychkln.lh1qk3x7efxnzcf5 \
        --discovery-token-ca-cert-hash sha256:17eeff907ae22d89655470a3c85b5e5d3532288494d7fe6eb2d6b6f9e62814b7 \
        --control-plane --certificate-key 188f74f3502cbe19769469f4d0ab955ccc61cf13acee7d64f122e2e3c99ec9cc

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.20.65.75:6443 --token ychkln.lh1qk3x7efxnzcf5 \
        --discovery-token-ca-cert-hash sha256:17eeff907ae22d89655470a3c85b5e5d3532288494d7fe6eb2d6b6f9e62814b7

3.3 init 后的三件事（k8s-cp1）¶

# ① kubectl 配置
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

⚠️ cp1 本机 kubectl 可能报 TLS 错误：NLB 四层 TCP 转发在 control-plane 节点「自己连自己」时会破坏 TLS 握手（RSA 签名校验失败：crypto/rsa: verification error，非通常的自签证书报错）。直连本机 IP 或 127.0.0.1 绕开 NLB 即可——cp1 是 apiserver 所在节点，直连不依赖负载均衡；从 worker 或运维机访问集群仍用 VIP 版 admin.conf，不受影响。 controller-manager / scheduler 虽同属 control-plane 节点，但 kubeadm 生成的 static Pod manifest 中 --master 参数指向本机 IP + 本地 CA，不经过 NLB，因此不受影响。

root@k8s-cp1:~# export KUBECONFIG=/etc/kubernetes/admin.conf
root@k8s-cp1:~# kubectl get nodes
# NAME      STATUS     ROLES           AGE     VERSION
# k8s-cp1   NotReady   control-plane   6m53s   v1.36.2

# 如果 kubectl get nodes 报 x509 / crypto/rsa 错误，改指本机 IP：
sed -i 's|server: https://172.20.65.75:6443|server: https://172.20.65.80:6443|' $HOME/.kube/config

# ② join 命令：init 输出末尾已直接给出 worker 版和控制面版（含 --control-plane --certificate-key），
#    直接保存即可。证书密钥 2 小时后过期，届时如需重新生成：
#    kubeadm token create --print-join-command
#    kubeadm init phase upload-certs --upload-certs

3.4 验证¶

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get nodes
# k8s-cp1   NotReady（CNI 未装，正常）
kubectl get pods -n kube-system -o wide
# kube-apiserver / kube-controller-manager / kube-scheduler / etcd 均 Running（静态 Pod）

4. kubeadm join：加入其余 2 个控制面¶

对比《Kubernetes 二进制部署实战》 9.1：不需要 scp 证书/二进制/配置——--upload-certs 的证书密钥 + join 命令一步完成。

4.1 在 k8s-cp2 / k8s-cp3 上执行¶

# 用 3.3 保存的 join 命令（控制面版，含 --control-plane --certificate-key）：
kubeadm join 172.20.65.75:6443 --token xxxxx \
  --discovery-token-ca-cert-hash sha256:xxxxx \
  --control-plane --certificate-key xxxxx

⚠️ 如果 join 报 TLS 错误（同 3.3 节的 crypto/rsa: verification error——cp2/cp3 join 时经 NLB 连 cp1，同属 NLB TCP 转发的 TLS 奇异性问题）：把 VIP 换成 cp1 本机 IP（172.20.65.80）直连。kubeadm 会从集群 ConfigMap 自动读取 control-plane-endpoint，生成的 kubeconfig 仍正确指向 VIP，不影响后续。

如果 join 报 kubelet.conf already exists / Port 10250 is in use（上次失败残留），执行后再重试：
kubeadm reset -f && rm -rf /etc/kubernetes/ /var/lib/kubelet/ /var/lib/etcd/
systemctl restart kubelet

4.2 验证（k8s-cp1）¶

kubectl get nodes
# 三个 control-plane 都出现（NotReady 正常）
kubectl get pods -n kube-system -o wide | grep etcd
# etcd 3 个 Pod 各在 3 台控制面上——stacked etcd 就位

5. kubeadm join：加入 3 个 worker¶

5.1 在 k8s-w1 / w2 / w3 上执行¶

# 用 3.3 保存的 worker 版 join 命令（不带 --control-plane）：
kubeadm join 172.20.65.75:6443 --token xxxxx \
  --discovery-token-ca-cert-hash sha256:xxxxx

worker 不是 NLB 后端，kubelet.conf 保持指向 VIP 即可利用 SLB 高可用：任何 control-plane 挂了，worker 的 kubelet 自动连到其他后端。

如果 join 报 kubelet.conf already exists / Port 10250 is in use（上次失败残留）：
kubeadm reset -f && rm -rf /etc/kubernetes/ /var/lib/kubelet/ && systemctl restart kubelet

5.2 验证¶

kubectl get nodes -o wide
# 6 台全部出现，STATUS 都是 NotReady（还差 CNI，下一节）

6. 部署 CNI（Flannel）¶

与《Kubernetes 二进制部署实战》 7.6 完全相同的套路（镜像源经验直接复用）：

6.1 下载配置（k8s-cp1）¶

wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
grep -A2 '"Network"' kube-flannel.yml    # 应为 10.244.0.0/16（与 --pod-network-cidr 一致）

# ⚠️ 国内镜像源：flannel 镜像在 ghcr.io（DaoCloud 403），改走南大镜像站
sed -i 's|ghcr.io/flannel-io/|ghcr.nju.edu.cn/flannel-io/|g' kube-flannel.yml
grep -E "image:" kube-flannel.yml | head -5

kubectl apply -f kube-flannel.yml
# namespace/kube-flannel created
# clusterrole.rbac.authorization.k8s.io/flannel created
# clusterrolebinding.rbac.authorization.k8s.io/flannel created
# serviceaccount/flannel created
# configmap/kube-flannel-cfg created
# daemonset.apps/kube-flannel-ds created

6.2 等待并验证¶

kubectl get pods -n kube-flannel -o wide
# 6 台各一个 flannel Pod，Running 即成功（首次拉镜像 1-3 分钟）
# NAME                    READY   STATUS    RESTARTS        AGE     IP             NODE      NOMINATED NODE   READINESS GATES
# kube-flannel-ds-26vts   1/1     Running   6 (4m10s ago)   7m22s   172.20.65.82   k8s-cp3   <none>           <none>
# kube-flannel-ds-2z8pf   1/1     Running   6 (4m2s ago)    7m22s   172.20.65.79   k8s-cp2   <none>           <none>
# kube-flannel-ds-7nvwn   1/1     Running   6 (3m54s ago)   7m22s   172.20.65.78   k8s-w2    <none>           <none>
# kube-flannel-ds-qb82g   1/1     Running   6 (4m10s ago)   7m22s   172.20.65.80   k8s-cp1   <none>           <none>
# kube-flannel-ds-vlbdh   1/1     Running   6 (4m4s ago)    7m22s   172.20.65.83   k8s-w3    <none>           <none>
# kube-flannel-ds-w98jc   1/1     Running   6 (3m50s ago)   7m22s   172.20.65.81   k8s-w1    <none>           <none>

kubectl get nodes
# 全部 Ready
# NAME      STATUS   ROLES           AGE   VERSION
# k8s-cp1   Ready    control-plane   59m   v1.36.2
# k8s-cp2   Ready    control-plane   25m   v1.36.2
# k8s-cp3   Ready    control-plane   24m   v1.36.2
# k8s-w1    Ready    <none>          23m   v1.36.2
# k8s-w2    Ready    <none>          23m   v1.36.2
# k8s-w3    Ready    <none>          23m   v1.36.2

若 ImagePullBackOff：kubectl describe pod -n kube-flannel <名> 看是哪个镜像拉不动，参考《Kubernetes 二进制部署实战》 7.6.2 的多源拉取 + ctr -n k8s.io images tag 套路（南大源已内置在 yaml 里，一般不会再遇到）。

若 CrashLoopBackOff 且日志含 stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory：br_netfilter 内核模块未加载。所有节点 modprobe br_netfilter 后 Flannel 自动恢复（1.3 节已配置持久化，但当前会话需手动加载）。

6.3 功能验证（沿用《Kubernetes 二进制部署实战》的链路）¶

kubectl create deployment web --image=nginx
kubectl expose deployment web --port=80 --type=NodePort
# 使用 describe 看看日志，没报错就多等一会儿
kubectl get pods -o wide
# NAME                   READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
# web-7887448d46-5hz68   1/1     Running   0          6m30s   10.244.5.2   k8s-w3    <none>           <none>
kubectl get svc web
# 通过任一 worker 的 NodePort 访问验证跨节点转发
# NAME   TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
# web    NodePort   10.0.0.254   <none>        80:30611/TCP   3m29s

kubectl describe pod web-7887448d46-5hz68 | tail -10

Pod Running 过后，验证跨节点访问：

# ① 等 Pod Ready
kubectl get pods -w    # Ctrl+C 退出

# ② 通过任意节点 IP + NodePort 访问 nginx（30611 是你 svc 输出的端口，这个不固定，通过 kubectl get svc web 检查实际端口）
curl http://172.20.65.80:30611    # cp1
curl http://172.20.65.78:30611    # w2

Pod 调度到了 w3，但从 cp1 访问也能通——这就是 Flannel vxlan 跨节点转发。两边都有返回说明网络正常。

7. 证书生命周期（对比《Kubernetes 二进制部署实战》的手动 cfssl）¶

kubeadm 签发的证书默认 1 年，到期前自动续期靠两个机制：

证书类型	续期机制
kubelet 证书	自动轮换（kubelet 每 80% 生命周期自动申请新证书）
控制面证书（apiserver/etcd 等）	`kubeadm certs renew` 手动触发（不会自动）

7.1 查看证书有效期¶

kubeadm certs check-expiration
# CERTIFICATE                EXPIRES   RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
# admin.conf                 Aug 04    364d            ca                      no
# apiserver                  Aug 04    364d            ca                      no
# ...
# kubelet-client             Aug 04    364d            ca                      no

7.2 模拟续期（提前一年演练）¶

# ① 续期全部证书（生产：在到期前 1-2 个月执行）
kubeadm certs renew all

# ② 重启静态 Pod 使新证书生效（kubeadm 不自动重启）
systemctl restart kubelet
# 静态 Pod 由 kubelet 拉起，重启 kubelet 即可触发 kube-apiserver, kube-controller-manager, kube-scheduler, etcd 等重新加载证书

# ③ 更新 kubeconfig（admin.conf 等内嵌证书需要同步）
# kubeadm init 不会把配置写到本地磁盘，需从集群 ConfigMap 导出：
kubectl get configmap -n kube-system kubeadm-config \
  -o jsonpath='{.data.ClusterConfiguration}' > /etc/kubernetes/kubeadm-config.yaml
kubeadm init phase kubeconfig all --config /etc/kubernetes/kubeadm-config.yaml

# ④ 验证
kubeadm certs check-expiration    # 有效期刷新为 365d
kubectl get nodes                 # 集群正常

《Kubernetes 二进制部署实战》对照：手动 cfssl 重签 + scp 分发 + 逐个重启组件——kubeadm 一条 renew all 完成，这是生产可维护性的核心差异。

8. 升级演练（v1.36.2 → v1.36.3）¶

升级顺序铁律：先控制面、后 worker；控制面内部逐个节点升级（先 cp1，再 cp2，再 cp3）。 1.36.3 为演习目标版本。执行前 apt list -a kubeadm 确认最新版本号。

8.1 升级首个控制面（k8s-cp1）¶

# ① 解除 hold 并升级 kubeadm
apt-mark unhold kubeadm && apt update && apt install -y "kubeadm=1.36.3-*" && apt-mark hold kubeadm

# ② 预检（列出可升级的组件）
kubeadm upgrade plan

# ③ 升级控制面（⚠️ 仅 cp1 用 apply，cp2/cp3 用 upgrade node）
kubeadm upgrade apply v1.36.3

# ④ 升级 kubelet/kubectl 并重启
apt-mark unhold kubelet kubectl && apt install -y "kubelet=1.36.3-*" "kubectl=1.36.3-*" && apt-mark hold kubelet kubectl
systemctl restart kubelet

# ⑤ 确认升级完成
kubectl get nodes

升级过程典型输出：

root@k8s-cp1:~#
root@k8s-cp1:~# apt-mark unhold kubeadm && apt update && apt install -y "kubeadm=1.36.3-*" && apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 http://mirrors.cloud.aliyuncs.com/debian trixie InRelease
Hit:2 http://mirrors.cloud.aliyuncs.com/debian trixie-updates InRelease
Hit:3 http://mirrors.cloud.aliyuncs.com/debian trixie-backports InRelease
Hit:4 http://mirrors.cloud.aliyuncs.com/debian-security trixie-security InRelease
Hit:5 https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.36/deb  InRelease
20 packages can be upgraded. Run 'apt list --upgradable' to see them.
Selected version '1.36.3-1.1' (isv:kubernetes:core:stable:v1.36:mirrors.tuna.tsinghua.edu.cn [amd64]) for 'kubeadm'
Upgrading:
  kubeadm

Summary:
  Upgrading: 1, Installing: 0, Removing: 0, Not Upgrading: 19
  Download size: 12.6 MB
  Space needed: 0 B / 36.5 GB available

Get:1 https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.36/deb  kubeadm 1.36.3-1.1 [12.6 MB]
Fetched 12.6 MB in 2s (7,209 kB/s)
apt-listchanges: Reading changelogs...
(Reading database ... 57620 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.36.3-1.1_amd64.deb ...
Unpacking kubeadm (1.36.3-1.1) over (1.36.2-2.1) ...
Setting up kubeadm (1.36.3-1.1) ...
kubeadm set on hold.

root@k8s-cp1:~# kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/config] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.36.2
[upgrade/versions] kubeadm version: v1.36.3
[upgrade/versions] Target version: v1.36.3
[upgrade/versions] Latest version in the v1.36 series: v1.36.3

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE      CURRENT   TARGET
kubelet     k8s-cp1   v1.36.2   v1.36.3
kubelet     k8s-cp2   v1.36.2   v1.36.3
kubelet     k8s-cp3   v1.36.2   v1.36.3
kubelet     k8s-w1    v1.36.2   v1.36.3
kubelet     k8s-w2    v1.36.2   v1.36.3
kubelet     k8s-w3    v1.36.2   v1.36.3

Upgrade to the latest version in the v1.36 series:

COMPONENT                 NODE      CURRENT   TARGET
kube-apiserver            k8s-cp1   v1.36.2   v1.36.3
kube-apiserver            k8s-cp2   v1.36.2   v1.36.3
kube-apiserver            k8s-cp3   v1.36.2   v1.36.3
kube-controller-manager   k8s-cp1   v1.36.2   v1.36.3
kube-controller-manager   k8s-cp2   v1.36.2   v1.36.3
kube-controller-manager   k8s-cp3   v1.36.2   v1.36.3
kube-scheduler            k8s-cp1   v1.36.2   v1.36.3
kube-scheduler            k8s-cp2   v1.36.2   v1.36.3
kube-scheduler            k8s-cp3   v1.36.2   v1.36.3
kube-proxy                          1.36.2    v1.36.3
CoreDNS                             v1.14.2   v1.14.2
etcd                      k8s-cp1   3.6.8-0   3.6.8-0
etcd                      k8s-cp2   3.6.8-0   3.6.8-0
etcd                      k8s-cp3   3.6.8-0   3.6.8-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.36.3

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________


root@k8s-cp1:~# kubeadm upgrade apply v1.36.3
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[upgrade/preflight] Running preflight checks
E0810 15:22:37.219016    6867 remote_runtime.go:1177] "RuntimeConfig from runtime service failed" err="rpc error: code = Unimplemented desc = method RuntimeConfig not implemented"
        [WARNING ContainerRuntimeVersion]: You must update your container runtime to a version that supports the CRI method RuntimeConfig. Falling back to using cgroupDriver from kubelet config will be removed in 1.37. For more information, see https://git.k8s.io/enhancements/keps/sig-node/4033-group-driver-detection-over-cri
[upgrade] Running cluster health checks
[upgrade/preflight] You have chosen to upgrade the cluster version to "v1.36.3"
[upgrade/versions] Cluster version: v1.36.2
[upgrade/versions] kubeadm version: v1.36.3
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster
[upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection
[upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[upgrade/control-plane] Upgrading your static Pod-hosted control plane to version "v1.36.3" (timeout: 5m0s)...
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests3257216693"
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Restarting the etcd static pod and backing up its manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-08-10-15-23-02/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 3 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-08-10-15-23-02/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-08-10-15-23-02/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-08-10-15-23-02/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/control-plane] The control plane instance for this node was successfully upgraded!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded!
W0810 15:24:36.461994    6867 postupgrade.go:105] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config-2026-08-10-15-24-36 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config-2026-08-10-15-24-36/config.yaml
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/bootstrap-token] Configuring bootstrap token and cluster-info RBAC rules
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Configured RBAC rules to allow the API server kubelet client certificate to access the kubelet API
[upgrade/addon] Skipping upgrade of addons because control plane instances [k8s-cp2 k8s-cp3] have not been upgraded
[upgrade/addon] Skipping upgrade of addons because control plane instances [k8s-cp2 k8s-cp3] have not been upgraded

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.36.3".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.
root@k8s-cp1:~#


root@k8s-cp1:~# apt-mark unhold kubelet kubectl && apt install -y "kubelet=1.36.3-*" "kubectl=1.36.3-*" && apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Selected version '1.36.3-1.1' (isv:kubernetes:core:stable:v1.36:mirrors.tuna.tsinghua.edu.cn [amd64]) for 'kubelet'
Selected version '1.36.3-1.1' (isv:kubernetes:core:stable:v1.36:mirrors.tuna.tsinghua.edu.cn [amd64]) for 'kubectl'
Upgrading:
  kubectl  kubelet

Summary:
  Upgrading: 2, Installing: 0, Removing: 0, Not Upgrading: 17
  Download size: 25.2 MB
  Space needed: 4,096 B / 35.9 GB available

Get:1 https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.36/deb  kubectl 1.36.3-1.1 [11.8 MB]
Get:2 https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.36/deb  kubelet 1.36.3-1.1 [13.4 MB]
Fetched 25.2 MB in 3s (7,897 kB/s)
apt-listchanges: Reading changelogs...
(Reading database ... 60728 files and directories currently installed.)
Preparing to unpack .../kubectl_1.36.3-1.1_amd64.deb ...
Unpacking kubectl (1.36.3-1.1) over (1.36.2-2.1) ...
Preparing to unpack .../kubelet_1.36.3-1.1_amd64.deb ...
Unpacking kubelet (1.36.3-1.1) over (1.36.2-2.1) ...
Setting up kubectl (1.36.3-1.1) ...
Setting up kubelet (1.36.3-1.1) ...
kubelet set on hold.
kubectl set on hold.
root@k8s-cp1:~#


root@k8s-cp1:~# systemctl restart kubelet


root@k8s-cp1:~# kubectl get nodes
NAME      STATUS   ROLES           AGE   VERSION
k8s-cp1   Ready    control-plane   27m   v1.36.3
k8s-cp2   Ready    control-plane   15m   v1.36.2
k8s-cp3   Ready    control-plane   14m   v1.36.2
k8s-w1    Ready    <none>          13m   v1.36.2
k8s-w2    Ready    <none>          13m   v1.36.2
k8s-w3    Ready    <none>          13m   v1.36.2
root@k8s-cp1:~#
root@k8s-cp1:~#

8.2 升级其余控制面（k8s-cp2 / k8s-cp3，每台依次）¶

# 与 cp1 相同的前两步，但③用 upgrade node（apply 只能跑在第一个控制面）
apt-mark unhold kubeadm && apt update && apt install -y "kubeadm=1.36.3-*" && apt-mark hold kubeadm
kubeadm upgrade node
apt-mark unhold kubelet kubectl && apt install -y "kubelet=1.36.3-*" "kubectl=1.36.3-*" && apt-mark hold kubelet kubectl
systemctl restart kubelet

kubeadm upgrade apply 的 --certificate-renewal=true（默认）会在升级时顺带续期证书——升级和续期可以一起做。

8.3 升级 worker（每台 worker 依次执行）¶

apt-mark unhold kubeadm && apt update && apt install -y "kubeadm=1.36.3-*" && apt-mark hold kubeadm
kubeadm upgrade node
apt-mark unhold kubelet kubectl && apt install -y "kubelet=1.36.3-*" "kubectl=1.36.3-*" && apt-mark hold kubelet kubectl
systemctl restart kubelet

《Kubernetes 二进制部署实战》对照：手动下载新二进制 → 停服务 → 换文件 → 改配置 → 重启，且要自己处理证书——kubeadm 的 upgrade plan/apply 自动完成版本校验、证书续期、组件滚动。

Worker 升级典型输出：

root@k8s-w1:~# apt-mark unhold kubeadm && apt update && apt install -y "kubeadm=1.36.3-*" && apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 http://mirrors.cloud.aliyuncs.com/debian trixie InRelease
Hit:2 http://mirrors.cloud.aliyuncs.com/debian trixie-updates InRelease
Hit:3 http://mirrors.cloud.aliyuncs.com/debian trixie-backports InRelease
Hit:4 http://mirrors.cloud.aliyuncs.com/debian-security trixie-security InRelease
Hit:5 https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.36/deb  InRelease
20 packages can be upgraded. Run 'apt list --upgradable' to see them.
Selected version '1.36.3-1.1' (isv:kubernetes:core:stable:v1.36:mirrors.tuna.tsinghua.edu.cn [amd64]) for 'kubeadm'
Upgrading:
  kubeadm

Summary:
  Upgrading: 1, Installing: 0, Removing: 0, Not Upgrading: 19
  Download size: 12.6 MB
  Space needed: 0 B / 37.0 GB available

Get:1 https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.36/deb  kubeadm 1.36.3-1.1 [12.6 MB]
Fetched 12.6 MB in 3s (3,880 kB/s)
apt-listchanges: Reading changelogs...
(Reading database ... 57620 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.36.3-1.1_amd64.deb ...
Unpacking kubeadm (1.36.3-1.1) over (1.36.2-2.1) ...
Setting up kubeadm (1.36.3-1.1) ...
kubeadm set on hold.
root@k8s-w1:~#
root@k8s-w1:~#

root@k8s-w1:~# kubeadm upgrade node
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
W0810 15:35:55.564025   10402 utils.go:69] The recommended value for "bindAddress" in "KubeProxyConfiguration" is: ::; the provided value is: 0.0.0.0
[upgrade/preflight] Running pre-flight checks
E0810 15:35:55.649199   10402 remote_runtime.go:1177] "RuntimeConfig from runtime service failed" err="rpc error: code = Unimplemented desc = method RuntimeConfig not implemented"
        [WARNING ContainerRuntimeVersion]: You must update your container runtime to a version that supports the CRI method RuntimeConfig. Falling back to using cgroupDriver from kubelet config will be removed in 1.37. For more information, see https://git.k8s.io/enhancements/keps/sig-node/4033-group-driver-detection-over-cri
[upgrade/preflight] Skipping prepull. Not a control plane node.
[upgrade/control-plane] Skipping phase. Not a control plane node.
[upgrade/kubeconfig] Skipping phase. Not a control plane node.
W0810 15:35:55.650766   10402 postupgrade.go:105] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config-2026-08-10-15-35-55 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config-2026-08-10-15-35-55/config.yaml
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/addon] Skipping the addon/coredns phase. Not a control plane node.
[upgrade/addon] Skipping the addon/kube-proxy phase. Not a control plane node.

root@k8s-w1:~# apt-mark unhold kubelet kubectl && apt install -y "kubelet=1.36.3-*" "kubectl=1.36.3-*" && apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Selected version '1.36.3-1.1' (isv:kubernetes:core:stable:v1.36:mirrors.tuna.tsinghua.edu.cn [amd64]) for 'kubelet'
Selected version '1.36.3-1.1' (isv:kubernetes:core:stable:v1.36:mirrors.tuna.tsinghua.edu.cn [amd64]) for 'kubectl'
Upgrading:
  kubectl  kubelet

Summary:
  Upgrading: 2, Installing: 0, Removing: 0, Not Upgrading: 17
  Download size: 25.2 MB
  Space needed: 4,096 B / 37.0 GB available

Get:1 https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.36/deb  kubectl 1.36.3-1.1 [11.8 MB]
Get:2 https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.36/deb  kubelet 1.36.3-1.1 [13.4 MB]
Fetched 25.2 MB in 4s (6,352 kB/s)
apt-listchanges: Reading changelogs...
(Reading database ... 57620 files and directories currently installed.)
Preparing to unpack .../kubectl_1.36.3-1.1_amd64.deb ...
Unpacking kubectl (1.36.3-1.1) over (1.36.2-2.1) ...
Preparing to unpack .../kubelet_1.36.3-1.1_amd64.deb ...
Unpacking kubelet (1.36.3-1.1) over (1.36.2-2.1) ...
Setting up kubectl (1.36.3-1.1) ...
Setting up kubelet (1.36.3-1.1) ...
kubelet set on hold.
kubectl set on hold.

root@k8s-w1:~# systemctl restart kubelet
root@k8s-w1:~#

9. 高可用验证（生产演练）¶

9.1 日常验证¶

# ① 通过 SLB 入口访问（任意机器）
curl -k https://172.20.65.75:6443/healthz
# ok

# ② 控制面组件健康（kubeadm 集群无 ComponentStatus 问题，直接看 Pod）
kubectl get pods -n kube-system -o wide

9.2 故障演练：关机一个 control-plane¶

# ① 在阿里云控制台停止 k8s-cp2（模拟宕机）

# ② 验证集群不受影响
kubectl get nodes          # cp2 NotReady，其余正常
kubectl get pods -A -o wide   # 业务 Pod 无感知
curl -k https://172.20.65.75:6443/healthz   # SLB 自动摘除 cp2，仍 ok

# ③ etcd quorum：3 节点挂 1 = 2/3，仍能选主（这就是 3 控制面的意义）
# ⚠️ 不要用 kubectl get pods 判断 etcd 状态——Pod 状态来自 APIServer 缓存快照，
#    节点宕机后 kubelet 无法上报新状态，Pod 会一直显示 Running（约 5 分钟后才被驱逐）
kubectl exec -n kube-system etcd-k8s-cp1 -- etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health --cluster
# 172.20.65.80 is healthy: successfully committed proposal
# 172.20.65.79 is unhealthy: context deadline exceeded  ← 已宕机
# 172.20.65.82 is healthy: successfully committed proposal
# Error: unhealthy cluster  ← 并非「集群挂了」，只是「不是 100% 健康」，2/3 集群正常工作

# ④ 恢复 k8s-cp2 后自动回归集群，无需操作
# 跟踪恢复过程：
#   cp2 本机: journalctl -u kubelet -f        （kubelet 拉起 static Pod）
#   cp1:      kubectl get nodes -w             （NotReady → Ready）
#   cp1:      kubectl get pods -n kube-system -w | grep k8s-cp2  （Pod 逐个 Running）

⚠️ 演练注意：别一次停 2 台（etcd 1/3 失去 quorum，集群只读/不可用）。etcdctl 的 Error: unhealthy cluster 是预期输出——只要还有 2 个节点标记为 healthy，quorum 就满足，集群正常运行。生产环境 etcd 数据在控制面本地盘——控制面节点关机前确认有备份或接受重建成本（stacked etcd 的代价，与《Kubernetes 二进制部署实战》独立 etcd 的取舍）。

9.3 worker 故障演练¶

# 停止 k8s-w3 → 其上 Pod 被重新调度到其他 worker（有副本的 Deployment）
kubectl get pods -o wide -w   # -w 持续观察 Pod 迁移全过程
# NAME                   READY   STATUS              AGE   IP           NODE     NOMINATED NODE
# web-7887448d46-5hz68   1/1     Running             35m   10.244.5.2   k8s-w3   <none>
# web-7887448d46-5hz68   1/1     Terminating         38m   10.244.5.2   k8s-w3   <none>
# web-7887448d46-tqrp8   0/1     Pending             0s    <none>       <none>   <none>
# web-7887448d46-tqrp8   0/1     ContainerCreating   0s    <none>       k8s-w2   <none>
# web-7887448d46-tqrp8   1/1     Running             2m    10.244.4.2   k8s-w2   <none>

Pod 重新调度约需 5 分 40 秒：kubelet 心跳超时 ~40s → 节点标记 NotReady → 等待 Pod 默认 toleration 到期（node.kubernetes.io/not-ready:NoExecute 默认 300s）→ 自动驱逐并调度到健康的 worker。

10. 对照总结：kubeadm 自动解决了哪些《Kubernetes 二进制部署实战》的坑¶

Kubernetes 二进制部署实战踩过的坑	kubeadm 下的状态
证书手动生成/分发/SAN 遗漏	✅ 自动生成 + `--apiserver-cert-extra-sans` 一步加 SAN
WorkingDirectory / flag+env 冲突	✅ 组件是静态 Pod，无 systemd 配置
klog 旧 flag 移除（logtostderr 等）	✅ 静态 Pod 参数由 kubeadm 维护
kubeconfig server 127.0.0.1 假 Healthy	✅ kubeadm 统一指向 control-plane-endpoint
v1.36 RBAC 最小权限补绑定	✅ 自动创建
kubelet-api-admin 绑定	✅ 自动创建
镜像国内拉取（pause/flannel/coredns）	✅ `--image-repository` 阿里云源 + 南大 flannel 源
Debian cni bin_dir / 内核模块 / iptables 三件套	⚠️ 仍需手动（2.1/1.3 节，环境层 kubeadm 不管）
NLB 客户端地址保持	⚠️ 关闭即可消除超时（3.1 节），关闭后仍有 TLS 奇异性（3.3 节：控制面节点 kubectl 指本机 IP 绕开）
SLB 健康检查源 IP 放行	⚠️ 仍需手动（3.1 节）

结论：kubeadm 把「K8s 自身」的复杂度全部接管（证书、配置、RBAC、升级）；「环境层」的坑（内核、运行时、网络、云厂商限制）与部署方式无关——这正是 Kubernetes 二进制部署实战先二进制后 kubeadm 的学习路径价值：二进制让你能看懂 kubeadm 在做什么，遇到环境层问题知道去哪查。

kubeadm 是不是比二进制部署方便多了？

扩展阅读¶

Kubernetes 二进制部署实战 — 二进制逐组件部署（原理篇）
Kubernetes 核心架构 — 认识控制平面、工作节点、核心组件
Vertica on Kubernetes 部署指南 — 基于本环境部署 Vertica
基于 MinIO 对象存储的 Vertica Eon on K8s 部署实战 — 基于本环境部署 Eon 模式（MinIO 公共存储）
Kubernetes 官方文档 — kubeadm 生产部署官方指南