在 Ansible 主控机上创建以下目录结构并将所有文件放入对应位置。# 在 Ansible 主控机上执行mkdir -p k8s-binary-offline/{files,roles,group_vars,inventory} cd k8s-binary-offline tree .预期目录结构如下k8s-binary-offline/ ├── ansible.cfg ├── inventory │ ├── hosts.ini │ └── group_vars│ ├── all.yml │ ├── k8s_master.yml │ └── k8s_node.yml ├── files│ ├── binaries/ │ │ ├── kubernetes-server-linux-amd64.tar.gz │ │ ├── cilium-linux-amd64.tar.gz │ │ ├── etcd-v3.5.12-linux-amd64.tar.gz │ │ ├── cfssl_linux-amd64 │ │ ├── cfssljson_linux-amd64 │ │ └── cfssl-certinfo_linux-amd64 │ ├── systemd/ │ │ ├── kube-apiserver.service │ │ ├── kube-controller-manager.service │ │ ├── kube-scheduler.service │ │ ├── kubelet.service │ │ ├── kube-proxy.service │ │ └── etcd.service │ ├── configs/ │ │ ├── kubelet-config.yaml │ │ ├── kube-proxy-config.yaml │ │ ├── bootstrap-kubelet.conf │ │ └── cilium-config.yaml │ └── certs/ │ └── (证书生成脚本和模板) ├── roles │ ├── common │ │ └── tasks │ │ └── main.yml │ ├── certs │ │ └── tasks │ │ └── main.yml │ ├── etcd │ │ └── tasks │ │ └── main.yml │ ├── master │ │ └── tasks │ │ └── main.yml │ ├── node │ │ └── tasks │ │ └── main.yml │ └── cilium │ └── tasks │ └── main.yml └── site.yml二、核心配置文件与脚本1. Ansible 主配置文件 (ansible.cfg)[defaults] inventory ./inventory/hosts.ini host_key_checking False remote_user root roles_path ./roles retry_files_enabled False gathering smart fact_caching jsonfile fact_caching_connection /tmp/ansible_factsfact_caching_timeout 600[ssh_connection] pipelining True ssh_args -o ControlMasterauto -o ControlPersist30m -o ControlPath/tmp/ansible-ssh-%h-%p-%r2. Inventory 文件 (inventory/hosts.ini)[k8s_master] master01 ansible_host10.10.1.10 ansible_userrootmaster02 ansible_host10.10.1.11 ansible_userroot master03 ansible_host10.10.1.12 ansible_userroot [k8s_node] node01 ansible_host10.10.1.21 ansible_userroot node02 ansible_host10.10.1.22 ansible_userroot node03 ansible_host10.10.1.23 ansible_userroot node04 ansible_host10.10.1.24 ansible_userroot node05 ansible_host10.10.1.25 ansible_userroot node06 ansible_host10.10.1.26 ansible_userroot [k8s_cluster:children] k8s_master k8s_node [etcd] master01 ansible_host10.10.1.10 master02 ansible_host10.10.1.11 master03 ansible_host10.10.1.12 [all:vars] ansible_ssh_private_key_file /root/.ssh/id_rsa ansible_become yes ansible_become_method sudo ansible_become_user root3. 全局变量文件 (group_vars/all.yml)# 集群基础配置 k8s_version: v1.30.10 etcd_version: v3.5.12 cilium_version: v1.16.6 cluster_name: k8s-cluster service_cidr: 10.96.0.0/12 pod_cidr: 10.244.0.0/16 cluster_dns: 10.96.0.10 # 网络配置 apiserver_vip: 10.10.1.100 apiserver_port: 6443 # 文件路径 local_files_dir: /root/k8s-binary-offline/files remote_install_dir: /opt/kubernetes remote_bin_dir: {{ remote_install_dir }}/bin remote_cfg_dir: {{ remote_install_dir }}/cfg remote_ssl_dir: {{ remote_install_dir }}/ssl remote_systemd_dir: /etc/systemd/system # 证书配置 cert_country: CN cert_state: Beijing cert_location: Beijing cert_organization: k8s cert_organizational_unit: System cert_email: admink8s.io # 系统配置 system_reserved: cpu200m,memory1Gi kubelet_eviction_hard: memory.available500Mi,nodefs.available10%,nodefs.inodesFree5%4. 主节点变量 (group_vars/k8s_master.yml)# Master 节点特定配置 apiserver_insecure_port: 8080 apiserver_secure_port: 6443 controller_manager_port: 10252 scheduler_port: 10251 etcd_client_port: 2379 etcd_peer_port: 2380 # Etcd 集群配置 etcd_cluster: etcd01https://10.10.1.10:2380,etcd02https://10.10.1.11:2380,etcd03https://10.10.1.12:2380 etcd_initial_cluster_token: etcd-cluster-k8s5. 工作节点变量 (group_vars/k8s_node.yml)# Node 节点特定配置 kubelet_port: 10250 kube_proxy_port: 10256 max_pods: 110三、离线二进制文件下载清单在能访问互联网的机器上下载以下文件然后上传到 Ansible 主控机的files/binaries/目录。组件版本下载地址备注Kubernetesv1.30.10https://dl.k8s.io/v1.30.10/kubernetes-server-linux-amd64.tar.gz官方二进制包Cilium CLIv1.16.6https://github.com/cilium/cilium-cli/releases/download/v1.16.6/cilium-linux-amd64.tar.gzCilium 命令行工具Ciliumv1.16.6https://github.com/cilium/cilium/releases/download/v1.16.6/cilium-1.16.6.tar.gzCilium 安装包etcdv3.5.12https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gzetcd 集群cfssl最新版https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64证书工具cfssljson最新版https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64证书工具runcv1.1.12https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64容器运行时containerdv1.7.18https://github.com/containerd/containerd/releases/download/v1.7.18/containerd-1.7.18-linux-amd64.tar.gz容器运行时下载脚本示例#!/bin/bash # download.sh - 在能联网的机器上执行 mkdir -p binaries cd binaries # Kuberneteswget https://dl.k8s.io/v1.30.10/kubernetes-server-linux-amd64.tar.gz # Cilium wget https://github.com/cilium/cilium-cli/releases/download/v1.16.6/cilium-linux-amd64.tar.gz wget https://github.com/cilium/cilium/releases/download/v1.16.6/cilium-1.16.6.tar.gz # etcd wget https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gz # cfssl wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 -O cfssl wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64 -O cfssljsonchmod x cfssl cfssljson # 容器运行时 wget https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64wget https://github.com/containerd/containerd/releases/download/v1.7.18/containerd-1.7.18-linux-amd64.tar.gz echo 下载完成请将 binaries/ 目录上传到 Ansible 主控机的 files/ 目录下四、Ansible Roles 核心任务1. 通用系统初始化 (roles/common/tasks/main.yml)--- - name: 禁用 swap shell: | swapoff -a sed -i / swap / s/^\(.*\)$/#\1/g /etc/fstab name: 设置主机名 hostname: name: {{ inventory_hostname }} - name: 配置 hosts 文件 blockinfile: path: /etc/hosts block: | 10.10.1.10 master0110.10.1.11 master02 10.10.1.12 master0310.10.1.21 node01 10.10.1.22 node0210.10.1.23 node03 10.10.1.24 node0410.10.1.25 node05 10.10.1.26 node06 name: 安装基础依赖 apt: name: curl - wget gnupg lsb-release apt-transport-https ca-certificates software-properties-common net-tools - ipvsadm - ipset sysstat - conntrack - nfs-common state: present update_cache: yes name: 配置内核参数 copy: src: {{ local_files_dir }}/configs/sysctl.conf dest: /etc/sysctl.d/k8s.conf owner: root group: root mode: 0644 notify: reload sysctl name: 加载内核模块 copy: content: | overlay br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack dest: /etc/modules-load.d/k8s.conf owner: root group: root mode: 0644 notify: load kernel modules - name: 配置时间同步 apt: name: chrony state: present notify: restart chrony name: 创建安装目录 file: path: {{ item }} state: directory owner: root group: root mode: 0755 loop: {{ remote_install_dir }} {{ remote_bin_dir }} {{ remote_cfg_dir }} {{ remote_ssl_dir }} /etc/cni/net.d /opt/cni/bin /var/lib/kubelet - /var/lib/kube-proxy - /var/lib/kube-controller-manager /var/lib/kube-scheduler /var/log/kubernetes2. 证书生成 (roles/certs/tasks/main.yml)--- - name: 复制证书工具 copy: src: {{ local_files_dir }}/binaries/{{ item }} dest: /usr/local/bin/ owner: root group: root mode: 0755 loop: cfssl - cfssljson name: 创建证书配置目录 file: path: /tmp/k8s-certs state: directory name: 生成 CA 配置文件 copy: content: | { signing: { default: { expiry: 87600h }, profiles: { kubernetes: { usages: [ signing, key encipherment, server auth, client auth ], expiry: 87600h } } } } dest: /tmp/k8s-certs/ca-config.json - name: 生成 CA 证书签名请求 copy: content: | { CN: kubernetes, key: { algo: rsa, size: 2048 }, names: [ { C: {{ cert_country }}, ST: {{ cert_state }}, L: {{ cert_location }}, O: {{ cert_organization }}, OU: {{ cert_organizational_unit }} } ] } dest: /tmp/k8s-certs/ca-csr.json - name: 生成 CA 证书和私钥 shell: | cd /tmp/k8s-certs cfssl gencert -initca ca-csr.json | cfssljson -bare ca args: creates: /tmp/k8s-certs/ca.pem name: 分发 CA 证书到所有节点 copy: src: /tmp/k8s-certs/{{ item }} dest: {{ remote_ssl_dir }}/ owner: root group: root mode: 0644 loop: ca.pem ca-key.pem - ca.csr delegate_to: {{ groups[k8s_cluster] | first }} run_once: true3. etcd 集群部署 (roles/etcd/tasks/main.yml)--- - name: 解压 etcd 二进制文件 unarchive: src: {{ local_files_dir }}/binaries/etcd-v3.5.12-linux-amd64.tar.gz dest: /tmp/ remote_src: yes- name: 安装 etcd 二进制文件 copy: src: /tmp/etcd-v3.5.12-linux-amd64/{{ item }} dest: {{ remote_bin_dir }}/ owner: root group: root mode: 0755 loop: etcd - etcdctl name: 创建 etcd 数据目录 file: path: /var/lib/etcd state: directory owner: root group: root mode: 0700 - name: 配置 etcd systemd 服务 template: src: {{ local_files_dir }}/systemd/etcd.service.j2 dest: {{ remote_systemd_dir }}/etcd.service owner: root group: root mode: 0644 notify: daemon-reload - restart etcd name: 启动 etcd 服务 systemd: name: etcd state: started enabled: yes daemon_reload: yes4. Master 节点部署 (roles/master/tasks/main.yml)--- - name: 解压 Kubernetes 二进制文件 unarchive: src: {{ local_files_dir }}/binaries/kubernetes-server-linux-amd64.tar.gz dest: /tmp/ remote_src: yes - name: 安装 Master 组件二进制文件 copy: src: /tmp/kubernetes/server/bin/{{ item }} dest: {{ remote_bin_dir }}/ owner: root group: root mode: 0755 loop: - kube-apiserver - kube-controller-manager kube-scheduler - kubectl kubelet kube-proxy name: 生成 apiserver 证书 shell: | # 证书生成逻辑 cd {{ remote_ssl_dir }} cat kube-apiserver-csr.json EOF { CN: kube-apiserver, hosts: [ 127.0.0.1, {{ apiserver_vip }}, 10.10.1.10, 10.10.1.11, 10.10.1.12, 10.96.0.1, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster, kubernetes.default.svc.cluster.local ], key: { algo: rsa, size: 2048 }, names: [ { C: {{ cert_country }}, ST: {{ cert_state }}, L: {{ cert_location }}, O: {{ cert_organization }}, OU: {{ cert_organizational_unit }} } ] } EOF cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver args: creates: {{ remote_ssl_dir }}/kube-apiserver.pem name: 配置 apiserver systemd 服务 template: src: {{ local_files_dir }}/systemd/kube-apiserver.service.j2 dest: {{ remote_systemd_dir }}/kube-apiserver.service notify: daemon-reload - restart kube-apiserver - name: 配置 controller-manager template: src: {{ local_files_dir }}/systemd/kube-controller-manager.service.j2 dest: {{ remote_systemd_dir }}/kube-controller-manager.service notify: daemon-reload - restart kube-controller-manager name: 配置 scheduler template: src: {{ local_files_dir }}/systemd/kube-scheduler.service.j2 dest: {{ remote_systemd_dir }}/kube-scheduler.service notify: - daemon-reload restart kube-scheduler - name: 生成 admin kubeconfig shell: | # kubeconfig 生成逻辑 kubectl config set-cluster kubernetes \ --certificate-authority{{ remote_ssl_dir }}/ca.pem \ --embed-certstrue \ --serverhttps://{{ apiserver_vip }}:{{ apiserver_port }} \ --kubeconfig{{ remote_cfg_dir }}/admin.kubeconfig kubectl config set-credentials admin \ --client-certificate{{ remote_ssl_dir }}/admin.pem \ --client-key{{ remote_ssl_dir }}/admin-key.pem \ --embed-certstrue \ --kubeconfig{{ remote_cfg_dir }}/admin.kubeconfig kubectl config set-context kubernetes \ --clusterkubernetes \ --useradmin \ --kubeconfig{{ remote_cfg_dir }}/admin.kubeconfig kubectl config use-context kubernetes \ --kubeconfig{{ remote_cfg_dir }}/admin.kubeconfig args: creates: {{ remote_cfg_dir }}/admin.kubeconfig - name: 启动 Master 服务 systemd: name: {{ item }} state: started enabled: yes daemon_reload: yes loop: kube-apiserver kube-controller-manager - kube-scheduler5. Node 节点部署 (roles/node/tasks/main.yml)--- - name: 安装 Node 组件二进制文件 copy: src: {{ local_files_dir }}/binaries/kubernetes/server/bin/{{ item }} dest: {{ remote_bin_dir }}/ owner: root group: root mode: 0755 loop: - kubelet - kube-proxy - name: 配置 kubelet template: src: {{ local_files_dir }}/configs/kubelet-config.yaml.j2 dest: {{ remote_cfg_dir }}/kubelet-config.yaml owner: root group: root mode: 0644 - name: 配置 kubelet systemd 服务 template: src: {{ local_files_dir }}/systemd/kubelet.service.j2 dest: {{ remote_systemd_dir }}/kubelet.service notify: - daemon-reload restart kubelet name: 配置 kube-proxy template: src: {{ local_files_dir }}/configs/kube-proxy-config.yaml.j2 dest: {{ remote_cfg_dir }}/kube-proxy-config.yaml owner: root group: root mode: 0644 - name: 配置 kube-proxy systemd 服务 template: src: {{ local_files_dir }}/systemd/kube-proxy.service.j2 dest: {{ remote_systemd_dir }}/kube-proxy.service notify: daemon-reload - restart kube-proxy name: 分发 bootstrap kubeconfig copy: src: {{ local_files_dir }}/configs/bootstrap-kubelet.conf dest: {{ remote_cfg_dir }}/bootstrap-kubelet.conf owner: root group: root mode: 0644 - name: 启动 Node 服务 systemd: name: {{ item }} state: started enabled: yes daemon_reload: yes loop: kubelet kube-proxy6. Cilium CNI 部署 (roles/cilium/tasks/main.yml)--- - name: 解压 Cilium CLI unarchive: src: {{ local_files_dir }}/binaries/cilium-linux-amd64.tar.gz dest: /usr/local/bin/ remote_src: yes name: 安装 Cilium shell: | cilium install \ --version {{ cilium_version }} \ --cluster-name {{ cluster_name }} \ --cluster-id 1 \ --ipam kubernetes \ --kube-proxy-replacementstrict \ --native-routing-cidr{{ pod_cidr }} \ --helm-set bpf.masqueradetrue \ --helm-set ipv4NativeRoutingCIDR{{ pod_cidr }} \ --helm-set kubeProxyReplacementtrue \ --helm-set k8sServiceHost{{ apiserver_vip }} \ --helm-set k8sServicePort{{ apiserver_port }} args: creates: /var/run/cilium/cilium.sock - name: 验证 Cilium 安装 shell: | cilium status --wait register: cilium_status until: cilium_status.rc 0 retries: 10 delay: 30- name: 验证网络连接 shell: | cilium connectivity test五、主 Playbook (site.yml)--- - name: 初始化所有节点 hosts: k8s_cluster roles: - common name: 生成证书 hosts: master01 roles: certs name: 部署 etcd 集群 hosts: etcd roles: - etcd - name: 部署 Master 节点 hosts: k8s_master roles: - master name: 部署 Node 节点 hosts: k8s_node roles: node - name: 部署 Cilium CNI hosts: master01 roles: cilium name: 验证集群状态 hosts: master01 tasks: - name: 检查节点状态 shell: | kubectl get nodes -o wide register: node_status until: Ready in node_status.stdout retries: 20 delay: 10 - name: 检查 Pod 状态 shell: | kubectl get pods -A -o wide register: pod_status name: 显示集群信息 debug: msg: | 集群部署完成 节点状态 {{ node_status.stdout }} Pod 状态 {{ pod_status.stdout }} 访问集群 export KUBECONFIG{{ remote_cfg_dir }}/admin.kubeconfig kubectl cluster-info六、部署执行命令在 Ansible 主控机上执行# 1. 确保所有文件就位 cd /root/k8s-binary-offline ls -la files/binaries/ # 确认所有二进制文件已下载 # 2. 配置 SSH 免密登录在所有节点上执行 ssh-keygen -t rsa -N -f ~/.ssh/id_rsa for host in 10.10.1.{10..12} 10.10.1.{21..26}; do ssh-copy-id root$host done # 3. 执行 Ansible Playbook ansible-playbook -i inventory/hosts.ini site.yml -v # 4. 验证部署 ssh root10.10.1.10 kubectl get nodes ssh root10.10.1.10 kubectl get pods -A七、故障排查要点问题现象可能原因解决方案Ansible 连接失败SSH 配置问题检查/etc/ssh/sshd_config的PasswordAuthentication和PermitRootLogin证书生成失败时间不同步所有节点执行chronyc sources检查时间同步etcd 启动失败端口冲突或证书问题检查journalctl -u etcd查看详细日志apiserver 无法启动证书 hosts 不匹配确保证书包含所有 Master 节点 IP 和 VIPkubelet 无法注册bootstrap token 过期重新生成bootstrap-kubelet.confCilium Pod CrashLoopBackOff内核版本不兼容Ubuntu 22.04 内核需 ≥5.4执行uname -r确认节点 NotReadyCNI 未就绪检查kubectl describe node node-name查看事件此方案基于二进制部署的最佳实践结合 Ansible 的幂等性设计确保一次性部署成功。所有配置文件已针对 Ubuntu 22.04 和 Kubernetes 1.30.10 优化Cilium 配置采用直接路由模式避免 overlay 网络性能损耗 。参考来源Ansible自动化部署Kubernetes_ansible_荧523-松山湖开发者村综合服务平台用Ansible实现二进制搭建K8s集群v1.13.7一主两从-CSDN博客Ubuntu22.04 安装 K8SKubernetes v1.22/v1.23 企业级高可用集群Ansible自动部署教程ubuntu24.04二进制安装k8s1.23.16使用ansible进行自动化安装离线安装 - CSDN文库