Kubernetes部署信息

kubernetes部署版本信息

  • OS: CentOS Linux release 7.5.1804 (Core)
  • Linux Kernel：4.14.15-1.el7.elrepo.x86_64
  • kubernetes: v1.12.2
  • etcd: v3.2.22
  • docker: docker-ce-18.09.0-3.el7.x86_64
  • NetPlugin: Calico
  • DNS: CoreDNS
  • Proxy-Mode: IPVS

    节点信息

  • Kube-node1: Master/Node 192.168.5.251
  • Kube-node2: Master/Node 192.168.5.252
  • Kube-node3: Master/Node 192.168.5.253
  • VIP: 192.168.5.250

安装前准备

所有节点互通，可连接互联网,kube-node1可无密码连接kube-node2和kube-node3

  • 三台加增加/etc/hosts
    • 192.168.5.251 kube-node1
    • 192.168.5.252 kube-node2
    • 192.168.5.253 kube-node3

  • 关闭Selinux、防火墙和交换分区

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    #禁用selinux
    setenforce 0
    vim /etc/selinux/config
    SELINUX=disabled

    #关闭防火墙
    systemctl stop firewalld &&　systemctl disbale firewalld

    #关闭交换分区
    swapoff -a
    yes | cp /etc/fstab /etc/fstab_bak
    cat /etc/fstab_bak |grep -v swap > /etc/fstab

  • 增加kernel设置

1
2
3
4
5
vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
sysctl -p
  • 如内核版本低于4的先升级内核。参考

  • 同步系统时间

    1
    2
    3
    yum install ntpdate -y

    ntpdate cn.pool.ntp.org

  • 设置kube-node1和kube-node2,kube-node3免密码连接

    1
    2
    3
    4
    5
    6
    #三台机生成密钥对
    ssh-keygen -t rsa
    #把本地主机的公钥复制到远程主机的authorized_keys文件上
    ssh-copy-id -i .ssh/id_rsa.pub kube-node1
    ssh-copy-id -i .ssh/id_rsa.pub kube-node2
    ssh-copy-id -i .ssh/id_rsa.pub kube-node3

所有节点安装docker-ce

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
 --add-repo \
 https://download.docker.com/linux/centos/docker-ce.repo
yum makecache fast
yum install -y docker-ce

#编辑docker.service启动文件
sed -i "13i ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT" /usr/lib/systemd/system/docker.service
#启动docker
systemctl daemon-reload
systemctl enable docker
systemctl start docker

#查看docker
docker info
#报以下警告
WARNING: Usage of loopback devices is strongly discouraged for production use

#解决方法：
vi /etc/docker/daemon.json
{
  "storage-driver": "devicemapper",
  "storage-opts": [
    "dm.directlvm_device=/dev/xdf",
    "dm.thinp_percent=95",
    "dm.thinp_metapercent=1",
    "dm.thinp_autoextend_threshold=80",
    "dm.thinp_autoextend_percent=20",
    "dm.directlvm_device_force=false"
  ]
}

systemctl daemon-reload
systemctl start docker

*注意新版本( docker-ce-18.09)的Docker在CentOS上官方不建议使用Overylay

Linux发行版 建议的存储驱动
Docker CE on Ubuntu aufs, devicemapper, overlay2 (Ubuntu 14.04.4 or later, 16.04 or later), overlay, zfs, vfs
Docker CE on Debian aufs, devicemapper, overlay2 (Debian Stretch), overlay, vfs
Docker CE on CentOS devicemapper, vfs
Docker CE on Fedora devicemapper, overlay2 (Fedora 26 or later, experimental), overlay (experimental), vfs

创建Kubernetes集群证书

  • kubernetes各组件之间用证书进行加密通讯，所以在创建集群前需要生成证书。本文用cfss来生成证书 https://github.com/cloudflare/cfssl
  • 生成的证书及密钥如下：
    • ca-key.pem
    • ca.pem
    • kubernetes-key.pem
    • kubernetes.pem
    • kube-proxy.pem
    • kube-proxy-key.pem
    • admin.pem
    • admin-key.pem

      证书使用说明

    • etcd: 使用ca.pem、kubernetes-key.pem、kubernetes.pem
    • kube-apiserver: 使用ca.pem、kubernetes-key.pem kubernetes.pem
    • kubelet: 使用ca.pem
    • kube-proxy: 使用ca.pem、kube-proxy-key.pem、kube-proxy.pem
    • kubectl: 使用ca.pem、admin-key.kem、admin.pem
      Kube-controller、kube-scheduler需要和kube-api部署在同一台服务器上且使用非安全端口通信，不需要证书；kube-proxy可以以daemonset的方式部署到node节点,本次版本以systemd方式部署，下篇文章用Kubeadmin自动部署时使用daemonset方式部署。
安装CFSSL（二进制方式）
1
2
3
4
5
6
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl &&　chmod +x /usr/local/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson &&　chmod +x /usr/local/bin/cfssljson
wget	https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo && chmod +x /usr/local/bin/cfssl-certinfo

创建CA证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
mkdir /root/ssl && cd /root/ssl
cfssl print-defaults config > ca-config.json
#编辑ca-connfig.json 如下：
{
    "signing": {
        "default": {
            "expiry": "87600h"
        },
        "profiles": {
            "kubernetes": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

  • 字段说明：

    • ca-config.json: 可以定义多个profiles，分别指定不同的使用场景和过期时间；后续在签名证书时用到。
    • signing: 表示该证书可以用来签名其它证书；生成的ca.pem证书中 CA=TRUE
    • server auth: 表示client可以用该CA对server提供的证书进行验证
    • client auth: 表示server可以用该CA对client提供的证书进行验证

      创建CA证书签名请求

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      cfssl	print-defaults	csr	>	ca-csr.json
      #编辑ca-csr.json如下：
      {
          "CN": "kubernetes",
          "key": {
              "algo": "rsa",
              "size": 2048
          },
          "names": [
              {
                  "C": "CN", 
                  "L": "Zhuhai",
                  "ST": "Guandong",
                  "O": "k8s",
                  "OU": "System"
              }
          ]
      }

  • 字段说明：

    • “CN”: Common Name,kube-apiserver从证书中提取该字段作为请求的用户名（User Name）;浏览器使用该字段验证网站是否合法：
    • “O”: oranization,kube-apiserver从证书中提取该字段作为请求用户所属的组（Group）;

生成CA证书和私钥

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem
```bash

#### 创建kubernetes证书
- 创建kubernetes证书签名证求
```bash
cfssl	print-defaults	csr	>	kubernetes-csr.json
#编辑 kubernetes-csr.json如下：
{
    "CN": "kubernetes",
    "hosts": [
        "127.0.0.1",
        "192.168.5.250",
        "192.168.5.251",
        "192.168.5.252",
        "192.168.5.253",
        "10.254.0.1",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local"

    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Zhuhai",
            "ST": "Guangdong",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

  • 如果hosts字段不为空则需要指定授权使用该证书的IP或域名列表，该证书后续会被kubernetes master和etcd集群使用，所以指定了它们的IP。kubernetes服务IP段为10.254.0.0/24,批定了它的第一个IP（kube-apiserver指定的service-cluster-ip-range）

生成kubernetes证书和私钥

1
cfssl	gencert	-ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes	kubernetes-csr.json	|	cfssljson	-bare	kubernetes

创建 admin证书

  • 创建admin证书签名请求
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    cfssl	print-defaults	csr	>	admin-csr.json
    #编辑admin-csr.json文件如下：
    {
        "CN": "admin",
        "hosts": [
        ],
        "key": {
            "algo": "rsa",
            "size": 2048
        },
        "names": [
            {
                "C": "CN",
                "L": "Zhuhai",
                "ST": "Guangdong",
                "O": "system:masters",
                "OU": "System"
            }
        ]
    }
    #生成admin证书和私钥
    cfssl	gencert	-ca=ca.pem	-ca-key=ca-key.pem	-config=ca-config.json	-profile=kubernetes	admin-csr.json	|	cfssljson	-bare	admin
    ls admin*
    admin.csr  admin-csr.json  admin-key.pem  admin.pem

创建kube-proxy证书

  • 创建kube-proxy证书签名请求

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    cfssl	print-defaults	csr	>	kube-proxy-csr.json
    #编辑文件如下：
    {
        "CN": "system:kube-proxy",
        "hosts": [],
        "key": {
            "algo": "rsa",
            "size": 2048
        },
        "names": [
            {
                "C": "CN",
                "L": "Zhuhai",
                "ST": "Guangdong",
                "O": "k8s",
                "OU": "System"
            }
        ]
    }
    #生成kube-proxy客户端证书和私钥
    cfssl	gencert	-ca=ca.pem	-ca-key=ca-key.pem	-config=ca-config.json -profile=kubernetes		kube-proxy-csr.json	|	cfssljson	-bare	kube-proxy
    ls kube-proxy*
    kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

  • CN指定该证书的User为system:kube-proxy

  • kube-apiserver预定义的RoleBinding cluster-admin将User system:kube-proxy与Role system:node-proxier绑定，该Role授予了调用 Proxy相关API的权限

使用cfssl-certinfo命令校验证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
cfssl-certinfo -cert kubernetes.pem
{
  "subject": {
    "common_name": "kubernetes",
    "country": "CN",
    "organization": "k8s",
    "organizational_unit": "System",
    "locality": "Zhuhai",
    "province": "Guangdong",
    "names": [
      "CN",
      "Guangdong",
      "Zhuhai",
      "k8s",
      "System",
      "kubernetes"
    ]
  },
  "issuer": {
    "common_name": "kubernetes",
    "country": "CN",
    "organization": "k8s",
    "organizational_unit": "System",
    "locality": "Zhuhai",
    "province": "Guandong",
    "names": [
      "CN",
      "Guandong",
      "Zhuhai",
      "k8s",
      "System",
      "kubernetes"
    ]
  },
  "serial_number": "312943317769421607384342025457645583677464040976",
  "sans": [
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local",
    "127.0.0.1",
    "192.168.5.251",
    "192.168.5.252",
    "192.168.5.253",
    "10.254.0.1"
  ],
  "not_before": "2018-11-19T04:27:00Z",
  "not_after": "2028-11-16T04:27:00Z",
 ...
  • 检查subject字段的内容是否和ca-csr.json中的一致

  • 和issuer字段的内容是否和ca-csr.json中的一致

    分发证书

    1
    2
    3
    4
    5
    6
    7
    for NODE in kube-node1 kube-node2 kube-node3; do 
      echo "--- $NODE ---"
      ssh ${NODE} "mkdir -p /etc/kubernetes/ssl"
      for FILE in ca.pem ca-key.pem kubernetes.pem kubernetes-key.pem kube-proxy.pem kube-proxy-key.pem; do 
        scp /root/ssl/${FILE} ${NODE}:/etc/kubernetes/ssl
      done
    done

创建kubeconfig文件

  • kubelet kube-proxy等节点进程和kube-apiserver进程通信时需要认证和授权。kubeconfig文件提供认证凭据来访问kube-apiserver
创建 TLS Bootstrapping Token

  • 待续…..

最后更新： 2023年08月27日 03:06

原始链接： https://blog.icanwen.com/2018/11/14/kubernetes_install_v1/