{{ cat }}'s docs

kubernetes 1.2.1 kubernetes集群安装(centos7裸机)


0. 背景介绍

1) 参照文档

不参照[CentOS][Using kubeadm to Create a Cluster]的原因是,前者已经废弃,后者在beta阶段。另外此文档只是一个大纲,这样能够更深入的了解kubernetes的组件和原理。

文档中有很多细节,实际操作之外的步骤大部分忽略掉了,推荐详读一遍文档。

2) 软件版本

items version comment
OS centos7
kubernetes 1.9.1 最新稳定版本
docker 17.09.0-ce
etcd 3.0.7
flannel 使用flannel做overlay网络,支持不同主机间pods间网络互通
  • docker(或者rkt)是必备的,因为kubernetes本身就是一个容器的编排工具
  • etcd给kubernetes和flannel提供数据存储支持,可部署在kubernetes master节点上,也可以单独启用一个集群
  • flannel给kubernetes提供了overlay网络支持(可选,也有其他选择,详细见文章开头的文档链接中的描述),实现了不同主机pods之间的直接互通
  • kubernetes包含以下组件
    • 在master节点上运行的kube-apiserver,kube-controller-manager,kube-scheduler
    • 在node节点上运行的kubelet,kube-proxy

3) 节点规划

hostname ip address service comment
master 172.16.1.100 etcd,kube-apiserver,kube-controller-manager,kube-scheduler,docker 主节点
node01 172.16.1.101 flannel,docker,kubelet,kube-proxy node 1
node02 172.16.1.102 flannel,docker,kubelet,kube-proxy node 2
node03 172.16.1.103 flannel,docker,kubelet,kube-proxy node 3

1. 主机环境

为了将系统环境和软件环境对安装的影响度降低,需要确保以下几项需求满足

  • 安装必要的工具包
    yum install -y wget vim iptables iptables-services

  • 关闭selinux

    sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
    setenforce 0
    
  • 关闭iptables-services和firewalld
    systemctl stop firewalld;systemctl stop iptables

    防火墙后期需要开启,并开放api服务的端口

  • 设定hostname到hosts文件中

    echo "172.16.1.100  master
    172.16.1.101  node01
    172.16.1.102  node02
    172.16.1.103  node03" >> /etc/hosts
    
  • 设定sysctl中的net.ipv4.ip_forward = 1

    echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
    sysctl -p
    

    net.ipv4.ip_forward = 1的配置确保了可以通过映射docker容器端口到外网,否则我们无法通过外网ip访问容器

  • 关闭系统swap

    swapoff -a
    

    注释swap的开机挂载项,修改/etc/fstab

    #/dev/mapper/VolGroup00-LogVol01 swap                    swap    defaults        0 0

    关闭系统swap,是为了严格的按照cpu和内存的限制,这样scheduler在规划pod的时候就不会把pod放进swap中了,这是为了性能考虑。


2. kubernetes master节点

1) 配置kubernetes环境变量(master节点)

echo 'export MASTER_IP=172.16.1.100
export SERVICE_CLUSTER_IP_RANGE=10.254.0.0/16
export CLUSTER_NAME=KubeTest
export PATH=$PATH:/usr/local/kubernetes/bin' > /etc/profile.d/kubernetes.sh
source /etc/profile.d/kubernetes.sh

规划集群中需要重复使用的内容为变量

  • MASTER_IP - master的静态ip
  • SERVICE_CLUSTER_IP_RANGE - service对象使用的ip范围
  • CLUSTER_NAME - kubernetes集群的名称

2) 获取kubernetes(master节点)

kubernetes的二进制包里面包含了kubernetes的二进制文件和支持的etcd版本

# 下载kubernetes
wget https://dl.k8s.io/v1.9.1/kubernetes-server-linux-amd64.tar.gz
tar zxvf kubernetes-server-linux-amd64.tar.gz

# 拷贝二进制文件到server端
mkdir -p /usr/local/kubernetes/{bin,security,conf}
cp kubernetes/server/bin/{kube-apiserver,kube-scheduler,kube-controller-manager,kubectl} /usr/local/kubernetes/bin/
chmod 750 /usr/local/kubernetes/bin/*
# 如果使用docker启动kube-apiserver,kube-scheduler,kube-controller-manager这三个服务的话,不需要拷贝它们的二进制文件,只需要拷贝kubectl即可

# 拷贝二进制文件到node端(提前做好ssh信任)
scp kubernetes/server/bin/{kubelet,kube-proxy} root@node01:/usr/local/bin
scp kubernetes/server/bin/{kubelet,kube-proxy} root@node02:/usr/local/bin
scp kubernetes/server/bin/{kubelet,kube-proxy} root@node03:/usr/local/bin

因为kubernetes这个项目是使用go语言编写,而go语言程序的部署方式很简单,就是拷贝二进制文件就可以,所以在这里,我们通过简单的复制各服务的二进制文件,就可以通过启动它们来启动相应的服务。

本文开头的参照文档中说:
node需要运行的kubelet,kube-proxy,docker,推荐直接在系统层面上启动服务;
而对于etcd, kube-apiserver, kube-controller-manager 和 kube-scheduler,推荐我们使用容器来运行它们,文档中给出了几种镜像的获取方式,当然,我们下载的二进制文件中也有这样的镜像文件(bin目录中tar结尾的文件)可以本地加载(使用docker load命令)镜像到本机的docker中。


3). 配置和安装kubernetes master服务

1) 部署etcd

kubernetes/cluster/images/etcd/Makefile中查找到对应的etcd版本

etcd 单点的安装可以参照etcd install single node with systemd

使用etcd储存flannel的网络配置

etcdctl --endpoints http://$MASTER_IP:2379 set /kube-centos/network/config '{ "Network": "10.5.0.0/16", "Backend": {"Type": "vxlan"}}'

为了测试,在主节点上只启动一个节点的etcd,etcd集群参照etcd 集群文档

2) 启动kubernets Apiserver, Controller Manager, 和 Scheduler服务

准备配置文件:

  • config, 通用配置
  • apiserver, kube-apiserver配置
  • controller-manager, kube-controller-manager配置
  • scheduler, kube-scheduler配置
cat > /usr/local/kubernetes/conf/config << EOF
###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"


# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://127.0.0.1:8080"
EOF

cat > /usr/local/kubernetes/conf/apiserver << EOF
###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

# The port on the local server to listen on.
KUBE_API_PORT="--insecure-port=8080"

# Port minions listen on
# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379,http://127.0.0.1:4001"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=$SERVICE_CLUSTER_IP_RANGE"

# default admission control policies
# KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"
KUBE_ADMISSION_CONTROL=""

# Add your own!
KUBE_API_ARGS="--service-node-port-range=1-65535"
EOF

cat > /usr/local/kubernetes/conf/controller-manager << EOF
###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS=""
EOF

cat > /usr/local/kubernetes/conf/scheduler << EOF
###
# kubernetes scheduler config

# default config should be adequate

# Add your own!
KUBE_SCHEDULER_ARGS=""
EOF

错误: No API token found for service account "default", retry after the token,解决办法是配置KUBE_ADMISSION_CONTROL=""禁用KUBE_ADMISSION_CONTROL

准备systemd unit文件:

  • kube-apiserver.service
  • kube-controller-manager.service
  • kube-scheduler.service
echo '[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
EnvironmentFile=-/usr/local/kubernetes/conf/config
EnvironmentFile=-/usr/local/kubernetes/conf/apiserver
User=kube
ExecStart=/usr/local/kubernetes/bin/kube-apiserver \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_ETCD_SERVERS \
        $KUBE_API_ADDRESS \
        $KUBE_API_PORT \
        $KUBELET_PORT \
        $KUBE_ALLOW_PRIV \
        $KUBE_SERVICE_ADDRESSES \
        $KUBE_ADMISSION_CONTROL \
        $KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/kube-apiserver.service

mkdir /usr/lib/systemd/system/kube-apiserver.service.d
echo '[Service]
PermissionsStartOnly=yes
ExecStartPre=/usr/bin/mkdir -p /var/run/kubernetes
ExecStartPre=/usr/bin/chown kube.kube /var/run/kubernetes' > /usr/lib/systemd/system/kube-apiserver.service.d/pre-start.conf

echo '[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/usr/local/kubernetes/conf/config
EnvironmentFile=-/usr/local/kubernetes/conf/controller-manager
User=kube
ExecStart=/usr/local/kubernetes/bin/kube-controller-manager \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/kube-controller-manager.service


echo '[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/usr/local/kubernetes/conf/config
EnvironmentFile=-/usr/local/kubernetes/conf/scheduler
User=kube
ExecStart=/usr/local/kubernetes/bin/kube-scheduler \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/kube-scheduler.service

依次启动kube-apiserver.service, kube-controller-manager.service, kube-scheduler.service

# 重载systemd unit文件
systemctl daemon-reload

# 创建spawn服务的用户kube(在配置文件中配置)
useradd -r -s /sbin/nologin kube
chown :kube /usr/local/kubernetes/bin/*

systemctl enable kube-apiserver.service
systemctl enable kube-controller-manager.service
systemctl enable kube-scheduler.service
systemctl start kube-apiserver.service
systemctl start kube-controller-manager.service
systemctl start kube-scheduler.service

3. node节点配置和安装基本软件

1) 部署flannel(node节点)

# 下载flannel
FLANNEL_VER=v0.9.1
wget https://github.com/coreos/flannel/releases/download/v0.9.1/flannel-${FLANNEL_VER}-linux-amd64.tar.gz
mkdir flannel
tar zxvf flannel-${FLANNEL_VER}-linux-amd64.tar.gz -C flannel
cp flannel/flanneld /usr/local/bin
mkdir -p /usr/libexec/flannel
cp flannel/mk-docker-opts.sh /usr/libexec/flannel/

# 准备flannel配置文件
## !!重点!! ##
# -iface,根据实际情况设定
# FLANNELD_PUBLIC_IP,每个节点不同
#############
cat > /etc/sysconfig/flanneld << EOF
FLANNELD_PUBLIC_IP="172.16.1.101"
FLANNELD_ETCD_ENDPOINTS="http://172.16.1.100:2379"
FLANNELD_ETCD_PREFIX="/kube-centos/network"
# Any additional options that you want to pass
FLANNELD_OPTIONS="-iface=eth1"
EOF

# 准备flannel systemd unit文件
echo '[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
ExecStart=/usr/local/bin/flanneld $FLANNELD_OPTIONS
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -c
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service' > /usr/lib/systemd/system/flannel.service

systemctl daemon-reload
systemctl enable flannel
systemctl start flannel

每个节点的flannel需要根据自己情况来填写配置文件

flannel启动后生成了以下文件:

  • /var/run/flannel/subnet.env, 从etcd中获取信息然后生成的flanneld配置文件
  • /run/docker_opts.env, flannel service文件中指定的/usr/libexec/flannel/mk-docker-opts.sh生成的docker环境变量文件

2) 安装docker(node节点)

# 安装docker底包
yum install -y git libcgroup libcgroup-tools
systemctl enable cgconfig
systemctl start cgconfig

# 下载安装docker
DOCKER_VER=17.09.0
wget https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_VER}-ce.tgz
tar zxvf docker-${DOCKER_VER}-ce.tgz
cp docker/* /usr/local/bin/
wget https://github.com/docker/compose/releases/download/1.17.1/docker-compose-Linux-x86_64
cp docker-compose-Linux-x86_64 /usr/local/bin/docker-compose
chmod 755 /usr/local/bin/*

# 准备systemd unit文件
echo '[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket flannel.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/docker_opts.env
ExecStart=/usr/local/bin/dockerd -H fd:// $DOCKER_OPTS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/docker.service


echo '[Unit]
Description=Docker Socket for the API
PartOf=docker.service

[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target' > /usr/lib/systemd/system/docker.socket

groupadd docker

systemctl daemon-reload
systemctl enable docker
systemctl start docker

docker systemd

3) 安装kubelet(node节点)

准备配置文件:

  • config, 通用配置
  • kubelet, kubelet配置
  • controller-manager, kube-controller-manager配置
mkdir /usr/local/kubernetes/conf -p

cat > /usr/local/kubernetes/conf/config << EOF
###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"


# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://172.16.1.100:8080"
EOF

cat > /usr/local/kubernetes/conf/kubelet << EOF
###
# kubernetes kubelet (minion) config

# --kubeconfig for kubeconfig
KUBELET_KUBECONFIG="--kubeconfig=/usr/local/kubernetes/conf/node-kubeconfig.yaml"

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"

# The port for the info server to serve on
# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override="

# Add your own!
KUBELET_ARGS=""
EOF

cat > /usr/local/kubernetes/conf/proxy << EOF
###
# kubernetes proxy config

# default config should be adequate

# Add your own!
KUBE_PROXY_ARGS=""
EOF

cat > /usr/local/kubernetes/conf/node-kubeconfig.yaml << EOF
apiVersion: v1
kind: Config
clusters:
- name: local
  cluster:
    server: http://master:8080
contexts:
- context:
    cluster: local
  name: kubelet-cluster.local
current-context: kubelet-cluster.local
EOF

准备systemd unit文件:

  • kubelet.service
  • kube-proxy.service
echo '[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/usr/local/kubernetes/conf/config
EnvironmentFile=-/usr/local/kubernetes/conf/kubelet
ExecStart=/usr/local/bin/kubelet \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBELET_KUBECONFIG \
        $KUBELET_ADDRESS \
        $KUBELET_PORT \
        $KUBELET_HOSTNAME \
        $KUBE_ALLOW_PRIV \
        $KUBELET_ARGS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/kubelet.service

echo '[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
EnvironmentFile=-/usr/local/kubernetes/conf/config
EnvironmentFile=-/usr/local/kubernetes/conf/proxy
ExecStart=/usr/local/bin/kube-proxy \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/kube-proxy.service

依次启动kubelet,kube-proxy服务

# 重载systemd units文件
systemctl daemon-reload

# 创建kubelet工作目录
mkdir /var/lib/kubelet

# 启动服务
systemctl enable kubelet
systemctl enable kube-proxy
systemctl start kubelet
systemctl start kube-proxy