分类:中行

离线安装rancher HA

RancherHA离线部署

介绍

本文主要目的在于记录rancher ha集群搭建步骤

服务器环境

节点 IP OS 软件
Master1 10.10.12.11 CentOS7.6(3.10.0-957.10.1.el7.x86_6) Docker,etcd, controlplane,worker
Master2 10.10.12.12 CentOS7.6(3.10.0-957.10.1.el7.x86_6) Docker,etcd, controlplane,worker
Master3 10.10.12.13 CentOS7.6(3.10.0-957.10.1.el7.x86_6) Docker,etcd, controlplane,worker
Nginx 10.10.12.14 CentOS7.6(3.10.0-957.10.1.el7.x86_6) Nginx

基础环境安装

环境设置

所有rancher节点执行

1.操作系统文件限制

vi /etc/security/limits.conf

在文件末尾添加以下内容:

root soft nofile 655350

root hard nofile 655350

* soft nofile 655350

* hard nofile 655350

2.关闭防火墙

systemctl stop firewalld

systemctl disable firewalld

3.关闭setlinx

将SELINUX值设置为disabled:

vim /etc/selinux/config

SELINUX=disabled

4.关闭swap

注释或删除swap交换分区:

vi /etc/fstab

#/dev/mapper/centos-swap swap                    swap    defaults        0 0

临时关闭:

swapoff -a

5.kernel调优

添加如下内容,

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

vm.swappiness=0

vm.max_map_count=655360

报错:

sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory

sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory

解决:

modprobe br_netfilter

6.创建用户并且添加到docker组,设置rancher用户密码:

useradd rancher -G docker

echo password |passwd –stdin rancher

报错:

useradd: group ‘docker’ does not exist

解决:

先安装docker即可

7.ssh免密登录

在11-13服务器上生成密钥对并执行下面命令(root/rancher用户相同操作):

rancher用户:

ssh-keygen

ssh-copy-id -i .ssh/id_rsa.pub rancher@10.10.12.11

ssh-copy-id -i .ssh/id_rsa.pub rancher@10.10.12.12

ssh-copy-id -i .ssh/id_rsa.pub rancher@10.10.12.13

root用户:

ssh-keygen

ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.10.12.11

ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.10.12.12

ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.10.12.13

8.修改hosts文件

cat /etc/hosts

10.10.12.11   master1

10.10.12.12   master2

10.10.12.13   master3

10.10.12.14   hi.rancher.cn

docker安装

上传packet.zip,解压进入目录

unzip packet.zip

cd packet

unzip docker.zip

cd docker

1.配置本地yum源安装docker(docker版本为18.09.6)

yum -y install containerd.io-1.2.5-3.1.el7.x86_64.rpm  docker-ce-18.09.6-3.el7.x86_64.rpm  docker-ce-cli-18.09.6-3.el7.x86_64.rpm

2.启动并开机自启动

systemctl start docker

systemctl enable docker

nginx安装

在10.10.12.14服务器上安装nginx用于rancher-server负载均衡。

1.安装nginx(通过nexus代理yum源)

yum install nginx -y

sudo systemctl enable nginx.service

2.修改配置文件

vim /etc/nginx/nginx.conf

worker_processes 4;

worker_rlimit_nofile 40000;

events {

    worker_connections 8192;

}

stream {

    upstream rancher_servers_http {

        least_conn;

        server 10.10.12.11:80 max_fails=3 fail_timeout=5s;

        server 10.10.12.12:80 max_fails=3 fail_timeout=5s;

        server 10.10.12.13:80 max_fails=3 fail_timeout=5s;

    }

    server {

        listen     80;

        proxy_pass rancher_servers_http;

    }

    upstream rancher_servers_https {

        least_conn;

        server 10.10.12.11:443 max_fails=3 fail_timeout=5s;

        server 10.10.12.12:443 max_fails=3 fail_timeout=5s;

        server 10.10.12.13:443 max_fails=3 fail_timeout=5s;

    }

    server {

        listen     443;

        proxy_pass rancher_servers_https;

    }

}

3.启动nginx:

sudo systemctl restart nginx.service

Rancher集群部署

安装必要工具

Master1节点

上传tools.zip并解压

unzip packet/tools.zip

cd packet/tools/

在10.10.12.11服务器上进行下面操作。

1.安装rke:

chmod +x rke

mv rke /usr/bin/rke

2.安装kubectl:

chmod +x kubectl

mv kubectl /usr/bin/kubectl

3.安装helm:

tar zxf helm-v3.0.1-linux-amd64.tar.gz

mv linux-amd64/helm /usr/bin/

其它工具下载地址:

https://www.cnrancher.com/docs/rancher/v2.x/cn/install-prepare/download/

加载镜像

上传镜像到所有rancher节点服务器并加载到docker

unzip packet/rancher-images.zip

cd packet/rancher-images/

for i in `ls`;do docker load -i $i ;done

rke安装k8s

1.切换到rancher用户

su – rancher

2.创建rancher集群配置文件(并修改pod地址池)

vim rancher-cluster.yml

nodes:

  – address: 10.10.12.11

    user: rancher

    role: [controlplane,worker,etcd]

  – address: 10.10.12.12

    user: rancher

    role: [controlplane,worker,etcd]

  – address: 10.10.12.13

    user: rancher

    role: [controlplane,worker,etcd]

services:

  etcd:

    snapshot: true

    creation: 6h

    retention: 24h

  kube-api:

    service_cluster_ip_range: 10.53.0.0/16

  kube-controller:

    cluster_cidr: 10.52.0.0/16

    service_cluster_ip_range: 10.53.0.0/16

  kubelet:  

   cluster_domain: local  

   cluster_dns_server: 10.53.0.10

注:如果之前操作失败,重新安装需要清理数据:

su – root

rm -rf /var/lib/rancher/etcd/*

rm -rf /etc/kubernetes/*

su – rancher

rke remove –config ./rancher-cluster.yml

3.启动集群

rke up –config ./rancher-cluster.yml

完成后,它应显示:Finished building Kubernetes cluster successfully。

4.配置环境变量:

切换到root用户su – root

vim /etc/profile

export KUBECONFIG=/home/rancher/kube_config_rancher-cluster.yml

保存,并执行:

source /etc/profile

修改metrics镜像下载策略

kubectl edit deployment metrics-server -n kube-system

5.通过kubectl测试您的连接,并查看您的所有节点是否处于Ready状态

6.检查集群pod运行情况

注:保存kube_config_rancher-cluster.yml和rancher-cluster.yml文件的副本,您将需要这些文件来维护和升级Rancher实例。

安装Rancher

1.安装证书

使用脚本生成自签名证书

sh create-ca.sh –ssl-domain=hi.rancher.cn –ssl-trusted-ip=10.10.12.11,10.10.12.12,10.10.12.13 –ssl-size=2048 –ssl-date=3650

kubectl create namespace cattle-system

kubectl -n cattle-system create secret tls tls-rancehr-ingress –cert=./tls.crt –key=./tls.key

kubectl -n cattle-system create secret generic tls-ca –from-file=cacerts.pem

注:必须把服务证书文件和key文件重命名为tls.crt和tls.key

2.helm安装rancher

helm  install rancher ./rancher-2.3.3.tgz –namespace cattle-system    –set hostname=hi.rancher.cn   –set ingress.tls.source=secret 

3.检查rancher状态

访问rancher

https://hi.rancher.cn    需要添加本地hosts

设置admin用户密码

为Cluster Pod添加主机别名(/etc/hosts)

如果你没有内部DNS服务器而是通过添加/etc/hosts主机别名的方式指定的Rancher server域名,那么不管通过哪种方式(自定义、导入、Host驱动等)创建K8S集群,K8S集群运行起来之后,因为cattle-cluster-agent Pod和cattle-node-agent无法通过DNS记录找到Rancher server,最终导致无法通信。

解决方法

可以通过给cattle-cluster-agent Pod和cattle-node-agent添加主机别名(/etc/hosts),让其可以正常通信(前提是IP地址可以互通)。

cattle-cluster-agent pod

kubectl -n cattle-system \

patch deployments cattle-cluster-agent –patch ‘{

    “spec”: {

        “template”: {

            “spec”: {

                “hostAliases”: [

                    {

                        “hostnames”:

                        [

                            “hi.rancher.cn”

                        ],

                            “ip”: “10.10.12.14”

                    }

                ]

            }

        }

    }

}’

cattle-node-agent pod

kubectl -n cattle-system \

patch daemonsets cattle-node-agent –patch ‘{

    “spec”: {

        “template”: {

            “spec”: {

                “hostAliases”: [

                    {

                        “hostnames”:

                        [

                            “hi.rancher.cn”

                        ],

                            “ip”: “10.10.12.14”

                    }

                ]

            }

        }

    }

}’

设置默认镜像仓库地址

1.登陆rancher UI设置管理员密码和url

2.在全局视图找到设置

3.找到system-default-registry点击编辑

4.设置镜像仓库地址为nexus代理仓库(不要加http/https前缀)

问题

1.rke安装k8s报错如下:

FATA[0001] Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) [10.10.12.13]

解决方案:尝试使用rancher用户执行命令docker info是否有权限

useradd rancher -G docker

openssh 版本必须高于6.7