阿里云k8s环境搭建测试
@ hanamichi · Wednesday, Jul 3, 2019 · 3 分钟阅读 · 更新于 7月 3, 2019

为之后在家自建nas,webdav等服务所预先测试

环境说明

  • 阿里云2c4g机器
  • all-in-one k8s集群
  • k8s 1.15.3
  • contour作为ingress
  • 域名配置A记录,*.master-node.whataspace.top指向虚拟机公网ip

环境安装

  1. 更改host

    hostnamectl set-hostname master-node
    
  2. 更改/etc/hosts文件

    yourip  master-node
    
  3. 关闭selinux

    sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
    
  4. 禁用swap

    swapoff -a
    

    编辑/etc/fstab,注释swap的那条记录

  5. 安装docker

    yum install -y yum-utils device-mapper-persistent-data lvm2
    
    yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    
    yum install docker-ce -y
    
  6. 将docker的cgroup driver更改成systemd

    cat <<EOF > /etc/docker/daemon.json
    {
      "exec-opts": ["native.cgroupdriver=systemd"],
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "100m"
      },
      "storage-driver": "overlay2",
      "storage-opts": [
        "overlay2.override_kernel_check=true"
      ]
    }
    EOF
    
  7. systemctl enable --now docker

  8. reboot

  9. 加载内核模块

    modprobe br_netfilter
    
  10. 安装k8s repo文件

    cat <<EOF > /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
    enabled=1
    gpgcheck=1
    repo_gpgcheck=1
    gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
    exclude=kube*
    EOF
    
  11. 安装kubeadm等工具

    yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
    
  12. 启动集群

    kubeadm init  --pod-network-cidr=192.168.0.0/16
    
  13. 根据提示拷贝kubeconfig文件

  14. 安装网络插件

    kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
    
  15. 使用kubectl get po --all-namespaces -w查看pod状态

  16. 部署成功后执行kubectl taint nodes --all node-role.kubernetes.io/master-使master节点可调度

ingress安装

ingress选用contour

安装方式见参考链接,在阿里云上使用,envoy的service使用nodeport方式暴露,具体yaml如下:

apiVersion: v1
kind: Service
metadata:
 name: envoy
 namespace: heptio-contour
spec:
 ports:
 - port: 80
   name: http
   protocol: TCP
   nodePort: 30080
 - port: 443
   name: https
   protocol: TCP
   nodePort: 30443
 selector:
   app: envoy
 type: NodePort

参考链接

使用 Contour 接管 Kubernetes 的南北流量

应用部署

k8s-dashboard

  1. 部署k8s-dashboard

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
    
  2. 定义ingressroute

    apiVersion: contour.heptio.com/v1beta1
    kind: IngressRoute
    metadata:
      name: dashboard
    spec:
      virtualhost:
        fqdn: yoursubdomain
        tls:
          passthrough: true
      tcpproxy:
        services:
          - name: kubernetes-dashboard
            port: 443
      routes:
        - match: /
          services:
            - name: kubernetes-dashboard
              port: 443
    

    ingressroute资源与dashboard均在kube-system下, 使用https://yoursubdomain:30443访问

  3. 创建集群管理员账户,使用以下文件创建sa

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: zhuxiaowei
      namespace: kube-system
    
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: zhuxiaowei
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
    - kind: ServiceAccount
      name: zhuxiaowei
      namespace: kube-system
    
  4. 获取token登录dashboard

    kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep zhuxiaowei | awk '{print $1}')
    

troubleshoot

  1. 默认情况下dashboard的https证书是空证书,在windows下使用谷歌浏览器存在无法访问的问题

    1. 使用以下命令创建自签证书:

      openssl genrsa -out ca.key 2048
      
      openssl req -new -x509 -key ca.key -out ca.crt -days 3650 -subj "/C=CN/ST=HB/L=WH/O=DM/OU=YPT/CN=CA"
      
      openssl x509 -in ca.crt -noout -text
      
      openssl genrsa -out dashboard.key 2048
      
      openssl req -new -sha256 -key dashboard.key -out dashboard.csr -subj "/C=CN/ST=HB/L=WH/O=DM/OU=YPT/CN=yourdomain"
      
      openssl x509 -req -sha256 -days 3650 -in dashboard.csr -out dashboard.crt -CA ca.crt -CAkey ca.key -CAcreateserial
      
      openssl x509 -in dashboard.crt -noout -text
      
    2. 创建kubernetes-dashboard-certs

      kubectl delete secret kubernetes-dashboard-certs -n kube-system
      
      kubectl create secret generic kubernetes-dashboard-certs --from-file="dashboard.crt,dashboard.key"
      

webdav

  1. 创建pv和pvc

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-volume-001
      labels:
        type: local
    spec:
      storageClassName: manual
      capacity:
        storage: 10Gi
      accessModes:
        - ReadWriteOnce
      hostPath:
        path: "/mnt/k8s-data"
    
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: webdav-data-claim
    spec:
      storageClassName: manual
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
    
  2. 使用以下deployment.yaml部署webdav服务

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: webdav
      name: webdav
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: webdav
      template:
        metadata:
          labels:
            app: webdav
        spec:
          containers:
          - image: twizzel/webdav:latest
            imagePullPolicy: IfNotPresent
            name: webdav
            env:
            - name: AUTH_TYPE
              value: Basic
            - name: USERNAME
              value: xxxxxxxxx
            - name: PASSWORD
              value: xxxxxxxxx
            - name: SSL_CERT
              value: selfsigned
            ports:
            - containerPort: 443
              name: https
              protocol: TCP
            volumeMounts:
            - name: webdav-data
              mountPath: /var/lib/dav
          volumes:
          - name: webdav-data
            persistentVolumeClaim:
              claimName: webdav-data-claim
    
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: webdav
      name: webdav
    spec:
      ports:
      - port: 443
        protocol: TCP
        name: https
        targetPort: 443
      selector:
        app: webdav
      sessionAffinity: None
      type: ClusterIP
    
    apiVersion: contour.heptio.com/v1beta1
    kind: IngressRoute
    metadata:
      name: webdav
    spec:
      virtualhost:
        fqdn: yoursubdomain
        tls:
          passthrough: true
      tcpproxy:
        services:
          - name: webdav
            port: 443
      routes:
        - match: /
          services:
            - name: webdav
              port: 443
    

关于我

关于我

Hanamichi 的 ❤️ 博客

记录一些 🌈 生活上,技术上的事

全平台玩家(pc-steam, ps4, switch)

职业是后端工程师

正在努力学习

对云原生微服务感兴趣

主要的技术栈是:

  • golang
  • kubernetes
  • 微服务架构

略懂一二:

  • python
  • C/C++
  • java

学习中:

  • Rust
  • JavaScript

目前在 99cloud 工作

– 2021 年 2 月 1 日更新

2020年的计划

2021年的计划

现在回头看看,因为疫情和自身自律性原因,2020年的计划大多都没有实现

  • github issue和wiki记录平时的一些想法,坚持整理然后写入博客

  • 继续阅读 程序员修炼之道代码整洁之道

  • 关注云原声CNCF开源社区动态,争取做些贡献(平时还是太懒,自律性不强)

  • 以 java spring 生态继续研究架构的世界

其他

其他

如果你喜欢我的文章 or 我的项目,或者它们可以给你带来帮助。

You can buy me a coffee. ~

我的微信赞赏码

wechat

社交链接