• 集群需部署后需配置 default storageclass，本文档不涉及此部分

部署

  • 官方部署文档：https://kubesphere.com.cn/docs/quick-start/minimal-kubesphere-on-k8s/

  • 开始安装(kubeasz 部署的集群安装 kubesphere 时会发生报错，详见实践)

1
2
3
4

$ wget https://github.com/kubesphere/ks-installer/releases/download/v3.2.1/kubesphere-installer.yaml
$ wget https://github.com/kubesphere/ks-installer/releases/download/v3.2.1/cluster-configuration.yaml
$ kubectl apply -f kubesphere-installer.yaml
$ kubectl apply -f cluster-configuration.yaml
  • 检查安装日志
1

$ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
  • 判断运行状态
1

$ kubectl get svc/ks-console -n kubesphere-system
  • 访问：确保在安全组中打开了端口 30880，并通过 NodePort (IP:30880) 使用默认帐户和密码 (admin/P@88w0rd) 访问 Web 控制台。

实践

部署脚本与集群通讯 SSL 报错

问题

  • 在运行完安装命令后，查看日志发现 ks-installer 容器运行 python 与集群通讯时存在报错
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

$  kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
2022-01-29T09:32:27+08:00 INFO     : shell-operator latest
2022-01-29T09:32:27+08:00 INFO     : HTTP SERVER Listening on 0.0.0.0:9115
2022-01-29T09:32:27+08:00 INFO     : Use temporary dir: /tmp/shell-operator
2022-01-29T09:32:27+08:00 INFO     : Initialize hooks manager ...
2022-01-29T09:32:27+08:00 INFO     : Search and load hooks ...
2022-01-29T09:32:27+08:00 INFO     : Load hook config from '/hooks/kubesphere/installRunner.py'
2022-01-29T09:32:27+08:00 INFO     : Load hook config from '/hooks/kubesphere/schedule.sh'
2022-01-29T09:32:27+08:00 INFO     : Initializing schedule manager ...
2022-01-29T09:32:27+08:00 INFO     : KUBE Init Kubernetes client
2022-01-29T09:32:27+08:00 INFO     : KUBE-INIT Kubernetes client is configured successfully
2022-01-29T09:32:27+08:00 INFO     : MAIN: run main loop
2022-01-29T09:32:27+08:00 INFO     : MAIN: add onStartup tasks
2022-01-29T09:32:27+08:00 INFO     : QUEUE add all HookRun@OnStartup
2022-01-29T09:32:27+08:00 INFO     : Running schedule manager ...
2022-01-29T09:32:27+08:00 INFO     : MSTOR Create new metric shell_operator_live_ticks
2022-01-29T09:32:27+08:00 INFO     : MSTOR Create new metric shell_operator_tasks_queue_length
2022-01-29T09:32:28+08:00 INFO     : GVR for kind 'ClusterConfiguration' is installer.kubesphere.io/v1alpha1, Resource=clusterconfigurations
2022-01-29T09:32:28+08:00 INFO     : EVENT Kube event '6a4a380c-3c07-4b9f-9c61-67d46575a822'
2022-01-29T09:32:28+08:00 INFO     : QUEUE add TASK_HOOK_RUN@KUBE_EVENTS kubesphere/installRunner.py
2022-01-29T09:32:30+08:00 INFO     : TASK_RUN HookRun@KUBE_EVENTS kubesphere/installRunner.py
2022-01-29T09:32:30+08:00 INFO     : Running hook 'kubesphere/installRunner.py' binding 'KUBE_EVENTS' ...
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)'))': /apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations/ks-installer
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)'))': /apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations/ks-installer
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)'))': /apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations/ks-installer

解决：

  • 更改容器内 /usr/local/lib/python3.9/site-packages/kubernetes/client/configuration.py 包，从依赖包层面忽略掉 SSL 校验

  • 步骤：

    • 将宿主机目录通过 volume 挂载入容器中，将 configuration.py 复制至宿主机挂载进容器的路径下
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    app: ks-install
spec:
  template:
    spec:
      # 容器挂载 volume
      containers:
        volumeMounts:
        - name: py-get
          mountPath: /py_get
          readOnly: false
      # 增加 volume
      volumes:
      - name: py-get
        hostPath:
          path: /root/py_get
          type: Directory

    1

    $ k apply -f kubesphere-installer.yaml
    • 查看 pod 对应的 node
    1
2
3

    $ k -n kubesphere-system get po -owide
NAME                            READY   STATUS    RESTARTS   AGE     IP             NODE           NOMINATED NODE   READINESS GATES
ks-installer-7bf885bc99-j8r2m   1/1     Running   0          4m39s   172.20.104.9   172.16.1.235   <none>           <none>
    • 调整 node /root/py_get 目录权限
    1

    $ sudo chmod o+w /root/py_get/
    • 拷贝文件
    1
2

    $ k -n kubesphere-system exec -it  ks-installer-7bf885bc99-j8r2m --  /bin/bash
bash-5.1$ cp /usr/local/lib/python3.9/site-packages/kubernetes/client/configuration.py /py_get/
    • 将 configuration.py 复制至 master node，更改配置，关闭 ssl 校验
    1
2

    # configuration.py 关闭 verify_ssl
self.verify_ssl = True ---> False
    • 更改 kubesphere-installer.yaml，挂载 configmap，覆盖容器镜像内 configuration.py 包
      • 注意：需去掉刚刚挂载 hostpath 那段配置
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

    # kubesphere-installer.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    app: ks-install
spec:
  template:
    spec:
      # 容器挂载 volume
      containers:
        volumeMounts:
        - name: py-configmap
          mountPath: /usr/local/lib/python3.9/site-packages/kubernetes/client/configuration.py
          subPath: configuration.py
          readOnly: true
      # 增加 volume
      volumes:
      - name: py-configmap
        configMap:
          name: py-configmap
    • apply kubesphere-installer.yaml
    1

    $ k apply -f kubesphere-installer.yaml
    • 将 configuration.py 注册入集群 configmap 中
      • 如果 apply kubesphere-installer.yaml 较长事件后才创建 cm，可通过直接 delete pod 的方式触发 pod 重启
    1

    $ k -n kubesphere-system create configmap py-configmap --from-file=configuration.py
    • apply cluster-configuration.yaml
    1

    $ k apply -f cluster-configuration.yaml
    • 查看安装日志
    1

    $ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

卸载

1
2

$ chmod u+x kubesphere-delete.sh
$ ./kubesphere-delete.sh

参考

  • 官方安装文档：https://kubesphere.com.cn/docs/quick-start/minimal-kubesphere-on-k8s/
  • 安装时 SSL 报错解决：https://github.com/kubernetes-client/python/issues/521
  • 卸载：https://kubesphere.com.cn/docs/installing-on-kubernetes/uninstall-kubesphere-from-k8s/