cilium/cilium-etcd-operator

重要提示：CoreDNS用户注意事项

为确保etcd节点间的TLS证书正常工作，pod IP的DNS反向查找必须映射回pod名称。如果使用CoreDNS，请检查CoreDNS ConfigMap，确保in-addr.arpa和ip6.arpa作为通配符列在kubernetes块中，如下所示：

kubectl -n kube-system edit cm coredns
[...]
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
    }

内容可能与上述不同，关键配置是确保in-addr.arpa和ip6.arpa作为通配符列在cluster.local旁边。

可通过在任何pod中使用host工具查找pod IP来验证：

host 10.60.20.86
86.20.60.10.in-addr.arpa domain name pointer cilium-etcd-972nprv9dp.cilium-etcd.kube-system.svc.cluster.local.

部署

部署cilium-etcd-operator时，仅当Kubernetes secret cilium-etcd-secrets不存在时才会自动创建。如果已配置Cilium使用外部etcd，可能使用相同的secret名称，因此部署cilium-etcd-operator不会覆盖该secret。

如果希望每次重启cilium-etcd-operator时覆盖证书，请设置以下环境变量：

- name: CILIUM_ETCD_OPERATOR_GENERATE_CERTS
  value: "true"

在cilium-etcd-operator.yaml文件中添加，并通过以下命令应用更改：

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-etcd-operator/master/cilium-etcd-operator.yaml

（可选）使用现有EtcdCluster自定义资源部署

从v2.0.6版本开始，cilium-etcd-operator支持重用用户部署的现有EtcdCluster。例如，可修改示例cilium-cr.yaml以添加etcd-operator提供的更多功能，如https://github.com/coreos/etcd-operator/blob/v0.9.4/doc/user/spec_examples.md#three-member-cluster-with-node-selector-and-anti-affinity-across-nodes%E6%88%96https://github.com/coreos/etcd-operator/blob/v0.9.4/pkg/apis/etcd/v1beta2/cluster.go#L127%E3%80%82%E8%AF%A5%E8%B5%84%E6%BA%90%E7%9A%84%E6%9E%B6%E6%9E%84%E5%8F%AF%E5%9C%A8https://github.com/coreos/etcd-operator/blob/v0.9.4/pkg/apis/etcd/v1beta2/cluster.go#L67%E6%89%BE%E5%88%B0%E3%80%82

首先，确保Kubernetes集群中已部署自定义资源定义etcdclusters.etcd.database.coreos.com。如果未部署，可通过以下命令部署：

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-etcd-operator/master/etcd-crd.yaml

之后，部署cilium的EtcdCluster自定义资源：

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-etcd-operator/master/cilium-cr.yaml

如果已设置etcdclusters.etcd.database.coreos.com CRD和EtcdCluster CR，可将cilium-etcd-operator ClusterRole的RBAC从：

yaml
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - delete
  - get
  - create

修改为：

yaml
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - get

验证

cilium-etcd-operator将启动etcd-operator并创建etcd集群。此过程可能需要几秒到几分钟。引导完成后，3节点etcd集群将启动：

kubectl -n kube-system get pods -l etcd_cluster=cilium-etcd
NAME                     READY   STATUS    RESTARTS   AGE
cilium-etcd-8k5czlw95m   1/1     Running   0          21h
cilium-etcd-mdwk9s99r5   1/1     Running   0          28h
cilium-etcd-zm52g4mqfv   1/1     Running   0          28h

如果不存在，它还将创建允许访问etcd的secrets：

kubectl -n kube-system get secret | grep cilium-
cilium-etcd-client-tls                           Opaque                                3      28h
cilium-etcd-peer-tls                             Opaque                                3      28h
cilium-etcd-server-tls                           Opaque                                3      28h
cilium-token-nj9dm                               kubernetes.io/service-account-token   3      28h

故障排除

检查etcd-operator状态：

kubectl -n kube-system get pods -l io.cilium/app=etcd-operator
NAME                             READY   STATUS    RESTARTS   AGE
etcd-operator-547c5c7f84-qqr2t   1/1     Running   1          29h

检查etcd-operator日志：

kubectl -n kube-system logs etcd-operator-547c5c7f84-qqr2t
[...]

检查故障的etcd集群成员：

kubectl -n kube-system get pods -l etcd_cluster=cilium-etcd
NAME                     READY   STATUS    RESTARTS   AGE
cilium-etcd-8k5czlw95m   1/1     Running   0          21h
cilium-etcd-mdwk9s99r5   1/1     Running   0          28h
cilium-etcd-zm52g4mqfv   1/1     Running   0          28h

检查单个etcd集群成员的日志：

kubectl -n kube-system logs cilium-etcd-8k5czlw95m

终止

终止cilium-etcd-operator将关闭operator本身，但etcd集群将继续运行。要删除etcd集群本身：

kubectl -n kube-system delete etcdclusters.etcd.database.coreos.com cilium-etcd

如果要清理所有状态，请运行清理脚本：

./cleanup.sh