bitnamicharts/etcd

Bitnami Secure Images Helm chart for Etcd

etcd is a distributed key-value store designed to securely store data across a cluster. etcd is widely used in production on account of its reliability, fault-tolerance and ease of use.

https://etcd.io/

Trademarks: This software listing is packaged by Bitnami. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement.

TL;DR

console
helm install my-release oci://REGISTRY_NAME/REPOSITORY_NAME/etcd

Note: You need to substitute the placeholders REGISTRY_NAME and REPOSITORY_NAME with a reference to your Helm chart registry and repository.

Introduction

This chart bootstraps a https://github.com/bitnami/containers/tree/main/bitnami/etcd deployment on a https://kubernetes.io cluster using the https://helm.sh package manager.

Before you begin

• Kubernetes 1.23+
• Helm 3.8.0+
• PV provisioner support in the underlying infrastructure

Installing the Chart

To install the chart with the release name my-release:

console
helm install my-release oci://REGISTRY_NAME/REPOSITORY_NAME/etcd

Note: You need to substitute the placeholders REGISTRY_NAME and REPOSITORY_NAME with a reference to your Helm chart registry and repository. For example, in the case of Bitnami, you need to use REGISTRY_NAME=registry-1.docker.io and REPOSITORY_NAME=bitnamicharts.

These commands deploy etcd on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation.

Tip: List all releases using helm list

Configuration and installation details

This section describes credentials, configuration, and other installation options.

Resource requests and limits

Bitnami charts allow setting resource requests and limits for all containers inside the chart deployment. These are inside the resources value (check parameter table). Setting requests is essential for production workloads and these should be adapted to your specific use case.

To make this process easier, the chart contains the resourcesPreset values, which automatically sets the resources section according to different presets. Check these presets in https://github.com/bitnami/charts/blob/main/bitnami/common/templates/_resources.tpl#L15. However, in production workloads using resourcesPreset is discouraged as it may not fully adapt to your specific needs. Find more information on container resource management in the https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/.

https://techdocs.broadcom.com/us/en/vmware-tanzu/bitnami-secure-images/bitnami-secure-images/services/bsi-doc/apps-tutorials-understand-rolling-tags-containers-index.html

It is strongly recommended to use immutable tags in a production environment. This ensures your deployment does not change automatically if the same tag is updated with a different image.

Bitnami will release a new chart updating its containers if a new version of the main container, significant changes, or critical vulnerabilities exist.

Prometheus metrics

This chart can be integrated with Prometheus by setting metrics.enabled to true. This will expose the etcd native Prometheus port in the container and service (if metrics.useSeparateEndpoint=true). It will all have the necessary annotations to be automatically scraped by Prometheus.

Prometheus requirements

It is necessary to have a working installation of Prometheus or Prometheus Operator for the integration to work. Install the https://github.com/bitnami/charts/tree/main/bitnami/prometheus or the https://github.com/bitnami/charts/tree/main/bitnami/kube-prometheus to easily have a working Prometheus in your cluster.

Integration with Prometheus Operator

The chart can deploy PodMonitor objects for integration with Prometheus Operator installations. To do so, set the value *.metrics.podMonitor.enabled=true. Ensure that the Prometheus Operator CustomResourceDefinitions are installed in the cluster or it will fail with the following error:

text
no matches for kind "PodMonitor" in version "monitoring.coreos.com/v1"

Install the https://github.com/bitnami/charts/tree/main/bitnami/kube-prometheus for having the necessary CRDs and the Prometheus Operator.

Update credentials

Bitnami charts configure credentials at first boot. Any further change in the secrets or credentials require manual intervention. Follow these instructions:

• Update the user password following https://etcd.io/docs/latest/op-guide/authentication/
• Update the password secret with the new values (replace the SECRET_NAME and PASSWORD placeholders)

shell
kubectl create secret generic SECRET_NAME --from-literal=etcd-root-password=PASSWORD --dry-run -o yaml | kubectl apply -f -

Cluster configuration

The Bitnami etcd chart can be used to bootstrap an etcd cluster, easy to scale and with available features to implement disaster recovery. It uses static discovery configured via environment variables to bootstrap the etcd cluster. Based on the number of initial replicas, and using the A records added to the DNS configuration by the headless service, the chart can calculate every advertised peer URL.

The chart makes use of some extra elements offered by Kubernetes to ensure the bootstrapping is successful:

• It sets a "Parallel" Pod Management Policy. This is critical, since all the etcd replicas should be created simultaneously to guarantee they can find each other.
• It records "not ready" pods in the DNS, so etcd replicas are reachable using their associated FQDN before they're actually ready.

Learn more about https://etcd.io/docs/current/op-guide/clustering/#discovery, https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-management-policies and https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-hostname-and-subdomain-fields.

Here is an example of the environment configuration bootstrapping an etcd cluster with 3 replicas:

The probes (readiness & liveness) are delayed 60 seconds by default, to give the etcd replicas time to start and find each other. After that period, the etcdctl endpoint health command is used to periodically perform health checks on every replica.

Scalability

The Bitnami etcd chart uses etcd reconfiguration operations to add/remove members of the cluster during scaling.

When scaling down, a "pre-stop" lifecycle hook is used to ensure that the etcdctl member remove command is executed. The hook stores the output of this command in the persistent volume attached to the etcd pod. This hook is also executed when the pod is manually removed using the kubectl delete pod command or rescheduled by Kubernetes for any reason. This implies that the cluster can be scaled up/down without human intervention.

Here is an example to explain how this works:

1. An etcd cluster with three members running on a three-nodes Kubernetes cluster is bootstrapped.
2. After a few days, the cluster administrator decides to upgrade the kernel on one of the cluster nodes. To do so, the administrator drains the node. Pods running on that node are rescheduled to a different one.
3. During the pod eviction process, the "pre-stop" hook removes the etcd member from the cluster. Thus, the etcd cluster is scaled down to only two members.
4. Once the pod is scheduled on another node and initialized, the etcd member is added again to the cluster using the etcdctl member add command. Thus, the etcd cluster is scaled up to three replicas.

If, for whatever reason, the "pre-stop" hook fails at removing the member, the initialization logic is able to detect that something went wrong by checking the etcdctl member remove command output that was stored in the persistent volume. It then uses the etcdctl member update command to add back the member. In this case, the cluster isn't automatically scaled down/up while the pod is recovered. Therefore, when other members attempt to connect to the pod, it may cause warnings or errors like the one below:

text
E | rafthttp: failed to dial XXXXXXXX on stream Message (peer XXXXXXXX failed to find local node YYYYYYYYY)
I | rafthttp: peer XXXXXXXX became inactive (message send to peer failed)
W | rafthttp: health check for peer XXXXXXXX could not connect: dial tcp A.B.C.D:2380: i/o timeout

Learn more about https://etcd.io/docs/current/op-guide/runtime-configuration/ and how to safely https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/.

Cluster updates

When updating the etcd StatefulSet (such as when upgrading the chart version via the helm upgrade command), every pod must be replaced following the StatefulSet update strategy.

The chart uses a "RollingUpdate" strategy by default and with default Kubernetes values. In other words, it updates each Pod, one at a time, in the same order as Pod termination (from the largest ordinal to the smallest). It will wait until an updated Pod is "Running" and "Ready" prior to updating its predecessor.

Learn more about https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies.

Disaster recovery

If, for whatever reason, (N-1)/2 members of the cluster fail and the "pre-stop" hooks also fail at removing them from the cluster, the cluster disastrously fails, irrevocably losing quorum. Once quorum is lost, the cluster cannot reach consensus and therefore cannot continue accepting updates. Under this circumstance, the only possible solution is usually to restore the cluster from a snapshot.

IMPORTANT: All members should restore using the same snapshot.

The Bitnami etcd chart solves this problem by optionally offering a Kubernetes cron job that periodically snapshots the keyspace and stores it in a RWX volume. In case the cluster disastrously fails, the pods will automatically try to restore it using the last available snapshot.

Learn how to enable this disaster recovery feature.

The chart also sets by default a "soft" Pod AntiAffinity to reduce the risk of the cluster failing disastrously.

Learn more about https://etcd.io/docs/current/op-guide/recovery, https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ and https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity

Enable security for etcd

The etcd chart can be configured with Role-based access control and TLS encryption to improve its security.

Configure RBAC

In order to enable Role-Based Access Control for etcd, set the following parameters:

text
auth.rbac.create=true
auth.rbac.rootPassword=ETCD_ROOT_PASSWORD

These parameters create a root user with an associate root role with access to everything. The remaining users will use the guest role and won't have permissions to do anything.

Configure TLS for server-to-server communications

In order to enable secure transport between peer nodes deploy the helm chart with these options:

text
auth.peer.secureTransport=true
auth.peer.useAutoTLS=true

Configure certificates for client communication

In order to enable secure transport between client and server, create a secret containing the certificate and key files and the CA used to sign the client certificates. In this case, create the secret and then deploy the chart with these options:

text
auth.client.secureTransport=true
auth.client.enableAuthentication=true
auth.client.existingSecret=etcd-client-certs

Learn more about the https://etcd.io/ and how to https://coreos.com/os/docs/latest/generate-self-signed-certificates.html.

Enable disaster recovery features

The Bitnami etcd Helm chart supports automatic disaster recovery by periodically snapshotting the keyspace. If the cluster permanently loses more than (N-1)/2 members, it tries to recover the cluster from a previous snapshot.

Enable this feature with the following parameters:

text
persistence.enabled=true
disasterRecovery.enabled=true
disasterRecovery.pvc.size=2Gi
disasterRecovery.pvc.storageClassName=nfs

If the startFromSnapshot.* parameters are used at the same time as the disasterRecovery.* parameters, the PVC provided via the startFromSnapshot.existingClaim parameter will be used to store the periodical snapshots.

NOTE: The disaster recovery feature requires volumes with ReadWriteMany access mode.

Backup and restore

Two different approaches are available to back up and restore this Helm Chart:

• Back up the data from the source deployment and restore it in a new deployment using etcd's built-in backup/restore tools.
• Back up the persistent volumes from the source deployment and attach them to a new deployment using Velero, a Kubernetes backup/restore tool.

Method 1: Backup and restore data using etcd's built-in tools

This method involves the following steps:

• Use the etcdctl tool to create a snapshot of the data in the source cluster.
• Make the snapshot available in a Kubernetes PersistentVolumeClaim (PVC) that supports ReadWriteMany access (for example, a PVC created with the NFS storage class)
• Restore the data snapshot in a new cluster using the <%= variable :catalog_name, :platform %> etcd Helm chart's startFromSnapshot.existingClaim and startFromSnapshot.snapshotFilename parameters to define the source PVC and source filename for the snapshot.

NOTE: Under this approach, it is important to create the new deployment on the destination cluster using the same credentials as the original deployment on the source cluster.

Method 2: Back up and restore persistent data volumes

This method involves copying the persistent data volumes for the etcd nodes and reusing them in a new deployment with https://velero.io/, an open source Kubernetes backup/restore tool. This method is only suitable when:

• The Kubernetes provider is https://velero.io/docs/latest/supported-providers/.
• Both clusters are on the same Kubernetes provider, as this is a requirement of https://velero.io/docs/latest/migration-case/.
• The restored deployment on the destination cluster will have the same name, namespace, topology and credentials as the original deployment on the source cluster.

This method involves the following steps:

• Install Velero on the source and destination clusters.
• Use Velero to back up the PersistentVolumes (PVs) used by the etcd deployment on the source cluster.
• Use Velero to restore the backed-up PVs on the destination cluster.
• Create a new etcd deployment on the destination cluster with the same deployment name, credentials and other parameters as the original. This new deployment will use the restored PVs and hence the original data.

Exposing etcd metrics

The metrics exposed by etcd can be exposed to be scraped by Prometheus. Metrics can be scraped from within the cluster using any of the following approaches:

• Adding the required annotations for Prometheus to discover the metrics endpoints, as in the example below:

yaml
podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics/cluster"
  prometheus.io/port: "9000"

• Creating a ServiceMonitor or PodMonitor entry (when the Prometheus Operator is available in the cluster)
• Using something similar to the https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml.

If metrics are to be scraped from outside the cluster, the Kubernetes API proxy can be utilized to access the endpoint.

Using custom configuration

In order to use custom configuration parameters, two options are available:

• Using environment variables: etcd allows setting environment variables that map to configuration settings. In order to set extra environment variables, you can use the extraEnvVars property. Alternatively, you can use a ConfigMap or a Secret with the environment variables using the extraEnvVarsCM or the extraEnvVarsSecret properties.

yaml
extraEnvVars:
  - name: ETCD_AUTO_COMPACTION_RETENTION
    value: "0"
  - name: ETCD_HEARTBEAT_INTERVAL
    value: "150"

• Using a custom etcd.conf.yml: The etcd chart allows mounting a custom etcd.conf.yml file as ConfigMap. In order to so, you can use the configuration property. Alternatively, you can use an existing ConfigMap using the existingConfigmap parameter.

Auto Compaction

Since etcd keeps an exact history of its keyspace, this history should be periodically compacted to avoid performance degradation and eventual storage space exhaustion. Compacting the keyspace history drops all information about keys superseded prior to a given keyspace revision. The space used by these keys then becomes available for additional writes to the keyspace.

autoCompactionMode, by default periodic. Valid values: "periodic", "revision".

• 'periodic' for duration based retention, defaulting to hours if no time unit is provided (e.g. "5m").
• 'revision' for revision number based retention. autoCompactionRetention for mvcc key value store in hour, by default 0, means disabled.

You can enable auto compaction by using following parameters:

console
autoCompactionMode=periodic
autoCompactionRetention=10m

Sidecars and Init Containers

If you have a need for additional containers to run within the same pod as the etcd app (e.g. an additional metrics or logging exporter), you can do so via the sidecars config parameter. Simply define your container according to the Kubernetes container spec.

yaml
sidecars:
  - name: your-image-name
    image: your-image
    imagePullPolicy: Always
    ports:
      - name: portname
       containerPort: 1234

Similarly, you can add extra init containers using the initContainers parameter.

yaml
initContainers:
  - name: your-image-name
    image: your-image
    imagePullPolicy: Always
    ports:
      - name: portname
        containerPort: 1234

Deploying extra resources

There are cases where you may want to deploy extra objects, such a ConfigMap containing your app's configuration or some extra deployment with a micro service used by your app. For covering this case, the chart allows adding the full specification of other objects using the extraDeploy parameter.

Setting Pod's affinity

This chart allows you to set your custom affinity using the affinity parameter. Find more information about Pod's affinity in the https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity.

As an alternative, you can use of the preset configurations for pod affinity, pod anti-affinity, and node affinity available at the https://github.com/bitnami/charts/tree/main/bitnami/common#affinities chart. To do so, set the podAffinityPreset, podAntiAffinityPreset, or nodeAffinityPreset parameters.

FIPS parameters

The FIPS parameters only have effect if you are using images from the https://go-vmware.broadcom.com/contact-us.

For more information on this new support, please refer to the https://techdocs.broadcom.com/us/en/vmware-tanzu/bitnami-secure-images/bitnami-secure-images/services/bsi-doc/security-frameworks-FIPS-compliance.html.

Persistence

The https://github.com/bitnami/containers/tree/main/bitnami/etcd image stores the etcd data at the /bitnami/etcd path of the container. Persistent Volume Claims are used to keep the data across statefulsets.

The chart mounts a https://kubernetes.io/docs/concepts/storage/persistent-volumes/ volume at this location. The volume is created using dynamic volume provisioning by default. An existing PersistentVolumeClaim can also be defined for this purpose.

If you encounter errors when working with persisten

Note: the README for this chart is longer than the DockerHub length limit of 25000, so it has been trimmed. The full README can be found at https://techdocs.broadcom.com/us/en/vmware-tanzu/bitnami-secure-images/bitnami-secure-images/services/bsi-app-doc/apps-charts-etcd-index.html