Encrypt Secrets in an etcd Database

Note: As of v1.8, Enterprise PKS has been renamed to VMware Tanzu Kubernetes Grid Integrated Edition. Some screenshots in this documentation do not yet reflect the change.

Page last updated:

This topic describes how to create and use a Kubernetes profile to encrypt a cluster’s etcd database.

For more information and other uses of Kubernetes profiles, see Using Kubernetes Profiles.

Create Kubernetes Profile

  1. Create a new a base64-encoded, 32-byte random key string to use as a secret.

    • On Linux or MacOS, you can generate the secret by running: head -c 32 /dev/urandom | base64
    • Example: jHc3NMp7s7T7JoJuZF7NUSkHVYCSikJCNJ+LrltbkJk=
  2. Create an encryption provider configuration file named encryption-provider-config.yml as follows:

    apiVersion: apiserver.config.k8s.io/v1
    kind: EncryptionConfiguration
    resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            keys:
            - name: key1
              secret: BASE-64-ENCODED-SECRET
    
    • Where BASE-64-ENCODED-SECRET is the secret you just created.
  3. Create a Kubernetes profile configuration file that uses your encryption provider configuration file to customize the kube-apiserver as follows:

    {
      "name": "profile1",
      "description": "Testing profile one",
      "customizations": [
        {
          "component": "kube-apiserver",
          "arguments": {
          },
          "file-arguments": {
            "encryption-provider-config": "/LOCAL-DIR/encryption-provider-config.yml"
          }
        }
      ]
    }
    
    • Where LOCAL-DIR is the directory containing your encryption provider configuration file.
  4. Use the TKGI CLI to create a Kubernetes profile based on the profile configuration file:

    $ tkgi create-k8s-profile /tmp/profile1.json
    Kubernetes profile profile1 successfully created
    

Create Kubernetes Cluster

Use the TKGI CLI to create a cluster based on the Kubernetes profile created above:

tkgi create-cluster CLUSTER -e EXAMPLE.COM -p small -n 1 --kubernetes-profile K8S-PROFILE
  • Where K8S-PROFILE is the filename of the Kubernetes profile. See tkgi create-cluster in the TKGI CLI topic.
    For example:
    $ tkgi create-cluster cluster1 -e cluster1-internal.com -p small -n 1 --kubernetes-profile profile1
    TKGI Version:              1.8.0-build.8
    Name:                     cluster1
    K8s Version:              1.17.5
    Plan Name:                small
    UUID:                     22f78823-0b70-4684-be2f-8457d6f3b1f1
    Last Action:              CREATE
    Last Action State:        in progress
    Last Action Description:  Creating cluster
    Kubernetes Master Host:   cluster1-internal.com
    Kubernetes Master Port:   8443
    Worker Nodes:             1
    Kubernetes Master IP(s):  In Progress
    Network Profile Name:
    Kubernetes Profile Name:  profile1
    Tags:
    

Running this tkgi create-cluster command sets the --encryption-provider-config flag on the kube-apiserver to point to the encryption provider configuration file, and restarts your kube-apiserver so that the change takes effect.

Verify that Data is Encrypted

After the kube-apiserver restarts, all existing data in its etcd database should be encrypted, and it should encrypt any new data that it stores.

You can check this encryption by storing and retrieving a new secret:

  1. Use the kubectl CLI create secret generic command to create a new secret called secret1 in the default namespace. For example:

    kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
    

  2. Using the etcdctl command line, read that secret out of etcd:

    ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C
    

    • Where […] are the additional arguments for connecting to the etcd server.

    For example, if you SSH into your master node and with admin privileges you could run the following command to get the secret and view it.

    ETCDCTL_API=3 /var/vcap/packages/etcdctl/etcdctl --endpoints https://master-0.etcd.cfcr.internal:2379 --cert /var/vcap/jobs/etcd/config/etcdctl.crt --key /var/vcap/jobs/etcd/config/etcdctl.key --cacert /var/vcap/jobs/etcd/config/etcdctl-ca.crt get /registry/secrets/default/secret1
    

  3. Verify the stored secret is prefixed with k8s:enc:aescbc:v1: which indicates the aescbc provider has encrypted the resulting data.

    master/3f9c5ca9-a2b1-469c-9007-6fdd0844a5ec:/var/vcap/bosh_ssh/bosh_2f4aaa514474422# ETCDCTL_API=3 /var/vcap/packages/etcdctl/etcdctl --endpoints https://master-0.etcd.cfcr.internal:2379 --cert /var/vcap/jobs/etcd/config/etcdctl.crt --key /var/vcap/jobs/etcd/config/etcdctl.key --cacert /var/vcap/jobs/etcd/config/etcdctl-ca.crt get /registry/secrets/default/secret1

    /registry/secrets/default/secret1 k8s:enc:aescbc:v1:key1:����@�8�����A������2cM7������uL�/

  4. Verify the secret is correctly decrypted when retrieved via the API:

    kubectl describe secret secret1 -n default
    
    Should match mykey: bXlkYXRh, mydata is encoded, check decoding a secret to completely decode the secret.

Rotate Encryption Key for Secrets in etcd Database

This section describes how to use encryption provider config to rotate key.

Changing the secret without incurring downtime requires a multi-step operation, especially for a highly-available deployment running multiple kube-apiserver processes.

  1. Generate a new key and add it as the second key entry for the current provider in your encryption-provider-config.yml keys: list:

    apiVersion: apiserver.config.k8s.io/v1
    kind: EncryptionConfiguration
    resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            keys:
            - name: key1
              secret: jHc3NMp7s7T7JoJuZF7NUSkHVYCSikJCNJ+LrltbkJk=
            - name: key2
              secret: MNI7xeE48/1dH+pE5LLHblhId6AjbzdN2I6rubh8AfE=
    
  2. Create a Kubernetes profile configuration file for profile2 that references the modified encryption-provider-config.yml.

    {
      "name": "profile2",
      "description": "Testing profile two",
      "customizations": [
        {
          "component": "kube-apiserver",
          "arguments": { },
          "file-arguments": {
            "encryption-provider-config": "/tmp/encryption-provider-config.yml"
          }
        }
      ]
    }
    
  3. Use the TKGI CLI to create a profile based on the configuration file:

    $ tkgi create-k8s-profile /tmp/profile2.json
    Kubernetes profile profile2 successfully created
    

  4. Update the cluster with this new profile profile2.

    $ tkgi update-cluster cluster1 --kubernetes-profile profile2
    
    Update summary for cluster cluster1:
    Kubernetes Profile Name: profile2
    Are you sure you want to continue? (y/n): y
    Use 'tkgi cluster cluster1' to monitor the state of your cluster
    

    This restarts kube-apiserver processes to ensure each server can decrypt using the new key.

  5. Edit the encryption provider configuration file so that it lists the new key as the first entry in the keys:, swapping its position with the old key.

    apiVersion: apiserver.config.k8s.io/v1
    kind: EncryptionConfiguration
    resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            keys:
            - name: key2
              secret: MNI7xeE48/1dH+pE5LLHblhId6AjbzdN2I6rubh8AfE=
            - name: key1
              secret: jHc3NMp7s7T7JoJuZF7NUSkHVYCSikJCNJ+LrltbkJk=
    
  6. Create a Kubernetes profile configuration file for profile3 that references the modified encryption-provider-config.yml.

    {
      "name": "profile3",
      "description": "Testing profile three",
      "customizations": [
        {
          "component": "kube-apiserver",
          "arguments": { },
          "file-arguments": {
            "encryption-provider-config": "/tmp/encryption-provider-config.yml"
          }
        }
      ]
    }
    
  7. Use the TKGI CLI to create a profile based on the new Kubernetes profile configuration file:

    $ tkgi create-k8s-profile /tmp/profile3.json
    Kubernetes profile profile2 successfully created
    
  8. Update the cluster with the new profile profile3.

    $ tkgi update-cluster cluster1 --kubernetes-profile profile3
    
    Update summary for cluster cluster1:
    Kubernetes Profile Name: profile3
    Are you sure you want to continue? (y/n): y
    Use 'tkgi cluster cluster1' to monitor the state of your cluster
    

    This restarts the kube-apiserver processes to ensure that each server now encrypts using the new key.

  9. To encrypt all existing secrets with the new key, run: kubectl get secrets --all-namespaces -o json | kubectl replace -f -

  10. Back up etcd with the new key.

  11. Edit the encryption-provider-config.yml again to remove the old decryption key from the config keys: list:

    apiVersion: apiserver.config.k8s.io/v1
    kind: EncryptionConfiguration
    resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            Keys:
            - name: key2
              secret: MNI7xeE48/1dH+pE5LLHblhId6AjbzdN2I6rubh8AfE=
    
  12. Create a Kubernetes profile configuration file for profile4 that references the modified encryption-provider-config.yml.

    {
      "name": "profile4",
      "description": "Testing profile four",
      "customizations": [
        {
          "component": "kube-apiserver",
          "arguments": { },
          "file-arguments": {
            "encryption-provider-config": "/tmp/encryption-provider-config.yml"
          }
        }
      ]
    }
    
  13. Use the TKGI CLI to create a profile based on the new Kubernetes profile configuration file:

    $ tkgi create-k8s-profile /tmp/profile4.json
    Kubernetes profile profile4 successfully created
    
  14. Update the cluster with the new profile profile4.

    $ tkgi update-cluster cluster1 --kubernetes-profile profile4
    
    Update summary for cluster cluster1:
    Kubernetes Profile Name: profile4
    Are you sure you want to continue? (y/n): y
    Use 'tkgi cluster cluster1' to monitor the state of your cluster
    
  15. Verify the stored secret is prefixed with k8s:enc:aescbc:v1:key2 which indicates the aescbc provider has encrypted the resulting data.

    master/3f9c5ca9-a2b1-469c-9007-6fdd0844a5ec:/var/vcap/bosh\_ssh/bosh\_2f4aaa514474422\#
    ETCDCTL\_API=3 /var/vcap/packages/etcdctl/etcdctl --endpoints
    https://master-0.etcd.cfcr.internal:2379 --cert
    /var/vcap/jobs/etcd/config/etcdctl.crt --key
    /var/vcap/jobs/etcd/config/etcdctl.key --cacert
    /var/vcap/jobs/etcd/config/etcdctl-ca.crt get
    /registry/secrets/default/secret1
    
    /registry/secrets/default/secret1
    
    k8s:enc:aescbc:v1:key2:����@�8�����A������2cM7������uL�/
    

Decrypt Secrets in the etcd Database

  1. To disable encryption of data at rest, place the identity provider as the first entry in the encryption provider configuration file:

    apiVersion: apiserver.config.k8s.io/v1
    kind: EncryptionConfiguration
    resources:
      - resources:
        - secrets
        providers:
        - identity: {}
        - aescbc:
            keys:
            - name: key1
              secret: <BASE 64 ENCODED SECRET>
    
  2. Create a Kubernetes profile configuration file for profile5 that references the modified encryption-provider-config.yml.

    {
      "name": "profile5",
      "description": "Testing profile five",
      "customizations": [
        {
          "component": "kube-apiserver",
          "arguments": { },
          "file-arguments": {
            "encryption-provider-config": "/tmp/encryption-provider-config.yml"
          }
        }
      ]
    }
    
  3. Use the TKGI CLI to create a profile based on the modified Kubernetes profile configuration file:

    $ tkgi create-k8s-profile /tmp/profile5.json
    Kubernetes profile profile5 successfully created
    
  4. Update the cluster with the new profile profile5:

    $ tkgi update-cluster cluster1 --kubernetes-profile profile5
    
    Update summary for cluster cluster1:
    Kubernetes Profile Name: profile5
    Are you sure you want to continue? (y/n): y
    Use 'tkgi cluster cluster1' to monitor the state of your cluster
    
  5. To force all secrets to be decrypted, run the command: kubectl get secrets --all-namespaces -o json | kubectl replace -f -.


Please send any feedback you have to pks-feedback@pivotal.io.