Backup and Restore StatefulSet App with Namespace

Page last updated:

This topic describes how to use Velero to backup and restore a statefulset application with namepsace.

Overview

This example demonstrates Velero backup and restore for a statefulset application with namespace. The CassandraDB app is used for demonstrating backup and restore with Velero.

When restoring a stateful application using Velero, the Storage Class that was used by the PVC in the application must be present on the Kubernetes cluster. If the PVC was using the default storage class, then the default storage class must also be present prior to initiating the restore operation with Velero.

Prerequisites

Install and configure Minio, Velero, and Restic.

The application we are going to use is the CassandraDB statefulset app. Download the CassandraDB YAML files to a local known directory:

  • cassandra-statefulset.yaml
  • cassandra-storageclass.yaml
  • headless-cassandra-service.yaml

Create the Storage Class

Modify the storage class cassandra-storageclass.yaml:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: demo-sts-sc
provisioner: kubernetes.io/vsphere-volume
parameters:
  diskformat: thin

Apply the storage class YAML file:

kubectl apply -f cassandra-storageclass.yaml

storageclass.storage.k8s.io/demo-sts-sc created

Verify the storage class:

kubectl get sc
NAME          PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE

demo-sts-sc   kubernetes.io/vsphere-volume   Delete          Immediate           false                  3s

Deploy CassandraDB App

Create the namespace:

kubectl create ns cassandra

namespace/cassandra created

Create the service:

kubectl apply -f headless-cassandra-service.yaml -n cassandra

service/cassandra created

Deploy CassandraDB app:

kubectl apply -f cassandra-statefulset.yaml -n cassandra

statefulset.apps/cassandra created

Verify:

kubectl get all -n cassandra

NAME              READY   STATUS    RESTARTS   AGE
pod/cassandra-0   1/1     Running   0          5m16s
pod/cassandra-1   1/1     Running   0          4m25s
pod/cassandra-2   1/1     Running   0          2m52s

NAME                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/cassandra   ClusterIP   None         <none>        <none>    5m21s

NAME                         READY   AGE
statefulset.apps/cassandra   3/3     5m16s
kubectl get pvc,pv -n cassandra

NAME                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/cassandra-data-cassandra-0   Bound    pvc-394770b3-64ba-4c35-af0c-f76a52695b25   100Gi      RWO            demo-sts-sc    5m46s
persistentvolumeclaim/cassandra-data-cassandra-1   Bound    pvc-5e0e4935-472e-46de-bba2-91d06d301f3b   100Gi      RWO            demo-sts-sc    4m55s
persistentvolumeclaim/cassandra-data-cassandra-2   Bound    pvc-cc4d55c4-b794-4a9f-871f-365ec0c3ad50   100Gi      RWO            demo-sts-sc    3m22s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                  STORAGECLASS   REASON   AGE
persistentvolume/pvc-394770b3-64ba-4c35-af0c-f76a52695b25   100Gi      RWO            Delete           Bound    cassandra/cassandra-data-cassandra-0   demo-sts-sc             5m46s
persistentvolume/pvc-5e0e4935-472e-46de-bba2-91d06d301f3b   100Gi      RWO            Delete           Bound    cassandra/cassandra-data-cassandra-1   demo-sts-sc             4m54s
persistentvolume/pvc-cc4d55c4-b794-4a9f-871f-365ec0c3ad50   100Gi      RWO            Delete           Bound    cassandra/cassandra-data-cassandra-2   demo-sts-sc             3m22s

Make sure CassandraDB instances are fully participating in the cluster:

kubectl exec -it cassandra-0 -n cassandra -- nodetool status

Datacenter: Demo-DataCenter
===========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.16.1.2  104.42 KiB  32           69.1%             6670d501-ce1a-41a6-b0ab-e5654be90d05  Demo-Rack
UN  172.16.1.3  75.87 KiB  32           60.8%             d4c65f54-6ba0-4caa-8b0c-9a2b945cb05c  Demo-Rack
UN  172.16.1.4  81.09 KiB  32           70.1%             fb0cca97-eb35-4e69-ad87-09ab18f739b2  Demo-Rack

Create and Populate a Database and Table in Cassandra

Create the DB:

kubectl exec -it cassandra-0 -n cassandra -- cqlsh

Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.

Create and populate table:

cqlsh> CREATE KEYSPACE demodb WITH REPLICATION = { ‘class’ : 'SimpleStrategy’, 'replication_factor’ : 3 }; cqlsh> use demodb; cqlsh:demodb> CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint); cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (100, 'Tom’, 'Cork’, 999, 1000000); cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (101, 'Andrew’, 'NY’, 1000, 1000000); cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (102, 'Lara’, 'Paris’, 1001, 1000000); cqlsh:demodb> select * from emp; “`

Verify:

 emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
    100 |     Cork |      Tom |       999 | 1000000
    102 |    Paris |     Lara |      1001 | 1000000
    101 |       NY |   Andrew |      1000 | 1000000

(3 rows)

Verify that the other Cassandra DB instances have the same information:

kubectl exec -it cassandra-1 -n cassandra -- cqlsh

Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> use demodb;
cqlsh:demodb> select * from emp;

 emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
    100 |     Cork |      Tom |       999 | 1000000
    102 |    Paris |     Lara |      1001 | 1000000
    101 |       NY |   Andrew |      1000 | 1000000

(3 rows)
kubectl exec -it cassandra-2 -n cassandra -- cqlsh

Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> use demodb;
cqlsh:demodb> select * from emp;

 emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
    100 |     Cork |      Tom |       999 | 1000000
    102 |    Paris |     Lara |      1001 | 1000000
    101 |       NY |   Andrew |      1000 | 1000000

(3 rows)

Add Annotations

Add annotations for the stateful pods with the volume name cassandra-data.

kubectl get pod -n cassandra

NAME          READY   STATUS    RESTARTS   AGE
cassandra-0   1/1     Running   0          19m
cassandra-1   1/1     Running   0          19m
cassandra-2   1/1     Running   0          17m

The pods cassandra-0, cassandra-1 and cassandra-2 must be annotated with cassandra-data.

kubectl -n cassandra annotate pod/cassandra-0 backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-0 annotated

kubectl -n cassandra annotate pod/cassandra-1 backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-1 annotated

kubectl -n cassandra annotate pod/cassandra-2 backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-2 annotated

Verify the annotations:

kubectl -n cassandra describe pod/cassandra-0 | grep Annotations
Annotations:  backup.velero.io/backup-volumes: cassandra-data

kubectl -n cassandra describe pod/cassandra-1 | grep Annotations
Annotations:  backup.velero.io/backup-volumes: cassandra-data

kubectl -n cassandra describe pod/cassandra-2 | grep Annotations
Annotations:  backup.velero.io/backup-volumes: cassandra-data

Backup the CassandraDB App using Namespace

Perform the Velero backup:

velero backup create cassandra-backup --include-namespaces cassandra

Backup request "cassandra-backup" submitted successfully.
Run `velero backup describe cassandra-backup` or `velero backup logs cassandra-backup` for more details.

Verify the backup:

velero backup create cassandra-backup --include-namespaces cassandra

Backup request "cassandra-backup" submitted successfully.
Run `velero backup describe cassandra-backup` or `velero backup logs cassandra-backup` for more details.
velero backup describe cassandra-backup --details
Name:         cassandra-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.17.8+vmware.1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=17

Phase:  Completed

...

Velero-Native Snapshots: <none included>

Restic Backups:
  Completed:
    cassandra/cassandra-0: cassandra-data
    cassandra/cassandra-1: cassandra-data
    cassandra/cassandra-2: cassandra-data

Get the backup:

velero backup get

NAME               STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
cassandra-backup   Completed   0        0          2020-07-29 08:26:18 -0700 PDT   29d       default            <none>

Use the Velero CRD commands to verify:

kubectl get crd
kubectl get backups.velero.io -n velero
NAME               AGE
cassandra-backup   94s
kubectl describe backups.velero.io cassandra-backup -n velero

Restore the CassandraDB App

Restore the CassandraDB app from the Velero backup. Note the following about the restore operation:

  • Pod annotation is still required for Velero backup of PV
  • The namespace 'cassandra’ was automatically re-created

Delete the namespace:

kubectl delete ns cassandra
namespace "cassandra" deleted

Verify that Cassandra resources are deleted:

kubectl get ns

NAME              STATUS   AGE
default           Active   21d
kube-node-lease   Active   21d
kube-public       Active   21d
kube-system       Active   21d
pks-system        Active   21d
velero            Active   7d18h
kubectl get pvc,pv --all-namespaces
No resources found

Verify that the storage class used by the application is still present:

kubectl get sc

NAME          PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
demo-sts-sc   kubernetes.io/vsphere-volume   Delete          Immediate           false                  36m

Restore the CassandraDB app from the backup:

velero restore create --from-backup cassandra-backup

Restore request "cassandra-backup-20200729083742" submitted successfully.
Run `velero restore describe cassandra-backup-20200729083742` or `velero restore logs cassandra-backup-20200729083742` for more details.

Verify that the app is restored:

velero restore describe cassandra-backup-20200729083742
Name:         cassandra-backup-20200729083742
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  Completed

Backup:  cassandra-backup

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Restic Restores (specify --details for more information):
  Completed:  3
velero restore get

NAME                              BACKUP             STATUS      ERRORS   WARNINGS   CREATED                         SELECTOR
cassandra-backup-20200729083742   cassandra-backup   Completed   0        0          2020-07-29 08:37:42 -0700 PDT   <none>
kubectl get ns

NAME              STATUS   AGE
cassandra         Active   80s
default           Active   21d
kube-node-lease   Active   21d
kube-public       Active   21d
kube-system       Active   21d
pks-system        Active   21d
velero            Active   7d18h
kubectl get all -n cassandra

NAME              READY   STATUS    RESTARTS   AGE
pod/cassandra-0   1/1     Running   0          2m6s
pod/cassandra-1   1/1     Running   2          2m6s
pod/cassandra-2   1/1     Running   2          2m6s

NAME                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/cassandra   ClusterIP   None         <none>        <none>    2m6s

NAME                         READY   AGE
statefulset.apps/cassandra   3/3     2m6s

Verify the persistent volume:

kubectl get pvc,pv -n cassandra

Check the content of each Cassandra DB instance:

kubectl exec -it cassandra-0 -n cassandra -- cqlsh

Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> use demodb;
cqlsh:demodb> select * from emp;

 emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
    100 |     Cork |      Tom |       999 | 1000000
    102 |    Paris |     Lara |      1001 | 1000000
    101 |       NY |   Andrew |      1000 | 1000000

(3 rows)

kubectl exec -it cassandra-1 -n cassandra – cqlsh

Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp;

emp_id emp_city emp_name emp_phone emp_sal
100 Cork Tom 999 1000000
102 Paris Lara 1001 1000000
101 NY Andrew 1000 1000000

(3 rows) ”`

kubectl exec -it cassandra-2 -n cassandra -- cqlsh

Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> use demodb;
cqlsh:demodb> select * from emp;

 emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
    100 |     Cork |      Tom |       999 | 1000000
    102 |    Paris |     Lara |      1001 | 1000000
    101 |       NY |   Andrew |      1000 | 1000000

(3 rows)

Please send any feedback you have to pks-feedback@pivotal.io.