Backing Up and Restoring in Tanzu MySQL for Kubernetes

This topic describes how to back up and restore VMware Tanzu™ SQL with MySQL for Kubernetes.

Overview

Tanzu MySQL for Kubernetes allows you to generate on-demand backups, configure schedules for automated backups, and restore backups to new TanzuMySQL instances.

For uploading and retrieving backup artifacts, Tanzu MySQL for Kubernetes currently supports S3, Minio, and other S3-compatible storage.

For backing up and restoring, Tanzu MySQL for Kubernetes uses four of the five Custom Resource Definitions (CRDs):

• TanzuMySQLBackup: References a TanzuMySQL backup artifact that exists in an external blobstore such as S3 or Minio. A new TanzuMySQLBackup resource is created every time an on-demand or scheduled backup is generated.

• TanzuMySQLBackupLocation: References an external blobstore and credentials necessary to access the blobstore.

• TanzuMySQLBackupSchedule: Represents a CronJob schedule on which to perform backups.

• TanzuMySQLRestore: References an instance of a restore that was performed. A new TanzuMySQLRestore resource is created every time a restore is performed.

For detailed information about the CRDs, see Controllers and Custom Resource Definitions (CRDs) in Tanzu MySQL for Kubernetes Architecture.

About Synchronization of Backups with the External Blobstore

Tanzu MySQL for Kubernetes syncs TanzuMySQLBackup resources in a Kubernetes cluster with the contents of the external blobstore. The external blobstore is treated as the source of truth. This means that, if a TanzuMySQLBackup resource is deleted on the Kubernetes cluster, but the associated backup artifact still exists in the external blobstore, Tanzu MySQL for Kubernetes re-creates the TanzuMySQLBackup resource to match the contents of the external blobstore.

Back Up Tanzu MySQL for Kubernetes Data

Performing backups for Tanzu MySQL for Kubernetes requires creating a TanzuMySQLBackupLocation resource that references an external blobstore. Both on-demand backups and scheduled backups use the TanzuMySQLBackupLocation to upload backup artifacts to the external blobstore.

Before starting the procedures for backing up a Tanzu MySQL for Kubernetes instance, ensure that you know the configuration details of your external blobstore and how often you want to perform scheduled backups.

Create a TanzuMySQLBackupLocation Resource

The purpose of creating this TanzuMySQLBackupLocation Resource is to configure the namespace with the location of the blobstore and the credentials to access it.

To create a TanzuMySQLBackupLocation resource:

1. Create a backuplocation.yaml file. You can use the template below:

   ---
   apiVersion: mysql.tanzu.vmware.com/v1alpha1
   kind: TanzuMySQLBackupLocation
   metadata:
     name: backuplocation-sample
   spec:
     storage:
       # For S3 or Minio:
       s3:
         bucket: "name-of-bucket"
         bucketPath: "my-backups-prefix/"
         # region: "us-east-1"
         endpoint:  "" # optional, default to AWS
         forcePathStyle: false
         disableSSL: false
         secret:
           name: backuplocation-sample-creds
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: backuplocation-sample-creds
   stringData:
     # S3 Credentials
     accessKeyId: "my-access-key-id"
     secretAccessKey: "my-secret-access-key"
   

2. Edit the backuplocation.yaml file with the configuration for your external blobstore. For an explanation of the properties that you can set in this file, see Properties for the TanzuMySQLBackupLocation Resource and Properties for the Secret.

3. Create the TanzuMySQLBackupLocation resource in the same namespace as the TanzuMySQL instances that you want to back up by running:

   kubectl apply -f backuplocation.yaml -n DEVELOPMENT-NAMESPACE
   

   Where is DEVELOPMENT-NAMESPACE is the namespace for the TanzuMySQL instance.

   For example:

   $ kubectl apply -f backuplocation.yaml -n my-namespace
   tanzumysqlbackuplocation.mysql.tanzu.vmware.com/backuplocation-sample created
   secret/backuplocation-sample-creds configured
   

4. Verify that the TanzuMySQLBackupLocation has been created by running:

   kubectl get tanzumysqlbackuplocation backuplocation-sample \
   -o jsonpath={.spec} -n DEVELOPMENT-NAMESPACE
   

   For example:

   $ kubectl get tanzumysqlbackuplocation backuplocation-sample -o jsonpath={.spec} -n my-namespace
   {
     "storage": {
       "s3": {
         "bucket": "bucket-sample",
         "disableSSL": false,
         "forcePathStyle": false,
         "region": "us-west-1",
         "secret": {
           "name": "backuplocation-sample-creds"
         }
       }
     }
   }
   

Create a TanzuMySQLBackupSchedule Resource

To set a schedule for automatic backups, create a TanzuMySQLBackupSchedule resource:

1. Create a backupschedule.yaml file. You can use the template below:

   ---
   apiVersion: mysql.tanzu.vmware.com/v1alpha1
   kind: TanzuMySQLBackupSchedule
   metadata:
     name: tanzumysqlbackupschedule-sample
   spec:
     backupTemplate:
       spec:
         location:
           name: backuplocation-sample
         cluster:
           name: tanzumysql-sample
     schedule: "@daily"
   

2. Edit the backupschedule.yaml file with the name of the TanzuMySQLBackupLocation resource that you created in Create a TanzuMySQLBackupLocation Resource and the name of the TanzuMySQL instance you want scheduled backups of. For an explanation of the properties that you can set in this file, see Properties for the TanzuMySQLBackupSchedule Resource.

3. Create the TanzuMySQLBackupSchedule resource in the same namespace as the TanzuMySQLBackupLocation and TanzuMySQL instance that you referenced in the TanzuMySQLBackupSchedule YAML file.

   kubectl apply -f backupschedule.yaml -n DEVELOPMENT-NAMESPACE
   

   For example:

   $ kubectl apply -f backupschedule.yaml -n my-namespace
   tanzumysqlbackupschedule.mysql.tanzu.vmware.com/backupschedule-sample created
   

4. Verify that the TanzuMySQLBackupSchedule has been created by running:

   kubectl get tanzumysqlbackupschedule tanzumysqlbackupschedule-sample -o jsonpath={.spec} -n DEVELOPMENT-NAMESPACE
   

   For example:

   $ kubectl get tanzumysqlbackupschedule tanzumysqlbackupschedule-sample -o jsonpath={.spec} -n my-namespace
   {
     "backupTemplate": {
       "spec": {
         "cluster": {
           "name": "demo-db"
         },
         "location": {
           "name": "demo-backuplocation"
         }
       }
     },
     "schedule": "@daily"
   }
   

   If you correctly configured both a TanzuMySQLBackupLocation resource and TanzuMySQLBackupSchedule resource for an existing TanzuMySQL instance, you see backups being generated and uploaded to the external blobstore.

Name and Location for Backup Artifacts

TanzuMySQLBackup resources that are automatically generated as a result of a TanzuMySQLBackupSchedule are named SCHEDULE-NAME-TIMESTAMP.

By default, Tanzu MySQL for Kubernetes stores backup artifacts under the subfolder structure yyyy > mm > dd. You can configure a custom path for backups so that backup artifacts are stored under the subfolder structure CUSTOM-PATH > yyyy > mm > dd.

Backup artifacts stored in the external blobstore are named DATETIME-RANDOM_STRING-backup.xb.

For example, if a TanzuMySQLBackupSchedule name is tanzumysqlbackupschedule-sample, the custom backup path is my-backups/, and a backup was taken on Thursday, December 10, 2020 at 8:51:03 PM GMT (timestamp 1607633463), then:

• The TanzuMySQLBackup resource on the Kubernetes cluster is named tanzumysqlbackupschedule-sample-1607633463
• The backup artifact in the external blobstore is named 20201210T205103-kzw54l-backup.xb
• The path to the artifact is my-backups/2020/12/10/.

Perform an On-Demand Backup

In addition to scheduled backups, you can take individual backups whenever you want.

Prerequisite: A TanzuMySQLBackupLocation resource that represents the external blobstore to which you upload the generated backup artifact. To configure the TanzuMySQLBackupLocation resource, see Create a TanzuMySQLBackupLocation Resource above.

To take a backup:

1. Create a backup.yaml file. You can use the template below:

   ---
   apiVersion: mysql.tanzu.vmware.com/v1alpha1
   kind: TanzuMySQLBackup
   metadata:
     name: backup-sample
   spec:
     location:
       name: backuplocation-sample
     cluster:
       name: tanzumysql-sample
   

   For an explanation of the properties that you can set in this file, see Properties for the TanzuMySQLBackup Resource.

2. Trigger the backup by creating the TanzuMySQLBackup resource in the same namespace as the instance by running:

   kubectl apply -f tanzumysqlbackup.yaml -n DEVELOPMENT-NAMESPACE
   

   For example:

   $ kubectl apply -f tanzumysqlbackup.yaml -n my-namespace
   tanzumysqlbackup.mysql.tanzu.vmware.com/backup-sample created
   

3. Verify that a backup has been generated and track its progress by running:

   kubectl get tanzumysqlbackup backup-sample -n DEVELOPMENT-NAMESPACE
   

   For example:

   $ kubectl get tanzumysqlbackup backup-sample -n my-namespace
   NAME            STATUS      TIME SCHEDULED                TIME STARTED                  TIME COMPLETED                AGE
   backup-sample   Succeeded   2020-12-01T21:49:26.138676Z   2020-12-01T21:49:26.148835Z   2020-12-01T21:49:30.609250Z   16m
   

   For an explanation of what each column means, see List Existing TanzuMySQLBackup Resources below.

List Existing TanzuMySQLBackup Resources

You might want to list existing TanzuMySQLBackup resources for various reasons, for example:

• To select a backup to restore. For steps to restore a backup, see Restore.
• To see the last successful backup.
• To verify that scheduled backups are running as expected.
• To find old backups that need to be cleaned up. For steps to delete backups, see Delete Old Backup Artifacts.

To see a list of existing TanzuMySQLBackup resources:

1. List existing TanzuMySQLBackup resources by running:

   kubectl get tanzumysqlbackup
   

   For example:

   $ kubectl get tanzumysqlbackup
   NAME            STATUS   TIME SCHEDULED                TIME STARTED   TIME COMPLETED   AGE
   backup-sample   Failed   2020-12-09T18:28:23.339156Z                                   23h
   

2. To understand the output, see the table below:

   Column Name	Meaning
   STATUS	Represents the current status of the backup. Allowed values are: Pending: The backup has been received but not scheduled on a TanzuMySQL Pod. Scheduled: The backup has been scheduled on a selected TanzuMySQL Pod but has not started. Running: The backup is being generated and streamed to the external blobstore. Succeeded: The backup has completed successfully. Failed: The backup has failed to complete. To troubleshoot a failed backup, see Troubleshoot Backup and Restore below.
   TIME SCHEDULED	The time at which the backup was scheduled on a selected TanzuMySQL Pod.
   TIME STARTED	The time that the backup process started.
   TIME COMPLETED	The time that the backup process finished. If the backup fails, this value is empty.
   AGE	The time since the backup was requested.

Delete Old Backup Artifacts

Tanzu MySQL for Kubernetes does not natively support retention policies for backup artifacts. You can configure retention policies on your external blobstore. If you do, you must also delete the associated TanzuMySQLBackup resources in the Kubernetes cluster, because those are not automatically deleted by Tanzu MySQL for Kubernetes.

To delete a backup:

1. Delete the backup in the external blobstore.

2. On your Kubernetes cluster, delete the TanzuMySQLBackup resource by running:

   kubectl delete tanzumysqlbackup BACKUP-NAME -n DEVELOPER-NAMESPACE
   

   For example:

   kubectl delete tanzumysqlbackup backup-sample -n my-namespace
   

Restore

This section discusses two kinds of restoring:

• Restore from a Backup
• Restore a Backup to a Different Namespace or Kubernetes Cluster

Restore from a Backup

TanzuMySQLRestores always restores to a new TanzuMySQL instance to avoid overwriting any data on an existing TanzuMySQL instance. The TanzuMySQL instance is created automatically when the restore is triggered.

Tanzu MySQL for Kubernetes does not allow you to restore a backup to an existing TanzuMySQL instance. Although you can perform this manually by copying the MySQL data from the backup artifact onto an existing TanzuMySQL instance, VMware strongly discourages you from doing this because you might overwrite existing data on the TanzuMySQL instance.

Prerequisites

Before you restore from a backup, you must have:

• An existing TanzuMySQLBackup in your current namespace. To select an existing TanzuMySQLBackup to restore, see List Existing TanzuMySQLBackup Resources.
• A TanzuMySQLBackupLocation that represents the bucket where the existing backup artifact is stored. See Create a TanzuMySQLBackupLocation Resource above.

Procedure

To restore from a backup:

1. Create a tanzumysqlrestore.yaml file. You can use the template below:

   ---
   apiVersion: mysql.tanzu.vmware.com/v1alpha1
   kind: TanzuMySQLRestore
   metadata:
     name: tanzumysqlrestore-sample
   spec:
     tanzuMysqlBackup:
       name: backup-sample
     tanzuMysqlTemplate:
       metadata:
         name: tanzumysql-sample
       spec:
         storageSize: 1Gi
         imagePullSecret: tanzu-mysql-image-registry
   

   For an explanation of the properties that you can set in this file, see Properties for the TanzuMySQLRestore Resource.

2. Trigger the restore by creating the TanzuMySQLRestore resource in the same namespace as the TanzuMySQLBackup and TanzuMySQLBackupLocation by running:

   kubectl apply -f tanzumysqlrestore.yaml -n DEVELOPMENT-NAMESPACE
   

   For example:

   $ kubectl apply -f tanzumysqlrestore.yaml -n my-namespace
   tanzumysqlrestore.mysql.tanzu.vmware.com/tanzumysqlrestore-sample created
   

3. Verify that a restore has been triggered and track the progress of your restore by running:

   kubectl get tanzumysqlrestore tanzumysqlrestore-sample -n DEVELOPMENT-NAMESPACE
   

   For example:

   $ kubectl get tanzumysqlrestore tanzumysqlrestore-sample -n my-namespace
   NAME                       STATUS      TIME STARTED                  TIME COMPLETED                AGE
   tanzumysqlrestore-sample   Succeeded   2020-12-01T21:52:30.000000Z   2020-12-01T21:53:09.163336Z   13m
   

4. To understand the output, see the table below:

   Column Name	Meaning
   STATUS	Represents the current status of the restore process. Allowed values are: Pending: The restore has been received but not yet scheduled on a TanzuMySQL Pod. Scheduled: The restore has been scheduled but has not started. Running: The restore is in progress. Succeeded: The restore has completed successfully. Failed: The restore failed. To troubleshoot, see Troubleshoot Backup and Restore below.
   TIME STARTED	The time that the restore process started.
   TIME COMPLETED	The time that the restore process finished. If the restore fails, this value is empty.
   AGE	The time since the restore was requested.

Restoring a Backup to a Different Namespace or Kubernetes Cluster

If you want to restore a backup to a different namespace or a different Kubernetes cluster, create a TanzuMySQLBackupLocation in the target namespace or Kubernetes cluster. Then, Tanzu MySQL for Kubernetes automatically creates TanzuMySQLBackup resources for the backup artifacts in the external blobstore.

To restore to a different namespace or Kubernetes cluster, you create a BackupLocation in the target namespace:

1. Target the destination cluster or namespace.

2. Create a TanzuMySQLBackupLocation resource that contains the backup artifact to restore. For how to do this, see Create a TanzuMySQLBackupLocation Resource.

3. Confirm that the TanzuMySQLBackup artifact to restore is included in the list by running:

   kubectl get tanzumysqlbackup
   

   For example:

   $ kubectl get tanzumysqlbackup
    NAME            STATUS      TIME SCHEDULED                TIME STARTED                  TIME COMPLETED                AGE
    sample-backup   Succeeded   2020-12-01T21:49:26.138676Z   2020-12-01T21:49:26.148835Z   2020-12-01T21:49:30.609250Z   5d17h
   

4. Trigger a restore by following steps in Restore from a Backup.

Troubleshoot Backup and Restore

Basic troubleshooting begins with reviewing the status for the resource and reading the messages associated with the resource events.

To troubleshoot problems with backup and restore:

1. Detect issues by monitoring the STATUS column of any Tanzu MySQL custom resource. If the status is Failed or is stuck in Pending, Scheduled, or Running, then one of the following might be the problem:

   • Misconfiguration
   • Problem with the external blobstore
   • Issues with the TanzuMySQL Operator

   In this example, the kubectl get command outputs a Failed status:

   $ kubectl get tanzumysqlbackup backup-sample
   NAME            STATUS   TIME SCHEDULED                TIME STARTED   TIME COMPLETED   AGE
   backup-sample   Failed   2020-12-09T18:28:23.339156Z                                   2m44s
   

2. Diagnose the issue by inspecting the Kubernetes events for the resource, for example:

   $ kubectl get events --field-selector involvedObject.name=backup-sample
   LAST SEEN   TYPE      REASON   OBJECT                           MESSAGE
   2m43s       Warning   Failed   tanzumysqlbackup/backup-sample   Secret "backuplocation-sample-creds" not found
   

3. Read the message in the MESSAGE column to understand why the failure occurred.
   
   

   In the example above, the backup-sample expected a Kubernetes secret called backuplocation-sample-creds to exist. Fix this problem by creating the backuplocation-sample-creds secret. The template for this secret is located in the backuplocation.yaml template.