Backing Up and Restoring MySQL Instances
This topic describes how to back up and restore VMware Tanzu™ SQL with MySQL for Kubernetes.
Overview
Tanzu MySQL for Kubernetes allows you to generate on-demand backups, configure schedules for automated backups, and restore backups to new MySQL instances.
For uploading and retrieving backup artifacts, Tanzu MySQL for Kubernetes currently supports S3, Minio, and other S3-compatible storage.
For backing up and restoring, Tanzu MySQL for Kubernetes uses four of the five Custom Resource Definitions (CRDs):
MySQLBackup: References a MySQL backup artifact that exists in an external blobstore such as S3 or Minio. A new MySQLBackup resource is created every time an on-demand or scheduled backup is generated.
MySQLBackupLocation: References an external blobstore and credentials necessary to access the blobstore.
MySQLBackupSchedule: Represents a CronJob schedule on which to perform backups.
MySQLRestore: References an instance of a restore that was performed. A new MySQLRestore resource is created every time a restore is performed.
For detailed information about the CRDs, see Controllers and Custom Resource Definitions (CRDs) in Architecture.
About Synchronization of Backups with the External Blobstore
Tanzu MySQL for Kubernetes syncs MySQLBackup resources in a Kubernetes cluster with the
contents of the external blobstore.
The external blobstore is treated as the source of truth.
This means that, if a MySQLBackup
resource is deleted on the Kubernetes cluster, but the
associated backup artifact still exists in the external blobstore, Tanzu MySQL for Kubernetes
re-creates the MySQLBackup
resource to match the contents of the external
blobstore.
Back Up Tanzu MySQL for Kubernetes Data
Performing backups for Tanzu MySQL for Kubernetes requires creating a MySQLBackupLocation resource that references an external blobstore. Both on-demand backups and scheduled backups use the MySQLBackupLocation to upload backup artifacts to the external blobstore.
Before starting the procedures for backing up a Tanzu MySQL for Kubernetes instance, ensure that you know the configuration details of your external blobstore and how often you want to perform scheduled backups.
Create a MySQLBackupLocation Resource
The purpose of creating this MySQLBackupLocation Resource is to configure the namespace with the location of the blobstore and the credentials to access it.
To create a MySQLBackupLocation resource:
Find the
backuplocation.yaml
deployment template that you downloaded in the Tanzu MySQL Deployment Templates TGZ file from VMware Tanzu Network. For how to download deployment templates, see Download the Resources in Installing the Tanzu SQL for Kubernetes Operator.Create a copy of the
backuplocation.yaml
file and give it a unique name.For example:
$ cp ~/Downloads/tanzu-mysql-deployment-templates-1.0.0/samples/backuplocation.yaml testbackuplocation.yaml
Edit the file with the configuration for your external blobstore. For an explanation of the properties that you can set in this file, see Properties for the MySQLBackupLocation Resource and Properties for the Secret.
Create the MySQLBackupLocation resource in the same namespace as the MySQL instances that you want to back up by running:
kubectl apply -f FILENAME -n DEVELOPMENT-NAMESPACE
- Where is
DEVELOPMENT-NAMESPACE
is the namespace for the MySQL instance. - Where
FILENAME
is the name of the configuration file you created in Step 2 above.
For example:
$ kubectl apply -f testbackuplocation.yaml -n my-namespace mysqlbackuplocation.with.sql.tanzu.vmware.com/backuplocation-sample created secret/backuplocation-sample-creds configured
- Where is
Verify that the MySQLBackupLocation has been created by running:
kubectl get mysqlbackuplocation backuplocation-sample \ -o jsonpath={.spec} -n DEVELOPMENT-NAMESPACE
For example:
$ kubectl get mysqlbackuplocation backuplocation-sample -o jsonpath={.spec} -n my-namespace { "storage": { "s3": { "bucket": "bucket-sample", "forcePathStyle": false, "region": "us-west-1", "secret": { "name": "backuplocation-sample-creds" } } } }
Create a MySQLBackupSchedule Resource
To set a schedule for automatic backups, create a MySQLBackupSchedule resource:
Find the
backupschedule.yaml
deployment template that you downloaded in the Tanzu MySQL Deployment Templates TGZ file from VMware Tanzu Network. For how to download deployment templates, see Download the Resources in Installing the Tanzu SQL for Kubernetes Operator.Create a copy of the
backupschedule.yaml
file and give it a unique name.For example:
$ cp ~/Downloads/tanzu-mysql-deployment-templates-1.0.0/samples/backupschedule.yaml testbackupschedule.yaml
Edit the file with the name of the MySQLBackupLocation resource that you created in Create a MySQLBackupLocation Resource and the name of the MySQL instance you want scheduled backups of. For an explanation of the properties that you can set in this file, see Properties for the MySQLBackupSchedule Resource.
Create the MySQLBackupSchedule resource in the same namespace as the MySQLBackupLocation and MySQL instance that you referenced in the MySQLBackupSchedule YAML file.
kubectl apply -f FILENAME -n DEVELOPMENT-NAMESPACE
Where
FILENAME
is the name of the configuration file you created in Step 2 above.For example:
$ kubectl apply -f testbackupschedule.yaml -n my-namespace mysqlbackupschedule.with.sql.tanzu.vmware.com/backupschedule-sample created
Verify that the MySQLBackupSchedule has been created by running:
kubectl get mysqlbackupschedule mysqlbackupschedule-sample -o jsonpath={.spec} -n DEVELOPMENT-NAMESPACE
For example:
$ kubectl get mysqlbackupschedule mysqlbackupschedule-sample -o jsonpath={.spec} -n my-namespace { "backupTemplate": { "spec": { "instance": { "name": "demo-db" }, "location": { "name": "demo-backuplocation" } } }, "schedule": "@daily" }
If you correctly configured both a
MySQLBackupLocation
resource andMySQLBackupSchedule
resource for an existing MySQL instance, you see backups being generated and uploaded to the external blobstore.
Name and Location for Backup Artifacts
MySQLBackup resources that are automatically generated as a result of a MySQLBackupSchedule are named
SCHEDULE-NAME-TIMESTAMP
.
By default, Tanzu MySQL for Kubernetes stores backup artifacts under the subfolder structure yyyy > mm > dd
.
You can configure a custom path for backups so that backup artifacts are stored under the subfolder
structure CUSTOM-PATH > yyyy > mm > dd
.
Backup artifacts stored in the external blobstore are named DATETIME-RANDOM_STRING-backup.xb
.
For example, if a MySQLBackupSchedule name is mysqlbackupschedule-sample
, the custom backup
path is my-backups/
, and a backup was taken on Thursday, December 10, 2020 at 8:51:03 PM GMT
(timestamp 1607633463
), then:
- The MySQLBackup resource on the Kubernetes cluster is named
mysqlbackupschedule-sample-1607633463
- The backup artifact in the external blobstore is named
20201210T205103-kzw54l-backup.xb
- The path to the artifact is
my-backups/2020/12/10/
.
Perform an On-Demand Backup
In addition to scheduled backups, you can take individual backups whenever you want.
Prerequisite: A MySQLBackupLocation resource that represents the external blobstore to which you upload the generated backup artifact. To configure the MySQLBackupLocation resource, see Create a MySQLBackupLocation Resource above.
To take a backup:
Find the
backup.yaml
deployment template that you downloaded in the Tanzu MySQL Deployment Templates TGZ file from VMware Tanzu Network. For how to download deployment templates, see Download the Resources in Installing the Tanzu SQL for Kubernetes Operator.Create a copy of the
backup.yaml
file and give it a unique name.For example:
$ cp ~/Downloads/tanzu-mysql-deployment-templates-1.0.0/samples/backup.yaml testbackup.yaml
Edit the file. For an explanation of the properties that you can set for the MySQLBackup resource, see Properties for the MySQLBackup Resource.
Trigger the backup by creating the MySQLBackup resource in the same namespace as the instance by running:
kubectl apply -f FILENAME -n DEVELOPMENT-NAMESPACE
Where
FILENAME
is the name of the configuration file you created in Step 2 above.For example:
$ kubectl apply -f testbackup.yaml -n my-namespace mysqlbackup.with.sql.tanzu.vmware.com/backup-sample created
Verify that a backup has been generated and track its progress by running:
kubectl get mysqlbackup backup-sample -n DEVELOPMENT-NAMESPACE
For example:
$ kubectl get mysqlbackup backup-sample -n my-namespace NAME STATUS SOURCE INSTANCE TIME STARTED TIME COMPLETED backup-sample Succeeded mysql-sample 2020-12-01T21:49:26Z 2020-12-01T21:49:30Z
For an explanation of what each column means, see List Existing MySQLBackup Resources below.
List Existing MySQLBackup Resources
You might want to list existing MySQLBackup resources for various reasons, for example:
- To select a backup to restore. For steps to restore a backup, see Restore.
- To see the last successful backup.
- To verify that scheduled backups are running as expected.
- To find old backups that need to be cleaned up. For steps to delete backups, see Delete Old Backup Artifacts.
To see a list of existing MySQLBackup resources:
List existing MySQLBackup resources by running:
kubectl get mysqlbackup
For example:
$ kubectl get mysqlbackup NAME STATUS SOURCE INSTANCE TIME STARTED TIME COMPLETED backup-sample Failed mysql-sample
To understand the output, see the table below:
Column Name Meaning STATUS
Represents the current status of the backup. Allowed values are: - Pending: The backup has been received but not scheduled on a MySQL Pod.
- Running: The backup is being generated and streamed to the external blobstore.
- Succeeded: The backup has completed successfully.
- Failed: The backup has failed to complete. To troubleshoot a failed backup, see Troubleshoot Backup and Restore below.
SOURCE INSTANCE
The MySQL instance the backup was taken from. TIME STARTED
The time that the backup process started. TIME COMPLETED
The time that the backup process finished. If the backup fails, this value is empty.
Delete Old Backup Artifacts
Tanzu SQL for Kubernetes does not natively support retention policies for backup artifacts. You can configure retention policies on your external blobstore. If you do, you must also delete the associated MySQLBackup resources in the Kubernetes cluster, because those are not automatically deleted by Tanzu SQL for Kubernetes.
To delete a backup:
Delete the backup in the external blobstore.
On your Kubernetes cluster, delete the MySQLBackup resource by running:
kubectl delete mysqlbackup BACKUP-NAME -n DEVELOPMENT-NAMESPACE
For example:
kubectl delete mysqlbackup backup-sample -n my-namespace
Restore
This section discusses two kinds of restoring:
Restore from a Backup
MySQLRestores always restores to a new MySQL instance to avoid overwriting any data on an existing MySQL instance. The MySQL instance is created automatically when the restore is triggered.
Tanzu MySQL for Kubernetes does not allow you to restore a backup to an existing MySQL instance. Although you can perform this manually by copying the MySQL data from the backup artifact onto an existing MySQL instance, VMware strongly discourages you from doing this because you might overwrite existing data on the MySQL instance.
Prerequisites
Before you restore from a backup, you must have:
- An existing MySQLBackup in your current namespace. To select an existing MySQLBackup to restore, see List Existing MySQLBackup Resources.
- A MySQLBackupLocation that represents the bucket where the existing backup artifact is stored. See Create a MySQLBackupLocation Resource above.
Procedure
To restore from a backup:
Find the
restore.yaml
deployment template that you downloaded in the Tanzu MySQL Deployment Templates TGZ file from VMware Tanzu Network. For how to download deployment templates, see Download the Resources in Installing the Tanzu SQL for Kubernetes Operator.Create a copy of the
restore.yaml
file and give it a unique name.For example:
$ cp ~/Downloads/tanzu-mysql-deployment-templates-1.0.0/samples/restore.yaml testrestore.yaml
Edit the file. For information about the properties that you can set for the MySQLRestore resource, see Property Reference for Backup and Restore.
Trigger the restore by creating the MySQLRestore resource in the same namespace as the MySQLBackup and MySQLBackupLocation by running:
kubectl apply -f FILENAME -n DEVELOPMENT-NAMESPACE
Where
FILENAME
is the name of the configuration file you created in Step 2 above.For example:
$ kubectl apply -f testrestore.yaml -n my-namespace mysqlrestore.with.sql.tanzu.vmware.com/restore-sample created
Verify that a restore has been triggered and track the progress of your restore by running:
kubectl get mysqlrestore restore-sample -n DEVELOPMENT-NAMESPACE
For example:
$ kubectl get mysqlrestore restore-sample -n my-namespace NAME STATUS SOURCE BACKUP TARGET INSTANCE TIME STARTED TIME COMPLETED restore-sample Succeeded backup-sample mysql-sample 2020-12-01T21:52:30Z 2020-12-01T21:53:09Z
To understand the output, see the table below:
Column Name Meaning STATUS
Represents the current status of the restore process.
Allowed values are:- Pending: The restore has been received but not yet scheduled on a MySQL Pod.
- Running: The restore is in progress.
- Succeeded: The restore has completed successfully.
- Failed: The restore failed. To troubleshoot, see Troubleshoot Backup and Restore below.
SOURCE BACKUP
The name of the backup being restored. TARGET INSTANCE
The name of the new MySQL instance to be restored with the backup contents. TIME STARTED
The time that the restore process started. TIME COMPLETED
The time that the restore process finished. If the restore fails, this value is empty.
Restoring a Backup to a Different Namespace or Kubernetes Cluster
If you want to restore a backup to a different namespace or a different Kubernetes cluster,
create a MySQLBackupLocation
in the target namespace or Kubernetes cluster.
Then, Tanzu MySQL for Kubernetes automatically creates MySQLBackup
resources
for the backup artifacts in the external blobstore.
To restore to a different namespace or Kubernetes cluster, you create a BackupLocation in the target namespace:
Target the destination Kubernetes cluster or namespace.
Create a MySQLBackupLocation resource that contains the backup artifact to restore. For how to do this, see Create a MySQLBackupLocation Resource.
Confirm that the MySQLBackup artifact to restore is included in the list by running:
kubectl get mysqlbackup
For example:
$ kubectl get mysqlbackup NAME STATUS SOURCE INSTANCE TIME STARTED TIME COMPLETED sample-backup Succeeded mysql-sample 2020-12-01T21:49:26Z 2020-12-01T21:49:30Z
Trigger a restore by following steps in Restore from a Backup.
Troubleshoot Backup and Restore
Basic troubleshooting begins with reviewing the status for the resource and reading the messages associated with the resource events.
To troubleshoot problems with backup and restore:
Detect issues by monitoring the
STATUS
column of any MySQL custom resource. If the status isFailed
or is stuck inPending
,Scheduled
, orRunning
, then one of the following might be the problem:- Misconfiguration
- Problem with the external blobstore
- Issues with the MySQL Operator
In this example, the
kubectl get
command outputs aFailed
status:$ kubectl get mysqlbackup backup-sample NAME STATUS SOURCE INSTANCE TIME STARTED TIME COMPLETED backup-sample Failed mysql-sample
Diagnose the issue by inspecting the Kubernetes events for the resource, for example:
$ kubectl get events --field-selector involvedObject.name=backup-sample LAST SEEN TYPE REASON OBJECT MESSAGE 2m43s Warning Failed mysqlbackup/backup-sample Secret "backuplocation-sample-creds" not found
Read the message in the
MESSAGE
column to understand why the failure occurred.In the example above, the
backup-sample
expected a Kubernetes secret calledbackuplocation-sample-creds
to exist. Fix this problem by creating thebackuplocation-sample-creds
secret. The template for this secret is located in thebackuplocation.yaml
template.