Restoring Tanzu Application Service for Kubernetes

This topic provides an overview of how to restore Tanzu Application Service for Kubernetes (TAS for Kubernetes) from a backup.

Overview

When restoring the TAS for Kubernetes database, the data for all the orgs, spaces, and apps are restored.

Database restore is not granular.

To restore TAS for Kubernetes:

Prerequisites

You must have access to the TAS for Kubernetes Velero backup artifacts you plan to restore.

Start TAS for Kubernetes Restore

To restore a TAS for Kubernetes backup:

Prepare the Velero Backup Artifacts

To prepare your TAS for Kubernetes Velero backup artifacts:

  1. Determine the name of the backup to be restored:

    $ velero backup get

  2. (Optional) To see backup metadata for any of the backups, generate a JSON file containing a snapshot of Cloud Foundry at the time of the backup.

    ./tanzu-application-service/config/_ytt_lib/github.com/pivotal/cf-for-k8s-disaster-recovery/backup-metadata/bin/get-backup-metadata.sh BACKUP-NAME OUTPUT-FILENAME

    Where:

    • BACKUP-NAME is the name of the backup you want to validate against.
    • OUTPUT-FILENAME is the name of the backup state JSON file to create.

    For example: 

    $ ./tanzu-application-service/config/ytt_lib/github.com/pivotal/cf-for-k8s-disaster-recovery/backup-metadata/bin/get-backup-metadata.sh backup1 output-file.json

  3. To prepare database resources for a restoration:

    kubectl -n postgres-dbs delete postgres.sql.tanzu.vmware.com/ccdb --wait
kubectl -n postgres-dbs delete pvc --selector postgres-instance=ccdb
kubectl -n postgres-dbs delete postgres.sql.tanzu.vmware.com/uaadb --wait
kubectl -n postgres-dbs delete pvc --selector postgres-instance=uaadb

    The --wait parameter is optional. Include the --wait parameter to follow the restore procedure progress.

    Note: This process deletes some Kubernetes resources, forcing Velero to restore the database Pod’s persistent volume to the backup version. This is required to restore the correct data.

Start a TAS for Kubernetes Velero Restore

To restore TAS for Kubernetes Tanzu Postgres Instances:

  1. To restore Tanzu Postgres Databases:

    velero create restore RESTORE-NAME \
  --from-backup BACKUP-NAME \
  --wait

    Where:

    • BACKUP-NAME is the name of the Velero backup artifact to be restored.
    • RESTORE-NAME is a name you choose to identify this restore attempt.

    The --wait parameter is optional. Include the --wait parameter to follow the restore procedure progress.

Monitor the Restore Progress

To inspect the status of a restore:

Review All Restores

To see the names and status of all restores:

  1. Open a command line.

  2. To display information about all restores:

    velero get restores

    For example:

    $ velero get restores

NAME    STATUS                       CREATED EXPIRES STORAGE LOCATION  SELECTOR
restore_1 Completed 2020-03-24 14:12:29 +0000 UTC    21d          default    <none>

Inspect a Single Restore

To review the status of a single restore:

  1. Determine the name of the restore. For more information, see Review All Restores above.

  2. To display information about a single restore:

    velero describe restore RESTORE-NAME

    For example:

    $ velero describe restore MY_RESTORE_NAME
Note: Add --details flag for highly verbose output

Name:         MY_RESTORE_NAME
Namespace:    default
Labels:       <none>
Annotations:  <none>

Phase:  Completed

Backup:  MY_BACKUP_NAME

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Review the Completed Restore

To review and validate a completed restore:

Review Restore Logs

You can inspect the logs for your restore after the restore has completed. Review the logs for any failures and their root causes.

To review the logs for a particular restore:

  1. Run:

    velero restore logs RESTORE-NAME

    Where RESTORE-NAME is name you provided as your restore name.

Validate the Restore

The Cloud Foundry Metadata reporting tool provides two scripts to capture the Cloud Foundry state from the backup, and then compare it to the live, restored state.

Note: The Cloud Foundry state from the backup is not a complete snapshot and should only be used as an aid to verifying the data. The validation follows the Cloud Foundry constructs of “users, orgs, spaces, and apps” to measure what data was in Cloud Foundry during a back up.

After your restore has completed, follow the steps below to validate the restored data.

Prerequisites

Before validating your backup:

  1. Confirm that the following scripts are in the tanzu-application-service/config/_ytt_lib/github.com/pivotal/cf-for-k8s-disaster-recovery/backup-metadata/bin directory:

    • get-backup-metadata.sh
    • compare-backup-metadata.sh

    If the scripts are not present, download the Tanzu Application Service from VMware Tanzu Network. For more information, see Prepare Your Installation Resources.

  2. Prepare the compare-backup-metadata.sh script for validating your restore:

    1. Open the compare-backup-metadata.sh script.

    2. To specify the container, modify the last line in the script to include -c cf-metadata:

      kubectl ${NAMESPACE} exec -it "${POD_NAME}" -c cf-metadata -- bash -ce 'backup-metadata compare' < "$1"

      Note: You must modify the compare-backup-metadata.sh script to explicitly specify the cf-metadata container because the cf-metadata container is not the default container in the Pod.

Procedure

To validate a completed restore:

  1. To determine the name of the backup used to restore your system:

    velero get backups

    For example:

    $ velero get backups

NAME    STATUS                       CREATED EXPIRES STORAGE LOCATION  SELECTOR
backup_1 Completed 2020-03-24 14:12:29 +0000 UTC     21d          default    <none>

  2. To generate a snapshot of the Cloud Foundry state at the time of the back up:

    tanzu-application-service/config/_ytt_lib/github.com/pivotal/cf-for-k8s-disaster-recovery/backup-metadata/bin/get-backup-metadata.sh BACKUP-NAME OUTPUT-FILENAME

    Where:

    • BACKUP-NAME is the name of the backup you restored.
    • OUTPUT-FILENAME is the name of the Cloud Foundry state file to create for the backup. The created state file is a JSON formatted file.

    For example: 

    $ ./tanzu-application-service/config/ytt_lib/github.com/pivotal/cf-for-k8s-disaster-recovery/backup-metadata/bin/get-backup-metadata.sh backup1 output-file.json

  3. To compare the backup state to the restored system state:

    tanzu-application-service/config/_ytt_lib/github.com/pivotal/cf-for-k8s-disaster-recovery/backup-metadata/bin/compare-backup-metadata.sh OUTPUT-FILENAME NAMESPACE

    Where:

    • OUTPUT-FILENAME is the name of the backup state JSON file created in the previous step.
    • NAMESPACE is the Kubernetes namespace that the cf-metadata pod is running in. The default namespace is cf-system.

      The script displays the difference between the current restored Cloud Foundry state and the system’s state at the time the backup was created.