LATEST VERSION: 1.9 - CHANGELOG
Redis for PCF v1.9

Manually Backing up and Restoring Redis for PCF

Page last updated:

Note: This topic applies only to Dedicated-VM and Shared-VM service plans.

Triggering a Manual Backup

Backups of your Redis deployment will automatically occur per the Cron Schedule you set in your deployment. You can trigger manual backups at any time by following the steps below. The backup artifacts will be sent to the destination configured in Ops Manager for automatic backups.

  • Follow these steps to log in to your Ops Manager installation and target the Redis tile deployment.
  • Identify the VM which holds your instance by running bosh vms.
    • For the shared-vm plan this will be the job name containing cf-redis-broker. Running manual-backup will back up all of the shared-VM instances and the broker state in the statefile.json file.
    • For the dedicated-vm plan this will be the job name containing dedicated-node. Running manual-backup will back up the Redis dump.rdb file for that dedicated-VM instance.
    • You can identify the exact node for your dedicated-vm service instance by comparing the IP Address from your application bindings.

An example output from bosh vms:

OpsManager VMs view

  • Target the manifest of your deployed Redis with bosh deployment PATH-TO-MANIFEST.yml. If you do not have this file, you can download it by running bosh download manifest DEPLOYMENT-NAME.
  • bosh ssh into the node you wish to back up (or the cf-redis-broker node in a shared-vm plan).

After you have connected to the node, a manual backup can be triggered with these steps:

  1. Switch to root using sudo -i.
  2. Run /var/vcap/jobs/service-backup/bin/manual-backup

Notes

Triggering a manual backup of a large dump.rdb could take sufficiently long that your SSH connection will timeout. Ensure that you have given yourself enough of a timeout to complete the backup.

Back ups are currently not available for On-Demand instances.

As of v1.8.2, backups can be to all AWS S3 regions (previously limited to the standard region, us-east-1). This requires specifying the region of your backup and the signature version used for your region. You can find this information here. In most cases the default signature version will be sufficient.

Making Your Own Backups

It is possible to create a back up of a Redis instance by hand, bypassing the automated backup tool altogether.

Persistence is enabled on these plans through the use of RDB files, using the following Redis config rules:

save 900 1
save 300 10
save 60 10000

Shared-VM Plan

You can either take the latest RDB file held on disk, which is generated by the above the rules, or trigger a recent update by using the redis-cli to trigger a BGSAVE. Credentials to log in to the redis-cli can be obtained from VCAP_SERVICES for your bound application.

The redis-cli is located in /var/vcap/packages/redis/bin/redis-cli.

On this plan, the BGSAVE command is aliased to a random string. This can be obtained from Ops Manager in the credentials tab.

Steps to Backup

  1. bosh ssh into your desired node. See the above section on identifying the correct VM.
  2. Change to root using sudo -i.
  3. Copy the contents of the /var/vcap/store/cf-redis-broker directory to a .zip or .tar file.
  4. Backup the folder / compressed file to your chosen location.

The /var/vcap/store/cf-redis-broker has sub-directories for each instance created of this plan. The backup file for each instance is called dump.rdb.

For example, here are two instances:

root@66358f3e-3428-46df-9bb3-9acc7770b188:/var/vcap/store/cf-redis-broker# find -type f | xargs ls -1
./redis-data/3124f373-e9e2-44e1-ad12-a8865d8978b0/db/dump.rdb
./redis-data/3124f373-e9e2-44e1-ad12-a8865d8978b0/redis.conf
./redis-data/3124f373-e9e2-44e1-ad12-a8865d8978b0/redis-server.pid
./redis-data/62333bf9-f023-4566-b233-6686f26b8f4d/db/dump.rdb
./redis-data/62333bf9-f023-4566-b233-6686f26b8f4d/redis.conf
./redis-data/62333bf9-f023-4566-b233-6686f26b8f4d/redis-server.pid
./statefile.json

Dedicated-VM Plan

You can either take the latest RDB file on disk, as generated by the above rules, or trigger a more recent RDB file by executing the BGSAVE command using the redis-cli. Credentials can be obtained from the VCAP_SERVICES from your bound application. The redis-cli can be found in /var/vcap/packages/redis/bin/redis-cli.

Steps to Backup

  • bosh ssh into your desired node. See the above section on identifying the correct VM.
  • Change to root using sudo -i.
  • Copy the contents of the /var/vcap/store/redis directory to a .zip or .tar file.
  • Backup the folder / compressed file to your chosen location.

The backup file will be named dump.rdb.

Restore Redis Instance from a Backup

To a Local System

You can choose to restore the RDB file to a local Redis instance.

The steps to do this depend on your configuration and setup. Refer to the Redis documentation for more details.

To Pivotal Cloud Foundry

You can also restore your backup file to another instance of the Redis for PCF tile.

Prerequisites

  • Same resource configuration as the instance from which you backed up.
  • Ensure that the persistent disk is large enough to accommodate the temporary files used during the restore process. It should be 3.5x the amount of RAM in the VM.

To restore your backup file to another instance of a Redis for PCF tile service instance:

  1. Create a new instance of the plan that you wish to restore to.
  2. Identify the VM which the instance of your plan is located on by following the steps from the Manual Backups section above. If you are restoring an instance of shared-vm, this VM is the broker VM.
  3. bosh ssh into the identified VM. This is the broker VM if restoring a shared-vm instance.

Redis for PCF v1.7 and later provides a script to automatically restore data in a newly provisioned Redis instance.

Preparation

  1. Transfer your backup RDB file to a local path on the VM (PATH-TO-RDB-BACKUP-ON-VM has to be under /var/vcap/store.
  2. To verify that the RDB file hasn’t been corrupted, run md5sum PATH-TO-RDB-BACKUP-ON-VM and compare it against the contents of the .md5 file named after the backup file. The values should be the same. The .md5 file is located in the same bucket as the original backup file.
  3. Switch to root user sudo su

Dedicated-VM Plan

The restore script will restore the data for the specified dedicated-VM instance.

Execution

  1. Run /var/vcap/jobs/redis-backups/bin/restore --sourceRDB PATH-TO-RDB-BACKUP-ON-VM
  2. Tail the script logs at /var/vcap/sys/log/redis-backups/redis-backups.log to see progress. When the data restore has been successfully completed, you will see the message Redis data restore completed successfully

Debugging

The data restore script runs the steps listed below. It logs its process along the way and provides helpful messages in case of failure.

The script logs at /var/vcap/sys/log/redis-backups/redis-backups.log.
If a step has failed, resolve the reason that caused it to fail and execute the failed step and every next step manually. You can retrieve {instance_password} through the binding to your service instance: cf service-key {instance_name} {key_name}

  1. StopAll
    Run monit stop all
  2. WaitForStop
    Wait for monit services to enter the not monitored state, you can watch this with watch monit summary
  3. DeleteExistingPersistenceFiles
    Clean up existing Redis data files:
    • rm -f /var/vcap/store/redis/appendonly.aof
    • rm -f /var/vcap/store/redis/dump.rdb
  4. CopyBackupFileWithCorrectPermissions
    Restore your Redis backup file to /var/vcap/store/redis/dump.rdb and correct the owner and permissions with chown vcap:vcap /var/vcap/store/redis/dump.rdb && chmod 660 /var/vcap/store/redis/dump.rdb
  5. SetAppendOnly
    Edit the template Redis config file withvim $(find /var/vcap/data/jobs/ -name redis.conf)` and make the following line changes:
    • appendonly yes -> appendonly no
  6. StartAll
    Run monit start all
  7. WaitForStart
    Wait for monit services to enter the running state, you can watch this with watch monit summary
  8. RewriteAOF
    Run /var/vcap/packages/redis/bin/redis-cli -a {instance_password} BGREWRITEAOF
  9. RewriteAOF
    Run watch "/var/vcap/packages/redis/bin/redis-cli -a {instance_password} INFO | grep aof_rewrite_in_progress" until aof_rewrite_in_progress is 0
  10. StopAll
    Run monit stop all
  11. WaitForStop
    Wait for monit services to enter the not monitored state, you can watch this with watch monit summary
  12. ChownToUserAndGroup
    Set correct owner on appendonly.aof by running chown vcap:vcap /var/vcap/store/redis/appendonly.aof
  13. SetAppendOnly
    Edit the template Redis config file with vim $(find /var/vcap/data/jobs/ -name redis.conf) and make the following line changes:
    • appendonly no -> appendonly yes
  14. StartAll
    Run monit start all

Shared-VM Plan

The restore script will restore the data for the specified shared-VM instance.

Execution

  1. Retrieve {instance_guid} by running: cf service {instance_name} --guid
  2. Run /var/vcap/jobs/redis-backups/bin/restore --sourceRDB {path-to-rdb-backup-on-vm} --sharedVmGuid {instance_guid}
  3. Tail the script logs at /var/vcap/sys/log/redis-backups/redis-backups.log to see progress. When the data restore has been successfully completed, you will see the message Redis data restore completed successfully

Debugging

The data restore script runs the steps listed below. It logs its process along the way and provides helpful messages in case of failure.

The script logs at /var/vcap/sys/log/redis-backups/redis-backups.log.
If a step has failed, resolve the reason that caused it to fail and execute the failed step and every next step manually. You can retrieve {instance_password} and {port} through the binding to your service instance: cf service-key {instance_name} {key_name}

  1. StopAll
    Run monit stop all
  2. WaitForStop
    Wait for monit services to enter the not monitored state, you can watch this with watch monit summary
  3. SetConfigCommand
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/redis.conf and comment out the line:
    • rename-command CONFIG "configalias" -> #rename-command CONFIG "configalias"
  4. SetRewriteCommand
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/redis.conf and comment out the line:
    • rename-command BGREWRITEAOF "" -> #rename-command BGREWRITEAOF ""
  5. DeleteExistingPersistenceFiles
    Clean up existing Redis data files if they exist:
    • rm -f /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/db/appendonly.aof
    • rm -f /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/db/dump.rdb
  6. CopyBackupFileWithCorrectPermissions
    Restore your Redis backup file to /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/db/dump.rdb and correct the owner and permissions with chown vcap:vcap /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/db/dump.rdb && chmod 660 /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/db/dump.rdb
  7. SetAppendOnly
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/redis.conf and make the following line changes:
    • appendonly yes -> appendonly no
  8. StartAll
    Run monit start all
  9. WaitForStart
    Wait for monit services to enter the running state, you can watch this with watch monit summary
  10. RewriteAOF
    Run /var/vcap/packages/redis/bin/redis-cli -a {instance_password} BGREWRITEAOF
  11. RewriteAOF
    Run watch "/var/vcap/packages/redis/bin/redis-cli -a {instance_password} INFO | grep aof_rewrite_in_progress" until aof_rewrite_in_progress is 0
  12. StopAll
    Run monit stop all
  13. WaitForStop
    Wait for monit services to enter the not monitored state, you can watch this with watch monit summary
  14. ChownToUserAndGroup
    Set correct owner on appendonly.aof by running chown vcap:vcap /var/vcap/store/redis/appendonly.aof
  15. SetAppendOnly
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/redis.conf and make the following line changes:
    • appendonly no -> appendonly yes
  16. SetConfigCommand
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/redis.conf and uncomment the line: #rename-command CONFIG "configalias" -> rename-command CONFIG "configalias"
  17. SetRewriteCommand
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/{instance_guid}/redis.conf and uncomment the line: #rename-command BGREWRITEAOF "" -> rename-command BGREWRITEAOF ""
  18. StartAll
    Run monit start all

Recovering Redis Instances

In the event of a recovery of Cloud Foundry, it is possible to recover bound Redis instances to healthy states that are in sync with Cloud Foundry. There are a few caveats to being able to recover previous instance state fully that depend on your plan.

Shared-VM Plan Caveats

  • You need a backed up RDB Redis dump file - this would be stored in your S3 buckets if you have backups configured.
  • You need a backed up /var/vcap/store/cf-redis-broker/redis-data directory from the service broker node. You do not need to backup and *.aof or *.rdb files from subdirectories if you have backups configured.

Dedicated-VM Plan Caveats

  • You need a backed up RDB Redis dump file - this would be stored in your S3 buckets if you have backups configured.
  • You need a backed up /var/vcap/store/redis/statefile.json from the service broker node.

Note

This procedure assumes that a recovery of service information and service keys assigned to instances are restored with a restore of Cloud Foundry.

Recovery Procedure

After redeploying Redis, take the following steps.

Shared-VM Plan

  1. bosh ssh into the service broker node of your Redis deployment.
  2. Run monit stop all && pkill redis-server
  3. Wait for monit services to enter the not monitored state, you can watch this with watch monit summary
  4. Confirm no running instances of redis-server with ps aux | grep redis-server
  5. Copy the backed up redis-data directory into /var/vcap/store/cf-redis-broker
  6. Follow the instructions here for your plan, skipping the first four steps described here, for restoring your backed up Redis data.
  7. Your Redis instance is now recovered.

Dedicated-VM Plan

  1. bosh ssh into the service broker node of your Redis deployment.
  2. Run monit stop all
  3. Wait for monit services to enter the not monitored state, you can watch this with watch monit summary
  4. Copy the backed up /var/vcap/store/cf-redis-broker/statefile.json and ensure ownership and permissions are correct with chown vcap:vcap /var/vcap/store/redis/dump.rdb && chmod 660 /var/vcap/store/redis/dump.rdb
  5. Follow the instructions here for your plan, skipping the first three steps described here, for restoring your backed up Redis data.
  6. Your Redis instance is now recovered.
Create a pull request or raise an issue on the source for this page in GitHub