Migrating to Internal Percona MySQL

This topic describes the procedure for migrating Pivotal Application Service (PAS) internal MySQL databases from the less secure MariaDB infrastructure to a TLS-encrypted Percona stack.

In PAS v2.2 and PAS v2.3, you could use either MariaDB with cluster nodes communicating over plaintext, or a Percona Server that uses TLS to encrypt communication between server cluster nodes. PAS v2.4 no longer uses MariaDB, so you must migrate your internal databases to Percona and redeploy PAS v2.3 before you can upgrade to PAS v2.4.

WARNING: This procedure causes temporary downtime of PAS system functions, as described in Effects of MySQL Downtime and MySQL Migration Downtime. It does not interrupt apps hosted by PAS, unless they use TCP routing. Applications using TCP routes will not be routable during the migration.

Effects of MySQL Downtime

During internal MySQL server downtime, the Cloud Controller and other platform components that use system databases become unavailable. This prevents users from pushing apps, scaling apps, and running other cf CLI commands.

If the PAS UAA is configured to use an internal database, then the UAA also becomes unavailable. This outage temporarily prevents users from signing into PAS orgs and spaces, Apps Manager, and other PAS interfaces.

PAS UAA downtime caused by internal MySQL migration also temporarily prevents users from logging into any PAS apps that use the Single Sign-On (SSO) service.

Procedure

  1. Scale Down Your Cluster: If your internal databases MySQL cluster runs three or more nodes, scale it down horizontally to a single node before you migrate. In the PAS tile > Resource Config > MySQL Server settings, set Instances to 1. Click Save.

    Note: Do not click Review Pending Changes and Apply Changes yet.

  2. Scale Up the Persistent Disk of the Server Node: Migrating the database demands twice the normal disk space on the MySQL server. To ensure that the single remaining MySQL server node currently uses at most 40% of its persistent disk, do the following:

    1. Check the persistent disk usage of the MySQL Server job in the Status tab of the PAS tile.
    2. If the MySQL Server job disk usage exceeds 40%, scale it up vertically by increasing its persistent disk.
  3. Change the System Databases Location: In the PAS tile > Databases pane, under Choose the location of your system databases, select to Internal Databases - MySQL - Percona XtraDB Cluster. Click Save. System db tls

  4. Migrate the Database: Return to the Installation Dashboard, click Review Pending Changes, and click Apply Changes to re-deploy PAS with the new internal database. Consider the following when performing this step:

    • You do not need to run any errands for this step.
    • Internal system databases become temporarily unavailable to PAS components during this migration step, but PAS apps continue to run. See MySQL Migration Downtime for details.
  5. Scale the Cluster Back Up: If you want to scale the database cluster back up to multiple server nodes, increase the instance count under Resource Config > MySQL Server. Then click Save to save the change and Review Pending Changes and Apply Changes in the Installation Dashboard to re-deploy.

MySQL Migration Downtime

Upgrading existing internal system databases to the new option of Internal Databases - MySQL - Percona XtraDB Cluster causes temporary downtime of PAS system functions. It does not interrupt apps hosted by PAS.

The length of downtime depends on multiple factors, including database size, persistent disk type, CPU and disk capacity of VMs, and PAS configuration. There are two downtime events:

  • When you scale down and migrate the database: During the migration, PAS updates the single node and migrates the data.
  • When you scale back up: PAS updates the first node to make it part of a three-node cluster.

During migration, BOSH orphans the two disks used by the scaled-down nodes. This does not increase the risk of losing data written before the migration, and these orphaned disks can act as a backup, should the migration fail. If the migration fails, contact Pivotal Support.

Migration Length

To estimate how long a database migration takes, perform a manual backup without saving the backup file.

To perform a manual backup, run the command: time /var/vcap/packages/mariadb/bin/mysqldump -u root -p --all-databases > /dev/null

BOSH runs additional processes before and after a MySQL migration, but the total time the migration takes is not much longer than the manual backup.

Create a pull request or raise an issue on the source for this page in GitHub