Scaling Internal MySQL for PAS

Page last updated:

This topic describes scaling down your Pivotal Application Service (PAS) MySQL cluster and provides example sizing data from two environments that have significant load on their MySQL clusters.

For additional resources about scaling Internal MySQL, see the Deployments Using Internal MySQL section of the Scaling PAS topic and the Scalable Components section of the High Availability in PAS topic.

Note: The procedure does not apply to databases configured as external in the Databases pane of the PAS tile.

PAS components that use system databases include the Cloud Controller, Diego Brain, Gorouter, and User Authorization and Authentication (UAA) server. For more information, see PAS Components.

Scale Down Your MySQL Cluster

This procedure explains how to safely scale your MySQL cluster down to a single node. If you are already running the MySQL cluster with a single node, you do not need to do this procedure.

By default, internal MySQL deploys as a single node. To take advantage of the high availability features of MySQL, you may have scaled the configuration up to three or more server nodes.

Check the Health of Your Cluster

Before scaling down your MySQL cluster, you must ensure the cluster is healthy.

To check the health of your cluster:

  1. Use the Cloud Foundry Command Line Interface (cf CLI) to target the API endpoint of your Pivotal Cloud Foundry (PCF) deployment. Run:

    cf api api.YOUR-SYSTEM-DOMAIN
    

    Where YOUR-SYSTEM-DOMAIN is the system domain defined in the Domains pane of the PAS tile.

  2. To obtain your UAA admin user credentials:

    1. Click the Credentials tab of the PAS tile.
    2. Locate the Admin Credentials entry in the UAA section.
    3. Click Link to Credential.
  3. Log in with the UAA admin credentials you obtained in the previous step by running:

    cf login -u admin
    
  4. Create a test organization to verify the database across all nodes by running:

    cf create-org data-integrity-test-organization
    
  5. To obtain the IP addresses of your MySQL server:

    1. From the Ops Manager Installation Dashboard, click the PAS tile.
    2. Click the Status tab.
    3. Record the IP addresses for all instances of the MySQL Server job.
  6. To retrieve Cloud Controller database credentials from CredHub using the Ops Manager API:

    1. Follow the procedures in Using the Ops Manager API to authenticate and access the Ops Manager API.
    2. Retrieve a list of deployed products by running:

      curl "https://OPS-MANAGER-FQDN/api/v0/deployed/products" \
      -X GET \
      -H "Authorization: Bearer UAA-ACCESS-TOKEN"
      

      Where:

      • OPS-MANAGER-FQDN is the fully-qualified domain name (FQDN) of your Ops Manager deployment.
      • UAA-ACCESS-TOKEN is the access token you recorded in the previous step.
    3. In the response to the above request, locate the product with an installation_name starting with cf- and copy its guid.

    4. Run:

      curl "https://OPS-MANAGER-FQDN/api/v0/deployed/products/PRODUCT-GUID/variables?name=cc-db-credentials" \
      -X GET \
      -H "Authorization: Bearer UAA-ACCESS-TOKEN"
      

      Where:

      • OPS-MANAGER-FQDN is the fully-qualified domain name (FQDN) of your Ops Manager deployment.
      • PRODUCT-GUID is the value of guid you recorded in the previous step.
      • UAA-ACCESS-TOKEN is the access token you recorded in the previous step.
    5. Record the Cloud Controller database username and password from the response to the above request.

  7. SSH into the Ops Manager VM. Because the procedures vary by IaaS, see the SSH into Ops Manager section of the Advanced Troubleshooting with the BOSH CLI topic for instructions specific to your IaaS.

  8. For each of the MySQL server IP addresses recorded above:

    1. Query the new organization by running:

      mysql -h YOUR-IP -u YOUR-IDENTITY -D ccdb -p -e "select created_at, name from organizations where name = 'data-integrity-test-organization'"
      

      Where:

      • YOUR-IP is the IP address of the MySQL server.
      • YOUR-IDENTITY is the identity value of the CCDB credentials you obtained above.
    2. When prompted, provide the password value of the CCDB credentials you obtained above.

    3. Examine the output of the mysql command and verify the created_at date is recent. For example:

      +---------------------+----------------------------------+
      | created_at          | name                             |
      +---------------------+----------------------------------+
      | 2016-05-28 01:11:42 | data-integrity-test-organization |
      +---------------------+----------------------------------+
      

  9. If each MySQL server instance does not return the same created_at result, contact Pivotal Support before proceeding further or making any changes to your deployment. If each MySQL server instance does return the same result, then you can safely proceed to scaling down your cluster to a single node by performing the steps in the following section.

Set Server Instance Count to 1

To scale your server instance count to 1:

  1. Select Resource Config in the PAS tile.

  2. Use the dropdown to change the Instances count for MySQL Server to 1.

  3. Click Save, then Review Pending Changes and Apply Changes to apply the changes.

  4. Delete your test organization by running:

    cf delete-org data-integrity-test-organization
    

MySQL Cluster Sizing Examples

This topic describes two sizing examples for internal MySQL in PAS.

Use this data as guidance to ensure your MySQL Clusters are scaled to handle the number of app instances running on your deployment.

Example 1: Pivotal Web Services Production Environment

The information in this section comes from the Pivotal-managed Cloud Foundry deployment, Pivotal Web Services (PWS).

Note: This deployment differs from most PCF deployments in that the MySQL database is used for Diego, but not the Cloud Controller. This means that while there are a large number of queries per second, most of them are reading data, and the number of writes is not realistically reflected.

  • IAAS: AWS
  • App Instances: ~23,000
  • Average SQL Queries Per Minute: ~308,000
  • Average IOPS: 220
  • Storage Volume Usage: 80%

VM Sizing

The following table displays MySQL VM settings for this environment:

Setting Value
VM Type c4.2xlarge
Storage Volume Type io1
Storage Volume Size 98 GB
Storage Volume IOPS 2000

Example 2: Cloud Foundry Diego Test Environment

The information in this section comes from an environment used by the Diego to test the MySQL cluster with a high load similar to one generated large deployment.

  • IAAS: GCP
  • App Instances: 250,000
  • Average SQL Queries Per Minute: ~5,100,000
  • Average IOPS:
    • Reads: 3.63
    • Writes: 363.98

VM Sizing

The following table displays MySQL VM settings for this environment:

Setting Value
VM Type n1-standard-16
Storage Volume Type pd-ssd
Storage Volume Size 1 TB