Monitoring Node Health (HA Cluster)

Page last updated:

Warning: Highly available plans are currently in beta. HA clusters are for advanced use cases only.

This topic describes how to obtain the health status for each node in a MySQL database cluster node in a highly available (HA) cluster plan.

If you cannot write or connect to your app, you should check the health status of your nodes. You can observe the health status of nodes through a proxy using the Switchboard dashboard or API endpoint. You can also view the number of client connections routed through a proxy to each node. For more information about proxies, see MySQL Proxy.

Monitor Node Health

You can monitor the health status of your nodes by doing one of the following:

Prerequisite

To connect to Switchboard, you must obtain credentials.

To obtain credentials for accessing the Switchboard dashboard and API, do the following:

  1. Run the following command:

    cf service YOUR-HA
    

    Where YOUR-HA is the name of your HA cluster.

     $ cf service myHA
    Showing info of service myHA in org acceptance / space example as admin...
    
    name:         myHA
    service:      p.mysql
    tags:
    plan:         db-ha-small
    description:  Dedicated instances of MySQL
    documentation:
    dashboard:    proxy.123abc45-67d8-912e-34f5-g34612c10dba.org.dedicated-mysql.cf-app.com
    (username: 'abcdefghijklmno', password: '012345678912345')
    
    Showing status of last operation from service myHA...
    
    status:    update succeeded
    message:   Instance update completed
    started:   2018-11-20T01:25:55Z
    updated:   2018-11-20T01:30:33Z
    
  2. Record the dashboard hostname, username, and password.

Monitor Node Health Using the Dashboard

To monitor node health using the Switchboard dashboard, do the following:

  1. To view a list of proxies in your browser, navigate to the hostname that you recorded in Prerequisite above. For example, proxy.123abc45-67d8-912e-34f5-g34612c10dba.org.dedicated-mysql.cf-app.com

  2. When prompted, enter the username and password that you recorded in Prerequisite above.

  3. Click the link for the proxy that you want to use to monitor node health.

  4. If you are prompted, enter the username and password that you recorded in Prerequisite above.

    Switchboard all healthy

Monitor Node Health Using the API

You can also use the Switchboard API to obtain the information that is shown on the Switchboard dashboard.

For example, you might want to use the API to write your own app to monitor the cluster.

To monitor node health using the Switchboard API, do the following:

  1. To monitor node health, run the following command:

    curl https://USERNAME:PASSWORD@N-HOSTNAME/v0/backends
    

    Where:

    • USERNAME is the username you recorded in Prerequisite above.
    • PASSWORD is the password you recorded in Prerequisite above.
    • N is either 0, 1, or 2 depending on the proxy you want to connect to.
    • HOSTNAME is the hostname you recorded in Prerequisite above.

    The above command outputs a JSON object similar to the following:

    $ curl https://abcdefghijklmno:012345678912345@0-proxy.123abc45-67d8-912e-34f5-g34612c10dba.org.dedicated-mysql.cf-app.com/v0/backends
    [
      {
        "host": "a-b1234c5d6.e-f891.bosh",
        "port": 6033,
        "healthy": true,
        "name": "backend-0",
        "currentSessionCount": 0,
        "active": true,
        "trafficEnabled": true
      },
      {
        "host": "a-b1234c5d6.e-f891.bosh",
        "port": 6033,
        "healthy": true,
        "name": "backend-1",
        "currentSessionCount": 0,
        "active": false,
        "trafficEnabled": true
      },
      {
        "host": "a-b1234c5d6.e-f891.bosh",
        "port": 6033,
        "healthy": true,
        "name": "backend-2",
        "currentSessionCount": 0,
        "active": false,
        "trafficEnabled": true
      }
    ]
    

Node Health Status

When determining where to route traffic, the proxy queries an HTTP healthcheck process running on the node. This healthcheck can return as either healthy or unhealthy, or the node can be unresponsive.

Healthy

If the healthcheck process returns HTTP status code 200, the proxy includes the node in its pool of healthy nodes.

When a new or resurrected nodes rejoin the cluster, the proxy continues to route all connections to the currently active node. In the case of failover, the proxy considers all healthy nodes as candidates for new connections.

Switchboard all healthy

Unhealthy

If the healthcheck returns HTTP status code 503, the proxy considers the node unhealthy.

This happens when a node becomes non-primary. For more information, see Cluster Scaling Behavior.

The proxy severs existing connections to newly unhealthy node. The proxy routes new connections to a healthy node, assuming such a node exists. Clients are expected to handle reconnecting on connection failure should the entire cluster become inaccessible.

Switchboard unhealthy

Unresponsive

If node health cannot be determined due to an unreachable or unresponsive healthcheck endpoint, the proxy considers the node unhealthy. This may happen if there is a network partition or if the VM running the node and healthcheck died.