Installing and Configuring PCF Healthwatch

Warning: PCF Healthwatch v1.5 is no longer supported or available for download. PCF Healthwatch v1.5 has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes how to install and configure Pivotal Cloud Foundry (PCF) Healthwatch. For a list of compatible Ops Manager and Pivotal Application Service (PAS) versions, see Product Snapshot.

Install PCF Healthwatch

1. Download the product file from Pivotal Network.

2. Navigate to the Ops Manager Installation Dashboard and click Import a Product to upload the product file.  

3. Under the Import a Product button, click + next to the version number of PCF Healthwatch. This adds the tile to your staging area.

Configure PCF Healthwatch

To start using PCF Healthwatch, you need to configure the PCF Healthwatch tile.

Create a Network

This section describes when and how to create a network for PCF Healthwatch.

Note: If you have previously installed PCF Healthwatch, you can skip this section.

About Service Networks

When using PCF Healthwatch with Ops Manager v2.1 or later, you do not need to create a separate network as a service network. You can use any of your networks because BOSH handles IP allocation and removes the risk of IP address collision.

For more information, see IP Address Management (IPAM) Removed in the PCF Ops Manager v2.1 Release Notes.

Procedure

If you do not want to use an existing network for PCF Healthwatch, you can create a new network by following these steps:

1. Click the Ops Manager Director tile on the Ops Manager Installation Dashboard.
2. Navigate to Create Networks and click Add Network.
3. In the Name field, enter a name for the service network.
4. Enter the network details.
5. Click Save.

Configure the PCF Healthwatch Tile

To configure the PCF Healthwatch tile, perform the following steps:

1. Click the PCF Healthwatch tile on the Ops Manager Installation Dashboard.

2. Navigate to Assign AZs and Networks and do the following:

   1. Select an Availability Zone (AZ) for placing singleton jobs.

   2. Select one or more AZs for balancing other jobs.

      Note: To create a highly available environment, Pivotal recommends selecting multiple AZs.

   3. Select Network for installing PCF Healthwatch.

   4. Select Service Network for running the BOSH Health Check VM.

   5. Click Save.

3. Navigate to Healthwatch Component Config and do the following:

   1. Optional: Enter a foundation name for this PCF. The entered name is emitted as a tag on all Firehose metrics generated by PCF Healthwatch. If no name is set, then PCF Healthwatch defaults to using the system domain as the foundation name label when publishing its metrics. Do not use any of the special characters !, @, #, ^, &, $, % , *, parentheses or quotes in the chosen name. Use of -, _ or space is allowed.
   2. Set Ingestor Count to the number of Traffic Controller instances configured in the PAS tile. Evaluate performance by monitoring the number of ingestor disonnects. Consistent disconnects is an indicator of too few Ingestor instances. For more information about Ingestor disconnects, see Number of PCF Healthwatch Nozzle Disconnects from Firehose.
   3. Set Redis Worker Count to a ratio of one Redis Worker for every five Ingestor instances configured in the Healthwatch tile. If the data being displayed in the PCF Healthwatch UI is regularly delayed by more than one-minute, scale up the number of instances.
   4. To prevent sending events to the Event Alerts tile (EVA), deselect Publish events to Event Alerts Tile. All events are still calculated and stored in the alert table.
   5. Click Save.

      Note: In the default highly available installation, PCF Healthwatch requires up to 20 GB of free RAM on Diego for steady-state operations and up to 40 GB of RAM during deployments. You can lower this footprint by reducing the instance counts in this section.
      Failure to have enough available RAM on Diego typically results in the Insufficient Resources error message when running the push-apps errand.

4. Navigate to Health Check and do the following:

   1. Enter Ops Manager URL. The URL should include the protocol. This URL must be reachable from a cf pushed application.
   2. In BOSH Health Check Availability Zone, select an AZ for the BOSH Health Check VM.

      Note: On Azure environments, this dropdown is null because Azure does not support AZs.

   3. In BOSH Health Check VM Type, select a VM type for the BOSH Health Check VM. The recommended VM type is small. The VM resources should include at least:
      • 1 CPU
      • 8 GB Disk
      • 1 GB RAM
        If you have add-ons installed, you may need to increase the RAM of the BOSH Health Check VM type. For an example of add-on memory requirements, see the Prerequisites section of the ClamAV add-on documentation.
   4. Optionally, you can also enable the BOSH Deployment Checker. This adds context to the PCF Healthwatch UI graphs of when BOSH deployments have occurred.
      1. Select Enable in the Bosh Deployment Checker section.
      2. Enter a BOSH UAA client and secret with an authority of bosh.read.

         Note: In previous versions of PCF Healthwatch, the BOSH Deployment Checker used the BOSH Director admin credentials as default values. As of v1.3, the BOSH Deployment Checker defaults to disabled until configured.

         1. Follow the instructions for Creating UAA Clients for BOSH Director to target the BOSH UAA and log in with an admin credential. Ensure that you are targeting the BOSH UAA and not Ops Manager or PAS UAA. If your BOSH Director is SSO-enabled, run the following command to log into the UAA:

            uaac token sso get BOSH-DIRECTOR-UAA-LOGIN-CLIENT-USERNAME

         2. Create a UAA Client for the Healthwatch BOSH task check using the following command:

            uaac client add CLIENT-NAME --authorized_grant_types client_credentials --authorities bosh.read --secret CLIENT-SECRET

   5. Click Save.

5. Return to the Ops Manager Installation Dashboard and click Apply Changes.

(Optional) Disable Ops Manager Continuous Validation Testing

PCF Healthwatch continuously validates the health of Ops Manager. You may want to disable this test suite if, for example, you are using BOSH instead of Ops Manager to deploy PCF software. This can be accomplished either via the Ops Manager UI or BOSH manifest.

To disable the Ops Manager test suite via Ops Manager UI (an option available as of v1.1.4), do the following:

1. Click the PCF Healthwatch tile on the Ops Manager Installation Dashboard.
2. Navigate to Health Check section.
3. Select the Disable radio button in the Ops Manager Validation Testing field.

To disable the Ops Manager test suite via BOSH manifest, do the following:

1. In your BOSH manifest, set the healthchecker.opsman.disable property to true.
2. Redeploy PCF Healthwatch and run the push-apps errand.

When you turn the test suite off, the following changes happen:

• The opsmanager-health-check testing app is not deployed.
• The Ops Manager Health panel is not displayed in the PCF Healthwatch UI.
• The Ops Manager Health metric is not created and emitted.

(Optional) Configure Syslog Forwarding

PCF Healthwatch supports forwarding syslog to an external log management service such as Papertrail, Splunk, or a custom enterprise log sink. You may find the VM logs useful for debugging problems in the system.

To enable remote syslog forwarding for PCF Healthwatch, do the following:

1. Click the PCF Healthwatch tile on the Ops Manager Installation Dashboard.
2. Navigate to the Syslog pane.
3. Select Yes without encryption or Yes with TLS encryption to enable syslog forwarding.
4. For External Syslog Host, enter the address of syslog server where you want logs sent, such as logs.example.com.
5. For External Syslog Port, enter the port of your syslog server, such as 29279.
6. For External Syslog Protocol, select the protocol for sending the logs.
7. (TLS only) For External Syslog Permitted Peer, enter the fingerprint or name of the remote peer, such as *.example.com.
8. (TLS only) For *External Syslog TLS CA Certificate *, enter the CA certificate of the syslog destination.

(Optional) Allow Additional Users to Access the PCF Healthwatch UI

Any user with the healthwatch.read or healthwatch.admin scope can access the PCF Healthwatch UI. The UAA admin user has both of these scopes by default.

To allow other users read-only access to the Healthwatch UI, perform the following steps. This procedure adds the healthwatch.read scope to the desired UAA users using the uaac client.

1. Access the uaac server with uaac target UAA-URL.
2. Log in as admin client. Enter uaac token client get UAA-ADMIN-CLIENT-USERNAME. When prompted, enter the secret.
3. Optional: Add new UAA users using uaac user add NEW-USER-USERNAME --emails NEW-USER-EMAIL-ADDRESS. When prompted, enter the password.
4. Enter uaac member add healthwatch.read UAA-USERNAME to add the scope to the user.

The above procedure can be leveraged to add the healthwatch.admin scope as well.

Note: Only users that should be allowed to change settings in Healthwatch should be granted the healthwatch.admin scope. This scope allows a user to alter the default alerting threshold values for the entire product. In future versions of Healthwatch, this user scope may be allowed additional configuration capabilities. As such, this Admin scope should not be granted to any user you wish to remain read-only access.

(Optional) Update Default Alert Thresholds

PCF Healthwatch includes several configurable alerts by deafult. Pivotal recommends customizing the default thresholds for these alerts based on your environment. You can determine the best threshold for your environment by monitoring the metrics over time and noting the metric values that indicate acceptable and unacceptable system performance and health.

To configure Healthwatch alert thresholds, you first need a UAA client with healthwatch.admin authority.

To create a client to configure alert thresholds, perform the following steps:

1. Access the uaac server with uaac target UAA-URL.
2. Log in as admin client. Enter uaac token client get UAA-ADMIN-CLIENT-USERNAME. When prompted, enter the secret.
3. Add new UAA client using uaac client add --authorities 'healthwatch.admin' --authorized_grant_types 'client_credentials' --secret NEW-SECRET. When promoted, enter the new client id.
4. Log in as new client using uaac token client get NEW-CLIENT-ID. When prompted, enter the secret defined above.
5. Verify the client has correct permission using uaac curl 'https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations'. You should see a list of active alert thresholds.

For more information about the default alerts included with PCF Healthwatch, see Supported Alerts.

For more information about updating default alert configurations, see Update Alert Configurations.