Installing and Configuring Pivotal Healthwatch

This topic describes how to install and configure Pivotal Healthwatch. For a list of compatible Ops Manager and Pivotal Application Service (PAS) versions, see Product Snapshot.

Install Pivotal Healthwatch

  1. Download the product file from Pivotal Network.

  2. Navigate to the Ops Manager Installation Dashboard and click Import a Product to upload the product file.  

  3. Under the Import a Product button, click + next to the version number of Pivotal Healthwatch. This adds the tile to your staging area.

Configure Pivotal Healthwatch

To start using Pivotal Healthwatch, you need to configure the Pivotal Healthwatch tile.

Create a Network

This section describes when and how to create a network for Pivotal Healthwatch.

Note: If you have previously installed Pivotal Healthwatch, you can skip this section.

About Service Networks

When using Pivotal Healthwatch with Ops Manager v2.1 or later, you do not need to create a separate network as a service network. You can use any of your networks because BOSH handles IP allocation and removes the risk of IP address collision.

For more information, see IP Address Management (IPAM) Removed in the PCF Ops Manager v2.1 Release Notes.

Procedure

If you do not want to use an existing network for Pivotal Healthwatch, you can create a new network by following these steps:

  1. Click the Ops Manager Director tile on the Ops Manager Installation Dashboard.
  2. Navigate to Create Networks and click Add Network.
  3. In the Name field, enter a name for the service network.
  4. Enter the network details.
  5. Click Save.

Configure the Pivotal Healthwatch Tile

To configure the Pivotal Healthwatch tile, perform the following steps:

  1. Click the Pivotal Healthwatch tile on the Ops Manager Installation Dashboard.
  2. Navigate to Assign AZs and Networks and do the following:

    1. Select an Availability Zone (AZ) for placing singleton jobs.
    2. Select one or more AZs for balancing other jobs.

      Note: To create a highly available environment, Pivotal recommends selecting multiple AZs.

    3. Select Network for installing Pivotal Healthwatch.

    4. Select Service Network for running the BOSH Health Check VM.

    5. Click Save.

  3. Navigate to Healthwatch Component Config and do the following:

    1. Optional: Enter a foundation name for this Pivotal Platform. The entered name is emitted as a tag on all Firehose metrics generated by Pivotal Healthwatch. If no name is set, then Pivotal Healthwatch defaults to using the system domain as the foundation name label when publishing its metrics. Do not use any of the special characters !, @, #, ^, &, $, % , *, parentheses or quotes in the chosen name. Use of -, _ or space is allowed.
    2. Set Ingestor Count to the number of Traffic Controller instances configured in the PAS tile. Evaluate performance by monitoring the number of ingestor disonnects. Consistent disconnects is an indicator of too few Ingestor instances. For more information about Ingestor disconnects, see Number of Pivotal Healthwatch Nozzle Disconnects from Firehose.
    3. Set Redis Worker Count to a ratio of one Redis Worker for every five Ingestor instances configured in the Healthwatch tile. If the data being displayed in the Pivotal Healthwatch UI is regularly delayed by more than one-minute, scale up the number of instances.
    4. To prevent sending events to the Event Alerts tile (EVA), deselect Publish events to Event Alerts Tile. All events are still calculated and stored in the alert table.
    5. Click Save.

      Note: In the default highly available installation, Pivotal Healthwatch requires up to 20 GB of free RAM on Diego for steady-state operations and up to 40 GB of RAM during deployments. You can lower this footprint by reducing the instance counts in this section.
      Failure to have enough available RAM on Diego typically results in the Insufficent Resources error message when running the push-apps errand.

  4. Navigate to Health Check and do the following:

    1. Enter Ops Manager URL. The URL should include the protocol. This URL must be reachable from a cf pushed application.
    2. In BOSH Health Check Availability Zone, select an AZ for the BOSH Health Check VM.

      Note: On Azure environments, this dropdown is null because Azure does not support AZs.

    3. In BOSH Health Check VM Type, select a VM type for the BOSH Health Check VM. The recommended VM type is small. The VM resources should include at least:
      • 1 CPU
      • 8 GB Disk
      • 1 GB RAM
        If you have add-ons installed, you may need to increase the RAM of the BOSH Health Check VM type. For an example of add-on memory requirements, see the Prerequisites section of the ClamAV add-on documentation.
    4. Optionally, you can also enable the BOSH Deployment Checker. This adds context to the Pivotal Healthwatch UI graphs of when BOSH deployments have occurred.
      1. Select Enable in the Bosh Deployment Checker section.
      2. Enter a BOSH UAA client and secret with an authority of bosh.read.

        Note: In previous versions of Pivotal Healthwatch, the BOSH Deployment Checker used the BOSH Director admin credentials as default values. As of v1.3, the BOSH Deployment Checker defaults to disabled until configured.

        1. Follow the instructions for Creating UAA Clients for BOSH Director to target the BOSH UAA and log in with an admin credential. Ensure that you are targeting the BOSH UAA and not Ops Manager or PAS UAA. If your BOSH Director is SSO-enabled, run the following command to log into the UAA:
          uaac token sso get BOSH-DIRECTOR-UAA-LOGIN-CLIENT-USERNAME
        2. Create a UAA Client for the Healthwatch BOSH task check using the following command:
          uaac client add CLIENT-NAME --authorized_grant_types client_credentials --authorities bosh.read --secret CLIENT-SECRET
    5. Click Save.
  5. Return to the Ops Manager Installation Dashboard and click Apply Changes.

(Optional) Disable Ops Manager Continuous Validation Testing

Pivotal Healthwatch continuously validates the health of Ops Manager. You may want to disable this test suite if, for example, you are using BOSH instead of Ops Manager to deploy Pivotal Platform software. This can be accomplished either via the Ops Manager UI or BOSH manifest.

To disable the Ops Manager test suite via Ops Manager UI (an option available as of v1.1.4), do the following:

  1. Click the Pivotal Healthwatch tile on the Ops Manager Installation Dashboard.
  2. Navigate to Health Check section.
  3. Select the Disable radio button in the Ops Manager Validation Testing field.

To disable the Ops Manager test suite via BOSH manifest, do the following:

  1. In your BOSH manifest, set the healthchecker.opsman.disable property to true.
  2. Redeploy Pivotal Healthwatch and run the push-apps errand.

When you turn the test suite off, the following changes happen:

  • The opsmanager-health-check testing app is not deployed.
  • The Ops Manager Health panel is not displayed in the Pivotal Healthwatch UI.
  • The Ops Manager Health metric is not created and emitted.

(Optional) Configure Syslog Forwarding

Pivotal Healthwatch supports forwarding syslog to an external log management service such as Papertrail, Splunk, or a custom enterprise log sink. You may find the VM logs useful for debugging problems in the system.

To enable remote syslog forwarding for Pivotal Healthwatch, do the following:

  1. Click the Pivotal Healthwatch tile on the Ops Manager Installation Dashboard.
  2. Navigate to the Syslog pane.
  3. Select Yes without encryption or Yes with TLS encryption to enable syslog forwarding.
  4. For External Syslog Host, enter the address of syslog server where you want logs sent, such as logs.example.com.
  5. For External Syslog Port, enter the port of your syslog server, such as 29279.
  6. For External Syslog Protocol, select the protocol for sending the logs.
  7. (TLS only) For External Syslog Permitted Peer, enter the fingerprint or name of the remote peer, such as *.example.com.
  8. (TLS only) For *External Syslog TLS CA Certificate *, enter the CA certificate of the syslog destination.

(Optional) Allow Additional Users to Access the Pivotal Healthwatch UI

Any user with the healthwatch.read or healthwatch.admin scope can access the Pivotal Healthwatch UI. The UAA admin user has both of these scopes by default.

To allow other users read-only access to the Healthwatch UI, perform the following steps. This procedure adds the healthwatch.read scope to the desired UAA users using the uaac client.

  1. Access the uaac server with uaac target UAA-URL.
  2. Log in as admin client. Enter uaac token client get UAA-ADMIN-CLIENT-USERNAME. When prompted, enter the secret.
  3. Optional: Add new UAA users using uaac user add NEW-USER-USERNAME --emails NEW-USER-EMAIL-ADDRESS. When prompted, enter the password.
  4. Enter uaac member add healthwatch.read UAA-USERNAME to add the scope to the user.

The above procedure can be leveraged to add the healthwatch.admin scope as well.

Note: Only users that should be allowed to change settings in Healthwatch should be granted the healthwatch.admin scope. This scope allows a user to alter the default alerting threshold values for the entire product. In future versions of Healthwatch, this user scope may be allowed additional configuration capabilities. As such, this Admin scope should not be granted to any user you wish to remain read-only access.

(Optional) Update Default Alert Thresholds

Pivotal Healthwatch includes several configurable alerts by deafult. Pivotal recommends customizing the default thresholds for these alerts based on your environment. You can determine the best threshold for your environment by monitoring the metrics over time and noting the metric values that indicate acceptable and unacceptable system performance and health.

To configure Healthwatch alert thresholds, you first need a UAA client with healthwatch.admin authority.

To create a client to configure alert thresholds, perform the following steps:

  1. Access the uaac server with uaac target UAA-URL.
  2. Log in as admin client. Enter uaac token client get UAA-ADMIN-CLIENT-USERNAME. When prompted, enter the secret.
  3. Add new UAA client using uaac client add --authorities 'healthwatch.admin' --authorized_grant_types 'client_credentials' --secret NEW-SECRET. When promoted, enter the new client id.
  4. Log in as new client using uaac token client get NEW-CLIENT-ID. When prompted, enter the secret defined above.
  5. Verify the client has correct permission using uaac curl 'https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations'. You should see a list of active alert thresholds.

For more information about the default alerts included with Pivotal Healthwatch, see Supported Alerts.

For more information about updating default alert configurations, see Update Alert Configurations.