Installing and Configuring Pivotal Healthwatch

This topic describes how to install and configure Pivotal Healthwatch. For a list of compatible Ops Manager and Tanzu Application Service for VMs (TAS for VMs) versions, see Product Snapshot.

Install Pivotal Healthwatch

  1. Download the product file from Pivotal Network.

  2. Navigate to the Ops Manager Installation Dashboard and click Import a Product to upload the product file.  

  3. Under the Import a Product button, click + next to the version number of Pivotal Healthwatch. This adds the tile to your staging area.

Configure Pivotal Healthwatch

To start using Pivotal Healthwatch, you need to configure the Pivotal Healthwatch tile.

Create a Network

This section describes when and how to create a network for Pivotal Healthwatch.

Note: If you have previously installed Pivotal Healthwatch, you can skip this section.

About Service Networks

When using Pivotal Healthwatch with Ops Manager v2.1 or later, you do not need to create a separate network as a service network. You can use any of your networks because BOSH handles IP allocation and removes the risk of IP address collision.

For more information, see IP Address Management (IPAM) Removed in the PCF Ops Manager v2.1 Release Notes.

Procedure

If you do not want to use an existing network for Pivotal Healthwatch, you can create a new network. To create a new network:

  1. Navigate to the Ops Manager Installation Dashboard.

  2. Click the BOSH Director tile.

  3. Navigate to Create Networks and click Add Network.

  4. In the Name field, enter a name for the service network.

  5. Enter the network details.

  6. Click Save.

Configure the Pivotal Healthwatch Tile

To configure the Pivotal Healthwatch tile:

  1. Navigate to the Ops Manager Installation Dashboard.

  2. Click the Pivotal Healthwatch tile.

  3. Select Assign AZs and Networks and configure the following fields:

    1. Select an Availability Zone (AZ) for placing singleton jobs.
    2. Select one or more AZs for balancing other jobs.

      Note: To create a highly available environment, Pivotal recommends selecting multiple AZs.

    3. Select Network for installing Pivotal Healthwatch.
    4. Select Service Network for running the BOSH Health Check VM.

  4. Click Save.

  5. Select Healthwatch Component Config and configure the following fields:

    1. (Optional) Enter a foundation name for this Ops Manager deployment. The entered name is emitted as a tag on all metrics generated by Pivotal Healthwatch. If no name is set, then Pivotal Healthwatch defaults to using the system domain as the foundation name label when publishing its metrics. Do not use any of the special characters !, @, #, ^, &, $, % , *, parentheses or quotes in the chosen name. Use of -, _ or space is allowed.
    2. Set Ingestor Count to the number of Traffic Controller instances configured in the TAS for VMs tile. Evaluate performance by monitoring the number of RLP drops. Consistent RLP drop rate is an indicator of too few Ingestor instances. For more information about RLP drops, see Reverse Log Proxy Egress Dropped Messages.
    3. Set Redis Worker Count to a ratio of one Redis Worker for every five Ingestor instances configured in the Healthwatch tile. If the data being displayed in the Pivotal Healthwatch UI is regularly delayed by more than one-minute, scale up the number of instances.
    4. To prevent sending events to the Event Alerts tile (EVA), deselect the Publish events to Event Alerts Tile checkbox. All events are still calculated and stored in the alert table.

      Note: In the default highly available installation, Pivotal Healthwatch requires up to 20 GB of free RAM on Diego for steady-state operations and up to 40 GB of RAM during deployments. You can lower this footprint by reducing the instance counts in this section.
      Failure to have enough available RAM on Diego typically results in the InsufficientResources error message when running the push-apps errand. For more information, see Insufficient Memory Resources in Troubleshooting Pivotal Healthwatch.

  6. Click Save.

  7. Select Health Check and configure the following fields:

    1. Enter Ops Manager URL. The URL should include the protocol. This URL must be reachable from a cf push app.
    2. In BOSH Health Check Availability Zone, select an AZ for the BOSH Health Check VM.

      Note: In Azure environments, this dropdown is null because Azure does not support AZs.

    3. In BOSH Health Check VM Type, select a VM type for the BOSH Health Check VM. The recommended VM type is small. The VM resources should include at least:
      • 1 CPU
      • 8 GB Disk
      • 1 GB RAM
        If you have add-ons installed, you may need to increase the RAM of the BOSH Health Check VM type. For an example of add-on memory requirements, see the Prerequisites section of the ClamAV add-on documentation.

    4. (Optional) Enable the BOSH Deployment Checker. This adds context to the Pivotal Healthwatch UI graphs of when BOSH deployments have occurred. To enable the BOSH Deployment Checker:

      1. Under Bosh Deployment Checker, select Enable.

      2. Enter a BOSH UAA client and secret with the bosh.read authority.

        Note: In previous versions of Pivotal Healthwatch, the BOSH Deployment Checker used the BOSH Director admin credentials as default values. As of Healthwatch v1.3, the BOSH Deployment Checker defaults to disabled until configured.

        1. Target the BOSH UAA instance and log in with an admin credential by following the procedure in Creating UAA Clients for BOSH Director. Ensure that you are targeting the BOSH UAA instance and not the Ops Manager or TAS for VMs UAA instance. If your BOSH Director is SSO-enabled, log in to the BOSH UAA instance by running:

          uaac token sso get BOSH-DIRECTOR-UAA-LOGIN-CLIENT-USERNAME

          Where BOSH-DIRECTOR-UAA-LOGIN-CLIENT-USERNAME is the username you created.

        2. Create a UAA client for the Healthwatch BOSH task check by running:

          uaac client add CLIENT-NAME --authorized_grant_types client_credentials --authorities bosh.read --secret CLIENT-SECRET

          Where:

          • CLIENT-NAME is the client name you want to set for the Healthwatch BOSH Deployment Checker.
          • CLIENT-SECRET is the client secret you want to set for the Healthwatch BOSH Deployment Checker.

  8. Click Save.

  9. Select Errands and ensure that the following errands are set:

    1. Push Monitoring Components is set to On.
    2. Metrics Troubleshooting is set to Off. This option is used for troubleshooting only.

  10. Return to the Ops Manager Installation Dashboard.

  11. Click Review Changes.

  12. Click Apply Changes.

(Optional) Disable Ops Manager Continuous Validation Testing

Pivotal Healthwatch continuously validates the health of Ops Manager. You may want to disable this test suite if, for example, you are using BOSH instead of Ops Manager to deploy Ops Manager software. As of Healthwatch v1.1.4, you can disable the test suite through the Ops Manager UI or your BOSH manifest.

To disable the Ops Manager test suite through the Ops Manager UI:

  1. Navitage to the Ops Manager Installation Dashboard.

  2. Click the Pivotal Healthwatch tile.

  3. Select Health Check.

  4. Under Ops Manager Validation Testing, select Disable.

To disable the Ops Manager test suite through the BOSH manifest:

  1. In your BOSH manifest, set the healthchecker.opsman.disable property to true.

  2. Redeploy Pivotal Healthwatch and run the push-apps errand.

When you turn the test suite off, the following changes happen:

  • The opsmanager-health-check testing app is not deployed.
  • The Ops Manager Health panel is not displayed in the Pivotal Healthwatch UI.
  • The Ops Manager Health metric is not created and emitted. For more information, see Ops Manager Health in Pivotal Healthwatch Metrics.

(Optional) Configure Syslog Forwarding

Pivotal Healthwatch supports forwarding syslog to an external log management service such as Papertrail, Splunk, or a custom enterprise log sink. You may find the VM logs useful for debugging problems in the system.

To enable remote syslog forwarding for Pivotal Healthwatch:

  1. Navitage to the Ops Manager Installation Dashboard.

  2. Click the Pivotal Healthwatch tile.

  3. Select Syslog.

  4. Enable syslog forwarding by selecting Yes without encryption or Yes with TLS encryption.

  5. For External Syslog Host, enter the address of syslog server where you want logs sent, such as logs.example.com.

  6. For External Syslog Port, enter the port of your syslog server, such as 29279.

  7. For External Syslog Protocol, select the protocol for sending the logs.

  8. If you selected Yes with TLS encryption:

    1. For External Syslog Permitted Peer, enter the fingerprint or name of the remote peer. For example, *.example.com.
    2. For External Syslog TLS CA Certificate, enter the CA certificate of the syslog destination.

(Optional) Allow Additional Users to Access the Pivotal Healthwatch UI

Any user with the healthwatch.read or healthwatch.admin scope can access the Pivotal Healthwatch UI. The UAA admin user has both of these scopes by default.

Users with the healthwatch.read scope have read-only access to the Healthwatch UI. Users with the healthwatch.admin scope have admin access to the Healthwatch UI. You can add the healthwatch.read and healthwatch.admin scopes to a user through the UAA Command Line Client (UAAC).

To add the healthwatch.read or healthwatch.admin scope to a user:

  1. Access the UAAC server by running:

    uaac target UAA-URL

    Where UAA-URL is the URL for the TAS for VMs UAA instance.

  2. Log in as the admin client by running:

    uaac token client get UAA-ADMIN-CLIENT-USERNAME

    Where UAA-ADMIN-CLIENT-USERNAME is the admin client username for the TAS for VMs UAA instance.

  3. When prompted, enter the UAA admin client secret.

  4. (Optional) Add new UAA users by running:

    uaac user add NEW-USER-USERNAME --emails NEW-USER-EMAIL

    Where:

    • NEW-USER-USERNAME is the username of the new user you want to add.
    • NEW-USER-EMAIL is the email of the new user you want to add.

  5. Add the healthwatch.read or healthwatch.admin scope to the user by running:

    uaac member add SCOPE USERNAME

    Where:

    • SCOPE is either healthwatch.read or healthwatch.admin.
    • USERNAME is the username of the user to which you want to add the scope.

The above procedure can be leveraged to add the healthwatch.admin scope as well.

Note: Only users that should be allowed to change settings in Healthwatch should be granted the healthwatch.admin scope. This scope allows a user to alter the default alerting threshold values for the entire product. In future versions of Healthwatch, this user scope may be allowed additional configuration capabilities. As such, this Admin scope should not be granted to any user you wish to remain read-only access.

(Optional) Update Default Alert Thresholds

Pivotal Healthwatch includes several configurable alerts by deafult. Pivotal recommends customizing the default thresholds for these alerts based on your environment. You can determine the best threshold for your environment by monitoring the metrics over time and noting the metric values that indicate acceptable and unacceptable system performance and health.

To configure Healthwatch alert thresholds, you must first add a UAA client with healthwatch.admin authority.

To create a client to configure alert thresholds:

  1. Access the UAAC server by running:

    uaac target UAA-URL

    Where UAA-URL is the URL for the TAS for VMs UAA instance.

  2. Log in as the admin client by running:

    uaac token client get UAA-ADMIN-CLIENT-USERNAME

    Where UAA-ADMIN-CLIENT-USERNAME is the admin client username for the TAS for VMs UAA instance.

  3. When prompted, enter the client secret.

  4. Add a new UAA client by running:

    uaac client add --authorities 'healthwatch.admin' --authorized_grant_types 'client_credentials' --secret NEW-CLIENT-SECRET

    Where NEW-CLIENT-SECRET is the secret you want to set for the new UAA client.

  5. When promoted, enter the ID you want to set for the new UAA client.

  6. Log in as the new UAA client by running:

    uaac token client get NEW-CLIENT-ID

    Where NEW-CLIENT-ID is the ID of the UAA client you created in the previous steps.

  7. When prompted, enter the client secret you set in a previous step.

  8. Verify that the new UAA client has the correct permissions by running:

    uaac curl 'https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations'

    Where SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile. For example, sys.example.com.

    The command returns a list of active alert thresholds.

For more information about the default alerts included with Pivotal Healthwatch, see Supported Alerts.

For more information about updating default alert configurations, see Update Alert Configurations.