Healthwatch Architecture

Page last updated:

This topic describes the architecture of the Healthwatch, Healthwatch Exporter for VMware Tanzu Application Service for VMs (TAS for VMs), and Healthwatch Exporter for Tanzu Kubernetes Grid Integrated Edition (TKGI) tiles. This topic also describes the possible configurations for monitoring metrics across multiple foundations.

Overview of Healthwatch Architecture

There are three tiles that form the Healthwatch architecture: Healthwatch, Healthwatch Exporter for TAS for VMs, and Healthwatch Exporter for TKGI.

A complete Healthwatch installation includes the Healthwatch tile, as well as at least one Healthwatch Exporter tile. However, you can deploy and use each tile separately as part of an alternate monitoring configuration.

You must install a Healthwatch Exporter tile on each Ops Manager foundation you want to monitor. You can install the Healthwatch tile on the same foundation or on a different foundation, depending on your desired monitoring configuration.

You can also configure the Healthwatch Exporter tiles to expose metrics to a service or database located outside your Ops Manager foundation, such as an external time-series database (TSDB) or an installation of the Healthwatch tile on the TKGI Control Plane. This does not require you to install the Healthwatch tile.

For a detailed explanation of the architecture for each tile, a list of open ports required for each component, and the possible configurations for monitoring metrics across foundations, see the following sections:

Healthwatch Tile Architecture

When you install the Healthwatch tile, Healthwatch deploys instances of Prometheus, Grafana, and MySQL. Healthwatch also deploys an Nginx proxy in front of the Prometheus instance for load balancing.

The Prometheus instance scrapes and stores metrics from the Healthwatch Exporter tiles, which are scrapable Prometheus endpoints that you install on your Ops Manager foundation. Prometheus also enables you to configure alerts with Alertmanager.

Healthwatch then exports these metrics to dashboards in the Grafana UI, where you can visualize the data in charts and graphs. You can also use Grafana to create customized dashboards for long-term monitoring and troubleshooting.

Note: The MySQL instance that the Healthwatch tile installs is used only to store your Grafana settings and does not store any time series data.

The diagram below illustrates how metrics flow from the Healthwatch Exporter tiles through Prometheus and to Grafana. It also shows how metrics flow through Prometheus to Alertmanager.

An arrow points from the Healthwatch Exporter into the Healthwatch Tile. Inside the tile, the arrow points from Prometheus to Grafana and from Prometheus through Alertmanager to alerts. The MySQL box sits next to Grafana.

High Availability

You can deploy the Healthwatch tile in high availability (HA) mode with three MySQL nodes and two MySQL Proxy nodes, or in non-HA mode with one MySQL node and one MySQL Proxy node.

Component Scaling

Healthwatch deploys a single Grafana VM by default. If you need Grafana to be HA, you can scale your Grafana instance horizontally.

Healthwatch deploys two Prometheus VMs by default. You can scale your Prometheus instance vertically, but you should not scale it horizontally.

Networking Rules for the Healthwatch Tile

The table below describes the ports you must open for each Healthwatch component:

This component … Must communicate with … Default TCP Port Notes
grafana
  • tsdb
  • pxc-proxy
  • External alerting URLs
  • External data sources
  • External authentication
  • External SMTP server
  • 4449
  • 3306
Additional networking rules may be required for any external connections listed. For example, 443 for UAA.
blackbox-exporter External canary target URLs N/A Additional networking rules may be required, depending on your external canary target URL configuration.
tsdb
  • blackbox-exporter
  • All VMs deployed by Healthwatch Exporter tiles
9090
tsdb (for TKGI cluster discovery) For each cluster:
  • Kube API Server
  • Kube Controller Manager
  • Kube Scheduler
  • etcd (Telegraf output plugin)
  • 8443
  • 10252
  • 10251
  • 10200
You only need to open these ports if you configure TKGI cluster discovery.

Healthwatch Exporter for TAS for VMs Architecture

The Healthwatch Exporter for TAS for VMs tile deploys metric exporter VMs to generate each type of metric related to the health of your TAS for VMs deployment.

Healthwatch Exporter for TAS for VMs sends metrics through the Loggregator Firehose to a Prometheus exposition endpoint on the associated metric exporter VMs. The Prometheus instance that exists within your metrics monitoring system then scrapes the exposition endpoints on the metric exporter VMs and imports those metrics into your monitoring system.

You can scale the VMs that Healthwatch Exporter for TAS for VMs deploys vertically, but should not scale them horizontally.

Networking Rules for Healthwatch Exporter for TAS for VMs

The table below describes the ports you must open for each Healthwatch Exporter for TAS for VMs component:

This component … Must communicate with … Default TCP Port
bosh-deployments-exporter
  • BOSH Director UAA
  • BOSH Director
  • 8443
  • 25555
bosh-health-exporter
  • BOSH Director UAA
  • BOSH Director
  • 8443
  • 25555
cert-expiration-exporter Ops Manager 443
pas-exporter-counter Reverse Log Proxy (RLP) nozzle 8082
pas-exporter-gauge RLP nozzle 8082
pas-exporter-timer RLP nozzle 8082
pas-sli-exporter
  • CAPI
  • UAA
  • 443
  • 443

Healthwatch Exporter for TKGI Architecture

The Healthwatch Exporter for TKGI tile deploys metric exporter VMs to generate SLIs related to the health of your TKGI deployment.

The Prometheus instance that exists within your metrics monitoring system then scrapes the Prometheus exposition endpoints on the metric exporter VMs and imports those metrics into your monitoring system.

You can scale the VMs that Healthwatch Exporter for TKGI deploys vertically, but should not scale them horizontally.

Networking Rules for Healthwatch Exporter for TKGI

The table below describes the ports you must open for each Healthwatch Exporter for TKGI component:

This component … Must communicate with … Default TCP Port
bosh-deployments-exporter
  • BOSH Director UAA
  • BOSH Director
  • 8443
  • 25555
bosh-health-exporter
  • BOSH Director UAA
  • BOSH Director
  • 8443
  • 25555
cert-expiration-exporter Ops Manager 443
pks-exporter
  • BOSH Director UAA
  • BOSH Director metrics agent
  • 8443
  • 25595
pks-sli-exporter
  • TKGI API UAA
  • TKGI API
  • 8443
  • 9021

Configuration Options

Healthwatch is flexible, allowing you to monitor metrics across a variety of platform and foundation configurations. The sections below describe the most common configuration scenarios:

Monitoring TAS for VMs on a Single Ops Manager Foundation

If you only want to monitor a single Ops Manager foundation that has TAS for VMs installed, install the Healthwatch tile and Healthwatch Exporter for TAS for VMs on the same foundation. The Healthwatch tile automatically detects Healthwatch Exporter for TAS for VMs on the same foundation and adds a scrape job for Healthwatch Exporter for TAS for VMs to the Prometheus instance.

For more information about installing and configuring the Healthwatch tile and Healthwatch Exporter for TAS for VMs, see the following topics:

Monitoring TKGI on a Single Ops Manager Foundation

If you only want to monitor a single Ops Manager foundation that has TKGI installed, install the Healthwatch tile and Healthwatch Exporter for TKGI on the same foundation. The Healthwatch tile automatically detects Healthwatch Exporter for TKGI on the same foundation and adds a scrape job for Healthwatch Exporter for TKGI to the Prometheus instance.

For more information about installing and configuring the Healthwatch tile and Healthwatch Exporter for TKGI, see the following topics:

Monitoring TAS for VMs and TKGI on a Single Ops Manager Foundation

If you only want to monitor a single Ops Manager foundation that has both TAS for VMs and TKGI installed, install the Healthwatch tile, Healthwatch Exporter for TAS for VMs, and Healthwatch Exporter for TKGI on the same foundation. The Healthwatch tile automatically detects Healthwatch Exporter for TAS for VMs and Healthwatch Exporter for TKGI on the same foundation and adds scrape jobs for both Healthwatch Exporter tiles to the Prometheus instance.

For more information about installing and configuring the Healthwatch tile, Healthwatch Exporter for TAS for VMs, and Healthwatch Exporter for TKGI, see the following topics:

Monitoring TAS for VMs on a Different Ops Manager Foundation

You can monitor several Ops Manager foundations that have TAS for VMs installed from a Healthwatch tile that you install on a separate Ops Manager foundation or the TKGI Control Plane. To do so, install the Healthwatch tile on either an Ops Manager foundation or the TKGI Control Plane. Then, install Healthwatch Exporter for TAS for VMs and open the ports for the metric exporter VMs that Healthwatch Exporter for TAS for VMs deploys on each Ops Manager foundation you want to monitor. For more information about the ports you must open for each metric exporter VM, see Networking Rules for Healthwatch Exporter for TAS for VMs above.

Once you have installed Healthwatch Exporter for TAS for VMs and opened the required ports on each Ops Manager foundation you want to monitor, add a scrape job for each Healthwatch Exporter for TAS for VMs tile in the Prometheus Configuration pane of the Healthwatch tile that you installed on your monitoring Ops Manager foundation or the TKGI Control Plane. To add a scrape job for a Healthwatch Exporter TAS for VMs tile:

  1. Retrieve the Ops Manager root certificate authority (CA) for the foundation you want to monitor. For more information, see Retrieve the Ops Manager Root CA in Managing Certificates with the Ops Manager API in the Ops Manager documentation.

  2. Nagivate to the Ops Manager Installation Dashboard for the foundation you want to monitor.

  3. Click the Healthwatch Exporter for Tanzu Application Service tile.

  4. Select the Credentials tab.

  5. In the row for Healthwatch Exporter Client Mtls, click Link to Credential.

  6. Record the credentials for Healthwatch Exporter Client Mtls.

  7. Navigate to the Healthwatch tile installed on your monitoring Ops Manager foundation or the TKGI Control Plane.

  8. Select Prometheus Configuration.

  9. Under Additional Scrape Config Jobs, click Add.

  10. For TSDB Scrape job, provide the configuration YAML for the scrape job for Healthwatch Exporter for TAS for VMs, similar to the following example:

    - job_name: FOUNDATION-NAME
      metrics_path: /metrics
      scheme: https
      static_configs:
        - targets:
          - "<gauge exporter ip>:9090"
          - "<counter exporter ip>:9090"
          - "<timer exporter ip>:9090"
    

    Where FOUNDATION-NAME is the name of the foundation you want to monitor.

  11. For TLS Config Certificate Authority, enter the Ops Manager root CA that you retrieved in a previous step.

  12. For TLS Config Certificate and Private Key, enter the certificate and private key from Healthwatch Exporter Client Mtls that you recorded from the Credentials tab in the Healthwatch Exporter for TAS for VMs tile in a previous step.

  13. For TLS Config Server Name, enter the name of the server that facilitates TLS communication between the Prometheus instance in the Healthwatch tile and the metric exporter VMs that Healthwatch Exporter for TAS for VMs deploys.

Monitoring TKGI on a Different Ops Manager Foundation

You can monitor several Ops Manager foundations that have TKGI installed from a Healthwatch tile that you install on a separate Ops Manager foundation or the TKGI Control Plane. To do so, install the Healthwatch tile on either an Ops Manager foundation or the TKGI Control Plane. Then, install Healthwatch Exporter for TKGI and open the ports for the metric exporter VMs that Healthwatch Exporter for TKGI deploys on each Ops Manager foundation you want to monitor. For more information about the ports you must open for each metric exporter VM, see Networking Rules for Healthwatch Exporter for TKGI above.

Once you have installed Healthwatch Exporter for TKGI and opened the required ports on each Ops Manager foundation you want to monitor, add a scrape job for each Healthwatch Exporter for TKGI tile in the Prometheus Configuration pane of the Healthwatch tile that you installed on your monitoring Ops Manager foundation or the TKGI Control Plane. To add a scrape job for a Healthwatch Exporter TKGI tile:

  1. Retrieve the Ops Manager root CA for the foundation you want to monitor. For more information, see Retrieve the Ops Manager Root CA in Managing Certificates with the Ops Manager API in the Ops Manager documentation.

  2. Nagivate to the Ops Manager Installation Dashboard for the foundation you want to monitor.

  3. Click the Healthwatch Exporter for Tanzu Kubernetes Grid - Integrated tile.

  4. Select the Credentials tab.

  5. In the row for Healthwatch Exporter Client Mtls, click Link to Credential.

  6. Record the credentials for Healthwatch Exporter Client Mtls.

  7. Navigate to the Healthwatch tile installed on your monitoring Ops Manager foundation or the TKGI Control Plane.

  8. Select Prometheus Configuration.

  9. Under Additional Scrape Config Jobs, click Add.

  10. For TSDB Scrape job, provide the configuration YAML for the scrape job for Healthwatch Exporter for TKGI, similar to the following example:

    - job_name: FOUNDATION-NAME
      metrics_path: /metrics
      scheme: https
      static_configs:
        - targets:
          - "<gauge exporter ip>:9090"
          - "<counter exporter ip>:9090"
          - "<timer exporter ip>:9090"
    

    Where FOUNDATION-NAME is the name of the foundation you want to monitor.

  11. For TLS Config Certificate Authority, enter the Ops Manager root CA that you retrieved in a previous step.

  12. For TLS Config Certificate and Private Key, enter the certificate and private key from Healthwatch Exporter Client Mtls that you recorded from the Credentials tab in the Healthwatch Exporter for TKGI tile in a previous step.

  13. For TLS Config Server Name, enter the name of the server that facilitates TLS communication between the Prometheus instance in the Healthwatch tile and the metric exporter VMs that Healthwatch Exporter for TKGI deploys.