Configuration Overview

Monitoring TAS from within Foundation

In order to monitor a single TAS installation, follow the instructions for Installing Healthwatch and the instructions for Installing Healthwatch Exporter for TAS on the same foundation where TAS is installed. Healthwatch will automatically detect that the Healthwatch Exporter for TAS has also been installed and will start scraping its metrics.

Monitoring PKS from within Foundation

In order to monitor a single PKS installation, follow the instructions for Installing Healthwatch on the same foundation where PKS is installed. Healthwatch will automatically detect that the Healthwatch Exporter for PKS is also installed and will begin scraping its metrics.

Monitoring TAS and PKS

In order to monitor a foundation with both TAS and PKS installed, follow the instructions for Installing Healthwatch, the instructions for Installing Healthwatch Exporter for TAS, and the instructions for Installing Healthwatch Exporter for PKS, on the same foundation where TAS and PKS are installed. Healthwatch will automatically detect that it is co-located with both Healthwatch Exporter for TAS and Healthwatch Exporter for PKS and will begin scraping both endpoints.

Monitoring TAS from the Control Plane

The Healthwatch tile can be used from the Control Plane to monitor any number of TAS foundations. To use Healthwatch in this way, follow the installation instructions for Installing Healthwatch on the control plane, then follow the installation instructions for Installing Healthwatch Exporter for TAS on each TAS foundation you would like to monitor.

Once you have finished installing Healthwatch Exporter for TAS and completed the necessary network configuration to expose the exporter VMs, add a scrape config to the TSDB Configuration section of the Healthwatch tile in the Control Plane to scrape the exporter VMs. You can get mTLS client credentials for this scrape config by grabbing the TAS Exporter Client Mtls credentials from the Credentials tab of the Healthwatch Exporter for TAS tile in Ops Manager. Your config will look something like the following:

- job_name: foundation_name
  metrics_path: /metrics
  scheme: https
  tls_config:
    ca: |
      -----BEGIN CERTIFICATE-----
      MIIEijCCA3KgAwIBAgIQVNDMqn2R/G08qg7VBDwoLzANBgkqhkiG9w0BAQsFADBU
      MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMSUw
      ...
      Kqj0ATjsh3/4L7paXAlnhrAzrlmEBclKUaWxY7xati5zqfkQIWXUey6JFbSlOqwl
      fOThhUwPPzIy/CtSCKY=
      -----END CERTIFICATE-----
    cert: |
      -----BEGIN CERTIFICATE-----
      MIIEijCCA3KgAwIBAgIQVNDMqn2R/G08qg7VBDwoLzANBgkqhkiG9w0BAQsFADBU
      3QM7YO2iIHA03VLkH2/Y8UPys2cjtRxMkiTBY3gYrdnP82ymw+6DgvHVodfCgVNk
      ...
      fQMxJ27wPIzEuB0NkOferZEi318PRwTJWkoEFE30Q+aKoXnWmWIs4chUTeGrNTNU
      fOTAAUwCCzIy/PIKWY=
      -----END CERTIFICATE-----
    key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEpAIBAAKCAQEA5OrJVeDocSD+LAC86vajwwzHk2Dflv3b3tCOMO/mO/9hH/x5
      wszRJv8wdckUkJrRv9GSbbZGDd0FmMsOl+/SP4WKaKFsQh6Uig1La/W0sMf4AK0M
      ...
      1qLvlv1R8T4ZKAi99VYt3g73NjhqgDXt/BtwEXWklfl72I4vIV/VcNaIGuAtCjrX
      TaHxfBvtOpqNAB1dKQ8tE3gXRxnyHlmQJkwKjUWLIeTkXgVOaZr6Pw==
      -----END RSA PRIVATE KEY-----
    server_name: TASexporter
  static_configs:
    - targets:
      - "<gauge exporter ip>:9090"
      - "<counter exporter ip>:9090"
      - "<timer exporter ip>:9090"

Note: Finding Certificates

The CA certificate is the one generated by Ops Manager on the foundation where the Exporter is installed. It can be retrieved from Ops-Manager-URL/api/v0/certificate_authorities. The certificates for the Exporter can be found on the same Ops Manager under the Exporter tile’s Credentials tab. The name of the credential is TAS Exporter Client Mtls.

Monitoring PKS from the Control Plane

The Healthwatch tile can be used from the Control Plane to monitor any number of PKS foundations. To use Healthwatch in this way, follow the installation instructions for Installing Healthwatch the control plane, then follow the installation instructions for Installing Healthwatch Exporter for PKS on each PKS foundation you would like to monitor.

Once you have finished installing Healthwatch Exporter for PKS and completed the necessary network configuration to expose the exporter VMs, add a scrape config to the TSDB Configuration section of the Healthwatch tile in the Control Plane to scrape the exporter VMs. You can get mTLS client credentials for this scrape config by grabbing the Pks Exporter Client Mtls credentials from the Credentials tab of the Healthwatch Exporter for PKS tile in Ops Manager. Your config will look something like the following:

- job_name: foundation_name
  metrics_path: /metrics
  scheme: https
  tls_config:
    ca: |
      -----BEGIN CERTIFICATE-----
      MIIEijCCA3KgAwIBAgIQVNDMqn2R/G08qg7VBDwoLzANBgkqhkiG9w0BAQsFADBU
      MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMSUw
      ...
      Kqj0ATjsh3/4L7paXAlnhrAzrlmEBclKUaWxY7xati5zqfkQIWXUey6JFbSlOqwl
      fOThhUwPPzIy/CtSCKY=
      -----END CERTIFICATE-----
    cert: |
      -----BEGIN CERTIFICATE-----
      MIIEijCCA3KgAwIBAgIQVNDMqn2R/G08qg7VBDwoLzANBgkqhkiG9w0BAQsFADBU
      3QM7YO2iIHA03VLkH2/Y8UPys2cjtRxMkiTBY3gYrdnP82ymw+6DgvHVodfCgVNk
      ...
      fQMxJ27wPIzEuB0NkOferZEi318PRwTJWkoEFE30Q+aKoXnWmWIs4chUTeGrNTNU
      fOTAAUwCCzIy/PIKWY=
      -----END CERTIFICATE-----
    key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEpAIBAAKCAQEA5OrJVeDocSD+LAC86vajwwzHk2Dflv3b3tCOMO/mO/9hH/x5
      wszRJv8wdckUkJrRv9GSbbZGDd0FmMsOl+/SP4WKaKFsQh6Uig1La/W0sMf4AK0M
      ...
      1qLvlv1R8T4ZKAi99VYt3g73NjhqgDXt/BtwEXWklfl72I4vIV/VcNaIGuAtCjrX
      TaHxfBvtOpqNAB1dKQ8tE3gXRxnyHlmQJkwKjUWLIeTkXgVOaZr6Pw==
      -----END RSA PRIVATE KEY-----
    server_name: pksexporter
  static_configs:
    - targets:
      - "<exporter ip>:9090"

Note: Finding Certificates

The CA certificate is the one generated by Ops Manager on the foundation where the Exporter is installed. It can be retrieved from Ops-Manager-URL/api/v0/certificate_authorities. The certificates for the Exporter can be found on the same Ops Manager under the Exporter tile’s Credentials tab. The name of the credential is Pks Exporter Client Mtls.

Adding Metrics to an Allowlist

In some deployments, it can be beneficial to only ingest certain metrics from a scrape job. For example, due to storage and cpu constraints, an operator may only want to store SLI metrics for a foundation, rather than the entirety of the Firehose. The following example shows how an operator could configure a scrape job to only ingest gauge metrics with the names some-metric or some-metric-2.

- job_name: allowlisted-job
  metric_relabel_configs:
  - source_labels: [__name__]
    regex: (some-metric|some-metric-2)
    action: keep
  metrics_path: /metrics
  scheme: https
  tls_config:
    ca: |
      -----BEGIN CERTIFICATE-----
      MIIEijCCA3KgAwIBAgIQVNDMqn2R/G08qg7VBDwoLzANBgkqhkiG9w0BAQsFADBU
      MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMSUw
      ...
      Kqj0ATjsh3/4L7paXAlnhrAzrlmEBclKUaWxY7xati5zqfkQIWXUey6JFbSlOqwl
      fOThhUwPPzIy/CtSCKY=
      -----END CERTIFICATE-----
    cert: |
      -----BEGIN CERTIFICATE-----
      MIIEijCCA3KgAwIBAgIQVNDMqn2R/G08qg7VBDwoLzANBgkqhkiG9w0BAQsFADBU
      3QM7YO2iIHA03VLkH2/Y8UPys2cjtRxMkiTBY3gYrdnP82ymw+6DgvHVodfCgVNk
      ...
      fQMxJ27wPIzEuB0NkOferZEi318PRwTJWkoEFE30Q+aKoXnWmWIs4chUTeGrNTNU
      fOTAAUwCCzIy/PIKWY=
      -----END CERTIFICATE-----
    key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEpAIBAAKCAQEA5OrJVeDocSD+LAC86vajwwzHk2Dflv3b3tCOMO/mO/9hH/x5
      wszRJv8wdckUkJrRv9GSbbZGDd0FmMsOl+/SP4WKaKFsQh6Uig1La/W0sMf4AK0M
      ...
      1qLvlv1R8T4ZKAi99VYt3g73NjhqgDXt/BtwEXWklfl72I4vIV/VcNaIGuAtCjrX
      TaHxfBvtOpqNAB1dKQ8tE3gXRxnyHlmQJkwKjUWLIeTkXgVOaZr6Pw==
      -----END RSA PRIVATE KEY-----
    server_name: TASexporter
  dns_sd_configs:
    - names:
        - q-s4.TAS-exporter-gauge.*.p sh.
      type: A
      port: 9090

Monitoring PKS Master Nodes

When Healthwatch is installed on the same foundation as PKS, it will automatically scrape the kube-scheduler and kube-controller-manager processes on the Master VM. More metrics (such as etcd metrics) can be made available by enabling the TSDB_client Telegraf output on the master VM. To do that, go to the Monitoring pane in the PKS tile, and enter the following configuration in the Setup Telegraf Outputs section:

[[outputs.TSDB_client]]
  listen = ":9273"

This will expose additional metrics on a /metrics endpoint on port 9273. In order to scrape these metrics, add the following scrape configuration to the TSDB Configuration pane of the Healthwatch tile:

- job_name: cluster_master_telegraf
  dns_sd_configs:
    - names:
        - q-s4.master.*.*.bosh.
      type: A
      port: 9273

This example uses Bosh DNS to automatically discover all master VMs, which now have Telegraf available on port 9273.

  1. Click Save.