Installing and Configuring AppDynamics Platform Monitoring for PCF

This topic describes how to install and configure AppDynamics Platform Monitoring for Pivotal Cloud Foundry (PCF). The tile allows you to deploy AppDynamics Firehose Nozzle that collects various KPI metrics and a Dashboard that automatically creates health rules and custom dashboards associated with the metrics.

Additionally the tile allows for advanced configuration to customize metric collection, configuring multi-foundation dashboard and overiding default configurations.

WARNING: Starting with 4.6 version the default AppDynamics Nozzle Application Name has been changed to appd-nozzle-v2 and a new dashboard with name appd-nozzle-v2-<{cf system domain}>-PCF KPI Dashboard will be created.

Overview

The tile deploys an AppDynamics Nozzle and Dashboard App as PCF apps to the org and space appdynamics-org/appdynamics-space. The diagram illustrates the AppDynamics Nozzle that is deployed by the tile. The Nozzle aggregates and forwards metrics from the  Loggregator Firehose to the AppDynamics Controller. The metrics are published in the Controller as custom metrics under the default application name, appd-nozzle-v2, and tier that corresponds to the system domain of the PCF foundation.

AppD Nozzle to Loggregator Firehose

The Dashboard App deployed by the tile will automatically create a set of KPI health rules and a Single Foundation dashboard in the controller configured in the Nozzle Configuration, which reflect the health and capacity of the foundation where the tile is installed.

AppD Nozzle to Loggregator Firehose

In addition, based on optional configuration added in the tile under AppDynamics Dashboard Configuration, the Dashboard App will create one or more Multi-Foundation Dashboards. These dashboards summarize the health and capacity of multiple foundations, assuming each foundation has the AppDynamics Platform Monitoring for PCF tile installed, and each tile is configured to report to the same AppDynamics controller, as indicated in the diagram.

AppD Nozzle to Loggregator Firehose

Before You Start

  1. Create a Pivotal Network account if you do not have one.

  2. Set up the AppDynamics Controller.

  3. Acquire at least one AppDynamics Go SDK Agent License. For more information, see License Management, and License Entitlements and Restrictions.

Configuring the AppDynamics Nozzle for PCF

  1. Log in to the Ops Manager.

  2. If the AppDynamics Platform Monitoring tile does not appear in the Ops Manager Installation Dashboard, click Import a Product in the left-side menu to import the AppDynamics Platform Monitoring tile.

  3. Double-click the AppDynamics Platform Monitoring for PCF tile in the Installation Dashboard. A Settings tab is displayed.

  4. Fill in the details for the following forms on the tile.

AppDynamics Nozzle Configuration

This form takes in information to establish a connection with Cloud Foundry Loggregator and AppDynamics Controller.

Settings tab for AppDynamics Nozzle Configuration

  • Doppler Logging Endpoint: The Firehose endpoint address, doppler_logging_endpoint value.

    • Run the cf curl /v2/info command to obtain the values of The UAA token_endpoint and Doppler Logging Endpoint doppler_logging_endpoint URL respectively.
    $ cf curl /v2/info
    {
    ...
    "doppler_logging_endpoint": "wss://doppler.sys.pie-multi-az-blue.cfplatformeng.com:443",
    }
    
  • Nozzle Username: The username for a member account belonging to the doppler.firehose group. This is the authorized UAA user that AppDynamics Nozzle uses to access the Loggregator.

  • Nozzle Password: The password for the above member account.

  • Controller Account Name: The name of the account in the AppDynamics Controller. Access the account name in the AppDynamics Controller UI on the Account tab under License.

  • Account Access Key: The access key for the AppDynamics Controller account. Access the account key in the AppDynamics Controller UI on the Account tab under License.

  • Controller Host: The hostname of the AppDynamics Controller.

  • Controller Port: The port on which the AppDynamics Controller listens for Nozzle traffic.

  • Enable Controller SSL: Establishes secures communication with the Controller.

  • AppDynamics Controller Proxy: Proxy url, if necessary, used by the Nozzle and Dashboard App to communicate with the AppDynamics Controller. Example: https://user:password@proxy.com:9000

Nozzle Credentials

  • Cloud Foundry (CF) environments usually ship with an opentsdb-firehose-nozzle account which already belongs to the doppler.firehose group. To use the existing opentsdb-firehose-nozzle account shipped with CF, log in to the OPS Manager and select the PAS tile and navigate to credentials > UAA > Opentsdb Nozzle Credentials and copy the username and password.
  • (or) Create a new account in the doppler.firehose group with the correct permissions. For more information, see UAA Client.

AppDynamics Dashboard Configuration

This form takes in information to create a Dashboard and associated health rules on the AppDynamics Controller

Settings tab for AppDynamics Dashboard Configuration

  • Username for Reporting Custom Dashboard: Username of an account with custom dashboard creation permissions.
  • Password: Password for the Controller account.

Refer this section for setting up a controller account with required permissions

Multi-foundation Dashboard Configuration (Optional)

The Multi-foundation Dashboard configuration supports creating a single dashboard that displays the health and performance of multiple foundations. Each foundation that is included must have the platform tile installed and configure the Nozzle to report to the same controller.

One Multi-foundation Dashboard is created for each configuration added in the Multi- foundation Dashboard Configurations section. In the example below, ‘all-foundations’ is the name of the new configuration, and will be prepended to the name of the dashboard created in the controller, which uses the format <config name>-PCF Multi-foundation Dashboard.

08multidashboardconfig

The ‘Application regex’ and ‘Tier regex’ should match one or more app/tier combinations where AppDynamics Nozzles are reporting. Based on the default app and tier, the example uses ‘appd- nozzle-v2’ as the app regex, and sys.* as the tier regex, which will match any system domain.

The ‘Dashboard layout selection’ defaults to ‘Auto select layout’ where the Dashboard App will select a layout with the appropriate number of foundations per row based on the total number of foundations discovered. Or a specific layout can be chosen to control the density of the created dashboard.

Below are examples of the ‘Two foundations per row’ and ‘Four foundations per row’ layouts.

Four foundations per row layout

Two foundations per row layout

(Optional) Extending Metrics Collection

Starting with the 4.6.* version of the tile we have included a feature to collect additional metrics in addition to the default standard KPI metrics that are collected by AppDynamics Nozzle.

Settings tab for AppDynamics Advanced Metrics

  • Collect Container Metrics: Check this box to collect Container metrics for the applications.

  • Extended Metrics JSON Filter: AppDynamics Nozzle aplies a filter to limit the number of metrics that are processed. This JSON configures the filter. It is a simple JSON dictionary with metrics origin tag used as keys and the list of names of metrics that belong to the origin that are to be collected as value. Additionally, a wildcard "*" can also be used inside the list to collect all metrics under the origin.

{
    "<origin>":["<MetricName1>", "<MetricName2>"],
    "<origin>":["*"]
}

e.g: Consider this metric with origin set to uaa with name audit_service.principal_not_found_count

origin:"uaa" eventType:ValueMetric timestamp:1553201379027067175 deployment:"cf" job:"uaa" index:"1e586719-7e22-4cb4-944d-2176db9e01b8"
 ip:"192.168.16.21" tags:<key:"source_id" value:"uaa" >
 valueMetric:<name:"audit_service.principal_not_found_count" value:0 unit:"gauge" >

to report the metric the filter entry will be

{
    "uaa": ["audit_service.principal_not_found_count"]
}

(Optional) Overriding Defaults

For 4.4.x controllers it is mandatory to override the default Application, Tier, Tier ID and Node values.

Settings tab for AppDynamics Default Overrides

  • UAA Token Endpoint: UAA token end point of the cloudfoundry. This can be found by doing $ cf curl /v2/info | grep token_endpoint
  • Nozzle Application Name: Enter a name to identify the Nozzle Application on the Controller, such as test-app-nozzle.
  • Nozzle Tier Name: Enter a name for the Nozzle Tier that corresponds to the Nozzle Application Name, such as test-app-tier. This MUST be a Go SDK Tier.
  • Nozzle Node Name: Enter a name for the Nozzle Node that corresponds to the Nozzle Application Name, such as test-app-node.
  • Nozzle Tier ID: Enter the component ID of the GoLang SDK tier.

4.4.x Controllers

For 4.4.x controller, please create an applicaiton on the AppDynamics Controller and create a Go SDK tier associated with that application and record the tier-id. Enter the AppDynamics AppName, Tier, TierID and Node names in the corresponding fields of the Overriding Defaults form.

Validating an Install

Validating the Nozzle

Once the tile install completes successfully in the Ops Manager, confirm that the Nozzle is reporting metrics to the controller by navigating to the appd-nozzle-v2 app and opening the Metric Browser. If the Nozzle is successfully reporting metrics, it will create the path Application Infrastructure Performance|<tier>|Custom Metrics|PCF Firehose Monitor, where the tier corresponds to the system domain of the PCF foundation. Navigate to a specific metric such as Diego Cell Metrics|rep|cf|diego_cell||rep. CapacityRemainingMemory as shown below, and doubleclick it to add it to the timeseries graph and confirm metrics are reporting in the last 5 minutes.

Pcf foundation 05

If the metrics don’t appear, use the Apps Manager to tail the logs for the Nozzle appdnozzle- <version> under appdynamics-org/appdynamics-space. Tailing the logs from the Apps Manager will ensure getting the startup logs for the Nozzle, which are typically lost when using `cf logs. Follow this procedure:

  1. Set APPD_DEBUG to true in the Apps Manager under Settings/User Provided Environment Variables.

    Pcf foundation 06

    This will cause the agent logs to be forwarded to standard err rather than written to the container file system, so they’ll be visible in the app logs.

  2. Restage the Nozzle in the Apps Manager.

  3. In the Apps Manager, go to Logs and press the play button to tail the logs as the Nozzle restages. You should see some [ERR] log messages in red that are specific to the SDK agent and would indicate any problems communicating with the Controller. The [OUT] log messages should show ‘Handling x events’ where x is non-zero and would indicate successful communication with the Loggregator Firehose.

    Pcf foundation 07

Validating the Dashboard App

Check that the KPI health rules have been created in the appd-nozzle-v2 app in the controller configured in the tile under Nozzle Configuration. Navigate to Applications/appd-nozzle-v2/Alert & Respond/Health Rules. You should see a set of over 100 health rules whose names are prefixed with the system domain of the PCF foundation where the tile was installed.

Open the the Single Foundation dashboard under Dashboards & Reports with the name appd-nozzle-v2-<system domain>-PCF KPI Dashboard and check that there are no broken links and the widgets are showing recent data.

If any Multi-foundation dashboards were configured, check that they are also listed under Dashboards & Reports, with the name <config name>-PCF Multi-foundation Dashboard.

If the health rules or dashboards are not listed, restage and check the logs for the Dashboard App appdpcfdashboard-<version> in the Apps Manager under the org and space appdynamics-org/appdynamcis-space, as was described for the Nozzle. Optionally set the APPD_DEBUG flag to true.

The Dashboard App requires that the Nozzle successfully publish metrics to the Controller before it will create the KPI health rules and dashboards. If the Dashboard App does not find the metrics, it will output a message to its log:

"retrying fetching metrics exponentially"

and eventually:

  “error: failed to find PCF metric path in target controller…”

If you see these messages, first resolve the issue with the Nozzle by following the Nozzle validation steps described above.

AppDynamics Nozzle Scalability

Starting with the 4.6.* version of the tile, AppDynamics Nozzle is completely scalable to any number of desired instances. However, it is recommended to scale the number of nozzle instances to match the number of traffic controller instances. See the official cloud foundry recommendation.

Multiple PCF Foundations

Multiple AppDynamics Platform Monitoring tiles can be deployed on multiple PCF foundations, using a single AppDynamics Controller. The metrics corresponding to each of the foundations are recorded under tier names which defaults to the system domain’s value of the Cloud Foundry deployment, for example: sys.pie-multi-az-blue.cfplatformeng.com Optionally, the tier names can be customized by setting Nozzle Tier Name field in the Overriding Defaults form.