Upgrading GCP Stackdriver Nozzle for PCF

This topic describes how to upgrade GCP Stackdriver Nozzle for Pivotal Cloud Foundry (PCF) to v2.0 from a v1.x release.

WARNING: This release introduces significant breaking changes for Stackdriver Nozzle users. These changes require the operator to take manual steps as part of the upgrading process. The manual steps will reset aspects of the project and will result in irreversible historical data loss and the need to recreate dashboards.

Stackdriver Monitoring Changes

All exported metrics have changed in this release.

Metric changes:

  • Metrics derived from the Firehose now have the metric origin prefixed to the metric name, e.g. gorouter.total_requests.
  • Metrics derived from the Firehose are now exported under a configurable path prefix, which defaults to firehose.
  • CounterEvent metrics are now represented as COUNTERs rather than GAUGEs in Stackdriver, so per-second rates can be computed.
  • The nozzle exports its own custom metrics under the stackdriver-nozzle prefix.

Label changes:

  • Application, space and organization UUIDs are no longer exported.
  • Resolved application, space and organization names are concatenated into a single applicationPath label.
  • The application instance index is now exported.
  • Event envelope tags are now exported as a string of comma-separated key=value pairs.
  • Origin and EventType labels are only attached to logs, not metrics.
  • There is a new foundation label that is configurable per nozzle instance, to distinguish between multiple foundations in a GCP/Stackdriver project.

Clear Metric Descriptors

Follow this guide to reset metric descriptors for your Stackdriver Monitoring project. It needs this to stay under the default custom metric descriptor quota. Ensure the stackdriver-nozzle is not running during this process.

As an alternative to this procedure, you can provision a new Google Cloud Project and configure the updated stackdriver-nozzle release to use it.

Singleton Deployment of stackdriver-nozzle

The stackdriver-nozzle keeps track of counter metrics to send to Stackdriver Monitoring at the instance level by default. If multiple copies of the stackdriver-nozzle are reporting the same metrics for a Cloud Foundry deployment, they will report incorrect data. You can mitigate this by running a single instance of the nozzle. An upstream feature in Loggregator is being tracked to address this.

Operations Manager and Operations Manager Tile users are forced to run the stackdriver-nozzle job as a single instance.

It is recommended that you vertically scale the stackdriver-nozzle VM in case of resource saturation. A number of performance optimizations in this release make sure that a single instance of the nozzle is able to provide enough throughput for a medium-sized PCF installation.

If you find the need to scale the stackdriver-nozzle job horizontally, you must use the BOSH release and follow guidance for the 2.0 upgrade.

Create a pull request or raise an issue on the source for this page in GitHub