Pivotal Cloud Cache Operator Guide

This document describes how a Pivotal Cloud Foundry (PCF) operator can install, configure, and maintain Pivotal Cloud Cache (PCC).

Requirements for Pivotal Cloud Cache

  • The Networking for On-Demand Services section describes networking requirements for PCC.

  • As of PCC v1.5.3, PCC increases security by requiring TLS encryption for gfsh and Pulse. Follow the instructions in Preparing for TLS prior to installing the tile.

Preparing for TLS

WARNING: As of PCC v1.5.3, PCC increases security by requiring TLS encryption for gfsh and Pulse. Complete the procedures in this topic before installing the PCC tile as part of an upgrade.

This topic describes how to provide an existing Certificate Authority (CA) certificate to BOSH CredHub and how to generate a new CA certificate with BOSH CredHub, if you do not already have one.

WARNING: This procedure involves restarting all of the VMs in your PCF deployment in order to propagate a CA certificate. The operation can take a long time to complete.

Overview

Enabling TLS provisions PCC service instances with a certificate so that apps, gfsh, and Pulse can establish an encrypted connection with the PCC service instance.

The certificate deployed on the PCC service instance is a server certificate. The server certificate is generated by CredHub, a component designed for centralized credential management in PCF. CredHub is deployed on the same VM as the BOSH Director.

CredHub generates the server certificate using a Certificate Authority (CA) certificate. The CA certificate must be provided to CredHub by the operator or generated by CredHub.

Apps use the CA certificate to authenticate components of PCC service instances. Apps that communicate with PCC must have access to the CA certificate in order to validate that the server certificate can be trusted.

WARNING: An operator must rotate the CA certificate if it expires or if it becomes compromised. To rotate your CA certificate, see Rotating CA Certificates for Pivotal Cloud Foundry Services in the Pivotal Knowledge Base. Do not attempt to rotate a CA certificate on your own. Contact Pivotal Support and perform the procedure in the Pivotal Knowledge Base article with their assistance.

Provide or Generate a CA Certificate

Perform the following procedures to create a User Account and Authentication (UAA) client for CredHub, log in to CredHub, and provide or generate a CA certificate.

Create a UAA Client

Perform the following steps to create a UAA client for CredHub on your UAA server:

  1. Retrieve the IP address of the BOSH Director VM and the Director credentials by performing the steps in Gather Credential and IP Address Information.

    Both the UAA and CredHub servers are colocated on the BOSH Director VM.

  2. SSH into the Ops Manager VM by performing the steps in SSH into Ops Manager VM.

  3. From the Ops Manager VM, use the UAA Command Line Interface (UAAC) to target the UAA server on the BOSH Director VM. In the UAAC command, specify the IP address for the BOSH Director VM and port 8443.

    Run the following command:

    uaac target BOSH-DIRECTOR:8443
    


    where BOSH-DIRECTOR is the IP address of the BOSH Director VM. You retrieved this address from the Status tab of the Ops Manager Director tile in step 1.

    For example:

    $ uaac target 10.0.0.5:8443
    

  4. In the Credentials tab of the Ops Manager Director tile, retrieve the UAA Login Client Credentials and record the identity and password values.

    Note: These are the credentials for the UAA server colocated on the BOSH Director, not the UAA server colocated on Pivotal Application Service.

  5. Retrieve the UAA Admin User Credentials and record the identity and password values.

  6. From the Ops Manager VM, use the UAAC to get a token.

    Run the following command:

    uaac token owner get login --secret=UAA-LOGIN-CLIENT-CRED
    


    where UAA-LOGIN-CLIENT-CRED is the password value of the UAA Login Client Credentials that you retrieved in step 4.

    For example:

    $ uaac token owner get \
        login --secret=abcdefghijklm123456789
    

  7. When prompted for a user name and password, enter the values for identity and password of the UAA Admin User Credentials that you retrieved in step 5. For example:

    User name:  admin
    Password:  ********************************
    

  8. Add a UAA client for CredHub with the correct grants.

    Enter the following command:

    $ uaac client add \
        --authorized_grant_types client_credentials \
        --authorities credhub.read,credhub.write
    
  9. When prompted for Client ID, enter credhub. When prompted for New client secret, enter a secure password of your choice. For example:

    Client ID:  credhub
    New client secret:  *******
    Verify new client secret:  *******
      scope: uaa.none
      client_id: credhub
      resource_ids: none
      authorized_grant_types: client_credentials
      autoapprove:
      authorities: credhub.write credhub.read
      name: credhub
      required_user_groups:
      lastmodified: 1518198701452
      id: credhub
      created_by: f609e861-39ec-4a16-8aee-cba9e9b079e3
    

Add the CA Certificate

Perform the following steps to log in to CredHub, provide or generate a CA certificate, and add the certificate to Ops Manager:

  1. From the Ops Manager VM, set the API target of the CredHub CLI to your CredHub server.

    Run the following command:

    credhub api https://BOSH-DIRECTOR:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
    


    where BOSH-DIRECTOR is the IP address of the BOSH Director VM.

    For example:

    $ credhub api https://10.0.0.5:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
    

  2. Log in to CredHub.

    Run the following command:

    credhub login --client-name=credhub --client-secret=CLIENT-SECRET
    


    where CLIENT-SECRET is the client secret you set in step 9 above.

    For example:

    $ credhub login \
        --client-name=credhub \
        --client-secret=abcdefghijklm123456789
    

  3. Use the CredHub CLI to check whether a services CA certificate already is present.

    • Enter the following command:
      $ credhub get \
        --name="/services/tls_ca"
      
    • If you already have a certificate at the services/tls_ca path, skip to step 5.
  4. Use the CredHub CLI to generate a CA certificate or provide an existing one.

    Note: Your PCF deployment may have multiple CA certificates. Pivotal recommends a dedicated CA certificate for services.

    • If you do not have a CA certificate, use the CredHub CLI to generate one. Enter the following command:
      $ credhub generate \
          --name="/services/tls_ca" \
          --type="certificate" \
          --no-overwrite \
          --is-ca \
          --common-name="rootCA"
      
    • If you have an existing CA certificate that you want to use, create a new file called root.pem with the contents of the certificate. Then enter the following command, specifying the path to root.pem and the private key for the certificate:
      $ credhub set \
          --name="/services/tls_ca" \
          --type="certificate" \
          --certificate=./root.pem \
          --private=ERKSOSMFF...
      
  5. Use the BOSH CLI v2 to extract the certificate portion from the CA certificate and print it. Enter the following command:

    $ bosh2 interpolate <(credhub get --name=/services/tls_ca) \
    --path /value/certificate
    

  6. Record the output of the bosh2 interpolate command from step 4.

  7. Navigate to the Ops Manager Installation Dashboard and select the Ops Manager Director tile. Click Security.

  8. Paste the contents of the CA certificate into Trusted Certificates and click Save.

  9. The CA certificate must also be added for the Gorouter. Navigate to the PAS Settings tab. Click on Networking. Add the CA certificate to the box labeled Certificate Authorities Trusted by Router and HAProxy and click Save.

  10. Optionally, if you are using Ops Manager v2.3 or later, click Review Pending Changes (see Reviewing Pending Product Changes).

  11. Click Apply Changes.

Installing and Configuring Pivotal Cloud Cache

With an Ops Manager role (detailed in Understand Roles in Ops Manager) that has the proper permissions to install and configure, follow these steps to install PCC on PCF:

  1. Download the tile from the Pivotal Network.
  2. Click Import a Product to import the tile into Ops Manager.
  3. Click the + symbol next to the uploaded product description.
  4. Click on the Cloud Cache tile.
  5. Complete all the configuration steps in the Configure Tile Properties section below.
  6. Return to the Ops Manager Installation Dashboard. Optionally, if you are using Ops Manager v2.3 or later, click Review Pending Changes (see Reviewing Pending Product Changes).
  7. Click Apply Changes to complete the installation of the PCC tile.

Configure Tile Properties

Configure the sections listed on the left side of the page.

Tile Configuration Sections

As you complete a section, save it. A green check mark appears next to the section name. Each section name must show this green check mark before you can complete your installation.

Assign Availability Zones and Networks

To select AZs and networks for VMs used by PCC, do the following:

  1. Click Assign AZs and Networks.

  2. Configure the fields on the Assign AZs and Networks pane as follows:

    FieldInstructions
    Place singleton jobs in Select the region that you want for singleton VMs.
    Balance other jobs in Select the AZ(s) you want to use for distributing other GemFire VMs. Pivotal recommends selecting all of them.
    Network Select your PAS (or Elastic Runtime) network.
    Service Network Select the network to be used for GemFire VMs.

  3. Click Save.

Settings

Smoke Test Settings

The smoke-tests errand that runs after tile installation. The errand verifies that your installation was successful. By default, the smoke-test errand runs on the system org and the p-cloudcache-smoke-test space.

Note: Smoke tests will fail unless you enable global default application security groups (ASGs). You can enable global default ASGs by binding the ASG to the system org without specifying a space. To enable global default ASGs, use cf bind-running-security-group.

To select which plan you want to use for smoke tests, do the following:

Select a plan to use when the smoke-tests errand runs.

Ensure the selected plan is enabled and configured. For information about configuring plans, see Configure Service Plans below. If the selected plan is not enabled, the smoke-tests errand fails.

Pivotal recommends that you use the smallest four-server plan for smoke tests. Because smoke tests create and later destroy this plan, using a very small plan reduces installation time. Configuring Smoke Tests Plan

Settings: Allow Outbound Internet Access Settings

By default, outbound internet access is not allowed from service instances.

If BOSH is configured to use an external blob store, you need allow outbound internet access from service instances. Log forwarding and backups, which require external endpoints, might also require internet access.

To allow outbound internet access from service instance, do the following:

Select Allow outbound internet access from service instances (IaaS-dependent).

Outbound Internet Access

Note: Outbound network traffic rules also depend on your IaaS settings. Consult your network or IaaS administrator to ensure that your IaaS allows outbound traffic to the external networks you need.

Default Distributed System ID Setting

Every service instance has an integer identifier called a distributed system ID. The ID defaults to the value 0. Service instances that form a distributed system that communicates across a WAN will need distinct IDs. Those distinct ID values are set when creating the service instance.

To change the default distributed system ID value, replace the default value of 0 with your new default value. Acceptable values are integers greater than or equal to 0 and less than or equal to 255.

Default Distributed System ID setting

Configure Service Plans

You can configure five individual plans for your developers. Select the Plan 1 through Plan 5 tabs to configure each of them.

Configuring a Plan

The Plan Enabled option is selected by default. If you do not want to add this plan to the CF service catalog, select Plan Disabled. You must enable at least one plan.

The Plan Name text field allows you to customize the name of the plan. This plan name is displayed to developers when they view the service in the Marketplace.

The Plan Description text field allows you to supply a plan description. The description is displayed to developers when they view the service in the Marketplace.

The Enable metrics for service instances checkbox enables metrics for service instances created using the plan. Once enabled, the metrics are sent to the Loggregator Firehose.

The CF Service Access drop-down menu gives you the option to display or not display the service plan in the Marketplace. Enable Service Access displays the service plan the Marketplace. Disable Service Access makes the plan unavailable in the Marketplace. If you choose this option, you cannot make the plan available at a later time. Leave Service Access Unchanged makes the plan unavailable in the Marketplace by default, but allows you to make it available at a later time.

The Service Instance Quota sets the maximum number of PCC clusters that can exist simultaneously.

When developers create or update a service instance, they can specify the number of servers in the cluster. The Maximum servers per cluster field allows operators to set an upper bound on the number of servers developers can request. If developers do not explicitly specify the number of servers in a service instance, a new cluster has the number of servers specified in the Default Number of Servers field.

The Availability zones for service instances setting determines which AZs are used for a particular cluster. The members of a cluster are distributed evenly across AZs.

WARNING! After you’ve selected AZs for your service network, you cannot add additional AZs; doing so causes existing service instances to lose data on update.

The remaining fields control the VM type and persistent disk type for servers and locators. The total size of the cache is directly related to the number of servers and the amount of memory of the selected server VM type. We recommend the following configuration:

  • For the VM type for the Locator VMs field, select a VM that has at least 2 CPUs, 1 GB of RAM and 4 GB of disk space.
  • For the Persistent disk type for the Locator VMs field, select 10 GB or higher.
  • For the VM type for the Server VMs field, select a VM that has at least 2 CPUs, 4 GB of RAM and 8 GB of disk space.
  • For the Persistent disk type for the server VMs field, select 10 GB or higher.

When you finish configuring the plan, click Save to save your configuration options.

Configure a Dev Plan

A Dev Plan is a type of service plan. Use a Dev Plan for development and testing. The plan provides a single locator and server, which are colocated within a single VM.

The page for configuring a Dev Plan is similar to the page for configuring other service plans. To configure the Dev Plan, input information in the fields and make selections from the options on the Plan for test development page.

Dev Plan Configuration Sections

If you have enabled post-deploy scripts in your BOSH Director, a region is automatically created. To confirm that post-deploy scripts are enabled, navigate to the Director Config pane of Ops Manger Director and verify that Enable Post Deploy Scripts is selected.

Enable post deploy scripts

Syslog

By default, syslog forwarding is not enabled in PCC. However, PCC supports forwarding syslog to an external log management service (for example, Papertrail, Splunk, or your custom enterprise log sink). The broker logs are useful for debugging problems creating, updating, and binding service instances.

To enable remote syslog for the service broker, do the following:

  1. Click Syslog. Configuring Log Management Service
  2. Configure the fields on the Syslog pane as follows:

    FieldInstructions
    Enable Remote Syslog Select to enable.
    External Syslog Address Enter the address or host of the syslog server for sending logs, for example, logs.example.com.
    External Syslog Port Enter the port of the syslog server for sending logs, for example,29279.
    Enable TLS for Syslog Select to enable secure log transmission through TLS. Without this, remote syslog sends unencrypted logs. We recommend enabling TLS, as most syslog endpoints such as Papertrail and Logsearch require TLS.
    Permitted Peer for TLS Communication. This is required if TLS is enabled. If there are several peer servers that can respond to remote syslog connections, then provide a regex, such as *.example.com.
    CA Certificate for TLS Communication If the server certificate is not signed by a known authority, for example, an internal syslog server, provide the CA certificate of the log management service endpoint.
    Send service instance logs to external By default, only the broker logs are forwarded to your configured log management service. If you want to forward server and locator logs from all service instances, select this. This lets you monitor the health of the clusters, although it generates a large volume of logs.

    If you don’t enable this, you get only the broker logs which include information about service instance creation, but not about on-going cluster health.

  3. Click Save.

Service Instance Upgrades

A configurable number of service instances may be upgraded concurrently by entering a new value that is greater than one and less than the BOSH worker count for the Number of simultaneous upgrades.

Specify a set of service instances to act as canaries for the upgrade process by changing the Number of upgrade canary instances to a value greater than 0. If all canary instances successfully upgrade, the remaining instances are upgraded. If any canary instance fails to upgrade, the upgrade fails and no further instances are upgraded.

Click Save after changing values.

Instance Upgrades section

Security

The environment may be configured to more securely store service keys within CredHub, instead of within the cloud controller’s data store. To enable this functionality:

  1. Click Security.

  2. Click on the box labeled Enable Secure Service Instance Credentials to enable use of CredHub.

  3. An ‘X’ is required in the text box to promote the understanding that a TLS-enabled service instance cannot be created if the PCF environment is not set up to handle TLS. See Preparing for TLS for how to prepare the PCF environment.

  4. Click Save.

Security section

Errands

By default, post-deploy and pre-delete errands always run. Pivotal recommends keeping these defaults. However, if necessary, you can change these defaults as follows.

For general information about errands in PCF, see Managing Errands in Ops Manager

  1. Click Errands.

  2. Change the setting for the errands.

  3. Click Save.

Setting Service Instance Quotas

On-demand provisioning is intended to accelerate app development by eliminating the need for development teams to request and wait for operators to create a service instance. However, to control costs, operations teams and administrators must ensure responsible use of resources.

There are several ways to control the provisioning of on-demand service instances by setting various quotas at these levels:

After you set quotas, you can:

Create Global-level Quotas

Each Pivotal Cloud Foundry (PCF) service has a separate service broker. A global quota at the service level sets the maximum number of service instances that can be created by a given service broker. If a service has more than one plan, then the number of service instances for all plans combined cannot exceed the global quota for the service.

The operator sets a global quota for each PCF service independently. For example, if you have Redis for PCF and RabbitMQ for PCF, you must set a separate global service quota for each of them.

When the global quota is reached for a service, no more instances of that service can be created unless the quota is increased, or some instances of that service are deleted.

The global quota is set in the service tile in Ops Manager, shown for an example service below.

Note: This is an example image only. The following screen may look slightly different for your service or release version.

service_level_quotas

Create Plan-level Quotas

A service may offer one or more plans. You can set a separate quota per plan so that instances of that plan cannot exceed the plan quota. For a service with multiple plans, the total number of instances created for all plans combined cannot exceed the global quota for the service.

When the plan quota is reached, no more instances of that plan can be created unless the plan quota is increased or some instances of that plan are deleted.

The plan quota is set in the service tile in Ops Manager, shown for an example service plan below.

Note: This is an example image only. The following screen may look slightly different for your service or release version.

plan_level_quotas

Create and Set Org-level Quotas

An org-level quota applies to all PCF services and sets the maximum number of service instances an organization can create within PCF. For example, if you set your org-level quota to 100, developers can create up to 100 service instances in that org using any combination of PCF services.

When this quota is met, no more service instances of any kind can be created in the org unless the quota is increased or some service instances are deleted.

To create and set an org-level quota, do the following:

  1. Run this command to create a quota for service instances at the org level:

    cf create-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans
    

    where these variables are:


    QUOTA-NAME—A name for this quota
    TOTAL-MEMORY—Maximum memory used by all service instances combined
    INSTANCE-MEMORY—Maximum memory used by any single service instance
    ROUTES—Maximum number of routes allowed for all service instances combined
    SERVICE-INSTANCES—Maximum number of service instances allowed for the org


    For example:
    cf create-quota myquota -m 1024mb -i 16gb -r 30 -s 50 --allow-paid-service-plans

  2. Associate the quota you created above with a specific org by running the following command:

    cf set-quota ORG-NAME QUOTA-NAME
    

    For example: cf set-quota dev_org myquota

For more information on managing org-level quotas, see Creating and Modifying Quota Plans.

Create and Set Space-level Quotas

A space-level service quota applies to all PCF services and sets the maximum number of service instances that can be created within a given space in PCF. For example, if you set your space-level quota to 100, developers can create up to 100 service instances in that space using any combination of PCF services.

When this quota is met, no more service instances of any kind can be created in the space unless the quota is updated or some service instances are deleted.

To create and set a space-level quota, do the following:

  1. Run the following command to create the quota:

    cf create-space-quota QUOTA -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans
    

    where these variables are:


    QUOTA-NAME—A name for this quota
    TOTAL-MEMORY—Maximum memory used by all service instances combined
    INSTANCE-MEMORY—Maximum memory used by any single service instance
    ROUTES—Maximum number of routes allowed for all service instances combined
    SERVICE-INSTANCES—Maximum number of service instances allowed for the org


    For example: cf create-space-quota myspacequota -m 1024mb -i 16gb -r 30 -s 50 --allow-paid-service-plans

  2. Associate the quota you created above with a specific space by running the following command:

    cf set-space-quota SPACE-NAME QUOTA-NAME
    

    For example:
    cf set-space-quota myspace myspacequota

For more information on managing space-level quotas, see Creating and Modifying Quota Plans.

View Current Org and Space-level Quotas

To view org quotas, run the following command.

cf org ORG-NAME

To view space quotas, run the following command:

cf space SPACE-NAME

For more information on managing org and space-level quotas, see the Creating and Modifying Quota Plans.

Monitor Quota Use and Service Instance Count

Service-level and plan-level quota use, and total number of service instances, are available through the on-demand broker metrics emitted to Loggregator. These metrics are listed below:

Metric Name Description
on-demand-broker/SERVICE-NAME/quota_remaining Quota remaining for all instances across all plans
on-demand-broker/SERVICE-NAME/PLAN-NAME/quota_remaining Quota remaining for a specific plan
on-demand-broker/SERVICE-NAME/total_instances Total instances created across all plans
on-demand-broker/SERVICE-NAME/PLAN-NAME/total_instances Total instances created for a specific plan

Note: Quota metrics are not emitted if no quota has been set.

Calculate Resource Costs for On-Demand Plans

On-demand plans use dedicated VMs, disks, and various other resources from an IaaS, such as AWS. To calculate maximum resource cost for plans individually or combined, you multiply the quota by the cost of VM and Persistent Disk types selected in the plan configuration(s). The specific costs depend on your IaaS.

The image below shows an example of the VM type and persistent disk selected, as well as the quota for this plan.

plan resources

Important: Although operators can limit on-demand instances with plan quotas and a global quota, as described in the above topics, IaaS resource usage still varies based on the number of on-demand instances provisioned.

Calculate Maximum Resource Cost Per On-Demand Plan

To calculate the maximum cost of VMs and persistent disk for each plan, do the following calculation:

plan quota x cost of selected resources

For example, if you selected the options in the above image, you have selected a VM type micro.cpu and a persistent disk type 20 GB, and the plan quota is 15. The VM and persistent disk types have an associated cost for the IaaS you are using. Therefore, to calculate the maximum cost of resources for this plan, multiply the cost of the resources selected by the plan quota:

(15 x cost of micro.cpu VM type) + (15 x cost of 20 GB persistent disk)

Calculate Maximum Resource Cost for All On-Demand Plans

To calculate the maximum cost for all plans combined, add together the maximum costs for each plan. This assumes that the sum of your individual plan quotas is less than the global quota.

Here is an example:

(plan1 quota x plan1 resource cost) + ( plan2 quota x plan2 resource cost) = max cost for all plans

Calculate Actual Resource Cost of all On-Demand Plans

To calculate the current actual resource cost across all your on-demand plans:

  1. Find the number of instances currently provisioned for each active plan by looking at the total_instance metric for that plan.

  2. Multiply the total_instance count for each plan by that plan’s resource costs. Record the costs for each plan.

  3. Add up the costs noted in Step 2 to get your total current resource costs.

For example:

(plan1 total_instances x plan1 resource cost) + (plan2 total_instances x plan2 resource cost) = current cost for all plans

Monitoring Pivotal Cloud Cache Service Instances

PCC clusters and brokers emit service metrics. You can use any tool that has a corresponding Cloud Foundry nozzle to read and monitor these metrics in real time.

As an app developer, when you opt to use a data service, you should be prepared to:

  • monitor the state of that service
  • triage issues that occur with that service
  • be notified of any concerns

If you believe an issue relates to the underlying infrastructure (network, CPU, memory, or disk), you will need to capture evidence and notify your platform team. The metrics described in this section can help in characterizing the performance and resource consumption of your service instance.

Service Instance Metrics

In the descriptions of the metrics, KPI stands for Key Performance Indicator.

Member Count


serviceinstance.MemberCount

Description Returns the number of members in the distributed system.
Metric Type number
Suggested measurement Every second
Measurement Type count
Warning Threshold less than the manifest member count
Suggested Actions This depends on the expected member count, which is available in the BOSH manifest. If the number expected is different from the number emitted, this is a critical situation that may lead to data loss, and the reasons for node failure should be investigated by examining the service logs.
Why a KPI? Member loss due to any reason can potentially cause data loss.

Total Available Heap Size


serviceinstance.TotalHeapSize

Description Returns the total available heap, in megabytes, across all instance members.
Metric Type number
Suggested measurement Every second
Measurement Type pulse
Why a KPI? If the total heap size and used heap size are too close, the system might see thrashing due to GC activity. This increases latency.

Total Used Heap Size


serviceinstance.UsedHeapSize

Description Returns the total heap used across all instance members, in megabytes.
Metric Type number
Suggested measurement Every second
Measurement Type pulse
Why a KPI? If the total heap size and used heap size are too close, the system might see thrashing due to GC activity. This increases latency.

Total Available Heap Size as a Percentage


serviceinstance.UnusedHeapSizePercentage

Description Returns the proportion of total available heap across all instance members, expressed as a percentage.
Metric Type percent
Suggested measurement Every second
Measurement Type compound metric
Warning Threshold 40%
Critical Threshold 10%
Suggested Actions If this is a spike due to eviction catching up with insert frequency, then customers need to keep a close watch that it should not hit the RED marker. If there is no eviction, then horizontal scaling is suggested.
Why a KPI? If the total heap size and used heap size are too close, the system might see thrashing due to GC activity. This increases latency.

Per Member Metrics

Memory Used as a Percentage


member.UsedMemoryPercentage

Description RAM being consumed.
Metric Type percent
Suggested measurement Average over last 10 minutes
Measurement Type average
Warning Threshold 75%
Critical Threshold 85%

Count of Java Garbage Collections


member.GarbageCollectionCount

Description The number of times that garbage has been collected.
Metric Type number
Suggested measurement Sum over last 10 minutes
Measurement Type count
Warning Threshold Dependent on the IaaS and app use case.
Critical Threshold Dependent on the IaaS and app use case.
Suggested Actions Check the number of queries run against the system, which increases the deserialization of objects and increases garbage.
Why a KPI? If the frequency of garbage collection is high, the system might see high CPU usage, which causes delays in the cluster.

CPU Utilization Percentage


member.HostCpuUsage

Description This member’s process CPU utilization, expressed as a percentage.
Metric Type percent
Suggested measurement Average over last 10 minutes
Measurement Type average
Warning Threshold 85%
Critical Threshold 95%
Suggested Actions If this is not happening with high GC activity, the system is reaching its limits. Horizontal scaling might help.
Why a KPI? High CPU usage causes delayed responses and can also make the member non-responsive. This can cause the member to be kicked out of the cluster, potentially leading to data loss.

Average Latency of Get Operations


member.GetsAvgLatency

Description The average latency of cache get operations, in nanoseconds.
Metric Type number
Suggested measurement Average over last 10 minutes
Measurement Type average
Warning Threshold Dependent on the IaaS and app use case.
Critical Threshold Dependent on the IaaS and app use case.
Suggested Actions If this is not happening with high GC activity, the system is reaching its limit. Horizontal scaling might help.
Why a KPI? It is a good indicator of the overall responsiveness of the system. If this number is high, the service administrator should diagnose the root cause.

Average Latency of Put Operations


member.PutsAvgLatency

Description The average latency of cache put operations, in nanoseconds.
Metric Type number
Suggested measurement Average over last 10 minutes
Measurement Type average
Warning Threshold Dependent on the IaaS and app use case.
Critical Threshold Dependent on the IaaS and app use case.
Suggested Actions If this is not happening with high GC activity, the system is reaching its limit. Horizontal scaling might help.
Why a KPI? It is a good indicator of the overall responsiveness of the system. If this number is high, the service administrator should diagnose the root cause.

JVM pauses


member.JVMPauses

Description The quantity of JVM pauses.
Metric Type number
Suggested measurement Sum over 2 seconds
Measurement Type count
Warning Threshold Dependent on the IaaS and app use case.
Critical Threshold Dependent on the IaaS and app use case.
Suggested Actions Check the cached object size; if it is greater than 1 MB, you may be hitting the limitation on JVM to garbage collect this object. Otherwise, you may be hitting the utilization limit on the cluster, and will need to scale up to add more memory to the cluster.
Why a KPI? Due to a JVM pause, the member stops responding to “are-you-alive” messages, which may cause this member to be kicked out of the cluster.

File Descriptor Limit


member.FileDescriptorLimit

Description The maximum number of open file descriptors allowed for the member’s host operating system.
Metric Type number
Suggested measurement Every second
Measurement Type pulse
Why a KPI? If the number of open file descriptors exceeds number available, it causes the member to stop responding and crash.

Open File Descriptors


member.TotalFileDescriptorOpen

Description The current number of open file descriptors.
Metric Type number
Suggested measurement Every second
Measurement Type pulse
Why a KPI? If the number of open file descriptors exceeds number available, it causes the member to stop responding and crash.

Quantity of Remaining File Descriptors


member.FileDescriptorRemaining

Description The number of available file descriptors.
Metric Type number
Suggested measurement Every second
Measurement Type compound metric
Warning Threshold 1000
Critical Threshold 100
Suggested Actions Scale horizontally to increase capacity.
Why a KPI? If the number of open file descriptors exceeds number available, it causes the member to stop responding and crash.

Gateway Sender and Gateway Receiver Metrics

These are metrics emitted through the CF Nozzle for gateway senders and gateway receivers.

Queue Size for the Gateway Sender

gatewaySender.<sender-id>.EventQueueSize
Description The current size of the gateway sender queue.
Metric Type number
Measurement Type count

Events Received at the Gateway Sender

gatewaySender.<sender-id>.EventsReceivedRate
Description A count of the events coming from the region to which the gateway sender is attached. It is the count since the last time the metric was checked. The first time it is checked, the count is of the number of events since the gateway sender was created.
Metric Type number
Measurement Type count

Events Queued by the Gateway Sender

gatewaySender.<sender-id>.EventsQueuedRate
Description A count of the events queued on the gateway sender from the region. This quantity of events might be lower than the quantity of events received, as not all received events are queued. It is a count since the last time the metric was checked. The first time it is checked, the count is of the number of events since the gateway sender was created.
Metric Type number
Measurement Type count

Events Received by the Gateway Receiver

gatewayReceiver.EventsReceivedRate
Description A count of the events received from the gateway sender which will be applied to the region on the gateway receiver’s site. It is the count since the last time the metric was checked. The first time it is checked, the count is of the number of events since the gateway receiver was created.
Metric Type number
Measurement Type count

Disk Metrics

These are metrics emitted through the CF Nozzle for disks.

Average Latency of Disk Writes

diskstore.DiskWritesAvgLatency
Description The average latency of disk writes in nanoseconds.
Metric Type number
Measurement Type time in nanoseconds

Quantity of Bytes on Disk

diskstore.TotalSpace
Description The total number of bytes on the attached disk.
Metric Type number
Measurement Type count

Quantity of Available Bytes on Disk

diskstore.UseableSpace
Description The total number of bytes of available space on the attached disk.
Metric Type number
Measurement Type count

Total Memory Consumption

The BOSH mem-check errand calculates and outputs the quantity of memory used across all PCC service instances. This errand helps PCF operators monitor resource costs, which are based on memory usage.

From the director, run a BOSH command of the form:

bosh -d <service broker name> run-errand mem-check

With this command:

bosh -d cloudcache-service-broker run-errand mem-check

Here is an anonymized portion of example output from the mem-check errand for a two cluster deployment:

           Analyzing deployment xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx1...
           JVM heap usage for service instance xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx1
           Used Total = 1204 MB
           Max Total = 3201 MB

           Analyzing deployment xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx2...
           JVM heap usage for service instance xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx2
           Used Total = 986 MB
           Max Total = 3201 MB

           JVM heap usage for all clusters everywhere:
           Used Global Total = 2390 MB
           Max Global Total = 6402 MB

Monitoring PCC Service Instances with Prometheus

Prometheus is one of various tools you can use to monitor services instances. It is a monitoring and alerting toolkit that allows for metric scraping. You can use the Firehose exporter to export all the metrics from the Firehose, which you can then graph with Grafana to monitor your PCC cluster.

Follow the instructions here to deploy Prometheus alongside your PCF cluster.

Prometheus can be deployed on any IaaS. You need to verify that the Firehose exporter job can talk to your UAA VM. This might involve opening up firewall rules or enabling your VM to allow outgoing traffic.

Grafana Example

You can run queries on, and build a custom dashboard of, specific metrics that are important to you.

Upgrading Pivotal Cloud Cache

Upgrade minor release versions from your currently deployed version to the target version in sequential order. For example, PCC v1.2 must be upgraded to PCC v1.3 prior to upgrading to PCC v1.4. Note that each PCC release is compatible with two Pivotal Application Service (PAS) and Ops Manager versions, as specified in the Product Snapshot. Incorporate those upgrades to PAS and Ops Manager in your upgrade process as required to maintain compatibility, as described in Upgrading Pivotal Cloud Foundry.

Follow the steps below to upgrade PCC:

  1. Download the new version of the tile from the Pivotal Network.
  2. Upload the product to Ops Manager.
  3. Click Add next to the uploaded product.
  4. Click on the Cloud Cache tile and configure the upgrade options.

    • To try the upgrade on a small number of service instances first, set the quantity of canary service instances as described in Service Instance Upgrades.
    • Set the number of instances that are to be upgraded in parallel as described in Service Instance Upgrades.
    • Make sure that under the Errands section, the Upgrade All Service Instances Post-Deploy Errand is Default (On). Save the change.
  5. Optionally, if you are using Ops Manager v2.3 or later, click Review Pending Changes (see Reviewing Pending Product Changes).

  6. Click Apply Changes.

Migrating to a TLS-Enabled Cluster

An existing PCC service instance that does not use TLS encryption may be migrated to become a PCC service instance with TLS encryption enabled.

WARNING! This procedure will require downtime for the service instance during the migration.

Follow the procedure given here after these prerequisites have been met:

  • All steps within Preparing for TLS have been completed.
  • The service instance has been upgraded to PCC v1.5.2 or a more recent PCC version. There will be no PCC version change during the migration.

Follow this procedure to migrate the existing PCC service instance:

  1. As a PCF operator, stop all apps. First, list all apps to identify the APP_NAME.
    $ cf apps
    
    Then, stop each app with:
    $ cf stop APP_NAME
    
  2. For all non-persistent regions, use the gfsh command line tool to export the data.

    WARNING! Without an export, all non-persistent region entries will be irretrievably lost.

    • Complete the steps within Accessing a Service Instance to acquire the correct version of gfsh, run it, and connect to the cluster using the cluster operator role/credentials from the service key.
    • List the regions.
      gfsh>list regions 
    • For each region, use gfsh describe to determine if the region is persistent or not and to acquire a server name.
      gfsh>describe region --name=REGION_NAME 
    • For each non-persistent region, use this single gfsh command to export all the data within the region. The SERVER_NAME identifies which GemFire server receives the export command and propagates the command to all other GemFire servers within the cluster.
      gfsh>export data --parallel --region=REGION_NAME --member=SERVER_NAME --dir=/var/vcap/store/gemfire-server 
  3. Your PCF operator needs to target the BOSH Director in order to acquire the DEPLOYMENT_NAME.
    • Run
      $ cf service SERVICE_INSTANCE_NAME 
      to acquire the digits that uniquely identify the service instance. The digits (XXX-XXX in the following instructions) are those between cloudcache- and the period ..
    • Log in to the BOSH Director.
      $ bosh log-in 
    • The DEPLOYMENT_NAME will appear in the output of
      $ bosh deployments | grep XXX-XXX 
  4. Using PCF operator credentials, stop the BOSH deployment:
    $ bosh -d DEPLOYMENT_NAME stop 
    and type “y” when prompted.
  5. Acquire the BOSH manifest with:
    $ bosh -d DEPLOYMENT_NAME manifest > DEPLOYMENT_NAME-manifest.yml 
  6. Edit the acquired BOSH manifest. There are three locations within the manifest file that will require additions. These three locations are identified within this anonymized portion of the manifest file with the symbols , , and . The first part of the manifest file is omitted, as its listed values change based on the PCC version. Real passwords have been replaced with the placeholder password, and user names have been replaced with the placeholder userX within this example.
    instance_groups:
    - name: locator
      instances: 3
      jobs:
      - name: gemfire-locator
        release: gemfire
        properties:
          gemfire:  
            distributed-system-id: 0
            locator:
              bpm_enabled: true
              port: '55221'
              properties:
                enable-time-statistics: true
            persist-pdx: true
            security:
              internal_cluster_password: password
              internal_cluster_username: userX
              roles:
                cluster_operator:
                - CLUSTER:WRITE
                - CLUSTER:READ
                - DATA:MANAGE
                - DATA:WRITE
                - DATA:READ
                - CLUSTER:MANAGE:DEPLOY
                - CLUSTER:MANAGE
                - CLUSTER:MANAGE:GATEWAY
                developer:
                - CLUSTER:READ
                - DATA:WRITE
                - DATA:READ
                gateway:
                - DATA:WRITE
              users:
                cluster_operator_userX:
                  password: password
                  roles:
                  - cluster_operator
                developer_userX:
                  password: password
                  roles:
                  - developer
      - name: route_registrar
        release: routing
        consumes:
          nats:
            deployment: cf-NNNNNNNNNNN
            from: nats
        properties:
          route_registrar:
            routes:
            - name: cloudcache
              port: 8080  
              registration_interval: 20s
              uris:
              - cloudcache-XXX-XXX.example.com
      - name: bpm
        release: bpm
      vm_type: micro.cpu
      stemcell: stemcell
      persistent_disk_type: '10240'
      azs:
      - us-central1-f
      networks:
      - name: example-services-subnet
    - name: server
      instances: 4
      jobs:
      - name: gemfire-server
        release: gemfire
        properties:
          gemfire:
            server:
              bpm_enabled: true
              create-gateway-receiver: true
              development-mode: false
              properties:
                enable-time-statistics: true
                jmx-manager-start: true
              security:
                gateway_password: password
                gateway_username: gateway_sender_userX
      - name: prime-cluster-for-pcc
        release: gemfire
      - name: bpm
        release: bpm
      vm_type: medium.cpu
      stemcell: stemcell
      persistent_disk_type: '10240'
      azs:
      - us-central1-f
      networks:
      - name: example-services-subnet
    update:
      canaries: 1
      canary_watch_time: 1000-600000
      update_watch_time: 1000-600000
      max_in_flight: 32
      serial: true
    features:
      converge_variables: true  
    
    Add lines to the BOSH manifest, using the lines as shown in red in the following modified version of the manifest. Substitute your digits that uniquely identify your service instance for XXX-XXX within the added lines.
    instance_groups:
    - name: locator
      instances: 3
      jobs:
      - name: gemfire-locator
        release: gemfire
        properties:
          gemfire:  
            tls: true
            truststore_password: ((trust-store-password))
            keystore_password: ((key-store-password))
            certificate: ((gemfire-certificate))
            trusted_certs:
            - ((/cf/diego-instance-identity-root-ca))
            - ((/services/tls_ca))
            distributed-system-id: 0
            locator:
              bpm_enabled: true
              port: '55221'
              properties:
                enable-time-statistics: true
            persist-pdx: true
            security:
              internal_cluster_password: password
              internal_cluster_username: userX
              roles:
                cluster_operator:
                - CLUSTER:WRITE
                - CLUSTER:READ
                - DATA:MANAGE
                - DATA:WRITE
                - DATA:READ
                - CLUSTER:MANAGE:DEPLOY
                - CLUSTER:MANAGE
                - CLUSTER:MANAGE:GATEWAY
                developer:
                - CLUSTER:READ
                - DATA:WRITE
                - DATA:READ
                gateway:
                - DATA:WRITE
              users:
                cluster_operator_userX:
                  password: password
                  roles:
                  - cluster_operator
                developer_userX:
                  password: password
                  roles:
                  - developer
      - name: route_registrar
        release: routing
        consumes:
          nats:
            deployment: cf-NNNNNNNNNNN
            from: nats
        properties:
          route_registrar:
            routes:
            - name: cloudcache
              port: 8080  
              tls_port: 8080
              server_cert_domain_san: cloudcache-XXX-XXX.example.com
              registration_interval: 20s
              uris:
              - cloudcache-XXX-XXX.example.com
      - name: bpm
        release: bpm
      vm_type: micro.cpu
      stemcell: stemcell
      persistent_disk_type: '10240'
      azs:
      - us-central1-f
      networks:
      - name: example-services-subnet
    - name: server
      instances: 4
      jobs:
      - name: gemfire-server
        release: gemfire
        properties:
          gemfire:
            server:
              bpm_enabled: true
              create-gateway-receiver: true
              development-mode: false
              properties:
                enable-time-statistics: true
                jmx-manager-start: true
              security:
                gateway_password: password
                gateway_username: gateway_sender_userX
      - name: prime-cluster-for-pcc
        release: gemfire
      - name: bpm
        release: bpm
      vm_type: medium.cpu
      stemcell: stemcell
      persistent_disk_type: '10240'
      azs:
      - us-central1-f
      networks:
      - name: example-services-subnet
    update:
      canaries: 1
      canary_watch_time: 1000-600000
      update_watch_time: 1000-600000
      max_in_flight: 32
      serial: true
    features:
      converge_variables: true  
    variables:
    - name: trust-store-password
      type: password
    - name: key-store-password
      type: password
    - name: gemfire-certificate
      type: certificate
      options:
        ca: /services/tls_ca
        common_name: gemfire-ssl
        alternative_names:
        - gemfire-ssl
        - cloudcache-XXX-XXX.example.com
    
  7. Redeploy the BOSH manifest. Do a BOSH deploy using the edited BOSH manifest:
    $ bosh -d SERVICE-INSTANCE-NAME deploy SERVICE-INSTANCE-NAME-manifest.yml 
    and type “y” when prompted.
  8. Restart the cluster with a sequential BOSH start:
    $ bosh start -d SERVICE-INSTANCE-NAME  --max-in-flight=1 
    and type “y” when prompted.
  9. Run gfsh and follow the directions in Connect with gfsh over HTTPS to connect to the TLS-enabled cluster.
  10. Use gfsh to import all region data that was exported earlier in this procedure. For each earlier-exported region, do:
    gfsh>import data --parallel --region=REGION_NAME --member=SERVER_NAME --dir=/var/vcap/store/gemfire-server 
  11. Revise the app such that it works with a TLS-enabled PCC service instance by following the instructions within Developing an App Under TLS. Re-build, re-deploy, and start the app.

Updating Pivotal Cloud Cache Plans

Follow the steps below to update plans in Ops Manager.

  1. Click on the Cloud Cache tile.
  2. Click on the plan you want to update under the Information section.
  3. Edit the fields with the changes you want to make to the plan.
  4. Click Save button on the bottom of the page.
  5. Click on the PCF Ops Manager to navigate to the Installation Dashboard.
  6. Optionally, if you are using Ops Manager v2.3 or later, click Review Pending Changes (see Reviewing Pending Product Changes).
  7. Click Apply Changes.

Plan changes are not applied to existing services instances until you run the upgrade-all-service-instances BOSH errand. You must use the BOSH CLI to run this errand. Until you run this errand, developers cannot update service instances.

Changes to fields that can be overridden by optional parameters, for example num_servers or new_size_percentage, change the default value of these instance properties, but do not affect existing service instances.

If you change the allowed limits of an optional parameter, for example the maximum number of servers per cluster, existing service instances in violation of the new limits are not modified.

When existing instances are upgraded, all plan changes are applied to them. Upgrades and updates to service instances can cause a rolling restart of GemFire servers. Be aware that the rebalancing of data to maintain redundancy may impact the performance of the remainder of the servers within the service instance.

WARNING: Data loss may result from the restart of a cluster. See Restarting a Cluster for the conditions under which data loss occurs.

Uninstalling Pivotal Cloud Cache

To uninstall PCC, follow the steps from below from the Installation Dashboard:

  1. Click the trash can icon in the bottom-right-hand corner of the tile.
  2. Optionally, if you are using Ops Manager v2.3 or later, click Review Pending Changes (see Reviewing Pending Product Changes).
  3. Click Apply Changes.

Troubleshooting

View Statistics Files

You can visualize the performance of your cluster by downloading the statistics files from your servers. These files are located in the persistent store on each VM. To copy these files to your workstation, run the following command:

`bosh2 -e BOSH-ENVIRONMENT -d DEPLOYMENT-NAME scp server/0:/var/vcap/store/gemfire-server/statistics.gfs /tmp`

See the Pivotal GemFire Installing and Running VSD topic for information about loading the statistics files into Pivotal GemFire VSD.

Smoke Test Failures

Error: “Creating p-cloudcache SERVICE-NAME failed”

The smoke tests could not create an instance of GemFire. To troubleshoot why the deployment failed, use the cf CLI to create a new service instance using the same plan and download the logs of the service deployment from BOSH.

Error: “Deleting SERVICE-NAME failed”

The smoke test attempted to clean up a service instance it created and failed to delete the service using the cf delete-service command. To troubleshoot this issue, run BOSH logs to view the logs on the broker or the service instance to see why the deletion may have failed.

Error: Cannot connect to the cluster SERVICE-NAME

The smoke test was unable to connect to the cluster.

To troubleshoot the issue, review the logs of your load balancer, and review the logs of your CF Router to ensure the route to your PCC cluster is properly registered.

You also can create a service instance and try to connect to it using the gfsh CLI. This requires creating a service key.

Error: “Could not perform create/put on Cloud Cache cluster”

The smoke test was unable to write data to the cluster. The user may not have permissions to create a region or write data.

Error: “Could not retrieve value from Cloud Cache cluster”

The smoke test was unable to read back the data it wrote. Data loss can happen if a cluster member improperly stops and starts again or if the member machine crashes and is resurrected by BOSH. Run BOSH logs to view the logs on the broker to see if there were any interruptions to the cluster by a service update.

General Connectivity

Client-to-Server Communication

PCC Clients communicate to PCC servers on port 40404 and with locators on port 55221. Both of these ports must be reachable from the PAS (or Elastic Runtime) network to service the network.

Membership Port Range

PCC servers and locators communicate with each other using UDP and TCP. The current port range for this communication is 49152-65535.

If you have a firewall between VMs, ensure this port range is open.

Port Range Usage Across a WAN

Gateway receivers and gateway senders communicate across WAN-separated service instances. Each PCC service instance uses GemFire defaults for the gateway receiver ports. The default is the inclusive range of port numbers 5000 to 5499.

Ensure this port range is open when WAN-separated service instances will communicate.