LATEST VERSION: 1.1 - CHANGELOG
Pivotal Cloud Cache for PCF v1.1

Operator Guide

This document describes how a Pivotal Cloud Foundry (PCF) operator can install, configure, and maintain Pivotal Cloud Cache (PCC).

Requirements for Pivotal Cloud Cache

Service Network

You must have access to a Service Network in order to install PCC.

Minimum Version Requirements

PCC requires PCF v1.10 with PCF Elastic Runtime v1.10.3 or later or PCF v1.11 with PCF Elastic Runtime v1.11.0 or later.

Installing and Configuring Pivotal Cloud Cache

Follow the steps below to install PCC on PCF:

  1. Download the tile from the Pivotal Network.
  2. Click Import a Product to import the tile into Ops Manager.
  3. Click the + symbol next to the uploaded product description.
  4. Click on the PCC tile.
  5. Complete all the configuration steps in the Configure Tile Properties section below.
  6. Return to the Ops Manager Installation Dashboard and click Apply Changes to complete the installation of the PCC tile.

Configure Tile Properties

Configure the sections listed on the left side of the page.

Tile Configuration Sections

After you complete a section, a green circle containing a check mark appears next to the section name. Each section name must show this checked green circle before you can complete your installation.

Assign Availability Zones and Networks

To select AZs and networks for VMs used by PCC, do the following:

  1. Click Assign AZs and Networks.

  2. Configure the fields on the Assign AZs and Networks pane as follows:

    FieldInstructions
    Place singleton jobs in Select the region that you want for singleton VMs.
    Balance other jobs in Select the AZ(s) you want to use for distributing other GemFire VMs. Pivotal recommends selecting all of them.
    Network Select your Elastic Runtime network.
    Service Network Select the network to be used for GemFire VMs.

  3. Click Save.

Settings: Specify Smoke Test Errands and Outbound Internet Acces

  1. Click Settings. This section provides settings for smoke-test errands and VM options.

  2. Under Plan to use for smoke test, select a plan to use when the smoke-tests errand runs. The smoke-tests errand runs after tile installation and verifies that your installation was successful. By default, the smoke-test errand runs on the system org and the p-cloudcache-smoke-test space.

  3. Ensure that the selected plan is enabled and configured. For information about configuring plans, see Configure Service Plans below. If the selected plan is not enabled, the smoke-tests errand fails.

    Note: Pivotal recommends that you use the smallest four-server plan for smoke tests. Because smoke tests create and later destroy this plan, using a very small plan reduces installation time.

    Configuring Smoke Tests Plan

  4. Under VM options, allow or disable internet access.

    If BOSH is configured to use an external blob store, you need allow outbound internet access from service instances. Log forwarding and backups, which require external endpoints, might also require internet access.

    By default, outbound internet access is NOT allowed from service instances.

  5. To allow outbound internet access from service instance, click Allow outbound internet access from service instances (IaaS-dependent).

    Outbound Internet Access

  6. After you have configured smoke test errands and internet access options, click Save.

Set Up Syslog Forwarding

By default, syslog forwarding is not enabled in PCC. However, PCC supports forwarding syslog to an external log management service (for example, Papertrail, Splunk, or your custom enterprise log sink). The broker logs are useful for debugging problems creating, updating, and binding service instances.

To enable remote syslog for the service broker, do the following:

  1. Click Syslog. Configuring Log Management Service
  2. Configure the fields on the Syslog pane as follows:

    FieldInstructions
    Enable Remote Syslog Click to select this button.
    External Syslog Address/td> Enter the address or host of the syslog server for sending logs, for example, logs.example.com.
    External Syslog Port Enter the port of the syslog server for sending logs, for example,29279.
    Enable TLS for Syslog Click to enable secure log transmission through TLS. Otherwise, remote syslog sends unencrypted logs.
    Permitted Peer for TLS Communication. This is required if TLS is enabled. If there are several peer servers that can respond to remote syslog connections, then provide a regex, such as *.example.com.
    CA Certificate for TLS Communication If the server certificate is not signed by a known authority, for example, an internal syslog server, provide the CA certificate of the log management service endpoint.
    Send service instance logs to external By default, only the broker logs are forwarded to your configured log management service. If you want to forward server and locator logs from all service instances, select this. This lets you monitor the health of the clusters, although it generates a large volume of logs.

    If you don’t enable this, you get only the broker logs which include information about service instance creation, but not about on-going cluster health.

  3. Click Save.

Configure Service Plans

You can configure five individual plans for your developers. Select the Plan 1 through Plan 5 tabs to configure each of them.

Configuring a Plan

Configure settings for your service plan as described in the table below.

SettingInstructions
Service Plan Access Enable Plan is checked by default. If you do not want to add this plan to the CF service catalog, select Disable Plan. You must enable at least one plan.
Plan Name Allows you to customize the name of the plan. This plan name is displayed to developers when they view the service in the Marketplace..
Plan Description Allows you to supply a plan description. The description is displayed to developers when they view the service in the Marketplace.
CF Service Access Enabled by default. This setting determines whether the plan is displayed on the marketplace. Enable Service Access will display the service plan to all developers in the CF marketplace. Disable Service Access will not display the service plan to developers in the CF marketplace and cannot be enabled at a later time. Leave Service Access Unchanged will not display the service plan in the CF marketplace by default but can be enabled at a later time.
Maximum service instances Sets the maximum allowed PCC service instances.
Maximum servers per cluster Allows operators to set an upper limit on the number of servers developers can request.
Default Number of Servers The number of servers a new cluster will have by default if developers do not explicitly specify a number of servers in a service instance.
Availability zones for service instances Determines which AZs are used for a particular cluster. The members of a cluster are distributed evenly across AZs. NOTE: After you’ve selected AZs for your service network, you cannot add additional AZs; doing so causes existing service instances to lose data on update.
VM type for the Locator VMs Specifies the type of VM to be used for locator VMs. Recommended: select a VM that has at least 1 GB of RAM and 4 GB of disk space.
Persistent disk type for the Locator VMs Specifies the size of the persistent disk to be used for locator VMs. Recommended: select 10 GB or higher.
VM type for the Server VMs Specifies the type of VM to be used for server VMs. Recommended: select a VM that has at least 4 GB of RAM and 8 GB of disk space.
Persistent disk type for the Server VMs Specifies the size of the persistent disk to be used for server VMs. Recommended: select 10 GB or higher.

Note: The total size of the cache is directly related to the number of servers and the amount of memory of the selected server VM type.

When you finish configuring the plan, click Save to save your configuration options.

Errands

By default, Post-Deploy and Pre-Delete errands always run. Pivotal recommends keeping these defaults. However, if necessary, you can change these defaults, as follows.

For general information about errands in PCF, see Managing Errands in Ops Manager 1. Click Errands.

  1. Change the setting for the errands.

    Note: If you are using Ops Manager v1.10.0 to v1.10.7, do not set errands to When Changed. This is because errands set to When Changed sometimes fail to run when the PCC tile changes.

  2. Click Save.

Stemcell

Ensure you import the correct type of stemcell indicated on this tab.

You can download the latest available stemcells from Pivnet.

BOSH System Health Metrics

The BOSH layer that underlies PCF generates healthmonitor metrics for all VMs in the deployment. However, these metrics are not included in the Loggregator Firehose by default. To get these metrics, do either of the following:

  • To send BOSH HM metrics through the Firehose, install the open-source HM Forwarder.
  • To retrieve BOSH health metrics outside of the Firehose, install the JMX Bridge for PCF tile.

Upgrading Pivotal Cloud Cache

Follow the steps below to upgrade PCC on PCF:

  1. Download the new version of the tile from the Pivotal Network.
  2. Upload the product to Ops Manager.
  3. Click Add next to the uploaded product.
  4. Click on the Cloud Cache tile and review your configuration options.
  5. Click Apply Changes.

Updating Pivotal Cloud Cache Plans

Follow the steps below to update plans in Ops Manager.

  1. Click on the PCC tile.
  2. Click on the plan you want to update under the Information section.
  3. Edit the fields with the changes you want to make to the plan.
  4. Click Save button on the bottom of the page.
  5. Click on the PCF Ops Manager to navigate to the Installation Dashboard.
  6. Click Apply Changes.

Plan changes are not applied to existing services instances until you run the upgrade-all-service-instances BOSH errand. You must use the BOSH CLI to run this errand. Until you run this errand, developers cannot update service instances.

Changes to fields that can be overridden by optional parameters, for example num_servers or new_size_percentage, will change the default value of these instance properties, but will not affect existing service instances.

If you change the allowed limits of an optional parameter, for example the maximum number of servers per cluster, existing service instances in violation of the new limits will not be modified.

When existing instances are upgraded, all plan changes are applied to them.

Uninstalling Pivotal Cloud Cache

To uninstall PCC, follow the steps from below from the Installation Dashboard:

  1. Click the trash can icon in the bottom-right-hand corner of the tile.
  2. Click Apply Changes.

Troubleshooting

View Statistics Files

You can visualize the performance of your cluster by downloading the statistics files from your servers. These files are located in the persistent store on each VM. To copy these files to your workstation, run the following command:

bosh scp server/0:/var/vcap/store/gemfire-server/statistics.gfs /tmp

See the Pivotal GemFire Installing and Running VSD topic for information about loading the statistics files into Pivotal GemFire VSD.

Smoke Test Failures

Error: “Creating p-cloudcache SERVICE-NAME failed”

The smoke tests could not create an instance of GemFire. To troubleshoot why the deployment failed, use the cf CLI to create a new service instance using the same plan and download the logs of the service deployment from BOSH.

Error: “Deleting SERVICE-NAME failed”

The smoke test attempted to clean up a service instance it created and failed to delete the service using the cf delete-service command. To troubleshoot this issue, run bosh logs to view the logs on the broker or the service instance to see why the deletion may have failed.

Error: Cannot connect to the cluster SERVICE-NAME

The smoke test was unable to connect to the cluster.

To troubleshoot the issue, review the logs of your load balancer, and review the logs of your CF Router to ensure the route to your PCC cluster is properly registered.

You also can create a service instance and try to connect to it using the gfsh CLI. This requires creating a service key.

Error: “Could not perform create/put on Cloud Cache cluster”

The smoke test was unable to write data to the cluster. The user may not have permissions to create a region or write data.

Error: “Could not retrieve value from Cloud Cache cluster”

The smoke test was unable to read back the data it wrote. Data loss can happen if a cluster member improperly stops and starts again or if the member machine crashes and is resurrected by BOSH. Run bosh logs to view the logs on the broker to see if there were any interruptions to the cluster by a service update.

General Connectivity

Client-to-Server Communication

PCC Clients communicate to PCC servers on port 40404 and with locators on port 55221. Both of these ports must be reachable from the Elastic Runtime network to service the network.

Membership Port Range

PCC servers and locators communicate with each other using UDP and TCP. The current port range for this communication is 49152-65535.

If you have a firewall between VMs, ensure this port range is open.

Create a pull request or raise an issue on the source for this page in GitHub