Troubleshooting

Service Installation Troubleshooting

Problem: Could not deploy a service instance in the Elastic Runtime

If you try to deploy (and allocate) more Session State Caching service instances than are available as defined by your Pivotal Cloud Foundry (PCF) Administrator, you may receive the following error message:

500 ERROR. YOU HAVE REACHED THIS PAGE BECAUSE AN ERROR OCCURRED. PLEASE CONTACT YOUR OPERATOR

You need to delete existing service instances or ask the admin to deploy more instances.

Problem: Service plan tile does not show up in the Developer Console Marketplace

Verify that the register errand under lifecycle errands is selected in the Ops Manager tile configuration screen.

Application Troubleshooting

Problem: Application does not pick up changes to the service instance

Make sure you restage (or re-push) your application if you have made configuration changes to the underlying service instance.

Service Broker Troubleshooting

The Service Broker is responsible for managing the lifecycle of service instances. If you encounter issues with service instance management (creating, destroying, binding, unbinding etc.), the broker logs may contain some clues.

The logs can be found on broker VMs at /var/vcap/sys/log/broker/broker.stdout.log and are logged in JSON format. You can use a utility such as jq to parse these logs and extract information. log_level of each message is number 0-3 corresponding to the following log levels: DEBUG, INFO, ERROR and FATAL. Here are some examples (using jq 1.5):

Display all ERROR logs:

jq '. | select(.log_level == 2)' broker.stdout

Search log for messages containing a substring:

jq '. | select(.message | contains("some-substring"))' broker.stdout

Check out the jq manual for more usage details.

Broker endpoints for advanced troubleshooting

  • GET /admin/credentials/{locator ip}: Given an IP address of a locator, will retrieve the credentials for the servers in the Gemfire cluster.
  curl 10.0.16.153:8080/admin/credentials/10.0.16.155
  {
    "credentials": [
      {
        "password": "apassword",
        "username": "ausername",
        "locators": [
          "10.0.16.154",
          "10.0.16.155"
        ]
      }
    ]
  }
  • GET /admin/wan/credentials/{service instance GUID}: Given the GUID of a Gemfire service instance, will retrieve the WAN credentials for that server
  curl 10.0.16.153:8080/admin/wan/credentials/7dd9446b-20c6-4fbc-a31f-8416fd96abbd
  {
    "password": "apassword",
    "username": "ausername",
    "locators": [
      "10.0.16.154[55221]",
      "10.0.16.155[55221]"
    ]
  }
  • GET /admin/instance_counts: Displays the number of available service instances associated with the broker
  curl 10.0.16.153:8080/admin/instance_counts
  [
    {
      "count": 3,
      "status": "AVAILABLE"
    }
  ]
  • GET /admin/cluster_config/{service instance GUID}: Retrieves the cluster config as a zip file which contains cluster.xml and cluster.properties
  curl 10.0.16.153:8080/admin/cluster_config/47574053-7050-4911-9277-5725b27e1e38 > cluster_config.zip
  • POST /admin/cluster_config/{service instance GUID}: Updates the cluster config of the specified service instance

Options:

  {
     'reset-defaults': bool,
     'clear-logs': bool,
     'clear-disk-stores': bool,
     'cluster_config': zip file (cluster/cluster.xml, cluster/cluster.properties and any custom code to be deployed on GemFire server),
     'cluster_properties': file, unsure
  }
  curl -v -F reset-defaults=true 10.0.16.153:8080/admin/cluster_config/47574053-7050-4911-9277-5725b27e1e38
  • GET /admin/archives/{service instance GUID}: Retrieves the log files of the cluster as a zip file
  curl 10.0.16.153:8080/admin/archives/47574053-7050-4911-9277-5725b27e1e38 > archives.zip`
  • DELETE /v2/service_instances/{service instance GUID} : Deletes a service instance.
  curl -v -X DELETE http://10.0.18.151:8080/v2/service_instances/608363f1-4811-480d-bd95-b2ad9833cac5
Create a pull request or raise an issue on the source for this page in GitHub