Troubleshooting On-Demand Instances

This topic provides techniques app developers can use to begin troubleshooting on-demand instances.

If Instances or Database are Inaccessible

You might experience the following during upgrades or in a leader-follower topology.

Temporary Outages

MySQL for PCF service instances can become temporarily inaccessible during upgrades and VM or network failures. For more information, see Service Interruptions.

Apps Cannot Write to the Database

If you have a leader-follower topology and your apps can no longer write to the database, the leader VM might be read-only. Contact your operator to check if the leader VM is read-only. For more information about troubleshooting this problem, see Both Leader and Follower Instances Are Read-Only.

If you can no longer read to the database as well, your persistent disk might be full. For more information about troubleshooting inoperable apps, see Apps are Inoperable.

Apps are Inoperable

If your apps become inoperable, your persistent disk might be full. In this state, read, write, and Cloud Foundry Command-Line Interface (cf CLI) opertations do not work.

Contact your operator to check if your persistent disk is full. For more information about troubleshooting this problem, see Persistent Disk is Full.

Troubleshoot Errors

You might see an error when using the Cloud Foundry Command-Line Interface (cf CLI) to perform basic operations on a MySQL for PCF service instance:

  • cf create
  • cf update
  • cf bind
  • cf unbind
  • cf delete

Understand a Cloud Foundry (CF) Error Message

Failed operations (create, update, bind, unbind, delete) result in an error message. You can retrieve the error message later by running the cf CLI command cf service INSTANCE-NAME. 

$ cf service myservice

Service instance: myservice
Service: super-db
Bound apps:
Tags:
Plan: dedicated-vm
Description: Dedicated Instance
Documentation url:
Dashboard:

Last Operation
Status: create failed
Message: Instance provisioning failed: There was a problem completing your request.
     Please contact your operations team providing the following information:
     service: redis-acceptance,
     service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089,
     broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac,
     task-id: 442,
     operation: create
Started: 2017-03-13T10:16:55Z
Updated: 2017-03-13T10:17:58Z

Use the information in the Message field to debug further. Provide this information to Pivotal Support when filing a ticket.

The task-id field maps to the BOSH task ID. For more information on a failed BOSH task, use the bosh task TASK-ID.

The broker-request-guid maps to the portion of the On-Demand Broker log containing the failed step. Access the broker log through your syslog aggregator, or access BOSH logs for the broker by typing bosh logs broker 0. If you have more than one broker instance, repeat this process for each instance.

Find Information about Your Service Instance

You might need to find the name, GUID, or other information about a service instance. To find this information, do the following:

  1. Log into the space containing the instance or failed instance. 

    $ cf login

  2. If you do not know the name of the service instance, run cf services to see a listing of all service instances in the space. The service instances are listed in the name column. 

    $ cf services
Getting services in org my-org / space my-space as user@example.com...
OK
name          service      plan        bound apps    last operation
my-instance   p.mysql      db-small                  create succeeded

  3. To retrieve more information about a specific instance, run cf service SERVICE-INSTANCE-NAME

  4. To retrieve the GUID of the instance, run cf service SERVICE-INSTANCE-NAME --guid

The GUID is useful for debugging.

Error: Failed to Set Credentials in Credential Store

If you encounter: 

error: failed to set credentials in credential store:
The request includes an unrecognized parameter 'mode'.
Please update or remove this parameter and retry your request.

after upgrading from PCF 2.3 to 2.4, it is likely because one or more installed on-demand brokers were not restarted during the upgrade. To clear the cached server version and enable your on-demand brokers to communicate with CredHub v2, do the procedure in Run BOSH Restart on Your On-Demand Service Brokers.

Use the Knowledge Base (Community)

Find the answer to your question and browse product discussions and solutions by searching the Pivotal Knowledge Base.

File a Support Ticket

You can file a support ticket here. Be sure to provide the error message from cf service YOUR-SERVICE-INSTANCE.

To expedite troubleshooting, if possible, provide your service broker logs, service instance logs, and BOSH task output. Your cloud operator should be able to obtain these from your error message.