Troubleshooting Errors

Start here if you’re responding to a specific errors or error messages.

Failed Install

  1. Certificate issues: The on-demand broker (ODB) requires valid certificates. Ensure that your certificates are valid and generate new ones if necessary.
  2. Deploy fails: Deploys can fail for a variety of reasons. View the logs using Ops Manager to determine why the deploy is failing.
  3. Networking problems:
    • Cloud Foundry cannot reach the service broker
    • Cloud Foundry cannot reach the service instances
    • The service network cannot access the BOSH director
  4. Register broker errand fails.
  5. The smoke test errand fails.
  6. Resource sizing issues: These occur when the resource sizes selected for a given plan are less than the service requires to function. Check your resource configuration in Ops Manager and ensure that the configuration matches that recommended by the service.
  7. Other service-specific issues.

Cannot Create or Delete Service Instances

If developers report errors such as the following:

Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: create

Log in to BOSH and target the service instance using the instructions on parsing a Cloud Foundry error message.

Retrieve the BOSH task ID from the error message and run bosh task TASK-ID.

If the BOSH error shows a problem with the deployment manifest:

  1. Download the manifest for the service instance by running bosh download manifest SERVICE-INSTANCE-GUID MY-SERVICE.yml.
  2. Check the manifest for configuration errors.

Access the broker logs and use the broker-request-id from the error message above to search the log for more information.

Check for:

Broker Request Timeouts

If developers report errors such as:

Server error, status code: 504, error code: 10001, message: The request to the service broker timed out: https://BROKER-URL/v2/service_instances/e34046d3-2379-40d0-a318-d54fc7a5b13f/service_bindings/aa635a3b-ef6d-41c3-a23f-55752f3f651b
  1. Validate that Cloud Foundry (CF) has network connectivity to the service broker.
  2. Check the BOSH queue size:
    • Log into BOSH as the admin user
    • Run bosh tasks
  3. If there are a large number of queued tasks then the system may be under load. BOSH is configured with two workers and one status worker. Advise app developers to try again later once the system is under less load.

In the future, Ops Manager will support configuring the number of BOSH workers available to the system.

Cannot Bind to or Unbind from Service Instances

Instance Does Not Exist

If developers report errors such as:

Server error, status code: 502, error code: 10001, message: Service broker error: instance does not exist`
  1. Run cf service MY-INSTANCE --guid to check that the service instance exists in BOSH and CF.

  2. Pre-pend the text service_instance- to the GUID you just obtained to create the ID that BOSH uses for your service instance (e.g. if your GUID is 1Hy6H, this becomes service_instance-1Hy6H).

  3. Run bosh vms INSTANCE-GUID with the service instance ID from the last step.

If the bosh deployment is not found then it has been deleted from BOSH. Contact Pivotal support for further assistance on recovery.

Other errors

If developers report errors such as:

Server error, status code: 502, error code: 10001, message: Service broker error: There was a problem completing your request. Please contact your operations team providing the following information: service: example-service, service-instance-guid: 8d69de6c-88c6-4283-b8bc-1c46103714e2, broker-request-id: 15f4f87e-200a-4b1a-b76c-1c4b6597c2e1, operation: bind

To find out the exact issue with the binding process:

  1. Access the service broker logs.

  2. Search the logs for the broker-request-id string listed in the error message above.

  3. Contact Pivotal support for further assistance if you are unable to resolve the problem.

  4. Check for:

Cannot Connect to a Service Instance

If developers report that their app cannot use service instances that they have successfully created and bound:

Ask the user to send application logs that show the connection error. If the error is originating from the service, then follow service-specific instructions. If the issue appears to be network-related, then:

  1. Check that application security groups are configured correctly. Access should be configured for the service network that the tile is deployed to.

  2. Ensure that the network the PCF Elastic Runtime tile is deployed to has network access to the service network. You can find the network definition for this service network in the Ops Manager Director tile.

  3. In Ops Manager go into the service tile and see the service network that is configured in the networks tab.

  4. In Ops Manager go into the ERT tile and see the network it is assigned to. Make sure that these networks can access each other.

Upgrade All Instances Fails

If the upgrade-all-service-instances errand fails, look at the errand output in the Ops Manager log.

If an instance fails to upgrade, debug and fix it before running the errand again to prevent any failure issues from spreading to other on-demand instances.

Once the Ops Manager log no longer lists the deployment as failing, re-run the errand to upgrade the rest of the instances.

Upgrade All Instances Takes a Long Time

  1. Depending on your IaaS and the number of service instances, the upgrade-all-service-instances task takes 15-20 minutes per instance if you changed IaaS resources and 5-10 minutes per instance if you reconfigured BOSH releases.
  2. To calculate how long the errand should take, multiply the number of instances by 20 minutes.
  3. If the errand goes on for much longer than this, you may want to:
    • ssh onto the errand VM
    • cd /var/vcap/sys/log and look for the errand output log
    • tail -f LOG and see if the errand is still attempting to upgrade instances. If it is, then allow the errand to continue. If it appears to be stuck:
    • bosh tasks and identify the errand task
    • bosh cancel task TASK-ID
  4. Retry running the errand.

Missing Logs and Metrics

If no logs are being emitted by the on-demand broker, check that your syslog forwarding address is correct in Ops Manager.

  1. Ensure you have configured syslog for the tile.
  2. Ensure that you have network connectivity between the networks that the tile is using and the syslog destination. If the destination is external, you need to use the public ip VM extension feature available in your Ops Manager tile configuration settings.
  3. Verify that the Firehose is emitting metrics:
    1. Install the cf nozzle plugin
    2. Run cf nozzle -f ValueMetric | grep --line-buffered "on-demand-broker/MY-SERVICE" to find logs from your service in the cf nozzle output.

If no metrics appear within five minutes, verify that the broker network has access to the Loggregator system on all required ports.

Contact Pivotal support if you are unable to resolve the issue.

File a Support Ticket

You can file a support ticket here. Please be sure to provide the error message from

$ cf service my-service

Please also provide your service broker logs, your service instance logs and BOSH task output, if a task-id is provided as part of the cf service <your-service> output, to help expedite troubleshooting.

Knowledge Base (Community)

Imagine a world where we autopopulate the first 10 hits from . For now, you’ll just have to click on the link.

Create a pull request or raise an issue on the source for this page in GitHub