On-Demand Services SDK v0.15

Troubleshooting Errors

Start here if you’re responding to a specific errors or error messages.

Failed Install

  1. Certificate issues: The on-demand broker (ODB) requires valid certificates. Ensure that your certificates are valid and generate new ones if necessary.
  2. Deploy fails: Deploys can fail for a variety of reasons. View the logs using Ops Manager to determine why the deploy is failing.
  3. Networking problems:
    • Cloud Foundry cannot reach the on-demand service broker
    • Cloud Foundry cannot reach the service instances
    • The service network cannot access the BOSH director
  4. Register broker errand fails.
  5. The smoke test errand fails.
  6. Resource sizing issues: These occur when the resource sizes selected for a given plan are less than the on-demand service requires to function. Check your resource configuration in Ops Manager and ensure that the configuration matches that recommended by the service.
  7. Other service-specific issues.

Cannot Create or Delete Service Instances

If developers report errors such as the following:

Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: create

Log in to BOSH and target the on-demand service instance using the instructions on parsing a Cloud Foundry error message.

Retrieve the BOSH task ID from the error message and run bosh task TASK-ID.

If the BOSH error shows a problem with the deployment manifest:

  1. Download the manifest for the on-demand service instance by running bosh download manifest service-instance_SERVICE-INSTANCE-GUID MY-SERVICE.yml.

  2. Check the manifest for configuration errors.

Access the broker logs and use the broker-request-id from the error message above to search the log for more information.

Check for:

Broker Request Timeouts

If developers report errors such as:

Server error, status code: 504, error code: 10001, message: The request to the service broker timed out: https://BROKER-URL/v2/service_instances/e34046d3-2379-40d0-a318-d54fc7a5b13f/service_bindings/aa635a3b-ef6d-41c3-a23f-55752f3f651b
  1. Validate that Cloud Foundry (CF) has network connectivity to the service broker.
  2. Check the BOSH queue size:
    • Log into BOSH as the admin user
    • Run bosh tasks
  3. If there are a large number of queued tasks then the system might be under load. BOSH is configured with two workers and one status worker. Advise app developers to try again later once the system is under less load.

In the future, Ops Manager will support configuring the number of BOSH workers available to the system.

Cannot Bind to or Unbind from Service Instances

Instance Does Not Exist

If developers report errors such as:

Server error, status code: 502, error code: 10001, message: Service broker error: instance does not exist`
  1. Run cf service MY-INSTANCE --guid to check that the on-demand service instance exists in BOSH and CF.

  2. Run bosh vms service-instance_GUID with the GUID you just obtained.

If the bosh deployment is not found then it has been deleted from BOSH. Contact Pivotal support for further assistance on recovery.

Other Errors

If developers report errors such as:

Server error, status code: 502, error code: 10001, message: Service broker error: There was a problem completing your request. Please contact your operations team providing the following information: service: example-service, service-instance-guid: 8d69de6c-88c6-4283-b8bc-1c46103714e2, broker-request-id: 15f4f87e-200a-4b1a-b76c-1c4b6597c2e1, operation: bind

To find out the exact issue with the binding process:

  1. Access the service broker logs.

  2. Search the logs for the broker-request-id string listed in the error message above.

  3. Contact Pivotal support for further assistance if you are unable to resolve the problem.

  4. Check for:

Cannot Connect to a Service Instance

If developers report that their app cannot use service instances that they have successfully created and bound:

Ask the user to send application logs that show the connection error. If the error is originating from the service, then follow service-specific instructions. If the issue appears to be network-related, then:

  1. Check that application security groups are configured correctly. Access should be configured for the service network that the tile is deployed to.

  2. Ensure that the network the Elastic Runtime or Pivotal Application Service (PAS) tile is deployed to has network access to the service network. You can find the network definition for this service network in the Ops Manager Director tile.

  3. From the Ops Manager Installation Dashboard, navigate to the service tile to see that the service network is configured in the Networks section.

  4. From the Ops Manager Installation Dashboard, navigate to the Elastic Runtime (or PAS) tile and see the network it is assigned to. Make sure that these networks can access each other.

Upgrade All Instances Fails

If the upgrade-all-service-instances errand fails, look at the errand output in the Ops Manager log.

If an instance fails to upgrade, debug and fix it before running the errand again to prevent any failure issues from spreading to other on-demand instances.

Once the Ops Manager log no longer lists the deployment as failing, re-run the errand to upgrade the rest of the instances.

Missing Logs and Metrics

If no logs are being emitted by the on-demand broker, check that your syslog forwarding address is correct in Ops Manager.

  1. Ensure you have configured syslog for the tile.
  2. Ensure that you have network connectivity between the networks that the tile is using and the syslog destination. If the destination is external, you need to use the public ip VM extension feature available in your Ops Manager tile configuration settings.
  3. Verify that the Firehose is emitting metrics:
    1. Install the cf nozzle plugin
    2. Run cf nozzle -f ValueMetric | grep --line-buffered "on-demand-broker/MY-SERVICE" to find logs from your service in the cf nozzle output.

If no metrics appear within five minutes, verify that the broker network has access to the Loggregator system on all required ports.

Contact Pivotal support if you are unable to resolve the issue.

Knowledge Base (Community)

Find the answer to your question and browse product discussions and solutions by searching the Pivotal Knowledge Base.

File a Support Ticket

You can file a support ticket here. Be sure to provide the error message from cf service YOUR-SERVICE-INSTANCE.

To help expedite troubleshooting, also provide your service broker logs, your service instance logs and BOSH task output, if your cf service YOUR-SERVICE-INSTANCE output includes a task-id.

Create a pull request or raise an issue on the source for this page in GitHub