LATEST VERSION: 0.17.2 - CHANGELOG

Troubleshooting Components

This topic contains guidance on checking for and fixing issues in the ODB components.

About the BOSH CLI

The BOSH CLI is available in two major versions, v1 and v2.

Where appropriate, this topic provides examples of using each version of the BOSH CLI. Consult the table below to determine which version of the CLI is supported for your Pivotal Cloud Foundry (PCF) installation.

PCF Version BOSH CLI Version
v1.10 CLI v1
v1.11 CLI v1 or CLI v2 (Pivotal recommends CLI v2)
v1.12 CLI v2

BOSH Problems

Missing BOSH Director UUID

BOSH v2 CLI: This error does not apply

If you’re using the BOSH v2 CLI, this error should not occur. The director_uuid is not a configurable field in the BOSH v2 CLI.

BOSH v1 CLI: Modifying the manifest

If using the BOSH v1 CLI, re-add the director_uuid to the manifest:

  1. Run bosh status --uuid and record the director_uuid value from the output.

  2. Edit the manifest and add the director_uuid: DIRECTOR-UUID from the last step at the top of the manifest.

For more, see Deployment Identification in the BOSH docs.

Large BOSH Queue

On-demand service brokers add tasks to the BOSH request queue, which can back up and cause delay under heavy loads. An app developer who requests a new service instance sees create in progress in the Cloud Foundry Command Line Interface (cf CLI) until BOSH processes the queued request.

Ops Manager currently deploys two BOSH workers to process its queue. Future versions of Ops Manager will let users configure the number of BOSH workers.

Configuration

Service instances in failing state

You may have configured a VM / Disk type in tile plan page in Ops Manager that is insufficiently large for the on-demand service instance to start. See tile-specific guidance on resource requirements.

Authentication

UAA Changes

If you have rotated any UAA user credentials then you may see authentication issues in the service broker logs.

To resolve this, redeploy the service tile in Ops Manager. This provides the broker with the latest configuration.

Note: You must ensure that any changes to UAA credentials are reflected in the Ops Manager credentials tab of the Elastic Runtime tile.

Networking

Common issues include:

  1. Network latency when connecting to the on-demand service instance to create or delete a binding.
    • Solution: Try again or improve network performance
  2. Network firewall rules are blocking connections from the on-demand service broker to the service instance.
    • Solution: Open the service tile in Ops Manager and check the two networks configured in the Networks pane. Ensure that these networks allow access to each other.
  3. Network firewall rules are blocking connections from the service network to the BOSH director network.
    • Solution: Ensure that service instances can access the Director so that the BOSH agents can report in.
  4. Apps cannot access the service network.
    • Solution: Configure Cloud Foundry application security groups to allow runtime access to the service network.
  5. Problems accessing BOSH’s UAA or the BOSH director.
    • Solution: Follow network troubleshooting and check that the BOSH director is online.

Validate Service Broker Connectivity to Service Instances

To validate you can bosh ssh onto the on-demand service broker, download the broker manifest and target the deployment, then try to reach the service instance.

If no BOSH task-id appears in the error message, look in the broker log using the broker-request-id from the task.

Note: This procedure is the same for v1 and v2 of the BOSH CLI.

Validate App Access to Service Instance

Use cf ssh to access to the app container, then try connecting to the on-demand service instance using the binding included in the VCAP_SERVICES environment variable.

Quotas

Plan Quota issues

If developers report errors such as:

Message: Service broker error: The quota for this service plan has been exceeded. 
Please contact your Operator for help.
  1. Check your current plan quota.
  2. Increase the plan quota.
  3. Log into Ops Manager.
  4. Reconfigure the quota on the plan page.
  5. Deploy the tile.
  6. Find who is using the plan quota and take the appropriate action.

Global Quota Issues

If developers report errors such as:

Message: Service broker error: The quota for this service has been exceeded. 
Please contact your Operator for help.
  1. Check your current global quota.
  2. Increase the global quota.
  3. Log into Ops Manager.
  4. Reconfigure the quota on the on-demand settings page.
  5. Deploy the tile.
  6. Find out who is using the quota and take the appropriate action.

Failing Jobs and Unhealthy Instances

To determine whether there is an issue with the on-demand service deployment, run bosh vms:

$ bosh vms --vitals service-instance_$guid

For additional information, run bosh instances:

$ bosh instances --ps --vitals

If the VM is failing. follow the service-specific information. Any unadvised corrective actions (such as running bosh restart on a VM) may cause issues in the service instance.

Note: This procedure is the same for v1 and v2 of the BOSH CLI.

Knowledge Base (Community)

Find the answer to your question and browse product discussions and solutions by searching the Pivotal Knowledge Base.

File a Support Ticket

You can file a support ticket here. Be sure to provide the error message from cf service YOUR-SERVICE-INSTANCE.

To help expedite troubleshooting, also provide your service broker logs, your service instance logs and BOSH task output, if your cf service YOUR-SERVICE-INSTANCE output includes a task-id.

Create a pull request or raise an issue on the source for this page in GitHub