LATEST VERSION: 0.17.2 - CHANGELOG

Techniques for Troubleshooting

This topic contains instructions on interacting with the on-demand service broker and on-demand service instance BOSH deployments, and on performing general maintenance and housekeeping tasks.

About the BOSH CLI

The BOSH CLI is available in two major versions, v1 and v2.

Where appropriate, this topic provides examples of using each version of the BOSH CLI. Consult the table below to determine which version of the CLI is supported for your Pivotal Cloud Foundry (PCF) installation.

PCF Version BOSH CLI Version
v1.10 CLI v1
v1.11 CLI v1 or CLI v2 (Pivotal recommends CLI v2)
v1.12 CLI v2

Parse a Cloud Foundry (CF) Error Message

Failed operations (create, update, bind, unbind, delete) result in an error message. You can retrieve the error message later by running the cf CLI command cf service INSTANCE-NAME.

$ cf service myservice

Service instance: myservice
Service: super-db
Bound apps:
Tags:
Plan: dedicated-vm
Description: Dedicated Instance
Documentation url:
Dashboard: 

Last Operation
Status: create failed
Message: Instance provisioning failed: There was a problem completing your request. 
     Please contact your operations team providing the following information: 
     service: redis-acceptance, 
     service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089,
     broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, 
     task-id: 442, 
     operation: create
Started: 2017-03-13T10:16:55Z
Updated: 2017-03-13T10:17:58Z

Use the information in the Message field to debug further. Provide this information to Pivotal Support when filing a ticket.

The task-id field maps to the BOSH task id. For further information on a failed BOSH task, use the bosh task TASK-ID command in the BOSH CLI.

The broker-request-guid maps to the portion of the On-Demand Broker log containing the failed step. Access the broker log through your syslog aggregator, or access BOSH logs for the broker by typing bosh logs broker 0. If you have more than one broker instance, repeat this process for each instance.

Note: This procedure is the same for v1 and v2 of the BOSH CLI.

Access Broker and Instance Logs and VMs

Before following the procedures below, log into the cf CLI and the BOSH CLI.

Access Broker Logs and VM(s)

BOSH CLI v2: Accessing logs

The following procedure is for v2 of the BOSH CLI.

  1. Run bosh deployments to identify the on-demand broker (ODB) deployment.
  2. Run bosh manifest ODB-DEPLOYMENT-NAME odb.yml to download the ODB manifest.
  3. Select the ODB deployment using bosh deployment odb.yml.
  4. View VMs in the deployment using bosh instances.
  5. Run bosh ssh INSTANCE-ID to SSH onto the VM.
  6. Run bosh logs INSTANCE-ID to download broker logs.
BOSH CLI v1: Accessing logs

The following procedure is for v1 of the BOSH CLI.

  1. Run bosh deployments to identify the on-demand broker (ODB) deployment.
  2. Run bosh download manifest ODB-DEPLOYMENT-NAME odb.yml to download the ODB manifest.
  3. Select the ODB deployment using bosh deployment odb.yml.
  4. View VMs in the deployment using bosh instances.
  5. Run bosh ssh INSTANCE-ID to SSH onto the VM.
  6. Run bosh logs INSTANCE-ID to download broker logs.

You can also access logs using Ops Manager by clicking on the Logs tab in the tile and downloading the broker logs.

The archive generated by BOSH or Ops Manager includes the following logs:

Log Name Description
broker.log Requests to the on-demand broker and the actions the broker performs while orchestrating the request (e.g. generating a manifest and calling BOSH). Start here when troubleshooting.
broker_ctl.log Control script logs for starting and stopping the on-demand broker.
post-start.stderr.log Errors that occur during post-start verification.
post-start.stdout.log Post-start verification.
drain.stderr.log Errors that occur while running the drain script.

Access Service Instance Logs and VMs

  1. To target an individual service instance deployment, retrieve the GUID of your service instance with the cf CLI command cf service MY-SERVICE --guid.

  2. Run bosh status --uuid to retrieve the BOSH Director GUID.

    Note: “GUID” and “UUID” mean the same thing.

  3. To download your BOSH manifest for the service, run bosh download manifest service-instance_SERVICE-INSTANCE-GUID MY-SERVICE.yml using the GUID you just obtained and a filename you want to save the manifest as.

  4. Edit the following line in the service instance manifest that you just saved, to include the current BOSH Director GUID:

        director_uuid: BOSH-DIRECTOR-GUID
    

  5. Run bosh deployment MY-SERVICE.yml to select the deployment using the Director UUID.

  6. Run bosh instances to view VMs in the deployment.

  7. Run bosh ssh INSTANCE-ID to SSH onto the VM.

  8. Run bosh logs INSTANCE-ID to download instance logs.

Run Service Broker Errands to Manage Brokers and Instances

From the BOSH CLI, you can run service broker errands that manage the service brokers and perform mass operations on the service instances that the brokers created. These service broker errands include:

BOSH CLI v1: Running the errand

If you use v1 of the BOSH CLI, you must configure the service broker manifest to run errands.

Set the service broker manifest in the BOSH CLI by doing the following:

  1. Run bosh deployments.
  2. In the Name column of the output, look or grep for a string of the form p-rabbit-GUID. This is the unique identifier for the broker in BOSH.
  3. Run bosh download manifest RABBIT-BROKER-GUID BROKER-MANIFEST.yml with the unique string from the previous step and any filename you want to give the broker deployment manifest.
  4. Run bosh deployment BROKER-MANIFEST.yml to select the broker deployment as the one to run broker errands against.

Register Broker

This errand registers the broker with Cloud Foundry and enables access to plans in the service catalog. Run this errand whenever the broker is re-deployed with new catalog metadata to update the Cloud Foundry catalog.

Plans with disabled service access are not visible to non-admin Cloud Foundry users (including Org Managers and Space Managers). Admin Cloud Foundry users can see all plans including those with disabled service access.

The errand does the following:

  • Registers the service broker with Cloud Controller.
  • Enables service access for any plans that have the radio button set to enabled in the tile plan page.
  • Disables service access for any plans that have the radio button set to disabled in the tile plan page.
  • Does nothing for any for any plans that have the radio button set to manual.
BOSH CLI v2: Running the errand

The following procedure is for v2 of the BOSH CLI.

Run this errand with the command bosh run-errand register-broker after you have selected the broker deployment with bosh deployment.

BOSH CLI v1: Running the errand

The following procedure is for v1 of the BOSH CLI.

Run this errand with the command bosh run errand register-broker.

Deregister Broker

This errand deregisters a broker from Cloud Foundry.

The errand does the following:

  • Deletes the service broker from Cloud Controller
  • Fails if there are any service instances, with or without bindings

Use the Delete All Service Instances errand to delete any existing service instances.

BOSH CLI v2: Running the errand

The following procedure is for v2 of the BOSH CLI.

Run this errand with the command bosh run-errand deregister-broker.

BOSH CLI v1: Running the errand

The following procedure is for v1 of the BOSH CLI.

Run this errand with the command bosh run errand deregister-broker after you have selected the broker deployment with bosh deployment.

Upgrade All Service Instances

If you have made changes to the plan definition or uploaded a new tile into Ops Manager, you may want to upgrade all the on-demand service instances to the latest software/plan definition.

The errand does the following:

  • Collects all of the service instances the on-demand broker has registered.
  • For each instance the errand serially:
    • Issues an upgrade command to the on-demand broker.
    • Re-generates the service instance manifest based on its latest configuration from the tile.
    • Deploys the new manifest for the service instance.
    • Waits for this operation to complete, then proceeds to the next instance.
  • Adds to a retry list any instances that have ongoing BOSH tasks at the time of upgrade.
  • Retries any instances in the retry list until all are upgraded.

If any instance fails to upgrade, the errand fails immediately. This prevents systemic problems from spreading to the rest of your service instances. Run the errand by following either of the procedures below.

BOSH CLI v2: Running the errand

The following procedure is for v2 of the BOSH CLI.

You can either select the errand through the Ops Manager UI and have it run when you click Apply Changes, or you can run the errand directly with the command bosh run-errand upgrade-all-service-instances.

BOSH CLI v1: Running the errand

The following procedure is for v1 of the BOSH CLI.

You can either select the errand through the Ops Manager UI and have it run when you click Apply Changes, or you can run the errand directly with the command bosh run errand upgrade-all-service-instances after you have selected the broker deployment with bosh deployment.

Delete All Service Instances

This errand deletes all service instances of your broker’s service offering in every org and space of Cloud Foundry. It uses the Cloud Controller API to do this, and therefore only deletes instances the Cloud Controller knows about. It will not delete orphan BOSH deployments.

Orphan BOSH deployments don’t correspond to a known service instance. While rare, orphan deployments can occur. Use the orphan-deployments errand to identify them.

The errand does the following:

  • Unbinds all applications from the service instances.
  • Deletes all service instances sequentially.
  • Checks if any instances have been created while the errand was running.
  • If newly-created instances are detected, the errand fails.

WARNING: This errand should only be used with extreme caution when you want to totally destroy all of the on-demand service instances in an environment.

BOSH CLI v2: Running the errand

The following procedure is for v2 of the BOSH CLI.

Run this errand with the command bosh run-errand delete-all-service-instances after you have selected the broker deployment with bosh deployment.

BOSH CLI v1: Running the errand

The following procedure is for v1 of the BOSH CLI.

Run this errand with the command bosh run errand delete-all-service-instances after you have selected the broker deployment with bosh deployment.

Detect Orphaned Instances Service Instances

A service instance is defined as ‘orphaned’ when the BOSH deployment for the instance is still running, but the service is no longer registered in Cloud Foundry.

The orphan-deployments errand collates a list of service deployments that have no matching service instances in Cloud Foundry and return the list to the operator. It is then up to the operator to remove the orphaned bosh deployments.

BOSH CLI v2: Running the errand

The following procedure is for v2 of the BOSH CLI.

Run this errand with the command bosh run-errand orphan-deployments after you have selected the broker deployment with bosh deployment.

BOSH CLI v1: Running the errand

The following procedure is for v1 of the BOSH CLI.

Run this errand with the command bosh run errand orphan-deployments after you have selected the broker deployment with bosh deployment.

If orphan deployments exist, the errand script will:

  • Exit with exit code 10
  • Output a list of deployment names under a [stdout] header
  • Provide a detailed error message under a [stderr] header

For example:

[stdout]
[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]

[stderr] Orphan BOSH deployments detected with no corresponding service instance in Cloud Foundry. Before deleting any deployment it is recommended to verify the service instance no longer exists in Cloud Foundry and any data is safe to delete.

Errand 'orphan-deployments' completed with error (exit code 10)

These details will also be available through the BOSH /tasks/ API endpoint for use in scripting:

$ curl 'https://bosh-user:bosh-password@bosh-url:25555/tasks/task-id/output?type=result' | jq .
{
  "exit_code": 10,
  "stdout": "[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]\n",
  "stderr": "Orphan BOSH deployments detected with no corresponding service instance in Cloud Foundry. Before deleting any deployment it is recommended to verify the service instance no longer exists in Cloud Foundry and any data is safe to delete.\n",
  "logs": {
    "blobstore_id": "d830c4bf-8086-4bc2-8c1d-54d3a3c6d88d"
  }
}

If no orphan deployments exist, the errand script will:

  • Exit with exit code 0
  • Stdout will be an empty list of deployments
  • Stderr will be None
[stdout]
[]

[stderr]
None

Errand 'orphan-deployments' completed successfully (exit code 0)

If the errand encounters an error during running it will:

  • Exit with exit 1
  • Stdout will be empty
  • Any error messages will be under stderr
BOSH CLI v2: Cleaning up orphaned instances

The following procedure is for v2 of the BOSH CLI.

To clean up orphaned instances, perform the following action on each:

$ bosh delete-deployment service-instance_SERVICE-INSTANCE-GUID

WARNING: This may leave IaaS resources in an unusable state.

BOSH CLI v1: Cleaning up orphaned instances

The following procedure is for v1 of the BOSH CLI.

To clean up orphaned instances, perform the following action on each:

$ bosh delete deployment service-instance_SERVICE-INSTANCE-GUID

WARNING: This may leave IaaS resources in an unusable state.

Select the BOSH Deployment for a Service Instance

BOSH CLI v1: Additional troubleshooting options

The following procedure is for v1 of the BOSH CLI. This troubleshooting tip does not apply to v2 of the BOSH CLI.

  1. Retrieve the GUID of your service instance with command cf service YOUR-SERVICE-INSTANCE --guid.

  2. To download your BOSH manifest for the service, run bosh download manifest service-instance_SERVICE-INSTANCE-GUID myservice.yml GUID you just obtained and a filename you want to save the manifest as.

  3. Run bosh deployment MY-SERVICE.yml to select the deployment.

Get Admin Credentials for a Service Instance

  1. Identify the service deployment by GUID.
  2. Log into BOSH.
  3. Download the manifest for the service instance and add the GUID if using the v1 BOSH CLI.
  4. Look in the manifest for the credentials, as described in the service documentation.

Reinstall a Tile

To reinstall a tile in the same environment where it was previously uninstalled:

  1. Ensure that the previous tile was correctly uninstalled as follows:
    1. Log in as an admin with cf login.
    2. Use cf m to confirm that the Marketplace does not list the service.
    3. Log in to BOSH as an admin with bosh login.
    4. Use bosh deployments to confirm that the output does not show a the service deployment.
  2. Run the “delete-all-service-instances” errand to delete every instance of the service.
  3. Run the “deregister-broker” errand to delete the service broker.
  4. Depending on which version of the BOSH CLI you are using, follow one of the steps below:
    1. For BOSH CLI v2: Use bosh delete-deployment BROKER-DEPLOYMENT-NAME to delete the service broker BOSH deployment.
    2. For BOSH CLI v1: Use bosh delete deployment BROKER-DEPLOYMENT-NAME to delete the service broker BOSH deployment.
  5. Reinstall the tile.

View Resource Saturation and Scaling

To view usage statistics for any service, select the service broker deployment. Then run bosh vms --vitals and bosh instances --vitals to view current resource utilization.

You can also view process-level information by using bosh instances --ps.

Note: This procedure is the same for v1 and v2 of the BOSH CLI.

Identify Apps using a Service Instance

If you want to identify which apps are using a specific service instance from the BOSH deployments name, you can run the following steps:

  1. Take the deployment name and strip the service-instance_ leaving you with the GUID.
  2. Login to CF as an admin.
  3. Obtain a list of all service bindings by running the following: cf curl /v2/service_instances/<GUID>/service_bindings
  4. The output from the above curl will give you a list of resources, with each item referencing a service binding, which contains the app_url. To find the name, org and space for the app, run the following:
    1. cf curl <app_url> and note the app name under entity.name
    2. cf curl <space_url> to obtain the space, using the entity.space_url from the above curl. Note the space name under entity.name
    3. cf curl <organization_url> to obtain the org, using the entity.organization_url from the above curl. Note the organization name under entity.name

Note: When running cf curl ensure that you query all pages, as the responses are limited to a certain number of bindings per page (default is 50). To find the next page simply curl the value under next_url

Monitor Quota Saturation and Service Instance Count

Quota saturation and total number of service instances are available through ODB metrics emitted to Loggregator. The metric names are shown below:

Metric Name Description
on-demand-broker/{service-name-marketplace}/quota_remaining global quota remaining for all instances across all plans
on-demand-broker/{service-name-marketplace}/{plan_name}/quota_remaining quota remaining for a particular plan
on-demand-broker/{service-name-marketplace}/total_instances total instances created across all plans
on-demand-broker/{service-name-marketplace}/{plan_name}/total_instances total instances created for a given plan

Note: Quota metrics are not emitted if no quota has been set.

Knowledge Base (Community)

Find the answer to your question and browse product discussions and solutions by searching the Pivotal Knowledge Base.

File a Support Ticket

You can file a support ticket here. Be sure to provide the error message from cf service YOUR-SERVICE-INSTANCE.

To help expedite troubleshooting, also provide your service broker logs, your service instance logs and BOSH task output, if your cf service YOUR-SERVICE-INSTANCE output includes a task-id.

Create a pull request or raise an issue on the source for this page in GitHub