LATEST VERSION: 1.10.5 - CHANGELOG
RabbitMQ for PCF v1.8.20

Troubleshooting On-Demand RabbitMQ for PCF

This topic provides operators with basic instructions for troubleshooting on-demand RabbitMQ® for Pivotal Cloud Foundry (PCF).

Troubleshooting Errors

Start here if you’re responding to a specific errors or error messages.

Failed Install

  1. Certificate issues: The on-demand broker (ODB) requires valid certificates. Ensure that your certificates are valid and generate new ones if necessary.
  2. Deploy fails: Deploys can fail for a variety of reasons. View the logs using Ops Manager to determine why the deploy is failing.
  3. Networking problems:
    • Cloud Foundry cannot reach the RabbitMQ for PCF service broker
    • Cloud Foundry cannot reach the service instances
    • The service network cannot access the BOSH director
  4. Register broker errand fails.
  5. The smoke test errand fails.
  6. Resource sizing issues: These occur when the resource sizes selected for a given plan are less than the RabbitMQ for PCF service requires to function. Check your resource configuration in Ops Manager and ensure that the configuration matches that recommended by the service.
  7. Other service-specific issues.

Cannot Create or Delete Service Instances

If developers report errors such as the following:

Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: create

Log in to BOSH and target the RabbitMQ for PCF service instance using the instructions on parsing a Cloud Foundry error message.

Retrieve the BOSH task ID from the error message and run bosh task TASK-ID.

If the BOSH error shows a problem with the deployment manifest:

  1. Download the manifest for the RabbitMQ for PCF service instance by running bosh download manifest service-instance_SERVICE-INSTANCE-GUID MY-SERVICE.yml.

  2. Check the manifest for configuration errors.

Access the broker logs and use the broker-request-id from the error message above to search the log for more information.

Check for:

BOSH CLI v2: This error does not apply

In the BOSH CLI v2, the manifest is not configurable. Open the manifest in a text editor or other program and troubleshoot it there.

BOSH CLI v1: Troubleshooting the deployment manifest

The following procedure is for v1 of the BOSH CLI.

If the BOSH error shows a problem with the deployment manifest:

  1. Download the manifest for the on-demand service instance by running bosh download manifest service-instance_SERVICE-INSTANCE-GUID MY-SERVICE.yml.
  2. Check the manifest for configuration errors.

Broker Request Timeouts

If developers report errors such as:

Server error, status code: 504, error code: 10001, message: The request to the service broker timed out: https://BROKER-URL/v2/service_instances/e34046d3-2379-40d0-a318-d54fc7a5b13f/service_bindings/aa635a3b-ef6d-41c3-a23f-55752f3f651b
  1. Validate that Cloud Foundry (CF) has network connectivity to the service broker.
  2. Check the BOSH queue size:
    • Log into BOSH as the admin user
    • Run bosh tasks
  3. If there are a large number of queued tasks then the system may be under load. BOSH is configured with two workers and one status worker. Advise app developers to try again later once the system is under less load.

In the future, Ops Manager will support configuring the number of BOSH workers available to the system.

Cannot Bind to or Unbind from Service Instances

Instance Does Not Exist

If developers report errors such as:

Server error, status code: 502, error code: 10001, message: Service broker error: instance does not exist`

Use your version of the BOSH CLI to troubleshoot.

BOSH CLI v2:

This procedure is for v2 of the BOSH CLI.

  1. Type cf service MY-INSTANCE --guid. This confirms that the the RabbitMQ for PCF service instance exists in BOSH and CF.

  2. Type bosh -d service-instance_YOUR-GUID vms, using the GUID you just obtained.

If the BOSH deployment is not found, it has been deleted from BOSH. Contact Pivotal support for further assistance.

BOSH CLI v1:

This procedure is for v1 of the BOSH CLI.

  1. Type cf service MY-INSTANCE --guid. This confirms that the the RabbitMQ for PCF service instance exists in BOSH and CF.

  2. Type bosh vms service-instance_GUID, using the GUID you just obtained.

If the BOSH deployment is not found, it has been deleted from BOSH. Contact Pivotal support for further assistance.

Other Errors

If developers report errors such as:

Server error, status code: 502, error code: 10001, message: Service broker error: There was a problem completing your request. Please contact your operations team providing the following information: service: example-service, service-instance-guid: 8d69de6c-88c6-4283-b8bc-1c46103714e2, broker-request-id: 15f4f87e-200a-4b1a-b76c-1c4b6597c2e1, operation: bind

To find out the exact issue with the binding process:

  1. Access the service broker logs.

  2. Search the logs for the broker-request-id string listed in the error message above.

  3. Contact Pivotal support for further assistance if you are unable to resolve the problem.

  4. Check for:

Cannot Connect to a Service Instance

If developers report that their app cannot use service instances that they have successfully created and bound:

Ask the user to send application logs that show the connection error. If the error is originating from the service, then follow RabbitMQ for PCF-specific instructions. If the issue appears to be network-related, then:

  1. Check that application security groups are configured correctly. Access should be configured for the service network that the tile is deployed to.

  2. Ensure that the network the PCF Elastic Runtime tile is deployed to has network access to the service network. You can find the network definition for this service network in the Ops Manager Director tile.

  3. In Ops Manager go into the service tile and see the service network that is configured in the networks tab.

  4. In Ops Manager go into the ERT tile and see the network it is assigned to. Make sure that these networks can access each other.

Upgrade All Instances Fails

If the upgrade-all-service-instances errand fails, look at the errand output in the Ops Manager log.

If an instance fails to upgrade, debug and fix it before running the errand again to prevent any failure issues from spreading to other on-demand instances.

Once the Ops Manager log no longer lists the deployment as failing, re-run the errand to upgrade the rest of the instances.

Missing Logs and Metrics

If no logs are being emitted by the on-demand broker, check that your syslog forwarding address is correct in Ops Manager.

  1. Ensure you have configured syslog for the tile.
  2. Ensure that you have network connectivity between the networks that the tile is using and the syslog destination. If the destination is external, you need to use the public ip VM extension feature available in your Ops Manager tile configuration settings.
  3. Verify that the Firehose is emitting metrics:
    1. Install the cf nozzle plugin
    2. Run cf nozzle -f ValueMetric | grep --line-buffered "on-demand-broker/MY-SERVICE" to find logs from your service in the cf nozzle output.

If no metrics appear within five minutes, verify that the broker network has access to the Loggregator system on all required ports.

Contact Pivotal support if you are unable to resolve the issue.

Troubleshooting Components

Guidance on checking for and fixing issues in on-demand service components.

BOSH problems

Missing BOSH Director UUID

If using the BOSH v1 CLI you need to re-add the director_uuid to the manifest:

  1. Run bosh status --uuid and record the director_uuid value from the output.

  2. Edit the manifest and add the director_uuid: DIRECTOR-UUID from the last step at the top of the manifest.

For more, see Deployment Identification in the BOSH docs.

Large BOSH Queue

On-demand service brokers add tasks to the BOSH request queue, which can back up and cause delay under heavy loads. An app developer who requests a new RabbitMQ for PCF instance sees create in progress in the Cloud Foundry Command Line Interface (cf CLI) until BOSH processes the queued request.

Ops Manager currently deploys two BOSH workers to process its queue. Future versions of Ops Manager will let users configure the number of BOSH workers.

Configuration

Service instances in failing state

You may have configured a VM / Disk type in tile plan page in Ops Manager that is insufficiently large for the RabbitMQ for PCF service instance to start. See tile-specific guidance on resource requirements.

Authentication

UAA Changes

If you have rotated any UAA user credentials then you may see authentication issues in the service broker logs.

To resolve this, redeploy the RabbitMQ for PCF tile in Ops Manager. This provides the broker with the latest configuration.

Note: You must ensure that any changes to UAA credentials are reflected in the OpsManager credentials tab of the Elastic Runtime tile.

Networking

Common issues include:

  1. Network latency when connecting to the RabbitMQ for PCF service instance to create or delete a binding.
    • Solution: Try again or improve network performance
  2. Network firewall rules are blocking connections from the RabbitMQ for PCF service broker to the service instance.
    • Solution: Open the RabbitMQ for PCF tile in Ops Manager and check the two networks configured in the Networks pane. Ensure that these networks allow access to each other.
  3. Network firewall rules are blocking connections from the service network to the BOSH director network.
    • Solution: Ensure that service instances can access the Director so that the BOSH agents can report in.
  4. Apps cannot access the service network.
    • Solution: Configure Cloud Foundry application security groups to allow runtime access to the service network.
  5. Problems accessing BOSH’s UAA or the BOSH director.
    • Solution: Follow network troubleshooting and check that the BOSH director is online.

Validate Service Broker Connectivity to Service Instances

To validate you can bosh ssh onto the RabbitMQ for PCF service broker, download the broker manifest and target the deployment, then try to reach the service instance.

If no BOSH task-id appears in the error message, look in the broker log using the broker-request-id from the task.

Validate App Access to Service Instance

Use cf ssh to access to the app container, then try connecting to the RabbitMQ for PCF service instance using the binding included in the VCAP_SERVICES environment variable.

Quotas

Plan Quota issues

If developers report errors such as:

Message: Service broker error: The quota for this service plan has been exceeded. 
Please contact your Operator for help.
  1. Check your current plan quota.
  2. Increase the plan quota.
  3. Log into Ops Manager.
  4. Reconfigure the quota on the plan page.
  5. Deploy the tile.
  6. Find who is using the plan quota and take the appropriate action.

Global Quota Issues

If developers report errors such as:

Message: Service broker error: The quota for this service has been exceeded. 
Please contact your Operator for help.
  1. Check your current global quota.
  2. Increase the global quota.
  3. Log into Ops Manager.
  4. Reconfigure the quota on the on-demand settings page.
  5. Deploy the tile.
  6. Find out who is using the quota and take the appropriate action.

Failing jobs and unhealthy instances

BOSH CLI v2: Troubleshooting the deployment

This procedure is for v2 of the BOSH CLI.

To determine whether there is an issue with the RabbitMQ for PCF service deployment, inspect the VMs.

$ bosh -d service-instance_SERVICE-INSTANCE-GUID vms --vitals 

If the VM is failing. follow the service-specific information. Any unadvised corrective actions (such as running bosh restart on a VM) may cause issues in the service instance.

BOSH CLI v1: Troubleshooting the deployment

To determine whether there is an issue with the RabbitMQ for PCF service deployment, run bosh vms:

$ bosh vms --vitals service-instance_SERVICE-INSTANCE-GUID

For additional information, run bosh instances:

$ bosh instances --ps --vitals

If the VM is failing. follow the service-specific information. Any unadvised corrective actions (such as running bosh restart on a VM) may cause issues in the service instance.

Techniques for Troubleshooting

Instructions on interacting with the on-demand service broker and on-demand service instance BOSH deployments, and on performing general maintenance and housekeeping tasks

Parse a Cloud Foundry (CF) Error Message

Failed operations (create, update, bind, unbind, delete) result in an error message. You can retrieve the error message later by running the cf CLI command cf service INSTANCE-NAME.

$ cf service myservice

Service instance: myservice
Service: super-db
Bound apps:
Tags:
Plan: dedicated-vm
Description: Dedicated Instance
Documentation url:
Dashboard: 

Last Operation
Status: create failed
Message: Instance provisioning failed: There was a problem completing your request. 
     Please contact your operations team providing the following information: 
     service: redis-acceptance, 
     service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089,
     broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, 
     task-id: 442, 
     operation: create
Started: 2017-03-13T10:16:55Z
Updated: 2017-03-13T10:17:58Z

Use the information in the Message field to debug further. Provide this information to Pivotal Support when filing a ticket.

Troubleshooting failed BOSH tasks

The task-id field maps to the BOSH task id. For further information on a failed BOSH task, use the bosh task TASK-ID command in the BOSH CLI.

Note: This procedure is the same for v1 and v2 of the BOSH CLI.

Accessing logs

The broker-request-guid maps to the portion of the On-Demand Broker log containing the failed step. Access the broker log through your syslog aggregator, or access BOSH logs for the broker by typing bosh logs broker 0. If you have more than one broker instance, repeat this process for each instance.

Note: This procedure is the same for v1 and v2 of the BOSH CLI.

Access Broker and Instance Logs and VMs

Before following the procedures below, log into the cf CLI and the BOSH CLI.

Access Broker Logs and VM(s)

BOSH CLI v2: Accessing logs

The following procedure is for v2 of the BOSH CLI.

  1. Run bosh deployments to identify the on-demand broker (ODB) deployment.
  2. View VMs in the deployment using bosh -d DEPLOYMENT-NAME instances.
  3. Run bosh -d DEPLOYMENT-NAME ssh INSTANCE-ID to SSH onto the VM.
  4. Run bosh -d DEPLOYMENT-NAME logs INSTANCE-ID to download broker logs.
BOSH CLI v1: Accessing logs

The following procedure is for v1 of the BOSH CLI.

  1. Run bosh deployments to identify the on-demand broker (ODB) deployment.
  2. Run bosh download manifest ODB-DEPLOYMENT-NAME odb.yml to download the ODB manifest.
  3. Select the ODB deployment using bosh deployment odb.yml.
  4. View VMs in the deployment using bosh instances.
  5. Run bosh ssh INSTANCE-ID to SSH onto the VM.
  6. Run bosh logs INSTANCE-ID to download broker logs.

You can also access logs using Ops Manager by clicking on the Logs tab in the tile and downloading the broker logs.

The archive generated by BOSH or Ops Manager includes the following logs:

Log Name Description
broker.log Requests to the on-demand broker and the actions the broker performs while orchestrating the request (e.g. generating a manifest and calling BOSH). Start here when troubleshooting.
broker_ctl.log Control script logs for starting and stopping the on-demand broker.
post-start.stderr.log Errors that occur during post-start verification.
post-start.stdout.log Post-start verification.

Access Service Instance Logs and VMs

BOSH CLI v2: Accessing logs and VMs

This procedure is for v2 of the BOSH CLI.

  1. To target an individual service instance deployment, retrieve the GUID of your service instance with the cf CLI command cf service MY-SERVICE --guid.

  2. Run bosh -d service-instance_YOUR-SERVICE-INSTANCE-GUID instances to view VMs in the deployment.

  3. Run bosh -d service-instance_YOUR-SERVICE-INSTANCE-GUID ssh INSTANCE-ID to SSH onto the VM.

  4. Run bosh -d service-instance_YOUR-SERVICE-INSTANCE-GUID logs INSTANCE-ID to download the instance logs.

BOSH CLI v1: Accessing logs and VMs

This procedure is for v1 of the BOSH CLI.

  1. To target an individual service instance deployment, retrieve the GUID of your service instance with the cf CLI command cf service MY-SERVICE --guid.

  2. Run bosh status --uuid to retrieve the BOSH Director GUID.

    Note: “GUID” and “UUID” mean the same thing.

  3. To download your BOSH manifest for the service, run bosh download manifest service-instance_SERVICE-INSTANCE-GUID MY-SERVICE.yml using the GUID you just obtained and a filename you want to save the manifest as.

  4. Edit the following line in the service instance manifest that you just saved, to include the current BOSH Director GUID:

        director_uuid: BOSH-DIRECTOR-GUID
    

  5. Run bosh deployment MY-SERVICE.yml to select the deployment using the Director UUID.

  6. Run bosh instances to view VMs in the deployment.

  7. Run bosh ssh INSTANCE-ID to SSH onto the VM.

  8. Run bosh logs INSTANCE-ID to download instance logs.

Run Service Broker Errands to Manage Brokers and Instances

From the BOSH CLI, you can run service broker errands that manage the service brokers and perform mass operations on the service instances that the brokers created. These service broker errands include:

Before running any of these errands, set the service broker manifest in the BOSH CLI by doing the following:

  1. Run bosh deployments.
  2. In the Name column of the output, look or grep for a string of the form p-rabbit-GUID. This is the unique identifier for the broker in BOSH.
  3. Run bosh download manifest RABBIT-BROKER-GUID BROKER-MANIFEST.yml with the unique string from the previous step and any filename you want to give the broker deployment manifest.
  4. Run bosh deployment BROKER-MANIFEST.yml to select the broker deployment as the one to run broker errands against.

Register Broker

This errand registers the broker with Cloud Foundry and enables access to plans in the service catalog. The errand should be run whenever the broker is re-deployed with new catalog metadata to update the Cloud Foundry catalog.

Plans with disabled service access are not visible to non-admin Cloud Foundry users (including Org Managers and Space Managers). Admin Cloud Foundry users can see all plans including those with disabled service access.

The errand does the following:

  • Registers the service broker with Cloud Controller
  • Enables service access for any plans that have the radio button set to enabled in the tile plan page.
  • Disables service access for any plans that have the radio button set to disabled in the tile plan page.
  • Does nothing for any for any plans that have the radio button set to manual
BOSH CLI v2: Running the errand

The following procedure is for v2 of the BOSH CLI.

Run this errand with the command bosh -d DEPLOYMENT-NAME run-errand register-broker.

BOSH CLI v1: Running the errand

The following procedure is for v1 of the BOSH CLI.

Run this errand with the command bosh run-errand register-broker after you have selected the broker deployment with bosh deployment.

Deregister Broker

This errand deregisters a broker from Cloud Foundry. It requires that there are no existing service instances. You can use the Delete All Service Instances errand to delete any existing service instances that are problematic.

The errand does the following:

  • Deletes the service broker from Cloud Controller
  • Fails if there are any service instances, with or without bindings

Run this errand with the command bosh run errand deregister-broker after you have selected the broker deployment with bosh deployment.

Upgrade All Service Instances

If you have made changes to the plan definition or uploaded a new tile into OpsManager you may want to upgrade all the RabbitMQ for PCF service instances to the latest software / plan definition.

To do this you can either select the errand through the OpsManager UI and have this happen along with the apply-changes action or run the errand directly with the command bosh run errand upgrade-all-service-instances after you have selected the broker deployment with bosh deployment.

The errand does the following:

  • Collects all of the service instances the on-demand broker has registered.
  • For each instance the errand serially:
    • Issues an upgrade command to the on-demand broker.
    • Re-generates the service instance manifest based on its latest configuration from the tile.
    • Deploys the new manifest for the service instance.
    • Waits for this operation to complete, then proceeds to the next instance.
  • Adds to a retry list any instances that have ongoing BOSH tasks at the time of upgrade.
  • Retries any instances in the retry list until all are upgraded.

If any instance fails to upgrade, the errand fails immediately. This prevents systemic problems from spreading to the rest of your service instances.

Delete All Service Instances

This errand deletes all service instances of your broker’s service offering in every org and space of Cloud Foundry. It uses the Cloud Controller API to do this, and therefore only deletes instances the Cloud Controller knows about. It will not delete orphan BOSH deployments: those that don’t correspond to a known service instance. Orphan BOSH deployments should never happen, but in practice they might. Use the orphan-deployments errand to identify them.

The errand does the following:

  • Unbinds all applications from the service instances.
  • Deletes all service instances sequentially.
  • Checks if any instances have been created while the errand was running.
  • If newly-created instances are detected, the errand fails.

WARNING: This errand should only be used with extreme caution when you want to totally destroy all of the on-demand service instances in an environment.

Run this errand with the command bosh run errand delete-all-service-instances after you have selected the broker deployment with bosh deployment.

Detect Orphaned Instances Service Instances

A service instance is defined as ‘orphaned’ when the BOSH deployment for the instance is still running, but the service is no longer registered in Cloud Foundry.

The orphan-deployments errand collates a list of service deployments that have no matching service instances in Cloud Foundry and return the list to the operator. It is then up to the operator to remove the orphaned bosh deployments.

Run this errand with the command bosh run errand orphan-deployments after you have selected the broker deployment with bosh deployment.

If orphan deployments exist, the errand script will:

  • Exit with exit code 10
  • Output a list of deployment names under a [stdout] header
  • Provide a detailed error message under a [stderr] header

For example:

[stdout]
[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]

[stderr] Orphan BOSH deployments detected with no corresponding service instance in Cloud Foundry. Before deleting any deployment it is recommended to verify the service instance no longer exists in Cloud Foundry and any data is safe to delete.

Errand 'orphan-deployments' completed with error (exit code 10)

These details will also be available through the BOSH /tasks/ API endpoint for use in scripting:

$ curl 'https://bosh-user:bosh-password@bosh-url:25555/tasks/task-id/output?type=result' | jq .
{
  "exit_code": 10,
  "stdout": "[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]\n",
  "stderr": "Orphan BOSH deployments detected with no corresponding service instance in Cloud Foundry. Before deleting any deployment it is recommended to verify the service instance no longer exists in Cloud Foundry and any data is safe to delete.\n",
  "logs": {
    "blobstore_id": "d830c4bf-8086-4bc2-8c1d-54d3a3c6d88d"
  }
}

If no orphan deployments exist, the errand script will:

  • Exit with exit code 0
  • Stdout will be an empty list of deployments
  • Stderr will be None
[stdout]
[]

[stderr]
None

Errand 'orphan-deployments' completed successfully (exit code 0)

If the errand encounters an error during running it will:

  • Exit with exit 1
  • Stdout will be empty
  • Any error messages will be under stderr

To clean up orphaned instances, perform the following action on each:

$ bosh delete deployment service-instance_SERVICE-INSTANCE-GUID

WARNING: This may leave IaaS resources in an unusable state.

Select the BOSH Deployment for a Service Instance

BOSH CLI v2: Selecting a BOSH deployment

  1. Retrieve the GUID of your service instance with command cf service YOUR-SERVICE-INSTANCE --guid.

  2. To download your BOSH manifest for the service, run bosh manifest -d service-instance_SERVICE-INSTANCE-GUID > MY-SERVICE.yml GUID you just obtained and a filename you want to save the manifest as.

BOSH CLI v1: Selecting a BOSH deployment

  1. Retrieve the GUID of your service instance with command cf service YOUR-SERVICE-INSTANCE --guid.

  2. To download your BOSH manifest for the service, run bosh download manifest service-instance_SERVICE-INSTANCE-GUID myservice.yml GUID you just obtained and a filename you want to save the manifest as.

  3. Run bosh deployment MY-SERVICE.yml to select the deployment.

Get Admin Credentials for a Service Instance

  1. Identify the service deployment by GUID.
  2. Log into BOSH.
  3. Download the manifest for the service instance and add the GUID if using the v1 BOSH CLI.
  4. Look in the manifest for the credentials, as described in the service documentation.

Reinstall a Tile

To reinstall a tile in the same environment where it was previously uninstalled:

  1. Ensure that the previous tile was correctly uninstalled as follows:
    1. cf login as an admin user.
    2. Run cf m and confirm that the Marketplace does not list RabbitMQ for PCF.
    3. bosh login as an admin user.
    4. Run bosh deployments and confirm that its output does not show a deployment for RabbitMQ for PCF. For example no p-concourse-GUID deployment exists.
  2. Run the delete-all-service-instances errand to delete all instances of the service.
  3. Run the deregister-broker errand to delete the service broker.
  4. Run bosh delete deployment YOUR-BROKER-DEPLOYMENT to delete the service broker BOSH deployment.
  5. Install the tile again.

View Resource Saturation and Scaling

To view usage statistics for any service, select the service broker deployment. Then run bosh vms --vitals and bosh instances --vitals to view current resource utilization.

You can also view process-level information by using bosh instances --ps.

Identify Service Instance Owner

If you have spotted a failing service instance deployment, you can identify which org/space owns the instance and list the apps bound to it by following these steps:

  1. bosh vms service-instance_SERVICE-INSTANCE-GUID shows a VM in a failing state.
  2. Take the deployment name and strip the service-instance_ leaving you with the GUID.
  3. Login to CF as an admin.
  4. Run cf curl /v2/service_instances/GUID | grep "space_url" and record the SPACE-GUID field from the output "space_url": "/v2/spaces/SPACE-GUID".
  5. run cf curl /v2/spaces/SPACE-GUID | grep -E "name|organization_url" with the SPACE-GUID above and record from the output:
    1. The SPACE-NAME field from the output "name": "MY-SPACE", and
    2. The ORG-GUID field from the output "organization_url": "/v2/organizations/ORG-GUID".
  6. Run cf curl /v2/organizations/ORG-GUID | grep "name" with the ORG-GUID above, and record the MY-ORG field from the output "name": "MY-ORG".
  7. Run cf target -o MY-ORG -s MY-SPACE with the values above to target the org and space.
  8. Run cf services to see all the services and bound apps in the space.
  9. Run cf curl /v2/spaces/SPACE-GUID/managers to find out who is the space manager.
  10. Use this information to contact the space manager if needed.

Monitor Quota Saturation and Service Instance Count

Quota saturation and total number of service instances are available through ODB metrics emitted to Loggregator. The metric names are shown below:

Metric Name Description
on-demand-broker/{service-name-marketplace}/quota_remaining global quota remaining for all instances across all plans
on-demand-broker/{service-name-marketplace}/{plan_name}/quota_remaining quota remaining for a particular plan
on-demand-broker/{service-name-marketplace}/total_instances total instances created across all plans
on-demand-broker/{service-name-marketplace}/{plan_name}/total_instances total instances created for a given plan

Note: Quota metrics are not emitted if no quota has been set.

Knowledge Base (Community)

Find the answer to your question and browse product discussions and solutions by searching the Pivotal Knowledge Base.

File a Support Ticket

You can file a support ticket here. Be sure to provide the error message from cf service YOUR-SERVICE-INSTANCE.

To help expedite troubleshooting, also provide your service broker logs, your service instance logs and BOSH task output, if your cf service YOUR-SERVICE-INSTANCE output includes a task-id.

Create a pull request or raise an issue on the source for this page in GitHub