Advanced Troubleshooting with the BOSH CLI

Page last updated:

This topic describes using the BOSH CLI to help diagnose and resolve issues with your Pivotal Cloud Foundry (PCF) deployment. Before using the information and techniques in this topic, you might want to review the Pivotal Cloud Foundry Troubleshooting Guide.

To follow the steps in this topic, you must log in to the BOSH Director. The BOSH Director runs on the virtual machine (VM) that Ops Manager deploys on the first install of the Ops Manager Director tile.

After authenticating into the BOSH Director, you can run specific commands using the BOSH Command Line Interface (BOSH CLI). BOSH Director diagnostic commands have access to information about your entire Pivotal Cloud Foundry (PCF) installation.

Note: Before running any BOSH CLI commands, verify that no BOSH Director tasks are running on the Ops Manager VM. See the Tasks section of BOSH CLI commands for more information.

Gather Credential and IP Address Information

Before you begin troubleshooting with the BOSH CLI, follow the instructions below to collect the information you need from the Ops Manager interface.

  1. Open the Ops Manager interface by navigating to the Ops Manager fully qualified domain name (FQDN) in a web browser.

  2. Click the Ops Manager Director tile and select the Status tab.

  3. Record the IP address for the Director job. This is the IP address of the VM where the BOSH Director runs.

    Ops mgr job ip

  4. Select the Credentials tab.

  5. Click Link to Credential to view the Director Credentials. Record these credentials.

    Bosh creds

  6. Return to the Installation Dashboard.

  7. (Optional) To prepare to troubleshoot the job VM for any other product, click the product tile and repeat the procedure above to record the IP address and VM credentials for that job VM.

  8. Log out of Ops Manager.

Note: Ensure that there are no Ops Manager installations or updates in progress while using the BOSH CLI.

SSH into Ops Manager

Use SSH to connect to the Ops Manager web application VM. Follow the instructions in one of the sections below to SSH into the Ops Manager VM.

vSphere:

To SSH into the Ops Manager VM in vSphere, you need the credentials used to import the PCF .ova or .ovf file into your virtualization system.

  1. From a command line, run ssh ubuntu@OPS-MANAGER-FQDN to SSH into the Ops Manager VM. Replace OPS-MANAGER-FQDN with the fully qualified domain name of Ops Manager.

  2. When prompted, enter the password that you set during the .ova deployment into vCenter. For example:

    $ ssh ubuntu@my-opsmanager-fqdn.example.com
    Password: ***********
    

AWS, Azure, and OpenStack:

To SSH into the Ops Manager VM in AWS, Azure, and OpenStack, follow these instructions:

  1. Locate the Ops Manager FQDN on the AWS EC2 instances page or the OpenStack Access & Security page.

  2. Run chmod 600 ops_mgr.pem to change the permissions on the .pem file to be more restrictive. For example:

    $ chmod 600 ops_mgr.pem
    
  3. Run ssh -i ops_mgr.pem ubuntu@OPS-MANAGER-FQDN to SSH into the Ops Manager VM. Replace OPS-MANAGER-FQDN with the fully qualified domain name of Ops Manager. For example:

    $ ssh -i ops_mgr.pem ubuntu@my-opsmanager-fqdn.example.com
    

GCP:

To SSH into the Ops Manager VM in GCP, follow these instructions:

  1. Confirm that you have installed the gcloud CLI. See the Google Cloud Platform documentation for more information.

  2. Run gcloud config set project MY-PROJECT to configure your Google Cloud Platform project. For example:

    $ gcloud config set project gcp
    

  3. Run gcloud auth login MY-GCP-ACCOUNT. For example:

    $ gcloud auth login user@example.com
    

  4. Run gcloud compute ssh MY-INSTANCE --zone MY-ZONE. For example:

    $ gcloud compute ssh om-pcf-1a --zone us-central1-b
    

  5. Run sudo su - ubuntu to switch to the ubuntu user.

Log in to the BOSH Director

Follow the steps below to log in to the BOSH Director.

Note: The procedures in this topic use BOSH CLI v2. The BOSH Director VM includes both BOSH CLI versions, but some BOSH CLI v1 commands are incompatible with PCF 1.12 deployments. Run the BOSH commands in this topic using bosh2.

Create a Local BOSH Director Alias

  1. Run the following command to create a local alias for the BOSH Director using the BOSH CLI: bosh2 alias-env MY-ENV -e DIRECTOR-IP-ADDRESS --ca-cert /var/tempest/workspaces/default/root_ca_certificate

    Replace the placeholder text with the following:

    • MY-ENV: Enter an alias for the BOSH Director, such as gcp.
    • DIRECTOR-IP-ADDRESS: Enter the IP address of your Ops Manager Director VM. For example:
      $ bosh2 alias-env gcp -e 10.0.0.3 --ca-cert /var/tempest/workspaces/default/root_ca_certificate
  2. Log in to the BOSH Director using one of the following options:

Log in to the BOSH Director with UAA

  1. Retrieve the Director password from the Ops Manager Director > Credentials tab. Alternatively, launch a browser and visit https://OPS-MANAGER-FQDN/api/v0/deployed/director/credentials/director_credentials to obtain the password. Replace OPS-MANAGER-FQDN with the fully qualified domain name of Ops Manager.

  2. Run bosh2 -e MY-ENV log-in to log in to the BOSH Director. Replace MY-ENV with the alias for your BOSH Director. For example:

    $ bosh2 -e gcp log-in
    Follow the BOSH CLI prompts and enter the Ops Manager Director credentials to log in to the BOSH Director.

Log in to the BOSH Director with SAML

  1. Log in to your identity provider and use the following information to configure SAML Service Provider Properties:

    • Service Provider Entity ID: bosh-uaa
    • ACS URL: https://DIRECTOR-IP-ADDRESS:8443/saml/SSO/alias/bosh-uaa
    • Binding: HTTP Post
    • SLO URL: https://DIRECTOR-IP-ADDRESS:8443/saml/SSO/alias/bosh-uaa
    • Binding: HTTP Redirect
    • Name ID: Email Address
  2. Run bosh2 -e MY-ENV log-in to log in to the BOSH Director. Replace MY-ENV with the alias for your BOSH Director. For example:

    $ bosh2 -e gcp log-in
    Follow the BOSH CLI prompts and enter your SAML credentials to log in to the BOSH Director.

    If you do not have browser access to the BOSH Director, run sshuttle on a local Linux workstation to browse the BOSH Director IP address as if it were a local address. Retrieve a UAA passcode using the browser:
    $ git clone https://github.com/apenwarr/sshuttle.git
    $ cd sshuttle
    $ ./sshuttle -r username@opsmanagerIP 0.0.0.0/0 -vv
    

  3. Click Log in with organization credentials (SAML).

    Login saml credentials

  4. Copy the Temporary Authentication Code that appears in your browser.

    Saml login temp auth code

  5. You see a login confirmation. For example:

    Logged in as admin@example.org
    

Use the BOSH CLI for Troubleshooting

This section describes three BOSH CLI commands commonly used during troubleshooting.

  • VMs: Lists the VMs in a deployment
  • Cloud Check: Runs a cloud consistency check and interactive repair
  • SSH: Starts an interactive session or executes commands with a VM

BOSH VMs

The bosh2 vms command provides an overview of the virtual machines that BOSH manages.

To use this command, run bosh2 -e MY-ENV -d MY-DEPLOYMENT vms. Replace MY-ENV with your environment, and MY-DEPLOYMENT with a deployment. -d MY-DEPLOYMENT is optional.

When troubleshooting an issue with your deployment, bosh2 vms may show a VM in an unknown state. Run bosh2 cloud-check on a VM in an unknown state to instruct BOSH to diagnose problems with the VM.

You can also run bosh2 vms to identify VMs in your deployment, then use the bosh2 ssh command to SSH into an identified VM for further troubleshooting.

bosh2 vms supports the following arguments:

  • --dns: Report also includes the DNS A record for each VM
  • --vitals: Report also includes load, CPU, memory usage, swap usage, system disk usage, ephemeral disk usage, and persistent disk usage for each VM

Note: The Status tab of the Elastic Runtime product tile displays information similar to the bosh2 vms output.

BOSH Cloud Check

Run the bosh2 cloud-check command to instruct BOSH to detect differences between the VM state database maintained by the BOSH Director and the actual state of the VMs. For each difference detected, bosh2 cloud-check can offer the following repair options:

  • Reboot VM: Instructs BOSH to reboot a VM. Rebooting can resolve many transient errors.
  • Ignore problem: Instructs BOSH to do nothing. You may want to ignore a problem in order to run bosh2 ssh and attempt troubleshooting directly on the machine.
  • Reassociate VM with corresponding instance: Updates the BOSH Director state database. Use this option if you believe that the BOSH Director state database is in error and that a VM is correctly associated with a job.
  • Recreate VM using last known apply spec: Instructs BOSH to destroy the server and recreate it from the deployment manifest that the installer provides. Use this option if a VM is corrupted.
  • Delete VM reference: Instructs BOSH to delete a VM reference in the Director state database. If a VM reference exists in the state database, BOSH expects to find an agent running on the VM. Select this option only if you know that this reference is in error. Once you delete the VM reference, BOSH can no longer control the VM.

To use this command, run bosh2 -e MY-ENV -d MY-DEPLOYMENT cloud-check. Replace MY-ENV with your environment, and MY-DEPLOYMENT with your deployment.

Example Scenarios

Unresponsive Agent

  $ bosh2 -e example-env -d example-deployment cloud-check
  ccdb/0 (vm-3e37133c-bc33-450e-98b1-f86d5b63502a) is not responding:

  - Ignore problem
  - Reboot VM
  - Recreate VM using last known apply spec
  - Delete VM reference (DANGEROUS!)

Missing VM

  $ bosh2 -e example-env -d example-deployment cloud-check
  VM with cloud ID `vm-3e37133c-bc33-450e-98b1-f86d5b63502a' missing:

  - Ignore problem
  - Recreate VM using last known apply spec
  - Delete VM reference (DANGEROUS!)

Unbound Instance VM

  $ bosh2 -e example-env -d example-deployment cloud-check
  VM `vm-3e37133c-bc33-450e-98b1-f86d5b63502a' reports itself as `ccdb/0' but does not have a bound instance:

  - Ignore problem
  - Delete VM (unless it has persistent disk)
  - Reassociate VM with corresponding instance

Out of Sync VM

  $ bosh2 -e example-env -d example-deployment cloud-check
  VM `vm-3e37133c-bc33-450e-98b1-f86d5b63502a' is out of sync:
  expected `cf-d7293430724a2c421061: ccdb/0', got `cf-d7293430724a2c421061: nats/0':

  - Ignore problem
  - Delete VM (unless it has persistent disk)

BOSH SSH

Use bosh2 ssh to SSH into the VMs in your deployment.

Follow the steps below to use bosh2 ssh:

  1. Identify a VM to SSH into. Run bosh2 -e MY-ENV -d MY-DEPLOYMENT vms to list the VMs in the given deployment. Replace MY-ENV with your environment alias and MY-DEPLOYMENT with the deployment name.

  2. Run bosh2 -e MY-ENV -d MY-DEPLOYMENT ssh VM-NAME/GUID. For example:

    $ bosh2 -e example-env -d example-deployment ssh diego-cell/abcd0123-a012-b345-c678-9def01234567

Create a pull request or raise an issue on the source for this page in GitHub