LATEST VERSION: 1.9 - CHANGELOG
JMX Bridge v1.8

Troubleshooting and Uninstalling JMX Bridge

Page last updated:

This topic describes how to resolve common issues with the JMX Bridge for Pivotal Cloud Foundry (PCF) tile and how to uninstall the tile if necessary.

Troubleshoot

The following sections provide help with troubleshooting JMX Bridge for PCF.

Dependency Error During Install

You might see the following error if your installed version of Elastic Runtime is not compatible with the version of JMX Bridge that you are attempting to install: JMX Bridge requires 'cf' version '~> 1.8.0.0' as a dependency.

Jmx ert install depedency

Follow the instructions below that correspond to your installed Elastic Runtime version:

  • Elastic Runtime v1.8.0 to v1.8.16:
    1. Download version v1.8.17 or later of the Elastic Runtime tile and import it into your Ops Manager Installation Dashboard.
    2. Click the newly added tile to review any configurable options. For more information, see Installing Pivotal Cloud Foundry and choose your IaaS.
    3. Click Apply Changes to install both Elastic Runtime and JMX Bridge.
  • Elastic Runtime v1.8.17 or later: Upgrade the JMX Bridge tile to v1.8.9 or later.

    Note: A known issue exists with older versions of JMX Bridge and the newer versions of Elastic Runtime. JMX Bridge v1.8.2 through v1.8.8 only work with Elastic Runtime v1.8.0 through v1.8.16.

Cyclic Dependency Error During Install

You might see the following error if you have Elastic Runtime and JMX Bridge installed.

Jmx ert cyclic depedency

  • JMX Bridge v1.8.7 or later; Elastic Runtime v1.8:
    1. The Collector component is no longer compatible with JMX Bridge. All metrics that were previously flowing through the Collector are now flowing through the Firehose in Elastic Runtime v1.8.
    2. Scale down the Collector in Elastic Runtime > Resource Config to 0
    3. Click Apply Changes and continue your JMX Bridge installation.

Note: This is only an issue for Elastic Runtime v1.8. The Collector does not exist in Elastic Runtime v1.9.

Missing Metrics from PCF Installation or Firehose

If you do not see expected metrics from Elastic Runtime in the JMX provider, verify that you installed Elastic Runtime before JMX Bridge. If you installed JMX Bridge first, perform the following steps:

  1. SSH into the opentsdb-firehose-nozzle VM. For information about how to use the BOSH CLI to SSH into a VM, see Advanced Troubleshooting with the BOSH CLI.
  2. Grant sudo access to the machine:
    $ sudo -i
  3. Restart the opentsdb-firehose-nozzle job.
    $ monit restart opentsdb-firehose-nozzle

Missing BOSH Metrics

If you do not see expected metrics from BOSH, try the following steps:

  1. Make sure the IP address in JMX Bridge > Status > JMX Provider matches the value entered in Ops Manager Director > Director Config > Metrics IP Address.
  2. If the JMX Provider IP address matches the Metrics IP address and you see no BOSH metrics in the system, contact Pivotal Support for help.

Validating JMX Bridge MBeans

If you do not see metrics from JMX Bridge in your third-party tooling integration as expected, first try the following steps to quickly debug whether there is an issue with the JMX Bridge product or if the issue is with the tooling integration:

  1. Verify Java 6+ is installed.
  2. Run jconsole:
    $ jconsole
  3. Select Remote Process and enter the IP of the JMX Provider VM with port 44444.
  4. Fill in the username and password for the JMX Provider that was entered during installation of JMX Bridge.
  5. Click Connect.

    Jmx jconsole connnect

  6. Allow Insecure connection if SSL was not enabled.

    Jmx jsonsole insecure

You can now view all MBeans emitted by JMX Bridge.

Jmx jconsole mbeans

Note: If you have enabled SSL, see Using SSL with a Self-Signed Certificate in JMX Bridge.

Set Up Port Forwarding for JMX

If you are connecting to jconsole from a location different from the install location (for example, deployed on AWS or GCP), you have to set up port forwarding to access the MBeans.

  1. Set up port forwarding on one tab of your console and keep it open:

    ssh -D 7777 username@pcf.domain.com -T

  2. Start jconsole in a new tab and set up the socksProxyPort to the forwarded port:

    jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777

  3. Navigate jconsole as normal.

Smoke Tests

If errors occur when the smoke tests run, you can find the errors in the ChangeLog for the installation. Some common failures are listed below.

Error internalMetricsAreSent() Fails
Cause The JMX Provider did not receive internal health metrics from the OpenTSDB Firehose Nozzle.
Solution Restart the OpenTSDB Firehose Nozzle VM and check the logs to verify it is running correctly.
Error receivingFirehoseMetrics() Fails
Cause The OpenTSDB Firehose Nozzle is not receiving metrics from the Firehose.
Solution Restart the OpenTSDB Firehose Nozzle VM and check the logs to verify it is connected to the Firehose. If you see a lot of reconnect attempts in the logs then you likely need to scale up the number of OpenTSDB Firehose Nozzle instances in the Resource Config tab.

Uninstall

To uninstall the JMX Bridge for PCF tile, see Deleting a Product.

Create a pull request or raise an issue on the source for this page in GitHub