Using RabbitMQ Queue Depth as a Scaling Metric

Page last updated:

This topic describes the RabbitMQ queue depth metric and how to configure App Autoscaler to use this metric when scaling apps.

Overview of RabbitMQ Queue Depth

RabbitMQ is a message broker that allows apps to communicate using message queues. Queue depth is the number of messages currently in a RabbitMQ queue that are waiting to be de-queued. You can configure Autoscaler to use RabbitMQ queue depth as the scaling metric for apps that consume messages from RabbitMQ queues for processing. When Autoscaler scales these apps up, they can process these RabbitMQ queues more quickly.

You can configure Autoscaler to use RabbitMQ queue depth as the scaling metric for an app in the following ways:

You can also configure Autoscaler to query queues on an external RabbitMQ cluster by creating a user-provided service that represents the external RabbitMQ cluster. For more information, see Use an External RabbitMQ Cluster below.

To monitor when Autoscaler scales an app based on changes in RabbitMQ queue depth, see Reviewing Autoscaling Events for Changes in RabbitMQ Queue Depth below.

For information about use cases that may complicate or prevent you from configuring RabbitMQ queue depth as the scaling metric for an app, see Special Considerations for Using RabbitMQ Queue Depth as a Scaling Metric below.

VMware recommends that you load-test your app to verify that the autoscaling rules you configured are effective. For more information, see Load-Testing Your App in Using Autoscaler in Production.

For more information about RabbitMQ queues, see the RabbitMQ documentation.

Configuring RabbitMQ Queue Depth as the Scaling Metric for an App Through the cf CLI

The procedures in this section describe how to configure Autoscaler to use RabbitMQ queue depth as the scaling metric for an app through the cf CLI.

You can configure configure Autoscaler to use RabbitMQ queue depth as the scaling metric for an app in the following ways:

For the procedures in this section, you must use the App Autoscaler CLI plugin. To download and install the App Autoscaler CLI plugin, see Install the App Autoscaler CLI Plugin in Using the App Autoscaler CLI.

Configure an Autoscaling Rule Using a Manifest File

You can configure autoscaling rules declaratively through a manifest file. This manifest file only configures Autoscaler, and does not interfere with any other existing app manifest files in your TAS for VMs deployment.

To configure an autoscaling rule that defines RabbitMQ queue depth as its scaling metric using a manifest file:

  1. In a terminal window, target the space in which the app you want to scale is deployed by running:

    cf target -o ORG-NAME -s SPACE-NAME
    

    Where:

    • ORG-NAME is the name of the org containing the space in which the app you want to scale is deployed.
    • SPACE-NAME is the name of the space in which the app you want to scale is deployed.
  2. If the space in which the app you want to scale is deployed does not already have an Autoscaler service instance of Autoscaler deployed in it, create an Autoscaler service instance by running:

    cf create-service app-autoscaler PLAN-NAME SERVICE-NAME
    

    Where:

    • PLAN-NAME is the name of the service plan you want to use for the Autoscaler service instance.
    • SERVICE-INSTANCE-NAME is the name you want to give the Autoscaler service instance. For example, autoscaler.

      If there is already an Autoscaler service instance in the space in which the app you want to scale is deployed, skip this step.
  3. Bind the Autoscaler service instance you created in the previous step to the app you want to scale by running:

    cf bind-service APP-NAME SERVICE-INSTANCE-NAME
    

    Where:

    • APP-NAME is the name of the app you want to scale.
    • SERVICE-INSTANCE-NAME is the name of the Autoscaler service instance in the previous step.
  4. To create a manifest file for Autoscaler that configures an autoscaling rule with RabbitMQ queue depth as its scaling metric, create a YAML file that includes the following configuration parameters:

    ---
    instance_limits:
      min: LOWER-SCALING-LIMIT
      max: UPPER-SCALING-LIMIT
    rules:
    - rule_type: rabbitmq
      rabbitmq_service: RABBITMQ-SERVICE-INSTANCE-NAME
      rule_sub_type: QUEUE-NAME
      threshold:
        min: MINIMUM-QUEUE-DEPTH-THRESHOLD
        max: MAXIMUM-QUEUE-DEPTH-THRESHOLD
    scheduled_limit_changes: []
    

    Where:

    • LOWER-SCALING-LIMIT is the minimum number of instances you want Autoscaler to create for the app.
    • UPPER-SCALING-LIMIT is the maximum number of instances you want Autoscaler to create for the app.
    • (Optional) RABBITMQ-SERVICE-INSTANCE-NAME is the name of the RabbitMQ service instance you want Autoscaler to query. Defining this parameter ensures that Autoscaler does not unnecessarily query other RabbitMQ service instances that are bound to the app.
    • QUEUE-NAME is the name of the queue you want Autoscaler to monitor. This queue must be on a RabbitMQ service instance that is bound to the app you want to scale.
    • MINIMUM-QUEUE-DEPTH-THRESHOLD is the minimum queue depth threshold. If the average queue depth falls below this number of messages, Autoscaler scales the number of app instances down.
    • MAXIMUM-QUEUE-DEPTH-THRESHOLD is the maximum queue depth threshold. If the average queue depth rises above this number of messages, Autoscaler scales the number of app instances up.

      The following example shows an Autoscaler manifest file monitoring the example_queue queue on the example-service RabbitMQ service instance, with a minimum queue depth threshold of 10 and a maximum queue depth threshold of 50:
    ---
    instance_limits:
      min: 10
      max: 100
    rules:
    - rule_type: rabbitmq
      rabbitmq_service: example-service
      rule_sub_type: example_queue
      threshold:
        min: 10
        max: 50
    scheduled_limit_changes: []
    
  5. Apply the autoscaling rule you configured in the previous step to the app you want to scale by running:

    cf configure-autoscaling APP-NAME MANIFEST-FILENAME
    

    Where:

    • APP-NAME is the name of the app.
    • MANIFEST-FILENAME is the filename of the manifest file you created in the previous step. For example, autoscaler.yml.

Configure an Autoscaling Rule Using CLI Commands

To configure an autoscaling rule that defines RabbitMQ queue depth as its scaling metric using CLI commands:

  1. In a terminal window, target the space in which the app you want to scale is deployed by running:

    cf target -o ORG-NAME -s SPACE-NAME
    

    Where:

    • ORG-NAME is the name of the org containing the space in which the app you want to scale is deployed.
    • SPACE-NAME is the name of the space in which the app you want to scale is deployed.
  2. If the space in which the app you want to scale is deployed does not already have a service instance of Autoscaler deployed in it, create an Autoscaler service instance by running:

    cf create-service app-autoscaler PLAN-NAME SERVICE-INSTANCE-NAME
    

    Where:

    • PLAN-NAME is the name of the service plan you want to use for the Autoscaler service instance.
    • SERVICE-INSTANCE-NAME is the name you want to give the Autoscaler service instance. For example, autoscaler.

      If there is already an Autoscaler service instance in the space in which the app you want to scale is deployed, skip this step.
  3. Bind the Autoscaler service instance you created in the previous step to the app you want to scale by running:

    cf bind-service APP-NAME SERVICE-INSTANCE-NAME
    

    Where:

    • APP-NAME is the name of the app you want to scale.
    • SERVICE-INSTANCE-NAME is the name of the Autoscaler service instance in the previous step.
  4. Configure upper and lower scaling limits for the app by running:

    cf update-autoscaling-limits APP-NAME LOWER-SCALING-LIMIT UPPER-SCALING-LIMIT
    

    Where:

    • APP-NAME is the name of the app.
    • LOWER-SCALING-LIMIT is the minimum number of instances you want Autoscaler to create for the app.
    • UPPER-SCALING-LIMIT is the maximum number of instances you want Autoscaler to create for the app.
  5. Allow Autoscaler to begin making scaling decisions for the app by running:

    cf enable-autoscaling APP-NAME
    

    Where APP-NAME is the name of the app.

  6. Create a rabbitmq autoscaling rule by running:

    cf create-autoscaling-rule APP-NAME rabbitmq MINIMUM-QUEUE-DEPTH-THRESHOLD MAXIMUM-QUEUE-DEPTH-THRESHOLD --rabbitmq-instance RABBITMQ-SERVICE-INSTANCE-NAME --subtype QUEUE-NAME
    

    Where:

    • APP-NAME is the name of the app for which you want to create an autoscaling rule.
    • MINIMUM-QUEUE-DEPTH-THRESHOLD is the minimum queue depth threshold. If the average queue depth falls below this number of messages, Autoscaler scales the number of app instances down.
    • MAXIMUM-QUEUE-DEPTH-THRESHOLD is the maximum queue depth threshold. If the average queue depth rises above this number of messages, Autoscaler scales the number of app instances up.
    • (Optional) RABBITMQ-SERVICE-INSTANCE-NAME is the name of the RabbitMQ service instance you want Autoscaler to query. Defining this parameter ensures that Autoscaler does not unnecessarily query other RabbitMQ service instances that are bound to the app.
    • QUEUE-NAME is the name of the queue you want Autoscaler to monitor. This queue must be on a RabbitMQ service instance that is bound to the app you want to scale.

      The following example command configures a rabbitmq autoscaling rule for the example-app app that monitors the example_queue queue on the example-service RabbitMQ service instance, with a minimum queue depth threshold of 10 and a maximum queue depth threshold of 50 milliseconds:
    cf create-autoscaling-rule example-app rabbitmq 10 50 --rabbitmq-instance example-service --subtype example_queue
    

Configure RabbitMQ Queue Depth as the Scaling Metric for an App Through Apps Manager

To configure Autoscaler to use RabbitMQ queue depth as the scaling metric for an app using Apps Manager:

  1. Log in to Apps Manager. For more information, see Logging In to Apps Manager.

  2. Select the org that contains the space in which the app you want to scale is deployed.

  3. Select the space in which the app you want to scale is deployed.

  4. Under Under Processes and Instances, click Manage Autoscaling. The Manage Autoscaling window appears.

  5. Next to Scaling Rules, click Edit. The Edit Scaling Rules window appears.

  6. Click Add rule. The Select type dropdown appears.

  7. From the Select type dropdown, select RabbitMQ. More configuration fields appear below.

  8. For Scale down if less than, enter the minimum queue depth threshold. If the average queue depth falls below this number of messages, Autoscaler scales the number of app instances down.

  9. For Scale up if more than, enter the maximum queue depth threshold. If the average queue depth rises above this number of messages, Autoscaler scales the number of app instances up.

  10. For Queue name, enter the name of the queue you want Autoscaler to monitor. This queue must be on a RabbitMQ service instance that is bound to the app you want to scale.

  11. Click Save.

Use an External RabbitMQ Cluster

If your RabbitMQ queue is hosted on an external RabbitMQ cluster, you can create a user-provided service that represents the external RabbitMQ cluster. Autoscaler can query this user-provided service to retrieve queue metadata.

To use a user-provided service for your external RabbitMQ cluster:

  1. Activate the RabbitMQ management plug-in on your RabbitMQ cluster by following the procedures in the RabbitMQ documentation. This allows Autoscaler to use the RabbitMQ HTTP Management API to discover the depth of the queues in the RabbitMQ cluster.

  2. In a terminal window, run:

    cf create-user-provided-service SERVICE-NAME -p '{"uri":"amqp://USERNAME:PASSWORD@RABBITMQ-FQDN","http_api_uri":"http://USERNAME:PASSWORD@RABBITMQ-FQDN:15672/api/","vhost":"VHOST"}' \ -t rabbitmq
    

    Where:

    • SERVICE-NAME is the name you want to give the user-provided service.
    • USERNAME is the username you want to configure for the external RabbitMQ cluster.
    • PASSWORD is the password you want to configure for the external RabbitMQ cluster.
    • RABBITMQ-FQDN is the fully-qualified domain name (FQDN) of the RabbitMQ cluster.
    • VHOST is the RabbitMQ virtual host you want the user-provided service to use.

      Warning: If you do not define the "vhost" parameter, Autoscaler requests queue metadata for all queues on the RabbitMQ cluster. This is a very expensive operation. For more information about RabbitMQ virtual hosts, see the RabbitMQ documentation.

  3. Bind the user-provided service to the app you want to scale by running:

    cf bind-service APP-NAME SERVICE-NAME
    

    Where:

    • APP-NAME is the name of the app you want to scale.
    • SERVICE-NAME is the name of the user-provided service you created in the previous step.

Note: Depending on the app to which you bind the user-provided service, you may need to configure the VCAP_SERVICES environment variable to allow the app to access the user-provided service. For more information, see the VMware Tanzu RabbitMQ for VMs documentation.

After you bind the user-provided service to the app, configure Autoscaler to use RabbitMQ queue depth as the scaling metric for an app by following one of the procedures in Configuring RabbitMQ Queue Depth as the Scaling Metric for an App Through the cf CLI or Configuring RabbitMQ Queue Depth as the Scaling Metric for an App Through Apps Manager above.

Reviewing Autoscaling Events for Changes in RabbitMQ Queue Depth

When Autoscaler scales the number of app instances up after the RabbitMQ queue depth metric increases above the maximum queue depth threshold, Autoscaler records an autoscaling event.

You can monitor the autoscaling events that Autoscaler records for changes in RabbitMQ queue depth in the following ways:

Review Autoscaling Events for Changes in RabbitMQ Queue Depth Through the cf CLI

To review the autoscaling events that Autoscaler records for changes in RabbitMQ queue depth through the cf CLI:

  1. In a terminal window, run:

    cf autoscaling-events APP-NAME
    

    Where APP-NAME is the name of the app for which you want to review autoscaling events.

    If Autoscaler has scaled the number of app instances up due to increases in the RabbitMQ queue depth metric, the above command returns output that contains autoscaling events similar to the following example:

    Time                   Description
    2022-05-24T20:47:34Z   Scaled up from 10 to 11 instances. The RabbitMQ message depth in example_queue is currently at an average of 50.35, which is above the maximum threshold of 50.00.
    

Review Autoscaling Events for Changes in RabbitMQ Queue Depth Through Apps Manager

To review the autoscaling events that Autoscaler records for changes in RabbitMQ queue depth through Apps Manager:

  1. Log in to Apps Manager. For more information, see Logging In to Apps Manager.

  2. Select the org that contains the space in which the app you want to scale is deployed.

  3. Select the space in which the app you want to scale is deployed.

  4. Under Under Processes and Instances, click Manage Autoscaling.

  5. Under Event History, click View More. A list of autoscaling events appears. If Autoscaler has scaled the number of app instances up due to increases in the RabbitMQ queue depth metric, the list of autoscaling events includes events similar to the following example:

    Scaled up from 10 to 11 instances. The RabbitMQ message depth in example_queue is currently at an average of 50.35, which is above the maximum threshold of 50.00.
    

Special Considerations for Using RabbitMQ Queue Depth as a Scaling Metric

This section describes use cases that may complicate or prevent you from configuring RabbitMQ queue depth as the scaling metric for an app. For more information, see the sections below:

RabbitMQ Service Binding Credentials in Runtime CredHub

In order to receive RabbitMQ queue depth metrics, Autoscaler must be able to retrieve service binding credentials for the RabbitMQ HTTP Management API from Cloud Controller. However, when a TAS for VMs deployment is configured to store credentials in Runtime CredHub, Autoscaler cannot retrieve service binding credentials from Cloud Controller.

For more information, see Autoscaler Does Not Support Runtime CredHub in Troubleshooting App Autoscaler.

Multiple RabbitMQ Service Instances

If an app is bound to more than one RabbitMQ service instance, and the autoscaling rules you configure for the app do not specify the name of a particular RabbitMQ service instance name, Autoscaler queries all RabbitMQ service instances that are bound to the app for the specified queue. To avoid this, specify the name of the RabbitMQ service instance that hosts the queue you want to monitor when you configure autoscaling rules for the app.