LATEST VERSION: 1.11.7 - CHANGELOG
RabbitMQ for PCF v1.11.7

Installing and Configuring the
Pre-Provisioned Service

This topic provides instructions to Pivotal Cloud Foundry (PCF) operators about how to install, configure, and deploy the RabbitMQ for PCF tile to provide a pre-provisioned service.

Note: For instructions about how to install, configure, and deploy the RabbitMQ for PCF tile as an on-demand service, see Installing and Configuring RabbitMQ for PCF as an On-Demand Service.

Download and Install the Tile

  1. Download the product file from Pivotal Network.

  2. Navigate to the Ops Manager Installation Dashboard and click Import a Product to upload the product file.

  3. Under the Import a Product button, click + next to the version number of RabbitMQ for PCF. This adds the tile to your staging area.

  4. Click the newly added RabbitMQ for PCF tile. This lets you begin configuring the tile. The installation is complete when you apply the changes from the configuration.

Configure Pre-Provisioned RabbitMQ for PCF

The configuration screen below appears when you click the RabbitMQ for PCF tile in Ops Manager. An orange circle beside a tab indicates that you must complete a configuration in the tab. A green checkmark indicates that the tab is preconfigured and you may optionally change its settings.

Rabbit Tile Configuration

Which Settings Tabs to Configure for the Pre-Provisioned Service

Configure the following tabs for the pre-provisioned service:

RabbitMQ Settings TabInstructions
Assign AZs and NetworksAssign AZs and Networks
RabbitMQRabbitMQ
SyslogSyslog
Dedicated Instance: Single Node PlanDedicated Instance: Single Node Plan
ErrandsErrands
StemcellStemcell

Assign AZs and Networks

Follow the steps below to configure the AZs and networks.

  1. In the Settings screen, click Assign AZs and Networks.

    Important: You cannot change the regions or networks after you have clicked Apply Changes in the final step below.

  2. Configure the fields on the Assign AZs and Networks as follows. All fields are required, though some do not apply to the pre-provisioned service.

    Required FieldsInstructions
    Place singleton jobs in Select a region. This selection only affects the on-demand service.
    Balance other jobs in Select additional region(s). This selection only affects the pre-provisioned service.
    Network Select a network for the RabbitMQ Broker.

    This should be a separate network from the one you select for Service Network. This network is represented by the Default Network, described in Default Network and Service Network. Typically, you select the network used for the Pivotal Application Service (PAS) or Elastic Runtime components.
    Service Network Select a network. This selection only affects the on-demand service.

    WARNING: Changing the Network after you have configured it, or changing the IP configuration, results in a failed deployment. For more information, see Changing Network or IP Addresses Results in a Failed Deployment.

  3. Click Save.

RabbitMQ

To configure the following sections in the RabbitMQ tab, in the Settings screen, click RabbitMQ.

RabbitMQ Admin User Credentials

Enter an admin username and password for RabbitMQ. This grants you full admin access to the RabbitMQ Management UI.

Enter the desired credentials in this section of the RabbitMQ tab:

Config credentials

Note: To rotate your administrator credentials, enter a new username and password, save your options, and redeploy by returning to the Ops Manager Installation Dashboard and clicking Apply Changes.

Plugins

Choose which plugins you want to enable in this section of the RabbitMQ tab.

You must leave the rabbitmq_management plugin enabled for this product to work.

Config plugins

For more information about RabbitMQ plugins, see the RabbitMQ documentation.

HAProxy Ports

Enter the ports HAProxy should load balance to the RabbitMQ nodes in this section of the RabbitMQ tab:

Config haproxy

  • All the default ports of all the available plugins are load-balanced by default. However, if you install extra protocol plugins, or provide a custom configuration that changes the ports RabbitMQ listens on, then you must update the list of load-balanced ports.

  • You must leave the management plugin listening on port 15672 and load balance that port.

  • If you change the topology of your RabbitMQ cluster, the HAProxy is automatically reconfigured during the deployment.

SSL

(Optional) Provide SSL certificates and keys for use by the RabbitMQ cluster in this section of the RabbitMQ tab:

Config ssl

  • SSL is simultaneously provided on the AMQPS port (5671) and the management port (15672). No other plugins are automatically configured for use with SSL.

  • If you provide SSL keys and certificates non-SSL support is disabled.

  • SSL settings are applied equally across all VMs in the cluster.

For more information about SSL support, see the RabbitMQ documentation.

(Optional) Provide an Erlang cookie to be used by the cluster in this section of the RabbitMQ tab. This is useful if you want to connect directly to the RabbitMQ cluster, for example with rabbitmqctl, or to connect other VMs running Erlang.

Config erlang

There are two known issues associated with the Erlang cookie when you do the following:

Cluster Scaling Known Issue

If you have not set the Erlang cookie and you want to scale out your cluster size without downtime, follow these steps:

  1. Follow the steps in one of the links below, up to and including the section Log in to the BOSH Director:

  2. Run one of the following commands depending on your Ops Manager version:

    • For Ops Manager v1.10 or earlier: bosh ssh rabbitmq-server/0
    • For Ops Manager v1.11 or later: bosh2 ssh rabbitmq-server/0
  3. Run the following commands:

    sudo -i
    echo $(cat /var/vcap/store/rabbitmq/.erlang.cookie)
    

  4. Paste the value returned from the last command into the Erlang cookie field in the RabbitMQ tab. This field is shown above.

  5. To increase the size of your cluster, navigate to the Resource Config tab and, in the first row, raise the value for the number of Instances of the RabbitMQ node.

  6. Return to the Ops Manager Installation Dashboard, and click Apply Changes.

Note: BOSH tells you that the cookie has changed—this is because the default value in the manifest is empty, which results in an auto-generated cookie. However, the value of the cookie on the server remains the same, so the known issue below does not apply.

Changing the Erlang cookie value requires cluster downtime. Pivotal strongly recommends that you do not change anything else during this time, because it is possible for the configuration to be inconsistently applied during this process.

The deployment may fail after this process. If so, redeploying fixes the issue.

RabbitMQ Configuration

(Optional) Provide a full rabbitmq.config file by pasting its contents in the RabbitMQ configuration field in the RabbitMQ tab. This rabbitmq.config file is then provided to all the nodes in the cluster.

Config rabbitmq

The input in this field must be Base64 encoded.

For example, suppose you want to configure the rates_mode of the rabbitmq_management stats below:

[
  {rabbitmq_management, [
    {rates_mode, detailed}
  ]}
].
  1. Encode the file into Base64:

    WwogIHtyYWJiaXRtcV9tYW5hZ2VtZW50LCBbCiAgICB7cmF0ZXNfbW9kZSwgZGV0YWlsZWR9CiAgXX0KXS4K
    
  2. Paste the above into the RabbitMQ configuration field:

    Config rabbitmq file

You can see an example rabbitmq.config file here. For more information about the RabbitMQ configuration, see the RabbitMQ documentation.

TLS Support

Configure TLS in this section of the RabbitMQ tab:

Config tls

  • TLS v1.0 is disabled by default, due to security issues.

  • TLS v1.1 and v1.2 are enabled by default and can be turned on and off.

External Load Balancer

(Optional) Enter a DNS name or IP address of an external load balancer to be returned in the binding credentials (VCAP_SERVICES) to application developers. Enter this in this section of the RabbitMQ tab:

Config elb

If you configure an external load balancer, to avoid an unnecessary VM deployment, in the Resource Config tab set the HAProxy for RabbitMQ instance count to 0.

Metrics Polling Interval

The metrics polling interval is set in this section of the RabbitMQ tab:

Metrics polling

The default setting is 30 seconds for all deployed components. Pivotal recommends that you do not change this interval. In order to avoid overwhelming components, do not set this below 10 seconds.

Changing this setting affects all deployed instances.

Disk Free Alarm Limit

Choose how much disk space RabbitMQ attempts to keep free at any given time in this section of the RabbitMQ tab:

Disk alarm threshold

RabbitMQ periodically checks if there is sufficient free space on disk. If there is not, RabbitMQ temporarily stops accepting new messages. This gives your apps time to consume existing messages, and thus free up some disk space. The RabbitMQ tile provides four options for this value:

  • 50MB is the minimum value. (Not Recommended)

    Selecting 50MB is not recommended and can cause data loss. For more information, see Dangers of Setting This Value Too Low and When to Use the 50MB Value below.

  • 100% Memory ensures that at the time when RabbitMQ checks the available disk, there must be enough space for RabbitMQ to page all memory-based messages out to disk.

  • 150% Memory is recommended. This is because it is possible that in between disk-space checks, RabbitMQ may:

    • Write persistent messages to disk (using up some disk space).
    • Accept more memory-based messages into various queues.
    • Page all memory-based messages to disk.

    In the above situations, RabbitMQ might require more free disk than it has memory.

  • 200% Memory is a conservative value used when the operator wants higher confidence that RabbitMQ never runs out of disk space.

For more information about disk alarms, see the RabbitMQ documentation.

Dangers of Setting This Value Too Low

If the disk of a given RabbitMQ node completely fills while RabbitMQ is running, that node crashes. This can lead to data loss, and loss of availability.

RabbitMQ reserves the right to page any and all messages in memory (even transient messages) to disk at any time. You must set your Disk free alarm limit high enough to ensure that RabbitMQ always has at least enough space to do this.

Disadvantages of Setting This Value Too High

If you set your Disk free alarm limit to a value larger than the size of your persistent disk, then RabbitMQ is not able to free up enough disk space to accept new messages. Ensure that you have a large enough disk to persist all the messages you intend to persist while also leaving enough space free to satisfy the Disk free alarm limit that you choose.

When to Use the 50MB Value

Pivotal does not recommend using this value in production. However, if you are experimenting with a development environment you may want to use a small disk to keep down costs, though this increases the possibility that RabbitMQ may crash and lose data.

Syslog

To enable monitoring for RabbitMQ for PCF, operators forward the syslog by designating an external syslog endpoint for RabbitMQ component log messages. This endpoint serves as the input to a monitoring platform such as Datadog, Papertrail, or SumoLogic.

To specify the destination for RabbitMQ for PCF log messages, do the following:

  1. From the Ops Manager Installation Dashboard, click the RabbitMQ tile.
  2. In the RabbitMQ tile, click the Settings tab. Ops Manager Syslog Forwarding Configuration
  3. Click Syslog.
  4. Configure the fields on the Syslog pane as follows:

    Option Description
    Syslog address IP or DNS address of the syslog server
    Syslog port Port of the syslog server
    Transport protocol Transport protocol of the syslog server. One of udp, tcp, relp.
    Format for logs Format for logs. Pivotal recommends RFC 5424, but Legacy Format can be used for compatibility reasons.
    Enable TLS Enable TLS to the syslog server.
    Permitted Peer If there are several peer servers that can respond to remote syslog connections, then you may provide a wildcard in the domain, such as *.example.com.
    Custom CA Certificate If the server certificate is not signed by a known authority, for example, an internal syslog server, provide the CA certificate of the log management service endpoint.

  5. Click Save.

  6. Return to the Ops Manager Installation Dashboard and click Apply Changes to redeploy with the changes.

    Dedicated Instance: Single Node Plan

This tab only applies to the on-demand service. However, you must complete the fields on this tab even if you are not using the on-demand service. Therefore, if you are not using the on-demand service:

  • Select or enter any values in the required fields, select the Acknowledge checkbox, and click Save.

For information on configuring the on-demand service, see Installing and Configuring the On-Demand Service.

Errands

(Optional) In the Errands tab, choose the defaults for when errands run.

Errands can be thought of as tasks. For example, when deploying or updating RabbitMQ for PCF, Ops Manager can optionally run a series of post-deploy errands. An example is the Smoke Tests errand, which checks the health of the RabbitMQ cluster after a deploy or upgrade.

You can decide whether to run post-deploy errands by toggling them on or off before you click Apply Changes to update a configuration in the Ops Manager Installation Dashboard:

Configuring One-time Errands

This is a one-time action before an update. You can change the above defaults in the Errands tab, as well as the defaults for pre-delete errands.

Important: In RabbitMQ for PCF v1.9.0 and later, all post-deploy errands are on by default. Pivotal recommends keeping these defaults, because the smoke tests can encounter unexpected issues, and on-demand instances of RabbitMQ for PCF may fall behind if the Upgrade All Service Instances errand is not on by default.

For more information on errand run rules, see Errand Run Rules.

Post-Deploy Errands

Errand Description
Broker Registrar Makes the pre-provisioned RabbitMQ service plans available in the Marketplace
Smoke Tests Checks that a pre-provisioned RabbitMQ service instance can be bound to a Cloud Foundry app, and that the app can publish and subscribe to a RabbitMQ cluster. See Smoke Test Process below.
Register On-Demand Service Broker Makes the on-demand RabbitMQ service plans available in the Marketplace. If you change the Service Plan Configuration, you must run this errand in order for the changes to be reflected in the Marketplace.
Dedicated Instance Smoke Tests Checks that on-demand RabbitMQ service instances can be bound to a Cloud Foundry app, and that the app can publish and subscribe to a RabbitMQ cluster. See Dedicated Instance Smoke Test Process.
Upgrade All Service Instances On-demand instances are updated and redeployed if there are changes to the Dedicated Instance settings or the tile is upgraded. If this errand is set to Off or When Changed, updates to Dedicated Instance settings will not be applied to existing service instances. Pivotal strongly recommends that this errand is configured to always run.

Smoke Test Process

Smoke tests run as a post-deployment errand. The smoke tests perform the following for each available service plan:

  1. Target the org system and create a cf space to run the tests.
  2. Deploy an instance of the CF RabbitMQ Example App in this cf space.
  3. Create a RabbitMQ service instance and bind it to the CF RabbitMQ Example App.
  4. Check that the CF RabbitMQ Example App can write to, and read from, the RabbitMQ service instance.
  5. Clean up the deployed Example App and all its service bindings.
  6. Delete the cf space created for the tests.

For more information, see Errands.

Pre-Delete Errands

Pre-delete errands run after an operator chooses to delete a product in the Ops Manager Installation Dashboard, but before Ops Manager finishes deleting the product.

Errand Description
Broker Deregistrar Removes the pre-provisioned RabbitMQ service from the Marketplace and deletes all associated service instances and bindings
Delete All Service Instances Unbinds and deletes existing dedicated service instances. The duration of this errand depends on the number of deployed on-demand instances.
Deregister On-Demand Service Broker Removes the on-demand RabbitMQ service from the Marketplace

Stemcell

  1. Click Stemcell.

  2. If prompted, import a new stemcell version. For more information, see the information about importing the stemcell for your IaaS: AWS, Azure, GCP, or vSphere.

Apply Configuration and Complete the Installation

Return to the Ops Manager Installation Dashboard and click Apply Changes to complete the installation of RabbitMQ for PCF.

Other Configuration Topics

Connecting to a Highly Available RabbitMQ Cluster

The RabbitMQ tile, allows for a highly available cluster through multiple HAProxy nodes. The hostnames, uris and hosts properties have been added and should be used in preference over the equivalent singular properties. The singular properties are maintained for backwards compatibility and always contain the first value from the equivalent plural property. The singular properties will eventually be deprecated.

For example, with two HAProxy jobs deployed, the following properties will be present:

"hostname": "10.0.0.41",
"hostnames": [
    "10.0.0.41",
    "10.0.0.51"]

Port to protocol mappings

  • 15672 = Management dashboard
  • 5672 = RabbitMQ
  • 5671 = RabbitMQ SSL
  • 1883 = MQTT
  • 8883 = MQTT SSL
  • 61613 = STOMP
  • 61614 = STOMP SSL
  • 15674 = Web STOMP
  • 4567 = RabbitMQ Service Broker
  • 3457 - 3459 = CF Loggregator

Security Groups

To enable access to the RabbitMQ tile service, you must ensure your security group allows access to the HAProxy and RabbitMQ Service Broker VMs configured in your deployment. You can obtain the IP addresses for these from the Ops Manager Status page for the RabbitMQ tile. Ensure the following ports are enabled for those VMs:

  • 15672
  • 5672
  • 5671
  • 1883
  • 8883
  • 61613
  • 61614
  • 15674
  • 4567
  • 3457 - 3459

The following is a template for configuring your Cloud Foundry security groups: [ {"protocol":"tcp","destination":"<haproxy-node-IP-addresses>","ports":"5671,5672,1883,8883,61613,61614,15672,15674"}, {"protocol":"tcp","destination":"<service-broker-node-IP-addresses>","ports":"4567"} ]

Application Security Groups

To allow this service to have network access, you must create Application Security Groups (ASGs).

Note: The service is unusable without Application Security Groups.

Application Container Network Connections

Application containers that use instances of the RabbitMQ service require the following outbound network connections:

Destination Ports Protocol Reason
HAProxy IPs 5672 tcp Application containers using AMQP
HAProxy IPs 5671 tcp Application containers using AMQP over SSL
HAProxy IPs 1883 tcp Application containers using MQTT
HAProxy IPs 8883 tcp Application containers using MQTT over SSL
HAProxy IPs 61613 tcp Application containers using STOMP
HAProxy IPs 61614 tcp Application containers using STOMP over SSL
HAProxy IPs 61613 tcp Application containers using Web STOMP

Create an ASG named rabbitmq-app-containers with the above configuration and bind it to either:

  • The appropriate space
  • The default-running ASG set if you want to provide access to all started apps. Then restart your apps.

If you are using an external load balancer, or have more than one IP address for HAProxy, you must also create egress rules for these. For example:

[
  {
      "ports": "5671-5672",
      "protocol": "tcp",
      "destination": "10.10.10.10/32"
  }
]

Assigned IPs

RabbitMQ for PCF does not support changing the IP addresses which have been assigned to the RabbitMQ deployments. For example, you cannot change the subnet into which the RabbitMQ cluster was originally provisioned. Doing so causes the deployment to fail. For more information, see Changing Network or IP Addresses Results in a Failed Deployment.

Preserving Dynamically Assigned IPs

You cannot switch from dynamically assigned IP addresses to a different set of static IP addresses. However, you can configure Ops Manager so the current set of dynamically assigned IP addresses always continue to be used. This may be useful when upgrading.

To do this, follow these steps:

  1. Go to the Status page in the RabbitMQ tile.
  2. Take note of the IP addresses for the RabbitMQ Server and HAProxy for RabbitMQ jobs, in the order nodes appear in the UI.
  3. Go to the Settings page, and click Networking.
  4. Enter the IP addresses you got from the Status page as a comma-separated list.


    Config static ip

  5. Click Save.

RabbitMQ Server Settings that Cannot be Overwritten

In all cases:

  • rabbit halt_on_upgrade_failure false
  • rabbitmq_mqtt subscription_ttl 1800000
  • log_levels [{connection,info}]
  • halt_on_upgrade_failure false
  • {rabbit, [ {collect_statistics_interval, 60000}] }
  • {rabbitmq_management, [ {rates_mode, none}] }

When SSL is enabled:

  • rabbit tcp_listeners []
  • rabbit ssl_listeners [5671]
  • rabbitmq_management listener [{port,15672},{ssl,false}]
  • rabbitmq_mqtt ssl_listeners [8883]
  • rabbitmq_stomp ssl_listeners [61614]
Create a pull request or raise an issue on the source for this page in GitHub