Installing and Configuring the
Pre-Provisioned Service

Warning: RabbitMQ for PCF v1.14 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic provides instructions to Pivotal Cloud Foundry (PCF) operators about how to install, configure, and deploy the RabbitMQ for PCF tile to provide a pre-provisioned service.

The RabbitMQ open source product provides additional documentation. For more information about getting started with RabbitMQ and ensuring production readiness, see the Production Checklist in the RabbitMQ Documentation.

Role-Based Access in Ops Manager

Ops Manager administrators can use Role-Based Access Control (RBAC) to manage which operators can make deployment changes, view credentials, and manage user roles in Ops Manager. Therefore, your role permissions might not allow you to perform every procedure in this operator guide.

For more information about roles in Ops Manager, see Understand Roles in Ops Manager.

Download and Install the Tile

Note: Before using RabbitMQ for PCF v1.14 and later, you must update any BOSH add-ons to support Xenial stemcells. See Update Add-ons to Run with Xenial Stemcell.

1. Download the product file from Pivotal Network.

2. Navigate to the Ops Manager Installation Dashboard and click Import a Product to upload the product file.

3. Under the Import a Product button, click + next to the version number of RabbitMQ for PCF. This adds the tile to your staging area.

4. Click the newly added RabbitMQ for PCF tile. This lets you begin configuring the tile. The installation is complete when you apply the changes from the configuration.

Configure Pre-Provisioned RabbitMQ for PCF

The configuration screen below appears when you click the RabbitMQ for PCF tile in Ops Manager. An orange circle beside a tab indicates that you must complete a configuration in the tab. A green checkmark indicates that the tab is preconfigured and you may optionally change its settings. An orange circle indicates that you are required to configure fields on the tab before you can deploy the tile.

Assign AZs and Networks

Follow the steps below to configure the AZs and networks.

1. In the Settings screen, click Assign AZs and Networks.

   Warning: You cannot change the regions or networks after you have clicked Apply Changes in the final step below.

2. Configure the fields on the Assign AZs and Networks as follows. All fields are required, though some do not apply to the pre-provisioned service.

   Required Fields	Instructions
   Place singleton jobs in	Select a region. This selection only affects the on-demand service.
   Balance other jobs in	Select additional region(s). This selection only affects the pre-provisioned service.
   Network	Select a network for the RabbitMQ Broker. This should be a separate network from the one you select for Service Network. This network is represented by the Default Network, described in Default Network and Service Network. Typically, you select the network used for Pivotal Application Service (PAS) components.
   Service Network	Select a network. This selection only affects the on-demand service.

   WARNING: Changing the Network after you have configured it, or changing the IP configuration, results in a failed deployment. For more information, see Changing Network or IP Addresses Results in a Failed Deployment.

3. Click Save.

Pre-Provisioned RabbitMQ

To configure the following sections in the Pre-Provisioned RabbitMQ tab, in the Settings screen, click Pre-Provisioned RabbitMQ.

RabbitMQ Admin User Credentials

In the RabbitMQ admin user credentials field of the RabbitMQ pane, enter an admin username and password:

You can use a combination of upper or lowercase alphanumerics and supported special characters: -=_+":;/?.><,~.

Note: You cannot use back quote ` or single quote '.

This grants you full admin access to the RabbitMQ Management UI.

Note: To rotate your administrator credentials, enter a new username and password, save your options, and redeploy by returning to the Ops Manager Installation Dashboard and clicking Apply Changes.

Plugins

Choose which plugins you want to enable in this section of the Pre-Provisioned RabbitMQ tab.

You must leave the rabbitmq_management plugin enabled for this product to work.

For more information about RabbitMQ plugins, see the RabbitMQ documentation.

Erlang Cookie

(Optional) Provide an Erlang cookie to be used by the cluster in this section of the Pre-Provisioned RabbitMQ tab. This is useful if you want to connect directly to the RabbitMQ cluster, for example with rabbitmqctl, or to connect other VMs running Erlang.

Note: Leaving this field blank is a security risk. See Security Issue with the Tile Generated Erlang Cookie below.

Known Issues with Erlang Cookie

There are three known issues associated with the Erlang cookie:

• Security Issue with the Tile Generated Erlang Cookie
• Cluster Scaling Known Issue
• Changing the Erlang Cookie Value Known Issue

Security Issue with the Tile Generated Erlang Cookie

If you leave the Erlang cookie field blank, the tile generates the cookie in a way that can be reverse-engineered. For more information about this security issue, see CVE-2018-1279: RabbitMQ cluster compromise due to deterministically generated cookie.

To avoid this issue, set the Erlang cookie to a secure password value. This requires cluster downtime, see Changing the Erlang Cookie Value Known Issue below.

Cluster Scaling Known Issue

If you have not set the Erlang cookie and you want to scale out your cluster size without downtime, follow these steps:

1. Follow the steps in the link below, up to and including the section Log in to the BOSH Director: Advanced Troubleshooting with the BOSH CLI

2. Run the following command:
   bosh ssh rabbitmq-server/0

3. Run the following commands:

   sudo -i
   echo $(cat /var/vcap/store/rabbitmq/.erlang.cookie)
   

4. Paste the value returned from the last command into the Erlang cookie field in the Pre-Provisioned RabbitMQ tab. This field is shown above.

5. To increase the size of your cluster, navigate to the Resource Config tab and, in the first row, raise the value for the number of Instances of the RabbitMQ node.

6. Return to the Ops Manager Installation Dashboard.

7. If you are using Ops Manager v2.3 or later, click Review Pending Changes. For more information about this Ops Manager page, see Reviewing Pending Product Changes.

8. Click Apply Changes.

Note: BOSH tells you that the cookie has changed—this is because the default value in the manifest is empty, which results in an auto-generated cookie. However, the value of the cookie on the server remains the same, so the known issue below does not apply.

Changing the Erlang Cookie Value Known Issue

Changing the Erlang cookie value requires cluster downtime. Pivotal strongly recommends that you do not change anything else during this time, because it is possible for the configuration to be inconsistently applied during this process.

The deployment might fail after this process. If so, redeploying fixes the issue.

Enable Custom Policy on New Instances

You can specify a custom RabbitMQ policy in this part of the Pre-Provisioned RabbitMQ tab. For information and instructions, see Setting Default Policies for the RabbitMQ Service.

External Load Balancer

(Optional) Enter a DNS name or IP address of an external load balancer to be returned in the binding credentials (VCAP_SERVICES) to app developers. Enter this in this section of the Pre-Provisioned RabbitMQ tab:

If you configure an external load balancer, to avoid an unnecessary VM deployment, in the Resource Config tab set the HAProxy for RabbitMQ instance count to 0.

HAProxy Ports

Enter the ports HAProxy should load balance to the RabbitMQ nodes in this section of the Pre-Provisioned RabbitMQ tab:

• Enter a comma-separated list of ports that are proxied to RabbitMQ nodes. If you enable other protocol plugins or have a custom configuration that changes the ports that RabbitMQ listens on, you can extend this list. The list defaults to the following:

  • 15672 - Management
  • 5672 - AMQP
  • 5671 - AMQP+SSL
  • 1883 - MQTT
  • 8883 - MQTT+SSL
  • 61613 - STOMP
  • 61614 - STOMP+SSL
  • 15674 - WebSTOMP

• You must leave the management plugin listening on port 15672 and load balance that port.

• If you change the topology of your RabbitMQ cluster, the HAProxy is automatically reconfigured during the deployment.

SSL

(Optional) Provide SSL certificates and keys for use by the RabbitMQ cluster in this section of the Pre-Provisioned RabbitMQ tab:

• SSL is simultaneously provided for AMQPS, STOMP and MQTT. No other plugins are automatically configured for use with SSL.

• If you provide SSL keys and certificates, non-SSL support is disabled. If you previously deployed this service without SSL support and have apps connected to the service, these apps lose their connections and must reconnect using SSL.

• SSL settings are applied equally across all VMs in the cluster.

• You can provide more then one CA certificate.

For more information about SSL support, see the RabbitMQ documentation.

TLS Support

Configure TLS in this section of the Pre-Provisioned RabbitMQ tab:

• TLS v1.0 is disabled by default, due to security issues.

• TLS v1.1 and v1.2 are enabled by default and can be turned on and off.

RabbitMQ Configuration

(Optional) Provide a full rabbitmq.config file by pasting its contents in the RabbitMQ configuration field in the Pre-Provisioned RabbitMQ tab. This rabbitmq.config file is then provided to all the nodes in the cluster.

The input in this field must be Base64 encoded.

For example, suppose you want to configure the rates_mode of the rabbitmq_management stats below:

[
  {rabbitmq_management, [
    {rates_mode, detailed}
  ]}
].

1. Encode the file into Base64:

   WwogIHtyYWJiaXRtcV9tYW5hZ2VtZW50LCBbCiAgICB7cmF0ZXNfbW9kZSwgZGV0YWlsZWR9CiAgXX0KXS4K
   

2. Paste the above into the RabbitMQ configuration field:

   

You can see an example rabbitmq.config file here. For more information about the RabbitMQ configuration, see the RabbitMQ documentation.

Select the Network Partition Behavior of the RabbitMQ Cluster

In this part of the Pre-Provisioned RabbitMQ tab, you can select one of two behaviors to occur in the event of a network partition.

1. In the Select the network partition behavior of the RabbitMQ cluster field, choose the desired behavior: pause_minority or autoheal.

For more information about these options and on RabbitMQ clusters and network partitions, see the RabbitMQ documentation.

For production purposes, Pivotal recommends that customers have at least three RabbitMQ server nodes and two HAProxies spread across low latency availability zones.

Disk Free Alarm Limit

Choose how much disk space RabbitMQ attempts to keep free at any given time in this section of the Pre-Provisioned RabbitMQ tab:

RabbitMQ periodically checks if there is sufficient free space on disk. If there is not, RabbitMQ temporarily stops accepting new messages. This gives your apps time to consume existing messages, and thus free up some disk space. The RabbitMQ tile provides four options for this value:

• 50MB is the minimum value. (Not Recommended)

  Selecting 50MB is not recommended and can cause data loss. For more information, see Dangers of Setting This Value Too Low and When to Use the 50MB Value below.

• 100% Memory ensures that at the time when RabbitMQ checks the available disk, there must be enough space for RabbitMQ to page all memory-based messages out to disk.

• 150% Memory is recommended. This is because it is possible that in between disk-space checks, RabbitMQ may:

  • Write persistent messages to disk (using up some disk space).
  • Accept more memory-based messages into various queues.
  • Page all memory-based messages to disk.

  In the above situations, RabbitMQ might require more free disk than it has memory.

• 200% Memory is a conservative value used when the operator wants higher confidence that RabbitMQ never runs out of disk space.

For more information about disk alarms, see the RabbitMQ documentation.

Dangers of Setting This Value Too Low

If the disk of a given RabbitMQ node completely fills while RabbitMQ is running, that node crashes. This can lead to data loss, and loss of availability.

RabbitMQ reserves the right to page any and all messages in memory (even transient messages) to disk at any time. You must set your Disk free alarm limit high enough to ensure that RabbitMQ always has at least enough space to do this.

Disadvantages of Setting This Value Too High

If you set your Disk free alarm limit to a value larger than the size of your persistent disk, then RabbitMQ is not able to free up enough disk space to accept new messages. Ensure that you have a large enough disk to persist all the messages you intend to persist while also leaving enough space free to satisfy the Disk free alarm limit that you choose.

When to Use the 50MB Value

Pivotal does not recommend using this value in production. However, if you are experimenting with a development environment you might want to use a small disk to keep down costs, though this increases the possibility that RabbitMQ crashes and loses data.

Specify Static IP Addresses

To specify static IP addresses, do the following:

1. In the Pre-Provisioned RabbitMQ tab, configure the following fields:
   

   • Pre-Provisioned HAProxy Static IPs: Enter a comma-delimited list of IP addresses grouped in the order of AZs configured. These IP addresses are assigned to your Pre-Provisioned HAproxy nodes.
     
     
   • Pre-Provisioned RabbitMQ Server Static IPs: Enter a comma-delimited list of IP addresses grouped in the order of AZs configured. These IP addresses are assigned to your Pre-Provisioned RabbitMQ Server nodes.
     
     
   • Pre-Provisioned Service Broker Static IP: Enter an IP address. This address must be in the network range of the network name specified in the Network drop-down list in the Assign AZs and Networks tab and must not be in the Service Network. This IP address is assigned to your Pre-Provisioned Service Broker node.

   Note: If any of the above fields are left blank, BOSH allocates an IP address.

   

2. Click Save.

Configure Syslog Forwarding and Metrics Polling Interval

To enable monitoring for RabbitMQ for PCF, operators forward the syslog by designating an external syslog endpoint for RabbitMQ component log messages. This endpoint serves as the input to a monitoring platform such as Datadog, Papertrail, or SumoLogic.

The metrics polling interval determines how often metrics are collected.

To specify the destination for RabbitMQ for PCF log messages, do the following:

1. From the Ops Manager Installation Dashboard, click the RabbitMQ tile.
2. In the RabbitMQ tile, click the Settings tab.
3. Click Syslog and Metrics.

4. Configure the fields on the Syslog pane as follows:

   Field	Description
   Metrics polling interval	The default setting is 30 seconds for all deployed components. Pivotal recommends that you do not change this interval. In order to avoid overwhelming components, do not set this below 10 seconds. Changing this setting affects all deployed instances.
   Syslog address	IP or DNS address of the syslog server
   Syslog port	Port of the syslog server
   Transport protocol	Transport protocol of the syslog server. One of udp, tcp, relp.
   Enable TLS	Enable TLS to the syslog server.
   Permitted Peer	If there are several peer servers that can respond to remote syslog connections, then you may provide a wildcard in the domain, such as *.example.com.
   Custom CA Certificate	If the server certificate is not signed by a known authority, for example, an internal syslog server, provide the CA certificate of the log management service endpoint.

5. Click Save.

6. Return to the Ops Manager Installation Dashboard.

7. If you are using Ops Manager v2.3 or later, click Review Pending Changes. For more information about this Ops Manager page, see Reviewing Pending Product Changes.

8. Click Apply Changes to redeploy with the changes.

Global Settings

Follow the steps below to enable the shareable instances beta feature. Sharing a service instance between spaces, allows apps in different spaces to share databases, messaging queues, and many other types of services. For more information, see Sharing Service Instances (Experimental).

WARNING: This is a beta feature based on an experimental feature in Cloud Foundry. Use the feature at your own risk in non-production environments.

1. Click Global Settings.

2. Select (Beta) Shareable Instances to enable the beta feature for sharing instances.
   

   

3. Click Save.

Dedicated Instance: Single Node Plan

This tab only applies to the on-demand service. However, you must complete the fields on this tab even if you are not using the on-demand service. Therefore, if you are not using the on-demand service:

• Select or enter any values in the required fields, select the Acknowledge checkbox, and click Save.

For information on configuring the on-demand service, see Installing and Configuring the On-Demand Service.

Errands

(Optional) In the Errands tab, choose the defaults for when errands run.

Errands can be thought of as tasks. For example, when deploying or updating RabbitMQ for PCF, Ops Manager can optionally run a series of post-deploy errands. An example is the Smoke Tests errand, which checks the health of the RabbitMQ cluster after a deploy or upgrade.

You can decide whether to run post-deploy errands by toggling them on or off before you click Apply Changes to update a configuration in the Ops Manager Installation Dashboard:

If you are using Ops Manager v2.3 or later, you can toggle errands on and off on the Review Pending Changes page.

This is a one-time action before an update. You can change these defaults by clicking Errands in the RabbitMQ for PCF Settings tab as well as the defaults for pre-delete errands.

Warning: In RabbitMQ for PCF v1.9.0 and later, post-deploy errands are on by default except the Recreate All Service Instances errand. Pivotal recommends keeping these defaults, because the smoke tests can encounter unexpected issues, and on-demand instances of RabbitMQ for PCF might fall behind if the Upgrade All Service Instances errand is not on by default.

For more information on errand run rules, see Errand Run Rules.

Post-Deploy Errands

Pre-Delete Errands

Pre-delete errands run after an operator chooses to delete a product in the Ops Manager Installation Dashboard, but before Ops Manager finishes deleting the product.

Stemcell

To verify that you have the correct stemcell, follow the procedure in Importing and Managing Stemcells.

Apply Configuration and Complete the Installation

1. Return to the Ops Manager Installation Dashboard.

2. If you are using Ops Manager v2.3 or later, click Review Pending Changes. For more information about this Ops Manager page, see Reviewing Pending Product Changes.

3. Click Apply Changes to complete the installation of RabbitMQ for PCF.

Other Configuration Topics

Connecting to a Highly Available RabbitMQ Cluster

The RabbitMQ tile, allows for a highly available cluster through multiple HAProxy nodes. The hostnames, uris and hosts properties have been added and should be used in preference over the equivalent singular properties. The singular properties are maintained for backwards compatibility and always contain the first value from the equivalent plural property. The singular properties will eventually be deprecated.

For example, with two HAProxy jobs deployed, the following properties will be present:

"hostname": "10.0.0.41",
"hostnames": [
    "10.0.0.41",
    "10.0.0.51"]

Port to protocol mappings

• 15672 = RabbitMQ Management UI
• 5672 = RabbitMQ
• 5671 = RabbitMQ SSL
• 1883 = MQTT
• 8883 = MQTT SSL
• 61613 = STOMP
• 61614 = STOMP SSL
• 15674 = Web STOMP
• 4567 = RabbitMQ Service Broker
• 3457 - 3459 = CF Loggregator

Security Groups

To enable access to the RabbitMQ tile service, you must ensure your security group allows access to the HAProxy and RabbitMQ Service Broker VMs configured in your deployment. You can obtain the IP addresses for these from the Ops Manager Status page for the RabbitMQ tile. Ensure the following ports are enabled for those VMs:

• 15672
• 5672
• 5671
• 1883
• 8883
• 61613
• 61614
• 15674
• 4567
• 3457 - 3459

The following is a template for configuring your Cloud Foundry security groups: [ {"protocol":"tcp","destination":"<haproxy-node-IP-addresses>","ports":"5671,5672,1883,8883,61613,61614,15672,15674"}, {"protocol":"tcp","destination":"<service-broker-node-IP-addresses>","ports":"4567"} ]

Application Security Groups

To allow this service to have network access, you must create Application Security Groups (ASGs).

Note: The service is unusable without Application Security Groups.

Application Container Network Connections

Application containers that use instances of the RabbitMQ service require the following outbound network connections:

Create an ASG named rabbitmq-app-containers with the above configuration and bind it to either:

• The appropriate space
• The default-running ASG set if you want to provide access to all started apps. Then restart your apps.

If you are using an external load balancer, or have more than one IP address for HAProxy, you must also create egress rules for these. For example:

[
  {
      "ports": "5671-5672",
      "protocol": "tcp",
      "destination": "10.10.10.10/32"
  }
]

Assigned IPs

RabbitMQ for PCF does not support changing the IP addresses which have been assigned to the RabbitMQ deployments. For example, you cannot change the subnet into which the RabbitMQ cluster was originally provisioned. Doing so causes the deployment to fail. For more information, see Changing Network or IP Addresses Results in a Failed Deployment.

Preserving Dynamically Assigned IPs

You cannot switch from dynamically assigned IP addresses to a different set of static IP addresses. However, you can configure Ops Manager so the current set of dynamically assigned IP addresses always continue to be used. This might be useful when upgrading.

To do this, follow these steps:

1. Go to the Status page in the RabbitMQ tile.
2. Take note of the IP addresses for the RabbitMQ Server and HAProxy for RabbitMQ jobs, in the order nodes appear in the UI.
3. Go to the Settings page, and click the Pre-Provisioned RabbitMQ tab.

4. Enter the IP addresses you got from the Status page as a comma-separated list.

   
   

5. Click Save.

RabbitMQ Server Settings that Cannot be Overwritten

In all cases:

• rabbit halt_on_upgrade_failure false
• rabbitmq_mqtt subscription_ttl 1800000
• log_levels [{connection,info}]
• halt_on_upgrade_failure false
• {rabbit, [ {collect_statistics_interval, 60000}] }
• {rabbitmq_management, [ {rates_mode, none}] }

When SSL is enabled:

• rabbit tcp_listeners []
• rabbit ssl_listeners [5671]
• rabbitmq_management listener [{port,15672},{ssl,false}]
• rabbitmq_mqtt ssl_listeners [8883]
• rabbitmq_stomp ssl_listeners [61614]