LATEST VERSION: 1.10.5 - CHANGELOG
RabbitMQ for PCF v1.8.20

Installing and Configuring RabbitMQ for PCF as a Pre-Provisioned Service

This topic provides instructions to PCF operators about how to install, configure, and deploy the RabbitMQ for PCF tile to provide a pre-provisioned service.

Note: For instructions on how to install, configure, and deploy the RabbitMQ for PCF tile as an on-demand service, see Installing and Configuring RabbitMQ for PCF as an On-Demand Service.

Download and Install the Tile

  1. Download the product file from Pivotal Network.

  2. Navigate to the Ops Manager Installation Dashboard and click Import a Product to upload the product file.

  3. Under the Import a Product button, click + next to the version number of RabbitMQ for PCF. This adds the tile to your staging area.

  4. Click the newly added RabbitMQ for PCF tile.

Configure the Tile for Pre-Provisioned Service

Management Dashboard

You must choose an admin username and password for RabbitMQ.

This will grant you full admin access to RabbitMQ through the Management UI.

Config credentials

Note: To rotate your administrator credentials, enter a new username and password, save your options, and redeploy by returning to the Ops Manager Installation Dashboard and clicking Apply Changes.

Plugins

You can choose which plugins you want to enable.

You must leave the management plugin enabled otherwise nothing will work.

Config plugins

Click here for more information about RabbitMQ plugins

HAProxy Ports

You can choose which ports HAProxy should load balance to the RabbitMQ nodes.

Config haproxy

By default, all the default ports of all the available plugins will be load-balanced.

However, if you install extra protocol plugins, or provide a custom configuration which changes the ports that RabbitMQ listens on then you must update the list of load-balanced ports.

Note that you must always leave the management plugin listening on port 15672 and load balance that port.

If you change the topology of your RabbitMQ cluster, the HAProxy is automatically reconfigured during the deployment.

Disk free alarm limit

You can choose how much disk space rabbit should attempt to keep free at any given time. For more information about how this works ‘under the hood’, see here.

If this is a fresh deployment please check that the 'Resource Config’ settings you choose for the RabbitMQ server, match the constraints which you choose as outlined below. For example the default resource configurations will not work if you choose the '150% Memory’ option, as it it optimized for high message volumes.

In summary, rabbit will periodically check if there is sufficient free space on disk. If there is not, rabbit will temporarily stop accepting new messages. This gives your apps time to consume existing messages, and thus free up some disk space. In the RabbitMQ Tile, we provide four options for this value:

Disk alarm threshold

  • 50MB is a bare minimum value.
  • 100% Memory ensures that at the time when rabbit checks the available disk, there must be enough space for rabbit to page all memory-based messages out to disk.
  • 150% Memory is recommended. This is because it is possible that in between disk-space-checks, rabbit may:
    • write persistent messages to disk (using up some disk space)
    • accept more memory-based messages into various queues
    • page all memory-based messages to disk. In these circumstances, rabbit may require more free disk than it has memory.
  • 200% Memory is a conservative value, for when the operator wants higher confidence that rabbit will never run out of disk.

Selecting 50MB is not recommended and could cause data loss. See below.

What are the dangers of setting this value too low?

If the disk of a given rabbit node completely fills while rabbit is running, that node will crash. This can lead to data loss, and loss of availability.

Rabbit reserves the right to page any and all messages in memory (even transient messages) to disk at any time. You should set your disk free alarm limit high enough to ensure that rabbit always has at least enough space to do this.

What are the disadvantages of setting this value too high?

If you set your disk free alarm limit to a value larger than the size of your persistent disk, then rabbit will never be able to free up enough disk space to accept new messages. You should ensure that you have a large enough disk to persist all the messages you intend to persist while also leaving enough space free to satisfy the disk free alarm limit that you have chosen.

When should I use the 50MB value?

This is not recommended in production. However, if you are experimenting with a development environment you might want to use a small disk to keep down costs, at the expense of increasing the chances that rabbit might crash and lose data.

Connecting to a Highly Available RabbitMQ Cluster

The RabbitMQ tile, allows for a highly available cluster through multiple HAProxy nodes. The hostnames, uris and hosts properties have been added and should be used in preference over the equivalent singular properties. The singular properties are maintained for backwards compatibility and will always contain the first value from the equivalent plural property. The singular properties will eventually be deprecated.

For example with two HAProxy jobs deployed the following properties will be present:

"hostname": "10.0.0.41",
"hostnames": [
    "10.0.0.41",
    "10.0.0.51"]

Port to protocol mappings

  • 15672 = Management dashboard
  • 5672 = RabbitMQ
  • 5671 = RabbitMQ SSL
  • 1883 = MQTT
  • 8883 = MQTT SSL
  • 61613 = STOMP
  • 61614 = STOMP SSL
  • 15674 = Web STOMP
  • 4567 = RabbitMQ Service Broker
  • 3457 - 3459 = CF Loggregator
  • 4001 = CF Loggregator - Doppler
  • 8080 = On Demand Service Broker
  • 8300 - 8301 = Consul

Security Groups

To enable access to the RabbitMQ tile service, you must ensure your security group allows access to the HAProxy and RabbitMQ Service Broker VMs configured in your deployment. You can obtain the IP addresses for these from the Ops Manager Status page for the RabbitMQ tile. Ensure the following ports are enabled for those VMs:

  • Inbound
Port(s) Protocol(s) Source Reason
15672 tcp Broker and internet(*) Allowing access to the RabbitMQ Management Dashboard & API
5671 - 5672 tcp All AMQP clients RabbitMQ will listen on those ports for AMQP
1883, 8883 tcp All MQTT clients RabbitMQ will listen on those ports for MQTT
61613, 61614 tcp All STOMP clients RabbitMQ will listen on those ports for STOMP
15674 tcp All Web STOMP clients RabbitMQ will listen on this port for STOMP-over-WebSockets
4567 tcp ERT ERT sends commands to the Service Broker for RabbitMQ
8080 tcp ERT ERT sends commands to the On Demand Service Broker for RabbitMQ
3457 - 3459 tcp ERT Between RabbitMQ and ERT network for Metrics
8300 - 8301 tcp, udp ERT Between RabbitMQ and ERT network for Consul

(*) Everyone that needs to access the RabbitMQ Management Dashboard & API externally

  • Outbound
Port(s) Protocol(s) Destination Reason
3457 - 3459 tcp ERT Between RabbitMQ and ERT network for Metrics
4001 tcp ERT From RabbitMQ to ERT (etcd) for Metron
8300 - 8301 tcp, udp ERT Between RabbitMQ and ERT network for Consul

The following is a template for configuring your Cloud Foundry security groups: [ {"protocol":"tcp","destination":"<haproxy-node-IP-addresses>","ports":"5671,5672,1883,8883,61613,61614,15672,15674"}, {"protocol":"tcp","destination":"<service-broker-node-IP-addresses>","ports":"4567"} ]

Application Security Groups

To allow this service to have network access you must create Application Security Groups (ASGs).

Note: The service is unusable without Application Security Groups.

Application Container Network Connections

Application containers that use instances of the RabbitMQ service require the following outbound network connections:

Destination Ports Protocol Reason
HAProxy IPs 5672 tcp Application containers using AMQP
HAProxy IPs 5671 tcp Application containers using AMQP over SSL
HAProxy IPs 1883 tcp Application containers using MQTT
HAProxy IPs 8883 tcp Application containers using MQTT over SSL
HAProxy IPs 61613 tcp Application containers using STOMP
HAProxy IPs 61614 tcp Application containers using STOMP over SSL
HAProxy IPs 61613 tcp Application containers using Web STOMP

Create an ASG name rabbitmq-app-containers with the above configuration and bind it to the appropriate space, or, to provide access to all started apps, bind it to the default-running ASG set and restart your apps. If you are using an external load balancer or have more than one IP address for HAProxy, you must also create egress rules for these. Example:

[
  {
      "ports": "5671-5672",
      "protocol": "tcp",
      "destination": "10.10.10.10/32"
  }
]

TLS

RabbitMQ can be configured to use TLS in all internal communications within the rabbit cluster, and also between the rabbit cluster and rabbit clients (these will usually be apps deployed to CF).

TLS support can include both: - Ensuring that communications between any two actors are encrypted - Ensuring that the identities of any two actors are known to each other

Configuring RabbitMQ TLS for Production

In a production environment it is recommended that you use TLS certificates which have been signed by a root Certificate Authority. In order to encrypt the communication channel, rabbit will need a public and private key. In addition the Certificate Authority used to generate the keys should be provided.

Configuring RabbitMQ TLS for Development or Testing

Ops Manager allow to create a pair of public and private key signed by its own Certificate Authority

Known TLS Vulnerabilities, and What To Do About Them

By default only TLSv1.1 and TLSv1.2 are enabled, TLSv1.0 is disabled by default because is vulnerable to attach such POODLE and BEAST. Is possible to enable TLSv1.0 to allow legacy application to connect (e.g. application based on JDK 6.0)

Config tls

TLS is simultaneously provided on the AMQPS port (5671) and the management port (15672).

If you provide TLS keys and certificates, you disable non-TLS support.

No other plugins are automatically configured for use with TLS.

TLS settings are applied equally across all machines in the cluster.

For more information about TLS support, see https://www.rabbitmq.com/ssl.html.

You can provide an Erlang cookie to be used by the cluster. This can be useful if you want to connect directly to the RabbitMQ cluster, such as with rabbitmqctl, or to connect other machines running Erlang.

Config erlang

Scaling Known Issue

If you have not set the Erlang cookie and you want to scale-out your cluster size without downtime, you’ll need to perform the following steps:

  • Follow the steps for troubleshooting with the BOSH CLI
  • bosh ssh rabbitmq-server/0
  • sudo -i
  • echo $(cat /var/vcap/store/rabbitmq/.erlang.cookie)
  • Paste the value from the above command into the Erlang cookie field displayed above.

You’ll then be able to adjust the size of your cluster and run Apply Changes.

Note: BOSH will tell you that the cookie has changed - this is because the default value in the manifest is empty, which results in an auto-generated cookie. The value of the cookie on the server will remain the same, so the section below will not apply.

Changing the Value Known Issue

If you want to change your Erlang cookie value, this will require cluster downtime. It is also strongly recommended that you do not change anything else, as it is possible for configuration to be inconsistently applied during this process.

The deployment may also fail—if it does, redeploying will fix the issue.

RabbitMQ Config

You can optionally provide a full rabbitmq.config file by pasting its contents in the RabbitMQ configuration field in the RabbitMQ tab.

Config rabbitmq

This rabbitmq.config file is then provided to all the nodes in the cluster.

You can see an example rabbitmq.config file here. For more information about the RabbitMQ configuration, see the RabbitMQ documentation.

The input must be Base64 encoded.

For example, to configure the rates_mode of the rabbitmq_management stats, which looks like this:

[
  {rabbitmq_management, [
    {rates_mode, detailed}
  ]}
].
  1. Encode the file into Base64:

    WwogIHtyYWJiaXRtcV9tYW5hZ2VtZW50LCBbCiAgICB7cmF0ZXNfbW9kZSwgZGV0YWlsZWR9CiAgXX0KXS4K
    
  2. Paste it into the RabbitMQ configuration field:

    Config rabbitmq file

TLS Support

TLS v1.0 is disabled by default, due to insecurities.

Config tls1

You can enable it again by selecting the checkbox.

TLS v1.1 and 1.2 are enabled by default and cannot be turned on or off.

External load balancer

Config elb

You can configure a DNS name or IP address of an external load balancer to be returned in the binding credentials (VCAP_SERVICES) to application developers.

Assigned IPs

RabbitMQ for PCF does not support changing the IP addresses that have been assigned to the RabbitMQ deployments. For example, you cannot change the subnet into which the RabbitMQ cluster was originally provisioned. Doing so causes the deployment to fail. For more information, see Changing Network or IP Addresses Results in a Failed Deployment.

Static IPs

Switching from dynamic IPs to static IPs (Upgrading)

It is not possible to switch from dynamic IPs to a different set of static IPs, but you can set up Ops Manager so the current set of dynamically assigned IPs will always continue to be used.

  1. Go to the Status page on the RabbitMQ product.
  2. Note the IPs for the RabbitMQ Server and HAProxy for RabbitMQ jobs, in the order nodes appear in the UI.
  3. Go to the Settings tab, and navigate to the Networking page.
  4. Fill the IP addresses you got from the Status page. IP addresses should be in a comma-separated list.

Config static ip

RabbitMQ Server settings that cannot be overwritten

  • rabbit halt_on_upgrade_failure false
  • rabbitmq_mqtt subscription_ttl 1800000
  • log_levels [{connection,info}]
  • halt_on_upgrade_failure false
  • {rabbit, [ {collect_statistics_interval, 60000}] }
  • {rabbitmq_management, [ {rates_mode, none}] }

When SSL is enabled:

  • rabbit tcp_listeners []
  • rabbit ssl_listeners [5671]
  • rabbitmq_management listener [{port,15672},{ssl,false}]
  • rabbitmq_mqtt ssl_listeners [8883]
  • rabbitmq_stomp ssl_listeners [61614]

Metrics polling

Apply Configuration and Complete the Installation

  1. Return to the Ops Manager Installation Dashboard and click Apply Changes to install RabbitMQ for PCF tile.

  2. Q: How does one follow the progress of the tile, what to check? how long does it take?

Create a pull request or raise an issue on the source for this page in GitHub