LATEST VERSION: 1.10.3 - CHANGELOG
RabbitMQ for PCF v1.9.8

Installing and Configuring RabbitMQ for PCF as a Pre-Provisioned Service

This topic provides instructions to PCF operators about how to install, configure, and deploy the RabbitMQ for PCF tile to provide a pre-provisioned service.

Note: For instructions on how to install, configure, and deploy the RabbitMQ for PCF tile as an on-demand service, see Installing and Configuring RabbitMQ for PCF as an On-Demand Service.

Download and Install the Tile

  1. Download the product file from Pivotal Network.

  2. Navigate to the Ops Manager Installation Dashboard and click Import a Product to upload the product file.

  3. Under the Import a Product button, click + next to the version number of RabbitMQ for PCF. This adds the tile to your staging area.

  4. Click the newly added RabbitMQ for PCF tile.

Configure the Tile for Pre-Provisioned Service

Management Dashboard

You must choose an admin username and password for RabbitMQ.

This will grant you full admin access to RabbitMQ through the Management UI.

Config credentials

Note: To rotate your administrator credentials, enter a new username and password, save your options, and redeploy by returning to the Ops Manager Installation Dashboard and clicking Apply Changes.

Plugins

You can choose which plugins you want to enable.

You must leave the management plugin enabled otherwise nothing will work.

Config plugins

Click here for more information about RabbitMQ plugins

HAProxy Ports

You can choose which ports HAProxy should load balance to the RabbitMQ nodes.

Config haproxy

By default, all the default ports of all the available plugins will be load-balanced.

However, if you install extra protocol plugins, or provide a custom configuration which changes the ports that RabbitMQ listens on then you must update the list of load-balanced ports.

Note that you must always leave the management plugin listening on port 15672 and load balance that port.

If you change the topology of your RabbitMQ cluster, the HAProxy is automatically reconfigured during the deployment.

Disk free alarm limit

You can choose how much disk space rabbit should attempt to keep free at any given time. For more information about how this works ‘under the hood’, see here. In summary, rabbit will periodically check if there is sufficient free space on disk. If there is not, rabbit will temporarily stop accepting new messages. This gives your apps time to consume existing messages, and thus free up some disk space. In the RabbitMQ Tile, we provide four options for this value:

Disk alarm threshold

  • 50MB is a bare minimum value.
  • 100% Memory ensures that at the time when rabbit checks the available disk, there must be enough space for rabbit to page all memory-based messages out to disk.
  • 150% Memory is recommended. This is because it is possible that in between disk-space-checks, rabbit may:
    • write persistent messages to disk (using up some disk space)
    • accept more memory-based messages into various queues
    • page all memory-based messages to disk. In these circumstances, rabbit may require more free disk than it has memory.
  • 200% Memory is a conservative value, for when the operator wants higher confidence that rabbit will never run out of disk.

Selecting 50MB is not recommended and could cause data loss. See below.

What are the dangers of setting this value too low?

If the disk of a given rabbit node completely fills while rabbit is running, that node will crash. This can lead to data loss, and loss of availability.

Rabbit reserves the right to page any and all messages in memory (even transient messages) to disk at any time. You should set your disk free alarm limit high enough to ensure that rabbit always has at least enough space to do this.

What are the disadvantages of setting this value too high?

If you set your disk free alarm limit to a value larger than the size of your persistent disk, then rabbit will never be able to free up enough disk space to accept new messages. You should ensure that you have a large enough disk to persist all the messages you intend to persist while also leaving enough space free to satisfy the disk free alarm limit that you have chosen.

When should I use the 50MB value?

This is not recommended in production. However, if you are experimenting with a development environment you might want to use a small disk to keep down costs, at the expense of increasing the chances that rabbit might crash and lose data.

Connecting to a Highly Available RabbitMQ Cluster

The RabbitMQ tile, allows for a highly available cluster through multiple HAProxy nodes. The hostnames, uris and hosts properties have been added and should be used in preference over the equivalent singular properties. The singular properties are maintained for backwards compatibility and will always contain the first value from the equivalent plural property. The singular properties will eventually be deprecated.

For example with two HAProxy jobs deployed the following properties will be present:

"hostname": "10.0.0.41",
"hostnames": [
    "10.0.0.41",
    "10.0.0.51"]

Port to protocol mappings

  • 15672 = Management dashboard
  • 5672 = RabbitMQ
  • 5671 = RabbitMQ SSL
  • 1883 = MQTT
  • 8883 = MQTT SSL
  • 61613 = STOMP
  • 61614 = STOMP SSL
  • 15674 = Web STOMP
  • 4567 = RabbitMQ Service Broker
  • 3457 - 3459 = CF Loggregator
  • 4001 = CF Loggregator - Doppler
  • 8300 - 8301 = Consul

Security Groups

To enable access to the RabbitMQ tile service, you must ensure your security group allows access to the HAProxy and RabbitMQ Service Broker VMs configured in your deployment. You can obtain the IP addresses for these from the Ops Manager Status page for the RabbitMQ tile. Ensure the following ports are enabled for those VMs:

  • Inbound
Port(s) Protocol(s) Source Reason
15672 tcp Broker and internet(*) Allowing access to the RabbitMQ Management Dashboard & API
5671 - 5672 tcp All AMQP clients RabbitMQ will listen on those ports for AMQP
1883, 8883 tcp All MQTT clients RabbitMQ will listen on those ports for MQTT
61613, 61614 tcp All STOMP clients RabbitMQ will listen on those ports for STOMP
15674 tcp All Web STOMP clients RabbitMQ will listen on this port for STOMP-over-WebSockets
4567 tcp ERT ERT sends commands to the Service Broker for RabbitMQ
8080 tcp ERT ERT sends commands to the On Demand Service Broker for RabbitMQ
3457 - 3459 tcp ERT Between RabbitMQ and ERT network for Metrics
8300 - 8301 tcp, udp ERT Between RabbitMQ and ERT network for Consul

(*) Everyone that needs to access the RabbitMQ Management Dashboard & API externally

  • Outbound
Port(s) Protocol(s) Destination Reason
3457 - 3459 tcp ERT Between RabbitMQ and ERT network for Metrics
4001 tcp ERT From RabbitMQ to ERT (etcd) for Metron
8300 - 8301 tcp, udp ERT Between RabbitMQ and ERT network for Consul

The following is a template for configuring your Cloud Foundry security groups: [ {"protocol":"tcp","destination":"<haproxy-node-IP-addresses>","ports":"5671,5672,1883,8883,61613,61614,15672,15674"}, {"protocol":"tcp","destination":"<service-broker-node-IP-addresses>","ports":"4567"} ]

Application Security Groups

To allow this service to have network access you must create Application Security Groups (ASGs).

Note: The service is unusable without Application Security Groups.

Application Container Network Connections

Application containers that use instances of the RabbitMQ service require the following outbound network connections:

Destination Ports Protocol Reason
HAProxy IPs 5672 tcp Application containers using AMQP
HAProxy IPs 5671 tcp Application containers using AMQP over SSL
HAProxy IPs 1883 tcp Application containers using MQTT
HAProxy IPs 8883 tcp Application containers using MQTT over SSL
HAProxy IPs 61613 tcp Application containers using STOMP
HAProxy IPs 61614 tcp Application containers using STOMP over SSL
HAProxy IPs 61613 tcp Application containers using Web STOMP

Create an ASG name rabbitmq-app-containers with the above configuration and bind it to the appropriate space, or, to provide access to all started apps, bind it to the default-running ASG set and restart your apps. If you are using an external load balancer or have more than one IP address for HAProxy, you must also create egress rules for these. Example:

[
  {
      "ports": "5671-5672",
      "protocol": "tcp",
      "destination": "10.10.10.10/32"
  }
]

SSL

You can provide SSL certificates and keys for use by the RabbitMQ cluster.

Config ssl

SSL is simultaneously provided on the AMQPS port (5671) and the management port (15672).

If you provide SSL keys and certificates, you disable non-SSL support.

No other plugins are automatically configured for use with SSL.

SSL settings are applied equally across all machines in the cluster.

For more information about SSL support, see https://www.rabbitmq.com/ssl.html.

You can provide an Erlang cookie to be used by the cluster. This can be useful if you want to connect directly to the RabbitMQ cluster, such as with rabbitmqctl, or to connect other machines running Erlang.

Config erlang

Cluster Scaling Known Issue

If you have not set the Erlang cookie and you want to scale-out your cluster size without downtime, you’ll need to perform the following steps:

  • Follow the steps for troubleshooting with the BOSH CLI
  • bosh ssh rabbitmq-server/0
  • sudo -i
  • echo $(cat /var/vcap/store/rabbitmq/.erlang.cookie)
  • Paste the value from the above command into the Erlang cookie field displayed above.

You’ll then be able to adjust the size of your cluster and run Apply Changes.

Note: BOSH will tell you that the cookie has changed - this is because the default value in the manifest is empty, which results in an auto-generated cookie. The value of the cookie on the server will remain the same, so the section below will not apply.

If you want to change your Erlang cookie value, this will require cluster downtime. It is also strongly recommended that you do not change anything else, as it is possible for configuration to be inconsistently applied during this process.

The deployment may also fail—if it does, redeploying will fix the issue.

RabbitMQ Config

You can optionally provide a full rabbitmq.config file by pasting its contents in the RabbitMQ configuration field in the RabbitMQ tab.

Config rabbitmq

This rabbitmq.config file is then provided to all the nodes in the cluster. You can see an example rabbitmq.config file here. For more information about the RabbitMQ configuration, see the RabbitMQ documentation.

The input must be Base64 encoded.

For example, to configure the rates_mode of the rabbitmq_management stats, which looks like this:

[
  {rabbitmq_management, [
    {rates_mode, detailed}
  ]}
].
  1. Encode the file into Base64:

    WwogIHtyYWJiaXRtcV9tYW5hZ2VtZW50LCBbCiAgICB7cmF0ZXNfbW9kZSwgZGV0YWlsZWR9CiAgXX0KXS4K
    
  2. Paste it into the RabbitMQ configuration field:

    Config rabbitmq file

TLS Support

TLS v1.0 is disabled by default, due to security issues.

Config tls

TLS v1.1 and v1.2 are enabled by default and can be turned on and off.

External load balancer

Config elb

You can configure a DNS name or IP address of an external load balancer to be returned in the binding credentials (VCAP_SERVICES) to application developers.

Assigned IPs

RabbitMQ for PCF does not support changing the IP addresses which have been assigned to the RabbitMQ deployments. For example, you cannot change the subnet into which the RabbitMQ cluster was originally provisioned. Doing so causes the deployment to fail. For more information, see Changing Network or IP Addresses Results in a Failed Deployment.

Static IPs

Switching from dynamic IPs to static IPs (Upgrading)

It is not possible to switch from dynamic IPs to a different set of static IPs, but you can set up Ops Manager so the current set of dynamically assigned IPs will always continue to be used.

  1. Go to the Status page on the RabbitMQ product.
  2. Note the IPs for the RabbitMQ Server and HAProxy for RabbitMQ jobs, in the order nodes appear in the UI.
  3. Go to the Settings tab, and navigate to the Networking page.
  4. Fill the IP addresses you got from the Status page. IP addresses should be in a comma-separated list.

Config static ip

RabbitMQ Server settings that cannot be overwritten

  • rabbit halt_on_upgrade_failure false
  • rabbitmq_mqtt subscription_ttl 1800000
  • log_levels [{connection,info}]
  • halt_on_upgrade_failure false
  • {rabbit, [ {collect_statistics_interval, 60000}] }
  • {rabbitmq_management, [ {rates_mode, none}] }

When SSL is enabled:

  • rabbit tcp_listeners []
  • rabbit ssl_listeners [5671]
  • rabbitmq_management listener [{port,15672},{ssl,false}]
  • rabbitmq_mqtt ssl_listeners [8883]
  • rabbitmq_stomp ssl_listeners [61614]

Configuring Metrics Polling Interval

The default polling interval is 30 seconds for all deployed components and it is recommended not to change it. The lowest interval setting is 10 seconds to avoid overwhelming components. This change will effect all deployed instances.

Metrics polling

Apply Configuration and Complete the Installation

  1. Return to the Ops Manager Installation Dashboard and click Apply Changes to install RabbitMQ for PCF tile.

  2. Q: How does one follow the progress of the tile, what to check? how long does it take?

Smoke Test Process

Smoke tests run as a post-deployment errand. The smoke tests perform the following for each available service plan:

  1. Target the org system and create a cf space to run the tests.
  2. Deploy an instance of the CF RabbitMQ Example App in this cf space.
  3. Create a RabbitMQ service instance and bind it to the CF RabbitMQ Example App.
  4. Check that the CF RabbitMQ Example App can write to, and read from, the RabbitMQ service instance.
  5. Clean up the deployed Example App and all its service bindings.
  6. Delete the cf space created for the tests.

For more information, see Errands.

Create a pull request or raise an issue on the source for this page in GitHub