LATEST VERSION: 1.9 - CHANGELOG
Redis for PCF v1.9

Redis for PCF Smoke Tests

Page last updated:

Redis for PCF runs a set of smoke tests during installation to confirm system health. The tests run in the org system and in the space redis-smoke-tests. The tests run as an application instance with a restrictive Application Security Group (ASG).

Smoke Test Steps

The smoke tests perform the following for each available service plan:

  1. Targets the org system and space redis-smoke-tests (creating them if they do not exist)
  2. Creates a restrictive security group, redis-smoke-tests-sg, and binds it to the space
  3. Deploys an instance of the CF Redis Example App to this space
  4. Creates a Redis instance and binds it to the CF Redis Example App
  5. Checks that the CF Redis Example App can write to and read from the Redis instance

Security Groups

Smoke tests create a new application security group for the CF Redis Example App (redis-smoke-tests-sg) and delete it once the tests finish. This security group has the following rules:

[
    {
      "protocol": "tcp",
      "destination": "<dedicated node IP addresses>",
      "ports": "6379" // Redis dedicated node port
    },
    {
      "protocol": "tcp",
      "destination": "<broker IP address>",
      "ports": "32768-61000" // Ephemeral port range (assigned to shared-vm instances)
    }
]

This allows outbound traffic from the test app to the Redis shared VM and dedicated VM nodes.

Smoke Tests Resilience

Smoke tests could fail due to reasons outside of the Redis deployment; for example network latency causing timeouts or the CloudFoundry instance dropping requests.

The smoke tests implement a retry policy for commands issued to CF, for two reasons: - to avoid smoke test failures due to temporary issues such as the ones mentioned above - to ensure that the service instances and bindings created for testing are cleaned up.

Smoke tests will retry failed commands against CF. They use a linear back-off with a baseline of 0.2 seconds, for a maximum of 30 attempts per command. Therefore, assuming that the first attempt is at 0s and fails instantly, subsequent retries will be at 0.2s, 0.6s, 1.2s and so on until either the command succeeds or the maximum number of attempts is reached.

The linear back-off was selected as a good middle ground between: - situations where the system is generally unstable-such as load-balancing issues-where max number of retries are preferred, and - situations where the system is suffering from a failure that lasts a few seconds-such as restart of a CloudFoundry VM-where it’s preferable to wait before reattempting the command.

Considerations

Because of the retry policy described in Smoke tests resilience, smoke tests will generally take longer than previous versions of PCF Redis to complete within environments with intermittent failures. If run regularly, e.g. as part of CI, they might last longer than the frequency of the automatic runs, queueing up executions. We recommend adjusting the frequency of the automatic smoke test runs based on the amount of time they take to complete.

The above retry policy does not guard against a more permanent CloudFoundry downtime or network connectivity issues. In this case, commands will fail after the maximum number of attempts and might leave claimed instances behind. We recommend disabling automatic smoke test runs and manually releasing any claimed instances in case of upgrades or scheduled downtimes.

Troubleshooting

If errors occur while the smoke tests run, they will be summarised at the end of the errand log output. Detailed logs can be found where the failure occurs. Some common failures are listed below.

Error Failed to target Cloud Foundry
Cause Your Pivotal Cloud Foundry is unresponsive
Solution Examine the detailed error message in the logs and check the PCF Troubleshooting Guide for advice
Error Failed to bind Redis service instance to test app
Cause Your deployment’s broker has not been registered with Pivotal Cloud Foundry
Solution Examine the broker-registrar installation step output and troubleshoot any problems.

When encountering an error when running smoke tests, it can be helpful to search the log for other instances of the error summary printed at the end of the tests, e.g. Failed to target Cloud Foundry. Lookout for TIP: ... in the logs next to any error output for further troubleshooting hints.

Create a pull request or raise an issue on the source for this page in GitHub