Installing the Service Broker

This topic describes how to install the Service Broker for AWS by configuring your AWS account and importing the product file into Ops Manager.

Installation Prerequisites

  • Pivotal Cloud Foundry version 1.7 or higher
  • An Amazon Web Services (AWS) account
  • Pivotal MySQL, or an already-created MySQL or PostgreSQL database

Step 1: Set up AWS

In this step you configure your AWS account to allow the Service Broker for AWS to create and manage AWS resources.

Create a new IAM user for the PCF cluster in the AWS Identity & Access Management console by following the procedure below.

Note: If you have more than one PCF deployment, you must create an IAM user for each deployment. Create a new IAM user in your AWS account (do not use the same IAM user that installed PCF on AWS as it has different policy restrictions).

  1. Select Policies and click Create Policy.
  2. Choose Create Your Own Policy.
  3. Enter PCFInstallationPolicy for Policy Name and Installation Policy for PCF for Description.
  4. Copy the following text into the Policy Document section:

        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": [
                        "s3:CreateBucket",
                        "s3:DeleteBucket",
                        "s3:PutBucketAcl",
                        "s3:PutBucketLogging",
                        "s3:PutBucketTagging",
                        "s3:GetObject",
                        "s3:ListBucket",
                        "iam:CreateAccessKey",
                        "iam:CreateUser",
                        "iam:GetUser",
                        "iam:DeleteAccessKey",
                        "iam:DeleteUser",
                        "iam:DeleteUserPolicy",
                        "iam:ListAccessKeys",
                        "iam:ListAttachedUserPolicies",
                        "iam:ListPolicies",
                        "iam:PutUserPolicy",
                        "iam:GetPolicy",
                        "iam:GetAccountAuthorizationDetails",
                        "rds:CreateDBCluster",
                        "rds:CreateDBInstance",
                        "rds:DeleteDBCluster",
                        "rds:DeleteDBInstance",
                        "rds:DescribeDBClusters",
                        "rds:DescribeDBInstances",
                        "rds:DescribeDBSnapshots",
                        "rds:DeleteDBSnapshot",
                        "rds:CreateDBParameterGroup",
                        "rds:ModifyDBParameterGroup",
                        "rds:DeleteDBParameterGroup",
                        "dynamodb:ListTables",
                        "dynamodb:DeleteTable",
                        "dynamodb:DescribeTable",
                        "sqs:CreateQueue",
                        "sqs:DeleteQueue"
                    ],
                    "Effect": "Allow",
                    "Resource": "*"
                }
            ]
        }
    
  5. Click Validate Policy to check for errors.

  6. Click Create Policy. This creates an AWS policy that you must apply to every IAM user you create for the Service Broker for AWS.

  7. For each IAM user, click the user name > Permissions tab > Attach Policy and select PCFInstallationPolicy.

  8. Record the AWS Access Key ID and Secret Access Key for each of your PCF IAM accounts for later use.

Note: The Service Broker for AWS also allows a custom name for the PCFInstallationPolicy.

To use the default name: follow the steps above for configuring the PCFInstallationPolicy policy.

To use a custom name: In the steps above, use a custom name instead of PCFInstallationPolicy. Later, during Broker Configuration, you will need to enter this custom name.

Service Key Policies

The Service Broker for AWS allows App Developers to create service keys for their service instances. Creating a service key creates an IAM user with a templated policy, and provides the App Developer with Access Key credentials to the actions listed in the policies below.

In V1.1.0, the service key Policy names are configurable. You can choose your own policy name and configure that name in each service by updating the Service Key Policy Name field. The default names are used below.

Create the policy templates for each service that you will enable, by logging into the AWS console, and doing the following:

  1. Select Policies and click Create Policy.
  2. Choose Create Your Own Policy.
  3. Enter PCFAppDeveloperPolicy-s3 for Policy Name and Service Broker for AWS Service Key policy for S3 for Description.
  4. Enter the following text in the Policy Document section, or create your own policy template

        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "allowtagging",
                    "Effect": "Allow",
                    "Action": [
                        "s3:GetBucketTagging",
                        "s3:PutBucketTagging"
                    ],
                    "Resource": [
                        "arn:aws:broker:resource::"
                    ]
                 }
             ]
         }
    
  5. Click Validate Policy to check for errors.

  6. Click Create Policy. This creates an AWS policy that you must apply to every IAM user you create for the Service Broker for AWS.

  7. Repeat steps 1-6 for PCFAppDeveloperPolicy-sqs PCFAppDeveloperPolicy-dynamodb and PCFAppDeveloperPolicy-rds; using or modifying the examples below.

PCFAppDeveloperPolicy-sqs Example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1471890189000",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues",
                "sqs:PurgeQueue",
                "sqs:ReceiveMessage",
                "sqs:SendMessage"
            ],
            "Resource": [
                "arn:aws:broker:resource::"
            ]
        }
    ]
}

PCFAppDeveloperPolicy-dynamodb Example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1471873911000",
            "Effect": "Allow",
            "Action": [
                "dynamodb:*"
            ],
            "Resource": [
                "arn:aws:broker:resource::"
            ]
        }
    ]
}

PCFAppDeveloperPolicy-rds Example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1471636048000",
            "Effect": "Allow",
            "Action": [
                "rds:ListTagsForResource",
                "rds:DescribeDbInstances"
            ],
            "Resource": [
                "arn:aws:broker:resource::"
            ]
        }
    ]
}

Step 2: Create Service Broker Database

The Service Broker for AWS requires access to a MySQL or PostgreSQL database that is highly available and regularly backed up, in order to store its configuration information. Loss of this data will severely affect the ability of the service broker to manage the configured service instances. However, the AWS resources will not be affected.

The database can be Pivotal MySQL or use an external MySQL or PostgreSQL database (including an AWS RDS database).

If you choose to use a Pivotal MySQL database, the service broker will automatically provision the database.

If you choose to use an RDS MySQL or RDS PostgreSQL database, follow these steps:

  1. In the AWS RDS console, create the appropriate database:

    • Choose the smallest DB instance class, db.t2.micro.
    • Select yes for Multi-AZ Deployment.
    • Choose Provisioned IOPS (SSD) for Storage Type. For a non-production database, you can choose General Purpose (SSD).
    • Enter the minimum accepted value ‘5’ to '100’ GB of Allocated Storage.
    • Leave Provisioned IOPS at the default value
    • Enter pcf-aws-services for the DB Instance Identifier.
    • Enter suitable values for the Master Username and Master Password
  2. Record the Database IP, Database Port, Admin user, Admin password and Database Name. The Service Broker for AWS stores configuration information in this database.

  3. Ensure that the Service Broker for AWS will be able to connect to your database. If necessary for your deployment, modify your AWS VPC settings to allow access to the RDS database from an external IP address. For more information, see the Accessing a DB Instance in a VPC topic in the AWS documentation.

Step 3: Application Security Groups

To allow this service to have network access you will need to create Application Security Groups (ASGs).

Note: Without Application Security Groups the service will not be usable.

Service Container Network Connections

This service is deployed as an application to the iaas-brokers space in the system org, and requires the following outbound network connections:

Destination Ports Protocol Reason
AWS_IP_RANGE 443, 5432, 1521, 1433, 3306 tcp This service accesses RDS/S3

Note: The AWS IP range is available from Amazon. You may choose to use curl and jq to convert the IP range as downloaded from Amazon into ASG rules:

    curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | \
    jq '[
      {
        ports: "443, 5432, 1521, 1433, 3306",
        protocol: "tcp",
        destination: .prefixes | map(select(.service == "AMAZON")) | .[] | .ip_prefix
      }
    ] | .' > aws-service-broker.json

Note: This creates a configuration which allows traffic for all AWS Regions. You may wish to further filter the list of entries in the Application Security Group config to a specific set of AWS Regions by adding additional `jq` `map(select(.region == “REGION_NAME”))` statements for each region.

Log in as an administrator and create an ASG called aws-service-broker, binding it to the iaas-brokers space in the system org:

$ cf create-security-group aws-service-broker aws-service-broker.json
$ cf bind-security-group aws-service-broker system iaas-brokers

Application Container Network Connections

Application containers that use instances of this service require the following outbound network connections:

Destination Ports Protocol Reason
AWS_IP_RANGE 443, 5432, 1521, 1433, 3306 tcp Application containers may access S3/RDS

Create an ASG called aws-service-broker-app-containers with the above configuration and bind it to the appropriate space or, to give all started apps access, the default-running ASG set. Example:

[
  {
      "ports": "443, 5432, 1521, 1433, 3306",
      "protocol": "all",
      "destination": "192.0.2.0/14"
  },
  {
      "ports": "443, 5432, 1521, 1433, 3306",
      "protocol": "all",
      "destination": "198.51.100.0/22"
  },
  ...rest of AWS IPs elided...
]

Step 4: Install in Ops Manager

To install the Service Broker for AWS tile in Ops Manager, complete the following steps:

  1. Download the Service Broker for AWS product file from Pivotal Network.

  2. From the Ops Manager Installation Dashboard, click Import a Product to upload the product file.

  3. Click Add next to the uploaded product description in the Available Products view to add this product to your staging area.

  4. Click on the newly added tile to configure the sections as described below.

AWS Config

The setup in Step 1 must be successfully completed before beginning this configuration. You will need the AWS and database parameters from that step to successfully configure the broker.

  1. Click AWS Config. Aws sb awsconfig
  2. Enter the AWS Access Key ID and the Secret Access Key from Step 1.
  3. Enter the Temporary Credentials Timeout value in minutes. This is the time after which service key credentials will expire. To disable expiry of the credentials, enter 0.
  4. Modify the AWS Endpoint ONLY if you are using Amazon’s Commercial Cloud Service. Otherwise, leave it at Default.
  5. For Default Region, enter the AWS region to use as the default. You can change this later for a particular service.
  6. For Default Availability Zone, enter the AWS Availability Zone to use as the default. This default Availability Zone will only be used where Multi-AZ is not selected.
  7. The broker automatically creates AWS tags for the service plan GUID and the org-space GUID. You can also configure default tags to add for each AWS resource by entering them in the Default Tags text box, separated by a space. For example, owner=operations env=production app=payroll.
  8. Click Save.

Broker Config

  1. Click Broker Config. Aws sb brokerconfig
  2. For App Instances, enter the number of instances of the service broker you want to run. The default is 2.
  3. Do not change the Broker IAM Policy Name unless you configured a custom policy name earlier.
  4. For Backing Database, choose the database type for the Service Broker Configuration.
  5. If you chose Pivotal MySQL enter the Plan Name to be used (e.g. 100mb) and a unique database name. The database will be created during deployment if it does not exist.
  6. If you chose External MySQL or External Postgres:
    1. For Database Host enter the IP address of the database where the Service Broker for AWS will store configuration information. You can use the database you created in the AWS RDS console in Step 1, or choose to use an existing database.
    2. For Database Port, enter the port number of your database. The default for PostgreSQL is 5432 and the default for MySQL is 3306.
    3. For Database Username and Database Password, enter your database credentials.
    4. For Database Name, enter the name of the database.
  7. Under Database SSL Connection, select Enabled if you want to encrypt traffic to your database and add your custom root certificate in the Root Certificate text box. Otherwise, select Disabled.
  8. Click Save.

Note: in the following sections that configure AWS Services, if you don’t want to offer that specific service in your PCF Marketplace then simply uncheck the box Enable in Marketplace

DynamoDB Config

  1. Click DynamoDB Config. Aws sb dynamodbconfig

  2. Under Override Default AWS Region select the default AWS region.

RDS Config

  1. Click RDS Config. Aws sb rdsconfig

  2. Under Override Default AWS Region select the default AWS region.

  3. Under Override Default AWS Availability Zone select the default Availability Zone for RDS database instances.

  4. The RDS Instance Prefix is set to “cf” by default. It is the prefix for creating RDS database instances.

  5. The Service Key Policy Name has a default setting. Do not change this unless you created a custom policy earlier.

  6. The Networking section allows you to configure whether to use the Default VPC or a Custom VPC, and whether the instance is publicly accessible. For a Custom VPC, enter the DB Subnet Group Name and the VPC Security Group(s).

  7. The setting Enable Storage Encryption is off by default. You can enable RDS encryption for a database instance. However, not all regions and DB instance classes support encryption so if you enable this option, you will need to change the default plans to remove those that don’t support encryption. More information is available here.

  8. Select the checkboxes to enable the RDS Services that you want to show in Marketplace

  9. Under Plans, for each RDS Database, the Service Broker for AWS includes several default plans for use.

    • basic: For small projects and during development.
    • standard: For a small production database (multi-AZ, 2vCPU, 7.5GB).
    • premium: For a mid-sized database (multi-AZ, 4 vCPU, 15GB).
    • enterprise: For a large database (multi-AZ, 8 vCPU, 32GB).
  10. You can view and edit the configuration of existing plans by clicking the arrow next to their names. Aws sb editplan

    Note: If you edit an existing service plan, only future instances of that services plan will be changed. Ask your Admin to modify existing instances.

  11. You can also create a new custom plan by clicking the Add button and completing the following fields:

    • Plan Name: The name of the plan in the Services Marketplace.
    • Plan Description: The description of the plan in the Services Marketplace.
    • Plan Features: The list of features displayed in Apps Manager.
    • Engine: This must be postgres or 'mysql’.
    • Engine Version: The version that your database uses.
    • DB Instance Class: The hardware that runs the database.
    • Allocated Storage (GB): The amount of storage allocated to the database.
    • Storage Type: This can be Magnetic, SSD or Provisioned IOPS (io1).
    • IOPS: The IOPS rate for the database instance. Only required if your Storage Type is io1.
    • Backup Retention Period: How long to retain backups (typically 7 days).
    • Preferred Backup Window: When to backup, must be at least 30min window. e.g. 13:00-14:00
    • Copy Tags to Snapshot: Whether to tag the database snapshots.
    • Multi AZ: Whether to automatically run in multiple Availability Zones.
    • The setting Enable Storage Encryption is off by default. If you have not enabled it globally, you can configure it per service plan.
    • For flexibility, the User Privileges allows configuration of database user privileges by service plan. Use this to secure the privileges you want for your developers or apps.
    • DB Parameter Group Name allows you to associate an existing DB Parameter Group Name with this service plan. Use the AWS Console to create the DB Parameter Group.

    Aws sb addplan

  12. Click Save.

S3 Config

  1. Click S3 Config. Aws sb s3config
  2. Under Override Default AWS Region, select the default AWS region for this service.
  3. Select Enable in Marketplace to enable the service in the Services Marketplace.
  4. The Service Key Policy Name has a default setting. Do not change this unless you created a custom policy earlier.
  5. The Quota Limit limits the total number of instances, setting it to 0 will disable it.
  6. Enter the Bucket Prefix, a short name to prefix all bucket names.
  7. Click Installation Dashboard to return to the Ops Manager Installation Dashboard.
  8. Click Apply Changes to install the product.

SQS Config

  1. Click SQS Config. Aws sb sqsconfig
  2. Under Override Default AWS Region, select the default AWS region for this service.
  3. Select Enable in Marketplace to enable the service in the Services Marketplace.
  4. The Service Key Policy Name has a default setting. Do not change this unless you created a custom policy earlier.
  5. The Quota Limit limits the total number of instances, setting it to 0 will disable it.
  6. For Queue Prefix, enter the queue name prefix that will be used for filtering list results.
  7. For Default Visibility Timeout, enter the length of time during which the queue will be unavailable once a message is delivered from the queue.
  8. For Message Retention Period, enter the number of seconds Amazon SQS retains a message.
  9. For Max Message Size, enter the limit of how many bytes a message can contain before Amazon SQS rejects it.
  10. For Delivery Delay, enter the time in seconds that the delivery of all messages in the queue will be delayed.
  11. For Rx Message Wait Time, enter the duration, in seconds, that the receiver action waits until a message is in the queue.
  12. The Redrive Policy specifies an existing dead letter queue to receive messages after the source queue fails to process a message a specified number of times. If enabled, this policy includes the following properties:
    • Dead Letter Queue: The Amazon Resource Name (ARN) of the dead letter queue.
    • Max Receives: The maximum number of times a message is delivered to the source queue before being sent to the dead letter queue.
  13. Click Apply Changes to install the product.
  14. Click Installation Dashboard to return to the Ops Manager Installation Dashboard.

More information on AWS SQS queue properties is available here. For more information on SQS Redrive Policy, refer to the documentation on Using Amazon SQS Dead Letter Queues

Step 5: Confirm Installation

Note: The Service Broker for AWS installs an app called aws-services-broker in the iaas-brokers space of the system org. Like other CF apps, you can view the application logs via the CF CLI by connecting to that org and space with the correct permissions.

  1. After Ops Manager finishes the installation, the Service Broker for AWS appears as a green tile in the Installation Dashboard.

    Aws sb install

  2. In Apps Manager, all orgs and spaces show the new services in the Services Marketplace. Users can create instances of these services through Apps Manager or via the cf CLI. Review the Creating and Managing Service Instances topic for more information.

    Aws sb marketplace

Create a pull request or raise an issue on the source for this page in GitHub