LATEST VERSION: 1.4 - RELEASE NOTES
PCF Service Broker for AWS v1.4

Installing the Service Broker

This topic describes how to install the Service Broker for AWS by configuring your AWS account and importing the product file into Ops Manager.

Installation Prerequisites

  • Pivotal Cloud Foundry (PCF) v2.3 or later
  • An Amazon Web Services (AWS) account
  • MySQL for PCF v2, or an existing MySQL or PostgreSQL database

Note: Pivotal Cloud Foundry (PCF) products and tiles released after July 2018 require Ubuntu Xenial stemcells instead of Ubuntu Trusty stemcells.

Step 1: Set Up AWS

In this step, you configure your AWS account to allow the Service Broker for AWS to create and manage AWS resources.

Create a new IAM user for the PCF cluster in the AWS Identity & Access Management console by following the procedure below.

Note: If you have more than one PCF deployment, you must create an IAM user for each deployment. Create a new IAM user in your AWS account (do not use the same IAM user that installed PCF on AWS as it has different policy restrictions).

  1. Select Policies and click Create Policy.

  2. Choose Create Your Own Policy.

  3. Enter PCFInstallationPolicy for Policy Name and Installation Policy for PCF for Description.

  4. Copy the following text into the Policy Document section:

        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": [
                        "s3:CreateBucket",
                        "s3:DeleteBucket",
                        "s3:PutBucketAcl",
                        "s3:PutBucketLogging",
                        "s3:PutBucketPolicy",
                        "s3:PutBucketTagging",
                        "s3:GetObject",
                        "s3:ListBucket",
                        "iam:CreateAccessKey",
                        "iam:CreateUser",
                        "iam:GetUser",
                        "iam:DeleteAccessKey",
                        "iam:DeleteUser",
                        "iam:DeleteUserPolicy",
                        "iam:ListAccessKeys",
                        "iam:ListAttachedUserPolicies",
                        "iam:ListUserPolicies",
                        "iam:ListPolicies",
                        "iam:PutUserPolicy",
                        "iam:GetPolicy",
                        "iam:GetAccountAuthorizationDetails",
                        "rds:CreateDBCluster",
                        "rds:CreateDBInstance",
                        "rds:DeleteDBCluster",
                        "rds:DeleteDBInstance",
                        "rds:DescribeDBClusters",
                        "rds:DescribeDBInstances",
                        "rds:DescribeDBSnapshots",
                        "rds:DeleteDBSnapshot",
                        "rds:CreateDBParameterGroup",
                        "rds:ModifyDBParameterGroup",
                        "rds:DeleteDBParameterGroup",
                        "dynamodb:ListTables",
                        "dynamodb:DeleteTable",
                        "dynamodb:DescribeTable",
                        "sqs:CreateQueue",
                        "sqs:DeleteQueue"
                    ],
                    "Effect": "Allow",
                    "Resource": "*"
                }
            ]
        }
    
  5. Click Validate Policy to check for errors.

  6. Click Create Policy. This creates an AWS policy that you must apply to every IAM user you create for the Service Broker for AWS.

  7. For each IAM user, click the user name, go to Permissions > Attach Policy, and select PCFInstallationPolicy.

  8. Record the AWS Access Key ID and Secret Access Key for each of your PCF IAM accounts for later use.

Note: The Service Broker for AWS also allows a custom name for the PCFInstallationPolicy.

To use the default name: Follow the steps above for configuring the PCFInstallationPolicy policy.

To use a custom name: In the steps above, use a custom name instead of PCFInstallationPolicy. Later, during Broker Configuration, you will need to enter this custom name.

Service Key Policies

The Service Broker for AWS allows App Developers to create service keys for their service instances. Creating a service key creates an IAM user with a templated policy, and provides the App Developer with Access Key credentials to the actions listed in the policies below.

The service key Policy names are configurable. You can choose your own policy name and configure that name in each service by updating the Service Key Policy Name field. The default names are used below.

To create the policy templates for each service that you will enable, log into the AWS console and do the following:

  1. Select Policies and click Create Policy.

  2. Choose Create Your Own Policy.

  3. Enter PCFAppDeveloperPolicy-s3 for Policy Name and Service Broker for AWS Service Key policy for S3 for Description.

  4. Enter the following text in the Policy Document section, or create your own policy template: { “Version”: “2012-10-17”, “Statement”: [ { “Sid”: “allowtagging”, “Effect”: “Allow”, “Action”: [ “s3:GetBucketTagging”, “s3:PutBucketTagging” ], “Resource”: [ “arn:aws:broker:resource::” ] } ] }

  5. Click Validate Policy to check for errors.

  6. Click Create Policy. This creates an AWS policy that you must apply to every IAM user you create for the Service Broker for AWS.

  7. Repeat steps 1-6 for PCFAppDeveloperPolicy-sqs, PCFAppDeveloperPolicy-dynamodb, and PCFAppDeveloperPolicy-rds, using or modifying the examples below.

PCFAppDeveloperPolicy-sqs Example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1471890189000",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues",
                "sqs:PurgeQueue",
                "sqs:ReceiveMessage",
                "sqs:SendMessage"
            ],
            "Resource": [
                "arn:aws:broker:resource::"
            ]
        }
    ]
}

PCFAppDeveloperPolicy-dynamodb Example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1471873911000",
            "Effect": "Allow",
            "Action": [
                "dynamodb:*"
            ],
            "Resource": [
                "arn:aws:broker:resource::"
            ]
        }
    ]
}

PCFAppDeveloperPolicy-rds Example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1471636048000",
            "Effect": "Allow",
            "Action": [
                "rds:ListTagsForResource",
                "rds:DescribeDbInstances"
            ],
            "Resource": [
                "arn:aws:broker:resource::"
            ]
        }
    ]
}

Step 2: Create Service Broker Database

To store its configuration information, the Service Broker for AWS requires access to a MySQL or PostgreSQL database that is highly available and regularly backed up. Loss of this data will severely affect the ability of the service broker to manage the configured service instances. However, the AWS resources will not be affected.

The database can be MySQL for PCF v2 or an external MySQL or PostgreSQL database, including an AWS RDS database.

If you choose to use a MySQL for PCF v2 database, the service broker will automatically provision the database.

If you choose to use an RDS MySQL or RDS PostgreSQL database, do the following:

  1. In the AWS RDS console, create the appropriate database:

    • Choose the smallest DB instance class, db.t2.micro.
    • Select yes for Multi-AZ Deployment.
    • Choose Provisioned IOPS (SSD) for Storage Type. For a non-production database, you can choose General Purpose (SSD).
    • Enter the minimum accepted value ‘5’ to '100’ GB of Allocated Storage.
    • Leave Provisioned IOPS at the default value.
    • Enter pcf-aws-services for the DB Instance Identifier.
    • Enter suitable values for the Master Username and Master Password.
  2. Record the Database IP, Database Port, Admin User, Admin Password, and Database Name. The Service Broker for AWS stores configuration information in this database.

  3. Ensure that the Service Broker for AWS will be able to connect to your database. If necessary for your deployment, modify your AWS VPC settings to allow access to the RDS database from an external IP address. For more information, see the Accessing a DB Instance in a VPC topic in the AWS documentation.

Step 3: Application Security Groups

To allow this service to have network access, you must create Application Security Groups (ASGs).

Note: Without Application Security Groups, the service cannot be used.

Service Container Network Connections

This service is deployed as an application to the iaas-brokers space in the system org, and requires the following outbound network connections:

Destination Ports Protocol Reason
AWS_IP_RANGE 443, 5432, 1521, 1433, 3306 tcp This service accesses RDS/S3

Note: The AWS IP range is available from Amazon. You may choose to use curl and jq to convert the IP range as downloaded from Amazon into ASG rules:

    curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | \
    jq '[
      {
        ports: "443, 5432, 1521, 1433, 3306",
        protocol: "tcp",
        destination: .prefixes | map(select(.service == "AMAZON")) | .[] | .ip_prefix
      }
    ] | .' > aws-service-broker.json

Note: This creates a configuration which allows traffic for all AWS Regions. You may wish to further filter the list of entries in the Application Security Group config to a specific set of AWS Regions by adding additional jq map(select(.region == "REGION_NAME")) statements for each region.

Log in as an administrator and create an ASG called aws-service-broker, binding it to the iaas-brokers space in the system org:

$ cf create-security-group aws-service-broker aws-service-broker.json
$ cf bind-security-group aws-service-broker system iaas-brokers

Application Container Network Connections

Application containers that use instances of this service require the following outbound network connections:

Destination Ports Protocol Reason
AWS_IP_RANGE 443, 5432, 1521, 1433, 3306 tcp Application containers may access S3/RDS

Create an ASG called aws-service-broker-app-containers with the above configuration and bind it to the appropriate space or, to give all started apps access, the default-running ASG set. For example:

[
  {
      "ports": "443, 5432, 1521, 1433, 3306",
      "protocol": "all",
      "destination": "192.0.2.0/14"
  },
  {
      "ports": "443, 5432, 1521, 1433, 3306",
      "protocol": "all",
      "destination": "198.51.100.0/22"
  },
  ...rest of AWS IPs elided...
]

Step 4: Install in Ops Manager

To install the Service Broker for AWS tile in Ops Manager, do the following:

  1. Download the Service Broker for AWS product file from Pivotal Network.

  2. From the Ops Manager Installation Dashboard, click Import a Product to upload the product file.

  3. Click Add next to the uploaded product description in the Available Products view to add this product to your staging area.

  4. Click on the newly added tile to configure the sections as described below.

AWS Config

The setup in Step 1 must be successfully completed before beginning this configuration. You need the AWS and database parameters from that step to successfully configure the broker.

  1. Click AWS Config. Aws sb awsconfig

  2. Enter the AWS Access Key ID and the Secret Access Key from Step 1.

  3. Enter the Temporary Credentials Timeout value in minutes. This is the time after which service key credentials will expire. To disable expiry of the credentials, enter 0.

  4. Modify the AWS Endpoint only if you are using Amazon’s Commercial Cloud Service. Otherwise, leave it set to the default.

  5. For Default Region, enter the AWS region to use as the default. You can change this later for a particular service.

  6. For Default Availability Zone, enter the AWS Availability Zone to use as the default. This default Availability Zone is only used where Multi-AZ is not selected.

  7. The broker automatically creates AWS tags for the service plan GUID and the org-space GUID. You can also configure default tags to add for each AWS resource by entering them in the Default Tags text box, separated by a space. For example, owner=operations env=production app=payroll.

  8. Click Save.

Broker Config

  1. Click Broker Config. Aws sb brokerconfig

  2. For App Instances, enter the number of instances of the service broker you want to run. The default is 2.

  3. Do not change the Broker IAM Policy Name unless you configured a custom policy name earlier.

  4. For Backing Database, choose the database type for the Service Broker Configuration.

  5. If you chose Pivotal MySQL (v2), enter the Plan Name to be used (e.g. db-small) and a unique database name. The database is created during deployment if it does not already exist.

  6. If you chose External MySQL or External Postgres:

    1. For Database Host enter the IP address of the database where the Service Broker for AWS will store configuration information. You can use the database you created in the AWS RDS console in Step 1, or choose to use an existing database.
    2. For Database Port, enter the port number of your database. The default for PostgreSQL is 5432, and the default for MySQL is 3306.
    3. For Database Username and Database Password, enter your database credentials.
    4. For Database Name, enter the name of the database.
  7. Under Database SSL Connection, select Enabled if you want to encrypt traffic to your database and add your custom root certificate in the Root Certificate text box. Otherwise, select Disabled.

  8. Click Save.

Note: In the following sections that configure AWS Services, if you do not want to offer that specific service in your PCF Marketplace, disable the Enable in Marketplace checkbox.

DynamoDB Config

  1. Click DynamoDB Config. Aws sb dynamodbconfig

  2. Under Override Default AWS Region, select the default AWS region.

RDS Config

  1. Click RDS Config. Aws sb rdsconfig

  2. Under Override Default AWS Region, select the default AWS region.

  3. Under Override Default AWS Availability Zone, select the default Availability Zone for RDS database instances.

  4. The RDS Instance Prefix is set to cf by default. This is the prefix for creating RDS database instances.

  5. The Service Key Policy Name has a default setting. Do not change this unless you created a custom policy earlier.

  6. The Networking section allows you to configure whether to use the Default VPC or a Custom VPC, and whether the instance is publicly accessible. For a Custom VPC, enter the DB Subnet Group Name and the VPC Security Group(s).

  7. The Enable Storage Encryption setting is disabled by default. You can enable RDS encryption for a database instance. However, not all regions and database instance classes support encryption. If you enable this setting, you must change the default plans to remove those that do not support encryption. For more information, see http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.html#d0e21710.

  8. Select the checkboxes of the RDS Services you want to show in the Services Marketplace.

  9. Under Plans, for each RDS Database, the Service Broker for AWS includes several default plans for use.

    • basic: For small projects and during development.
    • standard: For a small production database (multi-AZ, 2vCPU, 7.5 GB).
    • premium: For a mid-sized database (multi-AZ, 4 vCPU, 15 GB).
    • enterprise: For a large database (multi-AZ, 8 vCPU, 32 GB).
  10. You can view and edit the configuration of existing plans by clicking the arrow next to their names. Aws sb editplan

    Note: If you edit an existing service plan, only future instances of that services plan will be changed. Ask your administrator to modify existing instances.

  11. You can also create a new custom plan by clicking the Add button and completing the following fields:

    • Plan Name: The name of the plan in the Services Marketplace.
    • Plan Description: The description of the plan in the Services Marketplace.
    • Plan Features: The list of features displayed in Apps Manager.
    • Engine: This must be postgres or mysql.
    • Engine Version: The version that your database uses.
    • DB Instance Class: The hardware that runs the database.
    • Allocated Storage (GB): The amount of storage allocated to the database.
    • Storage Type: This can be Magnetic, SSD, or Provisioned IOPS (io1).
    • IOPS: The IOPS rate for the database instance. Only required if your Storage Type is io1.
    • Backup Retention Period: How long to retain backups (typically 7 days).
    • Preferred Backup Window: When to back up. Must be at least a 30-minute window (e.g. 13:00-14:00).
    • Copy Tags to Snapshot: Whether to tag the database snapshots.
    • Multi AZ: Whether to automatically run in multiple Availability Zones.
    • The Enable Storage Encryption setting is disabled by default. If you have not enabled it globally, you can configure it per service plan.
    • For flexibility, the User Privileges allows configuration of database user privileges by service plan. Use this to secure the privileges you want for your developers or apps.
    • DB Parameter Group Name allows you to associate an existing DB Parameter Group Name with this service plan. Use the AWS Console to create the DB Parameter Group.

    Aws sb addplan

  12. Click Save.

S3 Config

  1. Click S3 Config. Aws sb s3config

  2. Under Override Default AWS Region, select the default AWS region for this service.

  3. Select Enable in Marketplace to enable the service in the Services Marketplace.

  4. The Service Key Policy Name has a default setting. Do not change this unless you created a custom policy earlier.

  5. The Quota Limit limits the total number of instances. Setting it to 0 will disable it.

  6. Enter the Bucket Prefix, a short name to prefix all bucket names.

  7. S3 has one default plan called standard, but in Service Broker for AWS v1.4 and later, you can add more. A service plan can require an S3 bucket to be encrypted by creating an AWS KMS Key (see AWS SSE-KMS) and entering the KMS key ARN in the service plan configuration. Aws sb s3plans

  8. Click Save to store the changed settings.

SQS Config

  1. Click SQS Config. Aws sb sqsconfig

  2. Under Override Default AWS Region, select the default AWS region for this service.

  3. Select Enable in Marketplace to enable the service in the Services Marketplace.

  4. The Service Key Policy Name has a default setting. Do not change this unless you created a custom policy earlier.

  5. The Quota Limit limits the total number of instances. Setting it to 0 will disable it.

  6. For Queue Prefix, enter the queue name prefix that will be used for filtering list results.

  7. For Default Visibility Timeout, enter the length of time during which the queue will be unavailable once a message is delivered from the queue.

  8. For Message Retention Period, enter the number of seconds Amazon SQS retains a message.

  9. For Max Message Size, enter the limit of how many bytes a message can contain before Amazon SQS rejects it.

  10. For Delivery Delay, enter the time in seconds that the delivery of all messages in the queue will be delayed.

  11. For Rx Message Wait Time, enter the duration, in seconds, that the receiver action waits until a message is in the queue.

  12. The Redrive Policy specifies an existing dead letter queue to receive messages after the source queue fails to process a message a specified number of times. If enabled, this policy includes the following properties:

    • Dead Letter Queue: The Amazon Resource Name (ARN) of the dead letter queue.
    • Max Receives: The maximum number of times a message is delivered to the source queue before being sent to the dead letter queue.
  13. Click Installation Dashboard to return to the Ops Manager Installation Dashboard.

  14. If you are using Ops Manager v2.3 or later, click Review Pending Changes. For more information about this Ops Manager page, see Reviewing Pending Product Changes.

  15. Click Apply Changes to install the product.

For more information about AWS SQS queue properties, see AWS::SQS::Queue in the AWS documentation. For more information about SQS Redrive Policy, see Using Amazon SQS Dead Letter Queues in the AWS documentation.

Step 5: Confirm Installation

Note: The Service Broker for AWS installs an app named aws-services-broker in the iaas-brokers space of the system org. Like other Cloud Foundry apps, you can view the application logs via the cf CLI by connecting to that org and space with the correct permissions.

  1. After Ops Manager finishes the installation, the Service Broker for AWS appears as a green tile in the Installation Dashboard. Aws sb install

  2. In Apps Manager, all orgs and spaces show the new services in the Services Marketplace. Users can create instances of these services through Apps Manager or via the cf CLI. For more information, see Creating and Managing Service Instances. Aws sb marketplace

Create a pull request or raise an issue on the source for this page in GitHub