Creating and Managing Service Instances

Overview

This topic describes how developers set up, operate, and scale Amazon Web Services (AWS) resources from Pivotal Cloud Foundry (PCF) by creating and managing service instances using the Service Broker for AWS.

PCF operators must follow the instructions in the Installing the Service Broker topic to install the Service Broker for AWS before developers can use it. During installation, operators configure which AWS services they want to make available to developers in the Services Marketplace. They can also set up pre-defined service plans with specific resource configurations and securely manage their AWS credentials.

The current version of the Service Broker for AWS supports the following services:

  • AWS RDS for PostgreSQL
  • AWS S3
  • AWS RDS for MySQL
  • AWS RDS for Aurora
  • AWS RDS for SQL Server
  • AWS DynamoDB
  • AWS RDS for Oracle
  • AWS RDS for MariaDB
  • AWS SQS

Developers create and manage service instances of the Service Broker for AWS through the cf CLI or Apps Manager.

To perform the following procedures for creating and managing service instances, a developer must be logged in to the PCF deployment via the cf CLI.

View Services

  1. List available Marketplace Services with cf marketplace to show information about the Service Broker for AWS services.

    $ cf marketplace
    Getting services from marketplace in org system / space iaas-brokers as admin...
    OK
    service             plans                                      description
    app-autoscaler bronze, gold Scales bound applications in response to load
    aws-dynamodb standard* Create and manage Amazon DynamoDB tables
    aws-rds-aurora basic*, standard*, premium*, enterprise* Create and manage AWS RDS Aurora deployments
    aws-rds-mariadb basic*, standard*, premium*, enterprise* Create and manage AWS RDS MariaDB deployments
    aws-rds-mysql basic*, standard*, premium*, enterprise* Create and manage AWS RDS MySQL deployments
    aws-rds-oracle basic*, standard*, premium*, enterprise* Create and manage AWS RDS Oracle deployments
    aws-rds-postgres basic*, standard*, premium*, enterprise* Create and manage AWS RDS PostgreSQL deployments
    aws-rds-sqlserver basic*, standard*, premium*, enterprise* Create and manage AWS RDS SQL Server deployments
    aws-s3 standard* Create and manage Amazon S3 buckets
    aws-sqs standard* Create and manage Amazon SQS queues

    Note: These service plans have an associated cost. Creating a service instance will incur this cost.

  2. View descriptions for the plans of a service with cf marketplace -s SERVICE.

    $ cf marketplace -s aws-rds-postgres
    Getting service plan information for service aws-rds-postgres as admin...
    OK

    service plan description free or paid basic For small projects and during development. free standard For a small production database, multi-AZ, 2vCPU, 7.5GB free premium For a mid-sized database, multi-AZ, 4 vCPU, 15GB free enterprise For a large database, multi-AZ, 8 vCPU, 32GB free

Create Service Instances

Note: When installing the Service Broker for AWS, the PCF operator sets up an IAM account and policy and provides credentials to the Service Broker. Developers do not need access to the credentials to create service instances.

RDS for PostgreSQL

To create a service instance of the RDS for PostgreSQL service, use cf create-service to create an instance of aws-rds-postgres with or without custom settings.

To create an instance of aws-rds-postgres without custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE. The following example creates an instance named mypostgres1 with the standard plan:

$ cf create-service aws-rds-postgres standard mypostgres1

To create an instance of aws-rds-postgres with custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE with the -c flag and provide custom settings for the following elements:

  • Engine Version
  • Multi-AZ
  • Storage Type
  • AllocatedStorage
  • AvailabilityZone

The following example shows the syntax for each setting. You can omit settings you don’t want to explicitly set:

$ cf create-service aws-rds-postgres basic postgresdb -c '{ "CreateDbInstance": { "EngineVersion": "9.4.1", "MultiAZ": false, "StorageType": "gp2", "AllocatedStorage": 10, "AvailabilityZone": "us-east-1a", "Tags": [{"Key": "owner", "Value": "operations"}, {"Key": "Env", "Value": "staging"} ] } }'

RDS for MySQL

To create a service instance of the RDS for MySQL service, use cf create-service to create an instance of aws-rds-mysql with or without custom settings.

To create an instance of aws-rds-mysql without custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE. The following example creates an instance named mysqldb1 with the standard plan:

$ cf create-service aws-rds-mysql standard mysqldb1

To create an instance of aws-rds-mysql with custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE with the -c flag and provide custom settings for the following elements:

  • Engine Version
  • Multi-AZ
  • Storage Type
  • AllocatedStorage
  • AvailabilityZone

The following example shows the syntax for each setting. You can omit settings you don’t want to explicitly set:

$ cf create-service aws-rds-mysql basic mysqldb2 -c '{ "CreateDbInstance": { "EngineVersion": "5.6.27", "MultiAZ": false, "StorageType": "gp2", "AllocatedStorage": 20, "AvailabilityZone": "us-east-1a", "Tags": [{"Key": "owner", "Value": "operations"}, {"Key": "Env", "Value": "staging"} ] } }'

RDS for MariaDB

To create a service instance of the RDS for MariaDB service, use cf create-service to create an instance of aws-rds-mariadb with or without custom settings.

To create an instance of aws-rds-mariadb without custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE. The following example creates an instance named mariadbdb1 with the standard plan:

$ cf create-service aws-rds-mariadb standard mariadbdb1

To create an instance of aws-rds-mariadb with custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE with the -c flag and provide custom settings for the following elements:

  • Engine Version
  • Multi-AZ
  • Storage Type
  • AllocatedStorage
  • AvailabilityZone

The following example shows the syntax for each setting. You can omit settings you don’t want to explicitly set:

$ cf create-service aws-rds-mariadb basic mariadbdb2 -c '{ "CreateDbInstance": { "EngineVersion": "10.0.24", "MultiAZ": false, "StorageType": "gp2", "AllocatedStorage": 20, "AvailabilityZone": "us-east-1a", "Tags": [{"Key": "owner", "Value": "operations"}, {"Key": "Env", "Value": "staging"} ] } }'

RDS for Aurora

To create a service instance of the RDS for Aurora service, use cf create-service to create an instance of aws-rds-aurora with or without custom settings.

To create an instance of aws-rds-aurora without custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE. The following example creates an instance named auroradb1 with the standard plan:

$ cf create-service aws-rds-aurora standard auroradb1

To create an instance of aws-rds-aurora with custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE with the -c flag and provide custom settings for the following elements:

  • Multi-AZ
  • AvailabilityZone

The following example shows the syntax for each setting. You can omit settings you don’t want to explicitly set:

$ cf create-service aws-rds-aurora basic auroradb2 -c '{ "CreateDbInstance": { "MultiAZ": false, "AvailabilityZone": "us-east-1a", "Tags": [{"Key": "owner", "Value": "operations"}, {"Key": "Env", "Value": "staging"} ] } }'

RDS for SQL Server

To create a service instance of the RDS for SQL Server service, use cf create-service to create an instance of aws-rds-sqlserver with or without custom settings.

To create an instance of aws-rds-sqlserver without custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE. The following example creates an instance named sqlserverdb1 with the standard plan:

$ cf create-service aws-rds-sqlserver standard sqlserverdb1

To create an instance of aws-rds-sqlserver with custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE with the -c flag and provide custom settings for the following elements:

  • Engine Version
  • Multi-AZ
  • Storage Type
  • AllocatedStorage
  • AvailabilityZone

The following example shows the syntax for each setting. You can omit settings you don’t want to explicitly set:

$ cf create-service aws-rds-sqlserver basic sqlserverdb2 -c '{ "CreateDbInstance": { "EngineVersion": "12.00.4422.0.v1", "MultiAZ": true, "StorageType": "gp2", "AllocatedStorage": 20, "AvailabilityZone": "us-east-1a", "Tags": [{"Key": "owner", "Value": "operations"}, {"Key": "Env", "Value": "staging"} ] } }'

Note: For SQL Server setting Multi-AZ to true will enable Multi-AZ database mirroring. See the AWS documentation on Multi-AZ Deployments for Microsoft SQL Server with Database Mirroring for more details.

RDS for Oracle Database

To create a service instance of the RDS for Oracle service, use cf create-service to create an instance of aws-rds-oracle with or without custom settings.

To create an instance of aws-rds-oracle without custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE. The following example creates an instance named oracledb1 with the standard plan:

$ cf create-service aws-rds-oracle standard oracledb1

To create an instance of aws-rds-oracle with custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE with the -c flag and provide custom settings for the following elements:

  • Engine Version
  • Multi-AZ
  • Storage Type
  • AllocatedStorage
  • AvailabilityZone

The following example shows the syntax for each setting. You can omit settings you don’t want to explicitly set:

$ cf create-service aws-rds-oracle basic oracledb1 -c '{ "CreateDbInstance": { "EngineVersion": "12.1.0.2.v3", "MultiAZ": false, "StorageType": "gp2", "AllocatedStorage": 20, "AvailabilityZone": "us-east-1a", "Tags": [{"Key": "owner", "Value": "operations"}, {"Key": "Env", "Value": "staging"} ] } }'

DynamoDB

You can create and add items to NoSQL tables with DynamoDB. You can do this programmatically or through the AWS DynamoDB console. Pivotal recommends creating an IAM user at bind time and configuring access to a set of prefixed tables for operations such as creating a table and adding an item.

To create a service instance of the AWS DynamoDB service, use cf create-service to create an instance of aws-dynamodb.

To create an instance of aws-dynamodb, use cf create-service SERVICE PLAN SERVICE-INSTANCE. The following example creates an instance named mydynamodb with the standard plan:

$ cf create-service aws-dynamodb standard MYDYNAMODB

Binding an app to a DynamoDB service instance creates an IAM User that can create tables with a prefix corresponding to the guid of the service instance. You can find the credentials to create DynamoDB tables programatically in the VCAP_SERVICES environment variable of the app that service is bound to, along with the table prefix and region. For example:

{
 "VCAP_SERVICES": {
  "aws-dynamodb": [
   {
    "credentials": {
     "access_key_id": "AKIAIOSFODNN7EXAMPLE",
     "prefix": "0110c45b-e0be-4b05-8aa5-7bafd5866dff_",
     "region": "us-east-1",
     "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "label": "aws-dynamodb",
    "name": "mydynamodb",
    "plan": "standard",
    "provider": null,
   }
  ]
 }
}

The tables you create are only accessible from the app that service is bound to. Each bind creates an IAM user with full DynamoDB permissions on only the prefixed tables, for example arn:aws:dynamodb:REGION:ACCOUNT_ID:table/PREFIX_*.

S3

To create a S3 bucket, use cf create-service to create an instance of aws-s3 with or without custom settings.

To create an S3 bucket without custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE. The following example creates an instance named bucket1 with the standard plan:

$ cf create-service aws-s3 standard bucket1

Note: S3 service instances use default region and tags settings configured by the PCF operator during the installation of the Service Broker for AWS.

To create an S3 bucket with custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE with the -c flag. For instance, you can use custom settings to create a bucket in a specific region to lower latency for users. The following example creates a bucket in the Tokyo region with tag values k1=v1:

$ cf cs aws-s3 standard tokyobucket -c '{ "CreateBucket": { "CreateBucketConfiguration": { "LocationConstraint": "ap-northeast-1"} }, "PutBucketTagging": { "Tagging":{ "TagSet": [ {"Key" : "k1", "Value": "v1"}]}} }'

Note: Developers can supply additional configuration parameters to service instances at update and bind times.

SQS

To create an SQS queue, use cf create-service to create an instance of aws-sqs with or without custom settings.

To create an SQS bucket without custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE. The following example creates an instance named queue1 with the standard plan:

$ cf create-service aws-sqs standard queue1

Note: SQS queue instances use default region and property settings configured by the PCF operator during the installation of the Service Broker for AWS.

To create an SQS queue with custom settings, use cf create-service SERVICE PLAN SERVICE-INSTANCE with the -c flag. For instance, you can use custom settings to create a queue with a specific name and maximum message size. The following example creates a queue named “kb-queue” with a 1kb max message size:

$ cf cs aws-sqs standard kbqueue -c '{ "CreateQueue": {  "QueueName": "kb-queue", "Attributes": { "MaximumMessageSize": "1024"} } }'

Bind or Unbind a Service Instance

Binding a RDS database or S3 service instance to an app grants the app access to the RDS database or S3 bucket, and provides credentials in the environment variables. The access permissions are set at the least privilege required.

Run the following command to bind a service instance to an app:

$ cf bind-service YOUR-APP YOUR-SERVICE-INSTANCE

Note: When binding to an RDS database, a new user and password will be created for the binding. Oracle is an exception to this rule, where bound apps will be provided with the master username and password.

Unbinding a service instance from an app removes access to the database and removes database credentials from the environment variables.

Run the following command to unbind a service instance to an app:

$ cf unbind-service YOUR-APP YOUR-SERVICE-INSTANCE

Delete a Service Instance

Note: Before deleting a service instance, ensure there are no apps bound to the service instance.

Run the following command to delete a service instance:

$ cf delete-service YOUR-SERVICE-INSTANCE
Really delete the service YOUR-SERVICE-INSTANCE> y
Deleting service YOUR-SERVICE-INSTANCE in org system / space dev1 as appdev1...
OK
Delete in progress. Use 'cf services' or 'cf service YOUR-SERVICE-INSTANCE' to check operation status.

Using Service Keys for Other Commands

Creating service keys for a service instance allows app developers to perform additional operations against the underlying resources in AWS.

Run the following command to create a service key for a service instance:

$ cf create-service-key YOUR-SERVICE-INSTANCE SERVICE-KEY-NAME

This command creates a service key for “YOUR-SERVICE-INSTANCE” named “SERVICE-KEY-NAME”.

To view the corresponding credentials, run the following command:

$ cf service-key YOUR-SERVICE-INSTANCE SERVICE-KEY-NAME

This returns credentials in JSON format, containing the Access Key ID and Secret, and the ARN for the underlying resource in AWS. For example, service key credentials for an S3 bucket would return the following:

{
 "access_key_id": "AKIAIOSFODNN7EXAMPLE",
 "arn": "arn:aws:s3:::bucket-name",
 "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

Note: When creating a service key, a new IAM user will be created with a policy defined by the Operator. This policy can be configured to optionally time out after a time period specified by the Operator

App developers can use these keys to perform one-off tasks (creating RDS Snapshots, modifying parameter groups, adding permissions, etc.) against the underlying resource using the AWS CLI. The actions permitted are defined by the Operator using service key policy templates.

Run the following command to delete the service key:

$ cf delete-service-key YOUR-SERVICE-INSTANCE SERVICE-KEY-NAME

Deleting a service key removes access to the service instance, and deletes the underlying IAM user.

Troubleshooting

Problem

As a PCF operator, when deploying the tile by clicking Apply Changes, I get the following error: “A client error (InvalidClientTokenId) occurred when calling the GetUser operation: The security token included in the request is invalid.”

Reason

The AWS credentials are not valid, please recheck them or recreate the credentials.

Problem

As a PCF operator, when deleting the product tile, I get the following error: “Can not remove brokers that have associated service instances”.

Reason

Your service broker currently has service instances that are active. They must be deleted before the tile can be deleted.

Problem

As a developer, when trying to create a service, I get the following error: “Service broker error: InvalidClientTokenId: The security token included in the request is invalid.”

Reason

The AWS credentials are not valid, please ask your PCF operator to recheck them or recreate the credentials.

Create a pull request or raise an issue on the source for this page in GitHub