LATEST VERSION: 1.4 - CHANGELOG
Spring Cloud Services v1.4

Operator Information

Page last updated:

See below for information about Spring Cloud® Services’ deployment model and other information which may be useful in operating its services or client applications.

Tile Configuration Options

The Spring Cloud Services tile includes settings for various options. You can configure these by visiting the Installation Dashboard of Pivotal Cloud Foundry® Operations Manager, clicking the Spring Cloud Services tile, and then navigating to the Spring Cloud Services pane.

Configure Service Instance Limit

By default, the Spring Cloud Services broker will only provision up to 100 service instances. This service instance limit is also used to determine the memory quota for the org in which service instance backing applications are deployed; the org’s memory quota will be equal to 1.5G times the service instance limit. You can configure this limit in the Spring Cloud Services tile settings.

Configure service instance limit

In the Service Instance Limit field, enter the new service instance limit. Click Save, return to the Installation Dashboard, and apply your changes. The broker now can provision up to as many service instances as you specified.

Set Push Timeout for Backing Applications

By default, the Spring Cloud Services service broker allows four (4) minutes for a service instance backing application to be staged and start. To use another timeout interval, you can specify a different number of minutes in the Spring Cloud Services tile settings.

Set app push timeout

In the App push timeout field, enter the desired number of minutes. Click Save, return to the Installation Dashboard, and apply the change. The broker will now allow the specified number of minutes for service instance backing applications to start.

Set Buildpack for Service Instances

By default, the Spring Cloud Services service broker stages service instance backing applications using the buildpack chosen by the platform’s buildpack detection; normally, this will be the highest-priority Java buildpack. To cause the broker to use a particular Java buildpack regardless of priority, you can set the name of the buildpack to use in the Spring Cloud Services tile settings.

Set buildpack

In the Buildpack field, enter the name of the desired Java buildpack. Click Save, return to the Installation Dashboard, and apply your changes. The broker will now use the selected buildpack to stage service instance backing applications.

Skip SSL Certificate Validation

By default, Spring Cloud Services checks the Pivotal Cloud Foundry® Elastic Runtime SSL certificate for the domains included in the Security Requirements section of the Prerequisites topic. You can disable this validation by checking the Do not validate that SSL certificates are properly configured checkbox.

Disable ssl certificate validation

Click Save and return to the Installation Dashboard, then apply your changes. Spring Cloud Services will not validate the Elastic Runtime certificate as part of its installation process.

Errand Configuration

The Spring Cloud Services tile includes four lifecycle errands. You can view these by visiting the Installation Dashboard of Pivotal Cloud Foundry® Operations Manager, clicking the Spring Cloud Services tile, and then navigating to the Errands pane.

Errands

Spring Cloud Services includes three post-deploy errands and one pre-delete errand. These errands can be individually set to run conditionally (When Changed), to always run (On), or to never run (Off).

Important: To ensure that your PCF installation receives important updates to the tile, Pivotal recommends that all Spring Cloud Services lifecycle errands be set to On.

In the post-deploy errands, Broker Deployer pushes the Spring Cloud Services service broker and worker applications to PCF. Broker Registrar registers the service broker with the Cloud Controller and makes its service plans available to all orgs. Smoke Tests runs several tests to ensure service availability and correct configuration. The pre-delete errand, Broker Deregistrar, deletes all orgs and spaces specific to Spring Cloud Services and deregisters the broker from the Cloud Controller.

Service Orchestration and Instance Management

Spring Cloud Services provides a series of Managed Services on Pivotal Cloud Foundry (PCF). It uses Cloud Foundry’s Service Broker API to manage the three services—Config Server, Service Registry, and Circuit Breaker Dashboard—that it makes available. See below for information about Spring Cloud Services’s broker implementation and details about how it manages service instances.

Ops1

The Service Broker

The service broker’s functionality is divided between the following two Spring Boot applications, which are deployed in the “system” org to the “p-spring-cloud-services” space.

  • spring-cloud-broker (“the SB”): Implements the Service Broker API.
  • spring-cloud-broker-worker (“the Worker”): Acts on provision, deprovision, bind, and unbind requests.

The broker relies on two other Pivotal Cloud Foundry products, MySQL for Pivotal Cloud Foundry and RabbitMQ® for Pivotal Cloud Foundry, for the following service instances.

  • spring-cloud-broker-db: A MySQL database used by the SB.
  • spring-cloud-broker-rmq: A RabbitMQ queue used for communication between the SB and the Worker.

You can obtain the broker username and password from the Spring Cloud Services tile in Pivotal Cloud Foundry® Operations Manager. Click the Spring Cloud Services tile, and in the Credentials tab, follow the link to the Broker Credentials.

Broker credentials

Broker Upgrades

The Spring Cloud Services product upgrade process checks before redeploying the service broker to see whether the broker applications’ version has changed. If the version has not changed, the upgrade process will continue without redeploying either the SB or the Worker.

The SB application and the Worker application are deployed using a blue-green deployment strategy. During an upgrade of the service broker, the broker will continue processing requests to provision, deprovision, bind, and unbind service instances, without downtime.

Access Broker Applications in Apps Manager

To view the broker applications in Pivotal Cloud Foundry® Apps Manager, log into Apps Manager as an admin user and select the “system” org.

Broker apps apps manager org

The applications are deployed in the “p-spring-cloud-services” space.

Broker apps apps manager space

Service Instances

A service instance is backed by one or more Spring Boot applications deployed by the Worker in the “p-spring-cloud-services” org to the “instances” space.

A service instance is assigned a GUID at provision time. Backing applications for services include the GUID in their names:

Config Server

Service Registry

Circuit Breaker Dashboard

Access Service Instance Backing Applications in Apps Manager

To view a backing application for a service instance in Pivotal Cloud Foundry® Apps Manager, get the service instance’s GUID and look for the corresponding application in the “instances” space.

Target the org and space of the service instance.

$ cf target -o myorg -s development

API endpoint:   https://api.cf.wise.com (API version: 2.43.0)
User:           admin
Org:            myorg
Space:          development

$ cf services
Getting services in org myorg / space development as admin...
OK

name             service             plan       bound apps   last operation
config-server    p-config-server     standard   cook         update succeeded

Run cf service with the CF_TRACE environment variable set to true. Copy the value of resources.metadata.guid, which is the service instance GUID.

$ CF_TRACE=true cf service config-server

REQUEST: [2016-06-27T20:45:24-05:00]
[...]

RESPONSE: [2016-06-27T20:45:25-05:00]
[...]

{
  "total_results": 1,
  "total_pages": 1,
  "prev_url": null,
  "next_url": null,
  "resources": [
    {
      "metadata": {
        "guid": "51711835-4626-4823-b5a1-e5d91012f3f2",

Log into Apps Manager as an admin user and select the “p-spring-cloud-services” org. The applications are deployed in the “instances” space.

Instance apps apps manager org

Look for the backing application named as described above, with the prefix specific to the service type and the service instance GUID.

Instance apps apps manager space

Get Service Instance Information

Each backing application for a service instance has the Spring Boot Actuator info endpoint enabled and accessible at /info. You can obtain version and other information about a service instance by accessing this endpoint.

Locate a backing application for a service instance as described in the Access Service Instance Backing Applications in Apps Manager section. In Apps Manager, on the page for the backing application, click the Route tab and copy the route for the application.

Backing application route

Append /info to the route for the application. In the example above, the resulting URI would be:

https://config-021b3d73-3ffc-4cb8-a105-26f5aaa54dff.wise.com/info

Visit the URI in a browser or request it from another client (e.g. curl at a command line). The response is JSON, as in the following example for a Config Server service instance:

{
  "version":"5",
  "git":{
    "commit":{
      "time":1478639430000,
      "id":"6e139ee"
    },
    "branch":"HEAD"
  }
}

The response may include additional fields depending on the service type, as in the following example for a Service Registry service instance:

{
  "issuer": "https://p-spring-cloud-services.uaa.wise.com/oauth/token",
  "version": "5",
  "nodeCount": "1",
  "git": {
    "commit": {
      "time": 1478639430000,
      "id": "6e139ee"
    },
    "branch": "HEAD"
  },
  "peers": []
}

In addition to the version of the service instance (version), this response includes the OAuth 2 token issuer URI for the service instance (issuer), the number of nodes provisioned for the service instance (nodeCount), the list of peer service instances, if any (peers), and information about the Spring Cloud Services Git repository at build time for the service instance (git).

Note: Fields such as those for Git repository information are for diagnostic purposes and intended to provide Pivotal Support with information to help in troubleshooting.

UAA Identity Zones and Clients

Spring Cloud Services uses the multi-tenancy capabilities of the Cloud Foundry User Account and Authentication server (UAA). It creates a new Identity Zone (“the Spring Cloud Services Identity Zone”) for all authorization from Config Server and Service Registry service instances to client applications which have bindings to these service instances.

Within the platform Identity Zone (“the UAA Identity Zone”), Spring Cloud Services creates a p-spring-cloud-services UAA client for the SB and a p-spring-cloud-services-worker UAA client for the Worker. In the Spring Cloud Services Identity Zone, it creates a p-spring-cloud-services-worker UAA client for the Worker.

In the UAA Identity Zone, it also creates the following clients, where [GUID] is the service instance’s GUID:

  • A eureka-[GUID] UAA client per Service Registry service instance.
  • A hystrix-[GUID] UAA client per Circuit Breaker Dashboard service instance.

In the Spring Cloud Services Identity Zone, it also creates the following clients, where [GUID] is the service instance’s GUID:

  • A config-server-[GUID] UAA client per Config Server service instance.
  • A eureka-[GUID] UAA client per Service Registry service instance.

Get Access Token for Direct Requests to a Service Instance

To make requests directly against a Config Server service instance’s Spring Cloud Config Server backing application or a Service Registry service instance’s Spring Cloud Netflix Eureka backing application, you can get an access token using the binding credentials of an application that is bound to the service instance.

Note: The following procedure uses the jq command-line JSON processing tool.

Run cf env, giving the name of an application that is bound to the service instance:

$ cf services
Getting services in org myorg / space development as admin...
OK

name             service           plan        bound apps   last operation
config-server    p-config-server   standard    cook         create succeeded

$ cf env cook
Getting env variables for app cook in org myorg / space development as admin...
OK

System-Provided:
{
 "VCAP_SERVICES": {
  "p-config-server": [
   {
    "credentials": {
     "access_token_uri": "https://p-spring-cloud-services.uaa.cf.wise.com/oauth/token",
     "client_id": "p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73",
     "client_secret": "rU7dMUw6bQjR",
     "uri": "https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com"
    },
[...]

Then run the following Bash script, which fetches a token and uses the token to access an endpoint on a service instance backing application:

TOKEN=$(curl -k ACCESS_TOKEN_URI -u CLIENT_ID:CLIENT_SECRET -d
grant_type=client_credentials | jq -r .access_token); curl -k -H
"Authorization: bearer $TOKEN" -H "Accept: application/json"
URI/ENDPOINT | jq

In this script, replace the following placeholders using values from the cf env command above:

  • ACCESS_TOKEN_URI with the value of credentials.access_token_uri
  • CLIENT_ID with the value of credentials.client_id
  • CLIENT_SECRET with the value of credentials.client_secret
  • URI with the value of credentials.uri

Replace ENDPOINT with the relevant endpoint:

  • application/profile to retrieve configuration from a Config Server service instance
  • eureka/apps to retrieve the registry from a Service Registry service instance

The Service Instances Dashboard

To view the status of individual service instances, visit the Service Instances dashboard. You can access it at the following URL, where SCS_BROKER_URL is the URL of the SB (spring-cloud-broker):

SCS_BROKER_URL/admin/serviceInstances

Locate the SB as described in the Access Broker Applications in Apps Manager section. The Service Instances dashboard is also linked to on the main page of the SB.

The dashboard shows the version and status of each service instance, as well as other information including the number of applications that are bound to the instance. It also provides an Upgrade button for each service instance and an Upgrade Service Instances button; these buttons may be enabled if any service instances shown in the dashboard are not at the version included with the currently-installed Spring Cloud Services product. For more information about upgrading service instances, see the Service Instance Upgrades topic.

Si dashboard

You can click the Instance Name of a listed service instance to view the instance’s dashboard. Click the + icon beside the count of an instance’s Bound Apps or Service Keys to view the names of the applications or service keys that are bound to that instance.

Note: The Service Instances dashboard is updated frequently (close to real-time). If using the Cloud Foundry Command Line Interface tool (cf CLI), you may notice a discrepancy between the status given for a service instance by the cf CLI (e.g., by the cf service command) versus that given by the Service Instances dashboard. The status retrieved by the cf CLI is not updated as frequently and may take time to match that shown on the Service Instances dashboard.

Application Health and Status

For more visibility into how Spring Cloud Services service instances and the broker applications are behaving, or for troubleshooting purposes, you may wish to access those applications directly. See below for information about accessing their output.

Read Broker Application Logs

To access logs for the SB and Worker applications, target the “system” org and its “p-spring-cloud-services” space:

$ cf target -o system -s p-spring-cloud-services

API endpoint:   https://api.cf.wise.com (API version: 2.43.0)
User:           admin
Org:            system
Space:          p-spring-cloud-services
Ben-Kleins-MacBook-Pro:Work bklein$ cf apps
Getting apps in org system / space p-spring-cloud-services as admin...
OK

name                         requested state   instances   memory   disk   urls
spring-cloud-broker          started           1/1         1G       1G     spring-cloud-broker.apps.wise.com
spring-cloud-broker-worker   started           1/1         1G       1G     spring-cloud-broker-worker.apps.wise.com

Then you can use cf logs to tail logs for either the SB:

$ cf logs spring-cloud-broker
Connected, tailing logs for app spring-cloud-broker in org system / space p-spring-cloud-services as admin...

or the Worker:

$ cf logs spring-cloud-broker-worker
Connected, tailing logs for app spring-cloud-broker-worker in org system / space p-spring-cloud-services as admin...

Read Backing Application Logs

To access logs for service instance backing applications, target the “p-spring-cloud-services” org and its “instances” space:

$ cf target -o p-spring-cloud-services -s instances

API endpoint:   https://api.cf.wise.com (API version: 2.43.0)
User:           admin
Org:            p-spring-cloud-services
Space:          instances
$ cf apps
Getting apps in org p-spring-cloud-services / space instances as admin...
OK

name                                           requested state   instances   memory   disk   urls
config-86b38ce0-eed8-4c01-adb4-1a651a6178e2    started           1/1         1G       1G     config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com
eureka-493b6b17-512a-4961-8f85-6178251fe2fa    started           1/1         1G       1G     eureka-493b6b17-512a-4961-8f85-6178251fe2fa.apps.wise.com
hystrix-ff835bde-5b3a-4623-a57f-8d88028b6376   started           1/1         1G       1G     hystrix-ff835bde-5b3a-4623-a57f-8d88028b6376.apps.wise.com
turbine-ff835bde-5b3a-4623-a57f-8d88028b6376   started           1/1         1G       1G     turbine-ff835bde-5b3a-4623-a57f-8d88028b6376.apps.wise.com

Then you can use cf logs to tail logs for a backing application.

$ cf logs config-86b38ce0-eed8-4c01-adb4-1a651a6178e2
Connected, tailing logs for app config-86b38ce0-eed8-4c01-adb4-1a651a6178e2 in org p-spring-cloud-services / space instances as admin...

If the backing application is not running properly then detailed health information will be written to its log on each GET request sent to its health Actuator endpoint. No health details are logged if the backing application is healthy.

2017-05-04T13:37:17.43+0100 [APP/PROC/WEB/0]OUT 2017-05-04 12:37:17.431  INFO 14 --- [nio-8080-exec-7] i.p.s.c.h.ConfigServerHealthEndpoint     : Health status: DOWN. Health details: {git=DOWN {default=DOWN {repository={uri=https://github.com/spring-cloud-samples/cook-config}, error=https://github.com/spring-cloud-samples/cook-config: Authentication is required but no credentials have been configured}}, diskSpace=UP {total=1056858112, free=889167872, threshold=10485760}, refreshScope=UP {}}
2017-05-04T13:37:17.43+0100 [RTR/0]      OUT config-80593543-d141-424d-bcb9-1aa583be2356.olive.springapps.io - [2017-05-04T12:37:17.067+0000] "GET /health HTTP/1.1" 503 0 390 "-" "Java/1.8.0_121" "10.194.42.254:56706" "10.194.42.143:60820" x_forwarded_for:"10.194.42.143, 10.194.42.254" x_forwarded_proto:"https" vcap_request_id:"c04a99a8-c222-4e59-4acf-a9cfff62a3d5" response_time:0.369370172 app_id:"ee476250-6418-4e35-b3a4-9686947961d2" app_index:"0" x_b3_traceid:"2cfc3caf6fa0fa22" x_b3_spanid:"2cfc3caf6fa0fa22" x_b3_parentspanid:"-"

Access Actuator Endpoints

The SB and Worker applications, as well as backing applications for service instances, use Spring Boot Actuator. Actuator adds a number of endpoints to these applications; some of the added endpoints are summarized below.

ID Function
env Displays profiles, properties, and property sources from the application environment
health Displays information about the health and status of the application
metrics Displays metrics information from the application
mappings Displays list of @RequestMapping paths in the application
shutdown Allows for graceful shutdown of the application. Disabled by default; not enabled in Spring Cloud Services broker or instance-backing applications

See the Endpoints section of the Actuator documentation for the full list of endpoints.

Invoking Actuator Endpoints

The SB application, the Worker application, the Service Registry’s Eureka backing application, the Config Server’s backing application, and the Circuit Breaker Dashboard’s Hystrix backing application all use OAuth 2 authentication. To access Actuator endpoints on one of these applications, you must supply a valid OAuth 2 token in the request header.

  1. Log in to Apps Manager as an admin user and navigate to the relevant application: for the SB or Worker, access the SB application or Worker application as described in the The Service Broker section; for a Service Registry, Config Server, or Circuit Breaker Dashboard service instance, access the backing application as described in the Service Instances section.

    Hystrix

  2. From the Route tab, copy the value of the application’s route, as described in the Get Service Instance Information section.

  3. In a terminal where you have already authenticated to PCF using cf auth or cf login, send the request to the Actuator endpoint (e.g., using cURL), appending the endpoint ID to the route URL; e.g., for the health endpoint, this would be something like https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/health. The current OAuth 2 token can be obtained using the cf oauth-token command. In the example below, the response from cf oauth-token is used directly inside the header string.

    $ curl -k https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/health -H "Authorization: $(cf oauth-token)"
    {"status":"UP","git":{"status":"UP","default":{"status":"UP","repository":{"uri":"https://github.com/pivotal-cf/p-spring-cloud-services-acceptance"}},"app1":{"status":"UP","repository":{"uri":"https://github.com/pivotal-cf/p-spring-cloud-services-acceptance"}},"app2":{"status":"UP","repository":{"uri":"https://github.com/pivotal-cf/p-spring-cloud-services-acceptance"}}},"diskSpace":{"status":"UP","total":1056858112,"free":886517760,"threshold":10485760},"refreshScope":{"status":"UP"}}
    

You also can send a request to each application’s health Actuator endpoint without the authorization header. In that case, the response JSON object will contain only the summary status:

<pre class="terminal">
$ curl -k https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/health
{"status":"UP"}
</pre>

Sending an unauthenticated request to any of the other Actuator endpoints will return a response indicating insufficient authorization.

Invoke Config Server Configuration Endpoints

To view the configuration properties that a Config Server service instance is returning for an application, you can access the configuration endpoints on the service instance’s Spring Cloud Config Server backing application directly.

  1. Obtain an access token for interacting with the service instance, as described in the Get Access Token for Direct Requests to a Service Instance section. In Step 1 of that section, copy the URL in credentials.uri from the service instance’s entry in the VCAP_SERVICES environment variable.

  2. Make a request of the Config Server service instance backing application in the format http://SERVER/APPLICATION_NAME/PROFILE, where SERVER is the URL from credentials.uri as mentioned in Step 1, APPLICATION_NAME is the application name as set in the spring.application.name property, and PROFILE is the name of a profile that has a configuration file in the repository.


    An example request URL might look like this:

    https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/cook/production
    

    Or, for the default profile:

    https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/cook/default
    

    You can make the request using curl, providing the token—TOKEN in the below example—in a header with the -H or --header option:

    $ curl -H 'Authorization: bearer TOKEN' https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/cook/default
    {"name":"cook","profiles":["default"],"label":null,"version":"7b5da0d68d9237f2852f7ac6c5e7474fd433c3f3","propertySources":[{"name":"https://github.com/spring-cloud-services-samples/cook-config/cook.properties","source":{"cook.special":"Pickled Cactus"}}]}
    

For a list of path formats that the Config Server accepts, see the Request Paths section of the The Config Server topic in the Config Server documentation.

Capacity Requirements

Below are usage requirements of the applications backing a single service instance (SI) of each Spring Cloud Services service type.

Service Type Memory Limit / SI Disk Limit / SI
Config Server 1 GB 1 GB
Service Registry 1 GB 1 GB
Circuit Breaker Dashboard 2 GB 2 GB

Spring Cloud Services service types may also be backed by non-SCS service instances. For example, a Circuit Breaker Dashboard service instance uses a RabbitMQ for Pivotal Cloud Foundry service instance for communication between its two backing applications.

Create a pull request or raise an issue on the source for this page in GitHub