Installing PCF Isolation Segment

Page last updated:

This topic describes how to install the PCF Isolation Segment tile, which allows operators to isolate deployment workloads into dedicated resource pools called isolation segments.

Installing the tile installs a single isolation segment. However, you can install multiple isolation segments using the Replicator tool documented in Step 4.

After installing the tile, you must perform the steps in the Create an Isolation Segment section of the Managing Isolation Segments topic to create the isolation segment in the Cloud Controller Database (CCDB). The topic also includes information about managing an isolation segment.

For more information about how isolation segments work, see the Isolation Segments section of the Understanding Cloud Foundry Security topic.

Step 1: (Optional) Configure Routing

By default, the Pivotal Application Service (PAS) Router handles traffic for your isolation segment. However, you can deploy a dedicated router for your isolation segment instead. For information about configuring and managing routing for isolation segments, see the Routing for Isolation Segments topic.

To deploy a dedicated router, perform the following steps:

  1. Add a load balancer in front of the PAS Router. The steps to do this depend on your IaaS, but the setup of the load balancer should mirror the setup of the load balancer for the PAS Router that you configured in the PAS tile.
  2. Create a wildcard DNS entry for traffic routed to any app in the isolation segment. For example, *.iso.example.com.
  3. Attach the wildcard DNS entry to the load balancer you created.

Step 2: Install the Tile

Perform the following steps to install the PCF Isolation Segment tile:

  1. Download the product file from Pivotal Network.

  2. Navigate to YOUR-OPSMAN-FQDN in a browser to log in to the Ops Manager Installation Dashboard.

  3. Click Import a Product and select the downloaded product file.

  4. Under PCF Isolation Segment in the left column, click the plus sign.

Step 3: Configure the Tile

Click the orange PCF Isolation Segment tile to start the configuration process.

Pcf isolation segment

Assign AZs and Networks

Perform the following steps to configure the PCF Isolation Segment tile:

  1. Click Assign AZs and Networks.

    Az network

  2. Select an availability zone for your singleton jobs, and one or more availability zones to balance other jobs in.

  3. Select a network. This network does not need to be the same network where you deployed PAS. For most deployments, operators should create unique networks in which to deploy the Isolation Segment tile. These networks should maintain network reachability with the Diego components so that the cells can reach the Diego Brain and Diego Database VMs.

  4. Click Save.

Networking

Perform the following steps to configure the PCF Isolation Segment tile:

  1. Click Networking.

    Router ha ip

  2. (Optional): Under Router IPs, enter one or more static IP addresses for the routers that handle this isolation segment. These IP addresses must be within the subnet CIDR block that you defined in the Ops Manager network configuration for your Isolation Segment. If you have a load balancer, configure it to point to these IP addresses.

    Note: Entering the static IP addresses is not necessary for deployments running on a public IaaS such as AWS, GCP, or Azure because PCF users specify the IaaS load balancer in the Resource Config section of the PCF Isolation Segment tile.

  3. If you want to use HAProxy for this isolation segment, enter at least one address in the HAProxy IPs field. You should specify more than one IP address for high availability. Then configure your load balancer to forward requests for the domains you have set up for your deployment to these IP addresses. For more information, see Configuring SSL/TLS Termination at HAProxy.

    Note: If you rely on HAProxy for a feature in Pivotal Application Service (PAS) and you want isolated networking for this isolation segment, you may want to deploy the HAProxy provided by the Isolation Segment tile.

  4. Under Certificates and Private Key for HAProxy and Router, you must provide at least one Certificate and Private Key name and certificate keypair for HAProxy and Gorouter. The HAProxy and Gorouter are enabled to receive TLS communication by default. You can configure multiple certificates for HAProxy and Gorouter.

    1. Click the Add button to add a name for the certificate chain and its private keypair. This certificate is the default used by Gorouter and HAProxy. Networking haproxy router cert config
      You can either provide a certificate signed by a Certificate Authority (CA) or click on the Generate RSA Certificate link to generate a self-signed certificate in Ops Manager.
    2. If you want to configure multiple certificates for HAProxy and Gorouter, click the Add button and fill in the appropriate fields for each additional certificate keypair.

      For details about generating certificates in Ops Manager for your wildcard system domains, see the Providing a Certificate for Your SSL/TLS Termination Point topic.

  5. (Optional) When validating client requests using mutual TLS, the Gorouter trusts multiple certificate authorities (CAs) by default. If you want to configure the Gorouter and HAProxy to trust additional CAs, enter your CA certificates under Certificate Authorities Trusted by Router and HAProxy. All CA certificates should be appended together into a single collection of PEM-encoded entries. Networking router haproxy ca

  6. In the Minimum version of TLS supported by HAProxy and Router field, select the minimum version of TLS to use in HAProxy and Router communications. HAProxy and Router use TLS v1.2 by default. If you need to accommodate clients that use an older version of TLS, select a lower minimum version. For a list of TLS ciphers supported by the Gorouter, see Securing Traffic into Cloud Foundry. Networking min tls version

  7. Under Configure support for the X-Forwarded-Client-Cert header, configure PCF handles x-forwarded-client-cert (XFCC) HTTP headers based on where TLS is terminated for the first time in your deployment. Networking xforwarded client cert xfcc The following table indicates which option to choose based on your deployment layout.

    If your deployment is configured as follows: Then select the following option: Additional notes:
    • The Load Balancer is terminating TLS, and
    • Load balancer is configured to put the client certificate from a mutual authentication TLS handshake into the X-Forwarded-Client-Cert HTTP header
    TLS terminated for the first time at infrastructure load balancer (default). Both HAProxy and the Gorouter forward the XFCC header when included in the request.
    • The Load Balancer is configured to pass through the TLS handshake via TCP to the instances of HAProxy, and
    • HAProxy instance count is > 0
    TLS terminated for the first time at HAProxy. HAProxy sets the XFCC header with the client certificate received in the TLS handshake. The Gorouter forwards the header.

    Breaking Change: In the Router behavior for Client Certificates field, you cannot select the Router does not request client certificates option.

    • The Load Balancer is configured to pass through the TLS handshake via TCP to instances of the Gorouter

    TLS terminated for the first time at the Gorouter. The Gorouter strips the XFCC header if it is included in the request and forwards the client certificate received in the TLS handshake in a new XFCC header.

    If you have deployed instances of HAProxy, app traffic bypasses those instances in this configuration. If you have also configured your load balancer to route requests for ssh directly to the Diego Brain, consider reducing HAProxy instances to 0.

    Breaking Change: In the Router behavior for Client Certificates field, you cannot select the Router does not request client certificates option.


    For a description of the behavior of each configuration option, see Forward Client Certificate to Applications.

  8. To configure Gorouter behavior for handling client certificates, select one of the following options in the Router behavior for Client Certificate Validation field. Networking router client cert validate

    • Router does not request client certificates. This option is incompatible with the XFCC configuration options TLS terminated for the first time at HAProxy and TLS terminated for the first time at the Router in PAS because these options require mutual authentication. As client certificates are not requested, client will not provide them, and thus validation of client certificates will not occur.
    • Router requests but does not require client certificates. The Gorouter requests client certificates in TLS handshakes, validates them when presented, but does not require them. This is the default configuration.
    • Router requires client certificates. The Gorouter validates that the client certificate is signed by a Certificate Authority that the Gorouter trusts. If the Gorouter cannot validate the client certificate, the TLS handshake fails.

    WARNING: Requests to the platform will fail upon upgrade if your load balancer is configured with client certificates and the Gorouter does not have the certificate authority. To mitigate this issue, select Router does not request client certificates for Router behavior for Client Certificate Validation in the Networking pane.

  9. In the TLS Cipher Suites for Router field, review the TLS cipher suites for TLS handshakes between Gorouter and front-end clients such as load balancers or HAProxy. The default value for this field is ECDHE-RSA-AES128-GCM-SHA256:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384. If you want to modify the default configuration, use an ordered, colon-delimited list of Golang-supported TLS cipher suites in the OpenSSL format.
    Operators should verify that the ciphers are supported by any clients or front-end components that will initiate TLS handshakes with Gorouter. For a list of TLS ciphers supported by Gorouter, see Securing Traffic into Cloud Foundry. Networking tls router Verify that every client participating in TLS handshakes with Gorouter has at least one cipher suite in common with Gorouter.

    Note: Specify cipher suites that are supported by the versions configured in the Minimum version of TLS supported by HAProxy and Router field.

  10. In the TLS Cipher Suites for HAProxy field, review the TLS cipher suites for TLS handshakes between HAProxy and its clients such as load balancers and Gorouter. The default value for this field is the following:
    DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384 If you want to modify the default configuration, use an ordered, colon-delimited list of TLS cipher suites in the OpenSSL format.
    Operators should verify that the ciphers are supported by any clients or front-end components that will initiate TLS handshakes with HAProxy. Networking tls haproxy Verify that every client participating in TLS handshakes with HAProxy has at least one cipher suite in common with HAProxy.

    Note: Specify cipher suites that are supported by the versions configured in the Minimum version of TLS supported by HAProxy and Router field.

  11. Under HAProxy forwards requests to Router over TLS, select Enable or Disable based on your deployment layout. Networking haproxy router tls forward

    • Enable HAProxy forwarding of requests to Router over TLS
    If you want to: Encrypt communication between HAProxy and the Gorouter
    Then configure the following:
    1. Leave Enable selected.
    2. In the Certificate Authority for HAProxy Backend field, specify the Certificate Authority (CA) that signed the certificate you configured in the Certificate and Private Key for HAProxy and Router field.

      Note: If you used the Generate RSA Certificate link to generate a self-signed certificate, then the CA to specify is the Ops Manager CA, which you can locate at the CA endpoint in the Ops Manager API.

    3. Make sure that Gorouter and HAPRoxy have TLS cipher suites in common in the TLS Cipher Suites for Router and TLS Cipher Suites for HAProxy fields.
    See also:
    • Disable HAProxy forwarding of requests to Router over TLS
    If you want to: Use non-encrypted communication between HAProxy and Gorouter, or you are not using HAProxy
    Then configure the following:
    1. Select Disable.
    2. If you are not using HAProxy, set the number of HAProxy job instances to 0 on the Resource Config page. See Disable Unused Resources.
    See also:

  12. If you are not using SSL encryption or if you are using self-signed certificates, select Disable SSL certificate verification for this environment. Selecting this checkbox also disables SSL verification for route services.

    Note: For production deployments, Pivotal does not recommend disabling SSL certificate verification.

  13. (Optional) If you want HAProxy or the Gorouter to reject any HTTP (non-encrypted) traffic, select the Disable HTTP on HAProxy and Gorouter checkbox. When selected, HAProxy and Gorouter will not listen on port 80.
    Networking disable http haproxy gorouter

  14. (Optional) Select the Disable insecure cookies on the Router checkbox to set the secure flag for cookies generated by the router.

  15. (Optional) To disable the addition of Zipkin tracing headers on the Gorouter, deselect the Enable Zipkin tracing headers on the router checkbox. Zipkin tracing headers are enabled by default. For more information about using Zipkin trace logging headers, see Zipkin Tracing in HTTP Headers.

  16. (Optional) To stop the Router from writing access logs to local disk, deselect the Enable Router to write access logs locally checkbox. You should consider disabling this checkbox for high traffic deployments since logs may not be rotated fast enough and can fill up the disk.

  17. In the Choose whether or not to enable route services section, choose either Enable route services or Disable route services. Route services are a class of marketplace services that perform filtering or content transformation on application requests and responses. See the Route Services topic for details.

  18. (Optional) If you want to limit the number of app connections to the backend, enter a value in the Max Connections Per Backend field. You can use this field to prevent a poorly behaving app from all the connections and impacting other apps.

    To choose a value for this field, review the peak concurrent connections received by instances of the most popular apps in your deployment. You can determine the number of concurrent connections for an app from the httpStartStop event metrics emitted for each app request.

    If your deployment uses PCF Metrics, you can also obtain this peak concurrent connection information from Network Metrics. The default value is 500. Networking max connections backend

  19. Under Enable Keepalive Connections for Router, select Enable or Disable. Keepalive connections are enabled by default. For more information, see Keepalive Connections in HTTP Routing.
    Keepalive connections

  20. (Optional) To accommodate larger uploads over connections with high latency, increase the number of seconds in the Router Timeout to Backends field.

  21. (Optional) Use the Frontend Idle Timeout for Gorouter and HAProxy field to help prevent connections from your load balancer to Gorouter or HAProxy from being closed prematurely. The value you enter sets the duration, in seconds, that Gorouter or HAProxy maintains an idle open connection from a load balancer that supports keep-alive.

    In general, set the value higher than your load balancer’s backend idle timeout to avoid the race condition where the load balancer sends a request before it discovers that Gorouter or HAProxy has closed the connection.

    See the following table for specific guidance and exceptions to this rule:

    IaaS Guidance
    AWS AWS ELB has a default timeout of 60 seconds, so Pivotal recommends a value greater than 60.
    Azure By default, Azure load balancer times out at 240 seconds without sending a TCP RST to clients, so as an exception, Pivotal recommends a value lower than 240 to force the load balancer to send the TCP RST.
    GCP GCP has a default timeout of 600 seconds, so Pivotal recommends a value greater than 600.
    Other Set the timeout value to be greater than that of the load balancer’s backend idle timeout.

  22. (Optional) Increase the value of Load Balancer Unhealthy Threshold to specify the amount of time, in seconds, that the router continues to accept connections before shutting down. During this period, healthchecks may report the router as unhealthy, which causes load balancers to failover to other routers. Set this value to an amount greater than or equal to the maximum time it takes your load balancer to consider a router instance unhealthy, given contiguous failed healthchecks.

  23. (Optional) Modify the value of Load Balancer Healthy Threshold. This field specifies the amount of time, in seconds, to wait until declaring the Router instance started. This allows an external load balancer time to register the Router instance as healthy.

    Router lb thresholds

  24. (Optional) If app developers in your organization want certain HTTP headers to appear in their app logs with information from the Gorouter, specify them in the HTTP Headers to Log field. For example, to support app developers that deploy Spring apps to PCF, you can enter Spring-specific HTTP headers.

    Http Headers to Log

  25. If you expect requests larger than the default maximum of 16 Kbytes, enter a new value (in bytes) for HAProxy Request Max Buffer Size. You may need to do this, for example, to support apps that embed a large cookie or query string values in headers.

  26. If your PCF deployment uses HAProxy and you want it to receive traffic only from specific sources, use the following fields:

    • Protected Domains: Enter a comma-separated list of domains from which PCF can receive traffic.
    • Trusted CIDRs: Optionally, enter a space-separated list of CIDRs to limit which IP addresses from the Protected Domains can send traffic to PCF.
    Protected domains

  27. Select a sharding mode in the Router Sharding Mode field. The options are explained below. For more information, see Sharding Routers for Isolation Segments. Sharding options

    Option Name Description
    Isolation Segment Only The routers for the tile acknowledge requests only from apps deployed within the cells of the tile. All other requests fail.
    No Isolation Segment The routers for the tile reject requests for any isolation segment. Choose this option to add a group of routers for the PAS tile, such as when you want a private point of entry for the system domain.

  28. Click Save.

Application Containers

Perform the following steps to configure the PCF Isolation Segment tile:

  1. Click Application Containers.

    App containers

  2. (Optional): Under Private Docker Insecure Registry Whitelist, enter one or more private Docker image registries that are secured with self-signed certificates. Use a comma-delimited list in the format IP:Port or Hostname:Port.

  3. Under Segment Name, enter the name of your isolation segment. This name must be unique across your PCF deployment. You use this name when performing the steps in the Create an Isolation Segment section of the Managing Isolation Segments topic to create the isolation segment in the Cloud Controller Database (CCDB).

  4. Select your preference for Docker Images Disk-Cleanup Scheduling on Cell VMs. If you choose Clean up disk-space once threshold is reached, enter a Threshold of Disk-Used in megabytes. For more information about the configuration options and how to configure a threshold, see Configuring Docker Images Disk-Cleanup Scheduling.

  5. Under Enabling NFSv3 volume services, select Enable or Disable. NFS volume services allow application developers to bind existing NFS volumes to their applications for shared file access. For more information, see the Enabling NFS Volume Services topic.

    Note: In a clean install, NFSv3 volume services are enabled by default. In an upgrade, NFSv3 volume services match the setting of the previous deployment.

    Er config app vol svc

  6. (Optional) To configure LDAP for NFSv3 volume services, perform the following steps:

    • For LDAP Service Account User, enter the username of the service account in LDAP that will manage volume services.
    • For LDAP Service Account Password, enter the password for the service account.
    • For LDAP Server Host, enter the hostname or IP address of the LDAP server.
    • For LDAP Server Port, enter the LDAP server port number. If you do not specify a port number, Ops Manager uses port 389.
    • For LDAP Server Protocol, enter the server protocol. If you do not specify a protocol, Ops Manager uses TCP.
    • For LDAP User Fully-Qualified Domain Name, enter the fully qualified path to the LDAP service account. For example, if you have a service account called volume-services that belongs to organizational units (OU) called service-accounts and my-company, and your domain is called domain, the fully qualified path looks like the following:
      CN=volume-services,OU=service-accounts,OU=my-company,DC=domain,DC=com
  7. By default, PAS manages container images using the GrootFS plugin for Garden-runC. If you experience issues with GrootFS, you can disable the plugin and use the image plugin built into Garden-runC.

  8. To enable Gorouter to verify app identity using TLS, select the Router uses TLS to verify application identity checkbox. Verifying app identity using TLS improves resiliency and consistency for app routes. For more information about Gorouter route consistency modes, see Preventing Misrouting in HTTP Routing.

  9. Click Save.

System Logging

  1. In the System Logging menu, select an option underneath Do you want to configure syslog for system components?. No is selected by default. This setting only affects Diego cell, router, and HAProxy components within the isolation segment, not shared PAS system components. Iso logging no

  2. If you want to use syslog, select Yes. Ist tcp log export

  3. Enter the address of your external syslog aggregation service in the Address field. The address can be a hostname or IP address.

  4. Enter a port number in the Port field.

  5. Select a protocol from the Transport Protocol menu. This is the protocol the sytem uses to transmit logs to syslog.

  6. (Optional) Select the Enable TLS option if you want to transmit logs over TLS.

  7. Enter a Permitted Peer.

  8. Paste the certificate for your TLS certificate authority (CA) in the TLS CA Certificate field.

  9. (Optional) Select the Use TCP for file forwarding local transport to transmit logs over TCP, rather than UDP. This prevents log truncation, but may cause performance issues.

  10. Click Save.

Advanced Features

Edit the configurations in Advanced Features as desired.

Configure Router Resources

  1. Select Resource Config. Resource config
  2. If you are using a dedicated router for your isolation segment, follow the instructions below.

    Note: The configuration settings available in Resource Config vary depending on your IaaS.

    • If you have the Load Balancers column in your Resource Config:
      • Enter the wildcard DNS entry attached to your load balancer into the Router row under Load Balancers.
    • If you do not have the Load Balancers column in your Resource Config:
      • Navigate to the Networking section of the PCF Isolation Segment tile.
      • Specify Router IPs.
      • Attach the IP addresses to your load balancer manually.
  3. If you are not using HAProxy for this isolation segment, set the number of Instances to 0.

After Tile Configuration

After you configure the PCF Isolation Segment tile, perform the following steps:

  1. Create the isolation segment in the Cloud Controller Database (CCDB) by following the procedure in the Create an Isolation Segment section of the Managing Isolation Segments topic.

  2. Return to the Ops Manager Installation Dashboard and click Apply Changes to deploy the tile.

After the tile finishes deploying, see the Managing Isolation Segments topic for more information about managing an isolation segment.

Step 4: (Optional) Copy the Tile

If you want to create multiple isolation segments, perform the following steps to copy the PCF Isolation Segment tile with the Replicator tool:

  1. Download the Replicator tool from the Pivotal Cloud Foundry Isolation Segment section of Pivotal Network.

  2. Navigate to the directory where you downloaded the Replicator tool.

  3. Replicate the tile:

    ./replicator \
        -name "YOUR-NAME" \
        -path /PATH/TO/ORIGINAL.pivotal \
        -output /PATH/TO/COPY.pivotal
    

    Replace the values above with the following:

    • YOUR-NAME: Provide a unique name for the new PCF Isolation Segment tile. The name must be ten characters or less and only contain alphanumeric characters, dashes, underscores, and spaces.
    • /PATH/TO/ORIGINAL: Provide the absolute path to the original PCF Isolation Segment tile you downloaded from Pivotal Network.
    • /PATH/TO/COPY: Provide the absolute path for the copy that the Replicator tool produces.
  4. Follow the procedures in this topic using the new .pivotal file, starting with Step 1.

Create a pull request or raise an issue on the source for this page in GitHub