Installing PCF Isolation Segment

Page last updated:

This topic describes how to install the PCF Isolation Segment tile, which allows operators to isolate deployment workloads into dedicated resource pools called isolation segments.

Installing the tile installs a single isolation segment. However, you can install multiple isolation segments using the Replicator tool documented in Step 4.

After installing the tile, you must perform the steps in the Create an Isolation Segment section of the Managing Isolation Segments topic to create the isolation segment in the Cloud Controller Database (CCDB). The topic also includes information about managing an isolation segment.

For more information about how isolation segments work, see the Isolation Segments section of the Understanding Cloud Foundry Security topic.

Step 1: (Optional) Configure Routing

By default, the Elastic Runtime Router handles traffic for your isolation segment. However, you can deploy a dedicated router for your isolation segment instead. For information about configuring and managing routing for isolation segments, see the Routing for Isolation Segments topic.

To deploy a dedicated router, perform the following steps:

  1. Add a load balancer in front of the Elastic Runtime Router. The steps to do this depend on your IaaS, but the setup of the load balancer should mirror the setup of the load balancer for the Elastic Runtime Router that you configured in the Elastic Runtime tile.
  2. Create a wildcard DNS entry for traffic routed to any app in the isolation segment. For example, *.iso.example.com.
  3. Attach the wildcard DNS entry to the load balancer you created.

Step 2: Install the Tile

Perform the following steps to install the PCF Isolation Segment tile:

  1. Download the product file from Pivotal Network.

  2. Navigate to YOUR-OPSMAN-FQDN in a browser to log in to the Ops Manager Installation Dashboard.

  3. Click Import a Product and select the downloaded product file.

  4. Under PCF Isolation Segment in the left column, click the plus sign.

Step 3: Configure the Tile

Click the orange PCF Isolation Segment tile to start the configuration process.

Pcf isolation segment

Assign AZs and Networks

Perform the following steps to configure the PCF Isolation Segment tile:

  1. Click Assign AZs and Networks.

    Az network

  2. Select an availability zone for your singleton jobs, and one or more availability zones to balance other jobs in.

  3. Select a network. This network does not need to be the same network where you deployed Elastic Runtime. For most deployments, operators should create unique networks in which to deploy the Isolation Segment tile. These networks should maintain network reachability with the Diego components so that the cells can reach the Diego Brain and Diego Database VMs.

  4. Click Save.

Networking

Perform the following steps to configure the PCF Isolation Segment tile:

  1. Click Networking.

    Router ha ip

  2. (Optional): Under Router IPs, enter one or more static IP addresses for the routers that handle this isolation segment. These IP addresses must be within the subnet CIDR block that you defined in the Ops Manager network configuration for your Isolation Segment. If you have a load balancer, configure it to point to these IP addresses.

    Note: Entering the static IPs is not necessary for deployments running on a public IaaS such as AWS, GCP or Azure because PCF users specify the IaaS load balancer in the Resource Config section of the PCF Isolation Segment tile.

  3. If you want to use HAProxy for this isolation segment, enter at least one address in the HAProxy IPs field. You should specify more than one IP address for high availability. Then configure your load balancer to forward requests for the domains you have set up for your deployment to these IP addresses. For more information, see Configuring SSL/TLS Termination at HAProxy.

    Note: If you rely on HAProxy for a feature in Elastic Runtime and you want isolated networking for this isolation segment, you may want to deploy the HAProxy provided by the Isolation Segment tile.

  4. Under Certificate and Private Key for HAProxy and Router, you must provide an SSL Certificate and Private Key. Starting in PCF v.1.12, HAProxy and Gorouter are enabled to receive TLS communication by default. Networking haproxy router cert config
    You can either provide a certificate signed by a Certificate Authority (CA) or click on the Generate RSA Certificate link to generate a self-signed certificate in Ops Manager.

    For details about generating certificates in Ops Manager for your wildcard system domains, see the Providing a Certificate for Your SSL/TLS Termination Point topic.

  5. In the Minimum version of TLS supported by HAProxy and Router field, select the minimum version of TLS to use in HAProxy and Router communications. HAProxy and Router use TLS v1.2 by default. If you need to accommodate clients that use an older version of TLS, select a lower minimum version. For a list of TLS ciphers supported by the Gorouter, see Securing Traffic into Cloud Foundry. Networking min tls version

  6. In the TLS Cipher Suites for Router field, specify the TLS cipher suites to use for TLS handshakes between Gorouter and downstream clients like load balancers or HAProxy. Use an ordered, colon-delimited list of Golang-supported TLS cipher suites in the OpenSSL format. The recommended setting is ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384. Operators should verify that the ciphers are supported by any clients or downstream components that will initiate TLS handshakes with the Router. For a list of TLS ciphers supported by the Gorouter, see Securing Traffic into Cloud Foundry. Networking tls router Verify that whatever client is participating in the TLS handshake with Gorouter has at least one cipher suite in common with Gorouter.

    Note: Specify cipher suites that are supported by the versions configured in the Minimum version of TLS supported by HAProxy and Router field.

  7. In the TLS Cipher Suites for HAProxy field, specify the TLS cipher suites to use in HAProxy for TLS handshakes between HAProxy and its clients such as load balancers and Gorouter. Use an ordered, colon-delimited list of TLS cipher suites in the OpenSSL format. The recommended setting: DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384
    Operators should verify that the ciphers are supported by any clients or downstream components that will initiate TLS handshakes with the HAProxy. Networking tls haproxy Verify that whatever clients are participating in the TLS handshake with HAProxy have at least one cipher suite in common with HAProxy.

    Note: Specify cipher suites that are supported by the versions configured in the Minimum version of TLS supported by HAProxy and Router field.

  8. Under HAProxy forwards requests to Router over TLS, select Enable or Disable based on your deployment layout. Networking haproxy router tls forward

    • Enable HAProxy forwarding of requests to Router over TLS
      If you want to: Encrypt communication between HAProxy and the Gorouter
      Then configure the following:
      1. Leave Enable selected.
      2. In the Certificate Authority for HAProxy Backend field, specify the Certificate Authority (CA) that signed the certificate you configured in the Certificate and Private Key for HAProxy and Router field.

        Note: If you used the Generate RSA Certificate link to generate a self-signed certificate, then the CA to specify is the Ops Manager CA, which you can locate at the CA endpoint in the Ops Manager API.

      3. Make sure that Gorouter and HAPRoxy have TLS cipher suites in common in the TLS Cipher Suites for Router and TLS Cipher Suites for HAProxy fields.
      See also:
    • Disable HAProxy forwarding of requests to Router over TLS
      If you want to: Use non-encrypted communication between HAProxy and Gorouter, or you are not using HAProxy
      Then configure the following:
      1. Select Disable.
      2. If you are not using HAProxy, set the the number of HAProxy job instances to 0 on the Resource Config page. See Configuring Resources.
      See also:

  9. If you are not using SSL encryption or if you are using self-signed certificates, select Disable SSL certificate verification for this environment. Selecting this checkbox also disables SSL verification for route services.

    Note: For production deployments, Pivotal does not recommend disabling SSL certificate verification.

  10. (Optional) If you want HAProxy or the Gorouter to reject any HTTP (non-encrypted) traffic, select the Disable HTTP on HAProxy and Gorouter checkbox. When selected, HAProxy and Gorouter will not listen on port 80.
    Networking disable http haproxy gorouter

  11. Select the Disable insecure cookies on the Router checkbox to set the secure flag for cookies generated by the router.

  12. To disable the addition of Zipkin tracing headers on the Gorouter, deselect the Enable Zipkin tracing headers on the router checkbox. Zipkin tracing headers are enabled by default. For more information about using Zipkin trace logging headers, see Zipkin Tracing in HTTP Headers.

  13. Under Configure the CF Router support for the X-Forwarded-Client-Cert header, configure how the Gorouter handles x-forwarded-client-cert (XFCC) HTTP headers. Networking xforwarded client cert xfcc The following table indicates which option to choose based on your deployment layout.

    If your deployment is configured as follows: Then select the following option:
    • Load balancer is terminating TLS, and
    • Load balancer is configured to put the client certificate from a mutual authentication TLS handshake into the X-Forwarded-Client-Cert HTTP header, and
    • Requests to Gorouter are unencrypted (whether or not HAProxy is present.)
    Always forward the XFCC header in the request, regardless of whether the client connection is mTLS (default).
    • Load balancer is terminating TLS, and
    • Load balancer is configured to put the client certificate from a mutual authentication TLS handshake into the X-Forwarded-Client-Cert HTTP header, and
    • Requests to Gorouter are encrypted (whether or not HAProxy is present.)
    Forward the XFCC header received from the client only when the client connection is mTLS.
    • Load balancer is not terminating TLS (configured as pass through), and
    • Gorouter is terminating TLS
    Strip the XFCC header when present and set it to the client certificate from the mTLS handshake.

    For a description of the behavior of each configuration option, see Forward Client Certificate to Applications.

  14. (Optional) If you want to limit the number of app connections to the backend, enter a value in the Max Connections Per Backend field. You can use this field to prevent a poorly behaving app from all the connections and impacting other apps.

    To choose a value for this field, review the peak concurrent connections received by instances of the most popular apps in your deployment. You can determine the number of concurrent connections for an app from the httpStartStop event metrics emitted for each app request.

    If your deployment uses PCF Metrics, you can also obtain this peak concurrent connection information from Network Metrics. The default value of 0 means that there is no limit. Networking max connections backend

  15. Enter a value for Router Max Idle Keepalive Connections. See Considerations for Configuring max_idle_connections.

    Keepalive

  16. (Optional) To accommodate larger uploads over connections with high latency, increase the number of seconds in the Router Timeout to Backends field.

  17. (Optional) Use the Frontend Idle Timeout for Gorouter and HAProxy field to help prevent connections from your load balancer to Gorouter or HAProxy from being closed prematurely. The value you enter sets the duration, in seconds, that Gorouter or HAProxy maintains an idle open connection from a load balancer that supports keep-alive.

    In general, set the value higher than your load balancer’s backend idle timeout to avoid the race condition where the load balancer sends a request before it discovers that Gorouter or HAProxy has closed the connection.

    See the following table for specific guidance and exceptions to this rule:

    IaaS Guidance
    AWS AWS ELB has a default timeout of 60 seconds, so Pivotal recommends a value greater than 60.
    Azure By default, Azure load balancer times out at 240 seconds without sending a TCP RST to clients, so as an exception, Pivotal recommends a value lower than 240 to force the load balancer to send the TCP RST.
    GCP GCP has a default timeout of 600 seconds, so Pivotal recommends a value greater than 600.
    Other Set the timeout value to be greater than that of the load balancer’s backend idle timeout.

  18. (Optional) Increase the value of Load Balancer Unhealthy Threshold to specify the amount of time, in seconds, that the router continues to accept connections before shutting down. During this period, healthchecks may report the router as unhealthy, which causes load balancers to failover to other routers. Set this value to an amount greater than or equal to the maximum time it takes your load balancer to consider a router instance unhealthy, given contiguous failed healthchecks.

  19. (Optional) Modify the value of Load Balancer Healthy Threshold. This field specifies the amount of time, in seconds, to wait until declaring the Router instance started. This allows an external load balancer time to register the Router instance as healthy.

    Router lb thresholds

  20. (Optional) If app developers in your organization want certain HTTP headers to appear in their app logs with information from the Gorouter, specify them in the HTTP Headers to Log field. For example, to support app developers that deploy Spring apps to PCF, you can enter Spring-specific HTTP headers.

    Http Headers to Log

  21. If you expect requests larger than the default maximum of 16 Kbytes, enter a new value (in bytes) for HAProxy Request Max Buffer Size. You may need to do this, for example, to support apps that embed a large cookie or query string values in headers.

  22. If your PCF deployment uses HAProxy and you want it to receive traffic only from specific sources, use the following fields:

    • Protected Domains: Enter a comma-separated list of domains from which PCF can receive traffic.
    • Trusted CIDRs: Optionally, enter a space-separated list of CIDRs to limit which IP addresses from the Protected Domains can send traffic to PCF. Protected domains

  23. (Optional): Under Applications Network Maximum Transmission Unit, change the Maximum Transmission Unit (MTU). The default is 1454 bytes.

  24. (Optional): Enter a UDP port number in the VXLAN Tunnel Endpoint Port box. If you do not set a custom port, PCF Isolation Segment uses 4789. Ert c2c vxlan port

  25. To enable logging for app traffic, select Enable (will increase log volume) under Log traffic for all accepted/denied application packets. App traffic logging generates log messages as follows:

    • TCP traffic: Logs the first packet of every new TCP connection.
    • UDP traffic: Logs UDP packets sent and received, up to a maximum per-second rate for each container. Set this rate limit in the UDP logging interval field (default: 100).
    • Packets denied: Logs packets blocked by either a container-specific networking policy or by Application Security Group rules applied across the space, org, or deployment. Logs packet denials up to a maximum per-second rate for each container, set in the Denied logging interval field (default: 1).
      Log app traffic enable
      See Manage Logging for Container-to-Container Networking for more information.

  26. Select a sharding mode in the Router Sharding Mode field. The options are explained below. For more information, see Sharding Routers for Isolation Segments. Sharding options

    Option Name Description
    Isolation Segment Only The routers for the tile acknowledge requests only from apps deployed within the cells of the tile. All other requests fail.
    No Isolation Segment The routers for the tile reject requests for any isolation segment. Choose this option to add a group of routers for the Elastic Runtime tile, such as when you want a private point of entry for the system domain.

  27. Click Save.

Application Containers

Perform the following steps to configure the PCF Isolation Segment tile:

  1. Click Application Containers.

    App containers

  2. (Optional): Under Private Docker Insecure Registry Whitelist, enter one or more private Docker image registries that are secured with self-signed certificates. Use a comma-delimited list in the format IP:Port or Hostname:Port.

  3. Under Segment Name, enter the name of your isolation segment. This name must be unique across your PCF deployment. You use this name when performing the steps in the Create an Isolation Segment section of the Managing Isolation Segments topic to create the isolation segment in the Cloud Controller Database (CCDB).

  4. By default, containers use the same DNS servers as the host. If you want to override the DNS servers to be used in the containers in the isolation segment, enter a comma-separated list of servers in DNS Servers.

  5. Select your preference for Docker Images Disk-Cleanup Scheduling on Cell VMs. If you choose Clean up disk-space once threshold is reached, enter a Threshold of Disk-Used in megabytes. For more information about the configuration options and how to configure a threshold, see Configuring Docker Images Disk-Cleanup Scheduling.

  6. Under Enabling NFSv3 volume services, select Enable or Disable. NFS volume services allow application developers to bind existing NFS volumes to their applications for shared file access. For more information, see the Enabling NFS Volume Services topic.

    Note: In a clean install, NFSv3 volume services are enabled by default. In an upgrade, NFSv3 volume services match the setting of the previous deployment.

    Er config app vol svc

  7. (Optional) To configure LDAP for NFSv3 volume services, perform the following steps:

    • For LDAP Service Account User, enter the username of the service account in LDAP that will manage volume services.
    • For LDAP Service Account Password, enter the password for the service account.
    • For LDAP Server Host, enter the hostname or IP address of the LDAP server.
    • For LDAP Server Port, enter the LDAP server port number. If you do not specify a port number, Ops Manager uses 389.
    • For LDAP Server Protocol, enter the server protocol. If you do not specify a protocol, Ops Manager uses TCP.
    • For LDAP User Fully-Qualified Domain Name, enter the fully qualified path to the LDAP service account. For example, if you have a service account called volume-services that belongs to organizational units (OU) called service-accounts and my-company, and your domain is called domain, the fully qualified path looks like the following:
      CN=volume-services,OU=service-accounts,OU=my-company,DC=domain,DC=com
  8. By default, ERT manages container images using the GrootFS plugin for Garden-runC. If you experience issues with GrootFS, you can disable the plugin and use the image plugin built into Garden-runC.

  9. Click Save.

System Logging

  1. In the System Logging menu, select an option from the Do you want to configure syslog for system components? menu. No is selected by default. Iso logging no

  2. If you want to use syslog, select Yes. Iso logging yes

  3. Enter the address of your external syslog aggregation service in the Address field. The address can be a hostname or IP address.

  4. Enter a port number in the Port field.

  5. Select a protocol from the Transport Protocol menu. This is the protocol the sytem uses to transmit logs to syslog.

  6. (Optional) Select the Enable TLS option if you want to transmit logs over TLS.

  7. Enter a Permitted Peer.

  8. Paste the certificate for your TLS certificate authority (CA) in the TLS CA Certificate field.

  9. Click Save.

Advanced Features

Edit the configurations in Advanced Features as desired.

Configure Router Resources

  1. Select Resource Config. Resource config
  2. If you are using a dedicated router for your isolation segment, follow the instructions below.

    Note: The configuration settings available in Resource Config vary depending on your IaaS.

    • If you have the Load Balancers column in your Resource Config:
      • Enter the wildcard DNS entry attached to your load balancer into the Router row under Load Balancers.
    • If you do not have the Load Balancers column in your Resource Config:
      • Navigate to the Networking section of the PCF Isolation Segment tile.
      • Specify Router IPs.
      • Attach the IPs to your load balancer manually.
  3. If you are not using HAProxy for this isolation segment, set the number of Instances to 0.

Stemcell

Optionally, edit the Stemcell configuration.

After Tile Configuration

After you configure the PCF Isolation Segment tile, perform the following steps:

  1. Create the isolation segment in the Cloud Controller Database (CCDB) by following the procedure in the Create an Isolation Segment section of the Managing Isolation Segments topic.

  2. Return to the Ops Manager Installation Dashboard and click Apply Changes to deploy the tile.

After the tile finishes deploying, see the Managing Isolation Segments topic for more information about managing an isolation segment.

Step 4: (Optional) Copy the Tile

If you want to create multiple isolation segments, perform the following steps to copy the PCF Isolation Segment tile with the Replicator tool:

  1. Download the Replicator tool from the Pivotal Cloud Foundry Isolation Segment section of Pivotal Network.

  2. Navigate to the directory where you downloaded the Replicator tool.

  3. Replicate the tile:

    ./replicator \
        -name "YOUR-NAME" \
        -path /PATH/TO/ORIGINAL.pivotal \
        -output /PATH/TO/COPY.pivotal
    

    Replace the values above with the following:

    • YOUR-NAME: Provide a unique name for the new PCF Isolation Segment tile. The name must be ten characters or less and only contain alphanumeric characters, dashes, underscores, and spaces.
    • /PATH/TO/ORIGINAL: Provide the absolute path to the original PCF Isolation Segment tile you downloaded from Pivotal Network.
    • /PATH/TO/COPY: Provide the absolute path for the copy that the Replicator tool produces.
  4. Follow the procedures in this topic using the new .pivotal file, starting with Step 1.

Create a pull request or raise an issue on the source for this page in GitHub