Enabling and Configuring TCP Routing

Page last updated:

This topic describes what TCP Routing is and how to enable the feature in your Pivotal Application Service (PAS) deployment. id=

Overview

TCP Routing enables developers to run apps that serve requests on non-HTTP TCP protocols. You can use TCP routing to comply with regulatory requirements to terminate TLS as close to apps as possible so that packets are not decrypted before reaching the app level.

Architecture

The diagram below shows the layers of network address translation that occur in PAS in support of TCP Routing. The descriptions step through an example work flow that covers route ports, back end ports, and app ports.

Route Ports

  • A developer creates a TCP route for their app based on a TCP domain and a route port, and maps this route to one or more apps. For more information, see the Create a Route section of the Configuring Routes and Domains topic.

  • Clients make requests to the route. DNS resolves the domain name to the load balancer.

  • The load balancer listens on the port and forwards requests for the domain to the TCP routers. The load balancer must listen on a range of ports to support multiple TCP route creation. Additionally, PAS must be configured with this range, so that the platform knows what ports can be reserved when developers create TCP routes.

  • The TCP routers are organized into router groups. A router group is a cluster of identically configured routers. You can use router groups to reserve the same port for multiple TCP routes, thus increasing the capacity for TCP routes. However, only one router group is supported.

  • The TCP router can be dynamically configured to listen on the port when the route is mapped to an app. The domain the request was originally sent to is no longer relevant to the routing of the request to the app. The TCP router keeps a dynamically updated record of the back ends for each route port. The back ends represent instances of an app mapped to the route. The TCP router chooses a back end using a round-robin load balancing algorithm for each new TCP connection from a client. As the TCP router is protocol-agnostic, it does not recognize individual requests, only TCP connections. All client requests transit the same connection to the selected back end until the client or back end closes the connection. Each subsequent connection triggers the selection of a back end.

  • Because containers each have their own private network, the TCP router does not have direct access to app containers. When a container is created for an app instance, a port on the Diego Cell VM is randomly chosen and iptables are configured to forward requests for this port to the internal interface on the container. The TCP router then receives a mapping of the route port to the Diego Cell IP and port.

  • By default, the Diego Cell only routes requests to port 8080, the app port, on the container internal interface. The app port is the port on which apps must listen. Developers can use the Cloud Controller API to update the ports an app can receive requests on. For more information, see Configuring Apps to Listen on Custom Ports (Beta).

Enable TCP Routing

Pre-Deployment Steps

Note: If you have mutual TLS app identity verification enabled, app containers accept incoming communication only from the Gorouter. This disables TCP routing.

Before enabling TCP routing, you must set up networking requirements. To set up networking requirements:

  1. Choose a domain name from which your developers are to create TCP routes for their apps. For example, create a domain which is similar to your app domain but prefixed by the TCP subdomain: tcp.APPS-DOMAIN.com

  2. Configure DNS to resolve this domain name to the IP address of a highly-available load balancer that can forward traffic for the domain to the TCP routers. For more information, see the Domains section of the Configuring Routes and Domains topic. If you are operating an environment that does not require high availability, configure DNS to resolve the TCP domain name you have chosen directly to a single instance of the TCP router.

  3. (Optional) Choose IP addresses for the TCP routers and configure your load balancer to forward requests for the domain you chose in the step above to these addresses. Skip this step if you have configured DNS to resolve the TCP domain name to an instance of the TCP router. To configure your IP addresses for your PAS deployment, see the procedure for enabling TCP routing in the Configure Networking section of the Configuring PAS topic.

  4. (Optional) Decide on the number of TCP routes you want to support. For each TCP route, you must reserve a port. Configure your load balancer to forward the range of ports to the TCP routers. Skip this step if you have configured DNS to resolve the TCP domain name to an instance of the TCP router. For more information about configuring port reservations, see Modify TCP port reservations below.

  5. To configure your ports for your PAS deployment, enable TCP routing in PAS. For more information, see Configure Networking in Configuring PAS.

Post-Deployment Steps

In the following procedures, you use the Cloud Foundry Command Line Interface (cf CLI) to add the TCP shared domain and configure org quotas to grant developers the ability to create TCP routes. This requires an admin user account.

For more information about the cf CLI, see Using the Cloud Foundry Command Line Interface (cf CLI).

Configure PAS with

Your TCP Domain

After deployment, you must configure PAS with the domain that you configured in the pre-deployment step above so developers can create TCP routes from it.

  1. Run cf router-groups. You should see default-tcp as a response.

  2. Run cf create-shared-domain to create a shared domain and associate it with the router group.

    $ cf create-shared-domain tcp.APPS-DOMAIN.com --router-group default-tcp
    

  3. Run cf domains. Verify that next to your TCP domain, TCP appears under type.

Configure a Quota for TCP Routes

Since TCP route ports are a limited resource in some environments, quotas are configured to allow creation of zero TCP routes by default. After you deploy PAS, you can increase the maximum number of TCP routes for all orgs or for particular orgs and spaces. Because you reserve a route port for each TCP route, the quota for this resource is managed with the cf CLI command option --reserved-route-ports. For more information, see Creating and Modifying Quota Plans.

If you have a default quota that applies to all orgs, you can update it to configure the number of route ports that can be reserved by each org by running:

cf update-quota QUOTA --reserved-route-ports X

To create a new org quota that can be allocated to particular orgs, provide the following required quota attributes in addition to the number of reserved route ports by running:

cf create-quota QUOTA --reserved-route-ports X

You can also create a quota that governs the number of route ports that can be created in a particular space by running:

cf create-space-quota QUOTA --reserved-route-ports X

Disable TCP Routing

To disable TCP routing:

  1. Navigate to the Ops Manager Installation Dashboard.

  2. Click the PAS tile.

  3. Select the Networking side-tab.

  4. Under TCP routing, select Disable.

  5. Run cf delete-domain DOMAIN on a command line to remove the TCP routing domain, where DOMAIN is the name of your TCP routing domain.

Further TCP Routing Administration

Create a TCP Route

For information about creating a TCP Route, see the Create a TCP Route with a Port section of the Configuring Routes and Domains topic.

Modify TCP port reservations

After deployment, you can modify the range of ports available for TCP routes using cf curl commands, as demonstrated with the commands below. These commands require an admin user with the routing.router_groups.read and routing.router_groups.write scopes.

  1. In a terminal window, run cf curl /routing/v1/router_groups to view the reservable_ports. Record the guid from the output.
    For example:

    $ cf curl /routing/v1/router_groups
    [
      {
        "guid": "9d1c1da9-ed63-45e8-45ee-256f8579455c",
        "name": "default-tcp",
        "type": "tcp",
        "reservable_ports": "20000-20098"
      }
    ]
    

  2. To configure a new port, run cf curl /routing/v1/router_groups/GUID, where GUID is the GUID you recorded in the previous step.
    For example:

    $ cf curl \
    -X PUT -d '{"reservable_ports":"1024-1199"}' \
    /routing/v1/router_groups/f7392031-a488-4890-8835-c4a038a3bded
    
  3. To specify multiple ranges, enter a comma-separated list of ports or port ranges.
    For example:

    $ cf curl \
    -X PUT -d '{"reservable_ports":"1024-1199,1234-1248,1312"}' \
    /routing/v1/router_groups/f7392031-a488-4890-8835-c4a038a3bded
    

Note: Do not enter conflicting reservable_ports with other TCP router instances or ephemeral port ranges. Pivotal recommends port ranges within 1024-2047 and 18000-32767 on default installations. Check what ports are available on the TCP Router VMs to verify that no additional ports are in use. For more information, see TCP router fails to configure routes when there is a port conflict with a local process in GitHub.