Page last updated:
This topic describes how the Gorouter, the main component in the VMware Tanzu Application Service for VMs (TAS for VMs) routing tier, routes HTTP traffic within TAS for VMs.
For more information, see Routing in TAS for VMs Components.
The Gorouter has a limit of 1 MB for HTTP Headers.
The specific language, framework, and configuration of the back-end app container determine the effective header size limit. For example, the default header size for the Tomcat container is 8 kB.
X-Forwarded-Proto header gives the scheme of the HTTP request from the client.
If an incoming request includes the
X-Forwarded-Proto header, the Gorouter:
Appends it to the existing header
Sets the scheme to HTTP if the client made an insecure request, meaning a request on port
Sets the scheme to HTTPS if the client made a secure request, meaning a request on port
Developers can configure their apps to reject insecure requests by inspecting the
X-Forwarded-Proto HTTP header on incoming traffic. The header may have
multiple values represented as a comma-separated list, so developers must ensure the app rejects traffic that includes any
X-Forwarded-Proto values that are
X-Forwarded-For is present, the Gorouter appends the load balancer’s IP address to it and forwards the list. If
X-Forwarded-For is not present, the
Gorouter sets it to the IP address of the load balancer in the forwarded request (some load balancers masquerade the client IP). If a load balancer sends the
client IP using the PROXY protocol, then the Gorouter uses the client IP address to set
If your load balancer terminates TLS on the client side of the Gorouter, it must append these headers to requests forwarded to the Gorouter. For more information, see Securing Traffic into TAS for VMs.
Zipkin is a tracing system that allows app developers to troubleshoot failures or latency issues. Zipkin provides the ability to trace requests and responses across distributed systems. For more information, see Zipkin.io.
When Zipkin tracing is enabled in TAS for VMs, the Gorouter examines the HTTP request headers and performs the following:
X-B3-SpanIdHTTP headers are not present in the request, the Gorouter generates values for these and inserts the headers into the request forwarded to an app. The Gorouter access log message for the request includes
X-B3-SpanIdHTTP headers are present in the request, the Gorouter forwards them unmodified. In addition to these trace and span IDs, the Gorouter access log message for the request includes
You can then add Zipkin trace IDs to app logs in order to trace app requests and responses in TAS for VMs.
After adding Zipkin HTTP headers to app logs, you can correlate the trace and span IDs that the Gorouter logs with the trace IDs that the app logs by running
cf logs APP-NAME, where
APP-NAME is the name of the app. To correlate trace IDs for a request through multiple apps, each app must forward appropriate
values for the headers with requests to other apps.
For more information about Zipkin tracing, see Enabling Zipkin Tracing.
If you want to retrieve debug data for a specific instance of an app, you can use the HTTP header
X-Cf-App-Instance to make a request to the app instance
you want to debug.
To make an HTTP request to a specific app instance:
In a terminal window, retrieve the global unique identifier (GUID) of your app by running:
cf app APP-NAME --guid
APP-NAMEis the name of your app.
From the terminal output, record the GUID of your app.
List your app instances and retrieve the index number of the instance you want to debug by running:
cf app APP-NAME
APP-NAMEis the name of your app.
From the terminal output, record the the index number of the instance you want to debug.
Make a request to the app route by running:
curl APP-FQDN -H "X-Cf-App-Instance":"APP-GUID:INSTANCE-INDEX-NUMBER"
APP-FQDNis the fully-qualified domain name (FQDN) of your app. For example,
APP-GUIDis the app GUID that you recorded in a previous step.
INSTANCE-INDEX-NUMBERis the instance index number that you recorded in the previous step.
Note: You can only useIf either of the values you provide in the above command are invalid, Gorouter returns a
X-Cf-App-Instanceheader on the Diego architecture.
400error, and the response from Gorouter contains a
X-Cf-Routererrorheader with more information about the error. The following table describes the possible error responses:
X-Cf-Routererror Value Reason for Error Response Body
The value provided for
X-Cf-App-Instanceincludes an incorrectly formatted app GUID.
The value provided for
X-Cf-App-Instanceincludes a correctly formatted app GUID, but the app instance index number was not found for the requested route.
400 Bad Request: Requested instance ('1') with guid ('aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa') does not exist for route ('example-route.cf.com')
Apps that require mutual TLS (mTLS) need metadata from client certificates to authorize requests. TAS for VMs supports this use case without bypassing layer-7 load balancers and the Gorouter.
The HTTP header
X-Forwarded-Client-Cert (XFCC) may be used to pass the originating client certificate along the data path to the app. Each component in the
data path must trust that the back-end component has not allowed the header to be tampered with.
If you configure the load balancer to terminate TLS and set the XFCC header from the received client certificate, you must also configure the load balancer to strip this header if it is present in client requests. This configuration is required to prevent spoofing of the client certificate.
The sections below describe supported deployment configurations.
By default, the Gorouter forwards arbitrary headers that are not otherwise mentioned in the docs. This includes the XFCC header.
For apps to receive the XFCC header, configure your load balancer to set the XFCC header with the contents of the client certificate received in the TLS handshake.
To enable this mode:
Navigate to the Ops Manager Installation Dashboard.
Click the TAS for VMs tile.
Under TLS termination point, select Infrastructure load balancer.
If the Gorouter is the first component to terminate TLS, such that it receives the certificate of the originating client in the mutual TLS handshake, you should select this option. When selected, the Gorouter sets the XFCC header to the contents of the client certificate received in the TLS handshake and strips the XFCC header when present in a request.
This option requires you to configure the load balancer in front of the Gorouter to pass through TLS handshake to the Gorouter through TCP.
To enable this mode:
Navigate to the Ops Manager Installation Dashboard.
Click the TAS for VMs tile.
Under TLS termination point, select Gorouter.
The Gorouter trusts the Diego intermediate certificate authority. This trust is enabled automatically and permits mutual authentication between apps running on Ops Manager.
Depending on your needs, you can configure your TAS for VMs deployment to terminate TLS at the Gorouter, at the Gorouter and the load balancer, or at the load balancer only. For more information, see Securing Traffic into TAS for VMs.
The Gorouter supports TLS and mutual authentication to back-end destinations, including app instances, platform services, and any other routable endpoints.
This has the following benefits:
Improved availability for apps by keeping routes in the Gorouter’s routing table when TTL expires
Increased guarantees against misrouting by validating the identity of back ends before forwarding requests
Increased security by encrypting data in flight from the Gorouter to back ends
As TAS for VMs manages and balances apps, the internal IP address and ports for app instances change. To keep the Gorouter’s routing tables current, a Route-Emitter on each Diego Cell sends periodic messages for each app instance running on that Diego Cell to all Gorouters through NATS. Each message includes the location and a unique identifier for the app instance to verify its identity when using TLS to communicate with the instance.
Network partitions or NATS failures can cause the Gorouter’s routing table to fall out of sync, as TAS for VMs continues to re-create containers across hosts to keep apps running. This can lead to routing of requests to incorrect destinations.
Before forwarding traffic to an app instance, the Gorouter initiates a TLS handshake with an Envoy proxy running in each app container. In the TLS handshake, the Envoy proxy presents a certificate generated by Diego for each container which uniquely identifies the container using the same app instance identifier ent by the Route-Emitter, configured in the certificate as a domain Subject Alternative Name (SAN). For more information, see Envoyproxy.io.
If the Gorouter confirms that the app instance identifier in the certificate matches the one received in the route registration message, the Gorouter forwards the HTTP request over the TLS session, and the Envoy proxy then forwards it to the app process. If the instance identifiers do not match, the Gorouter removes the app instance from its routing table and transparently retries another instance of the app.
Currently, only Linux cells support the Gorouter validating app instance identities using TLS by default. With Windows cells, the Gorouter connects to back ends without TLS, forwarding requests to Windows apps over plain text and pruning based on route TTL.
Verifying app identity using TLS improves resiliency and consistency for app routes.
The App Containers pane of the TAS for VMs tile includes these options under Gorouter app identity verification:
The Gorouter uses TLS to verify app identity: Enables the Gorouter to verify app identity using TLS. This is the default option.
The Gorouter and apps use mutual TLS to verify each other’s identity: Enables the Gorouter and your apps to verify each other’s identity using TLS.
Breaking Change: If you have mutual TLS app identity verification enabled, app containers accept incoming communication only from the Gorouter. This disables TCP routing.
To enable TLS to backends running on Windows cells, the same options can be configured under (Beta) Enable TLS Connections From Router To Applications on the Advanced Features tab of the Windows tile.
The Gorouter can be configured to use different load balancing algorithms for routing incoming requests to app instances. The Gorouter maintains a dynamically updated list of app instances for each route. Depending on which algorithm is selected, it forwards to one of the app instances.
To configure the behavior, you can change the value of
router.balancing_algorithm manifest property to either of the following options:
round-robin. For more information, see Round-Robin Load Balancing below.
least-connection. For more information, see Least-Connection Load Balancing below.
By default, the Gorouter uses the round-robin algorithm.
Incoming requests for a given route are forwarded to all app instances one after another, looping back to the first one after they have each received a request. This algorithm is suitable for most use cases and evenly distributes the load between app instances.
Each request for a given route is forwarded to the app instance with the least number of open connections. This algorithm can be more suitable for some cases. For example, if app instances have long-lived connections and are scaled up, then new instances receive fewer connections, causing a disproportionate load. In this case, choosing a least-connection algorithm sends new connections to new instances to equalize the load.
WebSockets is a protocol providing bi-directional communication over a single, long-lived TCP connection, commonly implemented by web clients and servers. WebSockets are initiated through HTTP as an upgrade request. The Gorouter supports this upgrade handshake and holds the TCP connection open with the selected app instance.
To support WebSockets, the operator must configure the load balancer correctly. Depending on the configuration, clients may have to use a different port for WebSocket connections, such as port 4443, or a different domain name. For more information, see Supporting WebSockets.
The Gorouter supports session affinity, or sticky sessions, for incoming HTTP requests to compatible apps.
With sticky sessions, when multiple instances of an app are running on TAS for VMs, requests from a particular client always reach the same app instance. This allows apps to store session data specific to a user session.
To support sticky sessions, configure your app to return a sticky session cookie in responses. The default value for this field is
JSESSIONID. You can
configure the cookie names that the routing tier uses for sticky sessions.
To configure the names of the cookies, see Configure Networking in Configuring TAS for VMs.
If an app returns a sticky session cookie to a client request, the TAS for VMs routing tier generates a unique
VCAP_ID for the app instance
based on its GUID with the same expiry, sameSite, and secure attributes as
JSESSIONID. For example:
On subsequent requests, the client must provide both the sticky session and
The CF routing tier uses the
VCAP_ID cookie to forward client requests to the same app instance every time. The sticky session cookie is forwarded to the
app instance to enable session continuity. If the app instance identified by the
VCAP_ID crashes, the Gorouter attempts to route the request to a different
instance of the app. If the Gorouter finds a healthy instance of the app, it initiates a new sticky session.
For more information, see Session Affinity on GitHub.
Note: TAS for VMs does not persist or replicate HTTP session data across app instances. If an app instance crashes or is stopped, session data for that instance is lost. If you require session data to persist across crashed or stopped instances, or to be shared by all instances of an app, store session data in a TAS for VMs marketplace service that offers data persistence.
The Gorouter supports keep-alive connections from clients and does not close the TCP connection with clients immediately after returning an HTTP response. Clients are responsible for closing these connections.
If keep-alive connections are disallowed, the Gorouter closes the TCP connection with an app instance or system component after receiving an HTTP response.
If keep-alive connections are allowed, the Gorouter maintains established TCP connections to back ends. The Gorouter supports up to 100 idle connections to each back end:
If an idle connection exists for a given back end, the Gorouter reuses it to route subsequent requests.
If no idle connection exists for this back end, the Gorouter creates a new connection.
If the Gorouter cannot establish a TCP connection with a selected app instance, the Gorouter considers the instance ineligible for requests for 30 seconds and transparently attempts to connect to another app instance. Once the Gorouter has established a TCP connection with an app instance, the Gorouter forwards the HTTP request.
When you deploy an app that requires Diego Cells to restart or recreate, the app may not respond to a Gorouter request before the keep-alive connection breaks. The following table describes how the Gorouter behaves if it cannot establish a TCP connection to an app:
|If the Gorouter…||and the back end…||then the Gorouter…|
|cannot establish a TCP connection to a back end||N/A||retries another back end no more than three times|
|establishes a TCP connection to a back end and forwards the request||does not respond||waits 15 minutes for a response, and if it receives an error, does not retry another back end|
|establishes a TCP connection to a back end and forwards the request||returns a TCP connection error||returns an error to the client, marks the back end ineligible, and does not retry another back end|
In all cases, if the app still does not respond to the request, the Gorouter returns a
502 error. For more
information, see Troubleshooting Router Error Responses.