Using Proxies with Enterprise PKS on NSX-T

Page last updated:

Warning: VMware Enterprise PKS v1.6 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes how to use proxies with VMware Enterprise PKS with NSX-T.

Overview

If your environment includes HTTP proxies, you can configure Enterprise PKS with NSX-T to use these proxies so that Enterprise PKS-deployed Kubernetes master and worker nodes access public Internet services and other internal services through a proxy.

In addition, Enterprise PKS proxy settings apply to the PKS API instance. When an Enterprise PKS operator creates a Kubernetes cluster, the PKS API instance VM behind a proxy is able to manage NSX-T objects on the standard network.

You can also proxy outgoing HTTP/HTTPS traffic from Ops Manager and the BOSH Director so that all Enterprise PKS components use the same proxy service.

The following diagram illustrates the network architecture:

PKS Proxy Architecture

Enable PKS API and Kubernetes Proxy

To configure a global HTTP proxy for all outgoing HTTP/HTTPS traffic from the Kubernetes cluster nodes and the PKS API server, perform the following steps:

  1. Navigate to Ops Manager and log in.

  2. Click the Enterprise PKS tile.

  3. Click Networking.

  4. Under HTTP/HTTPS proxy, select Enabled. When this option is enabled, you can proxy HTTP traffic, HTTPS traffic, or both.


    Networking pane configuration
    Production environments can deny direct access to public Internet services and between internal services by placing an HTTP or HTTPS proxy in the network path between Kubernetes nodes and those services.

    Configuring Enterprise PKS to use these proxies allows Enterprise PKS-deployed Kubernetes nodes to access public Internet services and other internal services.

    Note: This setting will not set the proxy for running Kubernetes workloads or pods.

    To complete your global proxy configuration for all outgoing HTTP/HTTPS traffic from your Kubernetes clusters, perform the following steps:

    1. To proxy outgoing HTTP traffic, enter the URL of your HTTP proxy endpoint under HTTP Proxy URL. For example, http://myproxy.com:1234.
    2. (Optional) If your outgoing HTTP proxy uses basic authentication, enter the username and password in the HTTP Proxy Credentials fields.
    3. To proxy outgoing HTTPS traffic, enter the URL of your HTTP proxy endpoint under HTTPS Proxy URL. For example, http://myproxy.com:1234.

      Note: Using an HTTPS connection to the proxy server is not supported. HTTP and HTTPS proxy options can only be configured with an HTTP connection to the proxy server. You cannot populate either of the proxy URL fields with an HTTPS URL. The proxy host and port can be different for HTTP and HTTPS traffic, but the proxy protocol must be HTTP.

    4. (Optional) If your HTTPS proxy uses basic authentication, enter the username and password in the HTTPS Proxy Credentials fields.
    5. Under No Proxy, enter the comma-separated list of IP addresses that must bypass the proxy to allow for internal Enterprise PKS communication.

      The list should include 127.0.0.1 and localhost.

      Also include:
      • Your Enterprise PKS environment’s CIDRs, such as the service network CIDR where your Enterprise PKS cluster is deployed, the deployment network CIDR, the node network IP block CIDR, and the pod network IP block CIDR.

      • If a component is communicating with Enterprise PKS or a registry using a hostname, such as the Harbor API FQDN, instead of an IP address you will need to add the corresponding FQDN to the No Proxy list.

      • If you are upgrading and have an existing proxy configuration for reaching a Docker registry or other external services, add the following IP addresses to the No Proxy field to prevent the Enterprise PKS to IaaS traffic from going through the proxy: NSX Manager, vCenter Server, and all ESXi hosts.

      • Any additional IP addresses or domain names that should bypass the proxy.

        The No Proxy property for vSphere accepts wildcard domains denoted by a prefixed *. or ..

        For example:
        127.0.0.1,localhost,
        *.example1.com,
        .example2.com,
        example3.com,
        198.51.100.0/24,
        203.0.113.0/24,
        192.0.2.0/24
        

        Note: By default the 10.100.0.0/8 and 10.200.0.0/8 IP address ranges, .internal, .svc,.svc.cluster.local, .svc.cluster, and your Enterprise PKS FQDN are not proxied. This allows internal Enterprise PKS communication.

        Do not use the _ character in the No Proxy field. Entering an underscore character in this field can cause upgrades to fail.

        Because some jobs in the VMs accept *. as a wildcard, while others only accept ., we recommend that you define a wildcard domain using both of them. For example, to denote example.com as a wildcard domain, add both *.example.com and example.com to the No Proxy property.

  5. Save the changes to the Enterprise PKS tile.

  6. Proceed with any remaining Enterprise PKS tile configurations and deploy Enterprise PKS. See Installing Enterprise PKS on vSphere with NSX-T.

Enable Ops Manager and BOSH Proxy

To enable an HTTP proxy for outgoing HTTP/HTTPS traffic from Ops Manager and the BOSH Director, perform the following steps:

  1. Navigate to Ops Manager and log in.

  2. Select User Name > Settings in the upper right.

  3. Click Proxy Settings.

    PKS Proxy Settings

  4. Under HTTP Proxy, enter the FQDN or IP address of the HTTP proxy endpoint. For example, http://myproxy.com:80.

  5. Under HTTPS Proxy, enter the FQDN or IP address of the HTTPS proxy endpoint. For example, http://myproxy.com:80.

    Note: Using an HTTPS connection to the proxy server is not supported. Ops Manager and BOSH HTTP and HTTPS proxy options can be only configured with an HTTP connection to the proxy.

  6. Under No Proxy, include the hosts that must bypass the proxy. This is required.

    In addition to 127.0.0.1 and localhost, include the BOSH Director IP and the PKS VM IP. The BOSH Director IP is typically the first IP address in the deployment network CIDR, and the PKS VM IP is the second IP address in the deployment network CIDR. In addition, be sure to include the Ops Manager IP address in the No Proxy field as well.

    127.0.0.1,localhost,BOSH-DIRECTOR-IP,PKS-VM-IP,OPS-MANAGER-IP
    

    Note: Ops Manager does not allow the use of a CIDR range in the No Proxy field. You must specify each individual IP address to bypass the proxy.

    The No Proxy field does not accept wildcard domain notation, such as *.docker.io and *.docker.com. You must specify the exact IP or FQDN to bypass the proxy, such as registry-1.docker.io.

  7. Click Save.

  8. Return to the Ops Manager Installation Dashboard and click Review Pending Changes.

  9. Click Apply Changes to deploy Ops Manager and the BOSH Director with the updated proxy settings.


Please send any feedback you have to pks-feedback@pivotal.io.