Deploying PAS with NSX-T Networking

Page last updated:

This topic describes how to install Pivotal Application Service (PAS) on vSphere with NSX-T internal networking, using the VMware NSX-T Container Plug-in for PCF.

Overview

PAS uses a Container Network Interface (CNI) plugin to support secure and direct internal communication between containers. This plugin can be:

Prerequisites

Before deploying PAS with NSX-T networking, you must have the following:

  • An NSX-T environment with NSX-T components installed and configured. The NSX-T version must support the versions of NCP and PAS you intend to use. See the Support Matrix for relevant supported version combinations and see the VMware Documentation for NSX-T installation instructions.
  • BOSH and Ops Manager installed and configured on vSphere. See Deploying Ops Manager on vSphere and Configuring BOSH Director on vSphere.
  • The VMware NSX-T Container Plug-in for PCF tile downloaded from Pivotal Network and imported to the Ops Manager Installation Dashboard. See Adding and Importing Products for how to download and import Pivotal products to the Installation Dashboard.
  • The PAS tile downloaded from Pivotal Network and imported to the Ops Manager Installation Dashboard. The PAS tile must be in one of the following two states:
    • Configured but not deployed yet; you have not yet clicked Review Pending Changes, then Apply Changes on this version of PAS.
    • Deployed previously, with the Networking pane > Container Network Interface Plugin set to External.

      Note: Deploying PAS with its Container Network Interface (CNI) set to Silk configures Diego cells to use an internally-managed container network. Subsequently switching the CNI interface to External NSX-T leads to errors.

Architecture

The following diagram shows how to deploy an NSX-T machine to run PAS across multiple vSphere hardware clusters. NSX-T runs a Tier-0 (T0) router and multiple Tier-1 (T1) routers, each connecting to a network within Pivotal Cloud Foundry. Each vSphere hardware column cluster corresponds to an Availability Zone in PCF:

NSX-T & PAS Overview

When a developer pushes an app to a new Org for the first time, the NSX-T plugin triggers NSX-T to create a new T1 router and allocate an address range for the Org, on demand.

Install and Configure PAS and NSX-T

Installing NSX-T to run with PAS requires the following actions, which are described below:

  1. Set up NSX-T to Integrate with PAS

  2. Enable NSX-T Mode in the BOSH Director

  3. Configure PAS for External Container Networking

  4. Install and Configure the NSX-T Tile

Set up NSX-T to Integrate with PAS

To set up NSX-T to integrate with PAS, complete the procedures in the following sections:

Set up Logical Switches

To set up logical switches, do the following:

  1. In vSphere, create logical network switches to correspond to the networks that PCF uses.
    1. Log into the NSX-T Manager Dashboard.
    2. Open the Advanced Networking & Security tab at the top.
    3. Open the Switching tab on the left.
    4. For each of these networks…
      • Infrastructure (BOSH and Ops Manager, defined in BOSH Director tile > Assign AZs and Networks pane)
      • Deployment (PAS, defined in PAS tile > Assign AZs and Networks pane)
      • Services and Dynamic Services (marketplace services and on-demand services, also defined in the PAS tile)
      • Isolation Segment (optional, defined in the Isolation Segment tile > Assign AZs and Networks pane) …do the following:
        1. Click +ADD.
        2. Enter a name for the logical switch (e.g. PAS-Infrastructure, PAS-Deployment).
        3. Click ADD. NSX-T

Set up Routers

To set up routers, do the following:

  1. Create T0 network address translation (NAT) rules to communicate with Ops Manager

    1. Open the Advanced Networking & Security tab at the top
    2. Open the Routers tab on the left
    3. Select your T0 router
    4. Select the Services drop-down menu and choose NAT.
    5. Add a rule for destination NAT (DNAT) with:
      • The externally-specified destination IP address of incoming requests. If your Ops Manager has a DNS entry (e.g. opsmgr.example.com), this would be its IP address.
      • The internal network address of the Ops Manager NSX-T
    6. Add the corresponding source NAT (SNAT) rule with:
      • The externally-specified destination IP address
      • The internal network address of the Ops Manager NSX-T
    7. Add a rule for source NAT (SNAT) for the infrastructure and deployment networks:
      • The externally-specified destination IP address
      • The internal network address in CIDR notation NSX-T
  2. Create T1 Routers for PAS, to connect from the T0 router. For each PCF network, Infrastructure, Deployment, and so on, create a T1 router as follows:

  3. In the NSX-T Manager UI, navigate to Advanced Networking & Security > Routing > Routers

    1. Click +ADD > Tier-1 Router.
    2. Configure the router. Include the Edge Cluster and Edge Cluster Members; they are required to enable the Load Balancer. The Infrastructure network router configuration might look like: NSX-T
  4. Create T1 router downlink ports for PAS. For each T1 router you created, add a New Router Port as follows, to to allow traffic in and out:

    1. In the NSX-T Manager UI, select the T1 router.
    2. In Configuration > Router Ports, click +ADD to add a new router port.
    3. For Logical Switch, enter the name of the logical switch you defined for the network in Add New Logical Switch, above.
    4. For IP Address, use the first IP of the appropriate subnet. In this example, we have set aside 192.168.1.0/24 for Infrastructure (Operations Manager and BOSH Director), and 192.168.2.0/24 for the Deployment, so we used 192.168.1.1 and 192.168.2.1 respectively. NSX-T
  5. Advertise the routes of the T1 routers to the T0 router, so the T0 router can correctly route incoming requests based on their destination IP address:

    1. Select your T1 Router and navigate to Routing > Route Advertisement.
    2. Under Edit Route Advertisement Configuration, enable route advertisement by setting Status to Enabled.
    3. Set Advertise All Connected Routes to Yes.
    4. Set Advertise All LB VIP Routes to Yes (necessary if Load Balancing service is configured). NSX-T
  6. Allocate an IP Block for PAS Orgs.

    1. From the NSX-T Manager, navigate to Advanced Networking & Security > Networking > IPAM and click +ADD.
    2. Enter a name (e.g. PAS-container-ip-block). Later you will enter this IP Block name on the VMware NSX-T tile in the NCP section under IP Blocks of Container Networks.
    3. Enter a description, such as Subnets are allocated from this pool to each newly-created Org
    4. Enter a CIDR to allocate an address block large enough to accommodate all PAS apps. A /14 CIDR is large enough for ~1,000 Orgs with ~250 apps each. If you’re planning such a large foundation, pay heed to the VMware NSX-T PAS limits. NSX-T
  7. Create an external SNAT IP pool.

    1. Navigate to Advanced Networking & Security > Inventory > Groups > IP Pools and click +ADD.
    2. Enter a name (e.g. external-ip-pool) and a description (e.g. IP pool that provides 1 public IP for each Cloud Foundry Org). Later you will enter this pool name on the VMware NSX-T tile in the NCP section under IP Pools used to provide External (NAT) IP Addresses to Org Networks. NSX-T

Set up Load Balancer

To set up a load balancer, do the following:

  1. Create Active Health Monitors (health checks) for use by the Virtual Server later on:

    1. In the NSX-T Manager UI, navigate to Advanced Networking & Security > Networking > Load Balancing > Monitors > Active Health Monitors
    2. Create the health monitor for web load balancing:
    3. Click +ADD
    4. Fill out Monitor Properties:
      • Name: pas-web-monitor
      • Health Check Protocol: LbHttpMonitor
      • Monitoring Port: 8080
    5. Click Next
    6. Fill out Health Check Parameters:
      • HTTP Method: GET
      • HTTP Request URL: /health
      • HTTP Response Code: 200
    7. Click Finish
    8. Create the health monitor for TCP load balancing:
    9. Click +ADD
    10. Fill out Monitor Properties:
      • Name: pas-tcp-monitor
      • Health Check Protocol: LbHttpMonitor
      • Monitoring Port: 80
    11. Click Next
    12. Fill out Health Check Parameters:
      • HTTP Method: GET
      • HTTP Request URL: /health
      • HTTP Response Code: 200
    13. Create the health monitor for SSH load balancing:
    14. Click +ADD
    15. Fill out Monitor Properties:
      • Name: pas-ssh-monitor
      • Health Check Protocol: LbTcpMonitor
      • Monitoring Port: 2222
    16. Click Next > Finish
  2. Create Server Pools (collections of VMs which handle traffic) for use by the Virtual Server later on:

    1. In the NSX-T Manager UI, navigate to Advanced Networking & Security > Networking > Load Balancing > Server Pools
    2. Create the server pool for web load balancing:
    3. Click +ADD to add a new pool
    4. Fill out General Properties:
      • Name: pas-web-pool
    5. Click Next
    6. Fill out SNAT Translation:
      • Translation Mode: Auto Map
    7. Click Next
    8. Fill out Pool Members:
      • Membership Type: Static
    9. Click Next
    10. Fill out Health Monitors:
      • Active Health Monitor: pas-web-monitor
    11. Click Finish
    12. Create the server pool for TCP load balancing:
    13. Click +ADD to add new pool
    14. Fill out General Properties:
      • Name: pas-tcp-pool
    15. Click Next
    16. Fill out SNAT Translation:
      • Translation Mode: Transparent
    17. Click Next
    18. Fill out Pool Members:
      • Membership Type: Static
    19. Click Next
    20. Fill out Health Monitors:
      • Active Health Monitor: pas-tcp-monitor
    21. Click Finish
    22. Create the server pool for ssh load balancing:
    23. Click +ADD to add new pool
    24. Fill out General Properties:
      • Name: pas-ssh-pool
    25. Click Next
    26. Fill out SNAT Translation:
      • Translation Mode: Transparent
    27. Click Next
    28. Fill out Pool Members:
      • Membership Type: Static
    29. Click Next
    30. Fill out Health Monitors:
      • Active Health Monitor: pas-ssh-monitor
    31. Click Finish
  3. Create Virtual Servers:

    1. In the NSX-T Manager UI, navigate to Advanced Networking & Security > Networking > Load Balancing > Virtual Servers
    2. Create the Virtual Server which forwards unencrypted web (HTTP) traffic to the Foundation:

      Note: Foundations requiring end-to-end encryption should not enable the virtual server on port 80, or, if enabled, should configure it to redirect traffic to the encrypted port (443)

    3. Click +ADD
    4. Fill out General Properties:
      • Name: pas-web-vs
      • Application Type: Layer 4 (TCP)
      • Application Profile: nsx-default-lb-fast-tcp-profile
    5. Click Next
    6. Fill out Virtual Server Identifiers:
      • IP Address: use the address of the DNS record of *.system.YOUR-SYSTEM-DOMAIN.com
      • Port: 80,443
    7. Fill out Server Pool and Rules:
      • Default Server Pool: pas-web-pool
    8. Click Next several times, then Finish
    9. Create the Virtual Server which forwards traffic to apps with custom ports to the Foundation:
    10. Click +ADD to add a new Virtual Server
    11. Fill out General Properties:
      • Name: pas-tcp-vs
      • Application Type: Layer 4 (TCP)
      • Application Profile: nsx-default-lb-fast-tcp-profile
    12. Click Next
    13. Fill out Virtual Server Identifiers:
      • IP Address: use the address of the DNS record of tcp.apps.YOUR-SYSTEM-DOMAIN.com
      • Port: use the same ports as configured in the PAS Tile > Networking > TCP Routing Ports, e.g. 1024-1123,5900
    14. Click Next
    15. Fill out Server Pool and Rules:
      • Default Server Pool: pas-tcp-pool
    16. Click Next > Finish
    17. Create the Virtual Server which forwards ssh traffic to the Foundation:
    18. Click +ADD to add a new Virtual Server
    19. Fill out General Properties:
      • Name: pas-ssh-vs
      • Application Type: Layer 4 (TCP)
      • Application Profile: nsx-default-lb-fast-tcp-profile
    20. Click Next
    21. Fill out Virtual Server Identifiers:
      • IP Address: use the address of the DNS record of ssh.system.YOUR-SYSTEM-DOMAIN.com
      • Port: 2222
    22. Click Next
    23. Fill out Server Pool and Rules:
      • Default Server Pool: pas-ssh-pool
    24. Click Next > Finish
  4. Create the Load Balancer:

    1. In the NSX-T Manager UI, navigate to Advanced Networking & Security > Networking > Load Balancing > Load Balancers
    2. Click +ADD
    3. Fill out the fields:
      • Name: pas-lb
      • Load Balancer Size: Choose Small unless you have a larger Foundation
    4. Click OK
    5. Select pas-lb
    6. Click Actions > Attach to a Virtual Server, and then select pas-web-vs. Repeat this procedure for the Virtual Servers pas-tcp-vs and pas-ssh-vs.
    7. Click Action > Attach to a Logical Router, and then select T1-Router-PAS-Deployment

Enable NSX-T Mode in the BOSH Director

To enable NSX-T mode in the BOSH Director, do the following:

  1. From the Ops Manager Installation Dashboard, open the BOSH Director tile.

  2. In the vCenter Configs pane, click the pencil icon for the vCenter Config you want to edit. Then select NSX Networking. Configure NSX-T mode in BOSH tile

  3. Configure NSX networking by entering the following:

    • NSX Mode: Select NSX-T.
    • NSX Address: Enter the address of the NSX-T Manager.
    • NSX Username: Enter the username to connect to the NSX-T Manager.
    • NSX Password: The password for the username specified above.
    • NSX CA Cert: A CA certificate in PEM format that authenticates to the NSX-T server. If the NSX-T Manager generated a self-signed certificate, you can retrieve the CA certificate using OpenSSL with the command openssl s_client -host NSX-ADDRESS -port 443 -prexit -showcerts.

Configure PAS for External Container Networking

To configure PAS for external container networking, do the following:

  1. If you have not already done so, download the PAS tile from Pivotal Network and import it to the Installation Dashboard. See Adding and Importing Products for directions. Ops Manager Installation Dashboard with NSX-T tile

  2. Configure PAS, following the directions in Configuring PAS. When you configure Networking, select External under Container Networking Interface Plugin.
    NSX-T

  3. Update the server pool membership for the NSX-T Load Balancers by following Updating NSX-T Load Balancer Server Pool Membership. Make sure to make a VM extension for each of the three server pools: pas-web-pool, pas-tcp-pool, and pas-ssh-pool.

    Note: Without a VM extension, new or updated VMs are not added back into the NSX-T Load Balancer Server Pool.

Install and Configure the NSX-T tile

To install and configure the NSX-T tile, do the following:

  1. If you have not already done so, download the VMware NSX-T tile version 2.3 or later from Pivotal Network and import it to the Installation Dashboard. See Adding and Importing Products for directions. Ops Manager Installation Dashboard with NSX-T tile

  2. Click the VMware NSX-T tile to open its Settings tab, and configure the NSX Manager pane as follows:

    • NSX Manager Address: NSX-T Manager host address or IP address
    • Use Client Certificates or Username/Password: Select Basic Authentication with Username and Password and enter NSX Manager Admin Username and Admin Password credentials in the fields underneath.
    • NSX Manager CA Cert: Obtain this certificate from NSX-T Manager as follows:
      1. ssh into NSX-T Manager using the admin account that you created when you deployed NSX-T Manager.
      2. From the NSX-T Manager command line, run get certificate api to retrieve the certificate. NSX-T tile config: NSX-T Manager
  3. Open and configure the NCP (NSX-T Container Plugin) pane as follows:

    • PAS foundation Name: If unsure, use PAS. If multiple foundations co-exist on the same NSX-T Manager, choose a unique string, e.g. PAS-beta. NCP creates artifacts (e.g. T1 routers) and prefixes their names with this string for easy identification.
    • Overlay Transport Zone: A uniquely identifying string for the Transport Zone that you chose when you created logical switches for each network. This can be the name of the transport zone if no other zones in NSX-T share the same name, or else the UUID for the transport zone.
    • Tier-0 Router: A uniquely identifying string for the T0 router. This can be the tag string that you gave the router in NSX-T Manager if no other T0 routers in NSX-T share the same name, or else the UUID for the router.
    • Subnet Prefix of Container Networks: Subnet mask to set the address range size for apps in a single org. Defaults to 24. This number must be higher than the mask for all PAS orgs in the NSX-T Manager New IP Block pane, to define a each Org’s fraction of the total PAS address space.
    • Enable SNAT for Container Network: Enable this checkbox. NSX-T tile config: NCP
  4. In the NSX Node Agent pane, enable the Enable Debug Level of Logging for NSX Node Agent checkbox. NSX-T tile config: NSX-T Node Agent

  5. Click Save and return to the Installation Dashboard.

  6. After you have configured both the PAS tile and the VMware NSX-T tile, click Review Pending Changes, then Apply Changes to deploy PAS with NSX-T networking.

Upgrade PAS with NSX-T Networking

After you have deployed PAS with NSX-T, you may need to upgrade either Ops Manager, PAS, the NSX-T Container Plug-in (NCP) or NSX-T Data Center. If you upgrade one of these components, you may need to upgrade the other components as well.

For example, if you want to upgrade NSX-T Data Center, you may need to upgrade the NSX-T Container Plug-in first.

To upgrade PAS with NSX-T Networking, do the following:

  1. Plan the upgrade by determining the compatibility of NCP, NSX-T and PAS by checking the following documentation:

  2. Download the desired version of VMware NSX-T Container Plug-in for PCF tile from Pivotal Network.

  3. If you are upgrading from NSX-T Container Plug-in v2.3 to NSX-T Container Plug-in v2.4, delete any manually created Switching Profiles before proceeding with the upgrade. In the NSX-T Manager interface, do the following:

    • Select Networking, Switching, Switching Profiles
    • Select ncp-ha (for all clusters).
    • Click Delete.
  4. In Ops Manager, import the new version of the tile to the Installation Dashboard. See Adding and Importing Products for directions.

  5. Click Review Pending Changes and review your changes.

  6. Click Apply Changes.

  7. Continue with the upgrade of Ops Manager, PAS or NSX-T Data Center. For more information, see Upgrade NCP in a Pivotal Cloud Foundry Environment in the VMware NSX-T Data Center documentation.

Automation

For more information about automation, see the following resources:

  • Concourse Pipelines: Configure NSX-T for PAS: This sample Concourse pipeline provides jobs setup switches, routers, an IP block, and an IP pool to be used by PAS.

  • PyNSXT: This is a Python library that can be used as a CLI to run commands against a VMware NSX-T installation. It exposes operations for working with logical switches, logical routers, and pools. It uses version 2.1 of NSX-T for the swagger client.