Deploying PAS with NSX-T Networking

Page last updated:

This topic describes how to install Pivotal Application Service (PAS) on vSphere with NSX-T internal networking, using the VMware NSX-T Container Plug-in for PCF.

Introduction

PAS v2.3 uses a Container Network Interface (CNI) plugin to support secure and direct internal communication between containers. This plugin can be:

Requirements

  • An NSX-T 2.2 or later environment with NSX-T components installed and configured. See the NSX-T Cookbook for directions.
  • BOSH and Ops Manager v2.3 or later installed and configured on vSphere. See Deploying BOSH and Ops Manager to vSphere for directions.
  • The VMWare NSX-T Container Plug-in for PCF tile downloaded from Pivotal Network and imported to the Ops Manager Installation Dashboard. See Adding and Importing Products for how to download and import Pivotal products to the Installation Dashboard.
  • The PAS tile downloaded from Pivotal Network and imported to the Ops Manager Installation Dashboard. The PAS tile must be in one of the following two states:
    • Not deployed yet; you have not yet clicked Review Pending Changes, then Apply Changes on this version of PAS.
    • Deployed previously, with the Networking pane > Container Network Interface Plugin set to External.

      Note: Deploying PAS with its Container Network Interface (CNI) set to Silk configures Diego cells to use an internally-managed container network. Subsequently switching the CNI interface to External NSX-T leads to errors.


Architecture

The following diagram shows how to deploy an NSX-T machine to run PAS across multiple vSphere hardware clusters. NSX-T runs a Tier-0 (T0) router and multiple Tier-1 (T1) routers, each connecting to a network within Pivotal Cloud Foundry. Each vSphere hardware column cluster corresponds to an Availability Zone in PCF:

NSX & PAS Overview

When a developer pushes an app to a new Org for the first time, the NSX-T plugin triggers NSX-T to create a new T1 router and allocate an address range for the Org, on demand.

Install and Configure PAS and NSX-T

Installing NSX-T to run with PAS requires the following actions, which are described below:

  1. Set up NSX-T to Integrate with PAS

  2. Enable NSX-T Mode in the BOSH Director

  3. Configure PAS for External Container Networking

  4. Install and Configure the NSX-T Tile

Set up NSX-T to Integrate with PAS

  1. In vSphere, create logical network switches to correspond to the networks that PCF uses.

    1. Log into the NSX Manager Dashboard.
    2. Open the Switching tab.
    3. For each of these networks…
      • Infrastructure (BOSH and Ops Manager, defined in BOSH Director tile > Assign AZs and Networks pane)
      • Deployment (PAS, defined in PAS tile > Assign AZs and Networks pane)
      • Services and Dynamic Services (marketplace services and on-demand services, also defined in the PAS tile)
      • Isolation Segment (optional, defined in the Isolation Segment tile > Assign AZs and Networks pane)
        …do the following:
        1. Click Add New Logical Switch.
        2. Enter a name for the logical switch (e.g. infrastructure-ls, deployment-ls).
        3. Click Save. NSX
  2. Create T0 network address translation (NAT) rules to facilitate communication between Ops Manager and the BOSH Director.

    1. Create and configure a T0 router and router port for PCF.
    2. In NSX Manager, select your T0 Router and navigate to Services > NAT.
    3. Add a rule for destination NAT (DNAT) with:
      • The externally-specified destination IP address of incoming requests
      • The internal network address to translate the external address to NSX
    4. Add a rule for source NAT (SNAT) with:
      • The range of internal addresses that outgoing traffic may come from
      • The external IP address to put in each outgoing packet header, indicating its source NSX
  3. Create T1 Routers for PAS, to connect from the T0 router. For each PCF network, Infrastructure, Deployment, and so on, create a T1 router as follows:

    1. In the NSX Manager UI, navigate to Routing > Routers
    2. Click Add > Tier-1 Router.
    3. Configure the router. For example, the Infrastructure network router configuration might look like: NSX
  4. Create T1 router ports for PAS. For each T1 router you created, add a New Router Port as follows, to to allow traffic in and out:

    1. In the NSX Manager UI, select the T1 router.
    2. In Configuration > Router Ports, add a new router port.
    3. For Logical Switch, enter the name of the logical switch you defined for the network in Add New Logical Switch, above. NSX
  5. Advertise the routes of the T1 routers to the T0 router, so the T0 router can correctly route incoming requests based on their destination IP address:

    1. Select your T1 Router and navigate to Routing > Route Advertisement.
    2. Under Edit Route Advertisement Configuration, enable route advertisement by setting Status to Enabled.
    3. Set Advertise All NSX Connected Routes to Yes. NSX
  6. Add a tag for the T0 router.

    1. From NSX Manager, click router-t0 - Manage Tags
    2. Click + ADD and enter a unique identifier for the router, to associate it with PAS. NSX
  7. Allocate an IP Block for PAS Orgs.

    1. From NSX Manager, navigate to DDI > IPAM > New IP block.
    2. Enter a description, such as This is where IP addresses come from when a new Org is created in PAS.
    3. Enter a CIDR to allocate an address block large enough to accommodate all PAS apps. NSX
    4. Add a tag to the newly-created IP block. NSX
  8. Create a new switching profile with the following fields and tags:

    • Name: ncp-ha
    • Type: Spoof Guard
    • Tags: Add a tag with Scope ncp/ha, Tag True
    • Tags: Add a second tag with the Scope and Tag name of the T0 router tag you defined above. NSX
  9. Create an external SNAT IP pool.

    1. Navigate to Inventory > Groups > IP Pools > New IP Pool.
    2. Enter a name and a description. NSX
    3. Add two tags to the SNAT IP pool. NSX

Enable NSX-T Mode in the BOSH Director

  1. From the Ops Manager Installation Dashboard, open the BOSH Director tile.

  2. In the vCenter Configs pane, click the pencil icon for the vCenter Config you want to edit. Then select NSX Networking. Configure NSX-T mode in BOSH tile

  3. Configure NSX networking by entering the following:

    • NSX Mode: Select NSX-T.
    • NSX Address: Enter the address of the NSX Manager.
    • NSX Username: Enter the username to connect to the NSX Manager.
    • NSX Password: The password for the username specified above.
    • NSX CA Cert: A CA certificate in PEM format that authenticates to the NSX server. If the NSX Manager generated a self-signed certificate, you can retrieve the CA certificate using OpenSSL with the command openssl s_client -host NSX-ADDRESS -port 443 -prexit -showcerts.

      Note: To update NSX security group and load balancer information, see the Updating NSX Security Group and Load Balancer Information topic.

Configure PAS for External Container Networking

  1. If you have not already done so, download the PAS tile from Pivotal Network and import it to the Installation Dashboard. See Adding and Importing Products for directions. Ops Manager Installation Dashboard with NSX-T tile

  2. Configure PAS, following the directions in Deploying PAS on vSphere. When you configure Networking, select External under Container Networking Interface Plugin.
    NSX

Install and Configure the NSX-T tile

  1. If you have not already done so, download the VMware NSX-T tile version 2.3 or later from Pivotal Network and import it to the Installation Dashboard. See Adding and Importing Products for directions. Ops Manager Installation Dashboard with NSX-T tile

  2. Click the VMware NSX-T tile to open its Settings tab, and configure the NSX Manager pane as follows:

    • NSX Manager Address: NSX Manager host address or IP address
    • Use Client Certificates or Username/Password: Select Basic Authentication with Username and Password and enter NSX Manager Admin Username and Admin Password credentials in the fields underneath.
    • NSX Manager CA Cert: Obtain this certificate from NSX Manager as follows:
      1. ssh into NSX Manager using the admin account that you created when you deployed NSX Manager.
      2. From the NSX Manager command line, run get certificate api to retrieve the certificate. NSX-T tile config: NSX Manager
  3. Open and configure the NCP (NSX-T Container Plugin) pane as follows:

    • PAS Foundation Name: The tag string you gave to the t0 router in NSX Manager, above.
    • Overlay Transport Zone: A uniquely identifying string for the Transport Zone that you chose when you created logical switches for each network. This can be the name of the transport zone if no other zones in NSX share the same name, or else the UUID for the transport zone.
    • Tier-0 Router: A uniquely identifying string for the T0 router. This can be the tag string that you gave the router in NSX Manager if no other T0 routers in NSX share the same name, or else the UUID for the router.
    • Subnet Prefix of Container Networks: Subnet mask to set the address range size for apps in a single org. Defaults to 24. This number must be higher than the mask for all PAS orgs in the NSX Manager New IP Block pane, to define a each Org’s fraction of the total PAS address space.
    • Enable SNAT for Container Network: Enable this checkbox. NSX-T tile config: NCP
  4. In the NSX Node Agent pane, enable the Enable Debug Level of Logging for NSX Node Agent checkbox. NSX-T tile config: NSX Node Agent

  5. Click Save and return to the Installation Dashboard.

  6. After you have configured both the PAS tile and the VMware NSX-T tile, click Review Pending Changes, then Apply Changes to deploy PAS with NSX-T networking.

Automation

  • Concourse Pipelines: Configure NSX-T for PAS: This sample Concourse pipeline provides jobs setup switches, routers, an IP block, and an IP pool to be used by PAS.

  • PyNSXT: This is a Python library that can be used as a CLI to run commands against a VMWare NSX-T installation. It exposes operations for working with logical switches, logical routers, and pools. It uses version 2.1 of NSX-T for the swagger client.

Create a pull request or raise an issue on the source for this page in GitHub