Creating the Enterprise PKS Management Plane

Page last updated:

Prepare the vSphere and NSX-T infrastructure for the Enterprise PKS Management Plane where the PKS, Ops Manager, BOSH Director, and Harbor Registry VMs are deployed.

Prerequisites

Before you begin this procedure, make sure you have completed the following prerequisites for installing Enterprise PKS on vSphere with NSX-T:

About the Enterprise PKS Management Plane

The Enterprise PKS Management Plane is the network for PKS Management Plane components, including the PKS API Server, Ops Manager, BOSH Director, and Harbor Registry. The Enterprise PKS Management Plane includes a vSphere resource pool for Management Plane components, as well as a NSX Tier-1 Logical Switch, Tier-1 Logical Router, and Router Port, and NSX-T NAT rules on the Tier-0 Router.

For all types of NSX-T deployment topologies, create a Tier-1 (T1) Logical Switch and a Tier-1 Logical Router and Port. Enable route advertisement for the T1 Logical Router and advertise All NSX connected routes for the Management Plane VMs. Lastly, link the Tier-1 Router to the Tier-0 Router.

If you are using the NAT Topology, you will also need to create the following NAT rules on the Tier-0 Router:

  • Source NAT (SNAT) rule to allow the PKS Management VMs to communicate with your vCenter and NSX Manager environments. For example, an SNAT rule that maps 172.31.0.0/24 to 10.172.1.1, where 10.172.1.1 is a routable IP address from your PKS MANAGEMENT CIDR.
  • Destination NAT (DNAT) rule that maps an external IP address from the PKS MANAGEMENT CIDR to the IP where you deploy Ops Manager on the Management Plane logical switch. For example, a DNAT rule that maps 10.172.1.2 to 172.31.0.2, where 172.31.0.2 is the IP address you assign to Ops Manager when connected to ls-pks-mgmt.
  • Destination NAT (DNAT) rule that maps an external IP address from the PKS MANAGEMENT CIDR to the IP where you deploy Harbor on the Management Plane logical switch. For example, a DNAT rule that maps 10.172.1.3 to 172.31.0.3, where 172.31.0.3 is the IP address you assign to Harbor when connected to ls-pks-mgmt.

Lastly, if you want to provide users with remote access to the PKS CLI, you will need to define a DNAT rule that maps an external IP address from the PKS MANAGEMENT CIDR to the IP where you deploy the PKS API Server on the Management Plane logical switch. This rule lets developers use the PKS CLI remotely from their workstations or laptops. Such a rule is needed for both NAT and No-NAT topologies.

Step 1. Create vSphere Resource Pool for the Management Plane

  1. Log in to vCenter for your vSphere environment.
  2. Select Compute Cluster > New Resource Pool.
  3. Name the resource pool, such as RP-MGMT-PKS.
  4. Click OK.
  5. Verify resource pool creation.

Step 2. Create NSX-T Logical Switch for the Management Plane

  1. In NSX Manager, select Switching > Add.
  2. Create a new logical switch.
  3. Click Add.
  4. Verify logical switch creation.

Step 3. Create NSX-T Tier-1 Router for the Management Plane

Defining a Tier-1 Router involves creating the router and attaching it to the logical switch, creating a router port, and advertising the routes.

Create Tier-1 Router

  1. In NSX Manager, select Routing > Add > Tier-1 Router.

  2. Configure the Tier-1 Router.

  3. Click Add.

  4. Verify Tier-1 Router creation.

Create Tier-1 Router Port

  1. Select the Tier-1 Router you created.

  2. Select Configuration > Router Ports.

  3. Click Add and configure the Tier-1 Router Port.

    • Name: T1-MGMT-PKS-PORT
    • Logical Switch: Select LS-MGMT-PKS from the menu list
    • IP Address/mask: 10.0.0.1/24 (for example)
  4. Click Add.

  5. Verify Tier-1 Router Port creation.

  1. Select the Tier-1 Router > Routing > Route Advertisement.
  2. Advertise the Tier-1 routes as follows:
    • Status: enabled
    • Advertise all NSX connected routes: yes
  3. Click Save.
  4. Verify route advertisement.

Step 4. Connect the Tier-1 Router to the Tier-0 Router

Connect the Tier-1 router to the Tier-0 router and verify router-to-router connectivity.

  1. Select the Tier-1 Router > Overview screen and verify the configuration of the Tier-1 Router up to this point.

  2. At the Tier-1 Router > Overview screen, select the option Tier-0 Connection > Connect.

  3. At the Connect to Tier-0 Router, select the Tier-0 Router and click Connect.

  4. Verify connectivity between the Tier-1 and Tier-0 Routers.

  5. Select the Tier-1 Router > Configuration > Router ports.

The Tier-1 Router created for the Management Plane should have 2 port connections: one connected to the Tier-0 router, and a second port connected to the logical switch defined for the Management Plane. This second port is the default gateway for all VMs connected to this logical switch.

Step 5. Create SNAT Rule on the Tier-0 Router for vCenter and NSX Manager

Create a Source NAT (SNAT) rule on the Tier-0 Router for Enterprise PKS management components to access vCenter and NSX manager. The SNAT rule on the Tier-0 Router allows the Management Plane VMs to communicate with the vCenter and NSX-T Management environments. For example, create a SNAT rule that maps 172.31.0.0/24 to 10.172.1.1, where 10.172.1.1 is a routable IP address from your PKS MANAGEMENT CIDR.

Note: Limit the Destination CIDR for the SNAT rules to the subnets that contain your vCenter and NSX Manager IP addresses.

  1. Select Tier-0 Router > Services > NAT.
  2. Click ADD and configure the SNAT rule:

    • Priority: 1010 (for example)
    • Action: SNAT
    • Source: 10.0.0.0/24 (for example)
    • Destination IP: 10.40.206.0/24 (for example)
    • Translated IP: 10.40.14.2 (for example)
  3. Click Add.

  4. Verify SNAT rule creation.

Step 6. Create DNAT Rule on the Tier-0 Router for Ops Manager

Create a DNAT rule on the T0 Router to access the Ops Manager Web UI, which is required to deploy Enterprise PKS.

The Destination NAT (DNAT) rule on the Tier-0 Router maps an external IP address from the PKS MANAGEMENT CIDR to the IP where you deploy Ops Manager on the Management Plane logical switch. For example, a DNAT rule that maps 10.172.1.2 to 172.31.0.2, where 172.31.0.2 is the IP address you assign to Ops Manager when connected to ls-pks-mgmt.

To create a DNAT rule for Ops Manager:

  1. In the NSX-T Manager interface, select Routing > Routers.
  2. Select the T0 Router > Services > NAT.
  3. Add and configure a DNAT rule with the routable IP address as the destination and the internal IP address for Ops Manager as the translated IP:
    • Priority: 1000 (for example)
    • Action: DNAT (for example)
    • Destination IP: 10.40.14.1 (for example)
    • Translated IP: 10.0.0.2 (for example)
  4. Click Add.
  5. Verify the DNAT rule you defined.

Step 7. Create DNAT on the Tier-0 Router for Harbor Registry

If you are using VMware Harbor Registry with Enterprise PKS, create a DNAT rule on the Tier-0 router to access the Harbor Web UI. This DNAT rule maps the private Harbor IP address to a routable IP address from the floating IP pool on the Enterprise PKS Management network.

See Create DNAT Rule in the VMware Harbor Registry documentation for instructions.

Step 8. Create DNAT Rule on T0 Router for External Access to the PKS CLI

This DNAT rule is optional depending on whether or not you need to provide external access to the PKS CLI. If you do need to provide external access, this rule is needed for both NAT and no-NAT modes.

Note: You cannot create this rule until after Enterprise PKS is installed and the PKS API VM has an IP address.

  1. When the Enterprise PKS installation is completed, retrieve the Enterprise PKS endpoint by performing the following steps:
    1. From the Ops Manager Installation Dashboard, click the Enterprise PKS tile.
    2. Click the Status tab and record the IP address assigned to the Pivotal Container Service job.
  2. Create a DNAT rule on the shared Tier-0 router to map an external IP from the PKS MANAGEMENT CIDR to the Enterprise PKS endpoint. For example, a DNAT rule that maps 10.172.1.4 to 172.31.0.4, where 172.31.0.4 is the Enterprise PKS endpoint IP address on the ls-pks-mgmt NSX-T Logical Switch.

    Note: Ensure that you have no overlapping NAT rules. If your NAT rules overlap, you cannot reach Enterprise PKS Management Plane from VMs in the vCenter network.

Next Step

After you complete this procedure, follow the instructions in Creating the Enterprise PKS Compute Plane.


Please send any feedback you have to pks-feedback@pivotal.io.