LATEST VERSION: v1.3 - RELEASE NOTES
Pivotal Container Service v1.3

Creating the PKS Management Plane

Page last updated:

Prepare the vSphere and NSX-T infrastructure for the PKS Management Plane where the PKS, Ops Manager, BOSH Director, and Harbor Registry VMs are deployed.

Prerequisites

Before you begin this procedure, ensure that you have reviewed the following documentation for installing PKS on vSphere with NSX-T:

In addition, ensure that you have successfully deployed NSX-T for PKS. For more information, see Deploying NSX-T for PKS.

About the PKS Management Plane

The PKS Management Plane is the network for PKS Management components, including PKS, Ops Manager, and BOSH Director. The PKS Management Plane includes a vSphere resource pool for Management Plane components, as well as a NSX Tier-1 Logical Switch, Tier-1 Logical Router, and Router Port, as well as NSX NAT rules.

If you are using either the the NAT deployment topology or the No-NAT with Logical Switch deployment topology, create a Tier-1 (T1) Logical Switch, and a Tier-1 Logical Router and Port. Link the T1 logical router to the T0 logical router, and select the Edge Cluster defined for PKS. Enable route advertisement for the T1 Logical Router and advertise All NSX connected routes for the PKS Management Plane VMs (PKS, Ops Manager, and BOSH Director).

If you are using the NAT Topology, create the following NAT rules on the T0 Router.

  • Destination NAT (DNAT) rule that maps an external IP address from the PKS MANAGEMENT CIDR to the IP where you deploy Ops Manager on the PKS Management logical switch. For example, a DNAT rule that maps 10.172.1.2 to 172.31.0.2, where 172.31.0.2 is the IP address you assign to Ops Manager when connected to ls-pks-mgmt.
  • (Optional) Destination NAT (DNAT) rule that maps an external IP address from the PKS MANAGEMENT CIDR to the IP where you deploy Harbor on the PKS Management logical switch. For example, a DNAT rule that maps 10.172.1.3 to 172.31.0.3, where 172.31.0.3 is the IP address you assign to Harbor when connected to ls-pks-mgmt.
  • Source NAT (SNAT) rule to allow the PKS Management VMs to communicate with your vCenter and NSX Manager environments. For example, an SNAT rule that maps 172.31.0.0/24 to 10.172.1.1, where 10.172.1.1 is a routable IP address from your PKS MANAGEMENT CIDR.
  • SNAT rule for PKS management components to access ESXi Hosts.
  • (Optional) SNAT rules for access to other management servers, such as DNS, NTP, and LDAP/AD.

Lastly, for both NAT and no-NAT mode, if you want developers to be able to access the PKS API (that is, use the PKS CLI) from their workstations or laptops, you must share the PKS API endpoint to allow your organization to use the API to create, update, and delete clusters. For more information, see Creating Clusters.

Developers should use the DNAT IP address when logging in with the PKS CLI. For more information, see Using PKS. To create this DNAT rule, see Create DNAT Rule on T0 Router for External Access to the PKS CLI.

Step 1. Create vSphere Resource Pool for the PKS Management Plane

  1. Log in to vCenter for your vSphere environment.
  2. Select Compute Cluster > New Resource Pool.
  3. Name the resource pool, such as RP-MGMT-PKS.
  4. Click OK
  5. Verify resource pool creation.

Step 2. Create NSX-T Logical Switch for the PKS Management Plane

  1. In NSX Manager, select Switching > Add.
  2. Create a new logical switch. For example:
  3. Click Add.
  4. Verify logical switch creation.

Step 3. Create NSX-T Tier-1 Router for the PKS Management Plane

Defining a T1 router involves creating the router and attaching it to the logical switch, creating a router port, and advertising the routes.

Create T1 Router

  1. In NSX Manager, select Routing > Add > Tier-1 Router.
  2. Configure the T1 router. For example:
  3. Click Add.
  4. Verify T1 router creation.

Create T1 Router Port

  1. Select the T1 router you created.
  2. Select Configuration > Router Ports.
  3. Click Add and configure the T1 router port. For example:
    • Name: T1-MGMT-PKS-PORT
    • Logical Switch: select LS-MGMT-PKS
    • IP Address/mask: 10.0.0.1/24
  4. Click Add.
  5. Verify T1 router port creation.
  1. Select the T1 router > Routing > Route Advertisement.
  2. Advertise the T1 route as follows:
    • Status: enabled
    • Advertise all NSX connected routes: yes
  3. Click Save.s
  4. Verify route advertisement.

Verify T1 Router

  1. Select the T1 Router > Overview.
  2. Select Tier-0 Connection > Connect, then select the T0 router and click Connect.
  3. Verify connectivity between T1 and T0 routers.
  4. Select the T1 router > Router ports. The T1 router created for the PKS Management Plane should have 2 ports: one connected to the T0 router, and a second port connected to logical switch defined for the PKS Management Plane. This second port will be the default gateway for all VMs connected to this LS.

Step 4. Create DNAT Rule on T0 Router for Ops Manager

Create a DNAT rule on the T0 Router to access the Ops Manager Web UI, which is required to deploy PKS.

The Destination NAT (DNAT) rule on the T0 maps an external IP address from the PKS MANAGEMENT CIDR to the IP where you deploy Ops Manager on the PKS Management logical switch that you created on the T0 router. For example, a DNAT rule that maps 10.172.1.2 to 172.31.0.2, where 172.31.0.2 is the IP address you assign to Ops Manager when connected to ls-pks-mgmt.

To create a DNAT rule for Ops Manager:

  1. In NSX Manager, select Routing > Routers.
  2. Select the T0 Router > Services > NAT.
  3. Add and configure a DNAT rule with the routable IP address as the destination and the internal IP address for Ops Manager as the translated IP. For example:
    • Priority: 1000
    • Action: DNAT
    • Destination IP: 10.40.14.1
    • Translated IP: 10.0.0.2
  4. Click Add.
  5. Verify the DNAT rule.

Step 5. Create DNAT Rule on T0 Router for Harbor Registry

If you are using VMware Harbor Registry with PKS, create a similar DNAT rule on T0 router to access the Harbor Web UI. This DNAT rule maps the private Harbor IP address to a routable IP address from the floating IP pool on the PKS Management network. See Create DNAT Rule in the VMware Harbor Registry documentation for instructions.

Step 6. Create SNAT rule on T0 router for vCenter and NSX Manager

Create a SNAT rule on T0 router for PKS management components to access vCenter and NSX manager. The Source NAT (SNAT) rule on the T0 allows the PKS Management VMs to communicate with your vCenter and NSX Manager environments. For example, a SNAT rule that maps 172.31.0.0/24 to 10.172.1.1, where 10.172.1.1 is a routable IP address from your PKS MANAGEMENT CIDR.

Note: Limit the Destination CIDR for the SNAT rules to the subnets that contain your vCenter and NSX Manager IP addresses.

  1. Select T0 router > Services > NAT.
  2. Click ADD and configure the SNAT rule. For example:
    • Priority: 1010
    • Action: SNAT
    • Source: 10.0.0.0/24
    • Destination IP: 10.40.206.0/24
    • Translated IP: 10.40.14.2
  3. Click Add.
  4. Verify SNAT rule creation.

Step 7. Create SNAT Rules on T0 Router for DNS, NTP, and LDAP/AD

  1. In NSX Manager, select T0 router > Services > NAT.
  2. Add a SNAT rule for DNS. For example:
    • Priority: 1010
    • Action: SNAT
    • Source: 10.0.0.0/24
    • Destination IP: 10.20.20.1
    • Translated IP: 10.40.14.2
  3. Click Add.
  4. Add a SNAT rule for NTP. For example:
    • Priority: 1010
    • Action: SNAT
    • Source: 10.0.0.0/24
    • Destination IP: 10.113.60.176
    • Translated IP: 10.40.14.2
  5. Click Add.
  6. Add a SNAT rule for LDAP/AD. For example:
    • Priority: 1010
    • Action: SNAT
    • Source: 10.0.0.0/24
    • Destination IP: 10.40.207.0/24
    • Translated IP: 10.40.14.2
  7. Click Add.
  8. Verify SNAT rule creation.

Step 8. Create SNAT Rule on T0 Router for ESXi Hosts

Create a SNAT rule on T0 router for PKS management components to access ESXi Hosts (Management IP). The Destination IP is the Management IP subnet where ESXi Hosts are networked.

Note: Ops Manager and BOSH must use the NFCP protocol to the actual ESX hosts to which it is uploading stemcells. Specifically, Ops Manager & BOSH Director -> ESXi.

  1. Select T0 router > Services > NAT.
  2. Click Add and configure the SNAT rule. For example:
    • Priority: 1010
    • Action: SNAT
    • Destination IP: 10.115.40.0/24
    • Translated IP: 10.40.14.2
  3. Click Add.
  4. Verify SNAT rule creation:

(Optional) Step 9. Create DNAT Rule on T0 Router for External Access to the PKS CLI

This DNAT rule is optional depending on whether or not you need to provide external access to the PKS CLI. If you do need to provide external access, this rule is needed for both NAT and no-NAT modes.

Note: You cannot create this rule until after PKS is installed and the PKS API VM has an IP address.

  1. When the PKS installation is completed, retrieve the PKS endpoint by performing the following steps:
    1. From the Ops Manager Installation Dashboard, click the Pivotal Container Service tile.
    2. Click the Status tab and record the IP address assigned to the Pivotal Container Service job.
  2. Create a DNAT rule on the shared Tier-0 router to map an external IP from the PKS MANAGEMENT CIDR to the PKS endpoint. For example, a DNAT rule that maps 10.172.1.4 to 172.31.0.4, where 172.31.0.4 is PKS endpoint IP address on the ls-pks-mgmt NSX-T Logical Switch.

    Note: Ensure that you have no overlapping NAT rules. If your NAT rules overlap, you cannot reach PKS Management Plane from VMs in the vCenter network.

Next Step

After you complete this procedure, follow the instructions in Creating the PKS Compute Plane.


Please send any feedback you have to pks-feedback@pivotal.io.

Create a pull request or raise an issue on the source for this page in GitHub