Configuring BOSH Director with NSX-T for Enterprise PKS

Page last updated:

Warning: VMware Enterprise PKS v1.6 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes how to configure BOSH Director for vSphere with NSX-T integration for VMware Enterprise PKS.


Before you begin this procedure, ensure that you have successfully completed all preceding steps for installing Enterprise PKS on vSphere with NSX-T, including:

How Pivotal Platform Accesses NSX Manager

To create, delete, and modify NSX-T networking resources, Pivotal Platform tiles and APIs use a VMware NSX Manager account with the Enterprise Administrator role and permissions.

Users configure Pivotal Platform to authenticate to NSX Manager for different purposes in different tiles:

Step 1: Log in to Ops Manager

  1. Log in to Ops Manager with the username and password credentials that you set up in Configure Ops Manager for Enterprise PKS.

  2. Click the BOSH Director for vSphere tile.

    Config bosh 01

Step 2: Configure vCenter for Enterprise PKS

  1. Select vCenter Config.

    Config bosh 02

  2. Enter the following information:

    • Name: A name that you provide for your vCenter configuration. This field is used to identify the datacenter configuration in Ops Manager if you are configuring multiple datacenters.
    • vCenter Host: The hostname of the vCenter that manages ESXi/vSphere.
    • vCenter Username: A vCenter username with create and delete privileges for virtual machines (VMs) and folders.
    • vCenter Password: The password for the vCenter user specified above.
    • Datacenter Name: The name of the datacenter as it appears in vCenter.
    • Virtual Disk Type: The Virtual Disk Type to provision for all VMs. For guidance on selecting a virtual disk type, see vSphere Virtual Disk Types.
    • Ephemeral Datastore Names (comma delimited): The names of the datastores that store ephemeral VM disks deployed by Ops Manager.
    • Persistent Datastore Names (comma delimited): The names of the datastores that store persistent VM disks deployed by Ops Manager.

    Note: The vSphere datastore type must be Datastore. Enterprise PKS does not support the use of vSphere Datastore Clusters with or without Storage DRS. For more information, see Datastores and Datastore Clusters in the vSphere documentation.

  3. Select NSX Networking, then select NSX-T.

    Config bosh 03

  4. Configure NSX-T networking as follows:

    • NSX Address: Enter the IP address of the NSX Manager host.
    • NSX Username and NSX Password: Enter the NSX Manager username and password.
    • NSX CA Cert: Provide the CA certificate in PEM format that authenticates to the NSX server. Open the NSX CA Cert that you generated and copy/paste its content to this field.
  5. Configure the following folder names:

    • VM Folder: The vSphere datacenter folder where Ops Manager places VMs. Enter pks_vms.
    • Template Folder: The vSphere datacenter folder where Ops Manager places VMs. Enter pks_templates.
    • Disk path Folder: The vSphere datastore folder where Ops Manager creates attached disk images. You must not nest this folder. Enter pks_disk.

    Note: After your initial deployment, you cannot edit the VM Folder, Template Folder, and Disk path Folder names.

    Config bosh 04

  6. Click Save.

    Config bosh 05

Step 3: Configure BOSH Director

  1. Select Director Config.

    Config bosh 06

  2. In the NTP Servers (comma delimited) field, enter your NTP server addresses.

    Note: The NTP server configuration only updates after VM recreation. Ensure that you select the Recreate all VMs checkbox if you modify the value of this field.

  3. Leave the JMX Provider IP Address field blank.

  4. Leave the Bosh HM Forwarder IP Address field blank.

  5. Select the Enable VM Resurrector Plugin to enable BOSH Resurrector functionality.

  6. Select Enable Post Deploy Scripts to run a post-deploy script after deployment. This script allows the job to execute additional commands against a deployment.

    Note: You must enable post-deploy scripts to install Enterprise PKS.

  7. Select Recreate all VMs to force BOSH to recreate all VMs on the next deploy. This process does not destroy any persistent disk data.

  8. For typical Enterprise PKS deployments, the default settings for all other BOSH Director configuration parameters are suitable. Optionally you can apply additional configurations to BOSH Director. See Director Config Pane in Configuring BOSH Director on vSphere in the Pivotal Platform documentation for details.

    Note: If you need to be able to remotely access the BOSH Director VM using the BOSH CLI, and you are deploying Enterprise PKS with NSX-T in a NAT topology, you must provide the Director Hostname for BOSH at the time of installation. See Director Config Pane in Configuring BOSH Director on vSphere in the Pivotal Platform documentation for details.

  9. Click Save.

    Config bosh 07

Step 4: Create Availability Zones

Ops Manager Availability Zones correspond to your vCenter clusters and resource pools.

Multiple Availability Zones allow you to provide high-availability and load balancing to your applications. When you run more than one instance of an application, Ops Manager balances those instances across all of the Availability Zones assigned to the application.

For a highly available installation of your chosen runtime, use at least three availability zones.

Note: For more information about AZs and high availability in vSphere, see Compute and HA Considerations in vSphere Reference Architecture.

  1. Select Create Availability Zones.

    Config bosh 08

  2. Use the following steps to create one or more Availability Zones for Enterprise PKS to use:

    • Click Add and create the Enterprise PKS Management AZ.
    • Enter a unique Name for the Availability Zone, such as AZ-MGMT.
    • Select the IaaS configuration (vSphere/vCenter).
    • Enter the name of an existing vCenter Cluster to use as an Availability Zone, such as COMP-Cluster-1.
    • Enter the name of the Enterprise PKS Management Resource Pool in the vCenter cluster that you specified above, such as RP-MGMT-PKS. The jobs running in this Availability Zone share the CPU and memory resources defined by the pool.
    • Click Add Cluster and create at least one Enterprise PKS Compute AZ.
    • Specify the Cluster and the Resource Pool, such as RP-PKS-AZ. Alternatively, specify the Cluster and the Host Group. See Using vSphere Host Group for more information.
    • Add additional clusters as necessary. Click the trash icon to delete a cluster. The first cluster cannot be deleted.

    Config bosh 09

    Config bosh 11

    Config bosh 13

    Config bosh 29 host group

  3. Click Save.

    Config bosh 12

Step 5: Create Networks

  1. Select Create Networks.

    Config bosh 15

  2. Select Enable ICMP checks to enable ICMP on your networks. Ops Manager uses ICMP checks to confirm that components within your network are reachable.

  3. Click Add Network.

    Config bosh 16

  4. Create the following network:

    • NET-MGMT-PKS: Network for Ops Manager, BOSH Director, and the PKS API. This network maps to the NSX logical switch created for the Enterprise PKS Management Network. See Creating Enterprise PKS Management Plane.

    Note: NSX-T automatically creates the service network to be used by the master and worker nodes (VMs) for Kubernetes clusters managed by Enterprise PKS. You should not manually create this network.

    Use the following values as a guide when you define the network in BOSH. Replace the IP addresses with ranges you defined for the Enterprise PKS Management Network.. Reserve any IP addresses from the subnet that are already in use, such as the IP for Ops Manager and subnet gateway.

    Field Configuration
    vSphere Network Name LS-MGMT-PKS
    Reserved IP Ranges

  5. Select the AZ-MGMT Availability Zone to use with the NET-MGMT-PKS network.

    Note: Do not select the COMPUTE network at this point in the configuration. It will be configured at the end of the procedure.

  6. Click Save.

    Config bosh 17

Step 6: Assign AZs and Networks

  1. Select Assign AZs and Networks.

    Config bosh 18

  2. Use the drop-down menu to select a Singleton Availability Zone. The Ops Manager Director installs in this Availability Zone. For Enterprise PKS, this will be the AZ-MGMT availability zone.

  3. Use the drop-down menu to select a Network for BOSH Director. BOSH Director runs on the Enterprise PKS Management Plane network. Select the NST-MGTM-PKS network.

  4. Click Save.

    Config bosh 19

Step 7: Configure Security

  1. Select Security.

  2. In Trusted Certificates, enter a custom certificate authority (CA) certificate to insert into your organization’s certificate trust chain. This feature allows all BOSH-deployed components in your deployment to trust a custom root certificate.

    If you are using a private Docker registry, such as VMware Harbor, use this field to enter the certificate for the registry. See Integrating Harbor Registry with Enterprise PKS for details.

  3. Choose Generate passwords or Use default BOSH password. Use the Generate passwords option for increased security.

  4. Click Save. To view your saved Director password, click the Credentials tab.

Step 8: Configure BOSH DNS

  1. Select BOSH DNS Config.

  2. (Optional) In Excluded Recursors, enter a list of prohibited recursor addresses.

  3. (Optional) In Recursor Timeout, enter a time limit for contacting the connected recursors. This includes dialing, writing, and reading from the recursor. If any of these actions exceeds the time limit you set, the action fails.

    Note: This time limit must include one of the Go parse duration time units. For example, entering 5s sets the timeout limit to five seconds. For more information about supported time units, see func ParseDuration in the Go Programming Language documentation.

  4. (Optional) In Handlers, enter a list of custom domain handlers in JSON format.

  5. Click Save.

Step 9: Configure Logging

  1. Select Syslog.

  2. (Optional) To send BOSH Director system logs to a remote server, select Yes.

  3. In the Address field, enter the IP address or DNS name for the remote server.

  4. In the Port field, enter the port number that the remote server listens on.

  5. In the Transport Protocol dropdown menu, select TCP or UDP. This selection determines which transport protocol is used to send the logs to the remote server.

  6. (Optional) Select the Enable TLS checkbox to send encrypted logs to remote server with TLS. After you select the checkbox, perform the following steps:

    1. Enter either the name or SHA1 fingerprint of the remote peer in Permitted Peer.
    2. Enter the SSL certificate for the remote server in SSL Certificate.

      Note: For an optimal security configuration, enable TLS encryption when you are forwarding logs. Logs can contain sensitive information, such as cloud provider credentials.

  7. (Optional) Enter an integer in Queue Size. This value specifies the number of log messages held in the buffer. The default value is 100,000.

  8. (Optional) Select the checkbox to Forward Debug Logs to an external source. This option is deselected by default. If you select it, you may generate a large amount of log data.

  9. (Optional) Enter configuration details for rsyslog in the Custom rsyslog Configuration field. This field requires the rainerscript syntax.

  10. Click Save Syslog Settings.

Step 10: Configure Resources

  1. Select Resource Config.

  2. Adjust any values as necessary for your deployment. Under the Instances, Persistent Disk Type, and VM Type fields, choose Automatic from the drop-down menu to allocate the recommended resources for the job. If the Persistent Disk Type field reads None, the job does not require persistent disk space.

    Note: Ops Manager requires a Director VM with at least 8 GB memory.

    Note: If you set a field to Automatic and the recommended resource allocation changes in a future version, Ops Manager automatically uses the updated recommended allocation.

  3. Click Save.

Step 11: (Optional) Add Custom VM Extensions

Use the Ops Manager API to add custom properties to your VMs such as associated security groups and load balancers.

For more information, see Managing Custom VM Extensions.

Step 12: Deploy BOSH

Follow the steps below to deploy BOSH:

  1. Go to the Ops Manager Installation Dashboard.

    Config bosh 20

  2. Click Review Pending Changes.

    Config bosh 21

  3. Click Apply Changes.

    Config bosh 22

  4. Confirm changes applied successfully.

    Config bosh 23

  5. Check BOSH VM. Log in to vCenter and check for the p-bosh VM deployment in the Enterprise PKS Management resource pool.

    Config bosh 24

Step 12: Update Network Availability Zones

Ater BOSH is successfully deployed, update the network you defined above (NET-MGMT-PKS) to include each of the COMPUTE AZs that you defined. This ensures that both the Management AZ and the Compute AZs appear in the Enterprise PKS tile for the Plans.

  1. Return to the BOSH tile and click Create Networks.

    Config bosh 26

  2. Edit the network (NET-MGMT-PKS) and each COMPUTE AZ.

    Config bosh 27

  3. Click Save.

    Config bosh 28

  4. Review pending changes, and click Apply Changes to redeploy BOSH.

Next Step

Generate and Register the NSX Manager Superuser Principal Identity Certificate and Key for Enterprise PKS.

Please send any feedback you have to