Enterprise PKS Release Notes

Page last updated:

This topic contains release notes for Enterprise Pivotal Container Service (Enterprise PKS) v1.4.x.


v1.4.2

Release Date: August 19, 2019

Features

  • [Security Fix] This patch addresses CVE-2019-3794, the UAA client.write scope vulnerability.
  • [Bug Fix] Updated NCP timeout parameters to ensure that NCP does not overwhelm NSX-T in a high scale environment.

Product Snapshot

Element Details
Version v1.4.2
Release date June August 19, 2019
Compatible Ops Manager versions v2.4.2+, v2.5.x, v2.6.x
Compatible Ops Manager versions (Azure only) v2.4.2, v2.4.3, v2.4.13, v2.5.5 and later, v2.6.x
Xenial stemcell version v250.73
Kubernetes version v1.13.5
On-Demand Broker version v0.26.0
NSX-T versions v2.3.1, v2.4.0.1, v2.4.1, v2.4.2 (see below)
NCP version v2.4.1
Docker version v18.06.3-ce
docker-boshrelease
Backup and Restore SDK version v1.8.0
UAA version v71.2

Warning: NSX-T v2.4 implements a new password expiration policy. By default, the administrator password expires after 90 days. If the password expires, you cannot deploy new Kubernetes clusters using PKS with NSX-T. For more information, see the following VMware KB article and Updating the NSX Manager Password for NSX-T v2.4 in the Enterprise PKS documentation.

Note: NSX-T v2.4 implements a new user interface (UI) based on the NSX Policy API. PKS v1.4.2 does not support the NSX Policy API. Any objects created through the new UI cannot be used with PKS v1.4.2. If you are installing PKS v1.4.2 with NSX-T v2.4.x or upgrading to PKS v1.4.2 and NSX-T 2.4.x, you must use the Advanced Networking tab in NSX Manager to create, read, update, and delete all networking objects required for Enterprise PKS. For more information, see NSX-T v2.4 Management Interfaces in the Enterprise PKS documentation.

vSphere Version Requirements

If you are installing Enterprise PKS on vSphere or on vSphere with NSX-T Data Center, see the VMware Product Interoperability Matrices for compatibility information.

Feature Support by IaaS

AWS Azure GCP vSphere vSphere with NSX-T
Automatic Kubernetes Cluster API load balancer
HTTP proxy
Multi-AZ storage
Per-namespace subnets
Service type:LoadBalancer *

* For more information about configuring Service type:LoadBalancer on AWS, see the Access Workloads Using an Internal AWS Load Balancer section of Deploying and Exposing Basic Workloads.

Upgrade Path

The supported upgrade paths to PKS v1.4.2 are from PKS v1.4.0 and PKS v1.3.2 and later.

WARNING: Before you upgrade to PKS 1.4.x, you must increase the size of persistent disk for the PKS VM. For more information, see Upgrade Fails Due to Insufficient Disk Space in the Known Issues.

To upgrade, see Upgrading Enterprise PKS and Upgrading Enterprise PKS with NSX-T.

Breaking change: Upgrading to Ops Manager v2.6 on vSphere requires that you set a public SSH key in the OVF template prior to the upgrade. For more information, see Passwords Not Supported for Ops Manager VM on vSphere in the Breaking Changes section.

Known Issues

Enterprise PKS v1.4.2 has the following known issues:

Support for NSX-T v2.4.2 with Known Issue and Workaround

Enterprise PKS v1.4.2 supports NSX-T v2.4.2. However, there is a known issue with NSX-T v2.4.2 that can affect new and upgraded installations of Enterprise PKS v1.4.2 that use a NAT topology.

For NSX-T v2.4.2, the PKS Management Plane must be deployed on a Tier-1 distributed router (DR). If the PKS Management Plane is deployed on a Tier-1 service router (SR), the router needs to be converted. To convert an SR to a DR, refer to the East-West traffic between workloads behind different T1 is impacted, when NAT is configured on T0 (71363) KB article.

This issue will be addressed in a subsequent release of NSX-T such that it will not matter if the Tier-1 Router is a DR or an SR.

One Plan ID Is Longer Than Other Plan IDs

Symptom

One of your Plan IDs is one character longer than your other Plan IDs.

Explanation

Each Plan has a unique Plan ID. A Plan ID is normally a UUID consisting of 32 alphanumeric characters and 4 hyphens. The Plan 4 Plan ID is instead a UUID consisting of 33 alphanumeric characters and 4 hyphens.

You can safely configure and use Plan 4. The length of the Plan 4 Plan ID does not affect the functionality of Plan 4 clusters.

If you require all Plan IDs to have identical length, do not activate or use Plan 4.


v1.4.1

Release Date: June 20, 2019

Features

New features and changes in this release:

Product Snapshot

Element Details
Version v1.4.1
Release date June 20, 2019
Compatible Ops Manager versions v2.4.2+, v2.5.x, v2.6.x
Compatible Ops Manager versions (Azure only) v2.4.2, v2.4.3, v2.4.13, v2.5.5 and later, v2.6.x
Xenial stemcell version v250.25
Note: To address the Zombieload CVE, upload stemcell v250.48.
For more information, see the Mitigating Zombieload for Pivotal Container Service (PKS) KB article.
Kubernetes version v1.13.5
On-Demand Broker version v0.26.0
NSX-T versions v2.3.1, v2.4.0.1, v2.4.1 (recommended), v2.4.2 (see below)
NCP version v2.4.1
Docker version v18.06.3-ce
docker-boshrelease
Backup and Restore SDK version v1.8.0
UAA version v71.0

Warning: NSX-T v2.4 implements a new password expiration policy. By default, the administrator password expires after 90 days. If the password expires, you cannot deploy new Kubernetes clusters using PKS with NSX-T. For more information, see the following VMware KB article and Updating the NSX Manager Password for NSX-T v2.4 in the Enterprise PKS documentation.

Note: NSX-T v2.4 implements a new user interface (UI) based on the NSX Policy API. PKS v1.4.1 does not support the NSX Policy API. Any objects created through the new UI cannot be used with PKS v1.4.1. If you are installing PKS v1.4.1 with NSX-T v2.4.x or upgrading to PKS v1.4.1 and NSX-T 2.4.x, you must use the Advanced Networking tab in NSX Manager to create, read, update, and delete all networking objects required for Enterprise PKS. For more information, see NSX-T v2.4 Management Interfaces in the Enterprise PKS documentation.

vSphere Version Requirements

If you are installing Enterprise PKS on vSphere or on vSphere with NSX-T Data Center, see the VMware Product Interoperability Matrices for compatibility information.

Feature Support by IaaS

AWS Azure GCP vSphere vSphere with NSX-T
Automatic Kubernetes Cluster API load balancer
HTTP proxy
Multi-AZ storage
Per-namespace subnets
Service type:LoadBalancer *

* For more information about configuring Service type:LoadBalancer on AWS, see the Access Workloads Using an Internal AWS Load Balancer section of Deploying and Exposing Basic Workloads.

Upgrade Path

The supported upgrade paths to PKS v1.4.1 are from PKS v1.4.0 and PKS v1.3.2 and later.

WARNING: Before you upgrade to PKS 1.4.x, you must increase the size of persistent disk for the PKS VM. For more information, see Upgrade Fails Due to Insufficient Disk Space in the Known Issues.

To upgrade, see Upgrading Enterprise PKS and Upgrading Enterprise PKS with NSX-T.

Breaking change: Upgrading to Ops Manager v2.6 on vSphere requires that you set a public SSH key in the OVF template prior to the upgrade. For more information, see Passwords Not Supported for Ops Manager VM on vSphere in the Breaking Changes section.

Breaking Changes

Passwords Not Supported for Ops Manager VM on vSphere

Enterprise PKS v1.4.1 supports Ops Manager v2.6. Starting in Ops Manager v2.6, you can only SSH onto the Ops Manager VM in a vSphere deployment with a private SSH key. You cannot SSH onto the Ops Manager VM with a password.

To avoid upgrade failure and errors when authenticating, add a public key to the Customize Template screen of the the OVF template for the Ops Manager VM. Then, use the private key to SSH onto the Ops Manager VM.

Warning: You cannot upgrade to Ops Manager v2.6 successfully without adding a public key. If you do not add a key, Ops Manager shuts down automatically because it cannot find a key and may enter a reboot loop.

For more information about adding a public key to the OVF template, see Deploy Ops Manager in Deploying Ops Manager on vSphere.

For instructions to generate an SSH key pair, see one of the following: * Generate an SSH key pair for installing Ops Manager v2.6 on vSphere in the Pivotal Support Knowledge Base * How to generate an SSH key pair for installing Ops Manager v2.6 on vSphere in the VMware Knowledge Base.

Log Sink Entry Format Change

Log sink entries in PKS v1.4 are tagged with the human-readable cluster name instead of the host ID of the BOSH-defined VM.

Known Issues

Enterprise PKS v1.4.1 has the following known issues:

Support for NSX-T v2.4.2 with Known Issue and Workaround

Enterprise PKS v1.4.1 supports NSX-T v2.4.2. However, there is a known issue with NSX-T v2.4.2 that can affect new and upgraded installations of Enterprise PKS v1.4.1 that use a NAT topology.

For NSX-T v2.4.2, the PKS Management Plane must be deployed on a Tier-1 distributed router (DR). If the PKS Management Plane is deployed on a Tier-1 service router (SR), the router needs to be converted. To convert an SR to a DR, refer to the East-West traffic between workloads behind different T1 is impacted, when NAT is configured on T0 (71363) KB article.

This issue will be addressed in a subsequent release of NSX-T such that it will not matter if the Tier-1 Router is a DR or an SR.

PKS-provisioned Kubernetes cluster creation fails in a vSphere with NSX-T environment

Symptom

Kubernetes cluster creation fails with Enterprise PKS installed on vSphere with NSX-T.

Explanation

When configuring a plan for Kubernetes clusters, for the Worker VM Type setting, selecting either of the following Worker VM Types results in cluster creation failure because there is not enough disk to handle swap space.

  • medium (cpu: 2, ram: 4 GM, disk: 8 GB)
  • medium.mem (cpu: 1, ram: 8 GM, disk: 8 GB)

BOSH will assign the same amount of RAM to swap up to 50% of the ephemeral disk size. For more information, see How much Swap Space is Allocated for BOSH-Deployed VMs in the Pivotal Support Knowledge Base. Given the swap space requirements, the medium and medium.mem Worker VM types are only allocated 4 GB usable ephemeral disk, which is not enough for successful cluster deployment of the PKS tile with NCP.

Workaround

When selecting the Worker VM Type for Worker nodes, do not use the medium or medium.mem type in a vSphere with NSX-T environment. Use the default Worker VM Type medium.disk (cpu: 2, ram: 4 GB, disk: 32 GB) or any other Worker VM Type except medium or medium.mem.

Upgrade All Service Instances Errand Fails

Symptom

After clicking Apply Changes in Ops Manager, the upgrade-all-service-instances errand fails.

Explanation

The upgrade-all-service-instances errand fails if one or more of your existing clusters are in a FAILED state.

To check cluster status, run pks clusters.

Workaround

If you experience this issue, delete and re-create the failed cluster. Follow the procedure in Cannot Re-Create a Cluster that Failed to Deploy.

Upgrade Fails Due to Insufficient Disk Space

Symptom

The upgrade from PKS v1.3.x to Enterprise PKS v1.4.x fails with the following error in the BOSH debug log:

1 of 8 pre-start scripts failed. Failed Jobs:pxc-mysql

The pxc-mysql pre-start logs also include the following error message:

panic: Cannot continue, insufficient disk space to complete migration

Explanation

The upgrade to Enterprise PKS v1.4.x includes a MySQL migration from MaridDB to Percona.

When /var/vcap/store does not have enough space for the backup of existing MySQL files that happens during this migration, the Enterprise PKS upgrade fails.

Workaround

Before upgrading to Enterprise PKS v1.4.x, increase the persistent disk type configured for the PKS VM.

  1. In Ops Manager, click the Pivotal Container Service tile.
  2. Select the Resource Config pane.
  3. In Pivotal Container Service > Persistent Disk Type, select a larger disk type than the currently configured disk type. For example, if 10 GB is currently configured, select 20 GB.

If you experience this issue after attempting an upgrade, perform the steps above and start the upgrade again.

Kubectl CLI Commands Do Not Work after Changing an Existing Plan to a Different AZ

Symptom

After you reconfigure the AZ of an existing plan, kubectl cli commands do not work in the plan’s existing clusters.

Explanation

This issue occurs in IaaS environments which either limit or prevent attaching a disk across multiple AZs.

BOSH supports creating new VMs and attaching existing disks to VMs. BOSH cannot “move” VMs.

If the plan for an existing cluster is changed to a different AZ, the cluster’s new “intended” state is for the cluster to be hosted within the new AZ. To migrate the cluster from its original state to its intended state, BOSH will create new VMs for the cluster within the designated AZ and remove the cluster’s original VMs from the original AZ.

On an IaaS where attaching VM disks across AZs is not supported, the disks attached to the newly created VMs will not have the original disks’ content.

Workaround

If you have reconfigured the AZ of an existing cluster and afterwards could not run kubectl cli commands, contact Support for assistance.

One Plan ID Is Longer Than Other Plan IDs

Symptom

One of your Plan IDs is one character longer than your other Plan IDs.

Explanation

Each Plan has a unique Plan ID. A Plan ID is normally a UUID consisting of 32 alphanumeric characters and 4 hyphens. The Plan 4 Plan ID is instead a UUID consisting of 33 alphanumeric characters and 4 hyphens.

You can safely configure and use Plan 4. The length of the Plan 4 Plan ID does not affect the functionality of Plan 4 clusters.

If you require all Plan IDs to have identical length, do not activate or use Plan 4.


v1.4.0

Release Date: April 25, 2019

Features

New features and changes in this release:

  • Operators can configure up to ten sets of resource types, or plans, in the Enterprise PKS tile. All plans except the first can made available or unavailable to developers deploying clusters. Plan 1 must be configured and made available as a default for developers. For more information, see the Plans section of the installation topic for your IaaS, such as Installing Enterprise PKS on vSphere with NSX-T.

  • Operators can deploy up to 5 master nodes per plan. For more information, see the Plans section of the installation topic for your IaaS, such as Installing Enterprise PKS on vSphere with NSX-T.

  • Operators can install PKS and Pivotal Application Service (PAS) on the same instance of Ops Manager.

  • Improved workflow for managing cluster access. For more information, see Grant Cluster Access in Managing Users in Enterprise PKS with UAA.

  • Operators can create webhook ClusterSink resources. A webhook ClusterSink resource batches logs into 1 second units, wraps the resulting payload in JSON, and uses the POST method to deliver the logs to the address of your log management service. For more information see, Create a Webhook ClusterSink Resource with YAML and kubectl in Creating Sinks.

  • Operators can set quotas for maximum memory and CPU utilization in a PKS deployment. For more information, see Managing Resource Usage. This is a beta feature.

    Warning: This feature is a beta component and is intended for evaluation and test purposes only. Do not use this feature in a production environment. Product support and future availability are not guaranteed for beta components.

  • Operators can enable the PodSecurityPolicy admission plugin on a per-plan basis requiring cluster users to have policy, role, and role binding permissions to deploy pod workloads. See Pod Security Policies for more information.

  • Operators can enable the SecurityContextDeny admission plugin on a per-plan basis to prohibit the use of security context configurations on pods and containers. See Security Context Deny for more information.

  • Operators can enable the DenyEscalatingExec admission plugin on a per-plan basis to prohibit the use of certain commands for containers that allow host access. See Deny Escalating Execution for more information.

  • Operators using vSphere can use HostGroups to define Availability Zones (AZs) for clusters in BOSH. See Using vSphere Host Groups.

  • Operators using vSphere can configure compute profiles to specify which vSphere resources are used when deploying Kubernetes clusters in a PKS deployment. For more information, see Using Compute Profiles (vSphere Only).

    Warning: This feature is a beta component and is intended for evaluation and test purposes only. Do not use this feature in a production environment. Product support and future availability are not guaranteed for beta components.

  • Operators using vSphere with NSX-T can update a Network Profile and add to or reorder the Pods IP Block IDs. For more information, see the Change the Network Profile for a Cluster section of Using Network Profiles (NSX-T Only).

Product Snapshot

Element Details
Version v1.4.0
Release date April 25, 2019
Compatible Ops Manager versions v2.4.2+, v2.5.x
Compatible Ops Manager versions (Azure only) v2.4.2, v2.4.3, v2.4.13, v2.5.5 and later
Xenial stemcell version v250.25
Kubernetes version v1.13.5
On-Demand Broker version v0.26.0
NSX-T versions * v2.3.1, v2.4.0.1
NCP version v2.4.0
Docker version v18.06.3-ce
CFCR
Backup and Restore SDK version v1.8.0
UAA version v71.0

WARNING: NSX-T v2.4 implements a new password expiration policy. By default the administrator password expires after 90 days. If the password expires, you cannot deploy new Kubernetes clusters using PKS with NSX-T. For more information, refer to the following VMware KB article: https://kb.vmware.com/s/article/70691. See also Updating the NSX Manager Password for NSX-T v2.4 in the Enterprise PKS documentation.

Note: NSX-T v2.4 implements a new user interface (UI) based on the NSX Policy API. PKS v1.4.0 does not support the NSX Policy API. Any objects created via the new Policy-based UI cannot be used with PKS v1.4.0. If you are installing PKS v1.4.0 with NSX-T v2.4.x, or upgrading to PKS v1.4.0 and NSX-T 2.4.x, you must use the “Advanced Networking” tab in NSX Manager to create, read, update, and delete all networking objects required for PKS. See also NSX-T v2.4 Management Interfaces in the Enterprise PKS documentation.

vSphere Version Requirements

For Enterprise PKS installations on vSphere or on vSphere with NSX-T Data Center, refer to the VMware Product Interoperability Matrices.

Feature Support by IaaS

AWS Azure GCP vSphere vSphere with NSX-T
Automatic Kubernetes Cluster API load balancer
HTTP proxy
Multi-AZ storage
Per-namespace subnets
Service type:LoadBalancer *

* For more information about configuring Service type:LoadBalancer on AWS, see the Access Workloads Using an Internal AWS Load Balancer section of Deploying and Exposing Basic Workloads.

Upgrade Path

The supported upgrade paths to PKS v1.4.0 are from PKS v1.3.2 and later.

WARNING: Before you upgrade to PKS 1.4.x, you must increase the size of persistent disk for the PKS VM. For more information, see Upgrade Fails Due to Insufficient Disk Space in the Known Issues.

To upgrade, see Upgrading Enterprise PKS and Upgrading Enterprise PKS with NSX-T.

When upgrading to NSX-T 2.4:

  • Use the official VMware NSX-T Data Center 2.4 build.
  • Apply the NSX-T v2.4.0.1 hot-patch. For more information, see KB article 67499 in the VMware Knowledge Base.
  • To obtain the NSX-T v2.4.0.1 hot-patch, open a support ticket with VMware Global Support Services (GSS) for NSX-T Engineering.

Breaking Changes

Log sink entries in PKS v1.4 are tagged with the human-readable cluster name instead of the host ID of the BOSH-defined VM.

Known Issues

Enterprise PKS v1.4.0 has the following known issues:

Azure Default Security Group Is Not Automatically Assigned to Cluster VMs

Symptom

You experience issues when configuring a load balancer for a multi-master Kubernetes cluster or creating a service of type LoadBalancer. Additionally, in the Azure portal, the VM > Networking page does not display any inbound and outbound traffic rules for your cluster VMs.

Explanation

As part of configuring the Enterprise PKS tile for Azure, you enter Default Security Group in the Kubernetes Cloud Provider pane. When you create a Kubernetes cluster, Enterprise PKS automatically assigns this security group to each VM in the cluster. However, in Enterprise PKS v1.4, the automatic assignment may not occur.

As a result, your inbound and outbound traffic rules defined in the security group are not applied to the cluster VMs.

Workaround

If you experience this issue, manually assign the default security group to each VM NIC in your cluster.

Cluster Creation Fails When First AZ Runs out of Resources

Symptom

If the first availability zone (AZ) used by a plan with multiple AZs runs out of resources, cluster creation fails with an error like the following:

L Error: CPI error 'Bosh::Clouds::CloudError' with message 'No valid placement found for requested memory: 4096

Explanation

BOSH creates VMs for your Enterprise PKS deployment using a round-robin algorithm, creating the first VM in the first AZ that your plan uses. If the AZ runs out of resources, cluster creation fails because BOSH cannot create the cluster VM.

For example, if your three AZs each have enough resources for ten VMs, and you create two clusters with four worker VMs each, BOSH creates VMs in the following AZs:

AZ1 AZ2 AZ3
Cluster 1 Worker VM 1 Worker VM 2 Worker VM 3
Worker VM 4
Cluster 2 Worker VM 1 Worker VM 2 Worker VM 3
Worker VM 4

In this scenario, AZ1 has twice as many VMs as AZ2 or AZ3.

Azure Worker Node Communication Fails after Upgrade

Symptom

Outbound communication from a worker node VM fails after an upgrade to Enterprise PKS v1.4.0.

Explanation

Enterprise PKS 1.4.0 uses Azure Availability Sets to improve the uptime of workloads and worker nodes in the event of Azure platform failures. Worker node VMs are distributed evenly across Availability Sets.

Azure Standard SKU Load Balancers are recommended for the Kubernetes control plane and Kubernetes ingress and egress. This load balancer type provides an IP address for outbound communication using SNAT.

During an upgrade, when BOSH rebuilds a given worker instance in an Availability Set, Azure can time out while re-attaching the worker node network interface to the back-end pool of the Standard SKU Load Balancer.

For more information, see Outbound connections in Azure in the Azure documentation.

Workaround

You can manually re-attach the worker instance to the back-end pool of the Azure Standard SKU Load Balancer in your Azure console.

Upgrade Fails Due to Insufficient Disk Space

Symptom

The upgrade from PKS v1.3.x to Enterprise PKS v1.4.x fails with the following error in the BOSH debug log:

1 of 8 pre-start scripts failed. Failed Jobs:pxc-mysql

The pxc-mysql pre-start logs also include the following error message:

panic: Cannot continue, insufficient disk space to complete migration

Explanation

The upgrade to Enterprise PKS v1.4.x includes a MySQL migration from MaridDB to Percona.

When /var/vcap/store does not have enough space for the backup of existing MySQL files that happens during this migration, the Enterprise PKS upgrade fails.

Workaround

Before upgrading to Enterprise PKS v1.4.x, increase the persistent disk type configured for the PKS VM.

  1. In Ops Manager, click the Pivotal Container Service tile.
  2. Select the Resource Config pane.
  3. In Pivotal Container Service > Persistent Disk Type, select a larger disk type than the currently configured disk type. For example, if 10 GB is currently configured, select 20 GB.

If you experience this issue after attempting an upgrade, perform the steps above and start the upgrade again.

Kubectl CLI Commands Do Not Work after Changing an Existing Plan to a Different AZ

Symptom

After you reconfigure the AZ of an existing plan, kubectl cli commands do not work in the plan’s existing clusters.

Explanation

This issue occurs in IaaS environments which either limit or prevent attaching a disk across multiple AZs.

BOSH supports creating new VMs and attaching existing disks to VMs. BOSH cannot “move” VMs.

If the plan for an existing cluster is changed to a different AZ, the cluster’s new “intended” state is for the cluster to be hosted within the new AZ. To migrate the cluster from its original state to its intended state, BOSH will create new VMs for the cluster within the designated AZ and remove the cluster’s original VMs from the original AZ.

On an IaaS where attaching VM disks across AZs is not supported, the disks attached to the newly created VMs will not have the original disks’ content.

Workaround

If you have reconfigured the AZ of an existing cluster and afterwards could not run kubectl cli commands, contact Support for assistance.

One Plan ID Is Longer Than Other Plan IDs

Symptom

One of your Plan IDs is one character longer than your other Plan IDs.

Explanation

Each Plan has a unique Plan ID. A Plan ID is normally a UUID consisting of 32 alphanumeric characters and 4 hyphens. The Plan 4 Plan ID is instead a UUID consisting of 33 alphanumeric characters and 4 hyphens.

You can safely configure and use Plan 4. The length of the Plan 4 Plan ID does not affect the functionality of Plan 4 clusters.

If you require all Plan IDs to have identical length, do not activate or use Plan 4.


Please send any feedback you have to pks-feedback@pivotal.io.