LATEST VERSION: v1.1 - RELEASE NOTES
Pivotal Container Service v1.0

PKS Release Notes

PKS (Pivotal Container Service) is used to create and manage on-demand Kubernetes clusters via the PKS CLI.

v1.0.4

Release Date: May 21, 2018

Upgrade Procedure

Note: Upgrade to PKS v1.0.4 from either PKS v1.0.2 or PKS v1.0.3. Do not upgrade PKS v1.0.0 directly to v1.0.4. Instead, upgrade to v1.0.2, then v1.0.4. Alternatively, do a unique install of PKS v1.0.4.

To upgrade to PKS v1.0.4, follow the procedures in Upgrade PKS.

Features

  • Updates Kubernetes to v1.9.7.

Component Versions

PKS v1.0.4 includes or supports the following component versions:

Product Component Version Supported Notes
Pivotal Cloud Foundry Operations Manager (Ops Manager) 2.0.X and 2.1.X Separate download available from Pivotal Network
vSphere 6.5, 6.5 U1, and 6.5 U2 - Editions
  • vSphere Enterprise Plus Edition
  • vSphere with Operations Management Enterprise Plus
vSphere versions supported for Pivotal Container Service (PKS)
VMware Harbor Registry 1.4.2 Separate download available from Pivotal Network
NSX-T 2.1 Advanced Edition Available from VMware
Stemcell 3468.X Floating stemcell line available to download from Pivotal Network
Kubernetes 1.9.7* Packaged in the PKS Tile (CFCR)
CFCR (Kubo) 0.13 Packaged in the PKS Tile
Golang 1.9.5 Packaged in the PKS Tile
NCP 2.1.3 Packaged in the PKS Tile
Kubernetes CLI 1.9.7* Separate download available from the PKS section of Pivotal Network
PKS CLI 1.0.3-build.15 Separate download available from the PKS section of Pivotal Network
UAA 55
* Components marked with an asterisk have been patched to resolve security vulnerabilities or fix component behavior.

Known Issues

This section includes known issues with PKS v1.0.4 and corresponding workarounds.

Access to the Kubernetes API is Unavailable During Upgrades

PKS upgrades include upgrades to the master node. While the master node is undergoing an upgrade, the Kubernetes API is unavailable.

If you attempt to access the API during an upgrade, you will not be able to connect.

Stemcell Updates Cause Automatic VM Upgrading

Enabling the Upgrade all clusters errand allows automatic upgrading for VMs in your deployment. Pivotal recommends enabling this errand to ensure that all deployed cluster VMs are patched.

When you enable the Upgrade all clusters errand, the following actions can cause downtime:

  • Updating the PKS tile with a new stemcell triggers updating each VM in each cluster.
  • Updating other tiles in your deployment with new stemcells causes the upgrading of the PKS tile.

Upgrade Errand Fails with Failed Deployments

The Upgrade all clusters errand fails if any deployments are in a failed state.

To work around this issue, delete the failed cluster using the PKS CLI or redeploy the failed cluster with the BOSH CLI to ensure the cluster is in a successful state.

Pods Lose Network Connectivity After VM Cold Migration

When a Kubernetes cluster worker VM goes through cold migration in vSphere, newly provisioned pods lose network connectivity.

This issue can occur under the following conditions:

  • When the VM is powered off and is subject to cold migration, and the VM moves to a different ESXi host
  • When the VM is powering on and is subject to Distributed Resource Scheduler (DRS) before the powerup completes
  • When the vNIC of the VM is detached and reattached

To work around this issue, delete the worker VM. BOSH recreates the worker VM and restores network connectivity.


Kubernetes Cluster Creation Fails if NSX-T Manager Password Begins with Certain Special Characters

If you select NSX-T as a Container Network Type in PKS and your NSX-T Manager password begins with an @, $, ^, ', or space character, Kubernetes cluster creation fails. To resolve this issue, reset your NSX-T Manager password so that it does not begin with any of these characters. After resetting your NSX-T Manager password, reconfigure your NSX-T Manager credentials in the PKS tile with the updated password.

v1.0.3

Release Date: May 4, 2018

Upgrade Procedure

Note: The only supported upgrade path for PKS v1.0.3 is from PKS v1.0.2. Do not upgrade PKS v1.0.0 directly to v1.0.3. Instead, upgrade to v1.0.2, then v1.0.3. Alternatively, do a unique install of PKS v1.0.3.

To upgrade to PKS v1.0.3, perform the following steps:

  1. Download the latest 3468.x stemcell from Pivotal Network and configure the PKS tile with the stemcell.
  2. Create a new worker node service account.
  3. Follow the procedures in Upgrade PKS. When configuring the Kubernetes Cloud Provider configuration screen in the PKS tile, configure the new worker node credentials or service account key as appropriate for your IaaS.

Features

  • Separates the master and worker node credentials.
  • Updates Kubernetes to v1.9.6.
  • Updates Golang to v1.9.5.

Component Versions

PKS v1.0.3 includes or supports the following component versions:

Product Component Version Supported Notes
Pivotal Cloud Foundry Operations Manager (Ops Manager) 2.0.X and 2.1.X Separate download available from Pivotal Network
vSphere 6.5 and 6.5 U1 - Editions
  • vSphere Enterprise Plus Edition
  • vSphere with Operations Management Enterprise Plus
vSphere versions supported for Pivotal Container Service (PKS)
VMware Harbor Registry 1.4.1 Separate download available from Pivotal Network
NSX-T 2.1 Advanced Edition Available from VMware
Stemcell 3468.X Floating stemcell line available to download from Pivotal Network
Kubernetes 1.9.6* Packaged in the PKS Tile (CFCR)
CFCR (Kubo) 0.13 Packaged in the PKS Tile
Golang 1.9.5* Packaged in the PKS Tile
NCP 2.1.3* Packaged in the PKS Tile
Kubernetes CLI 1.9.6* Separate download available from the PKS section of Pivotal Network
PKS CLI 1.0.3-build.15* Separate download available from the PKS section of Pivotal Network
UAA 55*
* Components marked with an asterisk have been patched to resolve security vulnerabilities or fix component behavior.

Known Issues

This section includes known issues with PKS v1.0.3 and corresponding workarounds.

Access to the Kubernetes API is Unavailable During Upgrades

PKS upgrades include upgrades to the master node. While the master node is undergoing an upgrade, the Kubernetes API is unavailable.

If you attempt to access the API during an upgrade, you will not be able to connect.

Stemcell Updates Cause Automatic VM Upgrading

Enabling the Upgrade all clusters errand allows automatic upgrading for VMs in your deployment. Pivotal recommends enabling this errand to ensure that all deployed cluster VMs are patched.

When you enable the Upgrade all clusters errand, the following actions can cause downtime:

  • Updating the PKS tile with a new stemcell triggers updating each VM in each cluster.
  • Updating other tiles in your deployment with new stemcells causes the upgrading of the PKS tile.

Upgrade Errand Fails with Failed Deployments

The Upgrade all clusters errand fails if any deployments are in a failed state.

To work around this issue, delete the failed cluster using the PKS CLI or redeploy the failed cluster with the BOSH CLI to ensure the cluster is in a successful state.

Pods Lose Network Connectivity After VM Cold Migration

When a Kubernetes cluster worker VM goes through cold migration in vSphere, newly provisioned pods lose network connectivity.

This issue can occur under the following conditions:

  • When the VM is powered off and is subject to cold migration, and the VM moves to a different ESXi host
  • When the VM is powering on and is subject to Distributed Resource Scheduler (DRS) before the powerup completes
  • When the vNIC of the VM is detached and reattached

To work around this issue, delete the worker VM. BOSH recreates the worker VM and restores network connectivity.


StatefulSets Pod Failure After Recreating a VM

When using vSphere with NSX-T integration, if you recreate a node that hosts a StatefulSets pod, the pod can get stuck in a ContainerCreating state. The pod emits a warning event with a FailedCreatePodSandBox reason. This issue affects StatefulSets pods created before PKS v1.0.3.

A fix for this bug is included in PKS v1.0.3, but the fix applies only to StatefulSets created using PKS v1.0.2 or later. After upgrading PKS to v1.0.3, manually deleting and recreating all preexisting StatefulSets pods is recommended, even if they are in a running state.

To get all StatefulSets pods, run the following command on every Kubernetes cluster using the Kubernetes admin user permissions:

$ kubectl get pods -l "statefulset.kubernetes.io/pod-name" \
-o wide --all-namespaces

For each result, delete the pod by running the following command:

$ kubectl delete pod POD-NAME -n POD-NAMESPACE

You do not need to manually recreate the deleted pods. Kubernetes detects a StatefulSet with missing pods and automatically recreates the pods.

[Kubernetes Bug] Upgrading a Cluster Affects Persistent Workload Uptime

During an upgrade to v1.0.3 on vSphere, persistent storage volumes do not reattach to pods until all worker nodes have been upgraded, which results in workload downtime until the entire cluster is upgraded.

This issue occurs when you deploy a pod with persistent storage attached, drain the node, and then immediately delete the node VM.

The expected behavior is for persistent disks to reattach to the upgraded VMs after the pod is restored. However, a Kubernetes bug prevents the disk from reattaching. PKS v1.0.3 works around this bug by attaching the volumes after all workers are upgraded.

For more information, see the Kubernetes issue on GitHub.

In rare cases, pods with persistent volumes can stay in ContainerCreating state. If you see the error FailedMount Unable to mount volumes for pod POD-NAME, perform the following steps:

  1. Find the problem node by running kubectl describe pod POD-NAME.
  2. Prevent scheduling on the node that runs the pod by running kubectl cordon NODE-NAME.
  3. Delete pod by running kubectl delete pod POD-NAME.
  4. Wait for pod to be rescheduled and enter Running state. This may take several minutes.
  5. Resume scheduling on the node that runs the pod by running kubectl uncordon NODE-NAME.

Kubernetes Cluster Creation Fails if NSX-T Manager Password Begins with Certain Special Characters

If you select NSX-T as a Container Network Type in PKS and your NSX-T Manager password begins with an @, $, ^, ', or space character, Kubernetes cluster creation fails. To resolve this issue, reset your NSX-T Manager password so that it does not begin with any of these characters. After resetting your NSX-T Manager password, reconfigure your NSX-T Manager credentials in the PKS tile with the updated password.

v1.0.2

Release Date: April 12, 2018

Upgrade Procedure

To upgrade to PKS v1.0.2, perform the following steps:

  1. Download the docker_ctl script.
  2. Download the docker_ctl_update.sh script.
  3. Log in to the BOSH Director by running bosh -e MY-ENVIRONMENT log-in from a VM that can access your PKS deployment. Replace MY-ENVIRONMENT with the BOSH alias for your PKS environment. See Manage PKS Deployments with BOSH for more information.

    If you choose to log in from the Ops Manager VM, perform the following steps:
    1. Run sudo apt-get update.
    2. Run sudo apt-get install jq.
  4. Run export BOSH_ENVIRONMENT=MY-ENVIRONMENT. Replace MY-ENVIRONMENT with the BOSH alias for your PKS environment.
  5. Run the docker_ctl_update.sh script. This script contains the fix to correctly unmount Docker overlays. See the corresponding known issue for more information.
  6. Download the latest 3468.x stemcell from Pivotal Network and configure the PKS tile with the stemcell.
  7. Follow the procedures in Upgrade PKS.

Features

  • Updates Kubernetes to v1.9.5.
  • Updates Golang to v1.9.4.

Fixed Issues

General

  • Worker nodes are now drained before they stop in order to minimize workload downtime during a rolling upgrade.
  • UAA credentials and vCenter passwords no longer appear in BOSH logs.
  • BOSH DNS no longer causes worker nodes to fail after a manual restart.
  • The Kubernetes Controller Manager certificate no longer contains additional whitespace.
  • Drain user now has additional permissions to remove replication controller-owned pods.
  • Unmounting Docker overlay volumes no longer causes BOSH unmount failures.
  • Addresses upgrade issues in constrained environments.

vSphere

  • vSphere NSX-T integration now works with BOSH stemcell v3468.25 and later.
  • For vSphere with NSX-T, the pod logical switch port (LSP) is now updated when you recreate the VM that hosts the pod. See StatefulSets in the Kubernetes documentation and the known issue below for more information.
  • Added support for special characters #, &, ;, ", ', ^, \, space (), %, and ! in vCenter passwords in the Kubernetes Cloud Provider tile configuration page.
  • Drain script now deletes nodes to fix a vSphere issue where node names changed between 1.9.2 and 1.9.5.

Component Versions

PKS v1.0.2 includes or supports the following component versions:

Product Component Version Supported Notes
Pivotal Cloud Foundry Operations Manager (Ops Manager) 2.0.X and 2.1.X Separate download available from Pivotal Network
vSphere 6.5 and 6.5 U1 - Editions
  • vSphere Enterprise Plus Edition
  • vSphere with Operations Management Enterprise Plus
vSphere versions supported for Pivotal Container Service (PKS)
VMware Harbor Registry 1.4.1 Separate download available from Pivotal Network
NSX-T 2.1 Advanced Edition Available from VMware
Stemcell 3468.X* Floating stemcell line available to download from Pivotal Network
Kubernetes 1.9.5* Packaged in the PKS Tile (CFCR)
CFCR (Kubo) 0.13 Packaged in the PKS Tile
Golang 1.9.4* Packaged in the PKS Tile
NCP 2.1.2* Packaged in the PKS Tile
Kubernetes CLI 1.9.5* Separate download available from the PKS section of Pivotal Network
PKS CLI 1.0.2-build.4* Separate download available from the PKS section of Pivotal Network
* Components marked with an asterisk have been patched to resolve security vulnerabilities or fix component behavior.

Known Issues

This section includes known issues with PKS v1.0.2 and corresponding workarounds.

Access to the Kubernetes API is Unavailable During Upgrades

PKS upgrades include upgrades to the master node. While the master node is undergoing an upgrade, the Kubernetes API is unavailable.

If you attempt to access the API during an upgrade, you will not be able to connect.

Volume Unmount Failure After Stemcell Upgrade

During an upgrade to PKS v1.0.2, BOSH can fail to unmount the /var/vcap/store volume on worker nodes. This is due to an issue with the Docker BOSH release installed by the PKS v1.0.0 tile.

In this version of the BOSH release, Docker occasionally fails to unmount all overlays when stopping a node. When you upgrade the stemcell for the PKS tile, BOSH recreates VMs and can fail to correctly unmount Docker overlays.

To avoid this issue, follow the steps in the Upgrade Procedure section when you upgrade the PKS tile. The docker_ctl_update.sh script correctly unmounts Docker overlays by replacing the docker_ctl script on all worker nodes that have Docker deployed.

Stemcell Updates Cause Automatic VM Upgrading

Enabling the Upgrade all clusters errand allows automatic upgrading for VMs in your deployment. Pivotal recommends enabling this errand to ensure that all deployed cluster VMs are patched.

When you enable the Upgrade all clusters errand, the following actions can cause downtime:

  • Updating the PKS tile with a new stemcell triggers updating each VM in each cluster.
  • Updating other tiles in your deployment with new stemcells causes the upgrading of the PKS tile.

Upgrade Errand Fails with Failed Deployments

The Upgrade all clusters errand fails if any deployments are in a failed state.

To work around this issue, delete the failed cluster using the PKS CLI or redeploy the failed cluster with the BOSH CLI to ensure the cluster is in a successful state.

Pods Lose Network Connectivity After VM Cold Migration

When a Kubernetes cluster worker VM goes through cold migration in vSphere, newly provisioned pods lose network connectivity.

This issue can occur under the following conditions:

  • When the VM is powered off and is subject to cold migration, and the VM moves to a different ESXi host
  • When the VM is powering on and is subject to Distributed Resource Scheduler (DRS) before the powerup completes
  • When the vNIC of the VM is detached and reattached

To work around this issue, delete the worker VM. BOSH recreates the worker VM and restores network connectivity.


StatefulSets Pod Failure After Recreating a VM

When using vSphere with NSX-T integration, if you recreate a node that hosts a StatefulSets pod, the pod can get stuck in a ContainerCreating state. The pod emits a warning event with a FailedCreatePodSandBox reason. This issue affects StatefulSets pods created before PKS v1.0.2.

A fix for this bug is included in PKS v1.0.2, but the fix applies only to StatefulSets created using PKS v1.0.2 or later. After upgrading PKS to v1.0.2, manually deleting and recreating all preexisting StatefulSets pods is recommended, even if they are in a running state.

To get all StatefulSets pods, run the following command on every Kubernetes cluster using the Kubernetes admin user permissions:

$ kubectl get pods -l "statefulset.kubernetes.io/pod-name" \
-o wide --all-namespaces

For each result, delete the pod by running the following command:

$ kubectl delete pod POD-NAME -n POD-NAMESPACE

You do not need to manually recreate the deleted pods. Kubernetes detects a StatefulSet with missing pods and automatically recreates the pods.

[Kubernetes Bug] Upgrading a Cluster Affects Persistent Workload Uptime

During an upgrade to v1.0.2 on vSphere, persistent storage volumes do not reattach to pods until all worker nodes have been upgraded, which results in workload downtime until the entire cluster is upgraded.

This issue occurs when you deploy a pod with persistent storage attached, drain the node, and then immediately delete the node VM.

The expected behavior is for persistent disks to reattach to the upgraded VMs after the pod is restored. However, a Kubernetes bug prevents the disk from reattaching. PKS v1.0.2 works around this bug by attaching the volumes after all workers are upgraded.

For more information, see the Kubernetes issue on GitHub.

In rare cases, pods with persistent volumes can stay in ContainerCreating state. If you see the error FailedMount Unable to mount volumes for pod POD-NAME, perform the following steps:

  1. Find the problem node by running kubectl describe pod POD-NAME.
  2. Prevent scheduling on the node that runs the pod by running kubectl cordon NODE-NAME.
  3. Delete pod by running kubectl delete pod POD-NAME.
  4. Wait for pod to be rescheduled and enter Running state. This may take several minutes.
  5. Resume scheduling on the node that runs the pod by running kubectl uncordon NODE-NAME.

Kubernetes Cluster Creation Fails if NSX-T Manager Password Begins with Certain Special Characters

If you select NSX-T as a Container Network Type in PKS and your NSX-T Manager password begins with an @, $, ^, ', or space character, Kubernetes cluster creation fails. To resolve this issue, reset your NSX-T Manager password so that it does not begin with any of these characters. After resetting your NSX-T Manager password, reconfigure your NSX-T Manager credentials in the PKS tile with the updated password.

v1.0.0

Release Date: February 8, 2018

Features

  • Create, resize, delete, list, and show clusters through the PKS CLI
  • Native support for NSX-T and Flannel
  • Easily obtain kubeconfigs to use each cluster
  • Use kubectl to view the Kubernetes dashboard
  • Define plans that pre-configure VM size, authentication, default number of workers, and addons when creating Kubernetes clusters
  • User/Admin configurations for access to PKS API
  • Centralized logging through syslog

Component Versions

PKS v1.0.0 includes or supports the following component versions:

Product Component Version Supported Notes
Pivotal Cloud Foundry Operations Manager (Ops Manager) 2.0.0 - 2.0.5 Separate download available from Pivotal Network
vSphere 6.5 and 6.5 U1 - Editions
  • vSphere Enterprise Plus Edition
  • vSphere with Operations Management Enterprise Plus
vSphere versions supported for Pivotal Container Service (PKS)
VMware Harbor Registry 1.4.1 Separate download available from Pivotal Network
NSX-T 2.1 Advanced Edition Available from VMware
Stemcell 3468.21 Separate download available from Pivotal Network
Kubernetes 1.9.2 Packaged in the PKS Tile (CFCR)
CFCR (Kubo) 0.13 Packaged in the PKS Tile
NCP 2.1.0.1 Packaged in the PKS Tile
Kubernetes CLI 1.9.2 Separate download available from the PKS section of Pivotal Network
PKS CLI 1.0.0-build.3 Separate download available from the PKS section of Pivotal Network

Known Issues

This section includes known issues with PKS v1.0.0 and corresponding workarounds.

Access to the Kubernetes API is Unavailable During Upgrades

PKS upgrades include upgrades to the master node. While the master node is undergoing an upgrade, the Kubernetes API is unavailable.

If you attempt to access the API during an upgrade, you will not be able to connect.

Special Characters

In PKS v1.0.0, special characters, such as #, &, ;, ", ', ^, \, space (), !, and % cannot be used in vCenter passwords. To resolve this issue, reset your password so that it does not include any of the special characters listed above. After resetting your password in vCenter, reconfigure your credentials in the PKS tile with the updated password. PKS v1.0.2 adds support for the special characters listed above.

Stemcell Incompatibility with NSX-T

When deploying PKS v1.0.0 using NSX-T as the networking layer with a stemcell other than 3468.21, Kubernetes cluster deployments fail. PKS v1.0.2 adds support for stemcells v3468.25 and later.

Stemcell Updates Cause Automatic VM Upgrading

Enabling the Upgrade all clusters errand allows automatic upgrading for VMs in your deployment. Pivotal recommends enabling this errand to ensure that all deployed cluster VMs are patched.

When you enable the Upgrade all clusters errand, the following actions can cause downtime:

  • Updating the PKS tile with a new stemcell triggers the rolling of each VM in each cluster.
  • Updating other tiles in your deployment with new stemcells causes the rolling of the PKS tile.

Upgrade Errand Fails with Failed Deployments

The Upgrade all clusters errand fails if any deployments are in a failed state.

To work around this issue, delete the failed cluster using the PKS CLI or redeploy the failed cluster with the BOSH CLI to ensure the cluster is in a successful state.

Syslog Security Recommendations

BOSH Director logs contain sensitive information that should be considered privileged. For example, these logs may contain cloud provider credentials in PKS v1.0.0. If you choose to forward logs to an external syslog endpoint, using TLS encryption is strongly recommended to prevent information from being intercepted by a third party.


Please send any feedback you have to pks-feedback@pivotal.io.

Create a pull request or raise an issue on the source for this page in GitHub