Shutting Down and Restarting Enterprise PKS

Page last updated:

This topic lists and describes the shutdown and startup sequence for Enterprise Pivotal Container Service (Enterprise PKS) including Enterprise PKS-provisioned Kubernetes cluster nodes, PKS control plane components, and vSphere hosts.

Shutdown Sequence and Tasks

To perform a graceful shutdown of all Kubernetes, Enterprise PKS, and infrastructure components, complete the following tasks in sequence.

Step 1: Disable BOSH Resurrection

If you have the Enable VM Resurrector Plugin checkbox selected in the BOSH Director tile > Director Config pane, you must turn BOSH resurrection off before restarting PKS, to prevent BOSH from recreating VMs.

To do this, run the command bosh update-resurrection off.

Step 2: Shut Down Customer Apps

Shut down all customer apps running on Enterprise PKS-provisioned Kubernetes clusters.

Note: This task is optional. Perform it after considering the types of apps you have deployed. For example, stateful, stateless, or legacy apps.

Step 3: Shut Down Kubernetes Clusters

Shut down all Enterprise PKS-provisioned Kubernetes clusters following the procedure defined in the How to shutdown and startup a Multi Master PKS cluster knowledge base article.

For each Kubernetes cluster that you intend to shut down, do the following:

1. Using the BOSH CLI, retrieve the BOSH deployment name of your Enterprise PKS clusters by running the following command:

   bosh deployments
   

   Kubernetes cluster deployment names begin with service-instance_ and include a unique BOSH-generated hash.

2. Using the BOSH CLI, stop the Kubernetes worker nodes by running the following command:

   bosh -d service-instance_CLUSTER-UUID stop worker
   

   Where CLUSTER-UUID is the BOSH deployment name of your Enterprise PKS cluster. For example:

   $ bosh -d service-instance_aa1234567bc8de9f0a1c stop worker

   Note: When you use the BOSH stop command, all processes on the Kubernetes node are stopped. BOSH marks them stopped so that when the VM is powered back on, the processes do not start automatically.

3. Using the BOSH CLI, stop the Kubernetes master nodes by running the following command:

   bosh -d service-instance_CLUSTER-UUID stop master
   

   Where CLUSTER-UUID is the BOSH deployment name of your Enterprise PKS cluster. For example:

   $ bosh -d service-instance_aa1234567bc8de9f0a1c stop master

4. Using vCenter, shut down all Kubernetes node VMs. To do this, perform the following steps:

   1. Verify the node type by checking the “job” name in the Custom Attributes pane.

   2. Perform a graceful shutdown by right-clicking the target VM and selecting Power > Shut Down Guest OS.

      

      View a larger version of this image.

Step 4: Shut Down Enterprise PKS Control Plane

To shut down the PKS control plane VM, do the following:

1. Using the BOSH CLI, retrieve the BOSH deployment ID of your Enterprise PKS deployment by running the following command:

   bosh deployments
   

   Enterprise PKS deployment names begin with pivotal-container-service and include a unique BOSH-generated hash.

2. Using the BOSH CLI, stop the PKS control plane VM by running the following command:

   bosh -d pivotal-container-service-DEPLOYMENT-ID stop 
   

   Where DEPLOYMENT-ID is the BOSH-generated ID of your Enterprise PKS deployment. For example:

   $ bosh -d pivotal-container-service-1bf7b02738056cdc37e6 stop

3. Using vCenter, locate and gracefully shut down the PKS control plane VM.

   

   View a larger version of this image.

Step 5: Shut Down VMware Harbor Registry

To shut down the Harbor Registry VM, do the following:

1. Using the BOSH CLI, retrieve the BOSH deployment ID of your Harbor Registry deployment by running the following command:

   bosh deployments
   

   Harbor Registry deployment names begin with harbor-container-registry and include a unique BOSH-generated hash.

2. Using the BOSH CLI, stop the Harbor Registry VM by running the following command:

   bosh -d harbor-container-registry-DEPLOYMENT-ID stop
   

   Where DEPLOYMENT-ID is the BOSH-generated ID of your Harbor Registry deployment. For example:

   $ bosh -d harbor-container-registry-b4023f6857207b237399 stop

3. Using vCenter, locate and gracefully shut down the Harbor Registry VM.

   

   View a larger version of this image.

Step 6: Shut Down BOSH Director

Using vCenter, locate and gracefully shut down the BOSH Director VM.

View a larger version of this image.

Step 7: Shut Down Ops Manager

Using vCenter, locate and gracefully shut down the Ops Manager VM.

View a larger version of this image.

Step 8: Shut Down NSX-T Components

Using vCenter, gracefully shut down all NSX-T VMs in the following order:

1. NSX-T Manager
2. NSX-T Controllers
3. NSX-T Edge Nodes

View a larger version of this image.

Step 9: Shut Down vCenter Server

To shut down the vCenter Server VM, do the following:

1. Navigate to the vCenter Appliance Management Interface at https://YOUR-VCENTER-HOSTNAME-OR-IP-ADDRESS>:5480, where YOUR-VCENTER-HOSTNAME-OR-IP-ADDRESS is the hostname or IP address that you use to connect to vCenter through the vSphere Web Client.
2. Log in as root.
3. Select Actions > Shutdown from the menu and confirm the operation.

For more information about how to shut down the vCenter Server VM, see Reboot or Shut Down the vCenter Server Appliance in the vSphere documentation and the How to stop, start, or restart vCenter Server 6.x services KB article.

Note: After you shut down this vCenter VM, the vSphere Web Client will not be available.

View a larger version of this image

Step 10: Shut Down ESXi Hosts

To shut down each ESXi host in the vSphere cluster, do the following:

1. Put the ESXi host into maintenance mode by doing the following:

   1. Using a browser, navigate to the HTTPS IP address of the ESXi host, for example: https://10.196.146.20/.
   2. Log in using vSphere administrative credentials.
   3. Put the ESXi host in maintenance mode by selecting Actions > Enter maintenance mode. View a larger version of this image.

2. Power off the ESXi host. To do this, you have two options:

   • Use the EXSi web interface and select Actions > Shut down.
   • Use the remote management console for the host, such as Dell IDRAC or HP iLO. View a larger version of this image.

Startup Sequence and Tasks

To restart all Kubernetes, Enterprise PKS, and infrastructure components, complete the following tasks in the sequence presented.

Step 1: Start ESXi Hosts

To start the ESXi hosts, do the following:

1. Using the remote management console, such as Dell IDRAC or HP iLO, power on each ESXi host.
2. Connect to the web interface of each ESXi host and exit maintenance mode.

Step 2: Start vCenter

Connect to the web interface of the ESXi server that hosts the vCenter VM. Select the vCenter VM, and click Power On.

Step 3: Start NSX-T Components

To start the NSX-T components, perform the following steps:

1. Log into vCenter using the vSphere Client.
2. Power on the following VMs in the following order:
   1. NSX-T Manager
   2. NSX-T Controllers
   3. NSX-T Edge Nodes

Step 4: Start Ops Manager

1. Using vCenter, power on the Ops Manager VM.
2. Using a browser, go to the Ops Manager URL.
3. Enter the Ops Manager passphrase.
4. Log in to the Ops Manager UI.

Step 5: Start the BOSH Director

Using vCenter, power on the BOSH Director VM.

Note: BOSH is aware that all the VMs under its control were stopped. BOSH does not attempt to resurrect any VMs, which is the desired behavior.

It may take approximately 90 minutes for BOSH to start properly.

To speed up the BOSH startup process:

1. Obtain the BOSH Director VM Credentials from Ops Manager. For information about doing this, see Retrieving Credentials from Your Deployment in the Pivotal documentation.
2. SSH to the BOSH Director VM.

3. On the BOSH Director VM, run the following commands:

   sudo -i
   monit summary
   

4. If you see messages such as Process uaa Connection failed and Process credhub not monitored, then run the following command:

   monit restart uaa
   

5. After a few minutes, run the following command again:

   monit summary
   

   You should see that the uaa and credhub processes are now running. At this point, the BOSH Director should be fully up and running.

Step 6: Start the Enterprise PKS Control Plane

To start the PKS control plane, do the following:

1. Using vCenter, power on the PKS control plane VM.

2. Using the BOSH CLI, start the Enterprise PKS process on the VM by running the following command:

   bosh -d pivotal-container-service-DEPLOYMENT-ID start
   

   Where DEPLOYMENT-ID is the BOSH-generated ID of the Enterprise PKS deployment. For example:

   $ bosh -d pivotal-container-service-1bf7b02738056cdc37e6 start

   Note: Because you stopped the Enterprise PKS process using BOSH, you must restart it using BOSH.

Step 7: Start Harbor Registry

To start Harbor Registry, do the following:

1. Using vCenter, power on the Harbor VM.

2. Using the BOSH CLI, start the Harbor process on the VM by running the following command:

   bosh -d harbor-container-registry-DEPLOYMENT-ID start
   

   Where DEPLOYMENT-ID is the BOSH-generated ID of your Harbor Registry deployment. For example:

   $ bosh -d harbor-container-registry-b4023f6857207b237399 start

Step 8: Start the Kubernetes Clusters

For each Kubernetes cluster that you intend to start up, start the Kubernetes nodes in the following order:

1. Using the BOSH CLL, run ssh to access the first Enterprise PKS master node and start etcd.
2. Using the BOSH CLI, start the next Enterprise PKS master node.
3. Using the BOSH CLI, start all remaining Enterprise PKS master nodes including the master where you started etcd.
4. Using the BOSH CLI, start all Enterprise PKS worker nodes.

For exact Kubernetes node startup instructions, refer to the How to shutdown and startup a Multi Master PKS cluster knowledge base article.

Step 9: Start Customer Apps

Start all apps running on the Enterprise PKS-provisioned Kubernetes clusters.

Step 10: Re-enable BOSH Resurrection

Turn BOSH resurrection back on by running the command bosh update-resurrection on.