Shutting Down and Starting Up PKS

Page last updated:

This topic lists and describes the shutdown and startup sequence for Pivotal Container Service (PKS) including PKS-provisioned Kubernetes cluster nodes, PKS control plane components, and vSphere hosts.

Shutdown Sequence and Tasks

To perform a graceful shutdown of all Kubernetes, PKS, and infrastructure components, complete the following tasks in sequence.

Step 1: Shut Down Customer Apps

Shut down all customer apps running on PKS-provisioned Kubernetes clusters.

Note: This task is optional. Perform it after considering the types of apps you have deployed. For example, stateful, stateless, or legacy apps.

Step 2: Shut Down Kubernetes Clusters

Shut down all PKS-provisioned Kubernetes clusters following the procedure defined in the How to shutdown and startup a Multi Master PKS cluster knowledge base article.

For each Kubernetes cluster that you intend to shut down, do the following:

  1. Using the BOSH CLI, retrieve the BOSH deployment name of your PKS clusters by running the following command.

    bosh deployments
    

    Kubernetes cluster deployment names begin with service-instance_ and include a unique BOSH-generated hash.

  2. Using the BOSH CLI, stop the Kubernetes worker nodes by running the following command.

    bosh -d service-instance_CLUSTER-UUID stop worker
    

    Where CLUSTER-UUID is the BOSH deployment name of your PKS cluster. For example:

    $ bosh -d service-instance_aa1234567bc8de9f0a1c stop worker

    Note: When you use the BOSH stop command, all processes on the Kubernetes node are stopped. BOSH marks them stopped so that when the VM is powered back on, the processes do not start automatically.

  3. Using the BOSH CLI, stop the Kubernetes master nodes by running the following command.

    bosh -d service-instance_CLUSTER-UUID stop master
    

    Where CLUSTER-UUID is the BOSH deployment name of your PKS cluster. For example:

    $ bosh -d service-instance_aa1234567bc8de9f0a1c stop master
  4. Using vCenter, shut down all Kubernetes node VMs. To do this, perform the following steps:

    1. Verify the node type by checking the “job” name in the the Custom Attributes pane.
    2. Perform a graceful shutdown by right-clicking the target VM and selecting Power > Shut Down Guest OS.

      Shut Down Kubernetes Nodes

      View a larger version of this image.

Step 3: Shut Down PKS Control Plane

To shut down the PKS control plane VM, do the following:

  1. Using the BOSH CLI, retrieve the BOSH deployment ID of your PKS deployment by running the following command.

    bosh deployments
    

    PKS deployment names begin with pivotal-container-service and include a unique BOSH-generated hash.

  2. Using the BOSH CLI, stop the PKS control plane VM by running the following command.

    bosh -d pivotal-container-service-DEPLOYMENT-ID stop 
    

    Where DEPLOYMENT-ID is the BOSH-generated ID of your PKS deployment. For example:

    $ bosh -d pivotal-container-service-1bf7b02738056cdc37e6 stop
  3. Using vCenter, locate and gracefully shut down the PKS control plane VM.

    Shut Down PKS control plane VM

    View a larger version of this image.

Step 4: Shut Down VMware Harbor Registry

To shut down the Harbor Registry VM, do the following:

  1. Using the BOSH CLI, retrieve the BOSH deployment ID of your Harbor Registry deployment by running the following command.

    bosh deployments
    

    Harbor Registry deployment names begin with harbor-container-registry and include a unique BOSH-generated hash.

  2. Using the BOSH CLI, stop the Harbor Registry VM by running the following command.

    bosh -d harbor-container-registry-DEPLOYMENT-ID stop
    

    Where DEPLOYMENT-ID is the BOSH-generated ID of your Harbor Registry deployment. For example:

    $ bosh -d harbor-container-registry-b4023f6857207b237399 stop
  3. Using vCenter, locate and gracefully shut down the Harbor Registry VM.

    Shut Down Harbor Registry VM

    View a larger version of this image.

Step 5: Shut Down BOSH Director

Using vCenter, locate and gracefully shut down the BOSH Director VM.

Shut Down BOSH VM

View a larger version of this image.

Step 6: Shut Down Ops Manager

Using vCenter, locate and gracefully shut down the Ops Manager VM.

Shut Down Ops Manager VM

View a larger version of this image.

Step 7: Shut Down NSX-T Components

Using vCenter, gracefully shut down all NSX-T VMs in the following order:

  1. NSX-T Manager
  2. NSX-T Controllers
  3. NSX-T Edge Nodes

Shut Down NSX-T VMs

View a larger version of this image.

Step 8: Shut Down vCenter Server

To shut down the vCenter Server VM, do the following:

  1. Navigate to the vCenter Appliance Management Interface at https://YOUR-VCENTER-HOSTNAME-OR-IP-ADDRESS>:5480, where YOUR-VCENTER-HOSTNAME-OR-IP-ADDRESS is the hostname or IP address that you use to connect to vCenter through the vSphere Web Client.
  2. Log in as root.
  3. Select Actions > Shutdown from the menu and confirm the operation.

For more information about how to shut down the vCenter Server VM, see Reboot or Shut Down the vCenter Server Appliance in the vSphere documentation and the How to stop, start, or restart vCenter Server 6.x services KB article.

Note: After you shut down this vCenter VM, the vSphere Web Client will not be available.

Shut Down vCenter Server VM

View a larger version of this image

Step 9: Shut Down ESXi Hosts

To shut down each ESXi host in the vSphere cluster, do the following:

  1. Put the ESXi host into maintenance mode by doing the following:

    1. Using a browser, navigate to the HTTPS IP address of the ESXi host, for example: https://10.196.146.20/.
    2. Log in using vSphere administrative credentials.
    3. Put the ESXi host in maintenance mode by selecting Actions > Enter maintenance mode. Put ESXi Host into Maintenance Mode View a larger version of this image.
  2. Power off the ESXi host. To do this, you have two options:

Startup Sequence and Tasks

To restart all Kubernetes, PKS, and infrastructure components, complete the following tasks in the sequence presented.

Step 1: Start ESXi Hosts

To start the ESXi hosts, do the following:

  1. Using the remote management console, such as Dell IDRAC or HP iLO, power on each ESXi host.
  2. Connect to the web interface of each ESXi host and exit maintenance mode.

Step 2: Start vCenter

Connect to the web interface of the ESXi server that hosts the vCenter VM. Select the vCenter VM, and click Power On.

Step 3: Start NSX-T Components

To start the NSX-T components, perform the following steps:

  1. Log into vCenter using the vSphere Client.
  2. Power on the following VMs in the following order:
    1. NSX-T Manager
    2. NSX-T Controllers
    3. NSX-T Edge Nodes

Step 4: Start Ops Manager

Using vCenter, power on the Ops Manager VM.

Step 5: Start the BOSH Director

Using vCenter, power on the BOSH Director VM.

Note: BOSH is aware that all the VMs under its control were stopped. BOSH does not attempt to resurrect any VMs, which is the desired behavior.

It may take approximately 90 minutes for BOSH to start properly.

To speed up the BOSH startup process:

  1. Obtain the BOSH Director VM Credentials from Ops Manager. For information about doing this, see Retrieving Credentials from Your Deployment in the Pivotal documentation.
  2. SSH to the BOSH Director VM.
  3. On the BOSH Director VM, run the following commands:

    sudo -i
    monit summary
    
  4. If you see messages such as Process uaa Connection failed and Process credhub not monitored, then run the following command:

    monit restart uaa
    
  5. After a few minutes, run the following command again:

    monit summary
    

    You should see that the uaa and credhub processes are now running. At this point, the BOSH Director should be fully up and running.

Step 6: Start the PKS Control Plane

To start the PKS control plane, do the following:

  1. Using vCenter, power on the PKS control plane VM.
  2. Using the BOSH CLI, start the PKS process on the VM by running the following command.

    bosh -d pivotal-container-service-DEPLOYMENT-ID start
    

    Where DEPLOYMENT-ID is the BOSH-generated ID of the PKS deployment. For example:

    $ bosh -d pivotal-container-service-1bf7b02738056cdc37e6 start

    Note: Because you stopped the PKS process using BOSH, you must restart it using BOSH.

Step 7: Start Harbor Registry

To start Harbor Registry, do the following:

  1. Using vCenter, power on the Harbor VM.
  2. Using the BOSH CLI, start the Harbor process on the VM by running the following command:

    bosh -d harbor-container-registry-DEPLOYMENT-ID start
    

    Where DEPLOYMENT-ID is the BOSH-generated ID of your Harbor Registry deployment. For example:

    $ bosh -d harbor-container-registry-b4023f6857207b237399 start

Step 8: Start the Kubernetes Clusters

For each Kubernetes cluster that you intend to start up, start the Kubernetes nodes in the following order:

  1. Using the BOSH CLL, run ssh to access the first PKS master node and start etcd.
  2. Using the BOSH CLI, start the next PKS master node.
  3. Using the BOSH CLI, start all remaining PKS master nodes including the master where you started etcd.
  4. Using the BOSH CLI, start all PKS worker nodes.

For exact Kubernetes node startup instructions, refer to the How to shutdown and startup a Multi Master PKS cluster knowledge base article.

Step 9: Start Customer Apps

Start all apps running on the PKS-provisioned Kubernetes clusters.


Please send any feedback you have to pks-feedback@pivotal.io.