Shutting Down and Restarting Tanzu Kubernetes Grid Integrated Edition

Page last updated:

This topic lists and describes the shutdown and startup sequence for VMware Tanzu Kubernetes Grid Integrated Edition including Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes cluster nodes, TKGI components, and (vSphere only) vSphere hosts.

Many of these operations use your IaaS dashboard, such as vSphere Client, Azure Portal, AWS Management Console, or GCP Console.

Shutdown Sequence and Tasks

To perform a graceful shutdown of all Kubernetes, Tanzu Kubernetes Grid Integrated Edition, and infrastructure components, complete the following tasks in sequence.

Step 1: Disable BOSH Resurrection

If you have the Enable VM Resurrector Plugin checkbox selected in the BOSH Director tile > Director Config pane, you must turn BOSH resurrection off before restarting TKGI, to prevent BOSH from recreating VMs.

To do this, run the command bosh update-resurrection off.

Step 2: Shut Down Customer Apps

Shut down all customer apps running on Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes clusters.

Note: This task is optional. Perform it after considering the types of apps you have deployed. For example, stateful, stateless, or legacy apps.

Step 3: Shut Down Kubernetes Clusters

Shut down all Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes clusters following the procedure defined in the How to shutdown and startup a Multi Master TKGI cluster knowledge base article.

For each Kubernetes cluster that you intend to shut down, do the following:

  1. Using the BOSH CLI, retrieve the BOSH deployment name of your Tanzu Kubernetes Grid Integrated Edition clusters by running the following command:

    bosh deployments
    

    Kubernetes cluster deployment names begin with service-instance_ and include a unique BOSH-generated hash.

  2. Using the BOSH CLI, stop the Kubernetes worker nodes by running the following command:

    bosh -d service-instance_CLUSTER-UUID stop worker
    

    Where CLUSTER-UUID is the BOSH deployment name of your Tanzu Kubernetes Grid Integrated Edition cluster. For example:

    $ bosh -d service-instance_aa1234567bc8de9f0a1c stop worker

    Note: When you use the BOSH stop command, all processes on the Kubernetes node are stopped. BOSH marks them stopped so that when the VM is powered back on, the processes do not start automatically.

  3. Using the BOSH CLI, stop the Kubernetes master nodes by running the following command:

    bosh -d service-instance_CLUSTER-UUID stop master
    

    Where CLUSTER-UUID is the BOSH deployment name of your Tanzu Kubernetes Grid Integrated Edition cluster. For example:

    $ bosh -d service-instance_aa1234567bc8de9f0a1c stop master
  4. Using your IaaS dashboard, shut down all Kubernetes node VMs. To do this, perform the following steps:

    1. Verify the node type by checking the “job” name in the Custom Attributes pane.
    2. Perform a graceful shutdown by right-clicking the target VM and selecting Power > Shut Down Guest OS.

      Shut Down Kubernetes Nodes

      View a larger version of this image.

Step 4: Shut Down the Tanzu Kubernetes Grid Integrated Edition API and Database VMs

To shutdown Tanzu Kubernetes Grid Integrated Edition control plane TKGI API and TKGI Database VMs, complete the following:

  1. Stop Tanzu Kubernetes Grid Integrated Edition Control Plane Processes
  2. Shut Down the Tanzu Kubernetes Grid Integrated Edition API and Database VMs

Stop Tanzu Kubernetes Grid Integrated Edition Control Plane Processes

To stop Tanzu Kubernetes Grid Integrated Edition control plane processes and services, do the following:

  1. Using the BOSH CLI, retrieve the BOSH deployment ID of your Tanzu Kubernetes Grid Integrated Edition deployment by running the following command:

    bosh deployments
    

    The Tanzu Kubernetes Grid Integrated Edition deployment ID is pivotal-container-service- followed by a unique BOSH-generated hash.

  2. Using the BOSH CLI, stop the TKGI control plane VM by running the following command:

    bosh -d pivotal-container-service-DEPLOYMENT-ID stop
    

    Where DEPLOYMENT-ID is the BOSH-generated ID of your Tanzu Kubernetes Grid Integrated Edition deployment.

    For example:

    $ bosh -d pivotal-container-service-1bf7b02738056cdc37e6 stop
    

Shut Down the Tanzu Kubernetes Grid Integrated Edition API and Database VMs

To shut down the TKGI API and TKGI Database VMs, do the following:

  1. Run the bosh vms command to list your Tanzu Kubernetes Grid Integrated Edition control plane VMs:

    bosh -d pivotal-container-service-DEPLOYMENT-ID vms
    

    Where DEPLOYMENT-ID is the BOSH-generated ID of your Tanzu Kubernetes Grid Integrated Edition deployment.

    For example:

    $ bosh -d pivotal-container-service-1bf7b02738056cdc37e6 vms
  2. From the bosh vms output, record:

    • The TKGI API VM name, listed under Instances as pivotal-container-service/ followed by a unique BOSH-generated hash
    • The TKGI Database VM name, listed under Instances as pks-db/ followed by a unique BOSH-generated hash
  3. Using your IaaS dashboard, locate and gracefully shut down your Tanzu Kubernetes Grid Integrated Edition control plane VMs:

    • The TKGI API VM
    • The TKGI Database VM

    Shut Down TKGI API VM

    View a larger version of this image.

Step 5: Shut Down VMware Harbor Registry (vSphere Only)

To shut down the Harbor Registry VM, do the following:

  1. Using the BOSH CLI, retrieve the BOSH deployment ID of your Harbor Registry deployment by running the following command:

    bosh deployments
    

    Harbor Registry deployment names begin with harbor-container-registry and include a unique BOSH-generated hash.

  2. Using the BOSH CLI, stop the Harbor Registry VM by running the following command:

    bosh -d harbor-container-registry-DEPLOYMENT-ID stop
    

    Where DEPLOYMENT-ID is the BOSH-generated ID of your Harbor Registry deployment. For example:

    $ bosh -d harbor-container-registry-b4023f6857207b237399 stop
  3. Using vCenter, locate and gracefully shut down the Harbor Registry VM.

    Shut Down Harbor Registry VM

    View a larger version of this image.

Step 6: Shut Down BOSH Director

Using your IaaS dashboard, locate and gracefully shut down the BOSH Director VM.

Shut Down BOSH VM

View a larger version of this image.

Step 7: Shut Down Ops Manager

Using your IaaS dashboard, locate and gracefully shut down the Ops Manager VM.

Shut Down Ops Manager VM

View a larger version of this image.

Step 8: Shut Down NSX-T Components (vSphere NSX-T Only)

Using vCenter, gracefully shut down all NSX-T VMs in the following order:

  1. NSX-T Manager
  2. NSX-T Controllers
  3. NSX-T Edge Nodes

Shut Down NSX-T VMs

View a larger version of this image.

Step 9: Shut Down vCenter Server (vSphere Only)

To shut down the vCenter Server VM, do the following:

  1. Navigate to the vCenter Appliance Management Interface at https://YOUR-VCENTER-HOSTNAME-OR-IP-ADDRESS>:5480, where YOUR-VCENTER-HOSTNAME-OR-IP-ADDRESS is the hostname or IP address that you use to connect to vCenter through the vSphere Web Client.
  2. Log in as root.
  3. Select Actions > Shutdown from the menu and confirm the operation.

For more information about how to shut down the vCenter Server VM, see Reboot or Shut Down the vCenter Server Appliance in the vSphere documentation and the How to stop, start, or restart vCenter Server 6.x services KB article.

Note: After you shut down this vCenter VM, the vSphere Web Client will not be available.

Shut Down vCenter Server VM

View a larger version of this image

Step 11: Shut Down ESXi Hosts (vSphere NSX-T Only)

To shut down each ESXi host in the vSphere cluster, do the following:

  1. Put the ESXi host into maintenance mode by doing the following:

    1. Using a browser, navigate to the HTTPS IP address of the ESXi host, for example: https://10.196.146.20/.
    2. Log in using vSphere administrative credentials.
    3. Put the ESXi host in maintenance mode by selecting Actions > Enter maintenance mode. Put ESXi Host into Maintenance Mode View a larger version of this image.
  2. Power off the ESXi host. To do this, you have two options:

Startup Sequence and Tasks

To restart all Kubernetes, Tanzu Kubernetes Grid Integrated Edition, and infrastructure components, complete the following tasks in the sequence presented.

Step 1: Start ESXi Hosts (vSphere NSX-T Only)

To start the ESXi hosts, do the following:

  1. Using the remote management console, such as Dell IDRAC or HP iLO, power on each ESXi host.
  2. Connect to the web interface of each ESXi host and exit maintenance mode.

Step 2: Start vCenter (vSphere Only)

Connect to the web interface of the ESXi server that hosts the vCenter VM. Select the vCenter VM, and click Power On.

Step 3: Start NSX-T Components (vSphere NSX-T Only)

To start the NSX-T components, perform the following steps:

  1. Log into vCenter using the vSphere Client.
  2. Power on the following VMs in the following order:
    1. NSX-T Manager
    2. NSX-T Controllers
    3. NSX-T Edge Nodes

Step 4: Start Ops Manager

  1. Using your IaaS dashboard, power on the Ops Manager VM.
  2. Using a browser, go to the Ops Manager URL.
  3. Enter the Ops Manager passphrase.
  4. Log in to the Ops Manager UI.

Step 5: Start the BOSH Director

Using your IaaS dashboard, power on the BOSH Director VM.

Note: BOSH is aware that all the VMs under its control were stopped. BOSH does not attempt to resurrect any VMs, which is the desired behavior.

It may take approximately 90 minutes for BOSH to start properly.

To speed up the BOSH startup process:

  1. Obtain the BOSH Director VM Credentials from Ops Manager. For information about doing this, see Retrieving Credentials from Your Deployment in the Ops Manager documentation.
  2. SSH to the BOSH Director VM.
  3. On the BOSH Director VM, run the following commands:

    sudo -i
    monit summary
    
  4. If you see messages such as Process uaa Connection failed and Process credhub not monitored, then run the following command:

    monit restart uaa
    
  5. After a few minutes, run the following command again:

    monit summary
    

    You should see that the uaa and credhub processes are now running. At this point, the BOSH Director should be fully up and running.

Step 6: Start the Control Plane VMs

To start the TKGI Control Plane, do the following:

  1. Using your IaaS dashboard, power on the TKGI Database VM.

  2. Run bosh start to start the TKGI Database VM:

    bosh -d TKGI-DB-VM-ID start
    

    Where TKGI-DB-VM-ID is the name of the TKGI Database VM listed by the bosh vms command.

  3. Using your IaaS dashboard, power on the TKGI API VM.

  4. Run bosh start to start the TKGI API VM:

    bosh -d TKGI-API-VM-ID start
    

    Where TKGI-API-VM-ID is the name of the TKGI API VM.

Step 7: Start Harbor Registry (vSphere Only)

To start Harbor Registry, do the following:

  1. Using vCenter, power on the Harbor VM.
  2. Using the BOSH CLI, start the Harbor process on the VM by running the following command:

    bosh -d harbor-container-registry-DEPLOYMENT-ID start
    

    Where DEPLOYMENT-ID is the BOSH-generated ID of your Harbor Registry deployment. For example:

    $ bosh -d harbor-container-registry-b4023f6857207b237399 start

Step 8: Start the Kubernetes Clusters

For each Kubernetes cluster that you intend to start up, start the Kubernetes nodes in the following order:

  1. Using the BOSH CLL, run ssh to access the first Tanzu Kubernetes Grid Integrated Edition master node and start etcd.
  2. Using the BOSH CLI, start the next Tanzu Kubernetes Grid Integrated Edition master node.
  3. Using the BOSH CLI, start all remaining Tanzu Kubernetes Grid Integrated Edition master nodes including the master where you started etcd.
  4. Using the BOSH CLI, start all Tanzu Kubernetes Grid Integrated Edition worker nodes.

For exact Kubernetes node startup instructions, refer to the How to shutdown and startup a Multi Master TKGI cluster knowledge base article.

Step 9: Start Customer Apps

Start all apps running on the Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes clusters.

Step 10: Re-enable BOSH Resurrection

Turn BOSH resurrection back on by running the command bosh update-resurrection on.


Please send any feedback you have to pks-feedback@pivotal.io.