Verifying Deployment Health

Verify Kubernetes Health
Verify NCP Health (NSX-T Only)

Page last updated:

Warning: Pivotal Container Service (PKS) v1.3 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy.
To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes how to verify the health of your Pivotal Container Service (PKS) deployment.

Verify Kubernetes Health

Verify the health of your Kubernetes environment by following the steps below:

From the Ops Manager VM, run the following command:
```
bosh -e ENV-NAME login
```
Where ENV-NAME is the alias you set for your BOSH Director. For more information, see Managing PKS Deployments with BOSH.

For example:
```
$ bosh -e pks login
```
To verify that all nodes are in a ready state, run the following command for all Kubernetes contexts:
```
kubectl get nodes
```
To verify that all pods are running, run the following command for all Kubernetes contexts:
```
kubectl get pods --all-namespaces
```

To get the name of the target Kubernetes deployment, run the following command:

bosh deployments

For example:

$ bosh deployments
Using environment '30.0.0.10' as client 'ops_manager'

Name                                                   Release(s)                               Stemcell(s)                                      Team(s)
harbor-container-registry-b4023f6857207b237399         bosh-dns/1.10.0                          bosh-vsphere-esxi-ubuntu-xenial-go_agent/170.15  -
                                                       harbor-container-registry/1.7.3-build.2              
pivotal-container-service-7e64d53fc570503b5690         backup-and-restore-sdk/1.8.0             bosh-vsphere-esxi-ubuntu-xenial-go_agent/170.15  -
                                                       bosh-dns/1.10.0
                                                       bpm/0.13.0
                                                       cf-mysql/36.14.0
                                                       cfcr-etcd/1.8.0
                                                       docker/35.0.0
                                                       harbor-container-registry/1.7.3-build.2
                                                       kubo/0.25.9
                                                       kubo-service-adapter/1.3.3-build.1
                                                       nsx-cf-cni/2.3.1.10693410
                                                       on-demand-service-broker/0.24.0
                                                       pks-api/1.3.3-build.1
                                                       pks-helpers/50.0.0
                                                       pks-nsx-t/1.19.0
                                                       pks-telemetry/2.0.0-build.113
                                                       pks-vrli/0.7.0
                                                       sink-resources-release/0.1.15
                                                       syslog/11.4.0
                                                       uaa/64.0   
                                                       wavefront-proxy/0.9.0
service-instance_8de000ff-a87a-4930-81ba-106d42c2471e  bosh-dns/1.10.0                          bosh-vsphere-esxi-ubuntu-xenial-go_agent/170.15  pivotal-container-service-7e64d53fc570503b5690
                                                       bpm/0.13.0
                                                       cfcr-etcd/1.8.0
                                                       docker/35.0.0
                                                       harbor-container-registry/1.7.3-build.2
                                                       kubo/0.25.9
                                                       nsx-cf-cni/2.3.1.10693410    
                                                       pks-helpers/50.0.0
                                                       pks-nsx-t/1.19.0
                                                       pks-telemetry/2.0.0-build.113
                                                       pks-vrli/0.7.0
                                                       sink-resources-release/0.1.15
                                                       syslog/11.4.0
                                                       wavefront-proxy/0.9.0

3 deployments

In the example above, service-instance_8de000ff-a87a-4930-81ba-106d42c2471e is the Kubernetes deployment name.

Note: If you have deployed multiple Kubernetes clusters, determine the UUID using pks clusters, then match that UUID with the Kubernetes cluster deployment you are targeting.

To assess the status of the VMs comprising the target Kubernetes cluster, run the following command:

bosh -d K8-DEPLOYMENT vms

Where K8-DEPLOYMENT is the name of your Kubernetes cluster deployment. Kubernetes cluster deployment names begin with service-instance_ and include a unique BOSH-generated hash.

This command returns the name of each VM comprising the Kubernetes cluster, including each master and worker node.

For example:

$ bosh -d service-instance_8de000ff-a87a-4930-81ba-106d42c2471e vms Using environment '30.0.0.10' as client 'ops_manager'

Task 677. Done

Deployment 'service-instance_8de000ff-a87a-4930-81ba-106d42c2471e'

Instance Process State AZ IPs VM CID VM Type Active master/b6d3c263-1682-4c79-a9ab-35939127dedb running AZ-K8S 40.0.2.2 vm-60dbcf68-5538-4c4e-8c00-61edc003bb54 medium.disk true worker/d450548a-2b0c-4494-8144-cf9b7ef9c825 running AZ-K8S 40.0.2.4 vm-1bfdde6d-ce1d-4cdf-90d9-32bba260358f medium.disk true worker/d7f882f0-33dd-43d3-ab5d-058bcc505088 running AZ-K8S 40.0.2.3 vm-822cb573-411f-4c44-a32b-34e79520a7a6 medium.disk true worker/e5e25ffe-f448-4d19-990b-89546118c502 running AZ-K8S 40.0.2.5 vm-c6748604-8440-4b27-9cf4-10a70a02da24 medium.disk true

4 vms

Succeeded

To verify that all the processes in the Kubernetes cluster are in a running state, run the following command for each deployment:

bosh -d K8-DEPLOYMENT instances --ps

Where K8-DEPLOYMENT is the name of your Kubernetes cluster deployment. Kubernetes cluster deployment names begin with service-instance_ and include a unique BOSH-generated hash.

For example:

$ bosh -d service-instance_8de000ff-a87a-4930-81ba-106d42c2471e instances --ps Using environment '30.0.0.10' as client 'ops_manager'

Task 678. Done

Deployment 'service-instance_8de000ff-a87a-4930-81ba-106d42c2471e'

Instance Process apply-addons/ef0d09ae-d3ed-4832-8d3f-431d99730c26 - master/b6d3c263-1682-4c79-a9ab-35939127dedb - ~ blackbox ~ bosh-dns ~ bosh-dns-healthcheck ~ bosh-dns-resolvconf ~ etcd ~ kube-apiserver ~ kube-controller-manager ~ kube-scheduler ~ ncp ~ pks-helpers- worker/d450548a-2b0c-4494-8144-cf9b7ef9c825 - ~ blackbox ~ bosh-dns ~ bosh-dns-healthcheck ~ bosh-dns-resolvconf ~ docker ~ kube-proxy ~ kubelet ~ nsx-kube-proxy ~ nsx-node-agent ~ ovs-vswitchd ~ ovsdb-server ~ pks-helpers- worker/d7f882f0-33dd-43d3-ab5d-058bcc505088 - ~ blackbox ~ bosh-dns ~ bosh-dns-healthcheck ~ bosh-dns-resolvconf ~ docker ~ kube-proxy ~ kubelet ~ nsx-kube-proxy ~ nsx-node-agent ~ ovs-vswitchd ~ ovsdb-server ~ pks-helpers- worker/e5e25ffe-f448-4d19-990b-89546118c502 - ~ blackbox ~ bosh-dns ~ bosh-dns-healthcheck ~ bosh-dns-resolvconf ~ docker ~ kube-proxy ~ kubelet ~ nsx-kube-proxy ~ nsx-node-agent ~ ovs-vswitchd ~ ovsdb-server ~ pks-helpers- Process State AZ IPs - AZ-K8S - running AZ-K8S 40.0.2.2 running - - running - - running - - running - - running - - running - - running - - running - - running - - bosh-dns-resolvconf running - - running AZ-K8S 40.0.2.4 running - - running - - running - - running - - running - - running - - running - - running - - running - - running - - running - - bosh-dns-resolvconf running - - running AZ-K8S 40.0.2.3 running - - running - - running - - running - - running - - running - - running - - running - - running - - running - - running - - bosh-dns-resolvconf running - - running AZ-K8S 40.0.2.5 running - - running - - running - - running - - running - - running - - running - - running - - running - - running - - running - - bosh-dns-resolvconf running - -

51 instances

Verify NCP Health (NSX-T Only)

NCP runs as a BOSH host process. Each Kubernetes master node VM has one running NCP process. If your cluster has multiple master nodes, one NCP process is active while the others are on standby. For more information, see Architectural Changes.

Verify NCP health by following the steps below:

Perform the steps in Verify Kubernetes Health to view the VMs and process instances for your target Kubernetes cluster.

To verify the ncp process is running, re-run the bosh instances command on the master node:

bosh -d K8-DEPLOYMENT instances --ps

Where K8-DEPLOYMENT is the name of your Kubernetes cluster deployment. Kubernetes cluster deployment names begin with service-instance_ and include a unique BOSH-generated hash.

For example:

$ bosh -d service-instance_8de000ff-a87a-4930-81ba-106d42c2471e instances --ps
Using environment '30.0.0.10' as client 'ops_manager'

Task 678. Done

Deployment 'service-instance_8de000ff-a87a-4930-81ba-106d42c2471e'

Instance                                           Process                          Process State  AZ      IPs
apply-addons/ef0d09ae-d3ed-4832-8d3f-431d99730c26  -                                -              AZ-K8S  -
master/b6d3c263-1682-4c79-a9ab-35939127dedb        -                                running        AZ-K8S  40.0.2.2
~                                                  blackbox                         running        -       -
~                                                  bosh-dns                         running        -       -
~                                                  bosh-dns-healthcheck             running        -       -
~                                                  bosh-dns-resolvconf              running        -       -
~                                                  etcd                             running        -       -
~                                                  kube-apiserver                   running        -       -
~                                                  kube-controller-manager          running        -       -
~                                                  kube-scheduler                   running        -       -
~                                                  ncp                              running        -       -
~                                                  pks-helpers-bosh-dns-resolvconf  running        -       -

To SSH into the Kubernetes master node VM, run the following command:
```
bosh -e ENV-NAME -d PKS-DEPLOYMENT ssh VM-NAME/VM-ID
```
Where:
- ENV-NAME is the alias you set for your BOSH Director. For more information, see Managing PKS Deployments with BOSH.
- PKS-DEPLOYMENT is the name of your PKS installation. PKS deployment names begin with pivotal-container-service and include a unique BOSH-generated hash.
- VM-NAME is your Kubernetes master node VM name.
- VM-ID is your Kubernetes master node VM ID. This is a unique BOSH-generated hash.
  
  For example:
```
$ bosh -e pks -d pivotal-container-service-7e64d53fc570503b5690 ssh master/b6d3c263-1682-4c79-a9ab-35939127dedb
```
To verify the ncp process is running, run the following command from the master node VM:
```
monit summary
```
To check if the NCP process is active or on standby, run the following command:
```
/var/vcap/jobs/ncp/bin/nsxcli -c get ncp-master status
```
To restart the NCP process, run the following command:
```
monit restart ncp
```
To verify that the NCP process restarts successfully, run the following command:
```
monit summary
```

Please send any feedback you have to pks-feedback@pivotal.io.

Create a pull request or raise an issue on the source for this page in GitHub