Troubleshooting Windows Cells
This topic describes how to troubleshoot Windows Diego cells deployed by Pivotal Application Service (PAS) for Windows.
This section describes issues that may occur during the installation process.
You run the
winfs-injector and see the following error about certificates:
Get https://auth.docker.io/token?service=registry.docker.io& scope=repository:cloudfoundry/windows2016fs:pull: x509: failed to load system roots and no roots provided
Local certificates are needed to communicate with Docker Hub.
Install the necessary certificates on your local machine. On Ubuntu, you can install certificates with the
You run the
winfs-injector and see the following error about a missing file or directory:
open ...windows2016fs-release/VERSION: no such file or directory
You are using an outdated version of the
From Pivotal Network, download the recommended version of PAS for Windows File System Injector tool for the tile.
You click the + icon in Ops Manager to add the PAS for Windows tile to the Installation Dashboard, and you see the error:
The product file you are trying to upload does not contain the Windows Server container base image.
Delete the product file listing from Ops Manager by clicking its trash can icon under Import a Product.
Follow the PAS for Windows installation instructions to run the
winfs-injectortool locally on the product file. This step requires internet access, can take up to 20 minutes, and adds the Windows Server container base image to the product file.
Click Import a Product to upload the “injected” product file.
Click the + icon next to the product listing to add the PAS for Windows tile to the Installation Dashboard.
You can use Windows cell logs to troubleshoot Windows Diego cells. Windows cells generate the following types of logs:
BOSH job logs, such as
consul_agent_windows. These logs stream to the syslog server configured in the PAS tile System Logging pane, along with other PCF component logs. The names of these BOSH job logs correspond to the names of the logs emitted by Linux Diego cells.
Windows Event logs. These logs stream to the syslog server configured in the PAS for Windows System Logging pane.
You can forward BOSH job logs and Windows Event logs to an external syslog server in the following ways:
Configure a BOSH add-on to forward BOSH job logs. See the BOSH jobs logs step in Step 2: Install the Tile in Installing and Configuring PASW.
Configure PAS for Windows to forward Windows Event logs. See Forward Windows Event Logs to a Syslog Server below.
You can download the forwarded BOSH job logs and Window Event logs, in the PASW tile. See Download Cell Logs below.
To forward Windows Event logs to an external syslog server, do the following:
Navigate to the Ops Manager Installation Dashboard.
Click the PAS for Windows tile.
Under the Settings tab, select the System Logging pane.
Under Enable Syslog for VM logs?, click Enable
Under Address, enter the IP address of your syslog server.
Under Port, enter the port of your syslog server. The typical port for a syslog server is
Note: The host must be reachable from the PAS network. Ensure your syslog server listens on external interfaces.
Under Protocol, select the transport protocol to use when forwarding logs.
Select Enable System Metrics. For a list of the VM metrics that the System Metric Agent emits, see VM Metrics in GitHub.
To download Windows cell log, do the following:
- Navigate to the Ops Manager Installation Dashboard.
- Click the PAS for Windows tile.
- Click the Status tab.
- Under the Logs column, click the download icon for the Windows cell you want to retrieve logs from.
- Click the Logs tab.
- When the logs are ready, click the filename to download them.
- Unzip the file to examine the contents. Each component on the cell has its own logs directory:
Perform the following steps to connect to a Windows cell to run diagnostics:
Download and install a Remote Desktop Protocol (RDP) client.
Follow the steps in the Log into BOSH section of the Advanced Troubleshooting with the BOSH CLI topic to log in to your BOSH Director.
- The steps vary slightly depending on whether your PCF deployment uses internal authentication or an external user store.
- In PAS for Windows v2.6 and later, use of the
bosh sshcommand is enabled by default.
To retrieve the IP address of your Windows cell using the BOSH CLI, run the following command:
bosh -e ENV-NAME -d DEPLOYMENT-NAME
ENV-NAMEis the alias you assigned to your BOSH Director.
DEPLOYMENT-NAMEis your deployment’s name.
c:\Users\admin> bosh -e my-environ -d garden-windows Using environment '192.0.2.6' as client 'admin'
Name Release(s) Stemcell(s) Team(s) Cloud Config garden-windows ... ... - latest
Retrieve the Administrator password for your Windows cell by following the steps for your IaaS:
- On vSphere, this is the value of
consumer-vars.ymlfile you used to previously build a stemcell.
- On Amazon Web Services (AWS), navigate to the AWS EC2 console. Right-click on your Windows cell and select Get Windows Password from the drop-down menu. Provide the local path to the
ops_mgr.pemprivate key file you used when installing Ops Manager and click Decrypt password to obtain the Administrator password for your Windows cell.
- On Google Cloud Platform (GCP), navigate to the Compute Engine Dashboard. Under VM Instances, select the instance of the Windows VM. At the top of the page, click on Create or reset Windows password. When prompted, enter “Administrator” under Username and click Set. You will receive a one-time password for the Windows cell.
- You cannot RDP into Windows cells on Azure.
- On vSphere, this is the value of
Open your RDP client. The examples below use the Microsoft Remote Desktop app.
Click New and enter your connection information:
- Connection name: Enter a name for this connection.
- PC name: Enter the IP address of your Windows cell.
- User name: Enter
- Password: Enter the password of your Windows cell that you obtained above.
To mount a directory on your local machine as a drive in the Windows cell, perform the following steps:
- From the same Edit Remote Desktops window as above, click Redirection.
- Click the plus icon at the bottom left.
- For Name, enter the name of the drive as it will appear in the Windows cell. For Path, enter the path of the local directory.
- Click OK.
Close the Edit Remote Desktops window and double-click the newly added connection under My Desktops to open a RDP connection to the Windows cell.
In the RDP session, you can use the Consul CLI to diagnose problems with your Windows cell.
Perform the following steps to use the Consul CLI on your Windows cell to diagnose problems with your Consul cluster:
- In your RDP session, open a PowerShell window.
To change to the directory containing the Consul CLI binary, run the following:
CONSUL-CLI-DIRis the Consul CLI package’s directory path.
PS C:\Users\admin> cd C:\var\vcap\packages\consul-windows\bin\
To list the members of your Consul cluster, run the following:
PS C:\var\vcap\packages\consul-windows\bin> consul.exe members Node Address Status Type Build Protocol DC cell-windows-0 10.0.0.111:8301 alive client 0.6.4 2 dc1 cloud-controller-0 10.0.0.94:8301 alive client 0.6.4 2 dc1 cloud-controller-worker-0 10.0.0.99:8301 alive client 0.6.4 2 dc1 consul-server-0 10.0.0.96:8301 alive server 0.6.4 2 dc1 diego-brain-0 10.0.0.109:8301 alive client 0.6.4 2 dc1 diego-cell-0 10.0.0.103:8301 alive client 0.6.4 2 dc1 diego-cell-1 10.0.0.104:8301 alive client 0.6.4 2 dc1 diego-cell-2 10.0.0.107:8301 alive client 0.6.4 2 dc1 diego-database-0 10.0.0.92:8301 alive client 0.6.4 2 dc1 ha-proxy-0 10.0.0.254:8301 alive client 0.6.4 2 dc1 nfs-server-0 10.0.0.100:8301 alive client 0.6.4 2 dc1 router-0 10.0.0.105:8301 alive client 0.6.4 2 dc1 uaa-0 10.0.0.93:8301 alive client 0.6.4 2 dc1
Examine the output to ensure that the
cell-windows-0service is registered in the Consul cluster and is
alive. Otherwise, your Windows cell cannot communicate with your PCF deployment and developers cannot push .NET apps to the Windows cell. Check the configuration of your Consul cluster, and ensure that your certificates are not missing or misconfigured.