Troubleshooting Windows Cells

This topic describes how to troubleshoot Windows Diego cells deployed by Pivotal Application Service (PAS) for Windows.

Installation Issues

This section describes issues that may occur during the installation process.

Missing Local Certificates for Windows File System Injector

Symptom

You run the winfs-injector and see the following error about certificates:

Get https://auth.docker.io/token?service=registry.docker.io&
scope=repository:cloudfoundry/windows2016fs:pull: x509: 
failed to load system roots and no roots provided

Explanation

Local certificates are needed to communicate with Docker Hub.

Solution

Install the necessary certificates on your local machine. On Ubuntu, you can install certificates with the ca-certificates package.

Outdated Version for Windows File System Injector

Symptom

You run the winfs-injector and see the following error about a missing file or directory:

open ...windows2016fs-release/VERSION: no such file or directory

Explanation

You are using an outdated version of the winfs-injector.

Solution

From Pivotal Network, download the recommended version of PAS for Windows File System Injector tool for the tile.

Missing Container Image

Symptom

You click the + icon in Ops Manager to add the PAS for Windows tile to the Installation Dashboard, and you see the error:

Error invalid release uninjected tile

Explanation

The product file you are trying to upload does not contain the Windows Server container base image.

Solution

  1. Delete the product file listing from Ops Manager by clicking its trash can icon under Import a Product.

  2. Follow the PAS for Windows installation instructions to run the winfs-injector tool locally on the product file. This step requires internet access, can take up to 20 minutes, and adds the Windows Server container base image to the product file.

  3. Click Import a Product to upload the “injected” product file.

  4. Click the + icon next to the product listing to add the PAS for Windows tile to the Installation Dashboard.

Windows Cell Log Types

Windows cells generate two types of logs:

  • BOSH job logs, such as rep_windows and consul_agent_windows. These logs stream to the syslog server configured in the PAS tile System Logging pane, along with other PCF component logs. The names of these BOSH job logs correspond to the names of the logs emitted by Linux Diego cells.

  • Windows Event logs. These stream to the syslog server configured in the PAS for Windows System Logging pane and are downloadable through Ops Manager, as described below.

Access Windows Event Logs

PCF operators can access log messages from Windows Diego cells in two ways:

  • Configure PAS for Windows to send all Windows cell logs to an external syslog server.

  • Download archived logs from each Windows cell individually.

Send Cell Logs to a Syslog Server

To forward Windows cell log messages to an external syslog server, complete the following steps:

  1. Navigate to the Ops Manager Installation Dashboard.

  2. Click the PAS for Windows tile.

  3. Under the Settings tab, select the System Logging pane. Win syslog config

  4. Under Enable Syslog for VM logs?, click Enable

  5. Under Address, enter the IP address of your syslog server.

  6. Under Port, enter the port of your syslog server. The typical port for a syslog server is 514.

    Note: The host must be reachable from the PAS network. Ensure your syslog server listens on external interfaces.

  7. Under Protocol, select the transport protocol to use when forwarding logs.

  8. Click Save.

Download Cell Logs

Perform the following steps to retrieve the logs from a Windows cell:

  1. Navigate to the Ops Manager Installation Dashboard.
  2. Click the PAS for Windows tile.
  3. Click the Status tab.
  4. Under the Logs column, click the download icon for the Windows cell you want to retrieve logs from.
  5. Click the Logs tab.
  6. When the logs are ready, click the filename to download them.
  7. Unzip the file to examine the contents. Each component on the cell has its own logs directory:
    • /consul_agent_windows/
    • /garden-windows/
    • /metron_agent_windows/
    • /rep_windows/

Connect to a Windows Cell

Perform the following steps to connect to a Windows cell to run diagnostics:

  1. Download and install a Remote Desktop Protocol (RDP) client.

    • For Mac OS X, download the Microsoft Remote Desktop app from the Mac App Store.
    • For Windows, download the Microsoft Remote Desktop app from Microsoft.
    • For Linux/UNIX, download a RDP client like rdesktop.
  2. Follow the steps in the Log into BOSH section of the Advanced Troubleshooting with the BOSH CLI topic to log in to your BOSH Director. The steps vary slightly depending on whether your PCF deployment uses internal authentication or an external user store.

  3. Retrieve the IP address of your Windows cell using the BOSH CLI. Run the following command, replacing MY-ENV with the alias you assigned to your BOSH Director:

    $ bosh -e MY-ENV -d garden-windows
    Using environment 'DIRECTOR-IP' as client 'admin'

    Name Release(s) Stemcell(s) Team(s) Cloud Config garden-windows ... ... - latest

  4. Retrieve the Administrator password for your Windows cell by following the steps for your IaaS:

    • On vSphere, this is the value of WINDOWS_PASSWORD in the consumer-vars.yml file you used to previously build a stemcell.
    • On Amazon Web Services (AWS), navigate to the AWS EC2 console. Right-click on your Windows cell and select Get Windows Password from the drop-down menu. Provide the local path to the ops_mgr.pem private key file you used when installing Ops Manager and click Decrypt password to obtain the Administrator password for your Windows cell.
    • On Google Cloud Platform (GCP), navigate to the Compute Engine Dashboard. Under VM Instances, select the instance of the Windows VM. At the top of the page, click on Create or reset Windows password. When prompted, enter “Administrator” under Username and click Set. You will receive a one-time password for the Windows cell.
    • You cannot RDP into Windows cells on Azure.
  5. Open your RDP client. The examples below use the Microsoft Remote Desktop app.

  6. Click New and enter your connection information: Rdp connect

    • Connection name: Enter a name for this connection.
    • PC name: Enter the IP address of your Windows cell.
    • User name: Enter Administrator.
    • Password: Enter the password of your Windows cell that you obtained above.
  7. To mount a directory on your local machine as a drive in the Windows cell, perform the following steps:

    1. From the same Edit Remote Desktops window as above, click Redirection.
    2. Click the plus icon at the bottom left. Rdp redirection
    3. For Name, enter the name of the drive as it will appear in the Windows cell. For Path, enter the path of the local directory.
    4. Click OK.
  8. Close the Edit Remote Desktops window and double-click the newly added connection under My Desktops to open a RDP connection to the Windows cell.

  9. In the RDP session, you can use the Consul CLI to diagnose problems with your Windows cell.

Consul CLI

Perform the following steps to use the Consul CLI on your Windows cell to diagnose problems with your Consul cluster:

  1. In your RDP session, open a PowerShell window.
  2. Change into the directory that contains the Consul CLI binary:
    PS C:\Users\Administrator> cd C:\var\vcap\packages\consul-windows\bin\
    
  3. Use the Consul CLI to list the members of your Consul cluster:
    PS C:\Users\Administrator\var\vcap\packages\consul-windows\bin> .\consul.exe members
    Node                       Address          Status  Type    Build  Protocol  DC
    cell-windows-0             10.0.0.111:8301  alive   client  0.6.4  2         dc1
    cloud-controller-0         10.0.0.94:8301   alive   client  0.6.4  2         dc1
    cloud-controller-worker-0  10.0.0.99:8301   alive   client  0.6.4  2         dc1
    consul-server-0            10.0.0.96:8301   alive   server  0.6.4  2         dc1
    diego-brain-0              10.0.0.109:8301  alive   client  0.6.4  2         dc1
    diego-cell-0               10.0.0.103:8301  alive   client  0.6.4  2         dc1
    diego-cell-1               10.0.0.104:8301  alive   client  0.6.4  2         dc1
    diego-cell-2               10.0.0.107:8301  alive   client  0.6.4  2         dc1
    diego-database-0           10.0.0.92:8301   alive   client  0.6.4  2         dc1
    ha-proxy-0                 10.0.0.254:8301  alive   client  0.6.4  2         dc1
    nfs-server-0               10.0.0.100:8301  alive   client  0.6.4  2         dc1
    router-0                   10.0.0.105:8301  alive   client  0.6.4  2         dc1
    uaa-0                      10.0.0.93:8301   alive   client  0.6.4  2         dc1
    
  4. Examine the output to ensure that the cell-windows-0 service is registered in the Consul cluster and is alive. Otherwise, your Windows cell cannot communicate with your PCF deployment and developers cannot push .NET apps to the Windows cell. Check the configuration of your Consul cluster, and ensure that your certificates are not missing or misconfigured.
Create a pull request or raise an issue on the source for this page in GitHub