Creating a vSphere Windows Stemcell

This topic describes how to create the stemcell that Pivotal Application Service (PAS) for Windows needs to create VMs on vSphere.

The stemcell is based on Windows Server version 1709.

Overview

To create a Windows stemcell for vSphere, you create a base Windows VM from a volume-licensed ISO and subsequently maintain that base template with all Windows recommended security updates, but without the BOSH dependencies.

The VM with security updates will serve as the base for all future stemcells, produced from clones of that base VM. This enables you to build new stemcells without having to run Windows Updates from scratch each time. You can also use a “snapshot” feature to maintain an updated Windows image that does not contain the BOSH dependencies.

Pivotal recommends installing any available critical updates, and then rebuilding the stemcell from a clone of the original VM.

Prerequisites

You need the following to perform the procedures in this topic:

  • A Windows Server 1709 ISO, from MSDN or VLSC. You can use an evaluation copy for testing, but Pivotal does not recommend an evaluation copy for production, as the licensing expires.

    Note: Pivotal recommends maintaining a separate, updated Windows VM based on this ISO to serve as the basis for the installation steps below. This enables you to apply Windows Updates and create new stemcells without having to reinstall all updates from scratch.

  • Access to a vSphere account.

Files on Local Machine

As part of completing the procedures in this topic, you download the following files onto your local machine:

Files on Windows VM

As part of completing the procedures in this topic, you download the following files onto your Windows VM:

Step 1: Create Base VM for Stemcell

Upload the Windows ISO

Perform the following steps to upload the Windows ISO:

  1. Log in to vCenter.
  2. Click Storage in the vCenter menu.
  3. Choose a datastore and click on (or create) the directory where you want the Windows ISO.
  4. Click Upload a file to datastore, and upload the Windows ISO.

    Note: You may need to install the vSphere client web plugin to upload through your browser, or scp the file directly to the datastore server. For more information, see the VMware vSphere documentation.

Create and Customize a New VM

Perform the following steps to create and customize a new VM:

  1. If you are using an existing template, select the creation type Deploy from template and select a template.
  2. In Select compatibility, select ESXi 6.0 and later.
  3. For Guest OS Family, select Windows.
  4. For Guest OS version, select Windows Server 2016.
  5. In Customize hardware, perform the following steps:
    • Under New Hard disk, specify a size of 30GB or greater.
    • Under New CD\DVD Drive, perform the following steps:
      1. Select Datastore ISO File.
      2. Expand the menu and select Connect At Power On.
      3. Click Browse and select the ISO you uploaded to your datastore.

Install Windows Server

Perform the following steps to install Windows Server on the base VM:

  1. After creating the VM, click Power On in the Actions tab for your VM.
  2. Select Windows Server Standard.
  3. Select Custom installation.
  4. Complete the installation process, and enter a password for the Administrator user. BOSH will later randomize this password.

Verify OS

WARNING: You must complete the following procedure to verify your OS version before continuing.

Ensure you are using the correct the OS version by running the following PowerShell command on the Windows VM:

Get-CimInstance Win32_OperatingSystem | Select-Object 
Caption, Version, ServicePackMajorVersion, OSArchitecture, CSName, WindowsDirectory

The output should include Version: 10.0.16299.

Install VMware Tools

Perform the following steps to install VMware Tools on the base VM:

  1. Under the VM Summary tab, select Install VMware Tools.
  2. Navigate to the D: drive and run setup64.exe.

    Note: The VMWare Tools install window may appear behind the Command Prompt window.

  3. Restart the VM as required to finish the install.

Step 2: Install Windows Updates

Install Windows Updates

Install Windows updates on the Windows VM using your preferred procedure.

One way to install Windows updates on the Windows VM is by using the SConfig utility. Perform the following steps:

  1. On the Windows VM, run the SConfig utility.
  2. Select option number 6, Download and Install Updates.
  3. Select A for (A)ll updates.
  4. For the Select an option, select (A)ll updates.

You may need to reboot the Windows VM while installing updates.

Enable Meltdown Mitigation

WARNING: You must enable Meltdown mitigation. Not enabling Meltdown mitigation can lead to timeout issues while deploying the PASW tile.

Windows Server version 1709 should receive the update containing the Meltdown mitigation automatically when you install Windows updates.

After installing Windows update, ensure that the following registry keys are set to enable Meltdown mitigation:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" 
/v FeatureSettingsOverride /t REG_DWORD /d 0 /f

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" 
/v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization" 
/v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f

reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\QualityCompat" 
/f /v cadca5fe-87d3-4b96-b7fb-a231484277cc /t REG_DWORD /d 0

Step 3: Clone the VM

Clone the VM that has the Windows updates installed. Save the original VM so that you can run updates on it in the future.

Perform the following steps:

  1. In vCenter, right-click the current Windows VM
  2. Select Clone to Virtual Machine.
  3. Ensure that you can create the VM that can be used to create a stemcell for the next Patch Tuesday Monthly Updates.

Step 4: Install Required Software

You may need to specify an explicit execution policy for all of the PowerShell commands in the Step 4: Install Required Software section. You specify an execution policy with the -ExecutionPolicy flag.

For example:

powershell -ExecutionPolicy Bypass -Command "Install-CFFeatures"

Install the BOSH PS Modules

Perform the following steps to install the BOSH PS Modules:

  1. Locate the BOSH PS Modules download for the 1709 stemcell version you want to build, such as 1709.12.
  2. Transfer the bosh-psmodules.zip file to your Windows VM.

    Note: Use your preferred procedure for transferring files to your Windows VM. You may want to use folder sharing.

  3. Start PowerShell in the Windows VM and run the following command:

    Unblock-File PATH-TO-BOSH-PSMODULES.ZIP
    

    Where PATH-TO-BOSH-PSMODULES.ZIP is the full path to the location of bosh-psmodules.zip on your Windows VM.

  4. Unzip the archive with the following command:

    Expand-Archive PATH-TO-BOSH-PSMODULES.ZIP C:\Program Files\WindowsPowerShell\Modules
    

Install the Cloud Foundry Diego Cell Requirements

Perform the following steps to install the Cloud Foundry Diego Cell requirements:

  1. Start PowerShell in the Windows VM and run the following command:

    Install-CFFeatures
    

    The machine will restart automatically.

  2. Apply the recommended ingress and service configuration with the following command:

    Protect-CFCell
    

Install the BOSH Agent

Perform the following steps to install the BOSH Agent:

  1. Locate the BOSH Agent download for the 1709 stemcell version you want to build, such as 1709.12.
  2. Transfer the agent.zip file to your Windows VM.
  3. Start PowerShell in the Windows VM and run the following command:

    Unblock-File PATH-TO-AGENT.ZIP
    

    Where PATH-TO-AGENT.ZIP is the full path to the location of the agent.zip file on your Windows VM.

  4. Install the BOSH Agent with the following command:

    Install-Agent -IaaS vsphere -agentZipPath PATH-TO-AGENT.ZIP
    

Install OpenSSH

You can use the bosh ssh command on BOSH-deployed Windows VMs if you install the OpenSSH dependency on the Windows VM and then enable it during deploy time. This lets an operator enter into a CMD or PowerShell session on the VM as a user with administrator privileges.

Perform the following steps to install OpenSSH:

  1. Transfer the OpenSSH-Win64.zip file to the Windows VM and place it in C:\provision.
  2. Start PowerShell in the Windows VM and run the following command:

    Unblock-File 'C:\provision\OpenSSH-Win64.zip'
    
  3. Install OpenSSH with the following command:

    Install-SSHD -SSHZipFile 'C:\provision\OpenSSH-Win64.zip'
    
  4. When configuring the PAS for Windows tile, you must select the BETA: Enable BOSH-native SSH support on all VMs checkbox. For more information, see Installing and Configuring PAS for Windows.

Optimize and Compress the Disk

Note: Windows Server stemcells can be large, and can exceed the 10GB upload limit imposed by default by the BOSH Director.

Perform the following steps to reduce the stemcell size:

  1. Restart the VM.
  2. Start PowerShell in the Windows VM and run the following command to use dism to clear unnecessary files:

    Optimize-Disk
    
  3. Run the following command to defragment and zero out the disk:

    Compress-Disk
    

Step 5: Sysprep the System

This step “syspreps” the system, which ensures that each BOSH VM has a unique identity and applies the appropriate startup configuration at boot time.

The included policies help ensure the uptime and secure operations of the stemcell’s VMs, especially when deployed on PCF.

Note: This step disables services that could cause restarts, such as Windows Automatic Updates. OS restarts are not supported on BOSH-deployed Windows VMs, and the BOSH Director will “resurrect” the VM by destroying and repaving it.

Perform the following steps:

  1. Transfer the LGPO.ZIP file to the Windows VM.
  2. Start PowerShell in the Windows VM and run the following command:

    Expand-Archive PATH-TO-LGPO.ZIP C:\Windows
    
  3. Run the following command to sysrep the system:

    Invoke-Sysprep -IaaS vsphere 
    [-NewPassword PASSWORD]  
    [-Owner OWNER] [-Organization ORGANIZATION]
    

    Note: All of the flags of Invoke-Sysprep except for -IaaS are optional.

    Where:

    • PASSWORD is an optional flag that enables you to set a password of your choice. Do not use any special character in the password other than !. For example, Example12! is permitted but Example#12 is not. This is a known issue that will be fixed in future versions.
    • OWNER and ORGANIZATION are optional flags. Set them if your organization requires it.

      The sysrep command powers off the VM.

WARNING: Do not turn the VM back on before completing the procedure in Step 6: Export the VMDK File.

Step 6: Export the VMDK File

Perform the following steps to export the .VMDK file associated with the VM you powered off:

  1. In vCenter, right-click the VM and select Template > Export to OVF Template.
  2. Download the OVA to your local machine. You do not need to include files in the floppy or CD Drive.

    Note: You can also download the standalone vSphere client and select File > Export > Export OVF Template.

  3. Rename the downloaded OVA file to have a .tar extension.
  4. Expand the TAR archive and locate the VMDK file.

Step 7: Convert the VMDK File to a BOSH Stemcell

Note: This final step typically takes about ten to twenty minutes to complete.

Perform the following steps to convert the VMDK file to a BOSH stemcell:

  1. Download the latest release of the stembuild utility to your local machine and place the executable in your command-line path.
  2. Download ovftool to your local machine and place the executable in your command-line path.

    Note: On the Windows desktop, ovftool is installed by default in C:\Program Files\VMware\VMware OVF Tool.

    stembuild invokes ovftool to convert the disk image to the appropriate stemcell format and apply the proper configuration.

  3. Build the stemcell with the following command:

    stembuild -vmdk "PATH-TO-VMDK" -version YOUR-STEMCELL-VERSION -os 2016
    

    Where:

    • PATH-TO-VMDK is the path to the VMDK file.
    • YOUR-STEMCELL-VERSION is the 1709 stemcell version you want to build. For example, if you downloaded the BOSH PS Modules and BOSH Agent for the 1709.10 release, then specify 1709.10.

    stembuild will create the stemcell in the directory where you execute it. The file has a .tgz extension and a name similar to bosh-stemcell-1709.10-vsphere-esxi-windows2016-go_agent.tgz
.

    The stemcell is ready for use in conjunction with your BOSH deployment.

Step 8: Apply Monthly Patch Tuesday Updates

On Patch Tuesday, run Windows Updates on the base image, and then repeat Step 3: Clone the VM through Step 7: Convert the VMDK File to a BOSH Stemcell.

Troubleshooting

Garden Windows Logs Suggest Windows Features Not Installed

Symptom

You see the following error in your garden-windows job while deploying Windows 1709:

Missing required Windows Features: 
Web-Webserver, Web-WebSockets, AS-Web-Support, 
AS-NET-Framework, Web-WHC, Web-ASP.  
Please use the most recent stemcell.

Explanation

Install-CFFeatures may not have run successfully.

Solution

Run the following commands in PowerShell on your Windows VM to verify whether Install-CFFeatures ran successfully:

Get-WindowsFeature "Containers" | Where InstallState -Eq "Installed"
Get-WindowsFeature "Windows-Defender-Features" | Where InstallState -Eq "Removed"

Gorouter Returns a 502 Error When Accessing an App

Symptom

You cannot access a .NET app externally. When you attempt to access your app, Gorouter returns an HTTP 502 response status code. The app does not crash and is still running inside of its container.

Explanation

A race condition occurs when one app is deployed to a Diego cell and before it receives any traffic, a second app is deployed to the same Diego cell. As a result, the platform mistakenly removes your routing rules for the first app.

Solution

To resolve this issue, install Microsoft’s KB4074588 updates or later on your base Windows VM as described in Step 2: Install Windows Updates.

Once you install the updates and create your Windows stemcell for vSphere following the instructions in Step 3: Clone the VM through Step 7: Convert the VMDK File to a BOSH Stemcell, deploy the stemcell with PASW.

Create a pull request or raise an issue on the source for this page in GitHub