Creating a vSphere Windows Stemcell

This topic describes how to create the stemcell that Pivotal Application Service for Windows (PASW) needs to create VMs on vSphere.

Note: The instructions in this topic are based on vSphere 6.0 using vSphere Web Client.

Overview

To create a Windows stemcell for vSphere, you create a base Windows VM from a volume-licensed ISO and subsequently maintain that base template with all Windows recommended security updates, but without the BOSH dependencies.

Note: The stemcell you create in this topic is based on Windows Server, version 1803.

The VM with security updates serves as the base for all future stemcells, produced from clones of that base VM. This enables you to build new stemcells without having to run Windows Updates from scratch each time. You can also use a “snapshot” feature to maintain an updated Windows image that does not contain the BOSH dependencies.

Pivotal recommends installing any available critical updates, and then rebuilding the stemcell from a clone of the original VM.

Prerequisites

Before you create a vSphere Windows stemcell, you must have the following:

  • A Windows Server, version 1803 ISO, from Microsoft Developer Network (MSDN) or Volume Licensing Service Center (VLSC). You can use an evaluation copy for testing, but Pivotal does not recommend an evaluation copy for production, as the licensing expires.

    Note: Pivotal recommends maintaining a separate, updated Windows VM based on this ISO to serve as the basis for the installation steps below. This enables you to apply Windows Updates and create new stemcells without having to reinstall all updates from scratch.

  • A vSphere/vCenter account granted sufficient permissions to perform all of the following tasks:

    • Create a VM.
    • Configure a VM.
    • Open a VM in VM Remote Console on a local desktop.
    • Export a VM.
  • The ability to download/transfer files and software to a vCenter Windows VM.

Files on Local Machine

As part of completing the procedures in this topic, you download the following files to your local machine:

Files on Windows VM

As part of completing the procedures in this topic, you download the following files to your Windows VM:

Step 1: Create Base VM for Stemcell

This section describes how to create, configure, and verify a base Windows VM from a volume-licensed ISO.

Upload the Windows ISO

Perform the following steps to upload the Windows ISO:

  1. Log in to vCenter.
  2. Click Storage in the vCenter menu.
  3. Choose a datastore and click or create the directory where you want the Windows ISO.
  4. Click Upload a file to datastore, and upload the Windows ISO.

    Note: You might need to install the vSphere client web plugin to upload through your browser, or scp the file directly to the datastore server. For more information, see the VMware vSphere documentation.

Create and Customize a New VM

Perform the following steps to create and customize a new VM:

  1. In the vSphere client, click the VMs and Templates view to display the inventory objects.
  2. Right-click an object and select New Virtual Machine > New Virtual Machine….
  3. On the Select a creation type page, select Create a new virtual machine and click Next. New vm
  4. On the Select a name and folder page, perform the following steps:
    1. Enter a name for the VM.
    2. Select a location for the VM.
    3. Click Next.
  5. On the Select a compute resource page, select a compute resource to run the VM and click Next.
  6. On the Select storage page, perform the following steps:
    1. Select a VM Storage Policy.
    2. Select the destination datastore for the VM configuration files and virtual disks.
    3. Click Next.
  7. On the Select compatibility page, for the Compatible with configuration setting, select ESXi 6.0 and later and click Next.
  8. On the Select a guest OS page, perform the following steps:
    1. For Guest OS Family, select Windows.
    2. For Guest OS Version, select Microsoft Windows Server 2016.
    3. Click Next.
  9. On the Customize hardware page, configure the VM hardware and click Next. When configuring the VM hardware, select the following settings for New Hard disk and New CD\DVD Drive:
    1. For New Hard disk, specify 30 GB or greater.
    2. For New CD\DVD Drive, perform the following steps:
      1. Select Datastore ISO File.
      2. Select the ISO file you uploaded to your datastore and click OK.
      3. Enable the Connect At Power On checkbox.
  10. Review the configuration settings on the Ready to complete page and click Finish.

Install Windows Server

Perform the following steps to install Windows Server on the base VM:

  1. After creating the VM, click Power On in the Actions tab for your VM. Power on
  2. Select Windows Server Standard.
  3. Select Custom installation.
  4. Complete the installation process, and enter a password for the Administrator user. BOSH later randomizes this password.

Verify OS

WARNING: You must complete the following procedure to verify your OS version before continuing.

Ensure you are using the correct the OS version by running the following PowerShell command on the Windows VM:

Get-CimInstance Win32_OperatingSystem | Select-Object
Caption, Version, ServicePackMajorVersion, OSArchitecture, CSName, WindowsDirectory

The output includes Version: 10.0.17134.

Install VMware Tools

Perform the following steps to install VMware Tools on the base VM:

  1. Under the VM Summary tab, select Install VMware Tools.
  2. Navigate to the D: drive and run setup64.exe.

    Note: The VMware Tools install window might appear behind the Command Prompt window.

  3. Restart the VM as required to finish the install.

Step 2: Install Windows Updates

This section describes how to install Windows updates on your base Windows VM.

Install Windows Updates

Install Windows updates on the Windows VM using your preferred procedure.

One way to install Windows updates on the Windows VM is by using the SConfig utility. Perform the following steps:

  1. On the Windows VM, run the SConfig utility.
  2. Select option number 6, Download and Install Updates.
  3. Select A for (A)ll updates.
  4. For the Select an option, select (A)ll updates.

You might need to restart the Windows VM while installing updates.

Enable Meltdown Mitigation

WARNING: You must enable Meltdown mitigation. Not enabling Meltdown mitigation can lead to timeout issues while deploying the PASW tile.

Windows Server, version 1803 should receive the update containing the Meltdown mitigation automatically when you install Windows updates.

After installing Windows update, ensure that the following registry keys are set to enable Meltdown mitigation:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management"
/v FeatureSettingsOverride /t REG_DWORD /d 0 /f

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management"
/v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization"
/v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f

reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\QualityCompat"
/f /v cadca5fe-87d3-4b96-b7fb-a231484277cc /t REG_DWORD /d 0

Step 3: Clone the VM

Clone the VM that has the Windows updates installed. Save the original VM so that you can run updates on it in the future.

Perform the following steps:

  1. In the vSphere client, right-click the current Windows VM.
  2. Select Clone > Clone to Virtual Machine…. Clone vm
  3. Ensure that you can create the VM that can be used to create a stemcell for the next Patch Tuesday Monthly Updates.

Step 4: Install Required Software

You might need to specify an explicit execution policy for all of the PowerShell commands in the Step 4: Install Required Software section. You specify an execution policy with the -ExecutionPolicy flag.

For example:

powershell -ExecutionPolicy Bypass -Command "Install-CFFeatures"

Transfer Files to a Windows VM

Some of the procedures described in the sections below require transferring files to a Windows VM. Many different methods exist to transfer files to a Windows VM, such as folder sharing or the PowerShell Invoke-WebRequest cmdlet. Use whatever method that you prefer.

As an example, the following PowerShell Invoke-WebRequest command uses TLS v1.2 to transfer filename.zip from EXAMPLE-URL to the current location on the Windows VM:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest -Uri "EXAMPLE-URL/filename.zip" -OutFile ".\filename.zip"

Install the BOSH PS Modules

Perform the following steps to install the BOSH PS Modules:

  1. Locate the BOSH PS Modules download for the 1803 stemcell version you want to build, such as 1803.2.
  2. Transfer the bosh-psmodules.zip file to your Windows VM.
  3. Start PowerShell in the Windows VM and run the following command:

    Unblock-File PATH-TO-BOSH-PSMODULES.ZIP
    

    Where PATH-TO-BOSH-PSMODULES.ZIP is the full path to the location of bosh-psmodules.zip on your Windows VM.

  4. Unzip the archive with the following command:

    Expand-Archive PATH-TO-BOSH-PSMODULES.ZIP C:\Program Files\WindowsPowerShell\Modules
    

Install the Cloud Foundry Diego Cell Requirements

Perform the following steps to install the Cloud Foundry Diego cell requirements:

  1. Start PowerShell in the Windows VM and run the following command:

    Install-CFFeatures
    

    The machine restarts automatically.

  2. Apply the recommended ingress and service configuration with the following command:

    Protect-CFCell
    

Install the BOSH Agent

Perform the following steps to install the BOSH Agent:

  1. Locate the BOSH Agent download for the 1803 stemcell version you want to build, such as 1803.2.
  2. Transfer the agent.zip file to your Windows VM.
  3. Start PowerShell in the Windows VM and run the following command:

    Unblock-File PATH-TO-AGENT.ZIP
    

    Where PATH-TO-AGENT.ZIP is the full path to the location of the agent.zip file on your Windows VM.

  4. Install the BOSH Agent with the following command:

    Install-Agent -IaaS vsphere -agentZipPath PATH-TO-AGENT.ZIP
    

Install OpenSSH

You can use the bosh ssh command on BOSH-deployed Windows VMs if you install the OpenSSH dependency on the Windows VM and then enable it during deploy time. This lets an operator enter into a CMD or PowerShell session on the VM as a user with admin privileges.

Perform the following steps to install OpenSSH:

  1. Transfer the OpenSSH-Win64.zip file to the Windows VM and place it in C:\provision.
  2. Start PowerShell in the Windows VM and run the following command:

    Unblock-File 'C:\provision\OpenSSH-Win64.zip'
    
  3. Install OpenSSH with the following command:

    Install-SSHD -SSHZipFile 'C:\provision\OpenSSH-Win64.zip'
    
  4. When configuring the PAS for Windows tile, you must select the BETA: Enable BOSH-native SSH support on all VMs checkbox. For more information, see Installing and Configuring PAS for Windows.

Optimize and Compress the Disk

Note: Windows Server stemcells can be large, and can exceed the 10GB upload limit imposed by default by the BOSH Director.

Perform the following steps to reduce the stemcell size:

  1. Restart the VM.
  2. Start PowerShell in the Windows VM and run the following command to use dism to clear unnecessary files:

    Optimize-Disk
    
  3. Run the following command to defragment and zero out the disk:

    Compress-Disk
    

Step 5: Sysprep the System

This step “syspreps” the system, which ensures that each BOSH VM has a unique identity and applies the appropriate startup configuration at boot time.

The included policies help ensure the uptime and secure operations of the stemcell’s VMs, especially when deployed on PCF.

Note: This step disables services that could cause restarts, such as Windows Automatic Updates. OS restarts are not supported on BOSH-deployed Windows VMs, and the BOSH Director resurrects the VM by destroying and repaving it.

Perform the following steps:

  1. Transfer the LGPO.ZIP file to the Windows VM.
  2. Start PowerShell in the Windows VM and run the following command:

    Expand-Archive PATH-TO-LGPO.ZIP C:\Windows
    
  3. Run the following command to sysrep the system:

    Invoke-Sysprep -IaaS vsphere
    [-NewPassword PASSWORD]
    [-Owner OWNER] [-Organization ORGANIZATION]
    

    Note: All of the flags of Invoke-Sysprep except for -IaaS are optional.

    Where:

    • PASSWORD is an optional flag that enables you to set a password of your choice. Do not use any special character in the password other than !. For example, Example12! is permitted but Example#12 is not. This is a known issue.
    • OWNER and ORGANIZATION are optional flags. Set them if your organization requires it.

      The sysrep command powers off the VM.

WARNING: Do not turn the VM back on before completing the procedure in Step 6: Export the VMDK File.

Step 6: Export the VMDK File

Perform the following steps to export the .VMDK file associated with the VM you powered off:

  1. In vCenter, right-click the VM and select Template > Export to OVF Template.
  2. Download the OVA to your local machine. You do not need to include files in the floppy or CD Drive.

    Note: You can also download the standalone vSphere client and select File > Export > Export OVF Template.

  3. Rename the downloaded OVA file to have a .tar extension.
  4. Expand the TAR archive and locate the VMDK file.

Step 7: Convert the VMDK File to a BOSH Stemcell

Note: This final step typically takes about ten to twenty minutes to complete.

Perform the following steps to convert the VMDK file to a BOSH stemcell:

  1. Download the latest release of the stembuild utility to your local machine and place the executable in your command-line path.
  2. Download ovftool to your local machine and place the executable in your command-line path.

    Note: On the Windows desktop, ovftool is installed by default in C:\Program Files\VMware\VMware OVF Tool.

    stembuild invokes ovftool to convert the disk image to the appropriate stemcell format and apply the proper configuration.

  3. Build the stemcell with the following command:

    stembuild package -vmdk PATH-TO-VDMK -stemcell-version STEMCELL-VERSION -os 1803
    

    Where:

    • PATH-TO-VMDK is the path to the VMDK file.
    • STEMCELL-VERSION is the 1803 stemcell version you want to build. For example, if you downloaded the BOSH PS Modules and BOSH Agent for the 1803.2 release, then specify 1803.2.

    stembuild creates the stemcell in the directory where you execute it. The file has a .tgz extension and a name similar to bosh-stemcell-1803.2-vsphere-esxi-windows2016-go_agent.tgz
.

    The stemcell is ready for use in conjunction with your BOSH deployment.

Step 8: Apply Monthly Patch Tuesday Updates

On Patch Tuesday, run Windows Updates on the base image, and then repeat Step 3: Clone the VM through Step 7: Convert the VMDK File to a BOSH Stemcell.

Troubleshooting

Garden Windows Logs Suggest Windows Features Not Installed

Symptom

You see the following error in your garden-windows job while deploying Windows 1803:

Missing required Windows Features:
Web-Webserver, Web-WebSockets, AS-Web-Support,
AS-NET-Framework, Web-WHC, Web-ASP.
Please use the most recent stemcell.

Explanation

Install-CFFeatures might not have run successfully.

Solution

Run the following commands in PowerShell on your Windows VM to verify whether Install-CFFeatures ran successfully:

Get-WindowsFeature "Containers" | Where InstallState -Eq "Installed"
Get-WindowsFeature "Windows-Defender-Features" | Where InstallState -Eq "Removed"
Create a pull request or raise an issue on the source for this page in GitHub