Creating a Windows Stemcell for vSphere Manually

This topic describes how to create the Windows stemcell Pivotal Application Service for Windows (PASW), and Enterprise Pivotal Container Service (Enterprise PKS) use to create VMs on vSphere.

Note: The instructions in this topic are based on vSphere 6.0 using vSphere Web Client.


A BOSH stemcell is a versioned operating system image. You must create a BOSH stemcell for Windows before you can deploy either PASW or PKS Windows worker-based clusters, on vSphere.

The BOSH stemcell that you create in this topic is based on Windows Server 2019.

To create a Windows stemcell for vSphere, you create a base Windows VM from a volume-licensed ISO and subsequently maintain that base template with all Windows recommended security updates, but without the BOSH dependencies.

This Windows VM, with security updates, serves as the base for all future stemcells produced from clones of that base VM. This enables you to build new stemcells without having to run Windows Updates from scratch each time. You can also use a “snapshot” feature to maintain an updated Windows image that does not contain the BOSH dependencies.

Pivotal recommends installing any available critical updates, and then rebuilding the stemcell from a clone of the original VM.


Before you create a vSphere Windows stemcell, you must have:

  • A Windows Server 2019 ISO, build number: 17763, from Microsoft Developer Network (MSDN) or Volume Licensing Service Center (VLSC). You can use an evaluation copy for testing, but Pivotal does not recommend an evaluation copy for production, as the licensing expires.

    Note: Pivotal recommends maintaining a separate, updated Windows VM based on this ISO to serve as the basis for the installation steps below. This enables you to apply Windows Updates and create new stemcells without having to reinstall all updates from scratch.

    Note: The Windows Server 2019 ISO must be a clean, base ISO file. A clean ISO file has no custom scripts or tooling. For example, the ISO must have no logging or antivirus tools installed.

  • A vSphere/vCenter account granted sufficient permissions to perform all of the following tasks:

    • Create a VM.
    • Configure a VM.
    • Open a VM in VM Remote Console on a local desktop.
    • Export a VM.
  • The ability to download/transfer files and software to a vCenter Windows VM.

Files on Local Machine

As part of completing the procedures in this topic, you download the following files to your local machine:

  • ovftool.

  • stembuild.

    • To build a Windows stemcell for PASW, use the latest release of stembuild.
    • To build a Windows stemcell for PKS, use the vSphere stembuild CLI for Windows version 2019.7.

      Note: You must choose a stemcell version to build. Stemcells are versioned as MAJOR.MINOR, such as 2019.9.

      For more information about 2019 stemcells, see the Stemcell v2019.x (Windows Server 2019) Release Notes.

Files on Windows VM

As part of completing the procedures in this topic, download the following files to your Windows VM:

Step 1: Create Base VM for Stemcell

This section describes how to create, configure, and verify a base Windows VM from a volume-licensed ISO.

Upload the Windows ISO

To upload the Windows ISO:

  1. Log in to vCenter.

  2. Click Storage in the vCenter menu.

  3. Choose a datastore and click or create the directory where you want the Windows ISO.

  4. Click Upload a file to datastore, and upload the Windows ISO.

    Note: You might need to install the vSphere client web plugin to upload through your browser, or scp the file directly to the datastore server. For more information, see the VMware vSphere documentation.

Create and Customize a New VM

To create and customize a new VM:

  1. In the vSphere client, click the VMs and Templates view to display the inventory objects.

  2. Right-click an object and select New Virtual Machine > New Virtual Machine….

  3. On the Select a creation type page, select Create a new virtual machine and click Next. New vm

  4. On the Select a name and folder page:

    1. Enter a name for the VM.
    2. Select a location for the VM.
    3. Click Next.
  5. On the Select a compute resource page, select a compute resource to run the VM and click Next.

  6. On the Select storage page:

    1. Select a VM Storage Policy.
    2. Select the destination datastore for the VM configuration files and virtual disks.
    3. Click Next.
  7. On the Select compatibility page, for the Compatible with configuration setting, select ESXi 6.0 and later and click Next.

  8. On the Select a guest OS page:

    1. For Guest OS Family, select Windows.
    2. For Guest OS Version, select Microsoft Windows Server 2016.
    3. Click Next.
  9. On the Customize hardware page, configure the VM hardware and click Next. When configuring the VM hardware, select the following settings for New Hard disk and New CD\DVD Drive:

    1. For New Hard disk, specify 30 GB or greater.
    2. For New CD\DVD Drive, perform the following steps:
      1. Select Datastore ISO File.
      2. Select the ISO file you uploaded to your datastore and click OK.
      3. Enable the Connect At Power On checkbox.
  10. Review the configuration settings on the Ready to complete page and click Finish.

Install Windows Server

To install Windows Server on the base VM:

  1. After creating the VM, click Power On in the Actions tab for your VM. Power on

  2. Select Windows Server Standard.

  3. Select Custom installation.

  4. Complete the installation process, and enter a password for the Administrator user. BOSH later randomizes this password.

Verify OS

Warning: You must complete the following procedure to verify your OS version before continuing.

Ensure you are using the correct the OS version by running the following PowerShell command on the Windows VM:

Get-CimInstance Win32_OperatingSystem | Select-Object
Caption, Version, ServicePackMajorVersion, OSArchitecture, CSName, WindowsDirectory

The output includes Version: 10.0.17763.

Install VMware Tools

To install VMware Tools on the base VM:

  1. Under the VM Summary tab, select Install VMware Tools.

  2. Navigate to the D: drive and run setup64.exe.

    Note: The VMware Tools install window might appear behind the Command Prompt window.

  3. Restart the VM as required to finish the install.

Step 2: Install Windows Updates

This section describes how to install Windows updates on your base Windows VM.

Install Windows Updates

Install Windows updates on the Windows VM using your preferred procedure.

One way to install Windows updates on the Windows VM is by using the SConfig utility:

  1. On the Windows VM, run the SConfig utility.

  2. Select option number 6, Download and Install Updates.

  3. Select A for (A)ll updates.

  4. For the Select an option, select (A)ll updates.

You might need to restart the Windows VM while installing updates.

Enable Meltdown Mitigation

Warning: You must enable Meltdown mitigation. Not enabling Meltdown mitigation can lead to timeout issues while deploying your PASW or PKS tile.

Windows Server 2019 should receive the update containing the Meltdown mitigation automatically when you install Windows updates.

After installing Windows update, ensure that the following registry keys are set to enable Meltdown mitigation:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management"
/v FeatureSettingsOverride /t REG_DWORD /d 0 /f

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management"
/v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization"
/v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f

reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\QualityCompat"
/f /v cadca5fe-87d3-4b96-b7fb-a231484277cc /t REG_DWORD /d 0

Step 3: Clone the VM

Clone the VM that has the Windows updates installed. Save the original VM so that you can run updates on it in the future.

To clone the VM:

  1. In the vSphere client, right-click the current Windows VM.

  2. Select Clone > Clone to Virtual Machine…. Clone vm

  3. Ensure that you can create the VM that can be used to create a stemcell for the next Patch Tuesday Monthly Updates.

Step 4: Install Required Software

You might need to specify an explicit execution policy for all of the PowerShell commands in the Step 4: Install Required Software section. You specify an execution policy with the -ExecutionPolicy flag.

For example:

powershell -ExecutionPolicy Bypass -Command "Install-CFFeatures"

Transfer Files to a Windows VM

Some of the procedures described in the sections below require transferring files to a Windows VM. Many different methods exist to transfer files to a Windows VM, such as folder sharing or the PowerShell Invoke-WebRequest cmdlet. Use whatever method you prefer.

As an example, the following PowerShell Invoke-WebRequest command uses TLS v1.2 to transfer from EXAMPLE-URL to the current location on the Windows VM:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest -Uri "EXAMPLE-URL/" -OutFile ".\"

Install the BOSH PS Modules

To install the BOSH PS Modules:

  1. Locate the BOSH PS Modules download for the 2019 stemcell version you want to build, such as 2019.2.

  2. Transfer the file to your Windows VM.

  3. Start PowerShell in the Windows VM and run the following command:


    Where PATH-TO-BOSH-PSMODULES.ZIP is the full path to the location of on your Windows VM.

  4. Unzip the archive with the following command:

    Expand-Archive PATH-TO-BOSH-PSMODULES.ZIP C:\Program Files\WindowsPowerShell\Modules

Install the Cloud Foundry Diego Cell Requirements

To install the Cloud Foundry Diego cell requirements:

  1. Start PowerShell in the Windows VM and run the following command:


    The machine restarts automatically.

  2. Apply the recommended ingress and service configuration:


Install the BOSH Agent

To install the BOSH Agent:

  1. Locate the BOSH Agent download for the 2019 stemcell version you want to build, such as 2019.9.

  2. Transfer the file to your Windows VM.

  3. Start PowerShell in the Windows VM and run the following command:

    Unblock-File PATH-TO-AGENT.ZIP

    Where PATH-TO-AGENT.ZIP is the full path to the location of the file on your Windows VM.

  4. Install the BOSH Agent:

    Install-Agent -IaaS vsphere -agentZipPath PATH-TO-AGENT.ZIP

Install OpenSSH

You can use the bosh ssh command on BOSH-deployed Windows VMs if you install the OpenSSH dependency on the Windows VM and then enable it during deploy time. This lets an operator enter into a CMD or PowerShell session on the VM as a user with admin privileges.

  • To install OpenSSH for PASW:

    1. Transfer the file to the Windows VM and place it in C:\provision.
    2. Start PowerShell in the Windows VM and run the following command: Unblock-File 'C:\provision\'
    3. Install OpenSSH with the following command: Install-SSHD -SSHZipFile 'C:\provision\'
    4. When configuring the PAS for Windows tile, you must select the Enable BOSH-native SSH support on all VMs (beta) checkbox. For more information, see the Configure the Tile section of the Installing and Configuring PASW topic.
  • To install OpenSSH for PKS Windows clusters:

    1. Transfer the file to the Windows VM and place it in C:\provision.
    2. Start PowerShell in the Windows VM and run the following command: Unblock-File 'C:\provision\'
    3. Install OpenSSH: Install-SSHD -SSHZipFile 'C:\provision\'

Optimize and Compress the Disk

Note: Windows Server stemcells can be large, and can exceed the 10 GB upload limit imposed by default by the BOSH Director.

To reduce the stemcell size:

  1. Restart the VM.

  2. Start PowerShell in the Windows VM and run the following command to use dism to clear unnecessary files:

  3. Run the following command to defragment and zero out the disk:


Step 5: Sysprep the System

This step “syspreps” the system, which ensures that each BOSH VM has a unique identity and applies the appropriate startup configuration at boot time.

The included policies help ensure the uptime and secure operations of the stemcell’s VMs, especially when deployed on Pivotal Platform.

Note: This step disables services that could cause restarts, such as Windows Automatic Updates. OS restarts are not supported on BOSH-deployed Windows VMs, and the BOSH Director resurrects the VM by destroying and repaving it.

To sysprep the system:

  1. Transfer the LGPO.ZIP file to the Windows VM.

  2. Start PowerShell in the Windows VM and run the following command:

    Expand-Archive PATH-TO-LGPO.ZIP C:\Windows
  3. Sysprep the system:

    Invoke-Sysprep -IaaS vsphere
    [-NewPassword PASSWORD]
    [-Owner OWNER] [-Organization ORGANIZATION]

    Note: All of the flags of Invoke-Sysprep except for -IaaS are optional.


    • PASSWORD is an optional flag that enables you to set a password of your choice. Do not use any special character in the password other than !. For example, Example12! is permitted but Example#12 is not. This is a known issue.
    • OWNER and ORGANIZATION are optional flags. Set them if your organization requires it.

      The sysprep command powers off the VM.

Warning: Do not turn the VM back on before completing the procedure in Step 6: Export the VMDK File.

Step 6: Export the VMDK File

Export the .VMDK file associated with the VM you powered off:

  1. In vCenter, right-click the VM and select Template > Export to OVF Template.

  2. Download the OVA to your local machine. You do not need to include files in the floppy or CD Drive.

    Note: You can also download the standalone vSphere client and select File > Export > Export OVF Template.

  3. Rename the downloaded OVA file to have a .tar extension.

  4. Expand the TAR archive and locate the VMDK file.

Step 7: Convert the VMDK File to a BOSH Stemcell

Note: This final step typically takes about ten to twenty minutes to complete.

To convert the VMDK file to a BOSH stemcell:

  1. Download the latest release of the stembuild utility to your local machine and place the executable in your command-line path.

  2. Download ovftool to your local machine and place the executable in your command-line path.

    Note: On the Windows desktop, ovftool is installed by default in C:\Program Files\VMware\VMware OVF Tool.

    stembuild invokes ovftool to convert the disk image to the appropriate stemcell format and apply the proper configuration.

  3. Build the stemcell:

    stembuild package -vmdk PATH-TO-VDMK -stemcell-version STEMCELL-VERSION -os 2019


    • PATH-TO-VMDK is the path to the VMDK file.
    • STEMCELL-VERSION is the 2019 stemcell version you want to build. For example, if you downloaded the BOSH PS Modules and BOSH Agent for the 2019.2 release, then specify 2019.2. stembuild creates the stemcell in the directory where you execute it. The file has a .tgz extension and a name similar to bosh-stemcell-2019.2-vsphere-esxi-windows2019-go_agent.tgz

      The stemcell is ready for use in conjunction with your BOSH deployment.

Step 8: Apply Monthly Patch Tuesday Updates

On Patch Tuesday, run Windows Updates on the base image, and then repeat Step 3: Clone the VM through Step 7: Convert the VMDK File to a BOSH Stemcell.


Garden Windows Logs Suggest Windows Features Not Installed


You see the following error in your garden-windows job while deploying Windows Server 2019:

Missing required Windows Features:
Web-Webserver, Web-WebSockets, AS-Web-Support,
AS-NET-Framework, Web-WHC, Web-ASP.
Please use the most recent stemcell.


Install-CFFeatures might not have run successfully.


Run the following commands in PowerShell on your Windows VM to verify whether Install-CFFeatures ran successfully:

Get-WindowsFeature "Containers" | Where InstallState -Eq "Installed"
Get-WindowsFeature "Windows-Defender-Features" | Where InstallState -Eq "Removed"