Creating a Windows Stemcell for vSphere Manually

Warning: Pivotal Cloud Foundry (PCF) v2.5 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes how to create the stemcell that Pivotal Application Service for Windows needs to create VMs on vSphere.

Note: The instructions in this topic are based on vSphere 6.0 using vSphere Web Client.

Overview

To create a Windows stemcell for vSphere, you create a base Windows VM from a volume-licensed ISO and subsequently maintain that base template with all Windows recommended security updates, but without the BOSH dependencies.

Note: The stemcell you create in this topic is based on Windows Server 2019.

The VM with security updates serves as the base for all future stemcells, produced from clones of that base VM. This enables you to build new stemcells without having to run Windows Updates from scratch each time. You can also use a “snapshot” feature to maintain an updated Windows image that does not contain the BOSH dependencies.

Pivotal recommends installing any available critical updates, and then rebuilding the stemcell from a clone of the original VM.

Prerequisites

Before you create a vSphere Windows stemcell, you must have:

A vSphere environment. To ensure the VM hardware used by the stemcell is compatible with your deployment environment’s ESXi/ESX host and vCenter Server versions, see ESXi/ESX hosts and compatible virtual machine hardware versions list (2007240) in the VMware Knowledge Base.
A Windows Server 2019 ISO, build number: 17763, from Microsoft Developer Network (MSDN) or Volume Licensing Service Center (VLSC). The Windows Server 2019 ISO must be a clean, base ISO file. You can use an evaluation copy for testing, but Pivotal does not recommend an evaluation copy for production, because the licensing expires. For more information, see the Microsoft documentation or the Microsoft Volume Licensing Service Center website.
Note: A clean ISO file has no custom scripts or tooling. For example, the ISO must have no logging or antivirus tools installed.

Note: Pivotal recommends maintaining a separate, updated Windows VM based on this ISO to serve as the basis for the installation steps below. This enables you to apply Windows Updates and create new stemcells without having to reinstall all updates from scratch.
A vSphere/vCenter account granted sufficient permissions to perform all of the following tasks:
- Create a VM.
- Configure a VM.
- Open a VM in VM Remote Console on a local desktop.
- Export a VM.
The ability to download and transfer files and software to a vCenter Windows VM.

Files on Local Machine

As part of completing the procedures in this topic, you download the following files to your local machine:

The latest release of stembuild.
ovftool.

Files on Windows VM

As part of completing the procedures in this topic, you download the following files to your Windows VM:

lgpo.exe from the Microsoft Security Toolkit.
OpenSSH v7.7.2.0p1.
The BOSH PS Modules and BOSH Agent for the 2019 stemcell version you want to build.
Note: You must choose a stemcell version to build. Stemcells are versioned as MAJOR.MINOR, such as 2019.2. For more information about 2019 stemcells, see Stemcell v2019.x (Windows Server 2019) Release Notes.

Step 1: Create Base VM for Stemcell

This section describes how to create, configure, and verify a base Windows VM from a volume-licensed ISO.

Upload the Windows ISO

To upload the Windows ISO:

Log in to vCenter.
Click Storage in the vCenter menu.
Choose a datastore and click or create the directory where you want the Windows ISO.
Click Upload a file to datastore, and upload the Windows ISO.
Note: You might need to install the vSphere client web plugin to upload through your browser, or scp the file directly to the datastore server. For more information, see the VMware vSphere documentation.

Create and Customize a New VM

To create and customize a new VM:

In the vSphere client, click the VMs and Templates view to display the inventory objects.
Right-click an object and select New Virtual Machine > New Virtual Machine….
On the Select a creation type page, select Create a new virtual machine and click Next.
On the Select a name and folder page:
1. Enter a name for the VM.
2. Select a location for the VM.
3. Click Next.
On the Select a compute resource page, select a compute resource to run the VM and click Next.
On the Select storage page:
1. Select a VM Storage Policy.
2. Select the destination datastore for the VM configuration files and virtual disks.
3. Click Next.
On the Select compatibility page, for the Compatible with configuration setting, select ESXi 6.0 and later and click Next.
On the Select a guest OS page:
1. For Guest OS Family, select Windows.
2. For Guest OS Version, select Microsoft Windows Server 2016.
3. Click Next.
On the Customize hardware page, configure the VM hardware and click Next. When configuring the VM hardware, select the following settings for New Hard disk and New CD\DVD Drive:
1. For New Hard disk, specify 30 GB or greater.
2. For New CD\DVD Drive:
  1. Select Datastore ISO File.
  2. Select the ISO file you uploaded to your datastore and click OK.
  3. Enable the Connect At Power On checkbox.
Review the configuration settings on the Ready to complete page and click Finish.

Install Windows Server

To install Windows Server on the base VM:

After creating the VM, click Power On in the Actions tab for your VM.
Select Windows Server Standard.
Select Custom installation.
Complete the installation process, and enter a password for the administrator user. BOSH later randomizes this password.

Verify OS

Warning: You must complete the following procedure to verify your OS version before continuing.

To verify the OS:

Ensure you are using the correct the OS version by running the following PowerShell command on the Windows VM:

Get-CimInstance Win32_OperatingSystem | Select-Object
Caption, Version, ServicePackMajorVersion, OSArchitecture, CSName, WindowsDirectory

Confirm that the output includes:
```
Version: 10.0.17763
```

Install VMware Tools

To install VMware Tools on the base VM:

Under the VM Summary tab, select Install VMware Tools.
Navigate to the D: drive and run:
```
setup64.exe
```
Note: The VMware Tools install window might appear behind the Command Prompt window.
Restart the VM to complete the installation.

Step 2: Install Windows Updates

This section describes how to install Windows updates on your base Windows VM.

Install Windows Updates

Install Windows updates on the Windows VM using your preferred procedure.

One way to install Windows updates on the Windows VM is by using the SConfig utility. To use the SConfig utility:

On the Windows VM, run the SConfig utility.
Select option number 6, Download and Install Updates.
Select A for (A)ll updates.
For the Select an option, select (A)ll updates.

You might need to restart the Windows VM while installing updates.

Enable Meltdown Mitigation

Warning: You must enable Meltdown mitigation. Not enabling Meltdown mitigation can lead to timeout issues while deploying your PASW tile.

Windows Server 2019 should receive the update containing the Meltdown mitigation automatically when you install Windows updates.

After installing Windows update, ensure that the following registry keys are set to enable Meltdown mitigation:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management"
/v FeatureSettingsOverride /t REG_DWORD /d 0 /f

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management"
/v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization"
/v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f

reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\QualityCompat"
/f /v cadca5fe-87d3-4b96-b7fb-a231484277cc /t REG_DWORD /d 0

Step 3: Clone the VM

You must clone the VM that has the Windows updates installed. You should save the original VM so that you can run updates on it in the future.

To clone the VM:

In the vSphere client, right-click the current Windows VM.
Select Clone > Clone to Virtual Machine….
Ensure that you can create the VM that can be used to create a stemcell for the next Patch Tuesday Monthly Updates.

Step 4: Install Required Software

You might need to specify an explicit execution policy for all of the PowerShell commands in this section. You specify an execution policy with the -ExecutionPolicy flag. For example:

powershell -ExecutionPolicy Bypass -Command "Install-CFFeatures"

Transfer Files to a Windows VM

Some of the procedures described in the sections below require transferring files to a Windows VM. Many different methods exist to transfer files to a Windows VM, such as folder sharing or the PowerShell Invoke-WebRequest cmdlet. Use whichever method you prefer.

As an example, run the PowerShell Invoke-WebRequest command below to use TLS v1.2 to transfer filename.zip from EXAMPLE-URL to the current location on the Windows VM:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest -Uri "EXAMPLE-URL/filename.zip" -OutFile ".\filename.zip"

Install the BOSH PS Modules

To install the BOSH PS Modules:

Locate the BOSH PS Modules download for the 2019 stemcell version you want to build, such as 2019.2.
Transfer the bosh-psmodules.zip file to your Windows VM.
Start PowerShell in the Windows VM and run:
```
Unblock-File PATH-TO-BOSH-PSMODULES.ZIP
```
Where PATH-TO-BOSH-PSMODULES.ZIP is the full path to the location of bosh-psmodules.zip on your Windows VM.
Unzip the archive by running:
```
Expand-Archive PATH-TO-BOSH-PSMODULES.ZIP C:\Program Files\WindowsPowerShell\Modules
```
Where PATH-TO-BOSH-PSMODULES.ZIP is the full path to the location of bosh-psmodules.zip on your Windows VM.

Install the Pivotal Cloud Foundry (PCF) Diego Cell Requirements

To install the PCF Diego Cell requirements:

Start PowerShell in the Windows VM and run:
```
Install-CFFeatures
```
The machine restarts automatically.
Apply the recommended ingress and service configuration by running:
```
Protect-CFCell
```

Install the BOSH Agent

To install the BOSH Agent:

Locate the BOSH Agent download for the 2019 stemcell version you want to build, such as 2019.9.
Transfer the agent.zip file to your Windows VM.
Start PowerShell in the Windows VM and run:
```
Unblock-File PATH-TO-AGENT.ZIP
```
Where PATH-TO-AGENT.ZIP is the full path to the location of the agent.zip file on your Windows VM.
Install the BOSH Agent by running:
```
Install-Agent -IaaS vsphere -agentZipPath PATH-TO-AGENT.ZIP
```
Where PATH-TO-AGENT.ZIP is the full path to the location of the agent.zip file on your Windows VM.

Install OpenSSH

You can use the bosh ssh command on BOSH-deployed Windows VMs if you install the OpenSSH dependency on the Windows VM and then enable it during deploy time. This lets an operator enter into a CMD or PowerShell session on the VM as a user with admin privileges.

Perform the following steps to install OpenSSH:

Transfer the OpenSSH-Win64.zip file to the Windows VM and place it in C:\provision.
Start PowerShell in the Windows VM and run the following command:
```
Unblock-File 'C:\provision\OpenSSH-Win64.zip'
```

Install OpenSSH with the following command:

Install-SSHD -SSHZipFile 'C:\provision\OpenSSH-Win64.zip'

When configuring the PAS for Windows tile, you must select the BETA: Enable BOSH-native SSH support on all VMs checkbox. For more information, see Installing and Configuring PAS for Windows.

Optimize and Compress the Disk

Note: Windows Server stemcells can be large, and can exceed the 10 GB upload limit imposed by default by the BOSH Director.

To reduce the stemcell size:

Restart the VM.
Start PowerShell in the Windows VM and use dism to clear unnecessary files by running:
```
Optimize-Disk
```
Defragment and zero out the disk by running:
```
Compress-Disk
```

Step 5: Sysprep the System

This step “syspreps” the system, which ensures that each BOSH VM has a unique identity and applies the appropriate startup configuration at boot time.

The included policies help ensure the uptime and secure operations of the stemcell’s VMs, especially when deployed on PCF.

Note: This step disables services that could cause restarts, such as Windows Automatic Updates. OS restarts are not supported on BOSH-deployed Windows VMs, and the BOSH Director resurrects the VM by destroying and repaving it.

To sysprep the system:

Transfer the LGPO.ZIP file to the Windows VM.
Start PowerShell in the Windows VM and run:
```
Expand-Archive PATH-TO-LGPO.ZIP C:\Windows
```
Where PATH-TO-LGPO.ZIP is the full path to the location of the LGPO.ZIP file on your Windows VM.
Sysprep the system by running:
```
Invoke-Sysprep -IaaS vsphere
[-NewPassword PASSWORD]
[-Owner OWNER] [-Organization ORGANIZATION]
```
Note: All of the flags of Invoke-Sysprep except for -IaaS are optional.

Where:
- PASSWORD is an optional flag that enables you to set a password of your choice. Do not use any special character in the password other than !. For example, Example12! is permitted, but Example#12 is not. This is a known issue.
- OWNER and ORGANIZATION are optional flags. Set them if your organization requires it.
  
  The sysrep command powers off the VM.

Warning: Do not turn the VM back on before completing the procedure in Step 6: Export the VMDK File.

Step 6: Export the VMDK File

To export the .VMDK file associated with the VM you powered off:

In vCenter, right-click the VM and select Template > Export to OVF Template.
Download the OVA to your local machine. You do not need to include files in the floppy or CD drive.
Note: You can also download the standalone vSphere client and select File > Export > Export OVF Template.
Rename the downloaded OVA file to have a .tar extension.
Expand the TAR archive and locate the VMDK file.

Step 7: Convert the VMDK File to a BOSH Stemcell

Note: This final step typically takes about ten to twenty minutes to complete.

To convert the VMDK file to a BOSH stemcell:

Download the latest release of the stembuild utility to your local machine and place the executable in your command-line path.
Download ovftool to your local machine and place the executable in your command-line path.
Note: On the Windows desktop, ovftool is installed by default in C:\Program Files\VMware\VMware OVF Tool.
stembuild invokes ovftool to convert the disk image to the appropriate stemcell format and apply the proper configuration.
Build the stemcell by running:
```
stembuild package -vmdk PATH-TO-VDMK -stemcell-version STEMCELL-VERSION -os 2019
```
Where:
- PATH-TO-VMDK is the path to the VMDK file.
- STEMCELL-VERSION is the 2019 stemcell version you want to build. For example, if you downloaded the BOSH PS Modules and BOSH Agent for the 2019.2 release, then specify 2019.2. stembuild creates the stemcell in the directory where you execute it. The file has a .tgz extension and a name similar to bosh-stemcell-2019.2-vsphere-esxi-windows2019-go_agent.tgz .

The stemcell is ready for use in conjunction with your BOSH deployment.

Step 8: Apply Monthly Patch Tuesday Updates

On Patch Tuesday, run Windows updates on the base image, and then repeat Step 3: Clone the VM through Step 7: Convert the VMDK File to a BOSH Stemcell.

Troubleshooting

Garden Windows Logs Suggest Windows Features Not Installed

Symptom

You see the following error in your garden-windows job while deploying Windows Server 2019:

Missing required Windows Features:
Web-Webserver, Web-WebSockets, AS-Web-Support,
AS-NET-Framework, Web-WHC, Web-ASP.
Please use the most recent stemcell.

Explanation

Install-CFFeatures might not have run successfully.

Solution

Run the following commands in PowerShell on your Windows VM to verify whether Install-CFFeatures ran successfully:

Get-WindowsFeature "Containers" | Where InstallState -Eq "Installed"
Get-WindowsFeature "Windows-Defender-Features" | Where InstallState -Eq "Removed"

Create a pull request or raise an issue on the source for this page in GitHub