Creating a vSphere Stemcell Manually

This topic describes how to create a vSphere stemcell for Windows Server 2012R2 manually.

In vSphere, customers must build their own stemcells to use PCF Runtime for Windows in their on-premise deployments.

Overview

To build a vSphere stemcell, you start with an ISO of Windows Server 2012R2. Then you install the BOSH dependencies to create a .tgz file that can be uploaded to your BOSH Director and used with Cloud Foundry.

Considerations

The operator should maintain an updated template with all Windows recommended security updates. You can determine if your image needs updates by creating a VM with the image and navigating to Control Panel. If any critical or important updates are available, Pivotal recommends installing the updates first, then rebuilding the stemcell.

The instructions in this topic describe installing Windows from a Windows installation disk ISO, either a volume licensed copy or a retail copy from MSDN. Pivotal recommends maintaining a separate, updated Windows VM based on this ISO to serve as the basis for the installation steps below. This enables you to apply Windows Updates and create new stemcells without having to reinstall all updates from scratch. Ensure that your image has Hardware Compatibility set to version 9, and that it has VMware Tools installed.

On Patch Tuesday, run Windows Updates on the base image, then repeat the procedures in this topic starting at Step 3: Install BOSH PSModules.

Prerequisites

Component Notes
ovftool Ensure the ovftool command is available from your command line. This tool is installed by default in C:\Program Files\VMware\VMware OVF Tool.
Windows Server 2012 R2 ISO You can also use a custom base image.
VMware Fusion, VMware Workstation Pro v12+, or access to a vSphere account

Step 1: Create Base VM for Stemcell

You can create the base VM from VMware Fusion, VMware Workstation, or a vSphere account.

VMware Fusion

  1. Select File and then New.
  2. Select Installation Method: Create a custom virtual machine.
  3. Choose Operating System: Microsoft Windows, and then Windows Server 2012.
  4. Choose Virtual Disk: Create a new virtual disk. Use the default settings.
  5. Select Customize Settings. Save the VM before continuing. Use a name of your choosing.
  6. A Settings window will appear for your new VM. In the settings window, enter the following configurations:
    1. In the Removable Devices row, select CD/DVD, then click the Connect CD/DVD Drive checkbox.
    2. From This drive is configured to use the following:, select Choose a disc or disc image and specify your base Windows ISO.
    3. Click the Hard Disk settings.
    4. Click Advanced Options and deselect the Split into multiple files checkbox.
    5. Click Apply Changes.
  7. Start the VM. If you are starting from a new MSDN ISO, proceed through the Windows installation process with your ISO. Pivotal recommends that you select Windows Server 2012R2 Standard with a GUI when proceeding with installation.
  8. Install VMware Tools by performing the following steps:
    1. Click Virtual Machine.
    2. Click Install VMWare Tools.
    3. Click Complete Installation and follow the installation instructions.
    4. Restart the VM to finish the install.

For VMware Workstation

  1. Ensure VMware Workstation Pro v12+ is installed.
  2. Create a new VM by performing the following steps:
    1. Select Custom Advanced, then click Next.
    2. Select Workstation 9.x Compatibility, then click Next.
    3. Select I will install operating system later, then click Next.
    4. Select Microsoft Windows, then the Windows 2012 version. Click Next.
    5. Choose a name for your virtual machine, then click Next.
    6. Select BIOS, then click Next.
    7. Adjust the number of cores and processors as needed, then click Next.
    8. Adjust the settings for memory as needed, then click Next.
    9. Select the correct Network Type (NAT), then click Next.
    10. Select the LSI Logic SAS controller type, then click Next.
    11. Select the SCSI disk type, then click Next.
    12. Create a new virtual disk, then click Next.
    13. Adjust the maximum disk size as needed and select Store virtual disk as a single file, then click Next.
    14. Select Customize Hardware.
    15. Select New CD/DVD.
    16. Select Use ISO Image file and browse for your Windows ISO.
  3. Power on the new VM.
  4. Install Windows by performing the following steps:
    1. Select Server with GUI.
    2. Select Custom Installation.
    3. Continue the installation and select a password for the Admin user.
  5. After the VM starts successfully, right-click the machine name in Workstation and select Install VMware Tools. Restart the VM to finish the install.

For vCenter

  1. Upload your Windows ISO to your datastore by performing the following steps:

    1. Select Storage from the main vCenter menu.
    2. Select your desired datastore and directory.
    3. Select Upload a file to datastore and upload the ISO. This is the hard disk icon with a green plus sign.

    Note: To upload the ISO through your browser, you will be prompted to install a web plugin. You can also scpthe file directly to the datastore server.

  2. Create a new virtual machine by performing the following steps:

    1. If you use an existing template, select the creation type Deploy from template. Select the existing template, then click Next.
    2. In Select compatibility, select ESXi 5.5 and later. Click Next.
    3. Select Windows as Guest OS Family and Microsoft Windows Server 2012 as Guest OS version. Click Next.
  3. Under New CD\DVD Drive in Customize Hardware, perform the following steps:

    1. Select Datastore ISO File.
    2. Expand the menu and select Connect At Power On.
    3. Click Browse and select the ISO you uploaded to your datastore.
    4. Select IDE as the Virtual Device Node.
  4. After creating the VM, click Power On in the Actions tab for your VM, then install Windows by performing the following steps:

    1. Select Server with GUI.
    2. Select Custom Installation.
    3. Complete the installation process and add Select a password for Administrator user.
  5. In the vCenter web client, click Install VMware Tools in the VM Summary tab. Restart the VM to finish the install.

Step 2: Download BOSH PSModules

Download the bosh-psmodules.zip file for your desired release from the bosh-windows-stemcell-builder GitHub repo.

Warning: Pivotal does not recommend building the BOSH PSModules from the source. However, if you must do so, follow the steps below.

To build the BOSH PSModules from source, perform the following steps:

  1. Clone the bosh-windows-stemcell-builder GitHub repo on your host, not in the VM for your stemcell:
    $ git clone https://github.com/cloudfoundry-incubator/bosh-windows-stemcell-builder
    
  2. Expand the submodules and build the BOSH PSModules:
    $ cd bosh-windows-stemcell-builder
    $ git checkout DESIRED_STEMCELL_VERSION_TAG
    $ git submodule update --init --recursive
    $ gem install bundler
    $ bundle install
    $ rake package:psmodules
    
    Keep in the mind the following when completing this procedure:
    • Do not use the GitHub generated “Source Code” .zip and “Source Code” .tar.gz files from the releases page. These are not Git repositories and are missing submodules.
    • On Windows, you must clone directly into C:\, or the git clone and submodule update commands will fail due to file path lengths.

Step 3: Install BOSH PSModules

  1. Transfer the bosh-psmodules.zip file you either downloaded from the releases page of the bosh-windows-stemcell-builder GitHub repo or the build/bosh-psmodules.zip file you built from the source, to your Windows VM. If you are running VMware Workstation or Fusion, and have installed VMware Tools, you can drag and drop the file.
  2. Unzip the .zip file and copy the BOSH.* folders to C:\Program Files\WindowsPowerShell\Modules.

Step 4: Install Cloud Foundry Cell Requirements

  1. On your Windows VM, start powershell and run Install-CFFeatures. Depending on your Windows ISO, you may need to run powershell -ExecutionPolicy Bypass -Command Install-CFFeatures for this and all subsequent commands from BOSH PSModules.
  2. If you want to apply the recommended ingress and service configuration, run Protect-CFCell from PowerShell.
  3. To ensure you have the proper .NET Framework version(s) installed, follow this guide.

Step 5: Download and Install the BOSH Agent

  1. Download the agent.zip file for your desired release from the bosh-windows-stemcell-builder GitHub repo.
  2. Transfer the agent.zip file to your Windows VM.
  3. On your Windows VM, start powershell and run the following command:
    PS> Install-Agent -IaaS vsphere -agentZipPath PATH_TO_AGENT.ZIP
    

    Warning: Pivotal does not recommend building the BOSH Agent from the source. However, if you must do so, follow the steps below.

To build the BOSH Agent from the source, perform the following steps:

  1. Ensure you have the latest versions of Golang 1.8.x and Ruby 2.3.x installed.
  2. Run rake package:agent on your host. Do not run the rake package:agent command in the VM for your stemcell.
  3. Transfer the build/agent.zip file you built from the source to your Windows VM.
  4. On your Windows VM, start powershell and run Install-Agent -IaaS vsphere -agentZipPath PATH_TO_AGENT.zip.

Step 6: Optimize and Compress Disk

Windows 2012R2 stemcells can be large, and sometimes exceed the 10GB upload limit encoded into the BOSH Director. In order to reduce the stemcell size so you don’t have to change this limit, run the following PowerShell modules:

  • Optimize-Disk: This command is a wrapper that runs dism and clears unnecessary files.
  • Compress-Disk: This command can be used to defrag and zero out the disk.

Step 7: Sysprep and Apply Security Policies

Sysprepping and applying the recommended local security policy ensures uptime as well as the secure operations of the stemcell VMs, especially when deployed with Cloud Foundry.

Keep in mind that this procedure disables Windows Automatic Updates, which tends to restart the OS. When this occurs, BOSH will attempt to recreate the VM.

Perform the following steps:

  1. Download lgpo.exe to the Windows VM you are provisioning.
  2. Save lgpo.exe to C:\Windows\lgpo.exe.
  3. Run the following PowerShell command to sysprep and apply the recommended local security policy, replacing the placeholder values with your own information:
    PS> Invoke-Sysprep -IaaS vsphere -NewPassword NEW_PASSWORD -ProductKey PRODUCT_KEY -Owner OWNER -Organization ORGANIZATION
    
    Keep in mind the following:
    • If you are using a volume-licensed copy of Windows, leave PRODUCT_KEY blank.
    • Do not use any special character in the password other than !. For example, Example12! is acceptable, but not Example#12.
    • If you are invoking the sysprep command with powershell -ExecutionPolicy Bypass -Command Invoke-Sysprep..., wrap the entire sysprep invocation in double quotes:
      powershell -ExecutionPolicy Bypass -Command "Invoke-Sysprep -IaaS vsphere -NewPassword NEW_PASSWORD -Owner OWNER -Organization ORGANIZATION"
      
  4. Running the above command will power off the VM. Do not turn the VM back on before exporting.

For release versions 1079 and later of the BOSH Agent, you can sysprep the image without applying the recommended local security policy. Pivotal does not recommend this for production environments, but you may want it for development or testing environments that don’t require a high level of security, or if you are using your own local security policy.

To skip the application of the security policy, use the -SkipLGPO flag:

While it is not recommended, it is possible to sysprep the image without applying these policies. You may do this in development setups or testing scenarios that don’t require such security, or if you are using your own local policies. To do this, use the -SkipLGPO flag in PowerShell:

PS> Invoke-Sysprep -IaaS vsphere -NewPassword NEW_PASSWORD -ProductKey PRODUCT_KEY -Owner OWNER -Organization ORGANIZATION -SkipLGPO

This will power off the VM. Do not turn the VM back on before exporting.

Step 8: Convert VMDK File to BOSH Stemcell

Follow the instructions for VMware Fusion or VMware Workstation or vSphere.

VMware Fusion or Workstation

Perform the following steps to convert the VMDK file to a BOSH stemcell in VMware Fusion or Workstation:

  1. Locate the .vmdk file associated with the VM you powered off in the previous section. The default directory for VM disk images is in Documents in your home directory, such as ~/Documents/Virtual Machines/vm-name.vmwarevm/Virtual Disk.vmdk on Linux or C:\Users\Administrator\Documents\Virtual Machines\vm-name.vmwarevm\Virtual Disk.vmdk on Windows.
  2. Download the latest release of the stembuild utility, and choose a version for the stemcell.
  3. Use the stembuild utility to create a stemcell:
    $ stembuild -vmdk "~/Documents/My VMs/stemcell-base/Virtual Disk.vmdk" -version 1200.3
    The utility will create the stemcell in your current directory by default. The process may take ten to twenty minutes to complete.

vSphere

Perform the following steps to convert the VMDK file to a BOSH stemcell in vSphere:

  1. If you are using vCenter, right-click the VM and click Template > Export to OVF Template. If you are using the standalone vSphere client, click File > Export > Export OVF Template.
  2. Download the OVA file locally. You do not need to include files in the floppy or CD drive.
  3. Rename the OVA file to have a .tar extension.
  4. Decompress the .tar archive to find the VMDK file.
  5. Download the latest release of the stembuild utility, and choose a version for the stemcell.
  6. Use the stembuild utility to create a stemcell:
    $ stembuild -vmdk "~/Documents/My VMs/stemcell-base/Virtual Disk.vmdk" -version 1200.3
    The utility will create the stemcell in your current directory by default. The process may take ten to twenty minutes to complete.

Step 9: Test Stemcell

Using the bosh-windows-acceptance-tests helps you confirm that your stemcell is working as expected.

Perform the following steps to test the stemcell:

  1. Ensure you have the latest stable version of the BOSH CLI v2 installed. For more information, see the BOSH documentation.
  2. Ensure that you have the latest version of Golang 1.8.x installed.
  3. Set the environment variables in the following table. On Linux, set the variables by including export ENVIRONMENT_VARIABLE=VARIABLE_PATH_OR_VALUE in your Bash profile. On Windows, set the variables within the Advanced section of My Computer.

    Variable Notes
    BOSH_TARGET This is the IP of your BOSH director.
    BOSH_CLIENT This is the BOSH client name.
    BOSH_CLIENT_SECRET This is your BOSH client secret.
    BOSH_CA_CERT This is the content of your BOSH Director cert, including `\n` for new lines.
    BOSH_UUID This is the unique identifier for your BOSH deployment.
    STEMCELL_PATH This is the path to the stemcell tarball.
    AZ This is the AZ from the BOSH cloud config. For more information about the cloud config values, see the BOSH documentation.
    VM_TYPE This is the VM type from the BOSH cloud config.
    VM_EXTENSIONS This is a comma-separated string of VM extensions from the BOSH cloud config.
    NETWORK This is the network from the BOSH cloud config.


  4. Run the bosh-windows-acceptance-tests with the following rake commands:

    $ rake package: bwats
    $ rake run:bwats ["vsphere"]

Known Issues

This section lists known issues for creating a vSphere stemcell manually.

Special Characters

There is a known issue with using special characters in the password you create while completing the Step 7: Sysprep and Apply Security Policies section. Pivotal recommends using the exclamation point as a special character until this issue is completely resolved.

Create a pull request or raise an issue on the source for this page in GitHub