Creating a vSphere Stemcell Manually

In vSphere, customers must build their own stemcells to use their on-premise deployments. To create the vSphere stemcell manually, you must first begin with an ISO or other virtual machine (VM) image. This document describes how to use VMware Workstation, VMware Fusion, and vCenter to do the following:

  • Install the BOSH dependencies.
  • Create a .tgz file, the stemcell, that you can upload to your BOSH Director and use with Cloud Foundry.

Note: This process is based on the fact that the operator is maintaining an updated template with all Windows recommended security updates. Every release of this repository includes an updates.txt file that lists the currently recommended KB Microsoft hotfixes to have installed. You can determine if your image needs updates by creating a VM with the image and going to Control Panel. If any critical or important updates are available, we recommend installing updates first, then rebuilding the stemcell.

Requirements

Component Notes
ovftool Make sure the ovftool command is available from your command line. This tool is installed by default in C:\Program Files\VMware\VMware OVF Tool.
Windows ISO You can also use a custom base image.
Golang You should use the latest 1.8.x compiler.
Ruby You should use the latest latest 2.3.x version.
VMware Workstation, or VMware Fusion, or access to a vSphere account
git
tar.exe You should download this if you are on Windows.
VMware Tools This is installed via VMware Fusion, VMware Workstation, or vCenter.

Note: These are instructions for installing Windows from a Windows installation disk ISO, either a volume licensed copy or a retail copy from MSDN. We recommend maintaining a separate, updated Windows VM based on this ISO to serve as the basis for the installation steps below. This way, you may apply Windows Updates and create new stemcells without having to reinstall all updates from scratch. Make sure that your image has Hardware Compatibility set to version 9, and that it has VMware tools installed.

Step 1: Create base VM for stemcell

For VMware Fusion

  1. Select File and then New.
  2. Select Installation Method: Create a custom virtual machine.
  3. Choose Operating System: Microsoft Windows, and then Windows Server 2012. Despite selecting this Operating System, you will be selecting Windows Server 2012R2 in a subsequent step.
  4. Choose Virtual Disk: Create a new virtual disk. You can use the default settings.
  5. Select Customize Settings. Save the VM before continuing. You can use a name of your choosing.
  6. A Settings window will appear for your new VM. In the settings window, enter the following configurations:
    1. In the Removable Devices row, select Camera > Remove device.
    2. In the Removable Devices row, select CD/DVD, then select the Connect CD/DVD Drive checkbox.
    3. In the This drive is configured to use the following section, select Choose a disc or disc image, then input your base ISO.
    4. In the Advanced Options section select Bus type > SCSI, which is required for Hardware Version 9.
    5. In the Other row, select Compatibility > Advanced Options > Use Hardware Version number 9, then click Apply.
    6. In Processor & Memory, located within the System Settings row, keep the default settings for cores, processors, and memory.
  7. Start VM.
    1. If you are starting from a new MSDN ISO, proceed through the Windows installation process with your ISO. Pivotal recommends that you select Windows Server 2012R2 Standard with a GUI when proceeding with installation.
  8. Install VMware Tools.
    1. Select Virtual Machine, select Install VMWare Tools, Complete Installation, and follow the installation instructions.
    2. Restart the VM to finish the install.
  9. Shut down the VM.
  10. In the Settings window of the new VM, select CD/DVD, within the the Removable Devices row.
    1. Select Autodetect. This effectively removes the ISO.
    2. De-select Connect CD/DVD Drive.
    3. Select Advanced Options, and switch Bus type to IDE.
  11. Turn on and then turn off the VM. This is required to apply changes to CD/DVD.
  12. In the Other row, select Compatibility and then Advanced Options. Ensure that the Hardware Version is set to 9.

For VMware Workstation

  1. Install VMware workstation.
  2. Create a new Virtual Machine:
    1. Select Custom Advanced, then click Next.
    2. Select Workstation 9.x Compatibility, click Next.
    3. Select I will install operating system later, then click Next.
    4. Select Microsoft Windows, then the Windows 2012R2 version. Click Next.
    5. Choose a name for your virtual machine, then click Next.
    6. Select BIOS and click Next.
    7. Keep the default settings for number of cores and processors and click Next.
    8. Keep default settings for memory and click Next.
    9. Select the correct Network Type (NAT), then click Next.
    10. Create a new virtual disk, then click Next.
    11. Select Customize Hardware.
      1. Select New CD/DVD.
      2. Select Use ISO Image file and browse for the correct ISO.
  3. Power on the new VM and install Windows.
    1. Select Server with GUI.
    2. Select Custom Installation.
    3. Continue the installation and select a password for the Admin user.
  4. After the VM starts successfully, right-click the machine name in Workstation and select Install VMware Tools.
  5. Shut down the VM.
  6. Remove the ISO file from the CD/DVD drive.
    1. Select the settings for the VM.
    2. Select CD/DVD Remove.
    3. Add CD/DVD Drive.
    4. Select Use Physical drive > Auto Detect.
    5. De-select Connect at power on.
    6. Select Ok.
  7. Start the new VM.

For vCenter

  1. If you are using an ISO, upload it to your datastore.
    1. Select vCenter > Storage.
    2. Select your desired datastore and directory.
    3. Select Upload a file to datastore and upload the ISO. This is the hard disk icon with a green plus sign.
  2. If you use a web browser to upload the ISO to your datastore, you will be prompted to install a web plugin.
  3. Create a new virtual machine.

    1. If you use an existing template, select the creation type: Deploy from template. Select the existing template, then click Next.
  4. In Select compatibility, choose ESXi 5.5 and later. Click Next.

  5. Select Windows as Guest OS Family and Microsoft Windows Server 2012R2 as Guest OS version. Click Next.

  6. In Customize Hardware:

    1. Under New CD\DVD Drive:
      • Select Datastore ISO File.
      • Expand the menu and select Connect At Power On.
      • Click Browse and select the ISO you uploaded to your datastore.
      • Select IDE as the Virtual Device Node.
    2. Remove floppy drive, if present.
    3. Remove SATA Controller, if present.
  7. After creating the VM, click Power On in the Actions tab for your VM, then install Windows:

    1. Select Server with GUI.
    2. Select Custom Installation.
    3. Complete the installation process and add select a password for Administrator user.
  8. In the vCenter web client, Install VMware Tools in the VM Summary tab.

  9. With the VM powered off, select Edit Settings.

    • Remove CD/DVD drive.
    • Select New device > CD/DVD Drive and click Add.
    • Under New CD/DVD Drive, select Client Device.
    • Expand the menu and select IDE as the Virtual Device Node. Connect at Power On should not be selected.
  10. Remove New SATA Controller, if present.

Step 2: Download BOSH PSModules

Note: Do not use the GitHub generated “Source Code” .zip and “Source Code” .tar.gz files from the releases page. These are not Git repositories and are missing submodules. On Windows, you must clone directly into C:\\, or the git clone and submodule update will fail due to file path lengths.

  1. Download the bosh-psmodules.zip for your desired release here.
  2. Proceed to Step 3: Install BOSH PSModules.

Warning: Pivotal does not recommend building the BOSH PSModules from the source. However, if you must do so, follow the steps below.

To build the BOSH PSModules from the source:

  1. Clone this repo on your host, not in the VM for your stemcell.
  2. Expand the submodules by running the commands below:
    $ git clone https://github.com/cloudfoundry-incubator/bosh-windows-stemcell-builder
    $ cd bosh-windows-stemcell-builder
    $ git checkout DESIRED_STEMCELL_VERSION_TAG
    $ git submodule update --init --recursive
    $ gem install bundler
    $ bundle install
    $ rake package:psmodules
    

Step 3: Install BOSH PSModules

  1. Transfer the bosh-psmodules.zip file you either downloaded from the releases page or the build/bosh-psmodules.zip file you built from the source, to your Windows VM. You can drag and drop files if you have installed VMware tools in your VM and are running Workstation or Fusion.
  2. Unzip the .zip file and copy the BOSH.* folders to C:\Program Files\WindowsPowerShell\Modules.

Step 4: Install CloudFoundry Cell requirements

  1. On your windows VM, start powershell and run Install-CFFeatures.
  2. Apply the recommended ingress and service configuration:
    1. Run the following powershell command Protect-CFCell.

To make sure you have the proper .NET Framework version(s) installed, follow this guide.

Step 5: Download and Install the BOSH Agent

  1. Download agent.zip file. The agent.zip file is provided as a package in the release available on Github. We recommend downloading the .zip file from that package.
  2. Transfer the build/agent.zip file you downloaded from the releases page to your Windows VM.
  3. On your Windows VM, start powershell and run Install-Agent -IaaS vsphere -agentZipPath PATH_TO_agent.zip.
  4. Proceed to Step 6: Optimize and Compress Disk.

IMPORTANT: Pivotal does not recommend building the agent.zip from the source. However, if you must do so, run the rake commands below.

To build the agent.zip from the source:

  1. Run rake package:agent on your host. Do not run the rake package:agent command in the VM for your stemcell.
  2. Transfer the build/agent.zip file you built from the source to your Windows VM.
  3. On your Windows VM, start powershell and run Install-Agent -IaaS vsphere -agentZipPath PATH_TO_agent.zip.
  4. Proceed to Step 6: Optimize and Compress Disk.

Step 6: Optimize and Compress Disk

In order to reduce the stemcell size, you can run the following powershell modules:

  • Optimize-Disk: This command is a wrapper that runs dism and clears unnecessary files.
  • Compress-Disk: This command can be used to defrag and zero out the disk.

Step 7: Sysprep and apply security policies

Note: Applying the security policies supports the security of the stemcell and firewall rules that ensure secure operations on Cloud Foundry; therefore, Pivotal strongly recommends applying these policies. You can do so by completing the three steps below:

  1. Download lgpo.exe to the Windows VM you are provisioning.
  2. Save lgpo.exe to C:\Windows\lgpo.exe.
  3. Run the following powershell command, replacing values in ALL_CAPS with your own information: Invoke-Sysprep -IaaS vsphere -NewPassword NEW_PASSWORD -ProductKey PRODUCT_KEY -Owner OWNER -Organization ORGANIZATION

Completing the above will power off the VM. Do not turn the VM back on before exporting.

If you decide to sysprep the image without applying the recommended security policies:

  1. Run the following powershell command, replacing values in ALL_CAPS with your own information:

Invoke-Sysprep -IaaS vsphere -NewPassword NEW_PASSWORD -ProductKey PRODUCT_KEY -Owner OWNER -Organization ORGANIZATION -SkipLGPO

This will power off the VM. Do not turn the VM back on before exporting.

Step 8: Export image to OVA format

If you are using VMware Fusion or Workstation:

  1. Power off the VM.
  2. Locate the directory that has your VM’s .vmx file. This defaults to the Documents\Virtual Machines\VM-name\VM-name.vmx path in your user’s home directory. You can also right-click on the VM to find its location.
  3. Convert the .vmx file into an OVA archive using ovftool with the following command: ovftool <PATH_TO_VMX_FILE> image.ova

Step 9: Convert OVA file to a BOSH Stemcell

  1. Run gem install bundler.
  2. Run bundle install.
  3. Convert the OVA file by running the following command: rake package:vsphere_ova[PATH_TO_OVA,PATH_TO_STEMCELL_DESTINATION_DIRECTORY,STEMCELL_VERSION]

For example:

$ rake package:vsphere_ova[./build/image.ova,./build/stemcell,1035.0]

Currently, the OVA filename and destination path cannot contain spaces.

Step 10: Testing Stemcell with bosh-windows-acceptance-tests (BWATS)

Using bosh-windows-acceptance-tests helps you confirm that your stemcell is working as expected.

Requirements

Component Notes
bosh cli Latest stable version
Golang Latest 1.8.x compiler

bosh cli environment variables

Variable Notes
BOSH_TARGET This is the IP of your BOSH director.
BOSH_CLIENT This is the BOSH client name.
BOSH_CLIENT_SECRET This is your BOSH client secret.
BOSH_CA_CERT This is the contents of your bosh director cert, including \n for newlines.
BOSH_UUID This is the unique identifier for your BOSH deployment.
STEMCELL_PATH This is the path to the stemcell tarball.
AZ This is the area zone from the BOSH cloud config.
VM_TYPE This is the virtual machine type from the BOSH cloud config.
VM_EXTENSIONS This is a comma separated string of options, for example, 50GB_ephemeral_disk
NETWORK This is the network from the BOSH cloud config.


  1. Set the above environment variables by using your chosen input method. You can, for example, set the variable by running export ENVIRONMENT_VARIABLE=VARIABLE_PATH_OR_VALUE in your BASH profile if you are running Linux. If you are running Windows, you can set environment variables within the Advanced section of My Computer.
  2. Run bosh-windows-acceptance-tests (BWATS) with the following rake commands:
$ rake package: bwats
$ rake run:bwats ["vsphere"]

Known Issues

This section lists known issues for creating a vSphere stemcell manually.

Special Characters

There is a known issue with using special characters in the password you create while completing the Sysprep and apply security policies step. We recommend using the exclamation point, that is “!”, as a special character until this issue is completely resolved.

Create a pull request or raise an issue on the source for this page in GitHub