LATEST VERSION: 1.8 - CHANGELOG
PCF IPsec Add-On v1.8

Installing the IPsec Add-on for PCF

Page last updated:

This topic describes how to prepare your network for IPsec, create an IPsec manifest, and add IPsec to your deployment.

Prerequisites

To complete the IPsec installation, verify that you have satisfied the following prerequisites before you begin:

  • Google Cloud Platform (GCP), vSphere, Azure, Amazon Web Services (AWS), or OpenStack as your IaaS

  • Pivotal Cloud Foundry (PCF) operator administration rights

  • BOSH deployed through Ops Manager v1.8 or later

  • Set the MTU for your IaaS in the Elastic Runtime tile, under Networking. Pivotal recommends MTU values of 1354 on GCP, 1438 on Azure, and the default values on AWS and vSphere. For OpenStack, follow the recommendations of your Neutron/ML2 plugin provider, or empirically test the correct MTU for your environment.

Best Practices

  • IPsec may affect the functionality of other service tiles. As a result, Pivotal recommends deploying Elastic Runtime and each service tile to different isolated subnets. Alternatively, you can minimally deploy all service tiles to a single isolated subnet, apart from the Elastic Runtime subnet. Some service tiles, for example RabbitMQ, may not support IPsec, and must be placed in a non-IPsec subnet.
  • For IPsec on Linux VMs, Pivotal recommends any Ubuntu stemcells for vSphere, OpenStack, and HVM stemcells for AWS. These stemcells are available on Pivotal Network. If you use PV stemcells obtained from bosh.io, see the Packet Loss section of the Troubleshooting the IPsec Add-on for PCF topic to adjust MTU values.
  • For IPsec on Windows VMs, Pivotal recommends the Windows stemcells for AWS, GCP, or Azure available on Pivotal Network.

Configure Network Security

Refer to the appropriate section below for your IaaS network configuration details.

Google Cloud Platform

To configure your Google Cloud Platform (GCP) environment for IPsec, perform the following steps:

  1. Navigate to the Networking section of the GCP Console.
  2. Click Firewall rules.
  3. Click Create Firewall Rule.
  4. For Name, enter ipsec.
  5. For Network, select the network where Ops Manager is deployed. For example, opsmgr.
  6. For Source filter, select Allow from any source (0.0.0.0/0).
  7. For Allowed protocols and ports, enter udp:500; ah; esp.
  8. Click Create.
  9. Adjust the MTU value to 1354 by performing the procedure in the Packet Loss section of the Troubleshooting the IPsec Add-on for PCF topic.

vSphere

Confirm that your network allows the protocols listed in the table below.

Protocol Name Protocol Number Port(s)
AH 51 Any
ESP 50 Any
UDP 17 500

Azure

  1. Confirm that your network allows the protocols listed in the table below.

    Protocol Name Protocol Number Port(s)
    AH 51 Any
    ESP 50 Any
    UDP 17 500

  2. Adjust the MTU value to 1438 by performing the procedure in Packet Loss.

AWS

To configure your AWS environment for IPsec, perform the following steps:

  1. Navigate to EC2 Dashboard > Security Groups.

  2. Select the Security Group with the description PCF VMs Security Group and click Edit.

  3. Create the following Inbound Rules.

    Type Protocol Name Protocol Number Port Range Source
    Custom Protocol AH 51 All 10.0.0.0/16
    Custom Protocol ESP 50 All 10.0.0.0/16
    Custom UDP Rule UDP 17 500 10.0.0.0/16

Note: The default PCF VMs Security Group is typically specified with a subnet of 10.0.0.0/16. If your PCF subnet is deployed to a different CIDR block, adjust the source as needed.

OpenStack

Note: The following network configuration is optimized for Mirantis OpenStack, but other OpenStack distributions have a similar workflow.

To configure your Mirantis OpenStack environment for IPsec, perform the following steps:

  1. Navigate to Project / Access & Security.

  2. Select the security group and click Manage Rules.

  3. Create the following Ingress and Egress Rules. Adjust the source CIDR as needed for your environment.

    Protocol Name Protocol Number Port Range Source
    ESP 50 Any 0.0.0.0/0
    AH 51 Any 0.0.0.0/0
    UDP 17 500 0.0.0.0/0

Create the IPsec Manifest

To add IPsec to VMs in your deployment, you must first create a manifest file ipsec-addon.yml that configures IPsec add-on properties for Linux VMs, Windows VMs, or both.

Start by creating an IPsec manifest file ipsec-addon.yml, with the code below as a template.

releases:
- {name: ipsec, version: 1.0.0}
addons:

Add properties to the ipsec-addon.yml file as described below for Linux VMs and Windows VMs.

Enabling IPsec for Windows adds IPsec security to Windows VMs that users can create after you install PCF Runtime for Windows tile.

Add Linux VM Support to Your Manifest

Follow these steps to add IPsec to Linux VMs in your deployment.

  1. Add the following to your ipsec-addon.yml, after the addons: line.

    - name: ipsec-addon
    releases:
    - {name: ipsec, version: 1.0.0}
    addons: - name: ipsec-addon jobs: - name: ipsec release: ipsec include: stemcell: - os: ubuntu-trusty properties: ipsec: optional: false ipsec_subnets: - 10.0.1.1/20 no_ipsec_subnets: - 10.0.1.10/32 # bosh director - 10.0.1.4/32 # ops manager instance_certificate: | -----BEGIN CERTIFICATE----- MIIEMDCCAhigAwIBAgIRAIvrBY2TttU/LeRhO+V1t0YwDQYJKoZIhvcNAQELBQAw ... -----END CERTIFICATE----- instance_private_key: | -----BEGIN RSA PRIVATE KEY----- EXAMPLExRSAxPRIVATExKEYxDATAxEXAMPLExRSAxPRIVATExKEYxDATA ... -----END RSA PRIVATE KEY----- ca_certificates: - | -----BEGIN CERTIFICATE----- MIIFCTCCAvGgAwIBAgIBATANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDEwl0ZXN0 ... -----END CERTIFICATE----- - | -----BEGIN CERTIFICATE----- MIIFCTCCAvGgAwIBAgIBATAAYDVQQDEwl0ZXN0NBgkqhkiG9w0BAQsFADAUMRIwE ... -----END CERTIFICATE----- prestart_timeout: 30 esp_proposals: aes128gcm16! ike_proposals: aes128-sha256-modp2048! log_level: 1

  2. Replace the values listed in the template as follows:

    • releases: - version: Specify the version number of your IPsec download from Pivotal Network.
    • jobs: - name: Do not change the name of this job. It must be ipsec.
    • include: stemcell - os: IPsec for Linux is only supported on ubuntu-trusty.
    • optional: Makes IPsec enforcement optional. To add IPsec, set this flag to true. After IPsec has been successfully installed, set this flag back to false and redeploy.
    • ipsec_subnets: List the subnets that you want to be encrypted. You can include the entire deployment or a portion of the network. Encrypt any network that handles business-sensitive data.
    • no_ipsec_subnets: List the IP address of your BOSH Director and Ops Manager VM, along with any other IP addresses in your PCF deployment that you want to communicate without encryption. Pivotal recommends that you list the subnets that are used for PCF managed services. Subnets for PCF managed services that do not support IPsec must be listed under no-ipsec.

      Note: In GCP, if you use the default router for DNS instead of the Google public DNS at 8.8.8.8, you must add the IP address of the default router in your subnet to no_ipsec_subnets. For example, 10.0.0.1/32.

    • instance_certificate: Copy in the signed certificate that will be used by all your instance VMs. You must use one of the CAs in the ca_certificates property to sign this certificate. For a development or test environment, you can use a self-signed certificate. For more information, see Generate a Self-Signed Certificate above.
    • instance_private_key: Copy in the private key that corresponds to the instance_certificate above. This key must not use a pass phrase.
    • ca_certificates: Copy in CA certificates for the instance VM to trust during the validation process. In most cases, you only need the CA certificate used to sign the instance certificate. During CA credential rotation, you need two CA certificates.
    • prestart_timeout: You can modify the 30 second default prestart timeout value. This value limits the number of seconds allowed for IPsec to start before failing the attempt.
    • esp_proposals: You can modify the ESP (Quick Mode) encryption and integrity algorithms. The default, aes128gcm16!, is 128 bit AES-GCM with 128 bit ICV for both encryption and integrity.
    • ike_proposals: You can modify the IKE (Main Mode) encryption and integrity algorithms, and the Diffie-Hellman group. The default, aes128-sha256-modp2048!, is 128 bit AES-CBC for encryption, SHA2_256_128 HMAC for integrity, and Group 14 for Diffie-Hellman.
    • log_level: You can specify the IKE daemon numerical log level, ranging from -1 to 4. For more information, see Logger Configuration in the strongSwan documentation.

Add Windows VM Support to Your Manifest

Follow these steps to add IPsec to Windows VMs in your deployment.

  1. Add the following to your ipsec-addon.yml, under addons:. Add this code under the ipsec-addon section for Linux, if you included one above.

    - name: ipsec-windows-addon
      jobs:
      - name: ipsec-win
        release: ipsec
      include:
        stemcell:
          - os: windows2012R2
      properties:
        ipsec:
          optional: false
          ipsec_subnets:
          - 10.0.1.1/20
          no_ipsec_subnets:
          - 10.0.1.10/32  # bosh director
          - 10.0.1.4/32 # ops manager
          instance_certificate: |
            -----BEGIN CERTIFICATE-----
            MIIEMDCCAhigAwIBAgIRAIvrBY2TttU/LeRhO+V1t0YwDQYJKoZIhvcNAQELBQAw
            ...
            -----END CERTIFICATE-----
          instance_private_key: |
            -----BEGIN RSA PRIVATE KEY-----
            EXAMPLExRSAxPRIVATExKEYxDATAxEXAMPLExRSAxPRIVATExKEYxDATA
            ...
            -----END RSA PRIVATE KEY-----
          ca_certificates:
            - |
              -----BEGIN CERTIFICATE-----
              MIIFCTCCAvGgAwIBAgIBATANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDEwl0ZXN0
              ...
              -----END CERTIFICATE-----
            - |
              -----BEGIN CERTIFICATE-----
              MIIFCTCCAvGgAwIBAgIBATAAYDVQQDEwl0ZXN0NBgkqhkiG9w0BAQsFADAUMRIwE
              ...
              -----END CERTIFICATE-----
          quick_mode_proposals:
            - encryption: AESGCM128
              hash: AESGMAC128
          main_mode_proposals:
            - encryption: AES128
              hash: SHA256
              keyexchange: DH14

  2. Replace the values listed in the template as follows:

    • jobs: - name: Do not change the name of this job. It must be ipsec-win.
    • include: stemcell - os: IPsec for Windows is only supported on windows2012R2.
    • ipsec_subnets: Copy and paste the value from ipsec_subnets for Linux.
    • no_ipsec_subnets: Copy and paste the value from no_ipsec_subnets for Linux.
    • instance_certificate: Copy and paste the value from instance_certificate for Linux.
    • instance_private_key: Copy and paste the value from instance_private_key for Linux.
    • ca_certificates: Copy and paste the value from ca_certificates for Linux.
    • optional: Copy and paste the value from optional for Linux.
    • quick_mode_proposals: Array of Quick Mode algorithms for encryption and integrity. This value must match the list specified in esp_proposals for Linux. See the following default entry that matches the Linux default:
      - encryption: AESGCM128
        hash: AESGMAC128
      
    • main_mode_proposals: Array of Main Mode algorithms for encryption, integrity, and key exchange. This value must match the list specified in ike_proposals for Linux. See the following default entry that matches the Linux default: The default:
      - encryption: AES128
        hash: SHA256
        keyexchange: DH14
      

Linux and Windows Mixed Deployment Proposal Sets

Encryption Type Linux IPsec Property Windows IPsec Property
ike_proposals esp_proposals main_mode_proposals quick_mode_proposals
128 Bit Encryption
aes128-sha256-modp2048!
aes128gcm16!
- encryption: AES128
  hash: SHA256
  keyexchange: DH14
- encryption: AESGCM128
  hash: AESGMAC128
256 Bit Encryption
aes256-sha256-modp2048!
aes256gcm16!
- encryption: AES256
  hash: SHA256
  keyexchange: DH14
- encryption: AESGCM256
  hash: AESGMAC256

Download and Deploy the IPsec Add-on

After deploying Ops Manager, perform the following steps to download the IPsec binary, add your IPsec manifest to your BOSH manifest, and deploy the IPsec add-on:

  1. Download the IPsec add-on software binary from the Pivotal Network to your local machine.

  2. Copy the software binary to your Ops Manager instance.

    $ scp -i PATH/TO/PRIVATE/KEY ipsec-release.tar.gz ubuntu@YOUR-OPS-MANAGER-VM-IP:

  3. Copy the IPsec manifest file to your Ops Manager instance.

    $ scp -i PATH/TO/PRIVATE/KEY ipsec-addon.yml ubuntu@YOUR-OPS-MANAGER-VM-IP:

  4. SSH into Ops Manager.

    $ ssh -i PATH-TO-PRIVATE-KEY ubuntu@YOUR-OPS-MANAGER-VM-IP

  5. On the Ops Manager VM, navigate to the software binary location in your working directory.

    $ cd PATH-TO-BINARY

  6. On the Ops Manager VM, target the internal IP address of your BOSH Director. When prompted, enter your BOSH Director credentials. To retrieve your BOSH Director credentials, navigate to Ops Manager, click the Credentials tab, and click Link to Credential next to Director Credentials.

    $ bosh --ca-cert /var/tempest/workspaces/default/root_ca_certificate target YOUR-BOSH-DIRECTOR-INTERNAL-IP
    Target set to 'p-bosh'
    Your username: director
    Enter password: ******************
    Logged in as 'director'
    

  7. Upload your release.

    $ bosh upload release PATH-TO-BINARY/BINARY-NAME.tar

  8. Optionally, from the command line, confirm that the upload of the IPsec software binary completed. You should see the IPsec binary file.

    $ bosh releases

  9. Download your current runtime config and save as bosh-manifest.yml.

    $ bosh runtime-config > bosh-manifest.yml

  10. Append the contents of your IPsec manifest ipsec-addon.yml to bosh-manifest.yml.

  11. Update your runtime configuration to include the IPsec add-on.

    $ bosh update runtime-config PATH/bosh-manifest.yml

  12. Verify your runtime configuration changes match what you specified in the IPsec manifest file.

    $ bosh runtime-config
    Acting as user 'admin' on 'micro'
    
    releases:
     - {name: ipsec, version: 1.0.0}
    
    addons:
     name: ipsec-addon
      jobs:
      - name: ipsec
        release: ipsec
        ...
      - name: ipsec-win  # if using Windows
        release: ipsec
        ...
    
  13. If you have already deployed Elastic Runtime or are adding IPsec to an existing deployment:

    1. Set the optional flag to true.
    2. Navigate to your Installation Dashboard in Ops Manager.
    3. Click Apply Changes
    4. Wait for the installation to complete.
    5. Set the optional flag to false.
    6. Update the runtime config.
      $ bosh update runtime-config PATH/ipsec-addon.yml
    7. Navigate to your Installation Dashboard.
    8. Click Apply Changes.
  14. If Elastic Runtime tile is not yet installed:

    1. Navigate to your Installation Dashboard in Ops Manager.
    2. Click Apply Changes
    3. Deploy Elastic Runtime by following the installation instructions for your IaaS. For more information, see Installing Pivotal Cloud Foundry above.
  15. Secure the sensitive information in the ipsec-addon.yml file. Pivotal recommends encrypting the file and moving it to a secure location.

Verify Your IPsec Installation

After installing IPsec and deploying Elastic Runtime, perform the following steps to verify your IPsec installation:

  1. Run bosh vms to list the job VMs in your deployment. Choose the job name and index of any VM. The exact VM does not matter, because installing the IPsec add-on loads IPsec on all VMs deployed by Ops Manager.

  2. Run bosh ssh JOB-NAME/INDEX to open an SSH connection into the VM.

  3. Run sudo su - to enter the root environment with root privileges.

  4. Run monit summary to confirm that your ipsec job is listed as a bosh job.

    The Monit daemon 5.2.5 uptime: 18h 32m
    ...
    Process 'ipsec'                     running
    System 'system_localhost'           running
    

  5. Run PATH-TO-IPSEC/ipsec statusall to confirm that IPsec is running. If IPsec is not running, this command produces no output.

    $ /var/vcap/packages/strongswan-5.3.5/sbin/ipsec statusall
    Status of IKE charon daemon (strongSwan 5.3.5, Linux 3.19.0-56-generic, x86_64):
    uptime: 18 hours, since Mar 16 23:58:50 2016
    malloc: sbrk 2314240, mmap 0, used 1182400, free 1131840
    worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 206
    loaded plugins: charon aes sha1 sha2 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pem gmp xcbc cmac hmac attr kernel-netlink socket-default stroke
    Listening IP addresses:
    10.10.5.66
    Connections:
    ipsec-10.10.4.0/24:  %any...%any  IKEv1/2
    ipsec-10.10.4.0/24:   local:  [CN=test-cert-1-ca-1] uses public key authentication
    ipsec-10.10.4.0/24:    cert:  "CN=test-cert-1-ca-1"
    ipsec-10.10.4.0/24:   remote: uses public key authentication
    ipsec-10.10.9.0/24:   child:  10.10.5.66/32 === 10.10.9.0/24 TRANSPORT
    no-ipsec-10.10.4.1/32:  %any...%any  IKEv1/2
    no-ipsec-10.10.4.1/32:   local:  uses public key authentication
    no-ipsec-10.10.4.1/32:   remote: uses public key authentication
    no-ipsec-10.10.4.1/32:   child:  dynamic === 10.10.4.1/32 PASS
    Shunted Connections:
    no-ipsec-10.10.4.1/32:  dynamic === 10.10.4.1/32 PASS
    no-ipsec-10.10.5.1/32:  dynamic === 10.10.5.1/32 PASS
    no-ipsec-10.10.6.1/32:  dynamic === 10.10.6.1/32 PASS
    Routed Connections:
    ipsec-10.10.9.0/24{6}:  ROUTED, TRANSPORT, reqid 6
    ipsec-10.10.9.0/24{6}:   10.10.5.66/32 === 10.10.9.0/24
    ipsec-10.10.8.0/24{5}:  ROUTED, TRANSPORT, reqid 5
    ipsec-10.10.4.0/24{1}:   10.10.5.66/32 === 10.10.4.0/24
    Security Associations (45 up, 0 connecting):
    ipsec-10.10.4.0/24[459]: ESTABLISHED 13 seconds ago, 10.10.5.66[CN=test-cert-1-ca-1]...10.10.4.38[CN=test-cert-1-ca-1]
    ipsec-10.10.4.0/24{1527}:   10.10.5.66/32 === 10.10.4.38/32
    ...
    

  6. If you installed IPsec for Windows, follow these steps:

    1. From any Windows VM, open Windows Firewall with Advanced Security.
    2. Click Connection Security Rules.
    3. Confirm that you see rules for each ipsec and no-ipsec subnet that you listed in your manifest.

Generate a Self-Signed Certificate

Note: Use a self-signed certificate only for development or test environments. We recommend against using self-signed certificates for production deployments. The script below generate private keys in a certs subdirectory.

Follow the instructions to generate a self-signed certificate for your IPsec manifest.

Generate a Self-Signed Certificate with OpenSSL

  1. Download the openssl-create-ipsec-certs.sh bash script.
  2. Navigate to the directory where you downloaded the script:
    $ cd ~/workspace
  3. Change the permissions of the script:
    $ chmod u+x openssl-create-ipsec-certs.sh
  4. Run the script:
    $ ./openssl-create-ipsec-certs.sh
  5. Because this certificate expires in 365 days, set a calendar reminder to rotate the certificate within the year. For instructions on changing certificates, see Rotating IPsec Certificates.
Create a pull request or raise an issue on the source for this page in GitHub