Installing the Pivotal Cloud Foundry IPsec Add-On

Page last updated:

This topic describes how to prepare your network for IPsec, create an IPsec manifest, and add IPsec to your deployment.

Prerequisites

To complete the IPsec installation, verify that you have satisfied the following prerequisites before you begin:

  • Google Cloud Platform (GCP), vSphere, Azure, Amazon Web Services (AWS), or OpenStack as your IaaS

  • Pivotal Cloud Foundry (PCF) operator administration rights

  • BOSH deployed through Ops Manager 1.7 or later

  • Set the MTU for your IaaS in the Elastic Runtime tile, under Networking. Pivotal recommends MTU values of 1354 on GCP, 1438 on Azure, and the default values on AWS and vSphere. For OpenStack, follow the recommendations of your Neutron/ML2 plugin provider, or empirically test the correct MTU for your environment.

Limitations

Before installing the IPsec add-on, review the following list of limitations:

  • The IPsec add-on does not work on Windows.

  • IPsec Release v1.5.x series: During installation, the v1.5.x releases of the IPsec add-on may disrupt connectivity. For minimal downtime in new deployments, Pivotal recommends installing IPsec immediately after Ops Manager, and before installing the Elastic Runtime tile. For existing deployments, there will be downtime when applying IPsec. Upgrade to v1.6.x series to avoid downtime. For a cluster of approximately 40 VMs, you can expect approximately one hour of downtime.

Best Practices

  • IPsec may affect the functionality of other service tiles. As a result, Pivotal recommends deploying Elastic Runtime and each service tile to different isolated subnets. Alternatively, you can minimally deploy all service tiles to a single isolated subnet, apart from the Elastic Runtime subnet. Some service tiles, for example RabbitMQ, may not support IPsec, and must be placed in a non-IPsec subnet.
  • For IPsec, Pivotal recommends any Ubuntu stemcells for vSphere, OpenStack, and HVM stemcells for AWS. These stemcells are available on Pivotal Network. If you use PV stemcells obtained from bosh.io, see the Packet Loss section of the Troubleshooting the PCF IPsec Add-On topic to adjust MTU values.

Configure Network Security

Refer to the appropriate section below for your IaaS network configuration details.

Google Cloud Platform

To configure your Google Cloud Platform (GCP) environment for IPsec, perform the following steps:

  1. Navigate to the Networking section of the GCP Console.
  2. Click Firewall rules.
  3. Click Create Firewall Rule.
  4. For Name, enter ipsec.
  5. For Network, select the network where Ops Manager is deployed. For example, opsmgr.
  6. For Source filter, select Allow from any source (0.0.0.0/0).
  7. For Allowed protocols and ports, enter udp:500; ah; esp.
  8. Click Create.
  9. Adjust the MTU value to 1354 by performing the procedure in the Packet Loss section of the Troubleshooting the PCF IPsec Add-On topic.

vSphere

Confirm that your network allows the protocols listed in the table below.

Protocol Name Protocol Number Port(s)
AH 51 Any
ESP 50 Any
UDP 17 500

Azure

  1. Confirm that your network allows the protocols listed in the table below.

    Protocol Name Protocol Number Port(s)
    AH 51 Any
    ESP 50 Any
    UDP 17 500

  2. Adjust the MTU value to 1438 by performing the procedure in the Packet Loss section of the Troubleshooting the PCF IPsec Add-On topic.

AWS

To configure your AWS environment for IPsec, perform the following steps:

  1. Navigate to EC2 Dashboard > Security Groups.

  2. Select the Security Group with the description PCF VMs Security Group and click Edit.

  3. Create the following Inbound Rules.

    Type Protocol Name Protocol Number Port Range Source
    Custom Protocol AH 51 All 10.0.0.0/16
    Custom Protocol ESP 50 All 10.0.0.0/16
    Custom UDP Rule UDP 17 500 10.0.0.0/16

Note: The default PCF VMs Security Group is typically specified with a subnet of 10.0.0.0/16. If your PCF subnet is deployed to a different CIDR block, adjust the source as needed.

OpenStack

Note: The following network configuration is optimized for Mirantis OpenStack, but other OpenStack distributions have a similar workflow.

To configure your Mirantis OpenStack environment for IPsec, perform the following steps:

  1. Navigate to Project / Access & Security.

  2. Select the security group and click Manage Rules.

  3. Create the following Ingress and Egress Rules. Adjust the source CIDR as needed for your environment.

    Protocol Name Protocol Number Port Range Source
    ESP 50 Any 0.0.0.0/0
    AH 51 Any 0.0.0.0/0
    UDP 17 500 0.0.0.0/0

Create the IPsec Manifest

Follow these steps to create the IPsec manifest for your deployment:

  1. Create an IPsec manifest file ipsec-addon.yml, starting with the code below as a template.

    releases:
    - {name: ipsec, version: 1.0.0}
    addons: - name: ipsec-addon jobs: - name: ipsec release: ipsec include: stemcell: - os: ubuntu-trusty properties: ipsec: optional: false|true # Available since v1.6.x ipsec_subnets: - 10.0.1.1/20 no_ipsec_subnets: - 10.0.1.10/32 # bosh director - 10.0.1.4/32 # ops manager instance_certificate: | -----BEGIN CERTIFICATE----- MIIEMDCCAhigAwIBAgIRAIvrBY2TttU/LeRhO+V1t0YwDQYJKoZIhvcNAQELBQAw ... -----END CERTIFICATE----- instance_private_key: | -----BEGIN RSA PRIVATE KEY----- EXAMPLExRSAxPRIVATExKEYxDATAxEXAMPLExRSAxPRIVATExKEYxDATA ... -----END RSA PRIVATE KEY----- ca_certificates: - | -----BEGIN CERTIFICATE----- MIIFCTCCAvGgAwIBAgIBATANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDEwl0ZXN0 ... -----END CERTIFICATE----- - | -----BEGIN CERTIFICATE----- MIIFCTCCAvGgAwIBAgIBATAAYDVQQDEwl0ZXN0NBgkqhkiG9w0BAQsFADAUMRIwE ... -----END CERTIFICATE----- prestart_timeout: 30 esp_proposals: aes128gcm16! # Available since v1.6.x ike_proposals: aes128-sha256-modp2048! # Available since v1.6.x

  2. Replace the values listed in the template as follows:

    • For v1.5.x and above:
      • releases: - version: Specify the version number of your IPsec download from Pivotal Network.
      • jobs: - name: Do not change the name of this job. It must be ipsec.
      • include: stemcell - os: Do not change the OS type. IPsec is only supported on ubuntu-trusty.
      • ipsec_subnets: List the subnets that you want to be encrypted. You can include the entire deployment or a portion of the network. Encrypt any network that handles business-sensitive data.
      • no_ipsec_subnets: List the IP address of your BOSH Director and Ops Manager VM, along with any other IP addresses in your PCF deployment that you want to communicate without encryption. Pivotal recommends that you list the subnets that are used for PCF managed services. Subnets for PCF managed services that do not support IPsec must be listed under no-ipsec.

        Note: In GCP, if you use the default router for DNS instead of the Google public DNS at 8.8.8.8, you must add the IP address of the default router in your subnet to no_ipsec_subnets. For example, 10.0.0.1/32.

      • instance_certificate: Copy in the signed certificate that will be used by all your instance VMs. You must use one of the CAs in the ca_certificates property to sign this certificate. For a development or test environment, you can use a self-signed certificate. See the Generate a Self-Signed Certificate section of this topic for more information.
      • instance_private_key: Copy in the private key that corresponds to the instance_certificate above. This key must not use a pass phrase.
      • ca_certificates: Copy in CA certificates for the instance VM to trust during the validation process. In most cases, you only need the CA certificate used to sign the instance certificate. During CA credential rotation, you need two CA certificates.
      • prestart_timeout: You can modify the 30 second default prestart timeout value. This value limits the number of seconds allowed for IPsec to start before failing the attempt.
    • For v1.6.x and above:
      • optional: Makes IPsec enforcement optional. To add IPsec, set this flag to true. Once IPsec has been successfully installed, set this flag back to false and redeploy.
      • esp_proposals: You can modify the ESP (Quick Mode) encryption and integrity algorithms. The default, aes128gcm16!, is 128 bit AES-GCM with 128 bit ICV for both encryption and integrity.
      • ike_proposals: You can modify the IKE (Main Mode) encryption and integrity algorithms, and the Diffie-Hellman group. The default, aes128-sha256-modp2048!, is 128 bit AES-CBC for encryption, SHA2_256_128 HMAC for integrity, and Group 14 for Diffie-Hellman.

Note: To modify the configuration in an existing deployment, you must update the manifest file and redeploy.

Download and Deploy the IPsec Add-On

Warning: Adding the IPsec add-on version v1.5.x to an existing deployment results in app downtime. Deploy series v1.6.x and later to minimize downtime.

After deploying Ops Manager, perform the following steps to download and deploy the IPsec add-on:

  1. Download the IPsec add-on software binary from the Pivotal Network to your local machine.

  2. Copy the software binary to your Ops Manager instance.

    $ scp -i PATH/TO/PRIVATE/KEY ipsec-release.tar.gz ubuntu@YOUR-OPS-MANAGER-VM-IP:

  3. Copy the IPsec manifest file to your Ops Manager instance.

    $ scp -i PATH/TO/PRIVATE/KEY ipsec-addon.yml ubuntu@YOUR-OPS-MANAGER-VM-IP:

  4. SSH into Ops Manager.

    $ ssh -i PATH-TO-PRIVATE-KEY ubuntu@YOUR-OPS-MANAGER-VM-IP

  5. On the Ops Manager VM, navigate to the software binary location in your working directory.

    $ cd PATH-TO-BINARY

  6. On the Ops Manager VM, target the internal IP address of your BOSH Director. When prompted, enter your BOSH Director credentials. To retrieve your BOSH Director credentials, navigate to Ops Manager, click the Credentials tab, and click Link to Credential next to Director Credentials.

    $ bosh --ca-cert /var/tempest/workspaces/default/root_ca_certificate target YOUR-BOSH-DIRECTOR-INTERNAL-IP
    Target set to 'p-bosh'
    Your username: director
    Enter password: ******************
    Logged in as 'director'
    

  7. Upload your release.

    $ bosh upload release PATH-TO-BINARY/BINARY-NAME.tar

  8. Optionally, from the command line, confirm that the upload of the IPsec software binary completed. You should see the IPsec binary file.

    $ bosh releases

  9. Update your runtime configuration to include the IPsec add-on.

    $ bosh update runtime-config PATH/ipsec-addon.yml

  10. Verify your runtime configuration changes match what you specified in the IPsec manifest file.

    $ bosh runtime-config
    Acting as user 'admin' on 'micro'
    
    releases:
     - {name: ipsec, version: 1.0.0}
    
    addons:
     name: ipsec-addon
      jobs:
      - name: ipsec
        release: ipsec
    ...
    
  11. If you have already deployed Elastic Runtime and are adding IPsec to an existing deployment, complete the following steps.

    • For v1.5.x: Scale down the number of Consul instances to 1. See the Scaling Elastic Runtime topic for more information.
    • For v1.6.x and above: Set the optional flag to true. You do not need to scale the number of Consul instances.
  12. Navigate to your Installation Dashboard in Ops Manager.

Note: For v1.5.x, you cannot revert the following step without downtime. To minimize the impact, we recommend upgrading to the v1.6.x series before removing IPsec from the deployment.

  1. Click Apply Changes

    • For v1.6.x and above:
      • Wait for the installation to complete.
      • Set the optional flag to false.
      • Update the runtime config.
        $ bosh update runtime-config PATH/ipsec-addon.yml
      • Navigate to your Installation Dashboard.
      • Click Apply Changes.
  2. Secure the sensitive information in the ipsec-addon.yml file. Pivotal recommends encrypting the file and moving it to a secure location.

  3. For v1.5.x: If you scaled down the number of Consul instances in your existing Elastic Runtime deployment, scale the number of instances back to the previous value.

  4. If you have not installed the Elastic Runtime tile, deploy Elastic Runtime by following the installation instructions for your IaaS. See the Installing Pivotal Cloud Foundry topic for more information.

Verify Your IPsec Installation

After installing IPsec and deploying Elastic Runtime, perform the following steps to verify your IPsec installation:

  1. Run bosh vms to list the job VMs in your deployment. Choose the job name and index of any VM. The exact VM does not matter, because installing the IPsec add-on loads IPsec on all VMs deployed by Ops Manager.

  2. Run bosh ssh JOB-NAME/INDEX to open an SSH connection into the VM.

  3. Run sudo su - to enter the root environment with root privileges.

  4. Run monit summary to confirm that your ipsec job is listed as a bosh job.

    The Monit daemon 5.2.5 uptime: 18h 32m
    ...
    Process 'ipsec'                     running
    System 'system_localhost'           running
    

  5. Run PATH-TO-IPSEC/ipsec statusall to confirm that IPsec is running. If IPsec is not running, this command produces no output.

    $ /var/vcap/packages/strongswan-5.3.5/sbin/ipsec statusall
    Status of IKE charon daemon (strongSwan 5.3.5, Linux 3.19.0-56-generic, x86_64):
    uptime: 18 hours, since Mar 16 23:58:50 2016
    malloc: sbrk 2314240, mmap 0, used 1182400, free 1131840
    worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 206
    loaded plugins: charon aes sha1 sha2 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pem gmp xcbc cmac hmac attr kernel-netlink socket-default stroke
    Listening IP addresses:
    10.10.5.66
    Connections:
    ipsec-10.10.4.0/24:  %any...%any  IKEv1/2
    ipsec-10.10.4.0/24:   local:  [CN=test-cert-1-ca-1] uses public key authentication
    ipsec-10.10.4.0/24:    cert:  "CN=test-cert-1-ca-1"
    ipsec-10.10.4.0/24:   remote: uses public key authentication
    ipsec-10.10.9.0/24:   child:  10.10.5.66/32 === 10.10.9.0/24 TRANSPORT
    no-ipsec-10.10.4.1/32:  %any...%any  IKEv1/2
    no-ipsec-10.10.4.1/32:   local:  uses public key authentication
    no-ipsec-10.10.4.1/32:   remote: uses public key authentication
    no-ipsec-10.10.4.1/32:   child:  dynamic === 10.10.4.1/32 PASS
    Shunted Connections:
    no-ipsec-10.10.4.1/32:  dynamic === 10.10.4.1/32 PASS
    no-ipsec-10.10.5.1/32:  dynamic === 10.10.5.1/32 PASS
    no-ipsec-10.10.6.1/32:  dynamic === 10.10.6.1/32 PASS
    Routed Connections:
    ipsec-10.10.9.0/24{6}:  ROUTED, TRANSPORT, reqid 6
    ipsec-10.10.9.0/24{6}:   10.10.5.66/32 === 10.10.9.0/24
    ipsec-10.10.8.0/24{5}:  ROUTED, TRANSPORT, reqid 5
    ipsec-10.10.4.0/24{1}:   10.10.5.66/32 === 10.10.4.0/24
    Security Associations (45 up, 0 connecting):
    ipsec-10.10.4.0/24[459]: ESTABLISHED 13 seconds ago, 10.10.5.66[CN=test-cert-1-ca-1]...10.10.4.38[CN=test-cert-1-ca-1]
    ipsec-10.10.4.0/24{1527}:   10.10.5.66/32 === 10.10.4.38/32
    ...
    

Generate a Self-Signed Certificate

Note: Use a self-signed certificate only for development or test environments. Do not use a self-signed certificate for a production deployment. The scripts below generate private keys in a certs subdirectory.

To generate a self-signed certificate for your IPsec manifest, you can use either openssl or certstrap. Follow the instructions for your preferred method below.

Rerunning the scripts overwrites your current keys and the certificates.

Generate a Self-Signed Certificate with OpenSSL

  1. Download the openssl-create-ipsec-certs.sh bash script.
  2. Navigate to the directory where you downloaded the script:
    $ cd ~/workspace
  3. Change the permissions of the script:
    $ chmod u+x openssl-create-ipsec-certs.sh
  4. Run the script:
    $ ./openssl-create-ipsec-certs.sh

Generate a Self-Signed Certificate with Certstrap

  1. Download and install Go.
  2. Download the certstrap bash script.
  3. Navigate to the directory where you downloaded the script:
    $ cd ~/workspace
  4. Change the permissions of the script:
    $ chmod u+x certstrap-create-ipsec-certs.sh
  5. Run the script:
    $ ./certstrap-create-ipsec-certs.sh
Create a pull request or raise an issue on the source for this page in GitHub