Set Up an Additional Bidirectional Interaction

This section describes how to add a bidirectional TLS-encrypted WAN connection to an additional VMware Tanzu GemFire for VMs service instance, once an initial setup is in place for a first pair of Tanzu GemFire service instances.

Call the first pair of Tanzu GemFire service instances Cluster A and Cluster B, which reside on Foundation A and Foundation B, respectively. This set of directions sets up an interaction between Cluster A and and a third service instance, Cluster C, which resides on third foundation, Foundation C.

Assumptions

  • You have already established a TLS-encrypted WAN connection between a service instance on Foundation A, referred to here as Cluster A, and a service instance on Foundation B, which we’ll call Cluster B, following the Set Up a Bidirectional System procedure.
  • You wish to add an additional TLS-encrypted WAN connection between Cluster A and a third service instance on Foundation C, which we will call Cluster C.
  • The connection will be bidirectional, such that an operation on either cluster can be replicated in the other.
  • The Preparing for TLS procedure has been followed for Foundation C, establishing a CredHub “/services/tls_ca” certificate.
  • The Establishing Mutually Trusted TLS Credentials procedure has been followed, establishing mutually trusted TLS credentials between Foundations A and C.

Get Cluster A’s Remote Credentials

  1. Log into Foundation A using Foundation A Cloud Foundry credentials.

  2. View the Cluster A service key and save a copy of the remote_cluster_info element, which includes the remote_locators array and trusted_sender_credentials. These are the credentials that enable other clusters to communicate with Cluster A.

    $ cf service-key wan1 k1
    Getting key k1 for service instance wan1 as admin...
    
    {
     "distributed_system_id": "1",
     "gfsh_login_string": "connect
     --url=https://cloudcache-url.com/gemfire/v1
     --user=cluster_operator_user --password=pass --skip-ssl-validation",
     "locators": [
      "id1.locator.services-subnet.service-instance-id.bosh[55221]",
      "id2.locator.services-subnet.service-instance-id.bosh[55221]",
      "id3.locator.services-subnet.service-instance-id.bosh[55221]"
     ],
     "remote_cluster_info": {
      "recursors": {
       "services-subnet.service-instance-id.bosh": [
        "10.0.8.6:1053",
        "10.0.8.7:1053",
        "10.0.8.5:1053"
       ]
      },
      "remote_locators": [
       "id1.locator.services-subnet.service-instance-id.bosh[55221]",
       "id2.locator.services-subnet.service-instance-id.bosh[55221]",
       "id3.locator.services-subnet.service-instance-id.bosh[55221]"
      ],
      "trusted_sender_credentials": [
       {
        "password": "gws-GHI-password",
        "username": "gateway_sender_GHI"
       }
      ]
     },
     "tls-enabled": "true",
     "urls": {
      "gfsh": "https://cloudcache-1.example.com/gemfire/v1",
      "pulse": "https://cloudcache-1.example.com/pulse"
     },
     "users": [
      {
       "password": "cl-op-ABC-password",
       "roles": [
        "cluster_operator"
       ],
       "username": "cluster_operator_ABC"
      },
      {
       "password": "dev-DEF-password",
       "roles": [
        "developer"
       ],
       "username": "developer_DEF"
      }
     ],
     "wan": {
      "remote_clusters": [
       {
        "remote_locators": [
         "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
         "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
         "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]"
        ],
        "trusted_sender_credentials": [
         "gateway_sender_PQR"
        ]
       }
      ]
     }
    }
    

Create the New Cluster

The VMware Tanzu GemFire cluster within each service instance uses an identifier called a distributed_system_id. This example assumes the assignment of distributed_system_id = 1 for cluster A, distributed_system_id = 2 for cluster B, and distributed_system_id = 3 for cluster C.

  1. Log into Foundation C using Foundation C Cloud Foundry credentials.

  2. Use the cf create-service command to create a service instance, which we will call Cluster C in this example. In the cf create-service command, use the -c option to specify the following arguments:

    • "tls" : true to enable TLS
    • "distributed_system_id" : 3 to set the distributed system id of Cluster C to “3”
    • remote_clusters" : [CLUSTER-A-CREDENTIALS] to enable communication with Cluster A, where CLUSTER-A-CREDENTIALS are the contents of Cluster A’s remote_cluster_info element.

    The following command specifies the Cluster A credentials you copied in an earlier step, enabling Cluster C to communicate with and receive data from Cluster A:

    $ cf create-service p-cloudcache wan-cluster wan3 -c '{
    "tls" : true,
    "distributed_system_id" : 3,
      "remote_clusters":[
       {
        "recursors": {
          "services-subnet.service-instance-id.bosh": [
            "10.0.8.5:1053",
            "10.0.8.7:1053",
            "10.0.8.6:1053"
          ]
        },
        "remote_locators":[
          "id1.locator.services-subnet.service-instance-id.bosh[55221]",
          "id2.locator.services-subnet.service-instance-id.bosh[55221]",
          "id3.locator.services-subnet.service-instance-id.bosh[55221]"
        ],
        "trusted_sender_credentials":[
        {
          "username": "gateway_sender_GHI",
          "password":"gws-GHI-password"
        }]
      }]
    }'
    

    Verify the completion of service creation prior to continuing to the next step. Output from the cf services command will show the last operation as create succeeded when service creation is completed.

  3. While logged in to Foundation C, create a service key for Cluster C.

    $ cf create-service-key wan3 k3
    Creating service key wan3 for service instance k3 ...
    OK
    

    Display the service key and save a copy of the remote_cluster_info element, which includes the recursors array, remote_locators array, and trusted_sender_credentials. These are the credentials that enable other clusters to communicate with Cluster C.

    $ cf service-key wan3 k3
    
    Getting key k3 for service instance destination as admin...
    
    {
     "distributed_system_id": "3",
     "gfsh_login_string": "connect
     --url=https://cloudcache-url.com/gemfire/v1
     --user=cluster_operator_user --password=pass --skip-ssl-validation",
     "locators": [
      "id1.locator.services-subnet-3.service-instance-id-3.bosh[55221]",
      "id2.locator.services-subnet-3.service-instance-id-3.bosh[55221]",
      "id3.locator.services-subnet-3.service-instance-id-3.bosh[55221]"
     ],
     "remote_cluster_info": {
      "recursors": {
       "services-subnet-3.service-instance-id-3.bosh": [
        "10.2.32.7:1053",
        "10.2.32.6:1053",
        "10.2.32.8:1053"
       ]
      },
      "remote_locators": [
       "id1.locator.services-subnet-3.service-instance-id-3.bosh[55221]",
       "id2.locator.services-subnet-3.service-instance-id-3.bosh[55221]",
       "id3.locator.services-subnet-3.service-instance-id-3.bosh[55221]"
      ],
      "trusted_sender_credentials": [
       {
        "username": "gateway_sender_XYZ",
        "password": "gws-XYZ-password"
       }
      ]
     },
     "tls-enabled": "true",
     "urls": {
      "gfsh": "https://cloudcache-3.example.com/gemfire/v1",
      "pulse": "https://cloudcache-3.example.com/pulse"
     },
     "users": [
      {
       "password": "cl-op-STU-password",
       "roles": [
        "cluster_operator"
       ],
       "username": "cluster_operator_STU"
      },
      {
       "password": "dev-VWX-password",
       "roles": [
        "developer"
       ],
       "username": "developer_VWX"
      }
     ],
     "wan": {
      "remote_clusters": [
       {
        "remote_locators": [
          "id1.locator.services-subnet.service-instance-id.bosh[55221]",
          "id2.locator.services-subnet.service-instance-id.bosh[55221]",
          "id3.locator.services-subnet.service-instance-id.bosh[55221]"
        ],
        "trusted_sender_credentials": [
         "gateway_sender_GHI"
        ]
       }
      ]
     }
    }
    
  4. While logged in to Foundation A, update the Cluster A service instance to add the Cluster C credentials, using the -c option to specify a remote_clusters element that includes the contents of the Cluster C service key remote_cluster_info element, including the recursors array, remote_locators array, and trusted_sender_credentials. This allows Cluster A to communicate with Cluster C, and to accept data from Cluster C.

    IMPORTANT: You must also retain the credentials for all clusters with which Cluster A interacts, in this example those of Cluster B. The existing credentials can be copied from the remote_cluster_info entry in Cluster B’s service key, before updating the Cluster A service instance to append credentials for the additional cluster.

    The following example adds Cluster C’s credentials to Cluster A’s remote_clusters element:

    $ cf update-service wan1 -c '
    {
      "remote_clusters":[
       {
        "recursors": {
         "services-subnet-2.service-instance-id-2.bosh": [
          "10.1.16.7:1053",
          "10.1.16.6:1053",
          "10.1.16.8:1053"
         ]
        },
        "remote_locators": [
         "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
         "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
         "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]"
        ],
        "trusted_sender_credentials": [
         {
          "username": "gateway_sender_PQR",
          "password": "gws-PQR-password"
         }
        ]
       },
       {
        "recursors": {
         "services-subnet-3.service-instance-id-3.bosh": [
          "10.2.32.7:1053",
          "10.2.32.6:1053",
          "10.2.32.8:1053"
         ]
        },
        "remote_locators": [
         "id1.locator.services-subnet-3.service-instance-id-3.bosh[55221]",
         "id2.locator.services-subnet-3.service-instance-id-3.bosh[55221]",
         "id3.locator.services-subnet-3.service-instance-id-3.bosh[55221]"
        ],
        "trusted_sender_credentials": [
         {
          "username": "gateway_sender_XYZ",
          "password": "gws-XYZ-password"
         }
        ]
       }
     ]
    }'
    Updating service instance wan1 as admin
    

    Verify the completion of the service update prior to continuing to the next step. Output from the cf services command will show the last operation as update succeeded when service update is completed.

  5. To verify that a service instance has been correctly updated, delete and recreate the cluster service key. The recreated service key will have the same user identifiers and passwords as its predecessor, and will reflect the changes you specified in the recent cf update-service commands. In particular, the wan{} element at the end of a cluster’s service key should be populated with the other cluster’s remote connection information. For example, to verify that the Cluster A service key was updated correctly, log in as Cluster A administrator and issue these commands to delete and recreate the Cluster A service key:

    $ cf delete-service-key wan1 k1
      ...
    $ cf create-service-key wan1 k1
    

    Verify that the wan{} field of the Cluster A service key contains a remote_clusters element which specifies contact information for Cluster B, including Cluster B’s remote_locators array and trusted_sender_credentials:

    $ cf service-key wan1 k1
    
    Getting key k1 for service instance wan1 as admin...
    
    {
     "distributed_system_id": "1",
     "gfsh_login_string": "connect
     --url=https://cloudcache-url.com/gemfire/v1
     --user=cluster_operator_user --password=pass --skip-ssl-validation",
     "locators": [
      "id1.locator.services-subnet.service-instance-id.bosh[55221]",
      "id2.locator.services-subnet.service-instance-id.bosh[55221]",
      "id3.locator.services-subnet.service-instance-id.bosh[55221]"
     ],
     "remote_cluster_info": {
      "recursors": {
       "services-subnet.service-instance-id.bosh": [
        "10.0.8.6:1053",
        "10.0.8.7:1053",
        "10.0.8.5:1053"
       ]
      },
      "remote_locators": [
       "id1.locator.services-subnet.service-instance-id.bosh[55221]",
       "id2.locator.services-subnet.service-instance-id.bosh[55221]",
       "id3.locator.services-subnet.service-instance-id.bosh[55221]"
      ],
      "trusted_sender_credentials": [
       {
        "password": "gws-GHI-password",
        "username": "gateway_sender_GHI"
       }
      ]
     },
     "tls-enabled": "true",
     "urls": {
      "gfsh": "https://cloudcache-1.example.com/gemfire/v1",
      "pulse": "https://cloudcache-1.example.com/pulse"
     },
     "users": [
      {
       "password": "cl-op-ABC-password",
       "roles": [
        "cluster_operator"
       ],
       "username": "cluster_operator_ABC"
      },
      {
       "password": "dev-DEF-password",
       "roles": [
        "developer"
       ],
       "username": "developer_DEF"
      }
     ],
     "wan": {
      "remote_clusters":[
       {
        "remote_locators": [
         "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
         "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
         "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]"
        ],
        "trusted_sender_credentials": [
         "gateway_sender_PQR"
        ]
       },
       {
        "remote_locators": [
         "id1.locator.services-subnet-3.service-instance-id-3.bosh[55221]",
         "id2.locator.services-subnet-3.service-instance-id-3.bosh[55221]",
         "id3.locator.services-subnet-3.service-instance-id-3.bosh[55221]"
        ],
        "trusted_sender_credentials": [
         "gateway_sender_XYZ"
        ]
       }
      ]
     }
    }
    

Create Gateway Senders and Regions

  1. While logged in to Foundation A, use gfsh to create the cluster A gateway sender and alter the existing region.

    • Connect using gfsh following the instructions in Connect with gfsh over HTTPS with a role that is able to manage both the cluster and the data.
    • Create the cluster A gateway sender. The required remote-distributed-system-id option identifies the distributed-system-id of the destination cluster. It is 3 for this example:

      gfsh>create gateway-sender --id=send_to_3 --remote-distributed-system-id=3 --enable-persistence=true
      
    • Alter the existing cluster A region so that it specifies all gateway senders associated with the region. There are two gateway senders in this example, one that goes to cluster B and a second that goes to cluster C.

      gfsh>alter region --name=regionX --gateway-sender-id=send_to_2,send_to_3
      
  2. While logged in to Foundation C, use gfsh to create the cluster C gateway sender and region.

    • Connect using gfsh following the instructions in Connect with gfsh over HTTPS with a role that is able to manage both the cluster and the data.
    • Create the cluster C gateway sender:

      gfsh>create gateway-sender --id=send_to_1 --remote-distributed-system-id=1 --enable-persistence=true
      
    • Create the cluster C region:

      gfsh>create region --name=regionX --gateway-sender-id=send_to_1 --type=PARTITION_REDUNDANT