Restoring a WAN Connection
This sequence of steps restores a bidirectional WAN connection to a service instance when the service instance has been restored from backup.
In this description, cluster A is an existing, running service instance that was formerly connected to cluster B over a WAN connection, and cluster B is the service that has been freshly restored from backup, and needs to be reconnected.
At this point in the process, cluster B should have the following characteristics:
- All data restored from backup
- The same
distributed_system_id
as its predecessor - A freshly-generated service key
If the restored distributed_system_id
matches that in the backup, then there is no need to recreate
gateway senders and receivers; they are already defined. This procedure manually updates gateway senders with
pointers to remote locators and credentials, re-enabling their ability to connect across the WAN
without authentication errors.
Obtain the service key for cluster A. The service key contains generated credentials, in a JSON element called
remote_cluster_info
, that enable other clusters (Cluster B in this example) to communicate with Cluster A:$ cf service-key A k1
The contents of the service key differ based upon the cluster configuration, such as whether an authentication and enterprise single sign-on (SSO) system such as LDAP has been configured. Here is sample output from
cf service-key A k1
:Getting key k1 for service instance A as admin... { "distributed_system_id": "1", "gfsh_login_string": "connect --url=https://cloudcache-url.com/gemfire/v1 --user=cluster_operator_user --password=pass --skip-ssl-validation", "locators": [ "id1.locator.services-subnet.service-instance-id.bosh[55221]", "id2.locator.services-subnet.service-instance-id.bosh[55221]", "id3.locator.services-subnet.service-instance-id.bosh[55221]" ], "remote_cluster_info": { "recursors": { "services-subnet.service-instance-id.bosh": [ "10.0.8.6:1053", "10.0.8.7:1053", "10.0.8.5:1053" ] }, "remote_locators": [ "id1.locator.services-subnet.service-instance-id.bosh[55221]", "id2.locator.services-subnet.service-instance-id.bosh[55221]", "id3.locator.services-subnet.service-instance-id.bosh[55221]" ], "trusted_sender_credentials": [ { "password": "gws-GHI-password", "username": "gateway_sender_GHI" } ] }, "urls": { "gfsh": "https://cloudcache-1.example.com/gemfire/v1", "pulse": "https://cloudcache-1.example.com/pulse" }, "users": [ { "password": "cl-op-ABC-password", "roles": [ "cluster_operator" ], "username": "cluster_operator_ABC" }, { "password": "dev-DEF-password", "roles": [ "developer" ], "username": "developer_DEF" } ], "wan": {} }
Obtain the service key of cluster B:
$ cf service-key B k2
As above, the service key contains generated credentials, in a JSON element called
remote_cluster_info
, that enable other clusters (Cluster A in this example) to communicate with Cluster B. Here is sample output fromcf service-key B k2
:Getting key k2 for service instance destination as admin... { "distributed_system_id": "2", "gfsh_login_string": "connect --url=https://cloudcache-url.com/gemfire/v1 --user=cluster_operator_user --password=pass --skip-ssl-validation", "locators": [ "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]" ], "remote_cluster_info": { "recursors": { "services-subnet-2.service-instance-id-2.bosh": [ "10.1.16.7:1053", "10.1.16.6:1053", "10.1.16.8:1053" ] }, "remote_locators": [ "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]" ], "trusted_sender_credentials": [ { "password": "gws-PQR-password", "username": "gateway_sender_PQR" } ] }, "urls": { "gfsh": "https://cloudcache-2.example.com/gemfire/v1", "pulse": "https://cloudcache-2.example.com/pulse" }, "users": [ { "password": "cl-op-JKL-password", "roles": [ "cluster_operator" ], "username": "cluster_operator_JKL" }, { "password": "dev-MNO-password", "roles": [ "developer" ], "username": "developer_MNO" } ], "wan": {} }
Update the Cluster A service instance, using the
-c
option to specify aremote_clusters
element that includes the contents of the Cluster B service keyremote_cluster_info
element, including therecursors
array,remote_locators
array, andtrusted_sender_credentials
. This allows Cluster A to communicate with Cluster B, and to accept data from Cluster B:$ cf update-service A -c ' { "remote_clusters":[ { "recursors": { "services-subnet-2.service-instance-id-2.bosh": [ "10.1.16.7:1053", "10.1.16.6:1053", "10.1.16.8:1053" ] }, "remote_locators": [ "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]" ], "trusted_sender_credentials": [ { "password": "gws-PQR-password", "username": "gateway_sender_PQR" } ] } } ] }' Updating service instance A as admin
To verify that a service instance has been correctly updated, delete and recreate the cluster service key. The recreated service key will have the same user identifiers and passwords as its predecessor, and will reflect the changes you specified in the recent
cf update-service
commands. In particular, thewan{}
element at the end of a cluster’s service key should be populated with the other cluster’s remote connection information. For example, to verify that the Cluster A service key was updated correctly, log in as Cluster A administrator and issue these commands to delete and recreate the Cluster A service key:$ cf delete-service-key A k1 ... $ cf create-service-key A k1
Verify that the
wan{}
field of the Cluster A service key contains aremote_clusters
element which specifies contact information for Cluster B, including Cluster B’srecursors
array,remote_locators
array, andtrusted_sender_credentials
:Getting key k1 for service instance A as admin... { "distributed_system_id": "1", "gfsh_login_string": "connect --url=https://cloudcache-url.com/gemfire/v1 --user=cluster_operator_user --password=pass --skip-ssl-validation", "locators": [ "id1.locator.services-subnet.service-instance-id.bosh[55221]", "id2.locator.services-subnet.service-instance-id.bosh[55221]", "id3.locator.services-subnet.service-instance-id.bosh[55221]" ], "remote_cluster_info": { "recursors": { "services-subnet.service-instance-id.bosh": [ "10.0.8.6:1053", "10.0.8.7:1053", "10.0.8.5:1053" ] }, "remote_locators": [ "id1.locator.services-subnet.service-instance-id.bosh[55221]", "id2.locator.services-subnet.service-instance-id.bosh[55221]", "id3.locator.services-subnet.service-instance-id.bosh[55221]" ], "trusted_sender_credentials": [ { "password": "gws-GHI-password", "username": "gateway_sender_GHI" } ] }, "urls": { "gfsh": "https://cloudcache-1.example.com/gemfire/v1", "pulse": "https://cloudcache-1.example.com/pulse" }, "users": [ { "password": "cl-op-ABC-password", "roles": [ "cluster_operator" ], "username": "cluster_operator_ABC" }, { "password": "dev-DEF-password", "roles": [ "developer" ], "username": "developer_DEF" } ], "wan": { "remote_clusters": [ { "recursors": { "services-subnet-2.service-instance-id-2.bosh": [ "10.1.16.7:1053", "10.1.16.6:1053", "10.1.16.8:1053" ] }, "remote_locators": [ "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]" ], "trusted_sender_credentials": [ { "password": "gws-PQR-password", "username": "gateway_sender_PQR" } ] } ] } }
Update the Cluster B service instance, using the
-c
option to specify aremote_clusters
element that includes the contents of the Cluster A service keyremote_cluster_info
element, including therecursors
array,remote_locators
array, andtrusted_sender_credentials
. This allows Cluster B to communicate with Cluster A, and to accept data from Cluster A:$ cf update-service B -c ' { "remote_clusters":[ { "recursors": { "services-subnet.service-instance-id.bosh": [ "10.0.8.5:1053", "10.0.8.7:1053", "10.0.8.6:1053" ] } "remote_locators":[ "id1.locator.services-subnet.service-instance-id.bosh[55221]", "id2.locator.services-subnet.service-instance-id.bosh[55221]", "id3.locator.services-subnet.service-instance-id.bosh[55221]" "trusted_sender_credentials":[ { "username": "gateway_sender_GHI", "password":"gws-GHI-password" }] }] }' Updating service instance B as admin
To verify that the Cluster B service key was updated correctly, log in as Cluster B administrator and issue these commands to delete and recreate the Cluster B service key:
$ cf delete-service-key B k2 ... $ cf create-service-key B k2
Verify that the
wan{}
field of the Cluster B service key contains aremote_clusters
element which specifies contact information for Cluster A, including Cluster A’srecursors
array,remote_locators
array, andtrusted_sender_credentials
:Getting key k1 for service instance B as admin... { "distributed_system_id": "2", "gfsh_login_string": "connect --url=https://cloudcache-url.com/gemfire/v1 --user=cluster_operator_user --password=pass --skip-ssl-validation", "locators": [ "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]" ], "remote_cluster_info": { "recursors": { "services-subnet-2.service-instance-id-2.bosh": [ "10.1.16.7:1053", "10.1.16.6:1053", "10.1.16.8:1053" ] }, "remote_locators": [ "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]", "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]" ], "trusted_sender_credentials": [ { "password": "gws-PQR-password", "username": "gateway_sender_PQR" } ] }, "urls": { "gfsh": "https://cloudcache-2.example.com/gemfire/v1", "pulse": "https://cloudcache-2.example.com/pulse" }, "users": [ { "password": "cl-op-JKL-password", "roles": [ "cluster_operator" ], "username": "cluster_operator_JKL" }, { "password": "dev-MNO-password", "roles": [ "developer" ], "username": "developer_MNO" } ], "wan": { "remote_clusters": [ { "recursors": { "services-subnet.service-instance-id.bosh": [ "10.0.8.6:1053", "10.0.8.7:1053", "10.0.8.5:1053" ] }, "remote_locators": [ "id1.locator.services-subnet.service-instance-id.bosh[55221]", "id2.locator.services-subnet.service-instance-id.bosh[55221]", "id3.locator.services-subnet.service-instance-id.bosh[55221]" ], "trusted_sender_credentials": [ { "password": "gws-GHI-password", "username": "gateway_sender_GHI" } ] } ] } }
To verify that the connection has been successfully restored, see Verify Bidirectional WAN Setup.