Restoring a WAN Connection

This sequence of steps restores a bidirectional WAN connection to a service instance when the service instance has been restored from backup.

In this description, cluster A is an existing, running service instance that was formerly connected to cluster B over a WAN connection, and cluster B is the service that has been freshly restored from backup, and needs to be reconnected.

At this point in the process, cluster B should have the following characteristics:

All data restored from backup
The same distributed_system_id as its predecessor
A freshly-generated service key

If the restored distributed_system_id matches that in the backup, then there is no need to recreate gateway senders and receivers; they are already defined. This procedure manually updates gateway senders with pointers to remote locators and credentials, re-enabling their ability to connect across the WAN without authentication errors.

Obtain the service key for cluster A. The service key contains generated credentials, in a JSON element called remote_cluster_info, that enable other clusters (Cluster B in this example) to communicate with Cluster A:

$ cf service-key A k1

The contents of the service key differ based upon the cluster configuration, such as whether an authentication and enterprise single sign-on (SSO) system such as LDAP has been configured. Here is sample output from cf service-key A k1:

Getting key k1 for service instance A as admin...

{
 "distributed_system_id": "1",
 "gfsh_login_string": "connect
 --url=https://cloudcache-url.com/gemfire/v1
 --user=cluster_operator_user --password=pass --skip-ssl-validation",
 "locators": [
  "id1.locator.services-subnet.service-instance-id.bosh[55221]",
  "id2.locator.services-subnet.service-instance-id.bosh[55221]",
  "id3.locator.services-subnet.service-instance-id.bosh[55221]"
 ],
 "remote_cluster_info": {
  "recursors": {
   "services-subnet.service-instance-id.bosh": [
    "10.0.8.6:1053",
    "10.0.8.7:1053",
    "10.0.8.5:1053"
   ]
  },
  "remote_locators": [
   "id1.locator.services-subnet.service-instance-id.bosh[55221]",
   "id2.locator.services-subnet.service-instance-id.bosh[55221]",
   "id3.locator.services-subnet.service-instance-id.bosh[55221]"
  ],
  "trusted_sender_credentials": [
   {
    "password": "gws-GHI-password",
    "username": "gateway_sender_GHI"
   }
  ]
 },
 "urls": {
  "gfsh": "https://cloudcache-1.example.com/gemfire/v1",
  "pulse": "https://cloudcache-1.example.com/pulse"
 },
 "users": [
  {
   "password": "cl-op-ABC-password",
   "roles": [
    "cluster_operator"
   ],
   "username": "cluster_operator_ABC"
  },
  {
   "password": "dev-DEF-password",
   "roles": [
    "developer"
   ],
   "username": "developer_DEF"
  }
 ],
 "wan": {}
}

Obtain the service key of cluster B:

$ cf service-key B k2

As above, the service key contains generated credentials, in a JSON element called remote_cluster_info, that enable other clusters (Cluster A in this example) to communicate with Cluster B. Here is sample output from cf service-key B k2:

Getting key k2 for service instance destination as admin...

{
 "distributed_system_id": "2",
 "gfsh_login_string": "connect
 --url=https://cloudcache-url.com/gemfire/v1
 --user=cluster_operator_user --password=pass --skip-ssl-validation",
 "locators": [
  "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
  "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
  "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]"
 ],
 "remote_cluster_info": {
  "recursors": {
   "services-subnet-2.service-instance-id-2.bosh": [
    "10.1.16.7:1053",
    "10.1.16.6:1053",
    "10.1.16.8:1053"
   ]
  },
  "remote_locators": [
   "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
   "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
   "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]"
  ],
  "trusted_sender_credentials": [
   {
    "password": "gws-PQR-password",
    "username": "gateway_sender_PQR"
   }
  ]
 },
 "urls": {
  "gfsh": "https://cloudcache-2.example.com/gemfire/v1",
  "pulse": "https://cloudcache-2.example.com/pulse"
 },
 "users": [
  {
   "password": "cl-op-JKL-password",
   "roles": [
    "cluster_operator"
   ],
   "username": "cluster_operator_JKL"
  },
  {
   "password": "dev-MNO-password",
   "roles": [
    "developer"
   ],
   "username": "developer_MNO"
  }
 ],
 "wan": {}
}

Update the Cluster A service instance, using the -c option to specify a remote_clusters element that includes the contents of the Cluster B service key remote_cluster_info element, including the recursors array, remote_locators array, and trusted_sender_credentials. This allows Cluster A to communicate with Cluster B, and to accept data from Cluster B:

$ cf update-service A -c '
{
  "remote_clusters":[
  {
    "recursors": {
     "services-subnet-2.service-instance-id-2.bosh": [
      "10.1.16.7:1053",
      "10.1.16.6:1053",
      "10.1.16.8:1053"
     ]
    },
    "remote_locators": [
     "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
     "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
     "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]"
    ],
    "trusted_sender_credentials": [
     {
      "password": "gws-PQR-password",
      "username": "gateway_sender_PQR"
     }
    ]
   }
  }
 ]
}'
Updating service instance A as admin

To verify that a service instance has been correctly updated, delete and recreate the cluster service key. The recreated service key will have the same user identifiers and passwords as its predecessor, and will reflect the changes you specified in the recent cf update-service commands. In particular, the wan{} element at the end of a cluster’s service key should be populated with the other cluster’s remote connection information. For example, to verify that the Cluster A service key was updated correctly, log in as Cluster A administrator and issue these commands to delete and recreate the Cluster A service key:

$ cf delete-service-key A k1
  ...
$ cf create-service-key A k1

Verify that the wan{} field of the Cluster A service key contains a remote_clusters element which specifies contact information for Cluster B, including Cluster B’s recursors array, remote_locators array, and trusted_sender_credentials:

Getting key k1 for service instance A as admin...

{
 "distributed_system_id": "1",
 "gfsh_login_string": "connect
 --url=https://cloudcache-url.com/gemfire/v1
 --user=cluster_operator_user --password=pass --skip-ssl-validation",
 "locators": [
  "id1.locator.services-subnet.service-instance-id.bosh[55221]",
  "id2.locator.services-subnet.service-instance-id.bosh[55221]",
  "id3.locator.services-subnet.service-instance-id.bosh[55221]"
 ],
 "remote_cluster_info": {
  "recursors": {
   "services-subnet.service-instance-id.bosh": [
    "10.0.8.6:1053",
    "10.0.8.7:1053",
    "10.0.8.5:1053"
   ]
  },
  "remote_locators": [
   "id1.locator.services-subnet.service-instance-id.bosh[55221]",
   "id2.locator.services-subnet.service-instance-id.bosh[55221]",
   "id3.locator.services-subnet.service-instance-id.bosh[55221]"
  ],
  "trusted_sender_credentials": [
   {
    "password": "gws-GHI-password",
    "username": "gateway_sender_GHI"
   }
  ]
 },
 "urls": {
  "gfsh": "https://cloudcache-1.example.com/gemfire/v1",
  "pulse": "https://cloudcache-1.example.com/pulse"
 },
 "users": [
  {
   "password": "cl-op-ABC-password",
   "roles": [
    "cluster_operator"
   ],
   "username": "cluster_operator_ABC"
  },
  {
   "password": "dev-DEF-password",
   "roles": [
    "developer"
   ],
   "username": "developer_DEF"
  }
 ],
 "wan": {
  "remote_clusters": [
   {
    "recursors": {
     "services-subnet-2.service-instance-id-2.bosh": [
      "10.1.16.7:1053",
      "10.1.16.6:1053",
      "10.1.16.8:1053"
     ]
    },
    "remote_locators": [
     "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
     "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
     "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]"
    ],
    "trusted_sender_credentials": [
     {
      "password": "gws-PQR-password",
      "username": "gateway_sender_PQR"
     }
    ]
   }
  ]
 }
}

Update the Cluster B service instance, using the -c option to specify a remote_clusters element that includes the contents of the Cluster A service key remote_cluster_info element, including the recursors array, remote_locators array, and trusted_sender_credentials. This allows Cluster B to communicate with Cluster A, and to accept data from Cluster A:

$ cf update-service B -c '
{
  "remote_clusters":[
  {
    "recursors": {
      "services-subnet.service-instance-id.bosh": [
        "10.0.8.5:1053",
        "10.0.8.7:1053",
        "10.0.8.6:1053"
      ]
    }
    "remote_locators":[
      "id1.locator.services-subnet.service-instance-id.bosh[55221]",
      "id2.locator.services-subnet.service-instance-id.bosh[55221]",
      "id3.locator.services-subnet.service-instance-id.bosh[55221]"
    "trusted_sender_credentials":[
    {
      "username": "gateway_sender_GHI",
      "password":"gws-GHI-password"
    }]
  }]
}'
Updating service instance B as admin

To verify that the Cluster B service key was updated correctly, log in as Cluster B administrator and issue these commands to delete and recreate the Cluster B service key:

$ cf delete-service-key B k2
  ...
$ cf create-service-key B k2

Verify that the wan{} field of the Cluster B service key contains a remote_clusters element which specifies contact information for Cluster A, including Cluster A’s recursors array, remote_locators array, and trusted_sender_credentials:

Getting key k1 for service instance B as admin...

{
 "distributed_system_id": "2",
 "gfsh_login_string": "connect
 --url=https://cloudcache-url.com/gemfire/v1
 --user=cluster_operator_user --password=pass --skip-ssl-validation",
 "locators": [
  "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
  "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
  "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]"
 ],
 "remote_cluster_info": {
  "recursors": {
   "services-subnet-2.service-instance-id-2.bosh": [
    "10.1.16.7:1053",
    "10.1.16.6:1053",
    "10.1.16.8:1053"
   ]
  },
  "remote_locators": [
   "id1.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
   "id2.locator.services-subnet-2.service-instance-id-2.bosh[55221]",
   "id3.locator.services-subnet-2.service-instance-id-2.bosh[55221]"
  ],
  "trusted_sender_credentials": [
   {
    "password": "gws-PQR-password",
    "username": "gateway_sender_PQR"
   }
  ]
 },
 "urls": {
  "gfsh": "https://cloudcache-2.example.com/gemfire/v1",
  "pulse": "https://cloudcache-2.example.com/pulse"
 },
 "users": [
  {
   "password": "cl-op-JKL-password",
   "roles": [
    "cluster_operator"
   ],
   "username": "cluster_operator_JKL"
  },
  {
   "password": "dev-MNO-password",
   "roles": [
    "developer"
   ],
   "username": "developer_MNO"
  }
 ],
 "wan": {
  "remote_clusters": [
   {
    "recursors": {
     "services-subnet.service-instance-id.bosh": [
      "10.0.8.6:1053",
      "10.0.8.7:1053",
      "10.0.8.5:1053"
     ]
    },
    "remote_locators": [
     "id1.locator.services-subnet.service-instance-id.bosh[55221]",
     "id2.locator.services-subnet.service-instance-id.bosh[55221]",
     "id3.locator.services-subnet.service-instance-id.bosh[55221]"
    ],
    "trusted_sender_credentials": [
     {
      "password": "gws-GHI-password",
      "username": "gateway_sender_GHI"
     }
    ]
   }
  ]
 }
}

To verify that the connection has been successfully restored, see Verify Bidirectional WAN Setup.

Create a pull request or raise an issue on the source for this page in GitHub