Switch over to the secondary site

This topic describes the operational procedures necessary

This procedure switches from the primary site to the secondary site when using a setup as outlined in Concepts for active-passive deployments together with the blueprints outlined in Building blocks active-passive deployments.

When to use this procedure

Use this procedure to gracefully take the primary offline.

Once the primary site is back online, use the guides Recover from an out-of-sync passive site and Switch back to the primary site to return to the original state with the primary site being active.

See the Multi-site deployments guide for different operational procedures.

Procedures

Infinispan Cluster

For the context of this guide, Site-A is the primary site and Site-B is the secondary site.

When you are ready to take a site offline, a good practice is to disable the replication towards it. This action prevents errors or delays when the channels are disconnected between the primary and the secondary site.

Procedures to transfer state from secondary to primary site

  1. Log in into your secondary site

  2. Connect into Infinispan Cluster using the Infinispan CLI tool:

    Command:
    kubectl -n keycloak exec -it pods/infinispan-0 -- ./bin/cli.sh --trustall --connect https://127.0.0.1:11222

    It asks for the username and password for the Infinispan cluster. Those credentials are the one set in the Deploy Infinispan for HA with the Infinispan Operator guide in the configuring credentials section.

    Output:
    Username: developer
    Password:
    [infinispan-0-29897@ISPN//containers/default]>
    The pod name depends on the cluster name defined in the Infinispan CR. The connection can be done with any pod in the Infinispan cluster.
  3. Disable the replication to the primary site by running the following command:

    Command:
    site take-offline --all-caches --site=site-a
    Output:
    {
      "offlineClientSessions" : "ok",
      "authenticationSessions" : "ok",
      "sessions" : "ok",
      "clientSessions" : "ok",
      "work" : "ok",
      "offlineSessions" : "ok",
      "loginFailures" : "ok",
      "actionTokens" : "ok"
    }
  4. Check the replication status is offline.

    Command:
    site status --all-caches --site=site-a
    Output:
    {
      "status" : "offline"
    }

    If the status is not offline, repeat the previous step.

The Infinispan cluster in the secondary site is ready to handle requests without trying to replicate to the primary site.

AWS Aurora Database

Assuming a Regional multi-AZ Aurora deployment, the current writer instance should be in the same region as the active Keycloak cluster to avoid latencies and communication across availability zones.

Switching the writer instance of Aurora will lead to a short downtime. The writer instance in the other site with a slightly longer latency might be acceptable for some deployments. Therefore, this situation might be deferred to a maintenance window or skipped depending on the circumstances of the deployment.

To change the writer instance, run a failover. This change will make the database unavailable for a short time. Keycloak will need to re-establish database connections.

To fail over the writer instance to the other AZ, issue the following command:

aws rds failover-db-cluster  --db-cluster-identifier ...

Keycloak Cluster

No action required.

Route53

To force Route53 to mark the primary site as not available, edit the health check in AWS to point to a non-existent route (/lb-check-switched-over). After some minutes, the clients will notice the change and traffic will gradually move over to the secondary site.

Further reading

See Concepts to automate Infinispan CLI commands on how to automate Infinispan CLI commands.

On this page