Migrating from multi-cluster v1 to v2

Overview of changes

The migration involves the following changes:

Table 1. Components removed
Component	Reason
External Infinispan cluster	Authentication sessions, action tokens and login failure counters are stored in the database instead.
Infinispan cross-site replication	No inter-site cache replication is needed; the database handles all cross-site data.
Infinispan credentials and TLS secrets	No longer required without the external Infinispan cluster.
Fencing automation (for example, AWS Lambda)	Split-brain detection and fencing for Infinispan are no longer necessary.
Infinispan-specific monitoring and alerts	Prometheus alerts for Infinispan cache health and cross-site connectivity are no longer needed.

Table 2. Components retained
Component	Notes
Database	The existing synchronously replicated database is retained. Plan for approximately double the CPU usage and write IOPS compared to v1.
Load balancer	The external load balancer is retained. Remove any health checks that target Infinispan endpoints. The load balancer should probe the Keycloak `/lb-check` endpoint.

Table 3. Configuration changes to Keycloak
Change	Details
Enable the `stateless` feature	Replaces the `multi-site` feature.
Set a unique cluster name per site	Required for cross-cluster cache invalidation via the database outbox.
Increase the database connection pool size	Some data is now stored in database instead of Infinispan.
Remove external Infinispan connection options	Options such as `cache-remote-host`, `cache-remote-port`, `cache-remote-username`, and `cache-remote-password` are no longer used.

Prerequisites

A running multi-cluster v1 deployment as described in the Multi-cluster deployments (v1) guide.
Familiarity with the v2 architecture, its requirements, and limitations as described in the Multi-cluster deployments (v2) guide and the Concepts for multi-cluster deployments (v2) guide.
A database sized for the increased load. With v2, database CPU usage and write IOPS can roughly double. Review and adjust the database instance size, storage IOPS, and connection pool sizing accordingly. See the Concepts for database connection pools guide for details.
Network latency between database nodes that meets the requirements.

A round-trip latency of less than 5 ms is suggested and below 10 ms is required, together with a reliable network between the zones to avoid unexpected problems with latency, throughput or connectivity.

Network latency and latency spikes amplify in the response time of the service and can lead to queued requests, timeouts, and failed requests. Networking problems can cause downtimes until the failure detection isolates problematic nodes.

Impact on volatile data

Since user sessions are already stored in the database in multi-cluster v1, users remain logged in after migrating to v2.

The following volatile data is lost during the migration:

In-progress authentication sessions: Users who are mid-login need to restart their login flow.
Login failure counters: Brute-force detection counters are reset. Any ongoing lockouts are cleared.
OAuth authorization codes: OAuth authorization codes that were stored in Infinispan are lost. Users with in-progress OAuth flows need to restart the authorization.

To minimize user impact, perform the migration during a maintenance window with low traffic.

Migrating on Kubernetes

This procedure assumes the Keycloak Operator is installed in each Kubernetes cluster.

Perform the following steps in each Kubernetes cluster, unless stated otherwise:

Scale down Keycloak in all sites.

Scaling down ensures a clean transition. No requests should be served during the migration.
```
kubectl --namespace keycloak scale keycloak keycloak --replicas=0
```
Wait until all Keycloak Pods are terminated in all sites before proceeding.

Prepare the updated Keycloak CR.

The following changes are required:

Replace the multi-site feature with stateless:

  features:
    enabled:
      - stateless (1)

1	Replaces the `multi-site` feature from v1.

If you are using an optimized custom image, the stateless feature is a build-time option and cannot be enabled through the CR alone. You must rebuild the image with --features=stateless included in the build command before applying this CR. See Using custom Keycloak images for details.

Add a unique cluster name for each site:
```
  additionalOptions:
    - name: spi-cache-embedded--default--cluster-name
      value: CLUSTER1 (1)
```
1 Use a distinct name for each site (for example, CLUSTER1 and CLUSTER2). No two sites may share the same name.
Remove the external Infinispan connection options cache-remote-host, cache-remote-port, cache-remote-username, and cache-remote-password from additionalOptions.

Increase the database connection pool size to handle the additional load:

  db:
    poolMinSize: 30  (1)
    poolInitialSize: 30
    poolMaxSize: 30

1	Adjust this value based on your workload. With the `stateless` feature, authentication sessions and action tokens are stored in the database, which increases database load. See the Concepts for database connection pools guide for details.

See the Deploying Keycloak for HA with the Operator (v2) guide for a complete example of the v2 Keycloak CR.

Deploy the updated Keycloak CR to all sites:

kubectl --namespace keycloak apply -f keycloak.yaml

Verify the deployment.

Follow the verification steps in the Deploying Keycloak for HA with the Operator (v2) guide. Afterward, verify that the external load balancer routes traffic to all sites.

Remove the Infinispan deployment.

Once the v2 deployment is verified, remove the v1 infrastructure.

Delete the Infinispan Cache CRs:

kubectl --namespace keycloak delete cache actiontokens authenticationssessions loginfailures work

Delete the Infinispan CR:

kubectl --namespace keycloak delete infinispan infinispan

Remove the Infinispan credentials and cross-site secrets:

kubectl --namespace keycloak delete secret remote-store-secret
kubectl --namespace keycloak delete secret ispn-xsite-sa-token
kubectl --namespace keycloak delete secret xsite-keystore-secret
kubectl --namespace keycloak delete secret xsite-truststore-secret
kubectl --namespace keycloak delete secret xsite-token-secret

Adjust the secret names to match your deployment.

Optionally, uninstall the Infinispan Operator if it is no longer needed for other purposes.

Remove the fencing automation.

If using the AWS Lambda-based fencing as described in the Deploying an AWS Lambda to disable a non-responding site guide:
1. Delete the AWS Lambda function.
2. Remove the Prometheus AlertmanagerConfig and PrometheusRule CRs that triggered the Lambda.
3. Remove the SNS topic and any related IAM roles.
Adjust these steps based on your fencing implementation.
Remove Infinispan-specific monitoring.

Remove any Prometheus ServiceMonitor or PodMonitor resources that target the Infinispan deployment, as well as any Grafana dashboards for Infinispan cache metrics.

Changes to operational procedures

After migrating to v2, several operational procedures from v1 are no longer applicable:

Infinispan synchronization: The procedure described in the Synchronizing sites guide is no longer needed, as there is no external Infinispan to synchronize.
Site offline/online for Infinispan: The Infinispan-specific steps in the Taking a site offline and Bringing a site online guides are replaced by load balancer configuration changes. To take a site offline, remove it from the load balancer. To bring it back online, add it to the load balancer.
Infinispan health checks: The Infinispan health checks described in the Health checks for multi-cluster deployments guide are no longer applicable.

For upgrade procedures, see the Managing upgrades (v2) guide.