Concepts for multi-cluster deployments (v2)
Understand multi-cluster deployment with synchronous replication without an external Infinispan cluster.This topic describes a highly available multi-cluster setup and the behavior to expect. It outlines the requirements of the high availability architecture and describes the benefits and tradeoffs. See Multi-cluster deployments (v2) for the introduction.
When to use this setup
Use this setup to provide Keycloak deployments that are able to tolerate cluster failures, reducing the likelihood of downtime.
Multiple Keycloak clusters share a synchronously replicated database.
Unlike the standard Concepts for multi-cluster (v1) deployments, this setup does not require an external Infinispan cluster. This simplifies the deployment architecture and removes the need for an external monitoring solution that reconfigures the external Infinispan on failover. It also reduces operational procedures on failback.
Deployment, data storage and caching
Two or more independent Keycloak deployments running in different sites.
Users, realms, clients, user sessions, authentication sessions and other persisted entities are stored in a database that is replicated synchronously across all sites. Keycloak still maintains local in-memory caches for performance. Frequently read realm and authorization metadata, such as realm, client and group configuration, is cached in the embedded caches of each Keycloak cluster. Within a local Keycloak cluster, cache invalidations still use the embedded work caches. Between independent clusters, cache invalidations are propagated through the database-backed outbox mechanism.
Background jobs
Background jobs will run only in one of the active clusters and will be automatically redistributed to another cluster on a cluster failure.
Health check endpoints
Keycloak exposes two health check endpoints for different purposes:
-
/health/readyon the management port indicates whether an individual node is ready to serve traffic. Use this for the internal load balancer or Kubernetes readiness probes. -
/lb-checkindicates whether a site as a whole is available. Use this for the external load balancer that routes traffic between sites.
Updates to realm data and cache invalidation
This paragraph describes the internal implementation of how cache invalidation works across clusters with the stateless feature to allow administrators to understand the emerging observed behavior.
This internal implementation might change between minor or patch releases.
If it changes, this section will be updated.
|
When the realm data, including clients and groups, is changed in one Keycloak instance, that data is updated in the database. The instance sends a cache invalidation message to the other sites via the database, leveraging the outbox pattern with a queuing table. The other sites retrieve the message via polling. The polling interval is configurable and set to 100ms by default. The sending instance waits for five times the polling interval for the messages to be consumed by the receiver before returning to the caller. This allows callers in regular operations to ensure that all nodes have received the invalidation message and will use the new data from the database upon the next call.
Updates to realm data in these multi-cluster v2 setups are slower than in single-cluster setups, as the invalidation mechanism adds a delay of up to 100 ms to calls that modify realm-related data.
When planning for large batch updates of realm data, consider shutting down all clusters except for one to avoid the delay, and start up the other clusters once the update is complete.
Invalidation messages are queued for up to 60 seconds, which is in line with the cluster membership mechanism, which updates its information in the database every 30–45 seconds.
When a Keycloak instance temporarily loses its database connection, it may miss cache invalidation messages sent by other clusters during the outage. Once the database connection is restored, all local caches are cleared automatically to ensure consistency. As an additional safeguard, any cached realm information expires by default after one hour so that stale entries do not persist indefinitely.
Causes of data and service loss
While this setup aims for high availability, the following situations can still lead to service or data loss:
-
Keycloak site failure may result in requests failing in the period between the failure and the load balancer detecting it, as requests may still be routed to the failed site.
-
Once failures occur in the communication between the sites, manual steps may be necessary depending on the database vendor, for example to promote an instance to be the new writer.
-
Degraded setups can lead to service or data loss if additional components fail. Monitoring is necessary to detect degraded setups. Without monitoring, a degraded state may go unnoticed until a subsequent failure causes a full outage or data loss.
Failures which this setup can survive
| Failure | Expected behavior | RPO1 | RT2 |
|---|---|---|---|
Database node |
If the writer instance fails, the database can promote a reader instance in the same or other site to be the new writer. |
No data loss3 |
Seconds to minutes (depending on the database) |
Keycloak node |
Multiple Keycloak instances run in each site. If one instance fails, some incoming requests might receive an error message or be delayed for some seconds until the load balancer takes the node out of rotation. |
No data loss3 |
Seconds to minutes (depending on the load balancer) |
Database connectivity loss |
If the connectivity between sites is lost, the synchronous replication will fail. Some requests might receive an error message or be delayed for a few seconds. Manual operations might be necessary depending on the database. Keycloak instances that have access to a writable database instance will continue to serve traffic, while other nodes will mark themselves as not ready to the load balancer. |
No data loss3 |
Seconds to minutes (depending on the database and load balancer) |
Site failure |
If none of the Keycloak nodes are available, the load balancer will detect the outage and redirect the traffic to the other site. Some requests might receive an error message until the load balancer detects the failure. |
No data loss3 |
Seconds to minutes (depending on the load balancer) |
Table footnotes:
1 Tested Recovery Point Objective, assuming all parts of the setup were healthy at the time this occurred.
2 Estimated Recovery Time.
3 The statement "No data loss" depends on the setup not being degraded from previous failures.
For performance, Keycloak uses optimized database commit for ephemeral data such as sessions and events when supported by the configured database.
On a database crash, in-flight session data within a sub-second window may be lost, requiring affected users to re-authenticate.
Persistent data such as users, realms, and credentials always uses durable commit and is not affected.
To disable the optimization, set --spi-connections-jpa--quarkus--async-commit=false.
To restore the service, restart any failed Keycloak node and restore its database connectivity. No other recovery operations are necessary for Keycloak. A standard load balancer configuration should detect the active Keycloak nodes automatically and route traffic to them. Recovery operations on the database side might depend on the database vendor.
Known limitations
- Site Failure
-
A successful failover requires a setup not degraded from previous failures. Use monitoring to ensure degradations are detected and handled in a timely manner.
Questions and answers
- Why synchronous database replication?
-
A synchronously replicated database ensures that data written in one site is always available in the other site after site failures and no data is lost. It also ensures that the next request will not return stale data, independent of which site serves it.
- Why is a low-latency network between sites needed?
-
Synchronous replication defers the response to the caller until the data is received at the other site. A low latency is necessary as each request can potentially involve multiple interactions between the sites when data is updated, which would amplify the latency. See the Multi-cluster deployments (v2) for specific latency thresholds.
- Is a synchronous cluster less stable than an asynchronous cluster?
-
An asynchronous setup would handle network failures between the sites gracefully, while the synchronous setup would delay requests and throw errors to the caller where the asynchronous setup would have deferred the writes to the database on the other site. However, as the sites would never be fully up-to-date, this setup could lead to data loss during failures. This would include:
-
Lost changes leading to users being able to log in with an old password because database changes are not replicated to the other site at the point of failure when using an asynchronous database.
Therefore, tradeoffs exist between high availability and consistency. The focus of this topic is to prioritize consistency over availability with Keycloak.
-
Next steps
Continue reading the blueprints at Deploying Keycloak for HA with the Operator (v2) or Deploying Keycloak for HA on bare-metal or virtual machines (v2).
On this page