Keycloak allows deployments that consist of multiple Keycloak instances that connect to each other using its embedded Infinispan caches. Load balancers can distribute the load evenly across those instances. Such setups are intended for transparent networks; see Single-cluster deployments for more details.
A multi-cluster setup adds additional components to provide high availability that may be needed for some environments.
Unlike the standard Multi-cluster deployments (v1), it removes the requirement for an external Infinispan cluster. This simplifies the deployment architecture and lifts any requirement for a specific environment such as Kubernetes or AWS.
The multi-cluster deployment capabilities of Keycloak are targeted at use cases that:
Are constrained to a single region or an equivalent low-latency setup.
Permit planned outages for maintenance.
Fit within a defined user and request count.
Can accept the impact of periodic outages.
Provide data centers with the appropriate network latency and database configuration.
To run multi-cluster v2, use an infrastructure that meets the following requirements:
A highly available database with synchronous replication.
Without synchronous replication, a database failover or a stale database read can result in data loss or an inconsistent state, including user sessions, recent configuration changes, or authentication state.
Two or more clusters with low latency for read, write, and commit operations to the database.
A commit latency of less than 5 ms is suggested, and below 10 ms is required.
With synchronous replication, each commit includes a round-trip from the database primary to its replica to transfer the write-ahead log. High commit latency slows every write operation, including login and token requests, and can cause cascading timeouts and low throughput.
Synchronized clocks across all nodes.
Keycloak uses timestamps to detect cluster membership and to distribute cache invalidation messages between clusters. Clocks must be synchronized using NTP or an equivalent mechanism. A clock skew of more than 10 seconds between clusters can cause cache invalidation messages to not be delivered, leading to stale cached data.
An external load balancer to balance requests between the clusters and probe their availability.
Without a load balancer that actively probes the Keycloak health endpoints, site failures are not detected automatically, and traffic continues to be routed to a failed site or split-brain partitions until manual intervention.
While equivalent setups should work, you will need to verify the performance and failure behavior of your environment. We provide functional tests, failure tests and load tests in the Keycloak Benchmark Project.
Latency and latency spikes amplify in the response time of the service and can lead to queued requests, timeouts, and failed requests. Networking problems can cause downtimes until the failure detection isolates problematic nodes.
We regularly test Keycloak with the following configuration:
Two OpenShift single-AZ clusters, in the same AWS Region
Provisioned with Red Hat OpenShift Service on AWS (ROSA), using ROSA HCP.
All worker nodes reside in a single Availability Zone.
OpenShift version 4.21.
Amazon Aurora PostgreSQL database
High availability with a primary DB instance in one availability zone, and a synchronously replicated reader in the second availability zone.
Version 17.5
AWS Global Accelerator, sending traffic to both ROSA clusters
We regularly test Keycloak with the following load:
1,000,000 users
300 requests per second
|
It is imperative that your production deployments are integrated with an observability stack in order to identify issues early and facilitate troubleshooting when they do arise. Additional demand on Keycloak makes isolating issues harder, therefore this becomes increasingly pertinent as the total number of users and requests per second increases. |
While we did not see a hard limit in our tests with these values, we ask you to test for higher volumes with horizontally and vertically scaled Keycloak instances and databases.
During certain failure scenarios, there may be downtime of several minutes depending on the configuration of the database, the load balancer, or Keycloak. By default, Keycloak detects the failure of another node within one minute and either isolates it from the cluster or propagates its state via readiness probes to the load balancer.
After certain failure scenarios, manual intervention may be required depending on the database vendor.
The different guides introduce the necessary concepts and building blocks. For each building block, a blueprint shows how to set up a fully functional example. Additional performance tuning and security hardening are still recommended when preparing a production setup.