Server Installation and Configuration Guide

When running the server in standalone mode, there is a specific script you need to run to boot the server depending on your operating system. These scripts live in the bin/ directory of the server distribution.

To boot the server:

The bulk of this guide walks you through how to configure infrastructure level aspects of Keycloak. These aspects are configured in a configuration file that is specific to the application server that Keycloak is a derivative of. In the standalone operation mode, this file lives in …​/standalone/configuration/standalone.xml. This file is also used to configure non-infrastructure level things that are specific to Keycloak components.

The distribution has a mostly pre-configured app server configuration file for running within a cluster. It has all the specific infrastructure settings for networking, databases, caches, and discovery. This file resides in …​/standalone/configuration/standalone-ha.xml. There’s a few things missing from this configuration. You can’t run Keycloak in a cluster without configuring a shared database connection. You also need to deploy some type of load balancer in front of the cluster. The clustering and database sections of this guide walk you through these things.

You use the same boot scripts to start Keycloak as you do in standalone mode. The difference is that you pass in an additional flag to point to the HA config file.

To boot the server:

Various other chapters in this guide walk you through configuring various aspects like databases, HTTP network connections, caches, and other infrastructure related things. While standalone mode uses the standalone.xml file to configure these things, domain mode uses the …​/domain/configuration/domain.xml configuration file. This is where the domain profile and server group for the Keycloak server are defined.

Let’s look at some aspects of this domain.xml file. The auth-server-standalone and auth-server-clustered profile XML blocks are where you are going to make the bulk of your configuration decisions. You’ll be configuring things here like network connections, caches, and database connections.

The auth-server-standalone profile is a non-clustered setup. The auth-server-clustered profile is the clustered setup.

If you scroll down further, you’ll see various socket-binding-groups defined.

This configration defines the default port mappings for various connectors that are opened with each Keycloak server instance. Any value that contains ${…​} is a value that can be overridden on the command line with the -D switch, i.e.

The definition of the server group for Keycloak resides in the server-groups XML block. It specifies the domain profile that is used (default) and also some default boot arguments for the Java VM when the host controller boots an instance. It also binds a socket-binding-group to the server group.

Keycloak comes with two host controller configuration files that reside in the …​/domain/configuration/ directory: host-master.xml and host-slave.xml. host-master.xml is configured to boot up a domain controller, a load balancer, and one Keycloak server instance. host-slave.xml is configured to talk to the domain controller and boot up one Keycloak server instance.

To disable the load balancer server instance, edit host-master.xml and comment out or remove the "load-balancer" entry.

Another interesting thing to note about this file is the declaration of the authentication server instance. It has a port-offset setting. Any network port defined in the domain.xml socket-binding-group or the server group will have the value of port-offset added to it. For this example domain setup we do this so that ports opened by the load balancer server don’t conflict with the authentication server instance that is started.

Each Keycloak server instance defined in your host files creates a working directory under …​/domain/servers/{SERVER NAME}. Additional configuration can be put there, and any temporary, log, or data files the server instance needs or creates go there too. The structure of these per server directories ends up looking like any other WildFly booted server.

When running the server in domain mode, there is a specific script you need to run to boot the server depending on your operating system. These scripts live in the bin/ directory of the server distribution.

To boot the server:

When running the boot script you will need to pass in the host controlling configuration file you are going to use via the --host-config switch.

You can test drive clustering using the out-of-the-box domain.xml configuration. This example domain is meant to run on one machine and boots up:

To simulate running a cluster on two machines, you’ll need to run the domain.sh script twice to start two separate host controllers. The first will be the master host controller which will start a domain controller, an HTTP load balancer, and one Keycloak authentication server instance. The second will be a slave host controller that only starts up an authentication server instance.

As this is an advanced topic, we recommend you first read the following, which provide valuable background knowledge:

This section provides an introduction to the concepts and details of how Keycloak Cross-Datacenter Replication is accomplished.

Keycloak is stateful application. It uses the following as data sources:

In our example architecture, there are two data centers called site1 and site2. For Cross-Datacenter Replication, we must make sure that both sources of data work reliably and that Keycloak servers from site1 are eventually able to read the data saved by Keycloak servers on site2 .

Based on the environment, you have the option to decide if you prefer:

For more details, see Modes.

An end user’s browser sends an HTTP request to the front end load balancer. This load balancer is usually HTTPD or WildFly with mod_cluster, NGINX, HA Proxy, or perhaps some other kind of software or hardware load balancer.

The load balancer then forwards the HTTP requests it receives to the underlying Keycloak instances, which can be spread among multiple data centers. Load balancers typically offer support for sticky sessions, which means that the load balancer is able to always forward all HTTP requests from the same user to the same Keycloak instance in same data center.

HTTP requests that are sent from client applications to the load balancer are called backchannel requests. These are not seen by an end user’s browser and therefore can not be part of a sticky session between the user and the load balancer. For backchannel requests, the loadbalancer can forward the HTTP request to any Keycloak instance in any data center. This is challenging as some OpenID Connect and some SAML flows require multiple HTTP requests from both the user and the application. Because we can not reliably depend on sticky sessions to force all the related requests to be sent to the same Keycloak instance in the same data center, we must instead replicate some data across data centers, so the data are seen by subsequent HTTP requests during a particular flow.

According your requirements, there are two basic operating modes for Cross-Datacenter Replication:

The active/passive mode is better for performance. For more information about how to configure caches for either mode, see: SYNC or ASYNC backups.

Keycloak uses a relational database management system (RDBMS) to persist some metadata about realms, clients, users, and so on. See this chapter of the server installation guide for more details. In a Cross-Datacenter Replication setup, we assume that either both data centers talk to the same database or that every data center has its own database node and both database nodes are synchronously replicated across the data centers. In both cases, it is required that when a Keycloak server on site1 persists some data and commits the transaction, those data are immediately visible by subsequent DB transactions on site2.

Details of DB setup are out-of-scope for Keycloak, however many RDBMS vendors like MariaDB and Oracle offer replicated databases and synchronous replication. We test Keycloak with these vendors:

This section begins with a high level description of the Infinispan caches. More details of the cache setup follow.

In Keycloak we have the concept of authentication sessions. There is a separate Infinispan cache called authenticationSessions used to save data during authentication of particular user. Requests from this cache usually involve only a browser and the Keycloak server, not the application. Here we can rely on sticky sessions and the authenticationSessions cache content does not need to be replicated across data centers, even if you are in Active/Active mode.

We also have the concept of action tokens, which are used typically for scenarios when the user needs to confirm an action asynchronously by email. For example, during the forget password flow the actionTokens Infinispan cache is used to track metadata about related action tokens, such as which action token was already used, so it can’t be reused second time. This usually needs to be replicated across data centers.

Keycloak uses Infinispan to cache persistent data to avoid many unnecessary requests to the database. Caching improves performance, however it adds an additional challenge. When some Keycloak server updates any data, all other Keycloak servers in all data centers need to be aware of it, so they invalidate particular data from their caches. Keycloak uses local Infinispan caches called realms, users, and authorization to cache persistent data.

We use a separate cache, work, which is replicated across all data centers. The work cache itself does not cache any real data. It is used only for sending invalidation messages between cluster nodes and data centers. In other words, when data is updated, such as the user john, the Keycloak node sends the invalidation message to all other cluster nodes in the same data center and also to all other data centers. After receiving the invalidation notice, every node then invalidates the appropriate data from their local cache.

There are Infinispan caches called sessions, clientSessions, offlineSessions, and offlineClientSessions, all of which usually need to be replicated across data centers. These caches are used to save data about user sessions, which are valid for the length of a user’s browser session. The caches must handle the HTTP requests from the end user and from the application. As described above, sticky sessions can not be reliably used in this instance, but we still want to ensure that subsequent HTTP requests can see the latest data. For this reason, the data are usually replicated across data centers.

Finally the loginFailures cache is used to track data about failed logins, such as how many times the user john entered a bad password. The details are described here. It is up to the admin whether this cache should be replicated across data centers. To have an accurate count of login failures, the replication is needed. On the other hand, not replicating this data can save some performance. So if performance is more important than accurate counts of login failures, the replication can be avoided.

For more detail about how caches can be configured see Tuning the JDG cache configuration.

Keycloak uses multiple, separate clusters of Infinispan caches. Every Keycloak node is in the cluster with the other Keycloak nodes in same data center, but not with the Keycloak nodes in different data centers. A Keycloak node does not communicate directly with the Keycloak nodes from different data centers. Keycloak nodes use external JDG (actually Infinispan servers) for communication across data centers. This is done using the Infinispan HotRod protocol.

The Infinispan caches on the Keycloak side must be configured with the remoteStore to ensure that data are saved to the remote cache. There is separate Infinispan cluster between JDG servers, so the data saved on JDG1 on site1 are replicated to JDG2 on site2 .

Finally, the receiving JDG server notifies the Keycloak servers in its cluster through the Client Listeners, which are a feature of the HotRod protocol. Keycloak nodes on site2 then update their Infinispan caches and the particular user session is also visible on Keycloak nodes on site2.

See the Example Architecture Diagram for more details.

For this example, we describe using two data centers, site1 and site2. Each data center consists of 1 Infinispan server and 2 Keycloak servers. We will end up with 2 Infinispan servers and 4 Keycloak servers in total.

Our example setup assumes all that all 4 Keycloak servers talk to the same database. In production, it is recommended to use separate synchronously replicated databases across data centers as described in Database.

This section contains some tips and options related to Cross-Datacenter Replication.

For example, assume this scenario:

According to your environment, it may be more or less probable that the network between sites is unavailable or temporarily broken (split-brain). In case this happens, it is good that Infinispan servers on site1 are aware of the fact that Infinispan servers on site2 are unavailable, so they will stop trying to reach the servers in the jdg2 site and the backup failures won’t happen. This is called Take site offline .

There are 2 ways to take the site offline.

Manually by admin - Admin can use the jconsole or other tool and run some JMX operations to manually take the particular site offline. This is useful especially if the outage is planned. With jconsole or CLI, you can connect to the jdg1 server and take the site2 offline. More details about this are available in the JDG documentation.

Automatically - After some amount of failed backups, the site2 will usually be taken offline automatically. This is done due the configuration of take-offline element inside the cache configuration as configured in Infinispan server setup.

This example shows that the site will be taken offline automatically for the particular single cache if there are at least 3 subsequent failed backups and there is no any successful backup within 60 seconds.

Automatically taking a site offline is useful especially if the broken network between sites is unplanned. The disadvantage is that there will be some failed backups until the network outage is detected, which could also mean failures on the application side. For example, there will be failed logins for some users or big login timeouts. Especially if failure-policy with value FAIL is used.

Once your network is back and site1 and site2 can talk to each other, you may need to put the site online. This needs to be done manually through JMX or CLI in similar way as taking a site offline. Again, you may need to check all the caches and bring them online.

Once the sites are put online, it’s usually good to:

State transfer is a required, manual step. Infinispan server does not do this automatically, for example during split-brain, it is only the admin who may decide which site has preference and hence if state transfer needs to be done bidirectionally between both sites or just unidirectionally, as in only from site1 to site2, but not from site2 to site1.

A bidirectional state transfer will ensure that entities which were created after split-brain on site1 will be transferred to site2. This is not an issue as they do not yet exist on site2. Similarly, entities created after split-brain on site2 will be transferred to site1. Possibly problematic parts are those entities which exist before split-brain on both sites and which were updated during split-brain on both sites. When this happens, one of the sites will win and will overwrite the updates done during split-brain by the second site.

Unfortunately, there is no any universal solution to this. Split-brains and network outages are just state, which is usually impossible to be handled 100% correctly with 100% consistent data between sites. In the case of Keycloak, it typically is not a critical issue. In the worst case, users will need to re-login again to their clients, or have the improper count of loginFailures tracked for brute force protection. See the Infinispan/JGroups documentation for more tips how to deal with split-brain.

The state transfer can be also done on the Infinispan server side through JMX. The operation name is pushState. There are few other operations to monitor status, cancel push state, and so on. More info about state transfer is available in the Infinispan docs.

After split-brain it is safe to manually clear caches in the Keycloak admin console. This is because there might be some data changed in the database on site1 and because of the event, that the cache should be invalidated wasn’t transferred during split-brain to site2. Hence Keycloak nodes on site2 may still have some stale data in their caches.

To clear the caches, see Clearing Server Caches.

When the network is back, it is sufficient to clear the cache just on one Keycloak node on any random site. The cache invalidation event will be sent to all the other Keycloak nodes in all sites. However, it needs to be done for all the caches (realms, users, keys). See Clearing Server Caches for more information.

This section contains tips and options for configuring your JDG cache.

By default, the configuration of backup failure-policy in the Infinispan cache configuration in the JDG clustered.xml file is configured as FAIL. You may change it to WARN or IGNORE, as you prefer.

The difference between FAIL and WARN is that when FAIL is used and the Infinispan server tries to back data up to the other site and the backup fails then the failure will be propagated back to the caller (the Keycloak server). The backup might fail because the second site is temporarily unreachable or there is a concurrent transaction which is trying to update same entity. In this case, the Keycloak server will then retry the operation a few times. However, if the retry fails, then the user might see the error after a longer timeout.

When using WARN, the failed backups are not propagated from the Infinispan server to the Keycloak server. The user won’t see the error and the failed backup will be just ignored. There will be a shorter timeout, typically 10 seconds as that’s the default timeout for backup. It can be changed by the attribute timeout of backup element. There won’t be retries. There will just be a WARNING message in the Infinispan server log.

The potential issue is, that in some cases, there may be just some a short network outage between sites, where the retry (usage of the FAIL policy) may help, so with WARN (without retry), there will be some data inconsistencies across sites. This can also happen if there is an attempt to update the same entity concurrently on both sites.

How bad are these inconsistencies? Usually only means that a user will need to re-authenticate.

When using the WARN policy, it may happen that the single-use cache, which is provided by the actionTokens cache and which handles that particular key is really single use, but may "successfully" write the same key twice. But, for example, the OAuth2 specification mentions that code must be single-use. With the WARN policy, this may not be strictly guaranteed and the same code could be written twice if there is an attempt to write it concurrently in both sites.

If there is a longer network outage or split-brain, then with both FAIL and WARN, the other site will be taken offline after some time and failures as described in Bringing sites offline and online. With the default 1 minute timeout, it is usually 1-3 minutes until all the involved caches are taken offline. After that, all the operations will work fine from an end user perspective. You only need to manually restore the site when it is back online as mentioned in Bringing sites offline and online.

In summary, if you expect frequent, longer outages between sites and it is acceptable for you to have some data inconsistencies and a not 100% accurate single-use cache, but you never want end-users to see the errors and long timeouts, then switch to WARN.

The difference between WARN and IGNORE is, that with IGNORE warnings are not written in the JDG log. See more details in the Infinispan documentation.

The default configuration is using transaction in NON_DURABLE_XA mode with acquire timeout 0. This means that transaction will fail-fast if there is another transaction in progress for the same key.

The reason to switch this to 0 instead of default 10 seconds was to avoid possible deadlock issues. With Keycloak, it can happen that the same entity (typically session entity or loginFailure) is updated concurrently from both sites. This can cause deadlock under some circumstances, which will cause the transaction to be blocked for 10 seconds. See this JIRA report for details.

With timeout 0, the transaction will immediately fail and then will be retried from Keycloak if backup failure-policy with the value FAIL is configured. As long as the second concurrent transaction is finished, the retry will usually be successful and the entity will have applied updates from both concurrent transactions.

We see very good consistency and results for concurrent transaction with this configuration, and it is recommended to keep it.

The only (non-functional) problem is the exception in the Infinispan server log, which happens every time when the lock is not immediately available.

An important part of the backup element is the strategy attribute. You must decide whether it needs to be SYNC or ASYNC. We have 7 caches which might be Cross-Datacenter Replication aware, and these can be configured in 3 different modes regarding cross-dc:

If the SYNC backup is used, then the backup is synchronous and operation is considered finished on the caller (Keycloak server) side once the backup is processed on the second site. This has worse performance than ASYNC, but on the other hand, you are sure that subsequent reads of the particular entity, such as user session, on site2 will see the updates from site1. Also, it is needed if you want data consistency. As with ASYNC the caller is not notified at all if backup to the other site failed.

For some caches, it is even possible to not backup at all and completely skip writing data to the Infinispan server. To set this up, do not use the remote-store element for the particular cache on the Keycloak side (file KEYCLOAK_HOME/standalone/configuration/standalone-ha.xml) and then the particular replicated-cache element is also not needed on the Infinispan server side.

By default, all 7 caches are configured with SYNC backup, which is the safest option. Here are a few things to consider:

For the environments with slower network between data centers and probability of DoS, it is recommended to not backup the loginFailures cache at all.

Generally, if you are in doubt and performance is not a blocker for you, it’s safer to keep the caches in SYNC strategy.

Note the mode attribute of cache-configuration element.

The following tips are intended to assist you should you need to troubleshoot:

Unicode characters are properly handled provided the database was created with Unicode support in VARCHAR and CHAR fields (e.g. by using AL32UTF8 character set as the database character set). No special settings is needed for JDBC driver.

If the database character set is not Unicode, then to use Unicode characters in the special fields, the JDBC driver needs to be configured with the connection property oracle.jdbc.defaultNChar set to true. It might be wise, though not strictly necessary, to also set the oracle.jdbc.convertNcharLiterals connection property to true. These properties can be set either as system properties or as connection properties. Please note that setting oracle.jdbc.defaultNChar may have negative impact on performance. For details, please refer to Oracle JDBC driver configuration documentation.

Unicode characters are properly handled only for the special fields. No special settings of JDBC driver or database is necessary.

Unicode characters are properly handled provided the database was created with Unicode support in VARCHAR and CHAR fields in the CREATE DATABASE command (e.g. by using utf8 character set as the default database character set in MySQL 5.5. Please note that utf8mb4 character set does not work due to different storage requirements to utf8 character set ^[1]). Note that in this case, length restriction to non-special fields does not apply because columns are created to accommodate given amount of characters, not bytes. If the database default character set does not allow storing Unicode, only the special fields allow storing Unicode values.

At the side of JDBC driver settings, it is necessary to add a connection property characterEncoding=UTF-8 to the JDBC connection settings.

Unicode is supported when the database character set is UTF8. In that case, Unicode characters can be used in any field, there is no reduction of field length for non-special fields. No special settings of JDBC driver is necessary.

The character set of a PostgreSQL database is determined at the time it is created. You can determine the default character set for a PostgreSQL cluster with the SQL command

If the default character set is not UTF 8, then you can create the database with UTF8 as its character set like this:

If you are not using a reverse proxy or load balancer to handle HTTPS traffic for you, you’ll need to enable HTTPS for the Keycloak server. This involves

Outgoing HTTP requests sent by Keycloak can optionally use a proxy server based on a comma delimited list of proxy-mappings. A proxy-mapping denotes the combination of a regex based hostname pattern and a proxy-uri in the form of hostnamePattern;proxyUri, e.g.:

To determine the proxy for an outgoing HTTP request the target hostname is matched against the configured hostname patterns. The first matching pattern determines the proxy-uri to use. If none of the configured patterns match for the given hostname then no proxy is used.

If the proxy server requires authentication, include the proxy user’s credentials in this format username:password@. For example:

The special value NO_PROXY for the proxy-uri can be used to indicate that no proxy should be used for hosts matching the associated hostname pattern. It is possible to specify a catch-all pattern at the end of the proxy-mappings to define a default proxy for all outgoing requests.

The following example demonstrates the proxy-mapping configuration.

This can be configured via the following jboss-cli command. Note that you need to properly escape the regex-pattern as shown below.

The jboss-cli command results in the following subsystem configuration. Note that one needs to encode " characters with ".

When Keycloak invokes on remote HTTPS endpoints, it has to validate the remote server’s certificate in order to ensure it is connecting to a trusted server. This is necessary in order to prevent man-in-the-middle attacks. The certificates of these remote server’s or the CA that signed these certificates must be put in a truststore. This truststore is managed by the Keycloak server.

The truststore is used when connecting securely to identity brokers, LDAP identity providers, when sending emails, and for backchannel communication with client applications.

You can use keytool to create a new truststore file or add trusted host certificates to an existing one:

The truststore is configured within the standalone.xml, standalone-ha.xml, or domain.xml file in your distribution. The location of this file depends on your operating mode. You can add your truststore configuration by using the following template:

Possible configuration options for this setting are:

A few features in Keycloak rely on the fact that the remote address of the HTTP client connecting to the authentication server is the real IP address of the client machine. Examples include:

This can be problematic when you have a reverse proxy or loadbalancer in front of your Keycloak authentication server. The usual setup is that you have a frontend proxy sitting on a public network that load balances and forwards requests to backend Keycloak server instances located in a private network. There is some extra configuration you have to do in this scenario so that the actual client IP address is forwarded to and processed by the Keycloak server instances. Specifically:

Configuring your proxy to generate the X-Forwarded-For and X-Forwarded-Proto HTTP headers and preserving the original Host HTTP header is beyond the scope of this guide. Take extra precautions to ensure that the X-Forwarded-For header is set by your proxy. If your proxy isn’t configured correctly, then rogue clients can set this header themselves and trick Keycloak into thinking the client is connecting from a different IP address than it actually is. This becomes really important if you are doing any black or white listing of IP addresses.

Beyond the proxy itself, there are a few things you need to configure on the Keycloak side of things. If your proxy is forwarding requests via the HTTP protocol, then you need to configure Keycloak to pull the client’s IP address from the X-Forwarded-For header rather than from the network packet. To do this, open up the profile configuration file (standalone.xml, standalone-ha.xml, or domain.xml depending on your operating mode) and look for the urn:jboss:domain:undertow:10.0 XML block.

Add the proxy-address-forwarding attribute to the http-listener element. Set the value to true.

If your proxy is using the AJP protocol instead of HTTP to forward requests (i.e. Apache HTTPD + mod-cluster), then you have to configure things a little differently. Instead of modifying the http-listener, you need to add a filter to pull this information from the AJP packets.

Assuming that your reverse proxy doesn’t use port 8443 for SSL you also need to configure what port HTTPS traffic is redirected to.

Add the redirect-socket attribute to the http-listener element. The value should be proxy-https which points to a socket binding you also need to define.

Then add a new socket-binding element to the socket-binding-group element:

You can verify the reverse proxy or load balancer configuration by opening the path /auth/realms/master/.well-known/openid-configuration through the reverse proxy. For example if the reverse proxy address is https://acme.com/ then open the URL https://acme.com/auth/realms/master/.well-known/openid-configuration. This will show a JSON document listing a number of endpoints for Keycloak. Make sure the endpoints starts with the address (scheme, domain and port) of your reverse proxy or load balancer. By doing this you make sure that Keycloak is using the correct endpoint.

You should also verify that Keycloak sees the correct source IP address for requests. To check this, you can try to login to the admin console with an invalid username and/or password. This should show a warning in the server log something like this:

Check that the value of ipAddress is the IP address of the machine you tried to login with and not the IP address of the reverse proxy or load balancer.

This section covers configuring the built-in load balancer that is discussed in the Clustered Domain Example.

The Clustered Domain Example is only designed to run on one machine. To bring up a slave on another host, you’ll need to

See the load balancing section in the WildFly 18 Documentation for information how to use other software-based load balancers.

Some load balancers can be configured to add the route information by themselves instead of relying on the back end Keycloak node. However, as described above, adding the route by the Keycloak is recommended. This is because when done this way performance improves, since Keycloak is aware of the entity that is the owner of particular session and can route to that node, which is not necessarily the local node.

You are permitted to disable adding route information to the AUTH_SESSION_ID cookie by Keycloak, if you prefer, by adding the following into your RHSSO_HOME/standalone/configuration/standalone-ha.xml file in the Keycloak subsystem configuration: