Deploying Keycloak for HA with the Operator

Prerequisites

Kubernetes cluster running.
Understanding of a Basic Keycloak deployment of Keycloak with the Keycloak Operator.
AWS Aurora database deployed using the Deploying AWS Aurora in multiple availability zones guide.
Infinispan server deployed using the Deploying Infinispan for HA with the Infinispan Operator guide.

Procedure

Determine the sizing of the deployment using the Concepts for sizing CPU and memory resources guide.
Install the Keycloak Operator as described in the Keycloak Operator Installation guide.
Notice the configuration file below contains options relevant for connecting to the Aurora database from Deploying AWS Aurora in multiple availability zones
Notice the configuration file below options relevant for connecting to the Infinispan server from Deploying Infinispan for HA with the Infinispan Operator
Build a custom Keycloak image which is prepared for usage with the Amazon Aurora PostgreSQL database.
Secure the Amazon Aurora PostgreSQL database connection by downloading the certificate bundles. Create a ConfigMap with the certificate bundle using:
```
kubectl --namespace keycloak create configmap keycloak-aurora-rootcert --from-file aurora.pem=/path/to/bundle.pem
```

Deploy the Keycloak CR with the following values with the resource requests and limits calculated in the first step:

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  labels:
    app: keycloak
  name: keycloak
  namespace: keycloak
spec:
  hostname:
    hostname: <KEYCLOAK_URL_HERE>
  resources:
    requests:
      cpu: "2"
      memory: "1250M"
    limits:
      cpu: "6"
      memory: "2250M"
  db:
    vendor: postgres
    url: jdbc:aws-wrapper:postgresql://<AWS_AURORA_URL_HERE>:5432/keycloak?sslmode=verify-full&sslrootcert=/opt/keycloak/conf/truststores/configmap-keycloak-aurora-rootcert/aurora.pem (1)
    poolMinSize: 30 (2)
    poolInitialSize: 30
    poolMaxSize: 30
    usernameSecret:
      name: keycloak-db-secret
      key: username
    passwordSecret:
      name: keycloak-db-secret
      key: password
  image: <KEYCLOAK_IMAGE_HERE> (3)
  startOptimized: false (3)
  features:
    enabled:
      - multi-site (4)
  additionalOptions:
    - name: log-console-output
      value: json
    - name: metrics-enabled (5)
      value: 'true'
    - name: event-metrics-user-enabled
      value: 'true'
    - name: cache-remote-host
      value: "infinispan.keycloak.svc"
    - name: cache-remote-port
      value: "11222"
    - name: cache-remote-username
      secret:
        name: remote-store-secret
        key: username
    - name: cache-remote-password
      secret:
        name: remote-store-secret
        key: password
    - name: db-driver
      value: software.amazon.jdbc.Driver
  http:
    tlsSecret: keycloak-tls-secret
  instances: 3
  truststores:
    aurora:
      configMap:
        name: keycloak-aurora-rootcert (6)

1	Secure the database connection with the `sslmode=verify-full` and `sslrootcert` properties.
2	The database connection pool initial, max and min size should be identical to allow statement caching for the database. Adjust this number to meet the needs of your system. As most requests will not touch the database due to the Keycloak embedded cache, this change can serve several hundreds of requests per second. See the Concepts for database connection pools guide for details.
3	Specify the URL to your custom Keycloak image. If your image is optimized, set the `startOptimized` flag to `true`.
4	Enable additional features for multi-cluster support like the loadbalancer probe `/lb-check`.
5	To be able to analyze the system under load, enable the metrics endpoint.
6	Specify the `ConfigMap` name that contains the Amazon Aurora PostgreSQL database certificate bundle. The Operator will automatic mount the file in directory `/opt/keycloak/conf/truststores/configmap-<config map name>/<file-name>`

Verifying the deployment

Confirm that the Keycloak deployment is ready.

kubectl wait --for=condition=Ready keycloaks.k8s.keycloak.org/keycloak
kubectl wait --for=condition=RollingUpdate=False keycloaks.k8s.keycloak.org/keycloak

Optional: Load shedding

To enable load shedding, limit the number of queued requests.

Load shedding with max queued http requests

spec:
  additionalOptions:
    - name: http-max-queued-requests
      value: "1000"

All exceeding requests are served with an HTTP 503.

You might consider limiting the value for http-pool-max-threads further because multiple concurrent threads will lead to throttling by Kubernetes once the requested CPU limit is reached.

See the Concepts for configuring thread pools guide about load shedding for details.

Optional: Disable sticky sessions

When running on OpenShift and the default passthrough Ingress setup as provided by the Keycloak Operator, the load balancing done by HAProxy is done by using sticky sessions based on the IP address of the source. When running load tests, or when having a reverse proxy in front of HAProxy, you might want to disable this setup to avoid receiving all requests on a single Keycloak Pod.

Add the following supplementary configuration under the spec in the Keycloak Custom Resource to disable sticky sessions.

spec:
  ingress:
    enabled: true
    annotations:
      # When running load tests, disable sticky sessions on the OpenShift HAProxy router
      # to avoid receiving all requests on a single Keycloak Pod.
      haproxy.router.openshift.io/balance: roundrobin
      haproxy.router.openshift.io/disable_cookies: 'true'