This guide describes the procedures required to deploy Infinispan in a multiple-cluster environment (cross-site). For simplicity, this topic uses the minimum configuration possible that allows Keycloak to be used with an external Infinispan.
This guide assumes two OpenShift clusters named Site-A
and Site-B
.
This is a building block following the concepts described in the Concepts for active-passive deployments guide. See the Multi-site deployments guide for an overview.
This setup deploys two synchronously replicating Infinispan clusters in two sites with a low-latency network connection. An example of this scenario could be two availability zones in one AWS region.
Keycloak, loadbalancer and database have been removed from the following diagram for simplicity.
OpenShift or Kubernetes cluster running
Understanding of the Infinispan Operator
Install the Infinispan Operator
Configure the credential to access the Infinispan cluster.
Keycloak needs this credential to be able to authenticate with the Infinispan cluster.
The following identities.yaml
file sets the username and password with admin permissions
credentials:
- username: developer
password: strong-password
roles:
- admin
The identities.yaml
could be set in a secret as one of the following:
As a Kubernetes Resource:
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: connect-secret
namespace: keycloak
data:
identities.yaml: Y3JlZGVudGlhbHM6CiAgLSB1c2VybmFtZTogZGV2ZWxvcGVyCiAgICBwYXNzd29yZDogc3Ryb25nLXBhc3N3b3JkCiAgICByb2xlczoKICAgICAgLSBhZG1pbgo= (1)
1 | The identities.yaml from the previous example base64 encoded. |
Using the CLI
kubectl create secret generic connect-secret --from-file=identities.yaml
Check the Configuring Authentication documentation for more details.
These commands must be executed on both OpenShift clusters.
Create a service account.
A service account is required to establish a connection between clusters. The Infinispan Operator uses it to inspect the network configuration from the remote site and to configure the local Infinispan cluster accordingly.
For more details, see the Managing Cross-Site Connections documentation.
Create a service-account-token
secret type as follows.
The same YAML file can be used in both OpenShift clusters.
apiVersion: v1
kind: Secret
metadata:
name: ispn-xsite-sa-token (1)
annotations:
kubernetes.io/service-account.name: "xsite-sa" (2)
type: kubernetes.io/service-account-token
1 | The secret name. |
2 | The service account name. |
Create the service account and generate an access token in both OpenShift clusters.
Site-A
kubectl create sa -n keycloak xsite-sa
oc policy add-role-to-user view -n keycloak -z xsite-sa
kubectl create -f xsite-sa-secret-token.yaml
kubectl get secrets ispn-xsite-sa-token -o jsonpath="{.data.token}" | base64 -d > Site-A-token.txt
Site-B
kubectl create sa -n keycloak xsite-sa
oc policy add-role-to-user view -n keycloak -z xsite-sa
kubectl create -f xsite-sa-secret-token.yaml
kubectl get secrets ispn-xsite-sa-token -o jsonpath="{.data.token}" | base64 -d > Site-B-token.txt
The next step is to deploy the token from Site-A
into Site-B
and the reverse:
Site-B
token into Site-A
kubectl create secret generic -n keycloak xsite-token-secret \
--from-literal=token="$(cat Site-B-token.txt)"
Site-A
token into Site-B
kubectl create secret generic -n keycloak xsite-token-secret \
--from-literal=token="$(cat Site-A-token.txt)"
Create TLS secrets
In this guide, Infinispan uses an OpenShift Route for the cross-site communication. It uses the SNI extension of TLS to direct the traffic to the correct Pods. To achieve that, JGroups use TLS sockets, which require a Keystore and Truststore with the correct certificates.
For more information, see the Securing Cross Site Connections documentation or this Red Hat Developer Guide.
Upload the Keystore and the Truststore in an OpenShift Secret. The secret contains the file content, the password to access it, and the type of the store. Instructions for creating the certificates and the stores are beyond the scope of this guide.
To upload the Keystore as a Secret, use the following command:
kubectl -n keycloak create secret generic xsite-keystore-secret \
--from-file=keystore.p12="./certs/keystore.p12" \ (1)
--from-literal=password=secret \ (2)
--from-literal=type=pkcs12 (3)
1 | The filename and the path to the Keystore. |
2 | The password to access the Keystore. |
3 | The Keystore type. |
To upload the Truststore as a Secret, use the following command:
kubectl -n keycloak create secret generic xsite-truststore-secret \
--from-file=truststore.p12="./certs/truststore.p12" \ (1)
--from-literal=password=caSecret \ (2)
--from-literal=type=pkcs12 (3)
1 | The filename and the path to the Truststore. |
2 | The password to access the Truststore. |
3 | The Truststore type. |
Keystore and Truststore must be uploaded in both OpenShift clusters. |
Create a Cluster for Infinispan with Cross-Site enabled
The Setting Up Cross-Site documentation provides all the information on how to create and configure your Infinispan cluster with cross-site enabled, including the previous steps.
A basic example is provided in this guide using the credentials, tokens, and TLS Keystore/Truststore created by the commands from the previous steps.
Infinispan
CR for Site-A
apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
name: infinispan (1)
namespace: keycloak
annotations:
infinispan.org/monitoring: 'true' (2)
spec:
replicas: 3
security:
endpointSecretName: connect-secret (3)
service:
type: DataGrid
sites:
local:
name: site-a (4)
expose:
type: Route (5)
maxRelayNodes: 128
encryption:
transportKeyStore:
secretName: xsite-keystore-secret (6)
alias: xsite (7)
filename: keystore.p12 (8)
routerKeyStore:
secretName: xsite-keystore-secret (6)
alias: xsite (7)
filename: keystore.p12 (8)
trustStore:
secretName: xsite-truststore-secret (9)
filename: truststore.p12 (10)
locations:
- name: site-b (11)
clusterName: infinispan
namespace: keycloak (12)
url: openshift://api.site-b (13)
secretName: xsite-token-secret (14)
1 | The cluster name |
2 | Allows the cluster to be monitored by Prometheus. |
3 | If using a custom credential, configure here the secret name. |
4 | The name of the local site, in this case Site-A . |
5 | Exposing the cross-site connection using OpenShift Route. |
6 | The secret name where the Keystore exists as defined in the previous step. |
7 | The alias of the certificate inside the Keystore. |
8 | The secret key (filename) of the Keystore as defined in the previous step. |
9 | The secret name where the Truststore exists as defined in the previous step. |
10 | The Truststore key (filename) of the Keystore as defined in the previous step. |
11 | The remote site’s name, in this case Site-B . |
12 | The namespace of the Infinispan cluster from the remote site. |
13 | The OpenShift API URL for the remote site. |
14 | The secret with the access toke to authenticate into the remote site. |
For Site-B
, the Infinispan
CR looks similar to the above.
Note the differences in point 4, 11 and 13.
Infinispan
CR for Site-B
apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
name: infinispan (1)
namespace: keycloak
annotations:
infinispan.org/monitoring: 'true' (2)
spec:
replicas: 3
security:
endpointSecretName: connect-secret (3)
service:
type: DataGrid
sites:
local:
name: site-b (4)
expose:
type: Route (5)
maxRelayNodes: 128
encryption:
transportKeyStore:
secretName: xsite-keystore-secret (6)
alias: xsite (7)
filename: keystore.p12 (8)
routerKeyStore:
secretName: xsite-keystore-secret (6)
alias: xsite (7)
filename: keystore.p12 (8)
trustStore:
secretName: xsite-truststore-secret (9)
filename: truststore.p12 (10)
locations:
- name: site-a (11)
clusterName: infinispan
namespace: keycloak (12)
url: openshift://api.site-a (13)
secretName: xsite-token-secret (14)
Creating the caches for Keycloak.
Keycloak requires the following caches to be present: sessions
, actionTokens
, authenticationSessions
, offlineSessions
, clientSessions
, offlineClientSessions
, loginFailures
, and work
.
The Infinispan Cache CR allows deploying the caches in the Infinispan cluster.
Cross-site needs to be enabled per cache as documented by Cross Site Documentation.
The documentation contains more details about the options used by this guide.
The following example shows the Cache
CR for Site-A
.
Site-A
apiVersion: infinispan.org/v2alpha1
kind: Cache
metadata:
name: sessions
namespace: keycloak
spec:
clusterName: infinispan
name: sessions
template: |-
distributedCache:
mode: "SYNC"
owners: "2"
statistics: "true"
remoteTimeout: 14000
stateTransfer:
chunkSize: 16
backups:
mergePolicy: ALWAYS_REMOVE (1)
site-b: (2)
backup:
strategy: "SYNC" (3)
timeout: 13000
stateTransfer:
chunkSize: 16
1 | The cross-site merge policy, invoked when there is a write-write conflict.
Set this for the caches sessions , authenticationSessions , offlineSessions , clientSessions and offlineClientSessions , and do not set it for all other caches. |
2 | The remote site name. |
3 | The cross-site communication, in this case, SYNC . |
For Site-B
, the Cache
CR is similar except in point 2.
Site-B
apiVersion: infinispan.org/v2alpha1
kind: Cache
metadata:
name: sessions
namespace: keycloak
spec:
clusterName: infinispan
name: sessions
template: |-
distributedCache:
mode: "SYNC"
owners: "2"
statistics: "true"
remoteTimeout: 14000
stateTransfer:
chunkSize: 16
backups:
mergePolicy: ALWAYS_REMOVE (1)
site-a: (2)
backup:
strategy: "SYNC" (3)
timeout: 13000
stateTransfer:
chunkSize: 16
Confirm that the Infinispan cluster is formed, and the cross-site connection is established between the OpenShift clusters.
kubectl wait --for condition=WellFormed --timeout=300s infinispans.infinispan.org -n keycloak infinispan
kubectl wait --for condition=CrossSiteViewFormed --timeout=300s infinispans.infinispan.org -n keycloak infinispan
After Infinispan is deployed and running, use the procedure in the Connect Keycloak with an external Infinispan guide to connect your Keycloak cluster with the Infinispan cluster.