Scripted multi-site status check

Combine the different multi-site checks into a single shell script. Details on how to check the availability in a multi-site are available in the official multi-site docs.

Optional Bash script

You can use the cross-site-health-checks.sh script and extend it to perform the necessary checks and integrate this into your monitoring architecture.

To run the script, as a pre-requisite, you need to establish a session from your terminal to the target OCP cluster with a command.

oc login --token=sha256~masked-key --server=https://api.gh-keycloak-a.masked.openshiftapps.com:6443

Also note, it would be necessary to run this script against all the clusters in the cluster group, so the Administrator would have to repeat the above oc login command for all the clusters, which could be automated.

To run the script itself once you have an active oc session below is an example usage.

./cross-site-health-checks.sh \
-n runner-keycloak \
-l <KEYCLOAK_LB_URL> \
-k <KEYCLOAK_SITE_URL> \
-i <KEYCLOAK_ISPN_REST_URL> \
-u developer \
-p <ISPN_REST_URL_PWD> \
-c 3

Verify the Keycloak Load Balancer health check
Checking health for: KEYCLOAK_LB_URL/lb-check
"HEALTHY"

Verify the Load Balancer health check on the Site
Checking health for: KEYCLOAK_SITE_URL/lb-check
"HEALTHY"

Verify the default cache manager health in external ISPN
Checking health for: KEYCLOAK_ISPN_REST_URL/rest/v2/cache-managers/default/health/status
"HEALTHY"

Verify individual cache health
"HEALTHY"

ISPN Cluster Distribution
"HEALTHY"

ISPN Overall Status
"HEALTHY"

Verify for Keycloak condition in ROSA cluster
keycloak.k8s.keycloak.org/keycloak condition met
keycloak.k8s.keycloak.org/keycloak condition met

Usage of the script with details around the different options

Usage: [-n namespace] [-l keycloak_lb_url] [-k keycloak_site_url]
[-i infinispan_rest_url] [-u infinispan_user] [-p infinispan_pwd]
[-c expected_ispn_count]

  -n namespace: Kubernetes namespace
  -l keycloak_lb_url: Keycloak Load Balancer URL
  -k keycloak_site_url: Keycloak Site URL
  -i infinispan_rest_url: Infinispan REST URL
  -u infinispan_user: Infinispan user
  -p infinispan_pwd: Infinispan password
  -c expected_ispn_count: Expected Node Count in the Infinispan cluster