Multi-Cluster Deployment Using Google Cloud Multi-Cluster Ingress and

Author:

Lee Baines

Created at:

Jan 2022

Updated at:

May 2022

Google docs:
Multi-Cluster Ingress
Set Up Multi-Cluster Ingress
Deploy Multi-Cluster Ingress

Overview

This article explains how to deploy the ForgeRock Identity Platform across two different clusters, and configure proximity-based routing with multi-cluster ingress.

Features:

  • Fully meshed multi-cluster DS tolopology using CloudDNS for GKE

  • Global HTTP(S) load balancing across multiple clusters with Multi Cluster Ingress

  • Health checks to manage application level failover between clusters

  • Proximity-based routing

  • Active/active or failover

  • Blue/green deployments

Step 1: Prerequisites

1. Cloud SDK version

  • Cloud SDK version 290 or later.

2. Select Anthos pricing model
Anthos ingress controller is a Google hosted service which is used by Multi Cluster Ingress.
Choose your Anthos pricing model here:

3. Enable MCI APIs

gcloud services enable gkehub.googleapis.com
gcloud services enable anthos.googleapis.com # Enable if required based on the previous section
gcloud services enable multiclusteringress.googleapis.com

Step 2: Cluster provisioning and DS setup

  • Follow the clouddns readme to set up US and EU clusters and deploy DS using Cloud DNS for GKE.

  • Enable HTTP load balancing on the clusters. This can be applied either at cluster creation time or after the clusters are created.

  • After following the above steps you should have the following:

    • HTTP load balancing and workload identity enabled on both clusters

    • Secret Agent Operator deployed

    • DS deployed and communicating across both clusters

  • Now you need to register your new clusters to a fleet. Multi-cluster works only on clusters that are registered to the same fleet. Ensure:

  • <cluster_location> matches the location of the cluster master.

  • <cluster_name> matches the exact name of the cluster.

  • <project_id> matches your Google Cloud Project ID.

gcloud container hub memberships register <cluster_name> \
--gke-cluster <cluster_location>/<cluster_name> \
--enable-workload-identity \
--project=<project_id>

Verify membership by running the following command:

gcloud container hub memberships list --project=<project_id>

Step 3: Config cluster

For more info on configuring the config cluster, see a Google doc here: Setting up Multi Cluster Ingress  |  Google Kubernetes Engine (GKE)  |  Google Cloud.

Enable the US cluster as the config cluster:

gcloud alpha container hub ingress enable --config-membership=clouddns-us

Verify the config cluster:

gcloud alpha container hub ingress describe

Output should look something like:

multiclusteringressFeatureSpec:
  configMembership: projects/<project id>/locations/global/memberships/<cluster-membership-name>

Step 4: MCI Preparation

1. Create a Static IP
A static IP is required for the HTTP(S) load balancer. See the Google doc here: https://cloud.google.com/kubernetes-engine/docs/how-to/multi-cluster-ingress-setup#specifying_a_config_cluster

Add the IP address to the MultiClusterIngress file using the static-ip annotation:

annotations:
  networking.gke.io/static-ip: <static ip address>

2. SSL cert for the HTTP(S) load balancer There are various options for generating an SSL cert for external traffic to the HTTP(S) load balancer.

For the first two options, update the MultiClusterIngress file using the pre-shared-certs annotation:

annotations:
networking.gke.io/pre-shared-certs: "certname"

3. Configure FQDN
Add FQDN to the host property in the MultiClusterIngress file.

rules:
- host: <fqdn>

4. Deploy the custom resources to the config server

cd etc/multi-cluster/google-cloud/multi-cluster-ingress/mci
kubectl apply -f mci.yaml -n prod
kubectl apply -f mcs.yaml -n prod
kubectl apply -f backendconfig.yaml -n prod

For more information on the above resources see the Google doc links below:

Step 5: Deploy Frontend Apps

1. Deploy AM, IDM and the UIs
This example reflects a CDM medium size deployment which is the size used for testing. Please adjust to suit your requirements.

bin/forgeops install apps ui --medium

2. Verify deployment

Things to check:

Step 6: Delete the deployment

Carry out the following commands in each cluster:

bin/forgeops delete apps ui

Step 7: Delete the multi-cluster ingress configuration

1. Remove configs from config cluster

From the config cluster:

kubectl delete mcs --all
kubectl delete backendconfig --all
kubectl delete mci forgerock

2. Remove clusters from fleet

gcloud container hub memberships delete clouddns-eu
gcloud container hub memberships delete clouddns-us

Step 8: Pricing

https://backstage-community-prod.storage.googleapis.com/original/2X/b/b6c2ea1c77424a6cd28390d725434a18f9aa5e1b Google Cloud

/uploads/default/optimized/1X/f74dbc6eb29149f30e5241925000cd050ff4594f_2_690x362

Pricing  |  Google Kubernetes Engine (GKE)  |  Google Cloud

Review pricing for GKE

Step 9: Testing

The following tests have been carried out across two clusters in different regions unless specified (failover is triggered by scaling down product to 0 pods):

  • Verify geolocation.

  • Test failover between europe-west1 and europe-west2/europe-west2 and us-west-2

  • Failover IDM while creating users.

  • Test failover of AM across 3 clusters in different regions.

  • Test split load by running test against 1 location with low available resources.

  • Failover IDM based on idrepo being down.

  • Failover AM which running AuthN and Access Token simulations.

  • Test failover across 3 different clusters.

  • Running performance tests for 1,6 and 12 hour durations.

  • Running AM performance tests with stateful and stateless tokens.

  • Running load simulataneously across 2 clusters and forcing failover of AM.

TO DO

  • Use health check status to manage failovers:

    • Fail over IDM if AM is down

    • Fail over AM if idrepo or cts is down

    • Fail over IDM if idrepo is down

  • Enable SSL endpoint of AM

  • Resolve throuput ussues after 7 hours durin dual load performance testing

  • IG/RCS agent?
    +

Other Articles by This Author

https://community.forgerock.com/letter_avatar_proxy/v4/letter/l/6de8d8/48 Multi-Cluster Deployment for DS on GKE Using Cloud DNS for GKE [.badge-category__name#Setup#]

Cloud DNS for GKE is currently in Preview. Google documentation here. Overview This article explains how to deploy DS across two different GKE clusters, using one cluster in the US and one cluster in Europe. For different cluster names, just replace eu and us in the various files mentioned in the setup steps below. There are 5 major steps to the deployment: Prepare two clusters configured to use Cloud DNS with one in the US and one in Europe. Enable the use of shared secrets across the clus…

https://community.forgerock.com/letter_avatar_proxy/v4/letter/l/6de8d8/48 Deploying ForgeRock Directory Services on a Kubernetes Multi-Cluster using Google Cloud Multi-cluster Services (MCS) [.badge-category__name#Setup#]

This article shows you how to deploy a multi-region ForgeRock Directory Services (DS) solution that spans two Google Kubernetes Engine (GKE) clusters from different regions using a multi-cluster solution (MCS). This solution lets pods in one GKE cluster to discover pods in another GKE cluster, which simplifies configuration by automating creation of external DNS and firewall rules. Note: We will use the standard stateful applications (DS-CTS and DS-IDREPO) to deploy in each of the two GKE clust…