AM and IG: Zero Downtime Upgrade Strategy Using a Blue/Green

Author:

Charan Mann

Created at:

Apr 2022

Updated at:

Jun 2022

The standard deployment for the ForgeRock Identity Platform consists of multiple ForgeRock products such as IG, AM, IDM, and DS. As newer ForgeRock versions are released, deployments using older versions need to be migrated before they reach their end of life. Also, newer versions of ForgeRock products provide features such as intelligent authentication and the latest OAuth standards, which help businesses implement complex use cases.

https://backstage-community-prod.storage.googleapis.com/original/1X/7ecb81942112574ccd9a5f145604b36ba643ac7f

ForgeRock Deployment Components

Problem Statement

Traditionally, ForgeRock upgrades are handled via a rolling upgrade strategy using an in-place update. This strategy doesn’t suit all deployments due to the following constraints:

  • Many deployments don’t allow any downtime. This means production servers can’t be stopped for upgrade purposes.

  • Some deployments follow an immutable instances approach. This means no modification is allowed on the current running servers.

To resolve these constraints, a parallel deployment approach, or a blue/green strategy can be leveraged for upgrading ForgeRock servers.

Solution

This article provides a high-level approach for using a blue/green methodology for updating ForgeRock AM servers and related components like DS-ConfigStore, DS-CTS, AM-Agents, and IG servers. We plan to cover similar strategies for DS-UserStores and IDM in future articles.

In order to upgrade ForgeRock deployment, we need to first analyze the dependencies between various ForgeRock products and their impact on upgrade process:

Given the dependencies between ForgeRock products, it is generally advisable to upgrade AM before upgrading DS, AM agents, and others, as new versions of AM support older versions of DS and AM agents, but the converse may not be true.

Note: There can be some exceptions to this rule. For example:

  • Web policy agents 4.x are compatible with AM 6.0, but not with AM 6.5. This means the order of upgrade shall be existing version to AM 6.0 ⇒ AM Agent 4.x to 5.x ⇒ AM 6.0 to AM 6.5.x

  • If an AM-IDM integration is used, then both AM and IDM need to be upgraded at the same time.

Upgrade Units

https://backstage-community-prod.storage.googleapis.com/original/1X/7ad3e520838d5be3753d5c612c5477c32d333ff3

ForgeRock Upgrade Units

 +
A ForgeRock Identity Platform deployment can be divided into 4 units so
that upgrade of these units can be handled individually:

  • Unit 1: AM and its related stores (DS-Config and DS-CTS)

  • Unit 2: AM-Agents/IG

  • Unit 3: DS-UserStores

  • Unit 4: IDM and its datastore

The order of upgrade used by our approach shall be Unit 1⇒Unit 2⇒Unit 3⇒Unit 4.

Unit 1: AM Upgrade

https://backstage-community-prod.storage.googleapis.com/original/1X/0041e8ab8bf4d039c97f338f3f3655569c5d0f7e

Unit 1: AM Upgrade

Prerequisites/ Assumptions

1. This approach assumes that your infrastructure processes have the ability to install a parallel deployment for upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing AM deployment (like the 13.5.x version) and the green cluster reflects a new AM deployment (like the 6.5.x version).

3. There are N+1 AM servers and corresponding config stores deployed in your existing deployment. This means N servers are used for production load, and one server is reserved for maintenance activities like backup, upgrades, and others. If there is no such maintenance server, then you may need to remove one server from the production cluster (thereby reducing production load capacity) or install an additional node (AM server and corresponding config store) for this upgrade.

4. No sessions in CTS servers are replicated during blue/green switch; therefore, users are expected to re-authenticate after this migration. If your business use cases require users to remain authenticated, then these sessions (like OAuth Refresh tokens) need to be synced from the old to the new deployment. Mechanisms like ldif export/import or using IDM synchronization engine can be leveraged for syncing selective tokens from old to new deployments. Also, refer to the AM Release Notes on session compatibility across AM versions.

5. Review the Release Notes for all AM versions between existing and target AM deployment for new features, deprecated features, bug fixes, and so on for OpenAM 13.5 to AM 6.5 upgrade. Review the Release Notes for AM 5.0, 5.5, 6.0, and 6.5 versions.

Upgrade Process

1. Unconfigure replication for the DS-3 Config store. This ensures that the upgrade doesn’t impact an existing AM deployment.

2. Upgrade AM-3 in-place using the AM upgrade process. Note: You may need to handle new AM features in this process like AM 6.5 secrets, and others.

3. Export Amster configs from AM-3.

4. Transform Amster export so that the Amster export is aligned with a new green deployment such as DS hostname:port.

5. Install AM, DS-Config, and DS-CTS servers. Import the Amster export into a new green cluster. Note: For certain deployment patterns, such as ForgeRock immutable deployment, the Amster import needs to be executed for each AM node. If a shared config store is used, then the Amster import needs to be executed only once, and other nodes are required to be added to the existing AM site.

Switch Over to the New Deployment

6. After validating that the new deployment is working correctly, switch the load balancer from blue to green. This can also be done incrementally. If any issues occur, we can always roll back to the blue deployment.

Note: Any configuration changes made after the blue’s cluster’s Amster export should be applied to both blue and green deployments so that no configuration change is lost during switchover or rollback.

Post Go-Live

7. Stop the AM servers in the blue deployment.

8. Stop the Config and CTS DS servers in blue deployment.

9. De-provision the blue deployment.

Unit 2: AM-Agent/IG Upgrade

https://backstage-community-prod.storage.googleapis.com/original/1X/f23c75bf2da68e2a8f64d8f907d9f3ca69f59632

Unit 2: AM-Agent/IG Upgrade Process

AM-Agent

Prerequisites/ Assumptions

1. This approach assumes that your deployment (including applications protected by agents) has the ability to install a parallel deployment for upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing AM-Agent deployment and the green reflects new AM-Agent deployment.

3. A parallel base green deployment for protected app servers has already been created.

4. Create new Agent profiles for green deployment on AM servers.

5. This approach assumes both old and new AM-Agent versions are supported by the AM deployment version.

6. Refer to the Release Notes for latest and deprecated features in the new AM-Agent/IG version, such as the AM-Agent 5.6 Release Notes.

Upgrade Process

1. Install AM-Agents in the green deployment. Update agent profiles on the AM server (created in #4 above) for new agents deployed in the green deployment to match configurations used in agent profiles from the blue deployment. For certain AM versions, this process can be automated by retrieving existing Agent profiles and using these results to create new Agent profiles.

Switch Over to the New Deployment

2. After validating that the new deployment is working properly, switch the load balancer from blue to green.

Post Go-Live

3. Stop the app servers in the blue deployment.

4. Remove the blue agent profiles from AM deployment.

5. De-provision the blue deployment.

IG

Prerequisites/ Assumptions

1. This approach assumes that your deployment (including applications protected by agents) has the ability to install a parallel deployment for upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing IG deployment and the green reflects the new IG deployment.

3. This approach assumes both old and new IG versions are supported by the AM deployment version.

4. Create new Agent profiles for the green deployment on the AM servers required for IG servers.

5. Refer to the Release Notes for the latest and deprecated features in the new IG version, like IG 6.5 Release Notes.

Upgrade Process

1. Update the IG configs in the git repository as per the changes in the new version. You may create a different branch in your repository for the same.

2. Deploy the new green IG deployment by leveraging updated configurations.

Switch Over to the New Deployment

3. After validating that the new deployment is working fine, switch the load balancer from blue to green.

Post Go-Live

4. Stop the IG servers in the blue deployment.

5. De-provision the blue deployment.

Conclusion

Although a blue/green deployment requires a high level of deployment maturity, this approach provides an elegant way to minimize downtime for ForgeRock deployment upgrades. It is always advisable to practice an upgrade strategy in lower environments like dev, and stage before moving to a production environment.

Depending on the complexity of your deployment, there can be multiple things to be considered for these upgrade,s such as customizations, new FR features, migration to containers, and others. It is always recommended to break the entire upgrade process into multiple releases, like “base upgrade” followed by “leveraging new features”, and so on.

References

Other Articles by This Author

https://community.forgerock.com/letter_avatar_proxy/v4/letter/c/a4c791/48 DS: Zero Downtime Upgrade Strategy Using a Blue/Green Deployment [.badge-category__name#Architecture#]

Introduction This is the continuation of the previous blog about a Zero Downtime Upgrade Strategy Using a Blue/Green Deployment for AM. Traditionally, ForgeRock Directory Server (DS) upgrades are handled via a rolling upgrade strategy using an in-place update. As many deployments have constraints around this approach (zero downtime, immutable, etc.), a parallel deployment approach, also known as a blue/green strategy, can be leveraged for upgrading ForgeRock DS servers. Note: In this article, t…

https://community.forgerock.com/letter_avatar_proxy/v4/letter/c/a4c791/48 Configuring ForgeRock AM Active/Active Deployment Routing Using IG [.badge-category__name#Architecture#]

Introduction The standard deployment pattern for ForgeRock Identity Platform is to deploy the entire platform in multiple data centers/cloud regions. This is ensures the availability of services in case of an outage in one data center. This approach also provides performance benefits, as the load can be distributed among multiple data centers. Below is the example diagram for Active/Active deployment: [image] Problem Statement AM provides both stateful/CTS-based and stateless/client-based ses…