IDM: Zero Downtime Upgrade Strategy Using a Blue/Green Deployment

patrickdiligent · August 26, 2021, 4:00am

This article provides instructions for a zero downtime upgrade strategy for IDM using a blue/green deployment.

Introduction

This article is a continuation of an article about upgrading AM and IG using a in-place or by leveraging the migration service. As many deployments have constraints around this approach (zero downtime, immutable, ...), a parallel deployment approach or a blue/green strategy can be leveraged for upgrading ForgeRock IDM servers.

This corresponds to the Unit 4: IDM Upgrade in our overall ForgeRock upgrade approach.

Unit 4: IDM Upgrade

Determining the correct approach

IDM 6.5 is bundled with the migration service, allowing the migration of data from a blue environment to a green environment, table by table, row by row, leading to identical data on both sides. The constraint is that update traffic must be cut off in the blue environment to ensure consistent data in the green environment after the migration completes. It is not a Zero Downtime migration.

The method exposed here enables a Zero Downtime migration.

This article should be seen as a basis for your migration strategy. As IDM dependencies are very elaborated and varied, it is very difficult to design a perfect strategy that will work with all kinds of deployments. The table below exposes the major differences between the migration service method and the scripted CREST method:

Before engaging in using the methodology exposed in this article, please check that you really need Zero Downtime. It may be that the amount of data is too big to fall within an overnight window, which prevents it from using the migration service. The solution to this may be overcome with the following measures:

Verify that the blue system nodes are performance optimized to ensure the lowest possible response time.
Allocate more nodes in the green environment, and distribute the migration between the nodes by partitioning the mappings.

If none of the above is possible, then please read on.

Prerequisites/Assumptions

This approach assumes that your infrastructure processes have the ability to install a parallel deployment for upgrade or you are already using blue/green deployment.
In above diagram blue cluster reflects existing IDM deployment (like 4.5.x version) and green reflects new IDM deployment (like 6.5.x version).
Review release notes for all IDM versions between existing and target IDM deployment for new, deprecated features, bug fixes, and others, such as IDM 4.5 to IDM 6.5 upgrade, review the release notes for IDM 5.0, 5.5, 6.0 and 6.5 versions.
Verify that the deployment upgrade Zts with a blue/green deployment. It requires copying repository data over. When data transformation is not required, such as upgrading from 6.0 to 6.5, sharing the repository, which is not a pure blue/green deployment, might be the best fit.
When external systems are the source of truth for data, the IDM repository can be rebuilt through a full reconciliation. Therefore, the methodology exposed in this paper is not relevant for this case.

Upgrade the process using the CREST scripted connector

Prepare the migration:

Clone the crestidmmigration project from BitBucket, and read the project description so that you can decide whether this method fits the requirements. While implicit sync always occurs in the blue environment, decide which environment to initiate the reconciliation to.
Decide which strategy to employ for external resources (such as a ForgeRock® Directory Server). As described in the project, you can migrate the repository links table using the migration service, as the CREST migration preserves the managed object IDs. Alternatively, you may perform a full reconciliation after the migration. In all cases, turn off implicit sync for all mappings in the green environment to avoid unnecessary (and perhaps, conflictual) traffic to the external provisioning systems.
Follow the instructions in the Installation Guide to prepare for a migration, except that you are not going to perform a migration.
Provide all property mappings in sync.json. For encrypted values such as password, Step 2 in the Installation Guide ensures that both environments share the same encryption keys. Provide all necessary configuration to optimize the reconciliation (in particular, paged reconciliation). However, do not alter the correlationQuery.
Edit the onCreate and onUpdate scripts to add processing for all relationships. These scripts ensure that relationship properties are propagated to the target, but also filters out any relationship to resources that are not yet provisioned. This prevents duplicate entries.
Edit provisioner.openicf-scriptedcrest.json to include all properties that are to be migrated. Be careful not to change the configuration for the _id property, and examine how relationship properties are configured (see devices for multivalued, and owner for single value).
Perform a blue/green upgrade to deploy the CREST scripted connector for the 4.x or 5.x nodes. Ensure that implicit sync is disabled for all mappings involving the CREST connectors. The resulting green environment becomes the blue environment in the next migration phase.
Deploy the green environment (for example, 6.5), prepared for migration, including the CREST connector, if reconciliation is to be launched from the green nodes.

Turn on implicit sync in blue, and launch the reconciliation of each scripted CREST mapping, one after the other completes (so it does not run concurrently). You may use the migration service to migrate other data such as internal roles, internal users, and links.

Switch over to the new deployment:

After validating that the new deployment is working correctly, switch the load balancer from blue to green, and turn off implicit sync to external provisioning systems in blue. Then turn on implicit sync to external provisioning systems in green, and perform a reconciliation with the external provisioning systems, or use a custom method if provisioning is not performed through reconciliation.
In case of any issues, you can always roll back to blue deployment.

Note: Any managed object or configuration change made after switchover should be applied to both blue and green deployments so that no change is lost during rollback.

Post go-live instructions

Stop the migration service on the green deployment.
Stop blue IDM servers.
Deprovision blue deployment.

Conclusion

Although a blue/green deployment requires a high level of deployment maturity, this approach provides an elegant approach for minimising downtime for ForgeRock deployment upgrades. It is always advisable to practice upgrade strategy in lower environments like dev, stage before moving to the production environment.

Depending on the complexity of your deployment, multiple things may need to be considered for these upgrades such as customisations, new FR features, and others. It is always recommended to break the entire upgrade process into multiple releases like “base upgrade” followed by “leveraging new features” and so on.

When deploying IDM for the first time, it is always advisable to incorporate the upgrade strategy early on in the project so that any designed feature allows for seamless migration in the future. Also performing syncing to the green environment is going to impact the blue update performance as implicit syncs are all executed synchronously on the request’s thread (on 4.x, 5.x. and 6.0). Fortunately, this does not apply anymore to 6.5 when queued sync is enabled. The impact will be sensible with a high relationship cardinality, as the process requires interrogating the target system for each relationship before propagating it. This is why planning the upgrade strategy well in advance is important.

The provided CREST scripts and configuration are a starting point, and a proof of concept, which you can take as a basis to build your own based on the deployment requirements. The details are described at the crestidmmigration project. Note that a second alternative solution is proposed there, one that preserves relationships which allows running the migration service from the green environment (CREST+MIGRATION folder); however, this may contribute to a higher performance hit on the blue environment. Please note that crestidmmigration is an evolving project, and as such, some other variants could be proposed in the future, so stay tuned!