Reaching out to seek advise if anybody has used IDM (LDAP Connector - 1.5.20.17) to perform livesync.
We are migrating from our VM based infrastructure to use Kubernetes. We have deployed all the components in K8s and everything works fine.
We wanted to sync all the changes occurring on our K8s user-stores with our VM based user-store.
We tried creating IDM connector to sync changes between these user-stores but we are encountering lot of issues :
We plan to perform live sync every 30sec and we observe this process gets stuck sometimes for 15m or when there is slight load on K8s user-store its stuck for 30m.
For some of the attributes the timestamp is not correct. When sync happens on VM user-store it registers that time on contrary to the original time with our K8s user-store.
High memory requirement is noticed.
We played around with pool config and that also did not help.
The Connector UI has some problems we noticed most of the times we get 400 respone with “The resource instance account cannot be queried”
Before we proceed ahead with this approach wanted to understand if this solution is reliable and can coup up with daily updates (lastLoginTime) ?
Kindly let me know your opinion if you have used livesync in your deployments.
To answer your question, yes, I’ve used livesync with the LDAP connector. Not the particular version you’ve specified, and not specifically on 7.4, but I think I can still help with some troubleshooting.
First off, can you tell us a bit about the LDAP directories you are using? What products are you syncing between? Are you just syncing user objects, or groups as well?
Can you elaborate a bit about your comment on timestamps not being correct? I’m a bit confused as you state that for certain attributes timestamps are not correct, but individual attributes don’t have timestamps. What timestamp are you referencing?
How many users (and if applicable, groups) are in your source directory? Do you have any metrics on the rate of changes? If you are using useTimestampsForSync in the connector config, do you have the appropriate indexes in place? Where are you seeing the slowness? Reading from the source? Syncing to IDM? Publishing to target?
How functional is your configuration if you take liveSync out of the picture? Can you execute a reconciliation without issue? Can you successfully read from VM and write to K8s via the REST APIs?
So to recap, the short answer is yes, liveSync has been used with the IDM LDAP connector. There is, however, a lot more information that I feel we’d need in order to help you troubleshoot this.
We are using Ping DS as LDAP directories.
The Ping DS acts as idmRepository as well as identityStore.
With K8S DS we are on 7.4 version . We have separate Replication Servers configured.
While with VM DS we are using 7.2 and its a standalone instance i.e DS and RS are on same host.
We have created 2 LDAP connectors in K8S IDM to use livesync functionality to sync data from K8S DS servers to VM DS servers:
Customer - Connects to VM DS Servers for Customer
CustomerK8S - Connects to K8S DS server for Customer
We have creating 1 mapping to sync user objects data: systemCustomerk8sAccount_systemCustomerAccount - Maps Customer attributes in K8S DS to be synced to VM DS.
We have created below 1 scheduler for Livesync to be run after every 30 secs:
SyncCustomerFromK8sToVm
The attributes createDateTime & lastPasswordChange have property “writability” as “readOnlyDiscardWrites” in repo.ds.json, so these attribute values when synced to VM DS servers gets value as the time when sync gets completed. So when scheduler gets stuck, these values changes significantly leading to discrepancy in the data in VM DS servers.
We have 1.2 million records where we see daily logons around 800K. With every logon lastLoginTime gets updated.
Thank @mwtech for your response. I and @ajay.shedge work in the same team, I will try to answer your questions.
I will start with our infrastructures, we have VM based & K8S based infrastructure.
Our current environment is VM based where we have IDM instances hosted across 2 sites that connects to VM based Forgerock Directory Servers that acts as IDM repository & user store.
We are migrating to K8S based environment where also we have IDM instances hosted across 2 sites that connects to K8S based Directory Servers acting as IDM repository & user store. We have created connectors on K8S based IDM to initiate LiveSync from K8S DS to sync changes to VM based Directory servers.
Our plan is to decommission VM based IDM servers but retain VM based user stores for data redundancy. Hence, we are using LiveSync functionality to sync data from K8S DS to VM DS using K8S IDM connectors.
Regarding Livesync, we are using timestamps for livesync. We are seeing slowness when K8S IDM tries to sync data to VM based DS servers after checking the update in source (K8S DS). The scheduler gets stuck for around 15 mins and then when the request reaches VM based DS, it immediately syncs data.
We haven’t tried reconciliation yet.
Hope this information helps in understanding our requirement.
Do you have bidirectional recon mappings by any chance? What sync model are you using: DS (VM) ↔ DS (k8s), or DS (VM) ↔ Managed ↔ DS (K8s)? Any policy validation errors (in the latter case) with a sync retry in place?
Is there a reason why you’ve opted to use timestamps for the sync token vs the changelog, as recommended by ForgeRock (see Knowledge - ForgeRock BackStage)? In general, have you followed the advice in this KB article? It’s short but it’s good, and it cautions against the timestamp method due to performance issues.
Generally speaking, I like to get the connector configs working in basic use cases. This way, as I delve into more complex cases like liveSync I’m already confident in the core configurations and can focus on troubleshooting areas specific to my use case.
I’d like to suggest the following:
Read the linked KB article and ensure you follow the advice given. If you are unable to apply the best practices, please share the reasoning behind that to help our understanding.
Confirm some basic functionality in your connector configurations. At the system level, I’d try doing some basic read queries to ensure the connector is able to do a basic request without issues. If you encounter issues with that, troubleshoot further. Look at the DS access log to see the search and observe performance and see if there are red flags there.
After these steps have been followed, work to pinpoint the phase of the liveSync where the issues are occurring. This will help us understand where the bottleneck is and we can focus on resolving the specific issues.
One more question - are you testing this all out in a non-prod environment where you have flexibility to play with settings?
As we will be migrating to K8s and once we do a complete switchover to K8s we need the data sync from the K8 DS to VM DS for a seamless rollback option, hence we are using liveSync. The IDM instances of K8s have the firewall opened to connect to the VM DS and facilitate the sync from K8s to VM DS. Our IDM and AM points to the same OU in the DS. The changes to the objects(users and credentials) will be done by both IDM and AM. AM does the changes to the user/credential object such as updating the last login timestamp or the user’s active status (inetUserStatus) etc, so we are using livesync to have the AM changes included. The reason for chosing timestamps based is we don’t use any delete operations and also we have replication servers hosted on different instances instead on the same instance as that of DS servers.
We tried reconciliation but the problem with using reconcilation is that it can be run only once a day and it’s won’t serve our purpose for seamless rollback. Also, the attributes createDateTime & lastPasswordChange gets updated to the time when the recon finishes as these attributes have property “writability” as “readOnlyDiscardWrites” in repo.ds.json and we are not syncing these attributes to VM. So this leads to discrepancy in the data for the newly created objects in VM DS servers as sometimes lastlogintime is say morning’s but createdatetime is updated to night time when recon completes.
We tried setting the log level to FINEST and we observed that, IDM queries the Source which is the K8s DS when the livesync with timestamp is triggered and this query to the source is very fast, as it finishes with few millisecs. Then for the cases that get stuck, IDM does some processing internally which is taking time (and as for the logs available we see that this is the step where it takes time) before it connects to the target VM DS. In most cases, when the stuck happens we see that it takes around 15 minutes in IDM before it connects to the target VM DS to get the objects and do a compare or create new objects or sync to the target. In other words, before establishing the connection to the target VM DS, IDM spent 15 mins and we couldn’t understand where it was taking the time. Once it establishes a connection to the target VM then everything finishes within a few milliseconds.
And yes, we are testing this all out in a non-prod environment.
I’ve got a few more questions here, as I am a bit confused on how your systems are setup.
In the initial post it stated that you were moving from VM to K8S, but then said you want to sync everything from K8S to VM. Just to help with my understanding, can you clarify what is happening?
Is it correct to state that one of your DS instances is currently being used for both the AM user repo as well as the IDM repo? And the other DS instances are not being used at all, but will be in the future?
What sync model are you using: DS (VM) ↔ DS (k8s), or DS (VM) ↔ Managed ↔ DS (K8s)?
Is it correct to state that sometimes livesync works without issue, and that the large delay that you experience is intermittent? If so, how frequently does it happen? Is the delay always 15 minutes (or close to that)?
I think I’m confused about something regarding your setup, as it sounds like you have DS both as your IDM repo as well as a target system. Is it possible to ask for you to put together a simple diagram of the architecture and attach that? It would be great if you could capture the current state as well as the target state.
After we go-live with K8s and for some unforeseen reason we have to perform rollback the livesync option gave us quick rollback mechanism, hence we were exploring it .
If everything goes as per the plan we will decommission AM/IDM/IG applications hosted on VM but for data redundancy we will keep the Directory servers component for few more months hence the necessity to perform livesync from K8s (DS) <–>VM (DS).
With our current VM setup, All our directory servers instances have am-identity-store and idm-repo profile setup. All of directory servers are identical and are part of same replication topology.
AM/IDM connect to all this DS instances.
We have setup K8s DS instances similarly just the replication servers are hosted on different PODS.
This issue is not intermittent, we observe the livesync gets stuck daily multiple times sometimes for 15m or 30m.
We have reached out to Forgerock support and seeking help under a support ticket. The feedback provided was to use implicitSync but we observe AM changes are not getting synced. The sync model we are using DS (k8s) ↔ DS (VM)
So I may have missed something about all of this - is there a particular reason why you aren’t just using replication to keep the directory data in sync?
As with everything in life test thoroughly, but I believe this should be a suitable solution for your problem. I do suspect that with further troubleshooting we could identify the cause of the performance issues you are experiencing, but at this time I think that replication would be a much cleaner way to address your use case.
Thanks @mwtech for your response. Our DS(VM) is on 7.3.3 and DS(K8S) is on 7.4.1. In VM, we have replication servers and DS servers are on same host but in K8S, we have external replication servers. Do you think that replication would still work in that case? We might explore one of the DS VM instances in K8S replication topology.
I don’t see any reason why the viability of this proposed solution would be reduced due to the replication servers existing on different hosts. Their functionality remains the same regardless of whether or not they are colocated with the directory servers.