I’m working on a connector and it appears to be working(neat). As i am working on the recon or sync side of things I’m trying to understand. when does or what causes IDM to trigger the executequery method and the sync method. I’ve gotten the executequery method to trigger when I manually recon a mapping, but are there other things that trigger it? and what will cause the sync method to fire?
Hi Rob,
Let’s start by the easier: Sync. It is triggered whenever the system enpoint’s livesync action is invoked: Manage liveSync :: IDM 7.3.0.
For Query, it is either triggered by a CREST read or query at the system endpoint. Then the ICF framework is going to invoke the Query operation for an Update operation as well (after the update has been performed), and perhaps for the Create operation as well (before creating the resource). But as long as you have correctly implemented the connector operations and tested the functionality at the system endpoint, you should not worry much of this latter aspect. I must say that the Query operation is the one that needs the most care - and is probably the most involved - as it must obey to paging controls, sorting, and implement/convert the provided filter to the target resource API.
I may already shared this link with you, if not, this might be of interest (this is Groovy, but yet the methodology rightfully applies to Java as well): https://community.forgerock.com/t/developing-a-groovy-connector
Regards
Patrick
2 Likes
thank you for this, it was very helpful. I see better now how they work. One other question, I notice when I trigger a sync the getLastToken isn’t called. What triggers that?
How are you triggering the sync? Executing the livesync via the REST API directly or via the admin UI as shown in the link @patrick_diligent shared should both show you the latest token which getLastToken returned, either in the response from the call or via the UI.
@mwtech In fact getLatestSyncToken is called at the first ever livesync or after the sync token is reset. This call should return a token for the latest change in the resource, and not the last token returned by the previous livesync run.
So @rob.kimball to see getLatestSyncToken in action, reset the sync token this way (change according to connector name/object type):
DELETE /repo/synchronisation/pooledSyncStage/SYSTEMMSSQLACCOUNT
where the connector name is “mssql” and object type is “account”
Then launch a livesync: POST /system/?_action=liveSync&source=system/mssql/account
and you should see a trace for the getLatestSyncToken - but no sync occurring.
So if no update occurred in the resource since the sync token has been reset (e.g deleted), the livesync will not sync anything. That’s why it is important to have an initial reconciliation while livesync is already enabled.
So in this particular example above, that’s what shows in the log file after I reset the sync token:
Aug. 16, 2023 11:55:06 am DEBUG o.i.f.api.operations.SyncApiOp: Enter: getLatestSyncToken(ObjectClass: __ACCOUNT__) Method: getLatestSyncToken
Aug. 16, 2023 11:55:06 am DEBUG o.i.f.api.operations.SyncApiOp: Return: SyncToken: 44002 Method: getLatestSyncToken
Get Latest Sync Token is invoked first (no sync token yet in the DB), then the Sync operation shows up, but did not find anything to sync (normal, no changes occurred since resetting the sync token), and return the sync token passed into the operation, untouched.
Regards
Patrick
1 Like
Thanks for the correction there, and sorry for the misinformation. Guess that shows it’s been a couple of years since I last wrote a connector!
thank you both of you. it is very appreciated. One thing I’m trying to wrap my head around is though, if getlatestsynctoken is getting the current latest token, it should always result in a no change sync, what is the use case for that? I can see a use case to clear the token and get EVERYTHING(like a full recon situation)…I guess livesync isn’t’ meant to be treated that way?
Based on what @patrick_diligent said, getLatestSyncToken is only called if IDM does not have a sync token. IDM needs to know where to start querying from on the target system, and that is what the token is for. Once IDM has acquired a sync token, it persists that in the repo and uses that for subsequent liveSync operations.
For example, let’s say objects in your target system have an attribute named updateDate which tracks when the object was last updated and you are using this value for the sync token. When IDM doesn’t have a sync token for the system and getLatestSyncToken is called, your connector would find the object with the most recent updateDate value and return that value as the sync token. On subsequent liveSync calls to the connector IDM will pass in the sync token and the connector will query the target system for records with an updateDate greater than the sync token value and return those objects along with an updated syncToken reflecting what the most recent updateDate value now is. IDM then persists that new syncToken value, and that is used for the next liveSync call.
liveSync is usually used once an initial reconciliation has taken place, as a means of keeping IDM up-to-date with the latest data from the target system.
1 Like
To be clear in the order in which to apply reconciliation and livesync in the initial provisioning:
- Configure a livesync schedule
- Once the first livesync run has occurred (sync token established), launch a reconciliation
This to avoid missing updates happening after a reconciliation task complete (e.g sync of a single object in the reconciliation run) and before the first livesync run (as the sync token will be initialised with the latest change - which can be potentially later than the latest change captured by reconciliation if the source is subject to update traffic during the process). This is particularly important when there is no plan to establish a reconciliation schedule in production (which is a valid proposition for very large datasets e.g several 10M identities) but just rely on livesync for keeping the identities in sync.
1 Like
If you can’t tell, I come from an Oracle IDM background so some of the concepts are a bit new to me. So it sounds like… if bringing a new app on. once the connector is there. you’d do an initial recon(executequery) to bring in the records. Then configure livesync schedule to keep things up to date(sync) If the volume is excessively large you’d NOT do regular full recons and the ONLY incremental is done via livesync, otherwise livesync is for incremental and reconciliation is for full. Do I understand that all ok?
Hi Rob,
Always perform a full reconciliation after the first livesync run (the one that initialises the sync token) so that to capture all changes, then livesync will keep identities in sync as long as there is no network interruptions (so be sure to configure retries on error). If not, when the system is under update traffic, some changes will be forever missed (until the same resources are updated of course).
Whether or not to schedule reconciliations for a large dataset depend on many factors - the most important one is how long it takes (which depends on dataset size and system performances) - and secondly business requirement - what it the acceptable sync time? When livesync is in place, these matters are of a less concern, what matters then is recovering from livesync errors, where different strategies could be adopted. Could be one of the following, to name the least:
- Schedule a full reconciliation to recover missed changes (can take a long time)
- Capture missed changes from the log, and perform individual reconById actions. Means, have a robust monitoring solution in place.
Regards
Patrick
1 Like