ForgeRock Identity Cloud allows you to connect your tenant to an external data source such as an LDAP server or database. From time to time there is a need to count the total number of users (or objects) in such a datastore.
This blog presents a programmatic way to execute a count quickly and easily in Identity Cloud using the .../system endpoint.
As a pre-requisite, a Remote Connector Server (RCS) needs to be deployed close the datastore and a Connector needs to be configured in ForgeRock Identity Cloud in. See this link for more on this.
The implementation also makes use of paging to ensure the remote datastore is not overwhelmed when there are many millions of users or objects.
Note — some datastores do not support paging, be sure to test in lower environments first.
In order to execute the script either a tenant admin access token or an IDM scoped Service Account access token (see this blog for more on Service Accounts) is required.
In order to execute the script run:
./system_object_count.sh <access_token_here>
e.g.
./system_object_count.sh eyJ0eXAiOiJKV1QiLCJraWQiOiI3UEtESGd3Rnpp.....
The output will look something like this:
********************
********************
Execution cycle: 1
Result count from query: 100000
Total user count: 100000
Continuing - All user records not returned. Current Page Results cookie value is 100000
Request/response execution time: 0 minutes and 13 seconds
Total request/response execution time: 0 minutes and 13 seconds
********************
Execution cycle: 2
Result count from query: 100000
Total user count: 200000
Continuing - All user records not returned. Current Page Results cookie value is 200000
Request/response execution time: 0 minutes and 13 seconds
Total request/response execution time: 0 minutes and 26 seconds
********************
Execution cycle: 3
Result count from query: 100000
Total user count: 300000
Continuing - All user records not returned. Current Page Results cookie value is 300000
Request/response execution time: 0 minutes and 14 seconds
Total request/response execution time: 0 minutes and 40 seconds
********************
Execution cycle: 4
Result count from query: 18562
Total user count: 318562
Continuing - All user records not returned. Current Page Results cookie value is 400000
Request/response execution time: 0 minutes and 3 seconds
Total request/response execution time: 0 minutes and 43 seconds
Script Breakdown
The script defines a number of variables:
FQDN — this is the fully qualified domain name of the ForgeRock Identity Cloud tenant. For example https://openam-mytenant.forgerock.io
CONNECTOR — the name of the connector defined in Identity Cloud. To check this in the Platform UI > Native Consoles > Identity Management > Configure to menu > Connectors > The name is visible in the UI. In this example it’s LDAP
ACCOUNT_IDENTIFIER — the object type for the account. To check this in UI go to the above location > click the connector > Object Types tab. In this example ACCOUNT_IDENTIFIER should be account
The default for a ForgeRock Directory Services connector is account For databases this is typically __ACCOUNT__
PAGED_RESULTS_COOKIE — the start value for the paged results cookie, which is used to keep track of where the search query has got to. For ForgeRock Directory Services this should be set to '' For databases this is typically a value or 0 (note without quotes)
PAGE_SIZE — the number of objects to return from the remote datastore with each query. Default value is 100K, tune down if the remote datastore resource are limited or the dataset is small
DEBUG — if set to true the HTTP response for each call is output. Default value is false
The script then executes in a while loop, counting the number of objects per execution cycle, the total number of cumulative objects, and the response time for each call until the PAGED_RESULTS_COOKIE returns as null
Note you can hit Ctrl + C to cancel the script at any time if needed.
Conclusion
There you have it, an easy programatic approach for counting the total numbers of users and objects from a remote datastore in ForgeRock Identity Cloud.