How Often Are Metrics For DS Being Updated

Troubleshooting
, , ,
#1

We have the prometheus endpoint being scraped on our datastore servers. I am attempting to create a metric for the amount of time that it takes to perform ldap searches/binds/adds/deletes etc…

When I hit the metrics endpoint and grep for specific actions, it appears as though the metrics for those requests hardly ever change. For example, In Grafana, if I look at the ds_connection_handlers_ldap_request_seconds for ‘search’ on a specific host and the .5 quantile for that monitor has been the exact same for 7 days straight. There are thousands of search requests that hit hit that server every minute. Yet, the metric never changes off of the exact same value which is a little more than 2 milliseconds.

Here is our config on the prometheus endpoint it seems like we’re neither excluding metrics nor including them. Any thoughts on why those values are available, yet never changing?

    Property                 Value(s)
    --------------------------------------------
1)  authorization-mechanism  HTTP Basic
2)  base-path                /metrics/prometheus
3)  enabled                  true
4)  excluded-metric-pattern  -
5)  included-metric-pattern  -
#2

You can extract those statistics directly from DS. You need a service account that has access to cn=monitor data, see:
https://backstage.forgerock.com/docs/ds/7.3/monitoring-guide/ldap-monitoring.html
You can pull all stats for LDAPS:
$ bin/ldapsearch --nopropertiesfile --hostname localhost --port 1636 --bindDN uid=monitor --bindpassword password --searchScope base --baseDN "cn=LDAPS,cn=connection handlers,cn=monitor" --useSsl --trustAll "(&)" \*

Or target a specific operation (e.g. search with scope sub):
$ bin/ldapsearch --nopropertiesfile --hostname localhost --port 1636 --bindDN uid=monitor --bindpassword password --searchScope base --baseDN "cn=LDAPS,cn=connection handlers,cn=monitor" --useSsl --trustAll "(&)" ds-mon-requests-search-sub

You can also run a support extract job and get all stats in a file called monitor.ldif:
https://backstage.forgerock.com/knowledge/kb/article/a84846841

Having said that, I checked the Prometheus endpoint and my observations are similar to yours. I also connected Grafana, and the values being plotted do change, so I am not entirely sure why at this point.

1 Like
#3

Looking at the stats out of DS (which is the actual source) percentiles are re-calculated at frequent intervals but not exactly real time. So, live changes may not be visible right away. Also a lot of stats start from the moment DS is started, therefore if DS has been running for a very long time, it may be that some stats do not change much any more.
You may evaluate live performance with the provided rating tools:
https://backstage.forgerock.com/docs/ds/7.3/getting-started/performance.html
With regards

1 Like