Hi all, I would like to start by saying this article is as much for me as it is for you. It contains notes on how to do some things.
The ForgeOps team has been worked quite a lot on providing different deployment strategies for ForgeOps. I would like to particularly like to thank Lee Baines-Dark for looking into this issue and implementing the changes required.
Please note the forgeops repository is in continuous development and might have changed since this post was written. These steps were tested on branch 7.3-20230609.
Logging in a Kubernetes environment
Logging is enabled by default to some level in all of our products.
Logs inside of a pod can be accessed by running the kubectl logs <podName> command.
Use kubectl get pods command to obtain the pod name that is needed. An example output of the command looks like this:
NAME READY STATUS RESTARTS AGE
admin-ui-6645d49486-b8cwk 1/1 Running 0 147m
am-7b5bfb67bd-td75j 1/1 Running 0 84m
ds-idrepo-0 1/1 Running 0 20h
end-user-ui-74bd5fc9c4-22znb 1/1 Running 0 64m
idm-8548776488-8txzl 1/1 Running 0 84m
login-ui-74c7d4476d-6r9cz 1/1 Running 0 84m
To fetch the logs for IDM in this CDK deployment, would be kubectl logs idm-8548776488-8txzl. CDM deployments of ForgeOps will likely have more than one IDM pod.
Since kubectl logs needs to be run on each pod, it is very difficult to monitor the logs of all the pods in an environment at once. Therefore, Kubernetes logging does not provide a complete logging solution and further options need to be explored for production deployment.
They way Kubernetes captures logs is via a container’s standard output (stdout) and standard error (stderr) streams in a running pod. ForgeRock products provide handlers for directing this output to the pod filesystem. These are the options that will be used to modify the logging level.
Logging Limitations
The stdout and stderr logs are stored in a file called log-file.log located within a cluster node and not on a pod’s filesystem. Therefore logs do not increase disk space a pod needs. The disk space of the cluster node that hosts the pod is utilised instead.
Log files are rotated once they exceed 10MB or once a day. Kubernetes maintains 5 log files. Whenever the kubectl logs command is run, it fetches the last log file only.
The log files can also be accessed by connecting to the cluster node directly. If the node uses systemd, the logs can be found using journald, by running the journalctl command.
If the container within a pod is somehow terminated, the logs are retained.
If a pod contains no containers, it is terminated.
If the pod hosting the container is terminated, the logs are lost.
If the node hosting the pod is somehow terminated, the pod is also automatically terminated and restarted where there is compute resources are available.
ForgeRock recommends that a ForgeOps administrator implements a log aggregator to gather logs from many pods, and that logs are retained after incidents and outages.
Modifying the logging level by temporarily modifying the related idm-logging-properties configmap.
This action is not permanent and will be undone by reapplication and redeployment of ForgeOps. It is however useful for temporarily increasing the visibility into the logs and can be just as easily undone.
To ensure that the required resources are available and the version of 7.3-20230609 is used, run the kubectl get configmaps, the result of which should show as follows:
am-logback 1 3h11m
amster-files 2 22h
amster-retain 1 22h
dev-utils 6 22h
idm 10 22h
idm-logging-properties 1 22h
kube-root-ca.crt 1 20d
platform-config 24 22h
The configmap of interest is idm-logging-properties
To carry out the edit, please run:
$ kubectl edit configmap idm-logging-properties
This will open the configmap in vi by default or the value set in $EDITOR :
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
logging.properties: |
############################################################
# Levels in descending order are:
# SEVERE (highest value)
# WARNING
# INFO
# CONFIG
# FINE
# FINER
# FINEST (lowest value)
############################################################
# --- Handler ---
handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = ALL
java.util.logging.ConsoleHandler.formatter = org.forgerock.openidm.logger.SanitizedThreadIdLogFormatter
java.util.logging.ConsoleHandler.filter = org.forgerock.openidm.logging.util.LogFilter
# --- Loggers ---
# Global levels
.level = INFO
#org.forgerock.openidm.level = FINE
#org.forgerock.openidm.provisioner.level = FINER
# OpenICF is noisy at INFO level
org.forgerock.openicf.level = WARNING
# Logs the output from OSGi logging
org.forgerock.openidm.Framework.level = WARNING
# On restart the BarURLHandler can create warning noise
org.activiti.osgi.BarURLHandler.level = SEVERE
# Suppress warnings of failed connector loading
org.identityconnectors.framework.impl.api.local.LocalConnectorInfoManagerImpl.level = SEVERE
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"logging.properties":"############################################################\n# Levels in descending order are:\n# SEVERE (highest value)\n# WARNING\n# INFO\n# CONFIG\n# FINE\n# FINER\n# FINEST (lowest value)\n############################################################\n\n# --- Handler ---\nhandlers = java.util.logging.ConsoleHandler\njava.util.logging.ConsoleHandler.level = ALL\njava.util.logging.ConsoleHandler.formatter = org.forgerock.openidm.logger.SanitizedThreadIdLogFormatter\njava.util.logging.ConsoleHandler.filter = org.forgerock.openidm.logging.util.LogFilter\n\n# --- Loggers ---\n# Global levels\n.level = FINEST\n#org.forgerock.openidm.level = FINE\n#org.forgerock.openidm.provisioner.level = FINER\n\n# OpenICF is noisy at INFO level\norg.forgerock.openicf.level = WARNING\n# Logs the output from OSGi logging\norg.forgerock.openidm.Framework.level = WARNING\n# On restart the BarURLHandler can create warning noise\norg.activiti.osgi.BarURLHandler.level = SEVERE\n# Suppress warnings of failed connector loading\norg.identityconnectors.framework.impl.api.local.LocalConnectorInfoManagerImpl.level = SEVERE\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"idm","app.kubernetes.io/component":"idm","app.kubernetes.io/instance":"idm","app.kubernetes.io/name":"idm","app.kubernetes.io/part-of":"forgerock","tier":"middle"},"name":"idm-logging-properties","namespace":"suner"}}
creationTimestamp: "2023-06-19T14:31:24Z"
labels:
app: idm
app.kubernetes.io/component: idm
app.kubernetes.io/instance: idm
app.kubernetes.io/name: idm
app.kubernetes.io/part-of: forgerock
tier: middle
name: idm-logging-properties
namespace: suner
resourceVersion: "565925520"
Please note that unlike AM, the idm-logging-properties configmap does not contain any mention of its polling duration. This is because the file is only queried on start up. This means for changes to take effect the pod must be restarted.
The line that is important for this edit is the log level .level = INFO . By default the logging level is set to INFO, this level has a fuller description in the official IDM Documentation.
To change the logging level, this variable needs to be changed. For the purposes of this demonstration it will be altered to FINEST, as this is a very verbose output.
The pod must be restarted for the desired changes to take effect. This is done by executing the kubectl delete pod <nameOfIdmPod>.
The desired output can be verified by running kubectl logs <nameOfIdmPod> once the new pod has started:
...
FINEST: DECODE LDAP SEARCH RESULT(messageID=76, result=Result(resultCode=Success, matchedDn=, diagnosticMessage=, referrals=[], controls=[]))
[88] Jun 22, 2023 1:51:18 PM org.forgerock.i18n.slf4j.LocalizedLogger trace
FINEST: READ ASN.1 END SEQUENCE
[88] Jun 22, 2023 1:51:18 PM org.forgerock.i18n.slf4j.LocalizedLogger trace
FINEST: Rest2Ldap transactionId=2ddbaedc-5ee5-4648-9e6a-9cfc66b2f0f6-344 LDAP complete result for resource [generic]: 0 entry
[88] Jun 22, 2023 1:51:18 PM org.forgerock.i18n.slf4j.LocalizedLogger trace
FINEST: Rest2Ldap transactionId=2ddbaedc-5ee5-4648-9e6a-9cfc66b2f0f6-344 JSON complete result for resource [generic]: 0 entry
...
To return it to normal, simply repeat these steps with the variable set as INFO instead of FINEST
Modifying the logging level by permanently modifying the Kubernetes environment and setting a new default logging level
This action is permanent modification to the deployment.
This is done by modifying the kustomize/base folder to use a new default state for the application.
To begin, the logging.properties file will need to be modified. It is located in the /path/to/forgeops/kustomize/base/idm-cdk/config folder. An easy way to do so is to use the following command to run vim:
$ vim /path/to/fogeops/kustomize/base/idm-cdk/config/logging.properties
Once opened the file looks like this:
############################################################
# Levels in descending order are:
# SEVERE (highest value)
# WARNING
# INFO
# CONFIG
# FINE
# FINER
# FINEST (lowest value)
############################################################
# --- Handler ---
handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = ALL
java.util.logging.ConsoleHandler.formatter = org.forgerock.openidm.logger.SanitizedThreadIdLogFormatter
java.util.logging.ConsoleHandler.filter = org.forgerock.openidm.logging.util.LogFilter
# --- Loggers ---
# Global levels
.level = INFO
#org.forgerock.openidm.level = FINE
#org.forgerock.openidm.provisioner.level = FINER
# OpenICF is noisy at INFO level
org.forgerock.openicf.level = WARNING
# Logs the output from OSGi logging
org.forgerock.openidm.Framework.level = WARNING
# On restart the BarURLHandler can create warning noise
org.activiti.osgi.BarURLHandler.level = SEVERE
# Suppress warnings of failed connector loading
org.identityconnectors.framework.impl.api.local.LocalConnectorInfoManagerImpl.level = SEVERE
To modify the logging level, .level = INFO sets the global logging level. For the purposes of this article we will modify this to .level = FINEST and save the file.
To apply the changes to an existing cluster run:
$ bin/forgeops install idm --<size> --config-profile <yourConfigProfile> --namespace <yourNamespace>
If the ForgeOps deployment is initialised for the first the first time on a new cluster the --fqdn <yourFqdn> flag will need to be added.
The desired output can be verified by running kubectl logs <nameOfIdmPod>:
...
FINEST: READ ASN.1 END SEQUENCE
[86] Jun 22, 2023 1:57:39 PM org.forgerock.i18n.slf4j.LocalizedLogger trace
FINEST: DECODE LDAP SEARCH RESULT(messageID=568, result=Result(resultCode=Success, matchedDn=, diagnosticMessage=, referrals=[], controls=[]))
[86] Jun 22, 2023 1:57:39 PM org.forgerock.i18n.slf4j.LocalizedLogger trace
FINEST: READ ASN.1 END SEQUENCE
[86] Jun 22, 2023 1:57:39 PM org.forgerock.i18n.slf4j.LocalizedLogger trace
FINEST: Rest2Ldap transactionId=2ddbaedc-5ee5-4648-9e6a-9cfc66b2f0f6-1185 LDAP complete result for resource [generic]: 0 entry
[86] Jun 22, 2023 1:57:39 PM org.forgerock.i18n.slf4j.LocalizedLogger trace
FINEST: Rest2Ldap transactionId=2ddbaedc-5ee5-4648-9e6a-9cfc66b2f0f6-1185 JSON complete result for resource [generic]: 0 entry
...
To return it to normal, simply repeat these steps with the variable set as INFO instead of FINEST
Conclusion
There are two methods for enabling logging in IDM for ForgeOps:
- Temporarily modifying the
idm-logging-propertiesconfigmap - Permanently altering ForgeOps to set a new default logging level
The final piece of the puzzle would be the implementation of an external log ingester. By default, Google Kubernetes Engine provides logs in the form of Google Cloud Monitoring and Logging (Formerly Stackdriver). Enabling first-party tools or using third-party tools is recommended for Azure and AWS.