Audit Logging in AM and IDM for ForgeOps
Hi all, I would like to start with that these articles are as much for my benefit as they are for yours. They are my notes on how to do some things.
The ForgeOps team has been worked quite a lot on providing different deployment strategies for ForgeOps. I would like to particularly like to thank Lee Baines-Dark for looking into this issue and implementing the changes required.
Please note the forgeops repository is in continuous development and might have changed since this post was written.
Logging in a Kubernetes environment
Logs inside of a pod can be accessed by running the kubectl logs <podName> command.
Use kubectl get pods command to obtain the pod name that is needed. An example output of the command looks like this:
NAME READY STATUS RESTARTS AGE
admin-ui-6645d49486-b8cwk 1/1 Running 0 147m
am-7b5bfb67bd-td75j 1/1 Running 0 84m
ds-idrepo-0 1/1 Running 0 20h
end-user-ui-74bd5fc9c4-22znb 1/1 Running 0 64m
idm-8548776488-8txzl 1/1 Running 0 84m
login-ui-74c7d4476d-6r9cz 1/1 Running 0 84m
To fetch the logs for IDM in this CDK deployment, would be kubectl logs idm-8548776488-8txzl. CDM deployments of ForgeOps will likely have more than one IDM pod.
Since kubectl logs needs to be run on each pod, it is very difficult to monitor the logs of all the pods in an environment at once. Therefore, Kubernetes logging does not provide a complete logging solution and further options need to be explored for production deployment.
They way Kubernetes captures logs is via a container’s standard output (stdout) and standard error (stderr) streams in a running pod. ForgeRock products provide handlers for directing this output to the pod filesystem. These are the options that will be used to modify the logging level.
Logging Limitations
The stdout and stderr logs are stored in a file called log-file.log located within a cluster node and not on a pod’s filesystem. Therefore logs do not increase disk space a pod needs. The disk space of the cluster node that hosts the pod is utilised instead.
Log files are rotated once they exceed 10MB or once a day. Kubernetes maintains 5 log files. Whenever the kubectl logs command is run, it fetches the last log file only.
The log files can also be accessed by connecting to the cluster node directly. If the node uses systemd, the logs can be found using journald, by running the journalctl command.
If the container within a pod is somehow terminated, the logs are retained.
If a pod contains no containers, it is terminated.
If the pod hosting the container is terminated, the logs are lost.
If the node hosting the pod is somehow terminated, the pod is also automatically terminated and restarted where there is compute resources are available.
ForgeRock recommends that a ForgeOps administrator implements a log aggregator to gather logs from many pods, and that logs are retained after incidents and outages.
Audit Logging in AM
Default Behavior
Audit logs are enabled by default in a ForgeOps deployment. The built-in behaviour for AM is to output audit logs into stdout and stderr and can be identified by their format. The output is similar to other logs,but can be differentiated from other logs by the "source":"audit"tag. A common audit log that can be observed is by logging into the admin console.
This output is fetched using :
$ kubectl logs <nameOfAmPod>
And looks like this:
{"timestamp":"2023-07-12T10:37:24.764Z","eventName":"AM-LOGIN-MODULE-COMPLETED","transactionId":"d66b1499-8f96-4e43-b029-5653547d9325-295","trackingIds":["d66b1499-8f96-4e43-b029-5653547d9325-298"],"principal":["idm-resource-server"],"entries":[{"moduleId":"Application","info":{"authIndex":"module_instance","authControlFlag":"REQUIRED","moduleClass":"Application","ipAddress":"10.20.0.16","authLevel":"0"}}],"result":"SUCCESSFUL","realm":"/","component":"Authentication","source":"audit","topic":"authentication","level":"INFO","_eventId":"d66b1499-8f96-4e43-b029-5653547d9325-300"}
{"timestamp":"2023-07-12T10:37:24.767Z","eventName":"AM-LOGIN-COMPLETED","transactionId":"d66b1499-8f96-4e43-b029-5653547d9325-295","trackingIds":["d66b1499-8f96-4e43-b029-5653547d9325-298"],"userId":"id=idm-resource-server,ou=agent,ou=am-config","principal":["idm-resource-server"],"entries":[{"moduleId":"Application","info":{"authIndex":"module_instance","authIndexValue":"Application","ipAddress":"10.20.0.16","authLevel":"0"}}],"result":"SUCCESSFUL","realm":"/","component":"Authentication","source":"audit","topic":"authentication","level":"INFO","_eventId":"d66b1499-8f96-4e43-b029-5653547d9325-302"}
{"timestamp":"2023-07-12T10:37:24.804Z","eventName":"AM-ACCESS-OUTCOME","transactionId":"d66b1499-8f96-4e43-b029-5653547d9325-295","trackingIds":["d66b1499-8f96-4e43-b029-5653547d9325-298"],"userId":"id=idm-resource-server,ou=agent,ou=am-config","client":{"ip":"10.20.0.16","port":47832},"http":{"request":{"secure":true,"method":"POST","path":"https://am/am/oauth2/introspect","headers":{"accept":["application/json"],"content-type":["application/x-www-form-urlencoded"],"host":["am"],"user-agent":["Apache-HttpAsyncClient/4.1.4 (Java/11.0.18)"]}}},"request":{"detail":{"resourceOwnerId":"id=amadmin,ou=user,ou=am-config"}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":105,"elapsedTimeUnits":"MILLISECONDS","detail":{"scope":"fr:idm:*","token_type":"Bearer","client_id":"idm-admin-ui","username":"amadmin","active":true}},"realm":"/","component":"OAuth","source":"audit","topic":"access","level":"INFO","_eventId":"d66b1499-8f96-4e43-b029-5653547d9325-304"}
{"timestamp":"2023-07-12T10:37:25.111Z","eventName":"AM-ACCESS-OUTCOME","transactionId":"d66b1499-8f96-4e43-b029-5653547d9325-305","trackingIds":["d66b1499-8f96-4e43-b029-5653547d9325-208"],"userId":"id=amadmin,ou=user,ou=am-config","client":{"ip":"10.20.0.10","port":58168},"server":{"ip":"10.20.0.17","port":8081},"http":{"request":{"secure":true,"method":"GET","path":"https://dev.example.com/am/json/global-config/realms/","headers":{"accept":["application/json, text/plain, */*"],"accept-api-version":["protocol=2.0,resource=1.0"],"host":["dev.example.com"],"user-agent":["Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"],"x-forwarded-for":["38.92.97.65"],"x-forwarded-host":["dev.example.com"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-real-ip":["38.92.97.65"],"x-request-id":["30f71f60fac705676753b6962009ddde"],"x-scheme":["https"]}}},"request":{"protocol":"CREST","operation":"QUERY"},"response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":31,"elapsedTimeUnits":"MILLISECONDS"},"component":"Config","source":"audit","topic":"access","level":"INFO","_eventId":"d66b1499-8f96-4e43-b029-5653547d9325-309"}
This behaviour can be modified by editing options that are available in the admin console. First, the console must be selected from the left hand menu.
Inside the AM native console the location of the Audit logging configuration is inside of Configure/Global Services.
Within which, audit logging can be selected.
This appears like the example below and is slightly different from the on-prem AM audit logging services.
Making Modifications
Press the tick mark, to disable audit logging. Additionally the whitelist and blacklist filters can be applied here to control the information that is outputted. For more information visit the advanced options, under org.forgerock.openam.audit.identity.activity.events.blacklist
Additional granularity can be gained from modifying the Secondary Configurations window. This displays StdOut event handler.
Once the edit button is pressed, the configuration related to StdOut and the org.forgerock.openam.audit.events.handlers.JsonStdoutAuditEventHandlerFactory can be seen.
In this window the granularity of the topics that is controlled can be changed. Currently this window is fully populated. However events such as Access , Activity, Configuration and Authentication can be removed and added at a later date.
Once all the changes that need to be made are applied. Please remember to save the changes.
For the purposes of this exercise the change that will be implemented is simply disabling audit logging.
Implementing the new configuration
To implement the new configuration in ForgeOps a user must export the confing, rebuild images and push to their fork of ForgeOps, using the GitOps model.
From inside of a local fork of forgeops repository, the config must be exported:
$ bin/config export am <nameOfConfigProfile> --sort
Where <nameOfConfigProfile> is an uninterrupted string.
Once complete, a new AM image can be built using this config for quick implementation by running:
$bin/forgeops build am --config-profile <nameOfConfigProfile> --push-to <nameofRepo>
Where <nameOfRepo> is a viable Docker repository.
To permanently apply the changes onto a cluster, AM must be reinstalled. In this particular case it is reinstalled onto a CDK cluster using:
$ bin/forgeops install am --cdk --config-profile <nameOfConfigProfile>
This makes the changes to the cluster permanent and should the AM pod fail in any way, Kubernetes will restore the pod to a working condition using the configuration that has been exported.
To verify the changes have taken place the logs can be viewed using:
$ kubectl logs <nameOfAmPod>
In this output there are no more audit logs.
{"timestamp":"2023-07-12T14:26:22.234Z","level":"INFO","thread":"http-nio-8081-exec-1","mdc":{"transactionId":"0a215267-69b5-40e4-a803-7192978688f5-255"},"logger":"com.sun.identity.authentication.client.AuthClientUtils","message":"CookieDomains : 'null'","context":"default","transactionId":"0a215267-69b5-40e4-a803-7192978688f5-255"}
{"timestamp":"2023-07-12T14:26:23.326Z","level":"WARN","thread":"http-nio-8081-exec-3","mdc":{"transactionId":"0a215267-69b5-40e4-a803-7192978688f5-266"},"logger":"org.forgerock.oauth2.core.BasicOAuth2RequestImpl","message":"Could not parse request body: No content to map due to end-of-input\n at [Source: (BufferedReader); line: 1, column: 0]","context":"default","transactionId":"0a215267-69b5-40e4-a803-7192978688f5-266"}
{"timestamp":"2023-07-12T14:26:23.357Z","level":"INFO","thread":"http-nio-8081-exec-3","mdc":{"transactionId":"0a215267-69b5-40e4-a803-7192978688f5-266"},"logger":"com.sun.identity.authentication.client.AuthClientUtils","message":"CookieDomains : 'null'","context":"default","transactionId":"0a215267-69b5-40e4-a803-7192978688f5-266"}
{"timestamp":"2023-07-12T14:26:23.406Z","level":"WARN","thread":"http-nio-8081-exec-5","mdc":{"transactionId":"0a215267-69b5-40e4-a803-7192978688f5-271"},"logger":"org.forgerock.oauth2.core.BasicOAuth2RequestImpl","message":"Could not parse request body: No content to map due to end-of-input\n at [Source: (BufferedReader); line: 1, column: 0]","context":"default","transactionId":"0a215267-69b5-40e4-a803-7192978688f5-271"}
{"timestamp":"2023-07-12T14:26:23.421Z","level":"INFO","thread":"http-nio-8081-exec-5","mdc":{"transactionId":"0a215267-69b5-40e4-a803-7192978688f5-271"},"logger":"com.sun.identity.authentication.client.AuthClientUtils","message":"CookieDomains : 'null'","context":"default","transactionId":"0a215267-69b5-40e4-a803-7192978688f5-271"}
10.20.0.10 - - [12/Jul/2023:14:26:19 +0000] "GET /am/json/serverinfo/* HTTP/1.1" 200 527 95ms
10.20.0.10 - - [12/Jul/2023:14:26:20 +0000] "POST /am/json/realms/root/authenticate?goto=https%3A%2F%2Fdev.example.com%2Fam%2Foauth2%2Fauthorize%3Fredirect_uri%3Dhttps%3A%2F%2Fdev.example.com%2Fplatform%2FappAuthHelperRedirect.html%26client_id%3Didm-admin-ui%26response_type%3Dcode%26state%3D70Eo9TuLvy%26scope%3Dopenid%2520fr%3Aidm%3A*%26code_challenge%3DotXTMdA--pw9XU5TeDusWmUomIIhcJgorHwC_PelTDw%26code_challenge_method%3DS256 HTTP/1.1" 200 642 70ms
10.20.0.10 - - [12/Jul/2023:14:26:21 +0000] "POST /am/json/realms/root/authenticate?goto=https%3A%2F%2Fdev.example.com%2Fam%2Foauth2%2Fauthorize%3Fredirect_uri%3Dhttps%3A%2F%2Fdev.example.com%2Fplatform%2FappAuthHelperRedirect.html%26client_id%3Didm-admin-ui%26response_type%3Dcode%26state%3D70Eo9TuLvy%26scope%3Dopenid%2520fr%3Aidm%3A*%26code_challenge%3DotXTMdA--pw9XU5TeDusWmUomIIhcJgorHwC_PelTDw%26code_challenge_method%3DS256 HTTP/1.1" 200 441 90ms
10.20.0.10 - - [12/Jul/2023:14:26:21 +0000] "POST /am/json/users?_action=idFromSession HTTP/1.1" 200 481 18ms
Once satisfied with the quality of the update a git commit can be carried out to incorporate the changes into the local fork and continue on to be integrated into CDM and other environments.
Audit Logging in IDM
Default Behavior
Audit Logging in a ForgeOps Deployment is enabled by default. The built-in behaviour for IDM is to output audit logs into stdout and stderr and can be identified by their format. The output is similar to other logs, but it is differentiated by the "source":"audit"tag. A common action that can be observed in IDM audit logs is the log in, into the admin console.
The log output can be fetched by running the following command:
$ kubectl logs <nameOfIdmPod>
And can appear like this:
{"_id":"811575bb-a705-4d3d-abd6-70439ce86219-460","timestamp":"2023-07-13T10:54:21.424Z","eventName":"access","transactionId":"811575bb-a705-4d3d-abd6-70439ce86219-456","userId":"anonymous","client":{"ip":"10.20.0.1","port":50888},"server":{"ip":"10.20.0.11","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://10.20.0.11:8080/openidm/info/ping","headers":{"accept":["*/*"],"host":["10.20.0.11:8080"],"user-agent":["kube-probe/1.25"]}}},"request":{"protocol":"CREST","operation":"READ"},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":2,"elapsedTimeUnits":"MILLISECONDS"},"roles":["internal/role/openidm-reg"],"source":"audit","topic":"access","level":"INFO"}
{"_id":"811575bb-a705-4d3d-abd6-70439ce86219-483","timestamp":"2023-07-13T10:54:27.414Z","eventName":"access","transactionId":"811575bb-a705-4d3d-abd6-70439ce86219-479","userId":"anonymous","client":{"ip":"10.20.0.1","port":50890},"server":{"ip":"10.20.0.11","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://10.20.0.11:8080/openidm/info/ping","headers":{"accept":["*/*"],"host":["10.20.0.11:8080"],"user-agent":["kube-probe/1.25"]}}},"request":{"protocol":"CREST","operation":"READ"},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS"},"roles":["internal/role/openidm-reg"],"source":"audit","topic":"access","level":"INFO"}
{"_id":"811575bb-a705-4d3d-abd6-70439ce86219-498","timestamp":"2023-07-13T10:54:32.976Z","eventName":"access","transactionId":"811575bb-a705-4d3d-abd6-70439ce86219-494","userId":"anonymous","client":{"ip":"38.92.97.59","port":45606},"server":{"ip":"10.20.0.11","port":8080},"http":{"request":{"secure":true,"method":"GET","path":"https://dev.example.com/openidm/config/ui/themerealm","headers":{"accept":["application/json, text/plain, */*"],"host":["dev.example.com:443"],"user-agent":["Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"],"x-forwarded-for":["38.92.97.59"],"x-forwarded-host":["dev.example.com"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-real-ip":["38.92.97.59"],"x-request-id":["9df48b9e2c6fa009644acbcb4bf148af"],"x-requested-with":["XMLHttpRequest"],"x-scheme":["https"]}}},"request":{"protocol":"CREST","operation":"READ"},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":4,"elapsedTimeUnits":"MILLISECONDS"},"roles":["internal/role/openidm-reg"],"source":"audit","topic":"access","level":"INFO"}
{"_id":"811575bb-a705-4d3d-abd6-70439ce86219-505","timestamp":"2023-07-13T10:54:36.705Z","eventName":"access","transactionId":"811575bb-a705-4d3d-abd6-70439ce86219-501","trackingIds":["c196fae6-3603-40ab-b782-31c8c03cd989-163"],"userId":"openidm-admin","client":{"ip":"38.92.97.59","port":45606},"server":{"ip":"10.20.0.11","port":8080},"http":{"request":{"secure":true,"method":"GET","path":"https://dev.example.com/openidm/info/uiconfig","headers":{"accept":["application/json, text/plain, */*"],"host":["dev.example.com:443"],"user-agent":["Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"],"x-forwarded-for":["38.92.97.59"],"x-forwarded-host":["dev.example.com"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-real-ip":["38.92.97.59"],"x-request-id":["8a76e5555d8dc2d8c26cf4d9d237a90c"],"x-scheme":["https"]}}},"request":{"protocol":"CREST","operation":"READ"},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":11,"elapsedTimeUnits":"MILLISECONDS"},"roles":["internal/role/openidm-admin","internal/role/openidm-authorized"],"source":"audit","topic":"access","level":"INFO"}
{"_id":"811575bb-a705-4d3d-abd6-70439ce86219-511","timestamp":"2023-07-13T10:54:37.011Z","eventName":"access","transactionId":"811575bb-a705-4d3d-abd6-70439ce86219-507","trackingIds":["c196fae6-3603-40ab-b782-31c8c03cd989-163"],"userId":"openidm-admin","client":{"ip":"38.92.97.59","port":45606},"server":{"ip":"10.20.0.11","port":8080},"http":{"request":{"secure":true,"method":"POST","path":"https://dev.example.com/openidm/authentication","headers":{"accept":["application/json, text/plain, */*"],"host":["dev.example.com:443"],"user-agent":["Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"],"x-forwarded-for":["38.92.97.59"],"x-forwarded-host":["dev.example.com"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-real-ip":["38.92.97.59"],"x-request-id":["c7b2b27a982267c8267d0004160b1442"],"x-scheme":["https"]}}},"request":{"protocol":"CREST","operation":"ACTION","detail":{"action":"login"}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":7,"elapsedTimeUnits":"MILLISECONDS"},"roles":["internal/role/openidm-admin","internal/role/openidm-authorized"],"source":"audit","topic":"access","level":"INFO"}
This behaviour can be edited through the options available in the admin console provided by the ForgeRock Platform. To access these controls, first select the IDM native console.
The audit configuration settings reside within the System Preferences, that can be found under the Configure tab.
The first section on display is Auditing. There the three handlers can be observed as enabled by default in a ForgeOps Deployment. In particular the json-stdout handler is responsible for outputting audit logs to stdout where they can be ingested by Kubernetes API.
Once the pencil icon is pressed it will allow for further editing of the related audit event topics.
Here most of the options are already configured by default. With only recon not available, and can be enabled by selecting it, and pressing the Submit button.
Additionally the enabled toggle is on, letting us know that this Audit Event Handler is enabled.
Making Modifications
For the purposes of this exercise, the focus will be on a simple edit. In this case simply disabling this Audit Event Handler. This is achieved by pressing the toggle, and pressing the Submit button.
Once disabled there is a need to confirm the changes that are made. The user is alerted to this need using the notification that appears in the menu.
In order to apply the changes the Submit button must be pressed for the entirety of the System Preferences window. This is at the bottom of the page.
Implementing the new configuration
In order for the changes that were carried out to be saved, the user must undergo the GitOps model that is implement in ForgeOps.
From inside of the local fork of ForgeOps, please use the config export tool:
$ bin/config export idm <nameOfConfigProfile> --sort
Where <nameOfConfigProfile> is an uninterrupted string.
Once complete, a new IDM image can be build using the exported configuration:
$bin/forgeops build idm --config-profile <nameOfConfigProfile> --push-to <nameofRepo>
Where <nameOfRepo> is a viable Docker repository.
To perminantly apply the changes onto a cluster, IDM must be reinstalled. In this particular case it is reinstalled onto a CDK cluster using:
$ bin/forgeops install idm --cdk --config-profile <nameOfConfigProfile>
This makes the changes to the cluster permanent and should the IDM pod fail in any way, Kubernetes will restore the pod to a working condition using the configuration that has been exported.
To verify the changes have taken place we view the logs using:
$ kubectl logs <nameOfAIdmPod>
In this output there are no more audit logs.
[109] Jul 13, 2023 2:04:46 PM org.eclipse.jetty.io.ChannelEndPoint updateKey
FINE: Key interests updated 0 -> 1 on SocketChannelEndPoint@3757cc6e{l=/10.20.0.22:8080,r=/10.20.0.16:43780,OPEN,fill=FI,flush=-,to=1/300000}{io=1/1,kio=1,kro=1}->HttpConnection@61f00459[p=HttpParser{s=START,0 of -1},g=HttpGenerator@55cecf9b{s=START}]=>HttpChannelOverHttp@7818b4cb{s=HttpChannelState@42772aba{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=8,c=false/false,a=IDLE,uri=null,age=0}
[109] Jul 13, 2023 2:04:46 PM org.eclipse.jetty.io.ManagedSelector$SelectorProducer processUpdates
FINE: updates 0
[109] Jul 13, 2023 2:04:46 PM org.eclipse.jetty.io.ManagedSelector$SelectorProducer select
FINE: Selector sun.nio.ch.EPollSelectorImpl@69a66c74 waiting with 1 keys
[85] Jul 13, 2023 2:04:48 PM org.forgerock.openidm.quartz.RepoJobStore acquireNextTriggers
FINE: Attempting to acquire the next trigger batch
[85] Jul 13, 2023 2:04:48 PM org.forgerock.openidm.repo.opendj.impl.NestedOuTypePathVisitor existingPathForType
FINE: Getting path for type scheduler/triggers
[85] Jul 13, 2023 2:04:48 PM org.forgerock.openidm.quartz.RepoJobStore handleDeferredTriggerJobCompletion
FINE: Processing 0 deferred Trigger Job Completions
[85] Jul 13, 2023 2:04:48 PM org.forgerock.openidm.quartz.RepoJobStore acquireNextTriggers
FINE: Attempting to process misfired triggers
[85] Jul 13, 2023 2:04:48 PM org.forgerock.openidm.quartz.RepoJobStore acquireNextTriggers
FINE: No waiting triggers to acquire
[85] Jul 13, 2023 2:04:48 PM org.quartz.core.QuartzSchedulerThread run
FINE: batch acquisition of 0 triggers
Please note for the purposes of this exercise the FINER logging level is used, in such a way that there is log activity during the login event outside of Audit logs.
If the changes are successfully applied and the client is satisfied with the results, changes can be committed to the local ForgeOps fork, and implemented in a CDM environment.
Additional considerations
Please note that this is an incomplete audit logging solutions for a production environment. An important consideration is the retention of these logs. At this current moment in time logs related to audit logging disappear as soon as the related pod terminates. This behaviour will likely not achieve regulatory compliance.
Additionally, a production deployment has more than one AM pod and so the logs from one pod will offer an incomplete audit picture.
A further consideration is the retention policy for these logs, as well as the deletion protection around them. A typical solution would be to regularly copy log output to cold storage with limited permission that are only available to auditors.
ForgeRock recommends that each individual client carries out and implements a complete logging strategy that ensures security and business continuity using the tools that have been provided in addition to first and third part tools that match business needs.