AM 7.3, Install silently throws a java.net.ConnectException: Connection refused exception

zappee · July 14, 2023, 1:22am

Hello,

Unfortunately, I need to ask for help again. I am installing AM 7.3 using the AM-SSOConfiguratorTools.The tool throws a Connection refused exception immediately when I start it. I use this doc as a reference.

I have a well-installed DS 7.3 and I would like to use this server as a external config and identity store for this new AM instance. The DS server uses LDAPS (port 636) and it works properly. I can connect to this server using the Apache Directory Studio and the certificate that I see looks okay.

The interesting part is that if I use the web-based AM configurator (https://am.hello.com/openam) and I go through the custom config, finally I get a well-working AM server. So the manual configuration process works fine, I can add a new user using the AM web console, and I see the new entries in LDAP. But the AM configuration with openam-configurator-tool-14.1.3.18.jar does not work. In the config.properties file I use exactly the same parameters that I use during the web configuration. So I think that this Connection refused issue is not related to the LDAP server. This must be something else, but not sure. I do not know how this configurator tool works.

This is the command that I use:

type=PKCS12
store=/tmp/am.hello.com.p12
passwd=changeit

java \
   -Djava.security.debug=all \
   -Djavax.net.debug=all \
   -Djavax.net.ssl.trustStore=$store \
   -Djavax.net.ssl.trustStorePassword=$passwd \
   -Djavax.net.ssl.trustStoreType=$type \
   -Djavax.net.ssl.keyStore=$store \
   -Djavax.net.ssl.keyStorePassword=$passwd \
   -Djavax.net.ssl.keyStoreType=$type \
   -jar openam-configurator-tool-14.1.3.18.jar --file config.properties

I tried it with only the truststore Java options too, but I got the same result.

Inside the keystore, I have a server and a CA cert. I use a similar keystore on the DS side (server + same CA cert). The server certs are signed/issued by the CA.

This is the error log of the openam-configurator-tool, the config.properties, and the install.log of the manual AM installation process.

I am not sure where the configuration tool wants to connect but this connection does not work.

	at com.sun.identity.setup.OpenSSOConfigurator.postRequestToServer(OpenSSOConfigurator.java:246)
	at com.sun.identity.setup.OpenSSOConfigurator.execute(OpenSSOConfigurator.java:142)
	at com.sun.identity.setup.Main.main(Main.java:24)
java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)

I understand that the config tool JAR and the web openam WAR (tomcat) runs in two different JVM, but then…

Could you please help me?

zappee · July 14, 2023, 1:41pm

This issue is really tricky. My assumption was correct, the issue was not related to the LDAP server connection.

My first idea was to debug the openam-configurator-tool.jar, I can do it, no problem, I am a Java guy. But this is really time consuming and only God knows how it will end

Instead of the Java debug I installed tcpdump and I investigated the full network traffic of my machine while I run the ForgeRock config-tool.

I got this surprising result:

$ tcpdump --interface any -f "not port 22"

13:08:53.079133 lo    In  IP am.hello.com.32858 > am.hello.com.13034: Flags [S], seq 2208216060, win 65495, options [mss 65495,sackOK,TS val 2382642160 ecr 0,nop,wscale 7], length 0
13:08:53.079133 lo    In  IP am.hello.com.32860 > am.hello.com.13034: Flags [S], seq 2609563386, win 65495, options [mss 65495,sackOK,TS val 2382642160 ecr 0,nop,wscale 7], length 0
13:08:53.079148 lo    In  IP am.hello.com.13034 > am.hello.com.32858: Flags [R.], seq 0, ack 2208216061, win 0, length 0
13:08:53.079148 lo    In  IP am.hello.com.13034 > am.hello.com.32860: Flags [R.], seq 0, ack 2609563387, win 0, length 0

It seems that the config-tool wanna connect to port 13034 which is actually the public port of my Tomcat HTTPS port. I run everything in Docker containers and I use the following Docker port mapping config for HTTPS: "13034:443"

The Connection refused exception is correct, I have nothing listening on this port inside the container from where I run the config tool. Where this port comes from?

$ cat config.properties | grep 13034
SERVER_URL=https://am.hello.com:13034

This is the guilty guy. According to the docs, this value must point to the public URL of my AM server. But this is different inside and outside of the Docker container.

So I updated this value:
SERVER_URL=https://am.hello.com

And then, voala, the tool runs properly When the config-tool finished successfully almost immediately the LDAP TCP traffic appeared:

$ tcpdump --interface any -f "not port 22"

13:25:49.898466 eth0  In  IP ds.hello.com.docker_default.636 > am.hello.com.39216: Flags [P.], seq 1:51, ack 83, win 501, options [nop,nop,TS val 2034209853 ecr 1824831062], length 50
13:25:49.898479 eth0  Out IP am.hello.com.39216 > ds.hello.com.docker_default.636: Flags [.], ack 51, win 20754, options [nop,nop,TS val 1824831062 ecr 2034209853], length 0
13:25:49.898534 eth0  In  IP ds.hello.com.docker_default.636 > am.hello.com.46382: Flags [P.], seq 51:104, ack 83, win 501, options [nop,nop,TS val 2034209854 ecr 1824831062], length 53
13:25:49.898579 eth0  In  IP ds.hello.com.docker_default.636 > am.hello.com.39216: Flags [P.], seq 51:104, ack 83, win 501, options [nop,nop,TS val 2034209854 ecr 1824831062], length 53
13:25:49.898586 eth0  Out IP am.hello.com.39216 > ds.hello.com.docker_default.636: Flags [.], ack 104, win 20754, options [nop,nop,TS val 1824831063 ecr 2034209854], length 0
13:25:49.898588 eth0  Out IP am.hello.com.46382 > ds.hello.com.docker_default.636: Flags [.], ack 104, win 501, options [nop,nop,TS val 1824831063 ecr 2034209853], length 0
...

I am sharing this info, maybe this post can help others to save time.

But now I have a different question, please help if you could.
Now my AM server configuration points to a private port, and as you can see AM public port is different (13034). My question is, do I need to update this port somehow in order to AM can work properly?
If yes, then how can I do it from bash/command line?
Or maybe the best is to use the same HTTPS port inside and outside of the Docker container? But I do not wanna lose this flexibility…

Thanks

gery.ducatel · July 14, 2023, 3:41pm

The only thing I can think of is the JWKs URL is on port 13034, and AM believes it runs on port 443. In an OAuth 2.0 token AM will send the issuer value as https://am.hello.com (may be with the port :443), but the JWK URL is on the public port which could be rejected. This can happen also when AM is behind a firewall/load balancer. In this case you can override the issuer value in the configuration.
Now that you have AM up and running, have you observed something wrong?

NB: Out of curiosity, your jar file openam-configurator-tool.jar was it inside the container, or outside. If it was inside, I would expect that it would resolve the FQDN locally, and therefore it would find it on port 443.

zappee · July 14, 2023, 5:01pm

Inside of the container.

zappee · July 14, 2023, 5:17pm

Till now everything looks good, but I am just at the beginning of the learning curve.
ldap

gery.ducatel · July 17, 2023, 8:25am

Is there a DS Config Store, or are you using File Based Configuration (FBC)? From the picture it looks like you have a DS Config Store backend.
I cannot go into the merits of various architectures because the possibilities are too many. I will only add that ForgeRock ForgeOps relies on FBC for its static configuration data. If it is not something that you have considered yet. Please have a read here:
https://backstage.forgerock.com/docs/forgeops/7.3/index.html

zappee · July 18, 2023, 12:21am

Thanks for your useful note. I will check the mentioned docs.

zappee · July 21, 2023, 3:04pm

I went through the suggested docs but it is not clear how to install AM that way to use the mentioned File Based Configuration (FBC).

During the AM installation there are only two options for configuration store:

embedded - OpenAM embedded OpenDJ server
dirServer - ForgeRock OpenDJ or Oracle Directory Server Enterprise Edition

If I am correct all available options uses some kind of DS at the background. I can not see here FBC option to select. I am sure I misunderstood something, but what?