Tomcat version : Tomcat 8.5.60/61 below is applied configuration in our environment 1. Realm Configuration: ----------------------- <Realm className="org.apache.catalina.realm.CombinedRealm"> <Realm className="org.apache.catalina.realm.JNDIRealm" connectionURL="ldap://XX.XX.XX.XX:389" useStartTls="true" userPattern="uid={0},ou=people,ou=accounts,dc=netact,dc=net"/> <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/> </Realm> 2. LDAP and tomcat_users.xml has user credentials(ldapuser/ldapuser). with the above configuration , Tomcat will try to authenticate the user against JNDIRealm(LDAP) first and later the UserdatabaseRealm(tomcat_users.xml). User(ldapuser) tries to authenticates when ldap is down. User(ldapuser) authentication successful for the 1st time. User(ldapuser) tries to authenticates again when ldap is down. User(ldapuser) authentication is not successful for the 2nd time. looks like there is an hung when JNDIRealm.authenticate being invoked from CombinedRealm.authenticate. JNDIRealm.authenticate not returning any response as per the generated logs. Principal authenticate(String username, String credentials) Logs: ----- Dec 27, 2020 10:59:42 PM org.apache.catalina.authenticator.AuthenticatorBase invoke FINE: Calling authenticate() Dec 27, 2020 10:59:42 PM org.apache.catalina.authenticator.FormAuthenticator doAuthenticate FINE: Authenticating username 'ldapuser1' Dec 27, 2020 10:59:42 PM com.nokia.oss.esymac.realm.CustomLockOutRealm authenticate INFO: authenticate usernameldapuser1 , credentials ldapuser1 Dec 27, 2020 10:59:42 PM com.nokia.oss.esymac.realm.CustomLockOutRealm authenticate INFO: before authenticate.. Dec 27, 2020 10:59:42 PM com.nokia.oss.esymac.realm.DefaultLockOutRealm authenticate INFO: DefaultLockouRealm start authenticate..ldapuser1 Dec 27, 2020 10:59:42 PM com.nokia.oss.esymac.realm.DefaultLockOutRealm authenticate INFO: DefaultLockouRealm stabefore authenticate..ldapuser1 Dec 27, 2020 10:59:42 PM org.apache.catalina.realm.CombinedRealm authenticate FINE: Attempting to authenticate user [ldapuser1] with realm [org.apache.catalina.realm.CombinedRealm] Dec 27, 2020 10:59:42 PM org.apache.catalina.realm.CombinedRealm authenticate FINE: Attempting to authenticate user [ldapuser1] with realm [org.apache.catalina.realm.JNDIRealm] Note : With Tomcat 8.5.59 this issue not observed. Please let us know if any other information needed.
This is caused by the refactoring which added connection pooling. I think I have an idea of what the problem is (bad cleanup if the error is when calling get() in authenticate, the connection object could possibly remain locked), but can you provide a stack trace of the thread to make sure that is the issue ?
Created attachment 37654 [details] Tomcat 8.5.60/61 User authentication with JNDIRealm failure stacktrace
Below is the stcktarce "https-jsse-nio-8443-exec-4" #70 daemon prio=5 os_prio=0 tid=0x00007f6a784f4800 nid=0x1cc8b5 waiting on condition [0x00007f6a541f0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000e80f1a68> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.apache.catalina.realm.JNDIRealm.get(JNDIRealm.java:2475) at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:1310) at org.apache.catalina.realm.CombinedRealm.authenticate(CombinedRealm.java:198) at org.apache.catalina.realm.CombinedRealm.authenticate(CombinedRealm.java:198) at com.nokia.oss.esymac.realm.DefaultLockOutRealm.authenticate(DefaultLockOutRealm.java:135) at com.nokia.oss.esymac.realm.CustomLockOutRealm.authenticate(CustomLockOutRealm.java:201) at org.apache.catalina.authenticator.FormAuthenticator.doAuthenticate(FormAuthenticator.java:244) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:634) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:240) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:616) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1634) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) - locked <0x00000000ece7e440> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)
Thanks, that's what I thought. When using a single connection, then get() locks but never unlocks if there is an exception there. I should have a patch shortly, but I'm trying to verify with a test. As a workaround, you can probably use more than one connection (connectionPoolSize="2" or more, if your ldap server allows it) which will avoid the single connection lock.
could you please provide some information with sample example on how to configure connectionPoolSize(connectionPoolSize="2"). i will test and provide the results.
Thanks for your quick responses. could you please let me know , which version of Tomcat contains the fix for this issue.
(In reply to Satya from comment #6) > Thanks for your quick responses. > > could you please let me know , which version of Tomcat contains the fix for > this issue. It is not fixed yet. (In reply to Satya from comment #5) > could you please provide some information with sample example on how to > configure connectionPoolSize(connectionPoolSize="2"). i will test and > provide the results. It is an attribute on the JNDI realm (the documentation has been updated), so it looks like: <Realm className="org.apache.catalina.realm.JNDIRealm" connectionURL="ldap://XX.XX.XX.XX:389" useStartTls="true" userPattern="uid={0},ou=people,ou=accounts,dc=netact,dc=net" connectionPoolSize="2" />
I added a test case for this. The fix will be in Tomcat 10.0.1, 9.0.42 and 8.5.62.
Thanks for reply. we will test with Tomcat 8.5.62 once its released.