Bug 58244 - two way SSL loses client certificate after a few requests
Summary: two way SSL loses client certificate after a few requests
Alias: None
Product: Tomcat Native
Classification: Unclassified
Component: Library (show other bugs)
Version: 1.1.33
Hardware: PC All
: P2 normal with 10 votes (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
Depends on:
Reported: 2015-08-14 15:00 UTC by David Balažic
Modified: 2017-08-21 14:16 UTC (History)
1 user (show)

Test case to reproduce issue (9.72 KB, application/zip)
2015-08-26 11:25 UTC, David Balažic
100 % repeatable test case (17.30 KB, application/x-gzip)
2015-10-30 09:36 UTC, Petr Brouzda
Potential patch if OpenSSL decide this is a WONTFIX (2.06 KB, patch)
2016-02-21 16:49 UTC, Mark Thomas
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description David Balažic 2015-08-14 15:00:46 UTC
When accessing a web application that uses client certificate authentication run on Tomcat/APR (on Windows) with Firefox or Chrome, the client cert is "lost" after a short while. To the app it appears the client certificate was not sent.

Example code (JSP fragment, can be the only content of a JSP file):

User client cert data:
<%= ((java.security.cert.X509Certificate[])

After a few refreshes of the page (where it will show the client certificate DN) the page will fail with a NullPointerException as request.getAttribute will return null. It usually happens in less than a minute. To be more precise: when reloading about once per second, the problem occurs almost every time after 30 seconds. After that each request will fail the same way, until I restart tomcat.

This happens with Firefox (v39 and v40) and Chrome (v44), but not with IE v11.

It also occurs with different versions of tomcat and Java (and OS bitness) - see below for a list.

A simple test case using latest versions is:

    download and extract apache-tomcat-8.0.24-windows-x64.zip
    in the webapps folder create a folder named cert, there create a file named ccertA.jsp that contains the above code snippet

    in server.xml add a line:

    <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol" secure="true" scheme="https" maxThreads="150" URIEncoding="UTF-8" SSLVerifyClient="optional" SSLProtocol="TLSv1+TLSv1.1+TLSv1.2" SSLPassword="testing" SSLEnabled="true" SSLCertificateKeyFile="C:/your_server_key_private.pem" SSLCertificateFile="C:/ your_server_key _public.pem" SSLCACertificateFile="C:/supported_client_CAs.pem" />

    start tomcat by executing startup.bat
    open the page https://localhost:8443/cert/ccertA.jsp and keep refreshing it every few seconds
    After about 30 seconds it will show a NPE exception error page.

If I don't use APR (by deleting the tcnative-1.dll file and adapting the connector syntax for JSSE) the problem does not happen.

Tried versions, all having the issue:

    apache-tomcat-8.0.24-windows-x64 (also 32 bit version) - has APR 1.5.1 and TCN 1.1.33
    Java 1.6.0 Updates 12 and 45
    Java 1.8 Update 51
    Windows 7 Pro SP1 64 bit
    Windows 7 Pro SP1 32 bit
    Window 8.1 Pro 64 bit
    Windows 10 Home 64 bit
    Firefox versions 39.0 39.0.3 and 44
    Chrome v44
    Ubuntu 14.04 LTS 64 bit / tomcat 7.0.52-1ubuntu0.3 / libapr1:amd64 1.5.0-1 / libtcnative-1:amd64 1.1.29-1

A similar issue was discussed on the tomcat-users mailing list in 2010: "Client certificate gone after 1 minute timeout (SSL, APR)" [1] but with no solution. I posted there myself recently ( "Firefox SSL with APR - losing client certificate" [2] has a bit more details), but it is basically just my monologue.

Originally I tested with a server certificate issued by my private testing CA. Now I also tried with a "real" certificate issued by trusted CA. (I used my personal certificate. The browser complained about the hostname mitmatch which I clicked away).

Can upload test CA, server-cert/key and client cert/key for test if requiered.

[1] http://grokbase.com/t/tomcat/users/102pdv412y/client-certificate-gone-after-1-minute-timeout-ssl-apr
[2] http://www.mail-archive.com/users@tomcat.apache.org/msg118902.html
Comment 1 David Balažic 2015-08-26 11:25:09 UTC
Created attachment 33041 [details]
Test case to reproduce issue

The issue persists with the new apache-tomcat-8.0.26-windows-x64.

I attach a WAR file that can be used to reproduce the issue. Either unpack the single JSP file in it and put it into the webapps folder under its own folder (like tomcat/webapps/x/a.jsp) or deploy the WAR file under webapps/.

The WAR file also contains the CA's, server and client certificates that can be used for the test.

 - download and extract apache-tomcat-8.0.26-windows-x64.zip
 - (make sure you have a Java environment, JAVA_HOME must be set)
 - in conf/server.xml add:

<Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol" secure="true" scheme="https"
maxThreads="150" URIEncoding="UTF-8" SSLVerifyClient="optional"
SSLPassword="testing" SSLEnabled="true"
SSLCACertificateFile="C:\CAs.crt" />

The certificate files are in the WAR file, extract them to C:\ (or elsewhere).

 - put the a.war file to the webapps folder
 - start tomcat by startup.bat
 - import the my3.p12 certificate to Firefox (the password is : test )
 - in Firefox load the page https://localhost:8443/a/a.jsp (when asked, select the certifcate imported from my3.p12)
 - refresh the page (F5) every few seconds

Actual result: about 30 seconds after the first load of page, the page will show a NPE instead of the certificate name

Expected: no NPE

Note: the issue usually happens in 30 seconds, but sometimes it goes on without error for longer. If so, stop and restart tomcat and try again.
Comment 2 David Balažic 2015-09-17 19:04:12 UTC
The problem also goes away temporarily if I restart Firefox or just clear the "Active Logins" in the history deleting dialog. Then after 30 seconds it happens again.
Maybe related to SSL session length?

Any suggestion how to debug this is welcome.
Comment 3 Christopher Schultz 2015-09-21 20:20:02 UTC
If you use Wireshark or a similar packet-capture rig, can you see
whether the browser is changing the way it sends its data?

With Wireshark, you can install the server's private key and then you
can read all the encrypted traffic. Wireshark will disassemble all the
packets and even give you rich information at the protocol-level about
what's in there. You can probably tell the difference between what
Firefox or Chrome sends to the server both before and after the "loss"
of the certificate.

I'd like to confirm whether the browser or Tomcat is causing this problem... my suspicion is that it's the browser.
Comment 4 Petr Brouzda 2015-10-29 12:45:06 UTC
We have similar (maybe the same) problem.

We run
- Tomcat 8.0.24 with APR.
- HTTPS APR connector with SSLVerifyClient="require".
- on Debian 6

Client is a legacy application with no HTTPS support. So it uses "stunnel" (https://www.stunnel.org) for http-to-https proxy.

1) At the first request from this client ... server application sees client's certificate in javax.servlet.request.X509Certificate correctly.

2) Second and any subsequent requests within the same stunnel connection ... server application didn't see client's certificate, javax.servlet.request.X509Certificate is null.

3) After stunnel daemon is restarted, the first request is proceed correctly (with certificate info in javax.servlet.request.X509Certificate) and subsequent requests has javax.servlet.request.X509Certificate = null.

The difference is that (based on stunnel's logfile) the first request creates a new SSL session, and subsequent requests reuses that session.
Maybe there is a problem within APR that client certificate is not available when SSL session is reused.

(Other clients than stunnel works without problem.)
Comment 5 Petr Brouzda 2015-10-30 09:36:11 UTC
Created attachment 33232 [details]
100 % repeatable test case

Better test-case for this bug.
Comment 6 Petr Brouzda 2015-10-30 09:56:38 UTC
I've created a 100 % repeatable testcase.

In attached file (https://bz.apache.org/bugzilla/attachment.cgi?id=33232) you can find:
1) configuration for Tomcat and a simple web application from David Balazic's test case
2) direct SSL test from curl - works OK everytime
3) SSL test via stunnel - first request works OK, any subsequent requests fails, because Tomcat provide no certificate information.

Tested on centos6 (x64) with Tomcat 8.0.28, APR based Apache Tomcat Native library 1.1.33 using APR version 1.3.9.

Steps to reproduce the problem:

A) Prerequisities
sudo yum install stunnel curl

B) Run Tomcat
1) Install Tomcat 8.0.28 (I've used the /home/user/tomcat-test/apache-tomcat-8.0.28/ directory).
2) Build and configure Native APR .
3) Copy contents of apache-tomcat-8.0.28/ directory from .tar.gz to your Tomcat directory.
4) Check paths to certificates in conf/server.xml.
5) Start Tomcat. It should listen on port 8443, SSL with required client certificate. You can find client certificate in client-curl/client.pem.
6) There is a simple app: which prints user's certificate information.

C) Test the connection from curl
1) There is test.sh in test-curl/
2) Make it executable and run it.
3) It should connect to 8443 and run the test app. test.out = result from app. test.log = SSL log.
4) test.out should be like:
 User client cert data:
 CN=TEST CLIENT, O=Internet Widgits Pty Ltd, ST=CZ, C=CZ
 Time: Fri Oct 30 12:39:21 CET 2015

D) Run the stunnel test
1) There is run-stunnel.sh in client-stunnel/ 
2) Check the paths in this file (if you saved it elseqwhere) and run is as root (sudo ./run-stunnel.sh )
3) Stunnel creates listening port 8442, which accepts plain HTTP and forwards it to HTTPS 8443 with client certificate. Stunnel will run on console. Keep it running.
4) Open another console and run test.sh in the same directory. It will send the plain http request to 8442; request will be enveloped to https by stunnel and forwarded to 8443. Result will be printed on console:

   [user@localhost client-stunnel]$ ./test.sh
   User client cert data:
   CN=TEST CLIENT, O=Internet Widgits Pty Ltd, ST=CZ, C=CZ
   Time: Fri Oct 30 13:08:08 CET 2015

5) Run it once more. It will fail - no certificate is returned by Tomcat's request.getAttribute("javax.servlet.request.X509Certificate").

 [user@localhost client-stunnel]$ ./test.sh
 <!DOCTYPE html><html><head><title>Apache Tomcat/8.0.28 - Erro...

Any subsequent requests from the same stunnel connection will reuse SSL session and will produce the same error.
If you stop and start stunnel again, the first request will be OK (certificate passed to application) and subsequent requests will fail again.
Comment 7 Rainer Jung 2015-11-08 16:18:24 UTC
Wy is there an expectation, that a client certificate is present when an ssl session is resumed? Isn't the client cert only needed during the initial handshake? I wouldn't expect it to be persistet on the server side and by available for followup requests as long as the ssl session is resumed and no new full handshake happens.
Comment 8 Petr Brouzda 2015-11-08 17:11:00 UTC
(In reply to Rainer Jung from comment #7)

Two reasons:

1) It makes client certificate UNUSABLE for authentication if client cert information are not present on subsequent HTTP requests. SSL session can be started OUTSIDE of my application - for example when there are more than one apps on the server. 
So client works with application A... and then he send request to application B. It is single SSL session for his browser, so application B won't receive client certificate info? Bad behaviour, I think.

2) This behaviour is specific for Tomcat with APR.
In any other environment I know (Tomcat with standard JSSE, IBM WebSphere) it works correctly - client certificate information are present for every request, regardless of SSL sessions.
So application that works correctly with Tomcat/JSSE and with IBM WebSphere will fail on Tomcat/APR.
Comment 9 David Balažic 2015-11-25 17:53:40 UTC
News update:

with newer software vserions, the behavior changed slightly.

Server side:
 - before: Java 1.8.0u51 , apache 8.0.24/26
 - now   : Java 1.8.0u60 , apache 8.0.28

 - before: Firefox (v39 and v40) and Chrome (v44)
 - now   : Firefox (v42.0) and Chrome (v46)

New behavior:
 - Java 1.8.0u60 + Apache 8.0.28 + FF 42.0 = problem appears only if page is left with no activity for 60 seconds (unlike before when the problem appeared after 30 seconds even if refreshing/reloading the page every few seconds)

 - Java 1.8.0u60 + Apache 8.0.28 + Chrome 46 : when refreshing page, not problem even after 60 secs, if page left inactive: problem occurs after 60 seconds

 - Java 1.8.0u60 + Apache 8.0.28 + IE11: after 13 minutes of inactivity the problem does not occur (did not test longer, but the problem never occurred with IE before either)

* Apache 8.0.28 = http://www.apache.si/tomcat/tomcat-8/v8.0.28/bin/apache-tomcat-8.0.28-windows-x64.zip

New clients with old apache 8.0.26:

 - Chrome 46: when refreshing page, not problem even after 60 secs
if page left inactive: problem occurs after 60 seconds

 - FF 42.0: when refreshing page, not problem even after 60 secs
if page left inactive: problem occurs after 60 seconds

So it seems the newer browsers changed behavior.

Additional note: in both Chrome and Firefox after a longer period of inactivity (like 5 minutes) if the page is refreshed, it will work correctly. Apparently in such cases everything is done from new, as if it were connecting for the first time.
Comment 10 David Balažic 2015-11-25 18:59:33 UTC
With the Apache 8.0.29 released just now it is the same as with 8.0.28
Comment 11 David Balažic 2016-02-12 17:11:44 UTC
Same with tomcat version 8.0.32 which bundles OpenSSL 1.0.2e (see below)

The issue remains (with the change that now IE can not connect at all,
it complains about some TLS stuff, did not look into it).

Version details (from tomcat startup log):
Loaded APR based Apache Tomcat Native library 1.2.4 using APR version 1.5.1.
OpenSSL successfully initialized (OpenSSL 1.0.2e 3 Dec 2015)
Comment 12 Christopher Schultz 2016-02-16 15:35:13 UTC
David, you still haven't said whether this is a case of the browser not sending the certificate or the servlet ignoring it when it's sent. Using Wireshark should allow you to do that.

If the browser does not send the certificate, the only way for Tomcat to deal with that would be to store the certificate (and chain?) somewhere along with a mapping to the TLS session identifier (which should ALWAYS be available).

If a TLS session re-start occurs, the client cert should be presented again, and there should be continuity, there. Please confirm also that, when the cert disappears, that your TLS and HTTP sessions are not interrupted.
Comment 13 Mark Thomas 2016-02-18 23:44:36 UTC
It appears to be related to session tickets. If you set disableSessionTickets="true" the problem goes away.

Some quick background reading indicates that the session ticket should include the client cert so my current theory is that something isn't handling the session resumption correctly. I'll take a look but I'm not at all familiar with the code so if someone who is wants to take a look as well ...
Comment 14 Mark Thomas 2016-02-19 00:49:12 UTC
OK. I think I have found the problem.

Tomcat looks for two pieces of information when looking up client certs.
From AprSSLSupport:
int certLength = SSLSocket.getInfoI(socketRef, SSL.SSL_INFO_CLIENT_CERT_CHAIN);
byte[] clientCert = SSLSocket.getInfoB(socketRef, SSL.SSL_INFO_CLIENT_CERT);

In OpenSSL those map to

The problem is that in d2i_SSL_SESSION when the session is repopulated from the ticket, peer is populated but peer_chain is not. i2d_SSL_SESSION doesn't save the peer certificate chain either.

RFC5077 appears to allow full certificate chains to be present in the ticket.

Some more digging has unearthed this from the OpenSSL issue tracker:

It looks like addressing this is not a high priority for OpenSSL.

We might be able to work-around this on the Tomcat side to expose the client cert minus the chain and document that as a known restriction when using session tickets.
Comment 15 Remy Maucherat 2016-02-19 09:14:33 UTC
Ok, that sounds hard. The new OpenSSL code should avoid the problem in most cases since the certificates are cached in the SSL engine.
Comment 16 Mark Thomas 2016-02-21 16:49:43 UTC
Created attachment 33578 [details]
Potential patch if OpenSSL decide this is a WONTFIX

Working around this in Tomcat is quite simple. It does mean the full chain is only available on the initial connection. Subsequent connections only get the user cert. That is probably sufficient for most use cases. Where that isn't sufficient, the app can always cache the chain in the session. Another option is for the CLIENT-CERT authenticator to cache the chain in the session.

I'm following this up with the OpenSSL folks. If it is indeed a WONTFIX then we can apply this patch or something along these lines.
Comment 17 Mark Thomas 2017-08-21 14:16:28 UTC
It looks like the OpenSSL behaviour isn't going to change so I've gone ahead and handled this in the Tomcat code.

Fixed in:
- trunk for 9.0.0.M27 onwards
- 8.5.x for 8.5.21 onwards
- 8.0.x for 8.0.47 onwards
- 7.0.x for 7.0.82 onwards