Created attachment 33041 [details]
Test case to reproduce issue

The issue persists with the new apache-tomcat-8.0.26-windows-x64.

I attach a WAR file that can be used to reproduce the issue. Either unpack the single JSP file in it and put it into the webapps folder under its own folder (like tomcat/webapps/x/a.jsp) or deploy the WAR file under webapps/.

The WAR file also contains the CA's, server and client certificates that can be used for the test.

Steps:
 - download and extract apache-tomcat-8.0.26-windows-x64.zip
 - (make sure you have a Java environment, JAVA_HOME must be set)
 - in conf/server.xml add:

<Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol" secure="true" scheme="https"
maxThreads="150" URIEncoding="UTF-8" SSLVerifyClient="optional"
SSLPassword="testing" SSLEnabled="true"
SSLCertificateKeyFile="C:\snakeoil-rsa.key"
SSLCertificateFile="C:\snakeoil-rsa.crt"
SSLCACertificateFile="C:\CAs.crt" />

The certificate files are in the WAR file, extract them to C:\ (or elsewhere).

 - put the a.war file to the webapps folder
 - start tomcat by startup.bat
 - import the my3.p12 certificate to Firefox (the password is : test )
 - in Firefox load the page https://localhost:8443/a/a.jsp (when asked, select the certifcate imported from my3.p12)
 - refresh the page (F5) every few seconds

Actual result: about 30 seconds after the first load of page, the page will show a NPE instead of the certificate name

Expected: no NPE

Note: the issue usually happens in 30 seconds, but sometimes it goes on without error for longer. If so, stop and restart tomcat and try again.

The problem also goes away temporarily if I restart Firefox or just clear the "Active Logins" in the history deleting dialog. Then after 30 seconds it happens again.
Maybe related to SSL session length?

Any suggestion how to debug this is welcome.

If you use Wireshark or a similar packet-capture rig, can you see
whether the browser is changing the way it sends its data?

With Wireshark, you can install the server's private key and then you
can read all the encrypted traffic. Wireshark will disassemble all the
packets and even give you rich information at the protocol-level about
what's in there. You can probably tell the difference between what
Firefox or Chrome sends to the server both before and after the "loss"
of the certificate.

I'd like to confirm whether the browser or Tomcat is causing this problem... my suspicion is that it's the browser.

We have similar (maybe the same) problem.

We run
- Tomcat 8.0.24 with APR.
- HTTPS APR connector with SSLVerifyClient="require".
- on Debian 6

Client is a legacy application with no HTTPS support. So it uses "stunnel" (https://www.stunnel.org) for http-to-https proxy.

1) At the first request from this client ... server application sees client's certificate in javax.servlet.request.X509Certificate correctly.

2) Second and any subsequent requests within the same stunnel connection ... server application didn't see client's certificate, javax.servlet.request.X509Certificate is null.

3) After stunnel daemon is restarted, the first request is proceed correctly (with certificate info in javax.servlet.request.X509Certificate) and subsequent requests has javax.servlet.request.X509Certificate = null.

The difference is that (based on stunnel's logfile) the first request creates a new SSL session, and subsequent requests reuses that session.
Maybe there is a problem within APR that client certificate is not available when SSL session is reused.

(Other clients than stunnel works without problem.)

Created attachment 33232 [details]
100 % repeatable test case

Better test-case for this bug.

I've created a 100 % repeatable testcase.

In attached file (https://bz.apache.org/bugzilla/attachment.cgi?id=33232) you can find:
1) configuration for Tomcat and a simple web application from David Balazic's test case
2) direct SSL test from curl - works OK everytime
3) SSL test via stunnel - first request works OK, any subsequent requests fails, because Tomcat provide no certificate information.

Tested on centos6 (x64) with Tomcat 8.0.28, APR based Apache Tomcat Native library 1.1.33 using APR version 1.3.9.

Steps to reproduce the problem:

A) Prerequisities
sudo yum install stunnel curl

B) Run Tomcat
1) Install Tomcat 8.0.28 (I've used the /home/user/tomcat-test/apache-tomcat-8.0.28/ directory).
2) Build and configure Native APR .
3) Copy contents of apache-tomcat-8.0.28/ directory from .tar.gz to your Tomcat directory.
4) Check paths to certificates in conf/server.xml.
5) Start Tomcat. It should listen on port 8443, SSL with required client certificate. You can find client certificate in client-curl/client.pem.
6) There is a simple app: https://127.0.0.1:8443/test/a.jsp which prints user's certificate information.

C) Test the connection from curl
1) There is test.sh in test-curl/
2) Make it executable and run it.
3) It should connect to 8443 and run the test app. test.out = result from app. test.log = SSL log.
4) test.out should be like:
 User client cert data:
 CN=TEST CLIENT, O=Internet Widgits Pty Ltd, ST=CZ, C=CZ
 <br>
 Time: Fri Oct 30 12:39:21 CET 2015

D) Run the stunnel test
1) There is run-stunnel.sh in client-stunnel/ 
2) Check the paths in this file (if you saved it elseqwhere) and run is as root (sudo ./run-stunnel.sh )
3) Stunnel creates listening port 8442, which accepts plain HTTP and forwards it to HTTPS 8443 with client certificate. Stunnel will run on console. Keep it running.
4) Open another console and run test.sh in the same directory. It will send the plain http request to 8442; request will be enveloped to https by stunnel and forwarded to 8443. Result will be printed on console:

   [user@localhost client-stunnel]$ ./test.sh
   User client cert data:
   CN=TEST CLIENT, O=Internet Widgits Pty Ltd, ST=CZ, C=CZ
   <br>
   Time: Fri Oct 30 13:08:08 CET 2015

5) Run it once more. It will fail - no certificate is returned by Tomcat's request.getAttribute("javax.servlet.request.X509Certificate").

 [user@localhost client-stunnel]$ ./test.sh
 <!DOCTYPE html><html><head><title>Apache Tomcat/8.0.28 - Erro...

Any subsequent requests from the same stunnel connection will reuse SSL session and will produce the same error.
If you stop and start stunnel again, the first request will be OK (certificate passed to application) and subsequent requests will fail again.

Wy is there an expectation, that a client certificate is present when an ssl session is resumed? Isn't the client cert only needed during the initial handshake? I wouldn't expect it to be persistet on the server side and by available for followup requests as long as the ssl session is resumed and no new full handshake happens.

(In reply to Rainer Jung from comment #7)

Two reasons:

1) It makes client certificate UNUSABLE for authentication if client cert information are not present on subsequent HTTP requests. SSL session can be started OUTSIDE of my application - for example when there are more than one apps on the server. 
So client works with application A... and then he send request to application B. It is single SSL session for his browser, so application B won't receive client certificate info? Bad behaviour, I think.

2) This behaviour is specific for Tomcat with APR.
In any other environment I know (Tomcat with standard JSSE, IBM WebSphere) it works correctly - client certificate information are present for every request, regardless of SSL sessions.
So application that works correctly with Tomcat/JSSE and with IBM WebSphere will fail on Tomcat/APR.

News update:

with newer software vserions, the behavior changed slightly.

Server side:
 - before: Java 1.8.0u51 , apache 8.0.24/26
 - now   : Java 1.8.0u60 , apache 8.0.28

Clients:
 - before: Firefox (v39 and v40) and Chrome (v44)
 - now   : Firefox (v42.0) and Chrome (v46)

New behavior:
 - Java 1.8.0u60 + Apache 8.0.28 + FF 42.0 = problem appears only if page is left with no activity for 60 seconds (unlike before when the problem appeared after 30 seconds even if refreshing/reloading the page every few seconds)

 - Java 1.8.0u60 + Apache 8.0.28 + Chrome 46 : when refreshing page, not problem even after 60 secs, if page left inactive: problem occurs after 60 seconds

 - Java 1.8.0u60 + Apache 8.0.28 + IE11: after 13 minutes of inactivity the problem does not occur (did not test longer, but the problem never occurred with IE before either)


* Apache 8.0.28 = http://www.apache.si/tomcat/tomcat-8/v8.0.28/bin/apache-tomcat-8.0.28-windows-x64.zip


New clients with old apache 8.0.26:

 - Chrome 46: when refreshing page, not problem even after 60 secs
if page left inactive: problem occurs after 60 seconds

 - FF 42.0: when refreshing page, not problem even after 60 secs
if page left inactive: problem occurs after 60 seconds


So it seems the newer browsers changed behavior.

Additional note: in both Chrome and Firefox after a longer period of inactivity (like 5 minutes) if the page is refreshed, it will work correctly. Apparently in such cases everything is done from new, as if it were connecting for the first time.

With the Apache 8.0.29 released just now it is the same as with 8.0.28

Same with tomcat version 8.0.32 which bundles OpenSSL 1.0.2e (see below)

The issue remains (with the change that now IE can not connect at all,
it complains about some TLS stuff, did not look into it).

Version details (from tomcat startup log):
Loaded APR based Apache Tomcat Native library 1.2.4 using APR version 1.5.1.
OpenSSL successfully initialized (OpenSSL 1.0.2e 3 Dec 2015)

David, you still haven't said whether this is a case of the browser not sending the certificate or the servlet ignoring it when it's sent. Using Wireshark should allow you to do that.

If the browser does not send the certificate, the only way for Tomcat to deal with that would be to store the certificate (and chain?) somewhere along with a mapping to the TLS session identifier (which should ALWAYS be available).

If a TLS session re-start occurs, the client cert should be presented again, and there should be continuity, there. Please confirm also that, when the cert disappears, that your TLS and HTTP sessions are not interrupted.

It appears to be related to session tickets. If you set disableSessionTickets="true" the problem goes away.

Some quick background reading indicates that the session ticket should include the client cert so my current theory is that something isn't handling the session resumption correctly. I'll take a look but I'm not at all familiar with the code so if someone who is wants to take a look as well ...

OK. I think I have found the problem.

Tomcat looks for two pieces of information when looking up client certs.
From AprSSLSupport:
int certLength = SSLSocket.getInfoI(socketRef, SSL.SSL_INFO_CLIENT_CERT_CHAIN);
byte[] clientCert = SSLSocket.getInfoB(socketRef, SSL.SSL_INFO_CLIENT_CERT);

In OpenSSL those map to
SSL_SESSION->peer_chain
SSL_SESSION->peer

The problem is that in d2i_SSL_SESSION when the session is repopulated from the ticket, peer is populated but peer_chain is not. i2d_SSL_SESSION doesn't save the peer certificate chain either.

RFC5077 appears to allow full certificate chains to be present in the ticket.

Some more digging has unearthed this from the OpenSSL issue tracker:
https://rt.openssl.org/Ticket/Display.html?id=2288

It looks like addressing this is not a high priority for OpenSSL.

We might be able to work-around this on the Tomcat side to expose the client cert minus the chain and document that as a known restriction when using session tickets.

Ok, that sounds hard. The new OpenSSL code should avoid the problem in most cases since the certificates are cached in the SSL engine.

Created attachment 33578 [details]
Potential patch if OpenSSL decide this is a WONTFIX

Working around this in Tomcat is quite simple. It does mean the full chain is only available on the initial connection. Subsequent connections only get the user cert. That is probably sufficient for most use cases. Where that isn't sufficient, the app can always cache the chain in the session. Another option is for the CLIENT-CERT authenticator to cache the chain in the session.

I'm following this up with the OpenSSL folks. If it is indeed a WONTFIX then we can apply this patch or something along these lines.

It looks like the OpenSSL behaviour isn't going to change so I've gone ahead and handled this in the Tomcat code.

Fixed in:
- trunk for 9.0.0.M27 onwards
- 8.5.x for 8.5.21 onwards
- 8.0.x for 8.0.47 onwards
- 7.0.x for 7.0.82 onwards