Created attachment 38381 [details] logfile with debug When doing https request, the first request will be always much slower. For example testing jmeter.apache.org 4 times in sequence gives response times: 1234, 110,110,130 ms When using debug I see (removed lines in between): 2022-08-31 08:48:30,462 DEBUG o.a.j.p.h.s.HTTPHC4Impl: Start : sample https://jmeter.apache.org/ method GET followingRedirect false depth 0 2022-08-31 08:48:30,634 DEBUG o.a.j.u.JsseSSLManager: Creating threadLocal SSL context for: thread1 1-1 2022-08-31 08:48:31,312 DEBUG o.a.j.u.JsseSSLManager: Using threadLocal SSL context for: thread1 1-1 2022-08-31 08:48:31,347 DEBUG o.a.h.c.s.SSLConnectionSocketFactory: Starting handshake So creating the SSL context is taking 678ms. This is included in the response time of the first request. For the next (same) request is it using the already created SSL context. So the response time of this request is much faster. I have doubts if this "Creating threadLocal SSL context" should be part of the response time measured because it is not caused by the tested server/service. For load testing maybe not a big issue (because a lot of requests). But we use jmeter also for monitoring. So we are staring jmeter, repeating 1 request 1-3 times and stop jmeter.
Created attachment 38382 [details] jmx
I'am using java version "1.8.0_341"
Did some more testing: With jmeter 5.5 and java version 1.8.0_341 the problem is the same. When using jmeter 5.5 and openjdk 18 the problem is still there but the difference between the first en second request is lower (500ms).
The SSL context is not shared across threads, so technically speaking it should be included to "connect time" for properly attributing SSL overheads
Maybe, but it is not included in the next requests in the same thread. Even if keepalive is turned off. So for the next connect it is not included but making a new SLL connection. That is not consequent I think. It is used to setup the thread to prepare making a SSL connection one time. But when using jmeter (non-gui) for just 1 or 2 request it is very disturbing. Of course I want to include the ssl handshake every time but not the onetime setup of the client.
Well, historically, JMeter thread corresponds to a "user session", so per-thread caching is probably not that far from what typical applications would have. For instance, if you fully close web browser, and then open it again, I guess the very first request would be slower than the second one. I'm not sure how good JMeter emulates "opening a fresh browser window" when you "start test". Just in case: 1) I do not think removing "SSL cache" is an option, since it is expected that good apps would use SSL session caching. 2) I do not think invalidating SSL cache on each iteration is good (same reasons as in 1) 3) I do not think caching SSL forever and moving the initialization out of the measurement loop is good either: if real app executes just one request, they would face the same SSL startup cost. Frankly speaking, I have no clear answer how to approach all this, however, I could imagine: a) Adding an option like "poorman's warmup" where JMeter discards a couple of first iterations if there's more than one measurement. For instance, if there are two or three measurements, then JMeter could exclude the first (or two) from the report stats b) Adding something like "exclude first N seconds" or "exclude first N%" when computing the stats. This should let JMeter and the system under test to warm up a bit. However, in my experience, there were cases when 30min warmup load was needed. c) Adding something like "session management" where different threads could use the same session. For instance, like in a real application, there might be a shared "database connection pool", and app threads use the connections from the pool. The same might be workable for JMeter as well: there might be "HTTPS connection pool to the system under test", and JMeter threads could use it to send requests like the typical app would do using HttpClients. WDYT?
When this is for testing for services used by a browser I agree (mostly). I'm testing services which are use from another service. So no browser involved. The calling service is using connection pooling, so normally everything is already setup. So I'm only interested in the server response times (incl ssl handshake). Not in the client setup (before ssl handshake). The time for the setup will also be specific for which client is used. When using Jmeter with openjdk 18 of Java 1.8 I see already a big difference in setup time. So it will never be representative if there are different clients. Also for browsers. So better to leave it out :-) Doing a lot of requests, this will not be a big problem. With load testing you will gradually increase the number of parallel requests and wait until this is stable. You will use the response times in the stable period. I don't this can be handled by jmeter, I'm more a Loadrunner guy :-). So b can be useful for everyone. Also a could work. C would only work if the setup will be excluded from the first request response time. Making it more specific: I have 2 webservers with a loadbalancer in front of it. I do 1 request on the LB, 1 request on server 1 and 1 on server 2. This are are different "urls". So I don't repeat a request. Now 1 of the 3 shows a big difference in response time. So it is 1 iteration with 3 different requests. So then a won't work I think. Except when making an iteration of 2 for the first destination. Maybe just a checkbox on the request with the choice the exclude it from the stats (including jtl). Then I have to add a extra request as sort of dummy request.
This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/5698