Bug 65399 - Big difference in performance between non-gui and gui executions
Summary: Big difference in performance between non-gui and gui executions
Status: NEEDINFO
Alias: None
Product: JMeter
Classification: Unclassified
Component: Main (show other bugs)
Version: 5.4.1
Hardware: PC All
: P2 normal (vote)
Target Milestone: JMETER_5.5
Assignee: JMeter issues mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-06-24 18:35 UTC by Valentina
Modified: 2021-07-08 16:57 UTC (History)
0 users



Attachments
Test plan (4.79 KB, application/xml)
2021-06-29 15:55 UTC, Valentina
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Valentina 2021-06-24 18:35:52 UTC
Hi,
I've been comparing results of different load test executions and i've been getting a big difference in performance.
Executing in command line gives a longer response time (about 100% or 200% more).
The thread group i use is "Concurrency Thread Group". I run the test with 100 users and hold it for 10 seconds.
I've tested public apis (dog api, cat api, etc), i get this results with https://random.dog too. I've tried httpbin (https://httpbin.org/) and i get a similar difference.
I don't know if i'm doing something wrong. Any help or comment please?
Thanks in advance.
Comment 1 Felix Schumacher 2021-06-25 15:23:48 UTC
Can you give a minimal test plan, that shows the problem?

How are you comparing the response times?

Can you reproduce the behaviour without the third party plugin?
Comment 2 Valentina 2021-06-29 15:55:52 UTC
Created attachment 37921 [details]
Test plan

This is a simple test plan.
I've compared the response times using the "Aggregate report", using the "Report Dashboard" and generating .jtl files and then calculating percentiles by my own.
I can reproduce it without the plugin, the attachment file is a simple example.
Comment 3 Felix Schumacher 2021-06-29 19:22:52 UTC
Thanks for the test plan.

In my short tests, the first two times the GUI tests run faster, but the percentage of faster was nowhere near 100 or even 200 %. It was more 1 or 2 %. (I have done two tries, which is not really statistically relevant and a last test done after me writing this showed the non-GUI run faster 106 TX/s vs. 99 TX/s in the GUI)

I even expanded the test duration (I used a duration of 100 seconds) and the GUI is (in that one run I did a bit) slower than running it without the GUI. (I have set up a local test server in order to avoid DOSing some random service)

The results (from dashboard) were:
GUI: 1204597 Requests total, 12051.15 TX/s, 97 max, 7 Median, 12 95%, 22 99%
Non-GUI: 1235490 Requests total, 12358.61 TX/s, 83 max, 7 Median, 12 95%, 24 99% 

The reason for the (possibly true) advantage of the GUI for short test duration could be, that a JVM needs a bit to warm up and that we give it the time when loading a test plan and trying to find the right button to start it. In the non-GUI mode, that warm up phase is missing and it might hurt the performance of short tests (a wild guess).

If you are wondering about differences between the stats when viewed in the dashboard, the aggregate report or excel, you might like to have a look at bug 65353.
Comment 4 Valentina 2021-07-08 16:57:51 UTC
Thank you so much for your answer. Then i think it might be the JVM because i keep getting the difference.