Bug 62861 - Thread Group: Provide ability to configure whether a new iteration is a new user or same user (Would be applied on Cookie Manager, Cache Manager and httpclient.reset_state_on_thread_group_iteration)
Summary: Thread Group: Provide ability to configure whether a new iteration is a new u...
Status: RESOLVED FIXED
Alias: None
Product: JMeter
Classification: Unclassified
Component: HTTP (show other bugs)
Version: 5.0
Hardware: All All
: P1 enhancement (vote)
Target Milestone: JMETER_5.2
Assignee: JMeter issues mailing list
URL:
Keywords: FixedInTrunk, PatchAvailable
: 63535 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-10-29 10:12 UTC by Abhishek Rao
Modified: 2019-10-04 20:18 UTC (History)
2 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Abhishek Rao 2018-10-29 10:12:19 UTC
Hi,

I have a script with 5 http requests which upon executing for 10 iterations with keep-alive enabled for all requests, and following are my observations:
1. httpclient.reset_state_on_thread_group_iteration=true
a. With 1000ms think time per iteration
Expected Output: Connection per iteration
Actual Output: A total of 10 connections are created, and each connection was iteration dependent.
b. No think time throughout the iteration
Expected Output: Connection per iteration
Actual Output: A total of 10 connections are created, and each connection was iteration dependent.

2. httpclient.reset_state_on_thread_group_iteration=false
a. With 1000ms think time per iteration
Expected Output: Connection per thread, shared across multiple iterations
Actual Output: Network inspection shows that 5 connections were created in total, with an interval of ~2seconds before the client (JMeter) sent a FIN-ACK and closed the connection, which isn't the actual "full" reuse that we're looking for with this feature. 
b. No think time throughout the iteration
Expected Output: Connection per thread, shared across multiple iterations
Actual Output: Only one connection was created and used across all the iterations.

It seems though some underlying TCP configuration is being honoured, which is causing the client to create a new connection every 2 seconds after ~1+ seconds of inactivity due to our think time in the script. Is there a way to work around this and ensure full reuse of connection? I'm not sure whether this is a JMeter bug in the first place.
Comment 1 Philippe Mouawad 2018-10-29 10:25:06 UTC
(In reply to Abhishek Rao from comment #0)
> Hi,
> 
> I have a script with 5 http requests which upon executing for 10 iterations
> with keep-alive enabled for all requests, and following are my observations:
> 1. httpclient.reset_state_on_thread_group_iteration=true
> a. With 1000ms think time per iteration
> Expected Output: Connection per iteration
> Actual Output: A total of 10 connections are created, and each connection
> was iteration dependent.
> b. No think time throughout the iteration
> Expected Output: Connection per iteration
> Actual Output: A total of 10 connections are created, and each connection
> was iteration dependent.

This one looks ok, right ?

> 
> 2. httpclient.reset_state_on_thread_group_iteration=false
> a. With 1000ms think time per iteration
> Expected Output: Connection per thread, shared across multiple iterations
> Actual Output: Network inspection shows that 5 connections were created in
> total, with an interval of ~2seconds before the client (JMeter) sent a
> FIN-ACK and closed the connection, which isn't the actual "full" reuse that
> we're looking for with this feature. 
> b. No think time throughout the iteration
> Expected Output: Connection per thread, shared across multiple iterations
> Actual Output: Only one connection was created and used across all the
> iterations.
> 
> It seems though some underlying TCP configuration is being honoured, which
> is causing the client to create a new connection every 2 seconds after ~1+
> seconds of inactivity due to our think time in the script. Is there a way to
> work around this and ensure full reuse of connection? I'm not sure whether
> this is a JMeter bug in the first place.

Have a look at these 3 properties in http://jmeter.apache.org/usermanual/properties_reference.html:
httpclient4.idletimeout=0
httpclient4.validate_after_inactivity=1700
httpclient4.time_to_live=2000
Comment 2 Abhishek Rao 2018-10-29 11:26:57 UTC
Hi Philippe,

The first one is okay. 

The configuration you mentioned works, but how do I achieve pooling and indefinite reuse rather than reusing for an arbitrary time period? For example: I have scripts acting as clients using pooled connection architectures to talk to web-services internally where this feature will be really useful in simulating. Can this be a UI modifiable in future and parametric?

Also, how do I achieve hybrid simulations with one thread group using httpclient.reset_state_on_thread_group_iteration=true and one that needs it set to false?
Comment 3 Philippe Mouawad 2018-11-01 14:40:08 UTC
(In reply to Abhishek Rao from comment #2)
> Hi Philippe,
> 
> The first one is okay. 
> 
> The configuration you mentioned works, but how do I achieve pooling and
> indefinite reuse rather than reusing for an arbitrary time period? For
> example: I have scripts acting as clients using pooled connection
> architectures to talk to web-services internally where this feature will be
> really useful in simulating. 

What if you set:
httpclient4.idletimeout=0
httpclient4.validate_after_inactivity=1700
httpclient4.time_to_live=Very high value


> Can this be a UI modifiable in future and
> parametric?
> Also, how do I achieve hybrid simulations with one thread group using
> httpclient.reset_state_on_thread_group_iteration=true and one that needs it
> set to false?

This would be an enhancement request.
Workaround is to use distributed testing with 1 JVM in first mode and second JVM in second mode.
Comment 4 Abhishek Rao 2018-11-01 15:05:36 UTC
(In reply to Philippe Mouawad from comment #3)
> (In reply to Abhishek Rao from comment #2)
> > Hi Philippe,
> > 
> > The first one is okay. 
> > 
> > The configuration you mentioned works, but how do I achieve pooling and
> > indefinite reuse rather than reusing for an arbitrary time period? For
> > example: I have scripts acting as clients using pooled connection
> > architectures to talk to web-services internally where this feature will be
> > really useful in simulating. 
> 
> What if you set:
> httpclient4.idletimeout=0
> httpclient4.validate_after_inactivity=1700
> httpclient4.time_to_live=Very high value
> 
> 
> > Can this be a UI modifiable in future and
> > parametric?
> > Also, how do I achieve hybrid simulations with one thread group using
> > httpclient.reset_state_on_thread_group_iteration=true and one that needs it
> > set to false?
> 
> This would be an enhancement request.
> Workaround is to use distributed testing with 1 JVM in first mode and second
> JVM in second mode.


I dont think the workaround would apply for my scenario, because there's two thread groups in the same script and each requires a different setting. Distributed mode with different settings across JVMs would mean both the thread groups are affected, when the scenario calls for just one thread group to have the setting reflected. I think I'll split it into two tests for now and run them in parallel. I think the aforementioned will make for a great enhancement, I also found out that HC4 supports PooledConnectionManager which solves the problem.
Comment 5 Philippe Mouawad 2018-11-01 15:10:21 UTC
(In reply to Abhishek Rao from comment #4)
> (In reply to Philippe Mouawad from comment #3)
> > (In reply to Abhishek Rao from comment #2)
> > > Hi Philippe,
> > > 
> > > The first one is okay. 
> > > 
> > > The configuration you mentioned works, but how do I achieve pooling and
> > > indefinite reuse rather than reusing for an arbitrary time period? For
> > > example: I have scripts acting as clients using pooled connection
> > > architectures to talk to web-services internally where this feature will be
> > > really useful in simulating. 
> > 
> > What if you set:
> > httpclient4.idletimeout=0
> > httpclient4.validate_after_inactivity=1700
> > httpclient4.time_to_live=Very high value
> > 
> > 
> > > Can this be a UI modifiable in future and
> > > parametric?
> > > Also, how do I achieve hybrid simulations with one thread group using
> > > httpclient.reset_state_on_thread_group_iteration=true and one that needs it
> > > set to false?
> > 
> > This would be an enhancement request.
> > Workaround is to use distributed testing with 1 JVM in first mode and second
> > JVM in second mode.
> 
> 
> I dont think the workaround would apply for my scenario, because there's two
> thread groups in the same script and each requires a different setting.
> Distributed mode with different settings across JVMs would mean both the
> thread groups are affected, when the scenario calls for just one thread
> group to have the setting reflected. I think I'll split it into two tests
> for now and run them in parallel. I think the aforementioned will make for a
> great enhancement, I also found out that HC4 supports
> PooledConnectionManager which solves the problem.

What about my first note ?

Regarding workaround, just do the following:

- use for  threads field of each Thread Group  : 
   ${__P(threads_TG1,)}
   ${__P(threads_TG2,)}

Then run first jmeter-server with:

-Jthreads_TG1=X -Jthreads_TG2=0

And second with:

-Jthreads_TG1=0 -Jthreads_TG2=Y

I think it will work
Comment 6 Philippe Mouawad 2018-11-06 20:33:54 UTC
Hello,
Any feedback on answer ?

Thanks
Comment 7 Abhishek Rao 2018-11-07 07:11:35 UTC
(In reply to Philippe Mouawad from comment #6)
> Hello,
> Any feedback on answer ?
> 
> Thanks

Hi Philippe

Thanks for the work around. It's doing what we expected. :)

Abhishek
Comment 8 Philippe Mouawad 2018-11-07 07:29:21 UTC
Great, can you please answer my other questions so that I know if I can requalify bug to enhancement ?

Thanks
Comment 9 Philippe Mouawad 2018-11-10 11:15:49 UTC
Requalifying to enhancement.

Note there is a relation with this topic:
http://mail-archives.apache.org/mod_mbox/jmeter-dev/201811.mbox/%3CCAH9fUpau-3YWao86yU2DhM6mCAwssa61Wqew_iMYtFbmrioRiQ%40mail.gmail.com%3E


@Reporter, it would be really nice from you (since I spent time answering and providing workaround ) to answer comment 3 and confirm provided information fixed issue 2 and it is now an enhancement request and not a bug.
Comment 10 Abhishek Rao 2018-11-14 01:30:35 UTC
(In reply to Philippe Mouawad from comment #9)
> Requalifying to enhancement.
> 
> Note there is a relation with this topic:
> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201811.mbox/%3CCAH9fUpau-
> 3YWao86yU2DhM6mCAwssa61Wqew_iMYtFbmrioRiQ%40mail.gmail.com%3E
> 
> 
> @Reporter, it would be really nice from you (since I spent time answering
> and providing workaround ) to answer comment 3 and confirm provided
> information fixed issue 2 and it is now an enhancement request and not a bug.

Thought I confirmed in comment#7. Agree that this isn't necessarily a bug and just a behavior of setting TTL at HC level. It looks like we can go with the workaround for now:
a. Set TTL to desired value (long enough that connection persists throughout test) on specific jmeter-servers and leave to default for the rest
b. Configure run-time thread group enablement as mentioned in comment#5

Thanks for qualifying this as an enhancement. 

Quick question - will the enhancement be a PoolingHttpClientConnectionManager or httpclient.reset_state_on_thread_group_iteration just made threadgroup local and UI configurable?
Comment 11 Philippe Mouawad 2019-07-02 20:07:09 UTC
PR available:

https://github.com/apache/jmeter/pull/470
Comment 12 Philippe Mouawad 2019-07-02 20:11:10 UTC
This is an automated email from the ASF dual-hosted git repository.

pmouawad pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/jmeter.git


The following commit(s) were added to refs/heads/master by this push:
     new 629c643  This implements "Provide ability to configure whether a new iteration is a new user or same user" (BUG 62861) (#470)
629c643 is described below

commit 629c6430e17df7f7f75891d29e2129ab55a57d68
Author: Philippe M <pmouawad@users.noreply.github.com>
AuthorDate: Tue Jul 2 22:10:02 2019 +0200

    This implements "Provide ability to configure whether a new iteration is a new user or same user" (BUG 62861) (#470)
Comment 13 Philippe Mouawad 2019-07-07 13:00:24 UTC
*** Bug 63535 has been marked as a duplicate of this bug. ***
Comment 14 Vladimir Sitnikov 2019-10-04 09:29:07 UTC
@Philippe, was there a reason to implement "same user" vs "different user" as radio box?

It looks like a checkbox with label "Same user on each iteration" would be more consistent with other checkboxes.

WDYT?
Comment 15 UbikLoadPack support 2019-10-04 10:55:06 UTC
(In reply to Vladimir Sitnikov from comment #14)
> @Philippe, was there a reason to implement "same user" vs "different user"
> as radio box?
> 
> It looks like a checkbox with label "Same user on each iteration" would be
> more consistent with other checkboxes.
> 
> WDYT?

We followed recommendations in UX:

https://www.nngroup.com/articles/checkboxes-vs-radio-buttons/

If you use a checkbox, are you sure users will be aware of what is done when "Same user on each iteration" is unchecked ?

Maybe it would be better to switch other controls to Radio buttons.
Comment 16 Vladimir Sitnikov 2019-10-04 12:12:47 UTC
> We followed recommendations in UX:
> https://www.nngroup.com/articles/checkboxes-vs-radio-buttons/

Can you please clarify where is the recommendation to replace **single** checkbox with radio buttons?


Just in case, the article you mention starts with "Stay informed" sample with **two checkboxes**.

Then the very same article says that

> Ironically, with a single question, using a checkbox would have been correct

In other words, they recommend **single question using a checkbox**.


> are you sure users will be aware of
>what is done when "Same user on each iteration" is unchecked ?

1) If "same user" is checked, then the user is the same. If unchecked, then, "appareltly", the user is not the same.

2) Neither checkbox nor radio answer how "user" relates to a thread. For instance, what if I use TCP sampler. What "different user on each iteration" would mean? Nothing.

Frankly speaking, I'm not even sure the checkbox/radio should even be there, however it is already there, so we should at least try to reduce the damage.
Comment 17 Vladimir Sitnikov 2019-10-04 12:24:23 UTC
I've realized "old" test plans are opened as "different user on each iteration".

Was it really expected?
By default JMeter creates "same user on each iteration".
However, if I take an old plan and open it with new version, then it treats the plan as "different user" which might impact users that don't even know of the feature.
Comment 18 Philippe Mouawad 2019-10-04 20:03:12 UTC
This is an automated email from the ASF dual-hosted git repository.

pmouawad pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/jmeter.git


The following commit(s) were added to refs/heads/master by this push:
     new 538f8a2  Enhancement 62861: Fix old test plans opened as "different user on each iteration" while it should be "same user on each iteration"
538f8a2 is described below

commit 538f8a21ba1be380d35b5c47c137a54c099d7309
Author: pmouawad <p.mouawad@ubik-ingenierie.com>
AuthorDate: Fri Oct 4 22:02:03 2019 +0200

    Enhancement 62861: Fix old test plans opened as "different user on each
    iteration" while it should be "same user on each iteration"

    Also update release notes
    https://bz.apache.org/bugzilla/show_bug.cgi?id=62861
---
 .../java/org/apache/jmeter/threads/gui/ThreadGroupGui.java     | 10 ++++++----
 xdocs/changes.xml                                              |  1 +
 2 files changed, 7 insertions(+), 4 deletions(-)
Comment 19 Philippe Mouawad 2019-10-04 20:03:57 UTC
(In reply to Vladimir Sitnikov from comment #17)
> I've realized "old" test plans are opened as "different user on each
> iteration".
> 
> Was it really expected?
> By default JMeter creates "same user on each iteration".
> However, if I take an old plan and open it with new version, then it treats
> the plan as "different user" which might impact users that don't even know
> of the feature.

Fixed, thanks for finding it
Comment 20 Vladimir Sitnikov 2019-10-04 20:07:16 UTC
Did you just intentionally created a merge conflict with https://github.com/apache/jmeter/pull/513 ?

It is EXTREMELY sad that you create those merge conflicts for no reason.
Comment 21 Vladimir Sitnikov 2019-10-04 20:10:10 UTC
On top of that I guess your fix is incomplete.

It does not fix "read jmx" part of the thing, so "UI is good, but actual test would go in a different way".

https://github.com/apache/jmeter/pull/513 has the proper fix, so why hassle?
Comment 22 Philippe Mouawad 2019-10-04 20:14:09 UTC
(In reply to Vladimir Sitnikov from comment #20)
> Did you just intentionally created a merge conflict with
> https://github.com/apache/jmeter/pull/513 ?
> 
> It is EXTREMELY sad that you create those merge conflicts for no reason.

Vladimir, ABSOLUTELY NOT.
I fixed a bug you reported, for release preparation of tomorrow.

How can you think that ? 

We're working together in what I thought a gentle way in an OSS Team, it's not a war of EGO here...
Comment 23 Philippe Mouawad 2019-10-04 20:15:53 UTC
(In reply to Vladimir Sitnikov from comment #21)
> On top of that I guess your fix is incomplete.
> 
> It does not fix "read jmx" part of the thing, so "UI is good, but actual
> test would go in a different way".
> 
> https://github.com/apache/jmeter/pull/513 has the proper fix, so why hassle?

I tested:
- reading old JMX
- creating new one , saving , reloading

I didn't find an issue, I may have missed something as I am stupid, feel free to fix it
Comment 24 Vladimir Sitnikov 2019-10-04 20:18:56 UTC
@Philippe, would you please be so kind to add a unit test that would ensure old test plans are read appropriately?