Bug 64539 - View Result Tree: incorrect display of Assertion Result when defined globally
Summary: View Result Tree: incorrect display of Assertion Result when defined globally
Status: NEEDINFO
Alias: None
Product: JMeter
Classification: Unclassified
Component: HTTP (show other bugs)
Version: 5.3
Hardware: PC All
: P2 enhancement (vote)
Target Milestone: ---
Assignee: JMeter issues mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-19 12:19 UTC by Marc Stern
Modified: 2020-07-25 13:01 UTC (History)
1 user (show)



Attachments
screenshot (30.11 KB, image/png)
2020-06-19 12:19 UTC, Marc Stern
Details
Test with one assertion outside of a sampler (4.37 KB, application/xml)
2020-06-19 16:01 UTC, Felix Schumacher
Details
Test with one assertion outside of a sampler + another one outside the Thread Group (4.96 KB, application/xml)
2020-06-22 09:07 UTC, Marc Stern
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Marc Stern 2020-06-19 12:19:48 UTC
Created attachment 37317 [details]
screenshot

I have a JSR223 Assertion defined at the Thread group level.
It is correctly executed for each HTTP Request sampler.
When the assertion fails via AssertionResult.setFailure(true), it correctly add a red Assertion Result in the View Result Tree.
Problem: the HTTP Request in the View Result Tree stays green. When looking at the View Result Tree, all tests seem OK (see screenshot).

If the same assertion is added inside each HTTP Request, the View Result Tree correctly displays the HTTP Request in red (but it would require to duplicate the assertion to thousands of requests which is not manageable).
Comment 1 Felix Schumacher 2020-06-19 16:01:19 UTC
Created attachment 37320 [details]
Test with one assertion outside of a sampler
Comment 2 Felix Schumacher 2020-06-19 16:02:22 UTC
Can you give a simple test plan, that exhibits the failure? My test plan works as intended and the sampler gets marked as failed.
Comment 3 Marc Stern 2020-06-22 09:07:36 UTC
Created attachment 37321 [details]
Test with one assertion outside of a sampler + another one outside the Thread Group

I found the exact: it's an interaction with another assertion.
I attached your test modified: I added a Response Assertion at the global level.
It seems the Response Assertion - that's executed after the Groovy one - ovewrites the Sampler result.
Same result if I put both assertions inside the sampler: the sampler only "remembers" the last one.

To recap:
 - assertion 1: fail
 - assertion 2: ok
 result: ok
Comment 4 Marc Stern 2020-06-22 09:09:07 UTC
I never had that problem in JMeter 4
Comment 5 Felix Schumacher 2020-06-22 15:06:10 UTC
I checked your modified test plan with JMeter 4.0 and it showed the same behaviour. Are you sure, that you tried exactly the same things with JMeter 4.0?

The documentation for https://jmeter.apache.org/usermanual/component_reference.html#Response_Assertion and especially "ignore status" means to me, that this is not a bug.

The documentation states, that the response assertion has to be the first assertion to work properly.
Comment 6 Marc Stern 2020-06-23 10:43:28 UTC
I didn't make the link with the "ignore status".
I'm pretty sure I had the same test in JMeter 4 but I'm not 100% sure.

You're right, it's documented, so it's a feature :-(
I don't understand why the Response status is forced to successful instead of not changing the status in case we have an error code, but I imagine it's a technical limitation. Too bad!

There's actually no way to add a global assertion checking, for instance, for invalid headers without having a huge side-effect on the results (either flagging requests correctly returning a non-200, or not flagging eral errors).
I don't expect you to fix this or to continue the discussion, but I let you close the ticket.

Thanks for your answer.
Comment 7 Felix Schumacher 2020-06-24 14:33:53 UTC
Can you give a test plan for your setup to show exactly what kind of global assertions you want to have? I am not sure, that I get your expectations on this correctly.

You should be able to simulate the responses (with status and headers) using a JSR223 Sampler or an Java Request Sampler.

While this might not be a "bug", it could be an enhancement :)