Created attachment 37317 [details] screenshot I have a JSR223 Assertion defined at the Thread group level. It is correctly executed for each HTTP Request sampler. When the assertion fails via AssertionResult.setFailure(true), it correctly add a red Assertion Result in the View Result Tree. Problem: the HTTP Request in the View Result Tree stays green. When looking at the View Result Tree, all tests seem OK (see screenshot). If the same assertion is added inside each HTTP Request, the View Result Tree correctly displays the HTTP Request in red (but it would require to duplicate the assertion to thousands of requests which is not manageable).
Created attachment 37320 [details] Test with one assertion outside of a sampler
Can you give a simple test plan, that exhibits the failure? My test plan works as intended and the sampler gets marked as failed.
Created attachment 37321 [details] Test with one assertion outside of a sampler + another one outside the Thread Group I found the exact: it's an interaction with another assertion. I attached your test modified: I added a Response Assertion at the global level. It seems the Response Assertion - that's executed after the Groovy one - ovewrites the Sampler result. Same result if I put both assertions inside the sampler: the sampler only "remembers" the last one. To recap: - assertion 1: fail - assertion 2: ok result: ok
I never had that problem in JMeter 4
I checked your modified test plan with JMeter 4.0 and it showed the same behaviour. Are you sure, that you tried exactly the same things with JMeter 4.0? The documentation for https://jmeter.apache.org/usermanual/component_reference.html#Response_Assertion and especially "ignore status" means to me, that this is not a bug. The documentation states, that the response assertion has to be the first assertion to work properly.
I didn't make the link with the "ignore status". I'm pretty sure I had the same test in JMeter 4 but I'm not 100% sure. You're right, it's documented, so it's a feature :-( I don't understand why the Response status is forced to successful instead of not changing the status in case we have an error code, but I imagine it's a technical limitation. Too bad! There's actually no way to add a global assertion checking, for instance, for invalid headers without having a huge side-effect on the results (either flagging requests correctly returning a non-200, or not flagging eral errors). I don't expect you to fix this or to continue the discussion, but I let you close the ticket. Thanks for your answer.
Can you give a test plan for your setup to show exactly what kind of global assertions you want to have? I am not sure, that I get your expectations on this correctly. You should be able to simulate the responses (with status and headers) using a JSR223 Sampler or an Java Request Sampler. While this might not be a "bug", it could be an enhancement :)
This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/5338