Bug 65794 - JSON Assertion always successful with filter operators
Summary: JSON Assertion always successful with filter operators
Alias: None
Product: JMeter - Now in Github
Classification: Unclassified
Component: Main (show other bugs)
Version: 5.4.3
Hardware: PC All
: P2 major (vote)
Target Milestone: JMETER_5.5
Assignee: JMeter issues mailing list
Keywords: FixedInTrunk, PatchAvailable
Depends on:
Reported: 2022-01-09 19:45 UTC by Vladimir Rosu
Modified: 2022-01-30 11:29 UTC (History)
0 users

Break on indefinite paths, when an empty array is extracted and no assertion value given (1.84 KB, patch)
2022-01-17 19:37 UTC, Felix Schumacher
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Vladimir Rosu 2022-01-09 19:45:48 UTC
I'm testing an HTTP endpoint which returns the following content:
    "isCompressed": true,
    "rows": [
            "_2": "",
            "_4": "CustomString_1401",
            "_5": {
                "latitude": 0.0,
                "longitude": 0.0,
                "elevation": 0.0
            "_6": "CustomString_1401",
            "_8": "User0",
            "_9": "User",
            "_10": [],
            "_11": 1641561103991
            "_2": "",
            "_4": "CustomString_3930",
            "_5": {
                "latitude": 0.0,
                "longitude": 0.0,
                "elevation": 0.0
            "_6": "CustomString_3930",
            "_8": "User0",
            "_9": "User",
            "_10": [],
            "_11": 1641561045657
The following JSON assertion paths always return successful, even if those paths do not exist.
I believe this is a bug, as in case of assertion 1, there's no element which satisfies the filter, and in case of Filter 2, there's no key whose name is "_122".
Comment 1 Felix Schumacher 2022-01-15 17:06:00 UTC
I think you are stumbling upon an implementation detail of the jsonpath library we are using. According to https://github.com/json-path/JsonPath#what-is-returned-when the library will always return a list, if an expression '?(<expression>)' is used. JMeter will interpret this as an empty (but valid) result.

We might be able to tweak the parser using the options given at  https://github.com/json-path/JsonPath#tweaking-configuration
Currently we are using the default configuration. (./src/components/src/main/java/org/apache/jmeter/assertions/JSONPathAssertion.java line 112)

You might get better results (without patching JMeter), if you were asserting a value with the used assertion.
Comment 2 Vladimir Rosu 2022-01-17 08:23:48 UTC
I confirm that the asserting value approach work, but in my situation I did not want and need to generate the additional load of asserting the value.

After browsing to those links, I observed that a path, in JSON Path terminology, can be indefinite in multiple situations, not only when using Filter Operators.
Hence this situation might be triggered by other factors.

Considering the strange behavior I would like to add some helper text in the JSON assertion Help Center, here: https://jmeter.apache.org/usermanual/component_reference.html#JSON_Assertion

Is this solution acceptable? If yes, should I provide the text?
Comment 3 Felix Schumacher 2022-01-17 19:37:17 UTC
Created attachment 38165 [details]
Break on indefinite paths, when an empty array is extracted and no assertion value given

It would be great, if you would try to formulate the help text in a more correct way.

Attached is a patch, that would make the assertion fail, if
 * the given JSON Path is *indefinite*
 * and an empty array is extracted
 * and no assertion value has been given

I think it should fit with the current description better than silently ignoring this case.

What do you think?
Comment 4 Vladimir Rosu 2022-01-18 16:18:49 UTC
I agree that with this patch the user would get the required feedback. 
Two questions here.
1. Should I test a night build before releasing into mainstream? If so, do I need to build it?
2. I would still add this note in the Help Center:
"When using indefinite JSON Paths you must assert the value due to the existing JSON library implementation, otherwise the assertion could always return successful"
Comment 5 Felix Schumacher 2022-01-18 16:27:29 UTC
The patch has not been applied yet. If you want to test it, you would have to apply it by yourself and compile a version on your own. (If you feel, that you are not up to the task, I could provide a compiled one)

If no other dev speaks against applying this patch, I will commit it.

The docs should be updated in any case.
Comment 6 Felix Schumacher 2022-01-18 16:38:51 UTC
The documentation note has been added.

commit 6a5cac0ad75e72d5313f3c795845c6cea9ff6520
AuthorDate: Tue Jan 18 17:36:49 2022 +0100

    JSON Assertion always successful with indefinite paths
    Add a note to the component reference for JSON Path Assertions
    Bugzilla Id: 65794
 xdocs/changes.xml                        | 2 ++
 xdocs/usermanual/component_reference.xml | 3 +++
 2 files changed, 5 insertions(+)
Comment 7 Vladimir Rosu 2022-01-18 17:03:26 UTC
To save time, considering the patch is very low complexity, I do not need to text. Absolutely ok with the approach.
Comment 8 Vladimir Rosu 2022-01-19 16:17:40 UTC
I changed to New, because I'm not sure if I need to provide any other info.
Comment 9 Felix Schumacher 2022-01-19 19:17:05 UTC
@Vladimir, it would be great, if you could test next nightly or build from trunk and report back, whether this fixes your problem.

commit 278255de85829c850d36ad1655d1541e66d5368f
AuthorDate: Sat Jan 15 18:11:36 2022 +0100

    Fail JSONPath Assertion on indefinite paths and no assertion value
    The docs for JSONPath Assertion state, that it will fail the assertion,
    when no element is found with the given JSON path. This was currently
    not followed, when an indefinite path was used. In such a case, the
    JSONPath library would return an empty list, which the assertion logic
    would log as OK.
    With this change we let the assertion fail, when
     * an indefinite path was given
     * and an empty list is extracted
     * and no assertion value is given
    Bugzilla Id: 65794
 .../java/org/apache/jmeter/assertions/JSONPathAssertion.java   | 10 ++++++++--
 .../org/apache/jmeter/assertions/TestJSONPathAssertion.java    |  6 +++++-
 xdocs/changes.xml                                              |  2 +-
 xdocs/usermanual/component_reference.xml                       |  4 +++-
 4 files changed, 17 insertions(+), 5 deletions(-)
Comment 10 Felix Schumacher 2022-01-24 15:37:20 UTC
*ping* @Vladimir, have you had time to check the changes?
Comment 11 Vladimir Rosu 2022-01-24 16:21:00 UTC
I will confirm tomorrow - sorry about the delay.
Comment 12 Vladimir Rosu 2022-01-25 19:30:08 UTC
I confirm it works as discussed above. 100% ok from my PoV for release.
Comment 13 Vladimir Rosu 2022-01-25 19:31:34 UTC
Forgot to add that I tested with trunk 5.5-snapshot 85580b4
Comment 14 The ASF infrastructure team 2022-09-24 20:38:23 UTC
This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/5615