Bug 63527

Summary: HTTP Request Defaults : "URLs must match" -->"URLs Patterns to Include" + "URL Patterns to Exclude"
Product: JMeter - Now in Github Reporter: pierre.astruc
Component: HTTPAssignee: JMeter issues mailing list <issues>
Status: RESOLVED FIXED    
Severity: enhancement CC: sebb
Priority: P2 Keywords: FixedInTrunk
Version: 5.1.1   
Target Milestone: JMETER_5.4   
Hardware: All   
OS: All   
Attachments: Test plan to reproduce the problem

Description pierre.astruc 2019-06-24 19:31:42 UTC
Hello !

I'm using Jmeter to "retreive all embedded resouces" from some HTTP request, and it's working pretty well :)

Actually, I can whiteliste domains with regex like:
(?i).*(static\.apache\.net|media\.apache\.net|www\.apache\.net).*

Or blacklist some domains with regex like:
^((?!facebook\.com|googleapis\.com|abtasty\.com).)*$

As enhancement, I would like to see instead of "URLs must match" something like "HTTP(s) test Scrip Recorder", a way to declare URLs to keep, + a way to declare URLs to remove.
For example, I'm testing some whitelisted URLs domains with some 404 URLs pictures I would like to blacklist, and I can't seem to find a regex way to perform that with current 5.1.1 engine.

Thanks for reading !
Cheers
Comment 1 pierre.astruc 2020-08-01 19:53:51 UTC
I'm still looking for a solution to whitelist + blacklist some domains into the "retreive all embedded resouces" engine.
At the moment there is only one regex possible, so I can only whitelist or blacklist the url I think, am I wrong ?

Thanks !
Comment 2 Felix Schumacher 2020-08-02 13:36:39 UTC
Started an implementation at https://github.com/apache/jmeter/pull/607
Comment 3 pierre.astruc 2020-08-03 09:00:07 UTC
Awesome Felix, thanks again :)
Comment 4 Felix Schumacher 2020-08-08 12:34:21 UTC
Hi Pierre, could you please test next nightly and report, if this is what you wanted?

commit 0592978cb16ff8e7aea3ca96215e01141a78c2cd
AuthorDate: Sun Aug 2 15:25:18 2020 +0200

    Implement a new setting to allow the exclusion of embedded URLs
    
    At the moment we allow only one regular expression to control which URLs will
    be downloaded by the HTTP sampler, when embedded download is enabled.
    
    This can be used to exclude specific URLs, but it is not possible to exclude and allow
    URLs at the same time.
    
    This patch gives the user a second field which can be used to exclude URLs.
    
    Bugzilla Id: 63527
    Closes #607
---
 .../apache/jmeter/resources/messages.properties    |   1 +
 .../apache/jmeter/resources/messages_fr.properties |   1 +
 src/protocol/build.gradle.kts                      |   1 +
 .../protocol/http/config/gui/HttpDefaultsGui.java  |  62 +++++++++++++--------
 .../http/control/gui/HttpTestSampleGui.java        |  52 +++++++++--------
 .../protocol/http/sampler/HTTPSamplerBase.java     |  41 ++++++++++++--
 xdocs/changes.xml                                  |   1 +
 .../http-request-defaults-advanced-tab.png         | Bin 9455 -> 19213 bytes
 .../screenshots/http-request-advanced-tab.png      | Bin 6194 -> 18573 bytes
 xdocs/usermanual/component_reference.xml           |  22 ++++++--
 10 files changed, 123 insertions(+), 58 deletions(-)
Comment 5 pierre.astruc 2020-08-11 19:52:23 UTC
Of course Felix, I will try & report :)
Thanks for the dev !
Comment 6 pierre.astruc 2020-08-13 09:39:09 UTC
Created attachment 37391 [details]
Test plan to reproduce the problem

Hello Felix,

I made first tests with nightly #53873bc.
I just have a simple GET to https://www.evertest.com (test plan attached)

test1:
- when I'm using the reg (?i).*\.(js|css) to "URLs must not match": it's not working :(

test2:
- when I'm adding the reg (?i).*(evertest\.com).* to "URLs must match": it's working ! (jmeter only download from evertest.com + exclude *.js + *.css)

So the engine seems good, but it should also accept only "URLs must not match" info :)
At the moment, "URLs must not match" seems applied only when "URLs must match" are also defined.

Cheers !
Comment 7 Felix Schumacher 2020-08-13 10:52:27 UTC
Thanks for testing and the feedback.

Could you test again with the next nightly (or trunk build)?

commit 445df9f387d23cf5b8defa7c3d257608bff7904b
Author: Felix Schumacher <felix.schumacher@internetallee.de>
AuthorDate: Thu Aug 13 12:49:06 2020 +0200

    Fix to: Implement a new setting to allow the exclusion of embedded URLs
    
    A logic mistake would make the exclude filter be only applied, when
    a include filter was defined.
    
    Bugzilla Id: 63527
    Relates to #607
---
 .../java/org/apache/jmeter/protocol/http/sampler/HTTPSamplerBase.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Comment 8 pierre.astruc 2020-08-24 16:48:20 UTC
Hi Felix,

This is perfect, both filters are now working as needed ! (5.3.1-snapshot-ba27712)
A big thanks for improving the Embedded Resources download manager :)
Issue closed.

Pierre
Comment 9 Felix Schumacher 2020-09-06 07:59:50 UTC
*** Bug 53848 has been marked as a duplicate of this bug. ***
Comment 10 The ASF infrastructure team 2022-09-24 20:38:17 UTC
This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/5109