Bug 56141 - Application does not behave correctly when using HTTP Recorder
Summary: Application does not behave correctly when using HTTP Recorder
Status: NEEDINFO
Alias: None
Product: JMeter
Classification: Unclassified
Component: Main (show other bugs)
Version: 2.9
Hardware: All All
: P2 major (vote)
Target Milestone: ---
Assignee: JMeter issues mailing list
URL:
Keywords:
: 59681 (view as bug list)
Depends on:
Blocks:
 
Reported: 2014-02-15 10:57 UTC by Deeptish
Modified: 2016-06-11 08:15 UTC (History)
3 users (show)



Attachments
the fix (4.96 KB, patch)
2014-12-02 15:06 UTC, Dan
Details | Diff
Difference in wireshark (54.83 KB, image/png)
2014-12-02 23:57 UTC, Dan
Details
The recording options (164.85 KB, image/png)
2016-02-28 16:39 UTC, Dan
Details
The resulting httprequest (160.05 KB, image/png)
2016-02-28 16:40 UTC, Dan
Details
With my changes (62.36 KB, text/x-csrc)
2016-02-29 01:42 UTC, Dan
Details
Recording plan (8.33 KB, application/xml)
2016-03-07 12:30 UTC, Philippe Mouawad
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Deeptish 2014-02-15 10:57:52 UTC
1. go to http://projects.csmpl.com/nbpdcl/
user id: santa
password : password1

2. Go to Registration under Grievance Monitoring

3 i)   Consumer number 123456
  ii)  Name : Any name
  iii) Phone # : any cell number (should not DND activated)
  iv)  Gender : Fill as you want
  v)   District : Araia 
  vi)  Block : Araia
  vii) Panchayat : Araia Basti
  viii)Address1 : anything
  ix)  Complaint Category : Meter Installation
  x)   Complaint Sub-Category : New Meter
  x1 Complaint Details  : Some Text

4. Click on Submit button

With the above data & 

NO PROBELM : without proxy server we are able to register the complain
DEFECT :     But With proxy server & starting it through JMETER we are not able to do any thing even clicking on Submit(step 4) button application is clearing all the data entered
Comment 1 Sebb 2014-03-14 01:30:47 UTC
I can reproduce the problem.

There are some minor differences between what JMeter sends and what the browser sends, but so far I have not been able to determine exactly which differences are significant.
Comment 2 Philippe Mouawad 2014-03-22 21:57:08 UTC
Note that it works with Java Implementation, the issue is within the Submit request, as if I record all with HTTP4 and last with Java is works
Comment 3 Philippe Mouawad 2014-03-22 22:29:33 UTC
I note one difference:
KO:
Content-Type: multipart/form-data; boundary=94tKRetqHIuE-vOYibNRsao99Mo4g9; charset=UTF-8

OK:
Content-Type: multipart/form-data; boundary=---------------------------7d159c1302d0y0

Although recorded sampler is identical.
Comment 4 Dan 2014-10-29 14:42:23 UTC
I actually have a fix for this, but I'm not sure exactly how to go about checking in the code -
The problem is that it requires newer httpmime jar, and possibly the httpclient and http code ones. I'm using 4.3.5 in my fix. 

I modified the HTTPHC4Impl class, and therefore if I were to either check it in or create a new one, it would require a dependency change and I don't know if that is acceptable. A lot of code is deprecated. 

The way that I was able to solve the problem was that inside of the method org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sendPostData(HttpPost), inside of the  // Check if we should do a multipart/form-data or an
        // application/x-www-form-urlencoded post request
        if(getUseMultipartForPost()) {
I changed the code to the newer MultipartEntityBuilder and most importantly, had to call the setLaxMode(). 

if(getUseMultipartForPost()) {
            // If a content encoding is specified, we use that as the
            // encoding of any parameter values
            Charset charset = null;
            if(haveContentEncoding) {
                charset = Charset.forName(contentEncoding);
            }

            // Write the request to our own stream
            MultipartEntityBuilder mep = MultipartEntityBuilder.create();
            
            // Create the parts
            // Add any parameters
            PropertyIterator args = getArguments().iterator();
            while (args.hasNext()) {
               HTTPArgument arg = (HTTPArgument) args.next().getObjectValue();
               String parameterName = arg.getName();
               if (arg.isSkippable(parameterName)){
                   continue;
               }
               
               FormBodyPart formPart;
               mep.addBinaryBody(arg.getName(), arg.getValue().getBytes("ASCII"));
               mep.setLaxMode();
            }

            // Add any files
            // Cannot retrieve parts once added to the MultiPartEntity, so have to save them here.
            FileBody[] fileBodies = new FileBody[files.length];
            for (int i=0; i < files.length; i++) {
                HTTPFileArg file = files[i];
                fileBodies[i] = new FileBody(new File(file.getPath()));
                
                mep.addPart(file.getParamName(), fileBodies[i] );
            }
            
            HttpEntity pst = mep.build();
            post.setEntity(pst);
           
            if (pst.isRepeatable()){
                ByteArrayOutputStream bos = new ByteArrayOutputStream();
                for(FileBody fileBody : fileBodies){
                   // fileBody.hideFileData = true;
                }
                pst.writeTo(bos);
              
                bos.flush();
                // We get the posted bytes using the encoding used to create it
                postedBody.append(new String(bos.toByteArray(),
                        contentEncoding == null ? "US-ASCII" // $NON-NLS-1$ this is the default used by HttpClient
                        : contentEncoding));
                bos.close();
            } else {
                postedBody.append("<Multipart was not repeatable, cannot view what was sent>"); // $NON-NLS-1$
            }


        } else { // not multipart
Comment 5 Philippe Mouawad 2014-10-29 20:21:52 UTC
Hello Dan,
Thanks for investigation.

Could you submit a patch on this ? it will be easier to understand.
Also did you run ant test to check no regression is introduced by this change ?
Thanks
Comment 6 Dan 2014-12-01 15:04:36 UTC
I'm sorry, no I didn't run regression tests. I was going to submit a patch, however; the changes would require the use of newer jars. I would need to modify the ant properties to use the 4.3.5 version of the http client mime jar. If I did that, I would also want to use the same version for the other 2 http client jars. 

If such jars were to be used, and it were my project, I would also want to upgrade the code to leverage these jars, by changing the deprecated method calls and other improvements. 

So in order to provide a patch, I could take many different routes. I'm not honestly sure of the best approach. 

I just provided you with the solution for this one case scenario. I was really hoping that you could use the snippet I provided with the 4.3.5 jars, and sort of take it from there.
Comment 7 Sebb 2014-12-01 17:15:50 UTC
Would you be able to provide a patch for just the code that you changed?

It's very difficult to work out exactly what you have changed in the method HTTPHC4Impl.sendPostData(HttpPost) from the Bugzilla comment.

Other aspects of the change (updating jar versions) are relatively easy to do.
Comment 8 Dan 2014-12-02 15:06:34 UTC
Created attachment 32251 [details]
the fix

Ok, so I merged the fix I made from 2.11 into the trunk's code, and came up with this patch for you to look at. 

The important parts are 

Use MultipartEntityBuilder instead of the new MultipartEntity

and

 mep.setLaxMode();
Comment 9 Sebb 2014-12-02 16:22:10 UTC
Thanks, that makes it easier to review.

I think there are some problems with the patch:

987: the charset may not be ASCII, should use "charset" instead.

993: ViewableFileBody is required in order to support the Tree View Listener

and the trivial:
1042: spurious change to spacing (not relevant to the bug)

These are fixable, and I assume the patch would still work if these changes were made, but that is not easy for us to test.

Would you be able to fix test the above changes?
Comment 10 Dan 2014-12-02 20:46:01 UTC
Yes, I see what you're saying. 
Both points are valid and it shouldn't matter if you want to change those. I left that in by accident

And yes, I can test it again. I can easily reproduce it locally, and make sure this bug is still fixed. It might be a little bit before I can, but I will.

It seems as though it happens with an Oracle ADF webapp but only for certain pages. It's definitely the boundary string though
Comment 11 Sebb 2014-12-02 22:25:30 UTC
(In reply to Dan from comment #10)

> And yes, I can test it again. I can easily reproduce it locally, and make
> sure this bug is still fixed. It might be a little bit before I can, but I
> will.

Thanks.
 
> It seems as though it happens with an Oracle ADF webapp but only for certain
> pages. It's definitely the boundary string though

The boundary string value _should_ be irrelevant, however maybe the charset attribute is causing an issue (see comment 3 above)
Comment 12 Dan 2014-12-02 23:57:50 UTC
Created attachment 32255 [details]
Difference in wireshark

From wireshark, the top half is the request that fails - the bottom half works when you setLaxMode()
Comment 13 Dan 2014-12-03 00:03:46 UTC
From the MultipartEntityBuilder, the setLaxMode() changes something to allow it to work


    public MultipartEntityBuilder setLaxMode() {
        this.mode = HttpMultipartMode.BROWSER_COMPATIBLE;
        return this;
    }
    public MultipartEntityBuilder setStrictMode() {
        this.mode = HttpMultipartMode.STRICT;
        return this;
    }

I haven't delved deeper than this, only that it stopped the problem from happening when observing in wire shark, which was enough to get me rolling on what I was trying to accomplish

But you could very well be right. "The multipart dissector could not find the required boundary parameter." is the error description of wireshark's internal parser. 

If you know the underlying answer, I'd be interested!
Comment 14 Dan 2014-12-03 00:13:28 UTC
After looking at the differences between setLaxMode and STRICT, I see that the former uses the HttpBrowserCompatibleMultipart class to build the entity, and I wonder if the problem exists when it inserts

Content-Transfer-Encoding: 8bit\r\n\r\n 

The 3 bytes it says it has in the problem sample, include 0x0d0a so it is including the cr and nl in the data.
Comment 15 Sebb 2014-12-03 01:57:53 UTC
This does look suspicious.
I wonder why the extra CRLF is present?

Maybe there is some code that puts CRLF after lines in the output - if a line is empty for some reason the result would be two consecutive CRLF pairs.

It would be helpful to see the full headers for both scenarios
Comment 16 Dan 2014-12-04 15:08:39 UTC
Whenever you want to, can you attach a patch that you would like for me test against the trunk or another version? 

I can give it a whirl.
Comment 17 Philippe Mouawad 2015-12-29 21:51:08 UTC
Hi,
We updated recently to httpclient/httpmime 4.5.1 and httpcore-4.4.4.jar .
Could you give nightly build a try and see if it works for you ?
http://jmeter.apache.org/nightly.html
Installing JMeter runtime
Download the _bin and _lib files
Unpack the archives into the same directory structure
The other archives are not needed to run JMeter. 

Thanks
Comment 18 Philippe Mouawad 2015-12-29 21:53:43 UTC
Sorry, you will have to wait for httpclient 4.5.2 release, as there is a fix that might be the problem:

- https://issues.apache.org/jira/browse/HTTPCLIENT-1665

If you can build it and use it , it would be great otherwise wait for my comment.
Thanks in advance.
Comment 19 Philippe Mouawad 2016-02-27 08:57:34 UTC
Hi Dan,
NIghtly build integrating HttpClient 4.5.2 is available.
Could you give it a try ?
http://jmeter.apache.org/nightly.html
Installing JMeter runtime
Download the _bin and _lib files
Unpack the archives into the same directory structure
The other archives are not needed to run JMeter.
Comment 20 Philippe Mouawad 2016-02-27 09:02:57 UTC
Note that you might have to check "Browser-compatible headers" checkbox on Http Request
Comment 21 Dan 2016-02-28 05:13:35 UTC
(In reply to Philippe Mouawad from comment #20)
> Note that you might have to check "Browser-compatible headers" checkbox on
> Http Request

Philippe, it didn't work. I think it has to be the MultipartEntityBuilder instead of the MultipartEntity. (and setLaxMode)

I'll try to make changes in the morning and see if it works.
Comment 22 Philippe Mouawad 2016-02-28 12:15:32 UTC
Hi,
Thanks for feedback.
Did you check "Browser-compatible headers" in HttpRequest ?
Because MultipartEntityBuilder#setLaxMode is the same as using:

HttpMultipartMode.BROWSER_COMPATIBLE which we use when "Browser-compatible headers" is checked.


https://github.com/apache/jmeter/blob/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java#L1084

Regards
Comment 23 Dan 2016-02-28 16:39:12 UTC
Created attachment 33607 [details]
The recording options

There is isn't an option for browser headers in the recorder
Comment 24 Dan 2016-02-28 16:40:39 UTC
Created attachment 33608 [details]
The resulting httprequest

The browser compatble header option appears after the recording takes place.
Comment 25 Philippe Mouawad 2016-02-28 16:43:01 UTC
Ok, I understand now.

Could you debug using nightly build and see if you pass here when recording :
https://github.com/apache/jmeter/blob/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/DefaultSamplerCreator.java#L190
Thanks
Comment 26 Dan 2016-02-29 01:40:03 UTC
(In reply to Philippe Mouawad from comment #25)
> Ok, I understand now.
> 
> Could you debug using nightly build and see if you pass here when recording :
> https://github.com/apache/jmeter/blob/trunk/src/protocol/http/org/apache/
> jmeter/protocol/http/proxy/DefaultSamplerCreator.java#L190
> Thanks

Yes sir, I have verified that the line was called and yet it had the same result.

I made a few changes and got it working, and I'll attached the changed file
Comment 27 Dan 2016-02-29 01:42:31 UTC
Created attachment 33610 [details]
With my changes

Ok, I've added in my comments to each line I changed. 
With these changes it does work
Comment 28 Philippe Mouawad 2016-03-04 23:19:46 UTC
Author: pmouawad
Date: Fri Mar  4 23:12:51 2016
New Revision: 1733674

URL: http://svn.apache.org/viewvc?rev=1733674&view=rev
Log:
Bug 56141 - Application does not behave correctly when using HTTP Recorder
Bugzilla Id: 56141

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/DefaultSamplerCreator.java
    jmeter/trunk/xdocs/changes.xml
Comment 29 Philippe Mouawad 2016-03-04 23:23:20 UTC
Hi Dan,
Thanks for your tests and patch.
I cannot apply it like this as you force Lax mode everytime.

So I made the change in HTTP Test Script Recorder part to call sampler.setDoBrowserCompatibleMultipart(true); which will set LaxMode.
But this will only be done for Multipart requests.

It would be great if you could test it.
If it works please comment.
If it doesn't can you show the algorithm path in DefaultSamplerCreator#computeFromPostBody

Thanks
Comment 30 Philippe Mouawad 2016-03-04 23:46:51 UTC
Hi Again Dan,
You can directly get the nightly build (unzip and use) at:
https://builds.apache.org/job/JMeter-trunk/lastSuccessfulBuild/artifact/trunk/dist/


Thanks
Comment 31 Dan 2016-03-05 15:19:21 UTC
(In reply to Philippe Mouawad from comment #30)
> Hi Again Dan,
> You can directly get the nightly build (unzip and use) at:
> https://builds.apache.org/job/JMeter-trunk/lastSuccessfulBuild/artifact/
> trunk/dist/
> 
> 
> Thanks

Hi Phillippe, I did try it out this morning and it didn't work.
Comment 32 Philippe Mouawad 2016-03-05 15:32:25 UTC
Hi Dan,
Thanks for your test.
Do you have some public URL I could use ?
Otherwise can you show the algorithm path in DefaultSamplerCreator#computeFromPostBody that was followed when you recorded the bugged transaction ?
Thanks
Comment 33 Dan 2016-03-06 01:11:33 UTC
(In reply to Philippe Mouawad from comment #32)
> Hi Dan,
> Thanks for your test.
> Do you have some public URL I could use ?
> Otherwise can you show the algorithm path in
> DefaultSamplerCreator#computeFromPostBody that was followed when you
> recorded the bugged transaction ?
> Thanks

Unfortunately it is a very private url 

The path that it takes is always through line 151, and then either through the 
 if (!StringUtils.isEmpty(contentEncoding)) {
or the 
else...
And then enters the if (urlConfig != null) { 

So, going back to the change I made -> the existing HTTPHC4Impl class uses the org.apache.http.entity.mime.MultipartEntity which is deprecated, instead of the MultiPartEntityBuilder that I tried to add. And also, the setLaxMode() just changes the mode from STRICT to BROWSER_COMPATIBLE. Isn't that what you would want anyway? 

If the default doBrowserCompat... is set to false by default (since the prop isn't set) it will always be STRICT.
Comment 34 Philippe Mouawad 2016-03-06 08:14:59 UTC
Hi Dan,
First big thanks for your rapid feedback.

Do you confirm that we enter this part of code when recording:

https://github.com/apache/jmeter/blob/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/DefaultSamplerCreator.java#L190

if so LaxMode should be set.

Regarding your patch, yes it would be better to use new APIs as you did, but the patch considers we are always in laxmode, I cannot change it this way.

And the other part is that you use addBinary I would need more testing.

I will anyway try to switch to the new aPIs you used but I will need some feedback from you when done.Regards
Comment 35 Philippe Mouawad 2016-03-06 15:30:15 UTC
Author: pmouawad
Date: Sun Mar  6 15:29:40 2016
New Revision: 1733809

URL: http://svn.apache.org/viewvc?rev=1733809&view=rev
Log:
Bug 56141 - Application does not behave correctly when using HTTP Recorder
Migrate to up to date HttpMime APIs
Bugzilla Id: 56141

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java
Comment 36 Philippe Mouawad 2016-03-06 15:37:25 UTC
Hi Dan,
I migrated to new APIs.
And as you can see my code is not exactly yours as it would break the JUnit tests that also include tests on Strict mode.

I cannot call setLaxMode unless getDoBrowserCompatibleMultipart() returns true.
So if the problem is this one, the bug would be in DefaultSamplerCreator.
If the problem is somewhere else (in the way we add parts) I would need more information.


So if it still fails, would it be possible to use the attached JMX file, change the path I used in View Results Tree under HTTP(S) Test Script Recorder to something that exists on your disk.
Then start recording the part that fail and send me or attach the result here.

Thank you
Comment 37 Dan 2016-03-07 01:26:06 UTC
(In reply to Philippe Mouawad from comment #36)
> Hi Dan,
> I migrated to new APIs.
> And as you can see my code is not exactly yours as it would break the JUnit
> tests that also include tests on Strict mode.
> 
> I cannot call setLaxMode unless getDoBrowserCompatibleMultipart() returns
> true.
> So if the problem is this one, the bug would be in DefaultSamplerCreator.
> If the problem is somewhere else (in the way we add parts) I would need more
> information.
> 
> 
> So if it still fails, would it be possible to use the attached JMX file,
> change the path I used in View Results Tree under HTTP(S) Test Script
> Recorder to something that exists on your disk.
> Then start recording the part that fail and send me or attach the result
> here.
> 
> Thank you

I understand - I set the 
HTTPSampler.BROWSER_COMPATIBLE_MULTIPART$Boolean=true and she failed! But I verified that the property was set 

Also, yes it did hit that line 
 sampler.setDoBrowserCompatibleMultipart(true); // we are parsing browser input here

I didn't see the JMX file attached - where is that?
Comment 38 Philippe Mouawad 2016-03-07 12:30:37 UTC
Created attachment 33642 [details]
Recording plan
Comment 39 Philippe Mouawad 2016-03-07 12:31:13 UTC
(In reply to Dan from comment #37)
> (In reply to Philippe Mouawad from comment #36)
> > Hi Dan,
> > I migrated to new APIs.
> > And as you can see my code is not exactly yours as it would break the JUnit
> > tests that also include tests on Strict mode.
> > 
> > I cannot call setLaxMode unless getDoBrowserCompatibleMultipart() returns
> > true.
> > So if the problem is this one, the bug would be in DefaultSamplerCreator.
> > If the problem is somewhere else (in the way we add parts) I would need more
> > information.
> > 
> > 
> > So if it still fails, would it be possible to use the attached JMX file,
> > change the path I used in View Results Tree under HTTP(S) Test Script
> > Recorder to something that exists on your disk.
> > Then start recording the part that fail and send me or attach the result
> > here.
> > 
> > Thank you
> 
> I understand - I set the 
> HTTPSampler.BROWSER_COMPATIBLE_MULTIPART$Boolean=true and she failed! But I
> verified that the property was set 
> 
> Also, yes it did hit that line 
>  sampler.setDoBrowserCompatibleMultipart(true); // we are parsing browser
> input here
> 
> I didn't see the JMX file attached - where is that?

I have attached it now.
Comment 40 Philippe Mouawad 2016-03-11 20:26:48 UTC
Delaying to next version 3.1 as not enough information for now to fix it
Comment 41 Philippe Mouawad 2016-06-11 08:15:41 UTC
*** Bug 59681 has been marked as a duplicate of this bug. ***