I can reproduce the problem.

There are some minor differences between what JMeter sends and what the browser sends, but so far I have not been able to determine exactly which differences are significant.

Note that it works with Java Implementation, the issue is within the Submit request, as if I record all with HTTP4 and last with Java is works

I note one difference:
KO:
Content-Type: multipart/form-data; boundary=94tKRetqHIuE-vOYibNRsao99Mo4g9; charset=UTF-8

OK:
Content-Type: multipart/form-data; boundary=---------------------------7d159c1302d0y0

Although recorded sampler is identical.

I actually have a fix for this, but I'm not sure exactly how to go about checking in the code -
The problem is that it requires newer httpmime jar, and possibly the httpclient and http code ones. I'm using 4.3.5 in my fix. 

I modified the HTTPHC4Impl class, and therefore if I were to either check it in or create a new one, it would require a dependency change and I don't know if that is acceptable. A lot of code is deprecated. 

The way that I was able to solve the problem was that inside of the method org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sendPostData(HttpPost), inside of the  // Check if we should do a multipart/form-data or an
        // application/x-www-form-urlencoded post request
        if(getUseMultipartForPost()) {
I changed the code to the newer MultipartEntityBuilder and most importantly, had to call the setLaxMode(). 

if(getUseMultipartForPost()) {
            // If a content encoding is specified, we use that as the
            // encoding of any parameter values
            Charset charset = null;
            if(haveContentEncoding) {
                charset = Charset.forName(contentEncoding);
            }

            // Write the request to our own stream
            MultipartEntityBuilder mep = MultipartEntityBuilder.create();
            
            // Create the parts
            // Add any parameters
            PropertyIterator args = getArguments().iterator();
            while (args.hasNext()) {
               HTTPArgument arg = (HTTPArgument) args.next().getObjectValue();
               String parameterName = arg.getName();
               if (arg.isSkippable(parameterName)){
                   continue;
               }
               
               FormBodyPart formPart;
               mep.addBinaryBody(arg.getName(), arg.getValue().getBytes("ASCII"));
               mep.setLaxMode();
            }

            // Add any files
            // Cannot retrieve parts once added to the MultiPartEntity, so have to save them here.
            FileBody[] fileBodies = new FileBody[files.length];
            for (int i=0; i < files.length; i++) {
                HTTPFileArg file = files[i];
                fileBodies[i] = new FileBody(new File(file.getPath()));
                
                mep.addPart(file.getParamName(), fileBodies[i] );
            }
            
            HttpEntity pst = mep.build();
            post.setEntity(pst);
           
            if (pst.isRepeatable()){
                ByteArrayOutputStream bos = new ByteArrayOutputStream();
                for(FileBody fileBody : fileBodies){
                   // fileBody.hideFileData = true;
                }
                pst.writeTo(bos);
              
                bos.flush();
                // We get the posted bytes using the encoding used to create it
                postedBody.append(new String(bos.toByteArray(),
                        contentEncoding == null ? "US-ASCII" // $NON-NLS-1$ this is the default used by HttpClient
                        : contentEncoding));
                bos.close();
            } else {
                postedBody.append("<Multipart was not repeatable, cannot view what was sent>"); // $NON-NLS-1$
            }


        } else { // not multipart

Hello Dan,
Thanks for investigation.

Could you submit a patch on this ? it will be easier to understand.
Also did you run ant test to check no regression is introduced by this change ?
Thanks

I'm sorry, no I didn't run regression tests. I was going to submit a patch, however; the changes would require the use of newer jars. I would need to modify the ant properties to use the 4.3.5 version of the http client mime jar. If I did that, I would also want to use the same version for the other 2 http client jars. 

If such jars were to be used, and it were my project, I would also want to upgrade the code to leverage these jars, by changing the deprecated method calls and other improvements. 

So in order to provide a patch, I could take many different routes. I'm not honestly sure of the best approach. 

I just provided you with the solution for this one case scenario. I was really hoping that you could use the snippet I provided with the 4.3.5 jars, and sort of take it from there.

Would you be able to provide a patch for just the code that you changed?

It's very difficult to work out exactly what you have changed in the method HTTPHC4Impl.sendPostData(HttpPost) from the Bugzilla comment.

Other aspects of the change (updating jar versions) are relatively easy to do.

Created attachment 32251 [details]
the fix

Ok, so I merged the fix I made from 2.11 into the trunk's code, and came up with this patch for you to look at. 

The important parts are 

Use MultipartEntityBuilder instead of the new MultipartEntity

and

 mep.setLaxMode();

Thanks, that makes it easier to review.

I think there are some problems with the patch:

987: the charset may not be ASCII, should use "charset" instead.

993: ViewableFileBody is required in order to support the Tree View Listener

and the trivial:
1042: spurious change to spacing (not relevant to the bug)

These are fixable, and I assume the patch would still work if these changes were made, but that is not easy for us to test.

Would you be able to fix test the above changes?

Yes, I see what you're saying. 
Both points are valid and it shouldn't matter if you want to change those. I left that in by accident

And yes, I can test it again. I can easily reproduce it locally, and make sure this bug is still fixed. It might be a little bit before I can, but I will.

It seems as though it happens with an Oracle ADF webapp but only for certain pages. It's definitely the boundary string though

(In reply to Dan from comment #10)

> And yes, I can test it again. I can easily reproduce it locally, and make
> sure this bug is still fixed. It might be a little bit before I can, but I
> will.

Thanks.
 
> It seems as though it happens with an Oracle ADF webapp but only for certain
> pages. It's definitely the boundary string though

The boundary string value _should_ be irrelevant, however maybe the charset attribute is causing an issue (see comment 3 above)

Created attachment 32255 [details]
Difference in wireshark

From wireshark, the top half is the request that fails - the bottom half works when you setLaxMode()

From the MultipartEntityBuilder, the setLaxMode() changes something to allow it to work


    public MultipartEntityBuilder setLaxMode() {
        this.mode = HttpMultipartMode.BROWSER_COMPATIBLE;
        return this;
    }
    public MultipartEntityBuilder setStrictMode() {
        this.mode = HttpMultipartMode.STRICT;
        return this;
    }

I haven't delved deeper than this, only that it stopped the problem from happening when observing in wire shark, which was enough to get me rolling on what I was trying to accomplish

But you could very well be right. "The multipart dissector could not find the required boundary parameter." is the error description of wireshark's internal parser. 

If you know the underlying answer, I'd be interested!

After looking at the differences between setLaxMode and STRICT, I see that the former uses the HttpBrowserCompatibleMultipart class to build the entity, and I wonder if the problem exists when it inserts

Content-Transfer-Encoding: 8bit\r\n\r\n 

The 3 bytes it says it has in the problem sample, include 0x0d0a so it is including the cr and nl in the data.

This does look suspicious.
I wonder why the extra CRLF is present?

Maybe there is some code that puts CRLF after lines in the output - if a line is empty for some reason the result would be two consecutive CRLF pairs.

It would be helpful to see the full headers for both scenarios

Whenever you want to, can you attach a patch that you would like for me test against the trunk or another version? 

I can give it a whirl.

Hi,
We updated recently to httpclient/httpmime 4.5.1 and httpcore-4.4.4.jar .
Could you give nightly build a try and see if it works for you ?
http://jmeter.apache.org/nightly.html
Installing JMeter runtime
Download the _bin and _lib files
Unpack the archives into the same directory structure
The other archives are not needed to run JMeter. 

Thanks

Sorry, you will have to wait for httpclient 4.5.2 release, as there is a fix that might be the problem:

- https://issues.apache.org/jira/browse/HTTPCLIENT-1665

If you can build it and use it , it would be great otherwise wait for my comment.
Thanks in advance.

Hi Dan,
NIghtly build integrating HttpClient 4.5.2 is available.
Could you give it a try ?
http://jmeter.apache.org/nightly.html
Installing JMeter runtime
Download the _bin and _lib files
Unpack the archives into the same directory structure
The other archives are not needed to run JMeter.

Note that you might have to check "Browser-compatible headers" checkbox on Http Request

(In reply to Philippe Mouawad from comment #20)
> Note that you might have to check "Browser-compatible headers" checkbox on
> Http Request

Philippe, it didn't work. I think it has to be the MultipartEntityBuilder instead of the MultipartEntity. (and setLaxMode)

I'll try to make changes in the morning and see if it works.

Hi,
Thanks for feedback.
Did you check "Browser-compatible headers" in HttpRequest ?
Because MultipartEntityBuilder#setLaxMode is the same as using:

HttpMultipartMode.BROWSER_COMPATIBLE which we use when "Browser-compatible headers" is checked.


https://github.com/apache/jmeter/blob/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java#L1084

Regards

Created attachment 33607 [details]
The recording options

There is isn't an option for browser headers in the recorder

Created attachment 33608 [details]
The resulting httprequest

The browser compatble header option appears after the recording takes place.

Ok, I understand now.

Could you debug using nightly build and see if you pass here when recording :
https://github.com/apache/jmeter/blob/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/DefaultSamplerCreator.java#L190
Thanks

(In reply to Philippe Mouawad from comment #25)
> Ok, I understand now.
> 
> Could you debug using nightly build and see if you pass here when recording :
> https://github.com/apache/jmeter/blob/trunk/src/protocol/http/org/apache/
> jmeter/protocol/http/proxy/DefaultSamplerCreator.java#L190
> Thanks

Yes sir, I have verified that the line was called and yet it had the same result.

I made a few changes and got it working, and I'll attached the changed file

Created attachment 33610 [details]
With my changes

Ok, I've added in my comments to each line I changed. 
With these changes it does work

Author: pmouawad
Date: Fri Mar  4 23:12:51 2016
New Revision: 1733674

URL: http://svn.apache.org/viewvc?rev=1733674&view=rev
Log:
Bug 56141 - Application does not behave correctly when using HTTP Recorder
Bugzilla Id: 56141

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/DefaultSamplerCreator.java
    jmeter/trunk/xdocs/changes.xml

Hi Dan,
Thanks for your tests and patch.
I cannot apply it like this as you force Lax mode everytime.

So I made the change in HTTP Test Script Recorder part to call sampler.setDoBrowserCompatibleMultipart(true); which will set LaxMode.
But this will only be done for Multipart requests.

It would be great if you could test it.
If it works please comment.
If it doesn't can you show the algorithm path in DefaultSamplerCreator#computeFromPostBody

Thanks

Hi Again Dan,
You can directly get the nightly build (unzip and use) at:
https://builds.apache.org/job/JMeter-trunk/lastSuccessfulBuild/artifact/trunk/dist/


Thanks

(In reply to Philippe Mouawad from comment #30)
> Hi Again Dan,
> You can directly get the nightly build (unzip and use) at:
> https://builds.apache.org/job/JMeter-trunk/lastSuccessfulBuild/artifact/
> trunk/dist/
> 
> 
> Thanks

Hi Phillippe, I did try it out this morning and it didn't work.

Hi Dan,
Thanks for your test.
Do you have some public URL I could use ?
Otherwise can you show the algorithm path in DefaultSamplerCreator#computeFromPostBody that was followed when you recorded the bugged transaction ?
Thanks

(In reply to Philippe Mouawad from comment #32)
> Hi Dan,
> Thanks for your test.
> Do you have some public URL I could use ?
> Otherwise can you show the algorithm path in
> DefaultSamplerCreator#computeFromPostBody that was followed when you
> recorded the bugged transaction ?
> Thanks

Unfortunately it is a very private url 

The path that it takes is always through line 151, and then either through the 
 if (!StringUtils.isEmpty(contentEncoding)) {
or the 
else...
And then enters the if (urlConfig != null) { 

So, going back to the change I made -> the existing HTTPHC4Impl class uses the org.apache.http.entity.mime.MultipartEntity which is deprecated, instead of the MultiPartEntityBuilder that I tried to add. And also, the setLaxMode() just changes the mode from STRICT to BROWSER_COMPATIBLE. Isn't that what you would want anyway? 

If the default doBrowserCompat... is set to false by default (since the prop isn't set) it will always be STRICT.

Hi Dan,
First big thanks for your rapid feedback.

Do you confirm that we enter this part of code when recording:

https://github.com/apache/jmeter/blob/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/DefaultSamplerCreator.java#L190

if so LaxMode should be set.

Regarding your patch, yes it would be better to use new APIs as you did, but the patch considers we are always in laxmode, I cannot change it this way.

And the other part is that you use addBinary I would need more testing.

I will anyway try to switch to the new aPIs you used but I will need some feedback from you when done.Regards

Author: pmouawad
Date: Sun Mar  6 15:29:40 2016
New Revision: 1733809

URL: http://svn.apache.org/viewvc?rev=1733809&view=rev
Log:
Bug 56141 - Application does not behave correctly when using HTTP Recorder
Migrate to up to date HttpMime APIs
Bugzilla Id: 56141

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java

Hi Dan,
I migrated to new APIs.
And as you can see my code is not exactly yours as it would break the JUnit tests that also include tests on Strict mode.

I cannot call setLaxMode unless getDoBrowserCompatibleMultipart() returns true.
So if the problem is this one, the bug would be in DefaultSamplerCreator.
If the problem is somewhere else (in the way we add parts) I would need more information.


So if it still fails, would it be possible to use the attached JMX file, change the path I used in View Results Tree under HTTP(S) Test Script Recorder to something that exists on your disk.
Then start recording the part that fail and send me or attach the result here.

Thank you

(In reply to Philippe Mouawad from comment #36)
> Hi Dan,
> I migrated to new APIs.
> And as you can see my code is not exactly yours as it would break the JUnit
> tests that also include tests on Strict mode.
> 
> I cannot call setLaxMode unless getDoBrowserCompatibleMultipart() returns
> true.
> So if the problem is this one, the bug would be in DefaultSamplerCreator.
> If the problem is somewhere else (in the way we add parts) I would need more
> information.
> 
> 
> So if it still fails, would it be possible to use the attached JMX file,
> change the path I used in View Results Tree under HTTP(S) Test Script
> Recorder to something that exists on your disk.
> Then start recording the part that fail and send me or attach the result
> here.
> 
> Thank you

I understand - I set the 
HTTPSampler.BROWSER_COMPATIBLE_MULTIPART$Boolean=true and she failed! But I verified that the property was set 

Also, yes it did hit that line 
 sampler.setDoBrowserCompatibleMultipart(true); // we are parsing browser input here

I didn't see the JMX file attached - where is that?

Created attachment 33642 [details]
Recording plan

(In reply to Dan from comment #37)
> (In reply to Philippe Mouawad from comment #36)
> > Hi Dan,
> > I migrated to new APIs.
> > And as you can see my code is not exactly yours as it would break the JUnit
> > tests that also include tests on Strict mode.
> > 
> > I cannot call setLaxMode unless getDoBrowserCompatibleMultipart() returns
> > true.
> > So if the problem is this one, the bug would be in DefaultSamplerCreator.
> > If the problem is somewhere else (in the way we add parts) I would need more
> > information.
> > 
> > 
> > So if it still fails, would it be possible to use the attached JMX file,
> > change the path I used in View Results Tree under HTTP(S) Test Script
> > Recorder to something that exists on your disk.
> > Then start recording the part that fail and send me or attach the result
> > here.
> > 
> > Thank you
> 
> I understand - I set the 
> HTTPSampler.BROWSER_COMPATIBLE_MULTIPART$Boolean=true and she failed! But I
> verified that the property was set 
> 
> Also, yes it did hit that line 
>  sampler.setDoBrowserCompatibleMultipart(true); // we are parsing browser
> input here
> 
> I didn't see the JMX file attached - where is that?

I have attached it now.

Delaying to next version 3.1 as not enough information for now to fix it

*** Bug 59681 has been marked as a duplicate of this bug. ***

Hello Dan,
Could you take some time to test whether nightly build fixes this issue ?:

- http://jmeter.apache.org/nightly.html

Thanks a lot in advance.
Regards

Closing as it's been a long time since I haven't faced such issues.
There have been recently fixes in recorder.

If anybody faces issue, feel free to reopen with details

This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/3317