Bug 49039

Hello,
Could you provide the detailed steps of your test and if possible a front facing site  for this issue ?
Otherwise a capture of the dialog with the server.

Thank you
Regards

I can confirm this.

when recording HTTP data with JMeter, I'd assume that the request is sent to the server just like the browser would send it.
This is not the case in two examples I've run across.

As you'll know, when recording with HTTP Proxy Server, JMeter will be creating HTTP Samplers and then use their sample method to realize the request, returning the return value to the browser.
So we need to create a HTTP Sampler which produces the same request as the browser would have done.

When looking at HttpRequestHdr.populateSampler, I see that ConversionUtils.getEncodingFromContentType
is called. In case the encoding is not supported (returns null) we look in the pageEncodings and formEncodings maps (when and how are they populated??) and if we find nothing here, we use:
postData = new String(rawPostData);
meaning we convert the bytes to a string based on the platform-specific default encoding.

In my examples this breaks the communication, so not only replay but record does not work.
In one case I have no Content-Type header in the request, and new String(rawPostData) seems to break it, in the other case, I have a Content-Type: application/soap+msbin1 header, where "getEncodingFromContentType" returns null.

We really should make sure JMeter sends exactly what it received to the server when acting as proxy, and even better would be if this would work when executing the test.

For the former, the solution I found seems to be to change the line HttpRequestHdr: postData = new String(rawPostData);
to:
postData = new String(rawPostData, PostWriter.ENCODING);
to match the encoding which is used when executing the request in case no encoding is set.

I also have a problem with the line 
if (firstLine && !CharUtils.isAscii((char) x)){
which I can solve by commenting this out...

Hello Jens,
Thanks for your detailed description and investigation.

Is the site you are testing against a public one ?
If not would it be possible for you to capture your scenario with FIDDLER tool and make a Word document with screenshots of the Fiddler record. 
By having the 2 we will be able to reproduce and fix the issue.

Thank you
Regards
Philippe

The site is not a public one, but one from a customer and I don't have access any more, I'm afraid.

Probably it's enough to test if you create a http server returning some constant random byte garbage and a http client sending random bytes as post data, and checking whether there's a difference when you put JMeter as proxy in between.
As long as the Java VMs default encoding is not equal to PostWriter.ENCODING, you'll have a difference.

(In reply to comment #2)
> I can confirm this.
> 
> when recording HTTP data with JMeter, I'd assume that the request is sent to
> the server just like the browser would send it.
> This is not the case in two examples I've run across.
> 
> As you'll know, when recording with HTTP Proxy Server, JMeter will be creating
> HTTP Samplers and then use their sample method to realize the request,
> returning the return value to the browser.
> So we need to create a HTTP Sampler which produces the same request as the
> browser would have done.
> 
> When looking at HttpRequestHdr.populateSampler, I see that
> ConversionUtils.getEncodingFromContentType
> is called. In case the encoding is not supported (returns null) we look in the
> pageEncodings and formEncodings maps (when and how are they populated??) and if
> we find nothing here, we use:
> postData = new String(rawPostData);
> meaning we convert the bytes to a string based on the platform-specific default
> encoding.
> 
> In my examples this breaks the communication, so not only replay but record
> does not work.
> In one case I have no Content-Type header in the request, and new
> String(rawPostData) seems to break it, in the other case, I have a
> Content-Type: application/soap+msbin1 header, where
> "getEncodingFromContentType" returns null.
> 
> We really should make sure JMeter sends exactly what it received to the server
> when acting as proxy, and even better would be if this would work when
> executing the test.
> 
> For the former, the solution I found seems to be to change the line
> HttpRequestHdr: postData = new String(rawPostData);
> to:
> postData = new String(rawPostData, PostWriter.ENCODING);
> to match the encoding which is used when executing the request in case no
> encoding is set.
> 
This seems reasonable, as indeed Postwriter uses ENCODING const when encoding is null.
> I also have a problem with the line 
> if (firstLine && !CharUtils.isAscii((char) x)){
> which I can solve by commenting this out...
Regarding this, would it be possible to attach to Bugzilla à fiddler capture of the failing request.
CAN you give more information about the tested app, is it a binary soap webservice ?

It was another app than the other problem, but same here no access any more.
I can confirm a binary SOAP webservice was involved though I do not remember whether it was this request. The char in question was the German umlaut "ü".

This issue has been fixed as part of 52674 as of using :
new String(request.getRawPostData(), PostWriter.ENCODING);
when contentEncoding is not found.

Regarding  binary soap webservice , as it's a binary protocol recording would require implementing a dedicated SamplerCreator associated to mime type.

*** This bug has been marked as a duplicate of bug 52674 ***

This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/2358