Bug 63129 - JMeter can not identify encoding during first time page submission
Summary: JMeter can not identify encoding during first time page submission
Status: RESOLVED FIXED
Alias: None
Product: JMeter
Classification: Unclassified
Component: HTTP (show other bugs)
Version: 5.0
Hardware: All All
: P1 major (vote)
Target Milestone: JMETER_5.2
Assignee: JMeter issues mailing list
URL:
Keywords: FixedInTrunk
: 63417 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-01-29 17:10 UTC by Naveen Nandwani
Modified: 2019-05-19 20:58 UTC (History)
2 users (show)



Attachments
Application to reproduce the scenario. (993 bytes, application/zip)
2019-01-29 17:10 UTC, Naveen Nandwani
Details
Test Plan to reproduce the issue. (4.37 KB, application/xml)
2019-01-29 17:11 UTC, Naveen Nandwani
Details
Proposed changes in Source Code. Will create separate PR request. (1.27 KB, patch)
2019-01-29 17:20 UTC, Naveen Nandwani
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Naveen Nandwani 2019-01-29 17:10:40 UTC
Created attachment 36401 [details]
Application to reproduce the scenario.

Overview: When we post the Japanese characters from browser using JMeter HTTP(s) Test Script Recorder, we get garbage value in response when we post the request first time. However, if we post the same request again, we get actual output. So, from second execution onward it starts working fine.	

Cause of bug: JMeter is getting encoding as null in HTTP request. When JMeter gets encoding as null, it uses default encoding (UTF-8). therefore returning garbage value when encoding/decoding of characters(e.g. Japanese characters) other than UTF-8. However, JMeter records encoding when response comes from the web server, so it  starts working when we traverse same page again.

Impact of bug:
1. When End User traverse the page first time and posts any character String other than UTF-8 using JMeter proxy. End user gets garbage value in response of his first request. E.g. Shift-JIS characters are not supported by UTF-8 and return garbage value.
2. Many Japanese company uses JMeter for performance testing of their project So Whenever Japanese engineer, uses JMeter for their performance testing they did not get actual result after first request.

Reproduce condition: 

1. Page Encoding should be Shift_JIS.
2. Request  and response page encoding should be same.
3. Form method should be POST.

Steps to reproduce: 

Setup 
1. Deploy Application "jmeter_issue_Encoding_Store"(extract in apache-tomcat-9.0.12\webapps) in Tomcat and start the tomcat(execute "apache-tomcat-9.0.12\bin\startup.bat" .
2. Open the browser and change Proxy settings -> LAN Setting and set Address(eg: localhost) and port(e.g.8888).
3. Run JMeter
	3.1 click on file menu-> open ->open the attached "Test Plan.jmx" file.
	3.2 Set the same port number in “HTTP(s) Test Script Recorder” and in browser’s Network LAN Settings.
	3.3 Go to JMeter HTTP(s) Test Script Recorder -> Click Start -> Ok
	

Execution
1. Go to browser , open application (e.g. http://localhost:portNumber/jmeter_issue_Encoding_Store/index.jsp)

2. Click on Submit Button.
 
3. View result in browser.
Resultant Output: It is garbage text:  ?ソス?ソス?ソス?ソス?ソスl?ソス?ソス?ソスs
 
Expected Result: 初期大阪
Actual Result: ?ソス?ソス?ソス?ソス?ソスl?ソス?ソス?ソスs

4. click on "Back to previous page" and click on "Submit" button again.
5. View the result in browser, this time getting the actual result.

Expected Result: 初期大阪
Actual Result: 初期大阪
Comment 1 Naveen Nandwani 2019-01-29 17:11:57 UTC
Created attachment 36402 [details]
Test Plan to reproduce the issue.
Comment 2 Naveen Nandwani 2019-01-29 17:20:24 UTC
Created attachment 36403 [details]
Proposed changes in Source Code. Will create separate PR request.

Solution:

First Request is GET to load the base url. JMeter stores the encoding of the page (Shift-JIS).
Second Request is POST to submit "Japanese Character", but encoding is not available, as page is traversed for first time. If encoding of page of first request (Shift-JIS) is used, then Japanese Character will be correctly stored.

Following is the code snippet:
contentEncoding = pageEncodings.get(urlWithoutQuery);
if(contentEncoding == null && pageEncodings.size()>0) {
     contentEncoding = new ArrayList<String>(pageEncodings.values()).get(0);
}

Get Current page encoding is null, check the pageEncoding list, if last page encoding is available, use it.
Comment 3 Philippe Mouawad 2019-01-29 19:39:37 UTC
I think it's a duplicate of 63130, if I am wrong please reopen.

*** This bug has been marked as a duplicate of bug 63130 ***
Comment 4 Naveen Nandwani 2019-01-29 23:35:18 UTC
Scenario of this bug is similar to #63130, but root cause is different.
In this bug, JMeter does not have the encoding for the first time page POST request.

In case of #63130, encoding library is not working fine, in case of multi-byte Japanese character. Example of such character is mentioned in #63130.

Please let me know, if more details are required.
Comment 5 Naveen Nandwani 2019-02-13 11:25:57 UTC
Hi,

PR request is created for this bug, please refer below:
https://github.com/apache/jmeter/pull/441

Please let me know, if more information is required.

Regards,
Comment 6 Philippe Mouawad 2019-05-19 20:56:26 UTC
Author: pmouawad
Date: Sun May 19 20:55:31 2019
New Revision: 1859523

URL: http://svn.apache.org/viewvc?rev=1859523&view=rev
Log:
Bug 63129 - JMeter can not identify encoding during first time page submission

This closes #441
Bugzilla Id: 63129

Modified:
    jmeter/trunk/src/core/org/apache/jmeter/resources/messages.properties
    jmeter/trunk/src/core/org/apache/jmeter/resources/messages_fr.properties
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/AbstractSamplerCreator.java
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/Daemon.java
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/DefaultSamplerCreator.java
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/ProxyControl.java
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/proxy/gui/ProxyControlGui.java
    jmeter/trunk/xdocs/changes.xml
Comment 7 Philippe Mouawad 2019-05-19 20:58:04 UTC
*** Bug 63417 has been marked as a duplicate of this bug. ***