Bug 63130 - Jmeter is unable to decode Shift_JIS characters.
Summary: Jmeter is unable to decode Shift_JIS characters.
Status: RESOLVED DUPLICATE of bug 64075
Alias: None
Product: JMeter
Classification: Unclassified
Component: HTTP (show other bugs)
Version: 5.0
Hardware: PC All
: P1 critical (vote)
Target Milestone: JMETER_5.1.1
Assignee: JMeter issues mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-01-29 18:16 UTC by Naveen Nandwani
Modified: 2020-04-26 16:48 UTC (History)
1 user (show)



Attachments
Application to reproduce the scenario. (1010 bytes, application/zip)
2019-01-29 18:16 UTC, Naveen Nandwani
Details
Test Plan to reproduce the issue. (4.37 KB, application/xml)
2019-01-29 18:17 UTC, Naveen Nandwani
Details
Fix patch in one function. (2.29 KB, patch)
2019-01-29 18:20 UTC, Naveen Nandwani
Details | Diff
Patch to fix the problem. I will raise PR request separately. (2.07 KB, patch)
2019-01-29 18:22 UTC, Naveen Nandwani
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Naveen Nandwani 2019-01-29 18:16:50 UTC
Created attachment 36404 [details]
Application to reproduce the scenario.

Overview: 
JMeter is unable to decode Shift_JIS (Multi-Byte) characters and resulting garbage value. When we post the Shift_JIS  characters from browser using JMeter HTTP(s) Test Script Recorder, we get garbage value for Shift_JIS characters in response.
Below is the example of request and response:
		  
Request content:  初期値大阪市 
Response content: 初期?l大阪?s

Impact of bug:

1. When End User post Shift_JIS characters using JMeter proxy, get garbage value in response.
2. Shift_JIS (Shift Japanese Industrial Standards) encoding is used by many Japanese company So whenever Japanese engineer uses JMeter for their performance testing they did not get actual result.

Setup 
1. Deploy Application "jmeter_issue_Encoding_Library"(extract in apache-tomcat-9.0.12\webapps) in Tomcat and start the tomcat(execute "apache-tomcat-9.0.12\bin\startup.bat" .
2. Open the browser and change Proxy settings -> LAN Setting and set Address(eg: localhost) and port(e.g.8888).
3. Run JMeter
	3.1 click on file menu-> open ->open the attached "Test Plan.jmx" file.
	3.2 Set the same port number in “HTTP(s) Test Script Recorder” and in browser’s Network LAN Settings.
	3.3 Go to JMeter HTTP(s) Test Script Recorder -> Click Start -> Ok
	

Execution
1. Go to browser , open application (e.g. http://localhost:portNumber/jmeter_issue_Encoding_Library/index.jsp)

2. Click on Submit Button.
 
3. View result in browser.
 
Expected Result: 初期大阪
Actual Result: ?ソス?ソス?ソス?ソス?ソスl?ソス?ソス?ソスs

//Refer ticket #63129 (reported for the same issue)

4. click on "Back to previous page" and click on "Submit" button again.
5. View the result in browser, this time getting 2 characters as garbage.

Expected Result: 初期大阪
Actual Result: 初期?l大阪?s
Comment 1 Naveen Nandwani 2019-01-29 18:17:14 UTC
Created attachment 36405 [details]
Test Plan to reproduce the issue.
Comment 2 Naveen Nandwani 2019-01-29 18:20:34 UTC
Created attachment 36406 [details]
Fix patch in one function.

Cause of Bug:
JMeter uses java.net.URLDecoder class for URL encoding.
Its drawback : This library can not encode Shift_JIS correctly because of different specific interpretation between Java and IE.
Because of browser specification, char code is zipped.
Browser encode Japanese text as following.
初期値大阪市 ⇒ %8F%89%8A%FA%92l%91%E5%8D%E3%8Es
初 = %8F%89
期 = %8A%FA
値 = %92l      ← % is lack
大 = %91%E5
阪 = %8D%E3
市 = %8Es     ← % is lack

Solution:
JMeter should use org.apache.commons.codec.net.URLCodec class for URL encoding.
Refer to attached HTTPArgument.diff file for changes in library.
Comment 3 Naveen Nandwani 2019-01-29 18:22:18 UTC
Created attachment 36407 [details]
Patch to fix the problem. I will raise PR request separately.

Fix the issue in HTTPHC4Impl.java as well. 
Please find attached the patch.
Comment 4 Philippe Mouawad 2019-01-29 19:39:37 UTC
*** Bug 63129 has been marked as a duplicate of this bug. ***
Comment 5 Philippe Mouawad 2019-02-04 21:48:23 UTC
Hello,
Can you raise a PR following:
http://jmeter.apache.org/building.html

I am afraid this might be breaking many tests, but it's worth checking.

Regards
Comment 6 Naveen Nandwani 2019-02-07 06:49:49 UTC
(In reply to Philippe Mouawad from comment #5)
> Hello,
> Can you raise a PR following:
> http://jmeter.apache.org/building.html
> 
> I am afraid this might be breaking many tests, but it's worth checking.
> 
> Regards

Hello,

Thanks for considering this bug and patch.
We are working on creating PR request and will share in next week.

Regards,
Comment 7 Naveen Nandwani 2019-02-13 11:24:34 UTC
Hi,

PR Request is raised. Please refer to below:

https://github.com/apache/jmeter/pull/440

Please let me know, if more information is required.

Regards,
Comment 8 Philippe Mouawad 2020-04-26 16:48:17 UTC

*** This bug has been marked as a duplicate of bug 64075 ***