I try to record a test plan with the proxy server. My web app uses JSF 1.1. The view state is saved in HTML in hidden input control. A view state is a tree of JSF component state saved as a gzipped base64 string : Excerpt from ResponseStateManagerImpl of jsf-impl-1.1.jar ... ByteArrayOutputStream bos = new ByteArrayOutputStream(); if (compress) { zos = new GZIPOutputStream(bos); oos = new ObjectOutputStream(zos); } else { oos = new ObjectOutputStream(bos); } oos.writeObject(view.getStructure()); oos.writeObject(view.getState()); oos.close(); if (compress) { zos.close(); } bos.close(); hiddenField = " <input type=\"hidden\" name=\"" + RIConstants.FACES_VIEW + "\"" + " value=\"" + (new String(Base64.encode(bos.toByteArray()), "ISO-8859-1")) + "\" />\n "; ... sample : H4sIAAAAAAAAAO19CXgcxZV/ayRZh2UsyZLlS/YQ29gGPDrm0OGQIMuyLVY+sM3... When the current page is submitted by the browser, the proxy update the string into : H4sIAAAAAAAAAO19CXgcxZV/ayRZh2UsyZLlS/YQ29gGPDrm0OGQIMuyLVY sM3... the "+" char is missing at the end of the string !!! JSF can not deserialized the string into objects : it crashes. (UTFDataFormatException, ZipException, ...) I've remotly debugged jmeter to check when the value is altered. When Proxy class record an HTTP request every parameter of the request is always decoded. The HTTPArgument class is used to represent a request parameter. public HTTPArgument(String name, String value, boolean alreadyEncoded) { setAlwaysEncoded(true); if (alreadyEncoded) { try { name = JOrphanUtils.decode(name, "UTF-8"); value = JOrphanUtils.decode(value, "UTF-8"); } catch (UnsupportedEncodingException e) { // UTF-8 unsupported? You must be joking! log.error("UTF-8 encoding not supported!"); throw new Error(e.toString()); } } setName(name); setValue(value); setMetaData("="); } At least every parameter value is decoded with URLDecoder.decode(). My parameter value is corrupted because of this call : URLDecoder.decode("H4sIAAAAAAAAAO19CXgcxZVwz+iWZcuSJdv4HDDGds", "UTF-8") returns H4sIAAAAAAAAAO19CXgcxZVwz iWZcuSJdv4HDDGds
(In reply to comment #0) Are you using a POST or GET request to the web server ? From you description, it sounds like the problem is somewhere in the data flow from the web browser via the proxy to the Jmeter test plan. And not in the data flow from executing the test plan and sending request to the web server. Do you agree with that ? I've seen from the open bugs that there seems to be multiple problem with the proxy and handling of enconding, and I think your problem might be related.
(In reply to comment #1) > (In reply to comment #0) > > Are you using a POST or GET request to the web server ? POST. > From you description, it sounds like the problem is somewhere in the data flow > from the web browser via the proxy to the Jmeter test plan. And not in the data > flow from executing the test plan and sending request to the web server. > Do you agree with that ? yes. > > I've seen from the open bugs that there seems to be multiple problem with the > proxy and handling of enconding, and I think your problem might be related. > I've not found any but some special characters encoding problems (Japanese, ...). You can get a more complete description of my problem here : http://mail-archives.apache.org/mod_mbox/jakarta-jmeter-user/200703.mbox/browser (thread is "Parameter value corruption" )
I've looked into this now, but haven't gotten much further. If you can point us to a public web server where we can test, that would help a lot. Perhaps you know of some public sites which has demos of applications using JSF, where you and we could recreate the problem. If not, I suggest you try the following : Use the proxy to record the test plan. Then execute the test plan, and see which sample fails. Then I suggest you fix the value of that Request parameter by hand in the GUI, ie change the space to a +, and then re-execute the test to see if it works. That should work. If it still does not work, then I'm not sure if it is a Proxy decoding problem. Alternative, you could add a method in the org.apache.jmeter.protocol.http.util.TestHTTPArgument (the unit test for HTTPArgument), and write a unit test for the encoding and decoding of your problematic values, and supply that code, and we could have a look at it. Something like : public void testDecoding() throws Exception { // Test to try to recreate problem in bug 41837 // This is how the MultipartUrlConfig.addEncodedArgument method // creates and HTTPArgument for a parameter value that is // received as part of a POST request HTTPArgument arg = new HTTPArgument("test_for_bug_41837", "H4sIAAAAAAAAAO19CXgcxZV/ayRZh2UsyZLlS/YQ29gGPDrm0OGQIMuyLVY+sM3", true); if (arg.getName().equals(arg.getEncodedName()) && arg.getValue().equals(arg.getEncodedValue())) { arg.setAlwaysEncoded(false); } // The value must be encoded before it is sent to web server assertTrue(arg.isAlwaysEncoded()); } I've run that test code, and it seems to behave correct to me. Spaces as encoded as + in an URL, and then when the Proxy gets the value, it decodes the value, and put it into the HTTPSampler that is put into the test plan, because we prefer to show unencoded values in the GUI. What is the "Encode?" property value for the request parameter in the HTTP Request node that is put into the test plan ? If "Encode?" is true, then I don't think it is a problem that the "+" is shown as a " " in the GUI, because when the test is run again, it will be encoded into "+". Perhaps you could attach a test plan created by the proxy, which shows the problem ? Or you could attach a screengrab of the problematic HTTP Request in the test plan ? It would be good if you could run with the latest nightly build of jmeter. What is the "content-type" and "charset" for the web pages you are serving ? If you are using UTF-8, perhaps you could try to add "UTF-8" as the value of the "Content encoding" in the HTTP Request, you need the latest nightly build to have that field. I think you should try that first. That functionality was added as part of bug 41705. So if your application is expecting UTF8 data, but you have not set the "UTF-8" "Content encoding" on the HTTP Request, then you might get problems. I'll look more into this issue, if you can provide us with information so we get a few steps closer to solving this.
Created attachment 19750 [details] Contains 2 JSPs for testing purpose testviewid.jsp : contains 2 form with 2 different encodings. result.jsp: display the value of the submitted viewid When recording with JMeter proxy, the value encoded in first form is not corrupted while the same value encoded in second form is.
Created attachment 19751 [details] Test plan recorded by the proxy with multipart encoding The test plan shows the corrupted value '+' replaced by ' ' for the test_for_bug_41837 value.
Thanks for you long analysis. I've news ! The form encoding seems to be the cause of the corruption. You will find two useful attachments for testing purpose. When using "application/x-www-form-urlencoded" and post method, everything is fine. When using "multipart/form-data" and post method, the value is corrupted. As you can see with result.jsp, the proxy corrupts the value and then send the value to tomcat server.
Thanks for your info, I think we have solved this problem now. Analysis : Your jsp pages was helpful in analyzing this. I've added a bit of functionality to the results.jps page. I've added use of the commons-io.jar and commons-fileupload.ja, so that I can see what is posted when doing a "multipart/from-data" post. I'll attach the updated war file. First of all, if you are testing pages which are using multipart/form-data, then you should turn of the "Capture HTTP Headers" setting in the Proxy Server. See bug 27780. Bug 27780 also explains that when running the test plan after you have captured it, jmeter will issue a "application/x-www-form-urlencoded" request even though the proxy server received a multipart/form-data. But if you are a bit lucky, that should does not cause problems for you. I think the bug is in the MultipartUrlConfig class, which is handling the parsing of a multipart/form-data post request for the proxy server. If you are in a multipart/form-data post request, then the parameter values which are part of the multipart is not encoded by the browser. But the MultipartUrlConfig does the following : " } else if (parts[i].indexOf("name=") > -1) { int index = parts[i].indexOf("name=\"") + 6; String name = parts[i].substring(index, parts[i].indexOf("\"", index)); index = parts[i].indexOf("\n", index) + 2; String value = parts[i].substring(index).trim(); this.addEncodedArgument(name, value); } " So the MultipartUrlConfig assumes that parameter values which are part of a multipart/form-data has been URL encoded. I think that is the wrong assumption. I think it should do a "this.addArgument(name, value);", to add a parameter value which has not been URL encoded by the browser. I'll attach a patch to fix this. Read more on http://www.w3.org/TR/html4/interact/forms.html#h-17.13.4 And perhaps read section 6 og http://www.ietf.org/rfc/rfc2045.txt So the multipart/form-data can be encoded by specifying the "Content-Transfer-Encoding" in the multipart. But let us say that for now, the Proxy does not support that. It has been useful to do this investigation. I was already planning to fix bugs 27780, 33435, 41518 (which I think is the same as 27780), and 19128. I hope to have 27780 fixed within a couple of weeks. However, what I plan to do first, is to reorganize the "HTTP Request" GUI slightly, because I need to add an "HTTP POST" parameters section, so that the user can choose if he wants to do a "x-www-form-urlencoded" or a "multipart" post if only parameters are present. There also needs to be room in the GUI to add more files. I'm thinking about using tabs in the lower part of the GUI, one tab names "Options", one tab named "Parameters", one tab named "File uploads". That was all for now. It would be good if you could test the patch I'll add shortly, and see if things works then. And I guess Sebb will also look at the patch, and apply it if he agrees.
Created attachment 19758 [details] Suggested patch This patch makes parameter values which are part of a multipart/form-data post request treated as not url encoded, in fact not encoded at all. We assume the "default" "Content-Transfer-Encoding" has been used for the multipart/form-data. Support for different "Content-Transfer-Encoding" might come in the future.
Created attachment 19759 [details] Updated web app for testing multipart/form-data Added a bit more code to the jsp, and included commons-io.jar and commons-fileupload.jar
My updated test app is available at : http://www.kanonbra.com/testjmeter/testviewid.jsp at least for a couple of weeks. I testet recording a test plan using the patched proxy server now. I also attached a "View Results Tree" listener to the proxy server. I see that the browser/proxy has trouble sending the "multipart/form-data". I think this is because the "http headers" sent to the web server says "Content-Type: multipart/form-data;", while in fact the proxy is currently always sending HTTP Post "Content-Type: application/x-www-form-urlencoded" request. I'll look more into this now, because I guess you are not able to record much of a plan without having that fixed. But the recorded test plan is possible to play back correctly.
Created attachment 19760 [details] Updated patch, which also updates content-type header This updated patch also changes the "Content-Type" http header receives from web browser to "application/x-www-form-urlencoded" in the case where it is a multipart/form-data POST, and no file is being uploaded. With this patch, I'm able to record a test plan, and the browser shows the correct values. I'm not sure if this is the nicest patch, but I think it should be sufficient for you to record your plan, and test using jmeter, assuming that you web application allows "application/x-www-form-urlencoded". So this patch translates a multipart/form-data HTTP Post request into a HTTP Post "application/x-www-form-urlencoded" request, if no file is uploaded as part of the post request.
Thanks for the patch, which I have applied to SVN. It will be in the nightly builds after r521050
I think this bug should be closed as "Fixed". The HTTP Proxy server no longer changes values that contains "+" or other values that change when URL encoded, when it is a multipart/form-data request. The HTTP Proxy server has also been enhanced to support multipart/form-data even if files are not being uploaded, so I think all in this bug is fixed.
This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/1883