Created attachment 36685 [details] writing what I think the problem is which may be helpful to you to fix the bug. The product is not generating the HTML Dashboard report at the end of the test. I am pasting the Exception trace, Also writing what I think the problem is which may be helpful to you to fix the bug. Please see the attached file.
Hello, Can you share your jmeter.properties and user.properties and any other properties file if you have used one Also share the command line you used. Thank you
Today, 7/29/2019, Provided properties files and jmeter command line in the reply email.
(In reply to vemular from comment #2) > Today, 7/29/2019, Provided properties files and jmeter command line in the > reply email. Hello, Can you please attach the files to this ticket ? If you replied to the notification mail, it's not useful. Regards
Created attachment 36689 [details] user.properties
Created attachment 36690 [details] jmeter.properties
The Command Line I Used is "C:\apache-jmeter-5.1.1\bin\jmeter.bat" --nongui --remotestart 10.89.8.23,10.89.8.19,10.89.8.21,10.89.8.22,10.89.8.20 --testfile C:\JMeterLoadTests\SDIS_REST\SDIS_AND_QRS_REST_Sampler.jmx --logfile "C:\JMeterLoadTests\SDIS_REST\D07292019_110334\SDIS\log\20KB\SDIS.log" --reportatendofloadtests --reportoutputfolder "C:\JMeterLoadTests\SDIS_REST\D07292019_110334\SDIS\report\20KB\SDIS_75vusers" --remoteexit @@@@@@@@@@@@@@@@@@@ I am copying code chunk taken from the class of org.apache.jmeter.save.CSVSaveService of Source of Jmeter master branch(from GIT code repo). I beleive this code is unable to parse the entries of errored samples correctly. This code parses each line into a String Array. For example, if you compare a successful entry and a failed entry: Failed Entry from the report file: 2019/07/29 17:25:46.807,21034,ECMaaSRESTCreateDocSampler,500,Request Failed,10.89.8.21-Thread Group 1-2,,false,Test failed: text expected to contain /Successfully performed action/,14,0,15,15,null,0,0,0 Success Entry from report file: 2019/07/29 17:26:07.523,303,ECMaaSRESTCreateDocSampler,200,Successfully performed action: HOST_IPADDRESS =10.89.8.21,10.89.8.21-Thread Group 1-13,,true,,2539,0,15,15,null,0,0,0 The above both entries have same number of tokens. But the Jmeter code is wrongly tokenizing the failed entry. The column count and token count becoming unequal. @@@@@@@@@@@@@@@@ code from line # 988 to 1075 @@@@@@@@@@@@@@@ public static String[] csvReadFile(BufferedReader infile, char delim) throws IOException { int ch; ParserState state = ParserState.INITIAL; List<String> list = new ArrayList<>(); CharArrayWriter baos = new CharArrayWriter(200); boolean push = false; while (-1 != (ch = infile.read())) { push = false; switch (state) { case INITIAL: if (ch == QUOTING_CHAR) { state = ParserState.QUOTED; } else if (isDelimOrEOL(delim, ch)) { push = true; } else { baos.write(ch); state = ParserState.PLAIN; } break; case PLAIN: if (ch == QUOTING_CHAR) { baos.write(ch); throw new IOException( "Cannot have quote-char in plain field:[" + baos.toString() + "]"); } else if (isDelimOrEOL(delim, ch)) { push = true; state = ParserState.INITIAL; } else { baos.write(ch); } break; case QUOTED: if (ch == QUOTING_CHAR) { state = ParserState.EMBEDDEDQUOTE; } else { baos.write(ch); } break; case EMBEDDEDQUOTE: if (ch == QUOTING_CHAR) { baos.write(QUOTING_CHAR); // doubled quote => quote state = ParserState.QUOTED; } else if (isDelimOrEOL(delim, ch)) { push = true; state = ParserState.INITIAL; } else { baos.write(QUOTING_CHAR); throw new IOException( "Cannot have single quote-char in quoted field:[" + baos.toString() + "]"); } break; default: throw new IllegalStateException("Unexpected state " + state); } // switch(state) if (push) { if (ch == '\r') {// Remove following \n if present infile.mark(1); if (infile.read() != '\n') { infile.reset(); // did not find \n, put the character // back } } String s = baos.toString(); list.add(s); baos.reset(); } if ((ch == '\n' || ch == '\r') && state != ParserState.QUOTED) { break; } } // while not EOF if (ch == -1) {// EOF (or end of string) so collect any remaining data if (state == ParserState.QUOTED) { throw new IOException("Missing trailing quote-char in quoted field:[\"" + baos.toString() + "]"); } // Do we have some data, or a trailing empty field? if (baos.size() > 0 // we have some data || push // we've started a field || state == ParserState.EMBEDDEDQUOTE // Just seen "" ) { list.add(baos.toString()); } } return list.toArray(new String[list.size()]); }
Hello, your CSV file in word document uses | as separator which is not the default. How is this file generated ? You need to use the same config file for generation that was used for output. 1/ I see nowhere in user.properties or jmeter.properties that default_delimiter was set to |, so it looks to me regular that it fails. 2/ In last comment, you show other rows with ',' separator, did it fails for those data ? 3/ You say you generate report at end, what exactly do you mean ? As in your command line I don't see -e -o <report_folder> https://jmeter.apache.org/usermanual/generating-dashboard.html#report_after_load_test
Please see my notes inline. 1/ I see nowhere in user.properties or jmeter.properties that default_delimiter was set to |, so it looks to me regular that it fails. My response: [[Even with default delimiter which is comma (,) it gives the same error. The problem is not with the delimiter. Yes I changed the delimiter to default then I sent you the user.properties.]] ############## 2/ In last comment, you show other rows with ',' separator, did it fails for those data ? My response: Yes a changed from | delimiter to , delimiter. But even then I see same error of 2019-07-30 14:53:12,440 ERROR o.a.j.JMeter: Error generating dashboard: org.apache.jmeter.report.dashboard.GenerationException: Error while processing samples:Mismatch between expected number of columns:17 and columns in CSV file:4, check your jmeter.save.saveservice.* configuration or check line is complete. Even with fresh install of Jmeter I would get the error. ############# 3/ You say you generate report at end, what exactly do you mean ? As in your command line I don't see -e -o <report_folder> My response: If you see the command line I pasted in earlier comment, I used the expanded flags rather than the short flags, the flags of "--reportatendofloadtests --reportoutputfolder "C:\JMeterLoadTests\SDIS_REST\D07292019_110334\SDIS\report\20KB\SDIS_75vusers" --remoteexit" The "--reportatendofloadtests" will generate the dashboard report at end. The "--reportoutputfolder" specifies the output folder. These are same as "-e -o <report_folder>" ############ Please do Google search. I am pasting the Google search URL. You will find users are getting these errors. But they are not reporting as bug. https://www.google.com/search?safe=active&ei=V7ZAXbEcqbiCB8u4l3A&q=jmeter+%22error+while+processing+samples%3Amismatch+between+expected+number+of+columns%22&oq=jmeter+%22error+while+processing+samples%3Amismatch+between+expected+number+of+columns%22
(In reply to vemular from comment #8) > Please see my notes inline. > > 1/ I see nowhere in user.properties or jmeter.properties that > default_delimiter was set to |, so it looks to me regular that it fails. > > My response: > > [[Even with default delimiter which is comma (,) it gives the same error. > The problem is not with the delimiter. Yes I changed the delimiter to > default then I sent you the user.properties.]] > ############## > 2/ In last comment, you show other rows with ',' separator, did it fails for > those data ? > > My response: > > Yes a changed from | delimiter to , delimiter. But even then I see same > error of > > 2019-07-30 14:53:12,440 ERROR o.a.j.JMeter: Error generating dashboard: > org.apache.jmeter.report.dashboard.GenerationException: Error while > processing samples:Mismatch between expected number of columns:17 and > columns in CSV file:4, check your jmeter.save.saveservice.* configuration or > check line is complete. > > Even with fresh install of Jmeter I would get the error. > > ############# > > 3/ You say you generate report at end, what exactly do you mean ? As in your > command line I don't see > -e -o <report_folder> > > My response: > > If you see the command line I pasted in earlier comment, I used the expanded > flags rather than the short flags, the flags of "--reportatendofloadtests > --reportoutputfolder > "C: > \JMeterLoadTests\SDIS_REST\D07292019_110334\SDIS\report\20KB\SDIS_75vusers" > --remoteexit" > > The "--reportatendofloadtests" will generate the dashboard report at end. > The "--reportoutputfolder" specifies the output folder. These are same as > "-e -o <report_folder>" Indeed, I overlooked it. > > ############ > > Please do Google search. I am pasting the Google search URL. You will find > users are getting these errors. But they are not reporting as bug. > > https://www.google.com/ > search?safe=active&ei=V7ZAXbEcqbiCB8u4l3A&q=jmeter+%22error+while+processing+ > samples%3Amismatch+between+expected+number+of+columns%22&oq=jmeter+%22error+w > hile+processing+samples%3Amismatch+between+expected+number+of+columns%22 Can you just provide with the user.properties/jmeter.properties that you attached a failing csv file ? Thanks
(In reply to vemular from comment #6) . > > For example, if you compare a successful entry and a failed entry: > > Failed Entry from the report file: > > 2019/07/29 17:25:46.807,21034,ECMaaSRESTCreateDocSampler,500,Request > Failed,10.89.8.21-Thread Group 1-2,,false,Test failed: text expected to > contain /Successfully performed action/,14,0,15,15,null,0,0,0 > > Success Entry from report file: > > 2019/07/29 17:26:07.523,303,ECMaaSRESTCreateDocSampler,200,Successfully > performed action: HOST_IPADDRESS =10.89.8.21,10.89.8.21-Thread Group > 1-13,,true,,2539,0,15,15,null,0,0,0 > > > > The above both entries have same number of tokens. But the Jmeter code is > wrongly tokenizing the failed entry. The column count and token count > becoming unequal. It looks to me as you are trying to send data inside the 'message' (Successfully > performed action: HOST_IPADDRESS =10.89.8.21,10.89.8.21-Thread Group > 1-13,,true,). In the case of an exception those entries are MISSING (You could set an error message with the missing entries yourself though). As you are using the same delimiter for /your/ message as the data for JMeter, JMeters parser will fail.
Created attachment 36693 [details] Please read this first Please read this first. attaching more files.
Created attachment 36694 [details] JTL report entries
Created attachment 36695 [details] Jmeter log
Created attachment 36696 [details] user.properties
Created attachment 36697 [details] jmeter.properties
(In reply to vemular from comment #11) > Created attachment 36693 [details] > Please read this first > > Please read this first. attaching more files. Why can't you post your answer as text inline? It is really annoying and maybe even dangerous to have to open a word doc.
And the best thing is, you haven't answered -- in the word doc -- my question about whether you mix your csv message with JMeter's one.
(In reply to Felix Schumacher from comment #16) > (In reply to vemular from comment #11) > > Created attachment 36693 [details] > > Please read this first > > > > Please read this first. attaching more files. > > Why can't you post your answer as text inline? It is really annoying and > maybe even dangerous to have to open a word doc. A word doc allows to have pictures, different fonts for different topics, etc... The plain text does not have that luxury. Pictures make you understand well.
(In reply to vemular from comment #18) > (In reply to Felix Schumacher from comment #16) > > (In reply to vemular from comment #11) > > > Created attachment 36693 [details] > > > Please read this first > > > > > > Please read this first. attaching more files. > > > > Why can't you post your answer as text inline? It is really annoying and > > maybe even dangerous to have to open a word doc. > > A word doc allows to have pictures, different fonts for different topics, > etc... The plain text does not have that luxury. > Pictures make you understand well. It didn't help me at all. * I could not open it on my phone. * There was no new information available in the text (at least I did not find them). * the word document will probably not be indexed by search machines * bugzilla will not search inside it as it will do for text messages as this one * Pictures can be attached and shown directly by bugzilla * word documents can contain malicious content So, please don't attach word documents, if they are not needed.
(In reply to Felix Schumacher from comment #17) > And the best thing is, you haven't answered -- in the word doc -- my > question about whether you mix your csv message with JMeter's one. I do not think I understood your question completely. When you develop a Java Sampler there is API to set certain things into result, for example sampleResult.setResponseMessage(). For failed tests the response message is different. The response assertion does not pass when the expected responseMessage is different than the actual responseMessage. In case of assertion failure, the Jmeter itself putting this into the report file -> "Test failed: text expected to contain /Test is OK/". This string is not from my code. This is output by Jmeter into the report. Did you try to find why the error is -> "Mismatch between expected number of columns:17 and columns in CSV file:3," Every record has same 17 columns. Why does it say 3 columns? Here I am understanding "CSV file" in the error message as the JTL report file.
(In reply to vemular from comment #20) > (In reply to Felix Schumacher from comment #17) > > And the best thing is, you haven't answered -- in the word doc -- my > > question about whether you mix your csv message with JMeter's one. > > I do not think I understood your question completely. That is probably since I misread the example lines you gave. I thought that you passed extra values in the successful messages that contained the separator used for the rest of the line. Now on closer inspection it turns out that I have misread the lines. The data is not problematic. > > When you develop a Java Sampler there is API to set certain things into > result, for example sampleResult.setResponseMessage(). > > For failed tests the response message is different. The response assertion > does not pass when the expected responseMessage is different than the actual > responseMessage. > > In case of assertion failure, the Jmeter itself putting this into the report > file -> "Test failed: text expected to contain /Test is OK/". This string > is not from my code. This is output by Jmeter into the report. Yes. No problem here. > > Did you try to find why the error is -> "Mismatch between expected number of > columns:17 and columns in CSV file:3," > > Every record has same 17 columns. Why does it say 3 columns? > > Here I am understanding "CSV file" in the error message as the JTL report > file. That might be caused by a corrupted jtl file. Have you inspected the jtl file? Is it complete? The jtl file you attached shows no problem in my test report generation.
(In reply to Felix Schumacher from comment #21) > (In reply to vemular from comment #20) > > (In reply to Felix Schumacher from comment #17) > > > And the best thing is, you haven't answered -- in the word doc -- my > > > question about whether you mix your csv message with JMeter's one. > > > > I do not think I understood your question completely. > > That is probably since I misread the example lines you gave. I thought that > you passed extra values in the successful messages that contained the > separator used for the rest of the line. > > Now on closer inspection it turns out that I have misread the lines. The > data is not problematic. > > > > > When you develop a Java Sampler there is API to set certain things into > > result, for example sampleResult.setResponseMessage(). > > > > For failed tests the response message is different. The response assertion > > does not pass when the expected responseMessage is different than the actual > > responseMessage. > > > > In case of assertion failure, the Jmeter itself putting this into the report > > file -> "Test failed: text expected to contain /Test is OK/". This string > > is not from my code. This is output by Jmeter into the report. > > Yes. No problem here. > > > > > Did you try to find why the error is -> "Mismatch between expected number of > > columns:17 and columns in CSV file:3," > > > > Every record has same 17 columns. Why does it say 3 columns? > > > > Here I am understanding "CSV file" in the error message as the JTL report > > file. > > That might be caused by a corrupted jtl file. Have you inspected the jtl > file? Is it complete? So far I ran the test at least 20 times. And each time the JTL is generated successfully. Every time it is the same error for dashboard generation - "Mismatch between expected number blah blah...." There is no corruption of JTL file. > > The jtl file you attached shows no problem in my test report generation. Yes the JTL report is generated successfully. There is no issue in generating the JTL file. The next process after the generation of JTL process is generating the Dashboard report. The Dashboard report generation is failing.
in repository https://gitbox.apache.org/repos/asf/jmeter.git commit 5f41f757ba6cfaee6d95892f1c15eff1d56d638b AuthorDate: Thu Aug 1 20:49:56 2019 +0200 Log more details on broken CSV files When corrupt CSV files are read by ReportGenerator it is often not clear enough where the broken data in the file is. Give more details in log messages and exceptions. Inspired by Bugzilla Id: 63614 --- .../apache/jmeter/report/core/CsvSampleReader.java | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) commit 2d8e630039afd439ca852cc705cc6f666184e899 AuthorDate: Thu Aug 1 20:53:45 2019 +0200 Spacepolice Don't squash the exception cause directly to our own message. --- src/core/org/apache/jmeter/report/dashboard/ReportGenerator.java | 2 +- test/src/org/apache/jmeter/gui/action/HtmlReportGeneratorSpec.groovy | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
(In reply to vemular from comment #22) > (In reply to Felix Schumacher from comment #21) > > (In reply to vemular from comment #20) > > > (In reply to Felix Schumacher from comment #17) > > > > And the best thing is, you haven't answered -- in the word doc -- my > > > > question about whether you mix your csv message with JMeter's one. > > > > > > I do not think I understood your question completely. > > > > That is probably since I misread the example lines you gave. I thought that > > you passed extra values in the successful messages that contained the > > separator used for the rest of the line. > > > > Now on closer inspection it turns out that I have misread the lines. The > > data is not problematic. > > > > > > > > When you develop a Java Sampler there is API to set certain things into > > > result, for example sampleResult.setResponseMessage(). > > > > > > For failed tests the response message is different. The response assertion > > > does not pass when the expected responseMessage is different than the actual > > > responseMessage. > > > > > > In case of assertion failure, the Jmeter itself putting this into the report > > > file -> "Test failed: text expected to contain /Test is OK/". This string > > > is not from my code. This is output by Jmeter into the report. > > > > Yes. No problem here. > > > > > > > > Did you try to find why the error is -> "Mismatch between expected number of > > > columns:17 and columns in CSV file:3," > > > > > > Every record has same 17 columns. Why does it say 3 columns? > > > > > > Here I am understanding "CSV file" in the error message as the JTL report > > > file. > > > > That might be caused by a corrupted jtl file. Have you inspected the jtl > > file? Is it complete? > > So far I ran the test at least 20 times. And each time the JTL is generated > successfully. Every time it is the same error for dashboard generation - > "Mismatch between expected number blah blah...." Have you checked the jtl file? > > There is no corruption of JTL file. Sure? > > > > > The jtl file you attached shows no problem in my test report generation. > > Yes the JTL report is generated successfully. There is no issue in > generating the JTL file. The next process after the generation of JTL > process is generating the Dashboard report. The Dashboard report generation > is failing. The attached jtl works. You might want to attach the jtl file that is not working or send it to me directly if there is private data in it, that you don't want to have posted publicly. I have added more log detail to JMeter which should be available in the next nightly. It will give info about the line and the file with which JMeter has a problem.
(In reply to Felix Schumacher from comment #24) > (In reply to vemular from comment #22) > > (In reply to Felix Schumacher from comment #21) > > > (In reply to vemular from comment #20) > > > > (In reply to Felix Schumacher from comment #17) > > > > > And the best thing is, you haven't answered -- in the word doc -- my > > > > > question about whether you mix your csv message with JMeter's one. > > > > > > > > I do not think I understood your question completely. > > > > > > That is probably since I misread the example lines you gave. I thought that > > > you passed extra values in the successful messages that contained the > > > separator used for the rest of the line. > > > > > > Now on closer inspection it turns out that I have misread the lines. The > > > data is not problematic. > > > > > > > > > > > When you develop a Java Sampler there is API to set certain things into > > > > result, for example sampleResult.setResponseMessage(). > > > > > > > > For failed tests the response message is different. The response assertion > > > > does not pass when the expected responseMessage is different than the actual > > > > responseMessage. > > > > > > > > In case of assertion failure, the Jmeter itself putting this into the report > > > > file -> "Test failed: text expected to contain /Test is OK/". This string > > > > is not from my code. This is output by Jmeter into the report. > > > > > > Yes. No problem here. > > > > > > > > > > > Did you try to find why the error is -> "Mismatch between expected number of > > > > columns:17 and columns in CSV file:3," > > > > > > > > Every record has same 17 columns. Why does it say 3 columns? > > > > > > > > Here I am understanding "CSV file" in the error message as the JTL report > > > > file. > > > > > > That might be caused by a corrupted jtl file. Have you inspected the jtl > > > file? Is it complete? > > > > So far I ran the test at least 20 times. And each time the JTL is generated > > successfully. Every time it is the same error for dashboard generation - > > "Mismatch between expected number blah blah...." > > Have you checked the jtl file? > > > > > There is no corruption of JTL file. > > Sure? > > > > > > > > > The jtl file you attached shows no problem in my test report generation. > > > > Yes the JTL report is generated successfully. There is no issue in > > generating the JTL file. The next process after the generation of JTL > > process is generating the Dashboard report. The Dashboard report generation > > is failing. > > The attached jtl works. You might want to attach the jtl file that is not > working or send it to me directly if there is private data in it, that you > don't want to have posted publicly. > I have several JTL files. I zipped up all those. The path to them is D<MMDDYYYY_HHmmss>\SDIS\log\20KB. I am going to send to your email felix.schumacher@internetallee.de . The zipped up file is of 7MB. But each JTL file is around 30MB to 50MB. > I have added more log detail to JMeter which should be available in the next > nightly. It will give info about the line and the file with which JMeter has > a problem. Thank you for nightly. I will try to use it.
Only one of those seven jtl files shows a problem. With the more detailed error message you get: An error occurred: Error while processing samples: Mismatch between expected number of columns:17 and columns in CSV file:3, check your jmeter.save.saveservice.* configuration or check if line 58376 in '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log' is complete And in jmeter.log you can find: WARN o.a.j.r.c.CsvSampleReader: Short CSV read around line 58376 of file '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log'. Could only read 3 elements of 17 expected. Data is [2019/07/30 13:43:27.461, 121, ECMaaSRESTCreateDocSampl] And if you have a look at line 58376 (which is the last line in that file), you see: 2019/07/30 13:43:27.461,121,ECMaaSRESTCreateDocSampl Which seems to be a broken file to me.
(In reply to Felix Schumacher from comment #26) > Only one of those seven jtl files shows a problem. With the more detailed > error message you get: > > An error occurred: Error while processing samples: Mismatch between expected > number of columns:17 and columns in CSV file:3, check your > jmeter.save.saveservice.* configuration or check if line 58376 in > '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log' is complete > > And in jmeter.log you can find: > > WARN o.a.j.r.c.CsvSampleReader: Short CSV read around line 58376 of file > '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log'. Could only read 3 elements > of 17 expected. Data is [2019/07/30 13:43:27.461, 121, > ECMaaSRESTCreateDocSampl] > > And if you have a look at line 58376 (which is the last line in that file), > you see: > > 2019/07/30 13:43:27.461,121,ECMaaSRESTCreateDocSampl > > Which seems to be a broken file to me. Hello Today(Aug 2, 2019) I ran another test for 15 minutes. I used the JMeter jars that are complied from the latest source from GIT as of yerterday. The Jmeter log shows this below error. The error says there is a problem with line # 281268. I checked the line # 281268 in JTL file. The line # 281268 appears complete. All lines above & below this line also appear complete. I beleive the Dashboard generation process may be triggering in before all clients completely send and data to Master (in distributed load test). I beleive before the JTL file is completely written the Dashbord generation process is triggering. The dashboard generation process should be started only after that all clients are sent the "test completed" message AND the JTL file is completely written and it is closed. This is my guess. Also I beleive as the JTL file size increases Jmeter could not handle large JTL files. This is my assumption. I think the communication between the main thread and the thread that generates the DashBoard generation lack something which is necessary. One more thing is the timestamp in line #281268 is 2019/08/02 12:02:59.165,102. The timestamp of last line(# 281307) is 2019/08/02 12:02:59.644,3120. Probably the JTL file is being read before it is completely written and closed. I am sending the JTL file (44MB) to your email felix.schumacher@internetallee.de. 2019-08-02 12:03:30,530 ERROR o.a.j.JMeter: Error generating dashboard: org.apache.jmeter.report.dashboard.GenerationException: Error while processing samples: Mismatch between expected number of columns:17 and columns in CSV file:1, check your jmeter.save.saveservice.* configuration or check if line 281268 in 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is complete org.apache.jmeter.report.dashboard.GenerationException: Error while processing samples: Mismatch between expected number of columns:17 and columns in CSV file:1, check your jmeter.save.saveservice.* configuration or check if line 281268 in 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is complete at org.apache.jmeter.report.dashboard.ReportGenerator.generate(ReportGenerator.java:247) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] at org.apache.jmeter.JMeter$ListenToTest.generateReport(JMeter.java:1326) [ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] at org.apache.jmeter.JMeter$ListenToTest.run(JMeter.java:1310) [ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: org.apache.jmeter.report.core.SampleException: Mismatch between expected number of columns:17 and columns in CSV file:1, check your jmeter.save.saveservice.* configuration or check if line 281268 in 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is complete at org.apache.jmeter.report.core.CsvSampleReader.assertCorrectColumns(CsvSampleReader.java:205) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] at org.apache.jmeter.report.core.CsvSampleReader.nextSample(CsvSampleReader.java:189) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] at org.apache.jmeter.report.core.CsvSampleReader.readSample(CsvSampleReader.java:217) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] at org.apache.jmeter.report.processor.CsvFileSampleSource.produce(CsvFileSampleSource.java:180) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] at org.apache.jmeter.report.processor.CsvFileSampleSource.run(CsvFileSampleSource.java:238) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] at org.apache.jmeter.report.dashboard.ReportGenerator.generate(ReportGenerator.java:245) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] ... 3 more
(In reply to vemular from comment #27) > (In reply to Felix Schumacher from comment #26) > > Only one of those seven jtl files shows a problem. With the more detailed > > error message you get: > > > > An error occurred: Error while processing samples: Mismatch between expected > > number of columns:17 and columns in CSV file:3, check your > > jmeter.save.saveservice.* configuration or check if line 58376 in > > '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log' is complete > > > > And in jmeter.log you can find: > > > > WARN o.a.j.r.c.CsvSampleReader: Short CSV read around line 58376 of file > > '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log'. Could only read 3 elements > > of 17 expected. Data is [2019/07/30 13:43:27.461, 121, > > ECMaaSRESTCreateDocSampl] > > > > And if you have a look at line 58376 (which is the last line in that file), > > you see: > > > > 2019/07/30 13:43:27.461,121,ECMaaSRESTCreateDocSampl > > > > Which seems to be a broken file to me. > > Hello > > Today(Aug 2, 2019) I ran another test for 15 minutes. I used the JMeter jars > that are complied from the latest source from GIT as of yerterday. > > The Jmeter log shows this below error. The error says there is a problem > with line # 281268. I checked the line # 281268 in JTL file. The line # > 281268 appears complete. All lines above & below this line also appear > complete. > I beleive the Dashboard generation process may be triggering in before all > clients completely send and data to Master (in distributed load test). I > beleive before the JTL file is completely written the Dashbord generation > process is triggering. The dashboard generation process should be started > only after that all clients are sent the "test completed" message AND the > JTL file is completely written and it is closed. This is my guess. > Also I beleive as the JTL file size increases Jmeter could not handle large > JTL files. This is my assumption. > I think the communication between the main thread and the thread that > generates the DashBoard generation lack something which is necessary. > > One more thing is the timestamp in line #281268 is 2019/08/02 > 12:02:59.165,102. The timestamp of last line(# 281307) is 2019/08/02 > 12:02:59.644,3120. Probably the JTL file is being read before it is > completely written and closed. > > > I am sending the JTL file (44MB) to your email > felix.schumacher@internetallee.de. > > 2019-08-02 12:03:30,530 ERROR o.a.j.JMeter: Error generating dashboard: > org.apache.jmeter.report.dashboard.GenerationException: Error while > processing samples: Mismatch between expected number of columns:17 and > columns in CSV file:1, check your jmeter.save.saveservice.* configuration or > check if line 281268 in > 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is > complete > org.apache.jmeter.report.dashboard.GenerationException: Error while > processing samples: Mismatch between expected number of columns:17 and > columns in CSV file:1, check your jmeter.save.saveservice.* configuration or > check if line 281268 in > 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is > complete > at > org.apache.jmeter.report.dashboard.ReportGenerator.generate(ReportGenerator. > java:247) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] > at org.apache.jmeter.JMeter$ListenToTest.generateReport(JMeter.java:1326) > [ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] > at org.apache.jmeter.JMeter$ListenToTest.run(JMeter.java:1310) > [ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] > Caused by: org.apache.jmeter.report.core.SampleException: Mismatch between > expected number of columns:17 and columns in CSV file:1, check your > jmeter.save.saveservice.* configuration or check if line 281268 in > 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is > complete > at > org.apache.jmeter.report.core.CsvSampleReader. > assertCorrectColumns(CsvSampleReader.java:205) > ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] > at > org.apache.jmeter.report.core.CsvSampleReader.nextSample(CsvSampleReader. > java:189) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] > at > org.apache.jmeter.report.core.CsvSampleReader.readSample(CsvSampleReader. > java:217) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] > at > org.apache.jmeter.report.processor.CsvFileSampleSource. > produce(CsvFileSampleSource.java:180) > ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] > at > org.apache.jmeter.report.processor.CsvFileSampleSource. > run(CsvFileSampleSource.java:238) > ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] > at > org.apache.jmeter.report.dashboard.ReportGenerator.generate(ReportGenerator. > java:245) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802] > ... 3 more I hope you noted my message in comment 27. Now I am no more generating the dashboard in the same command as the main test, but running the dashboard generation as a separate command. That is I am not going to use the flag of "--reportatendofloadtests" in the command. I would think to post this message in user forums who are Jmeter users so they also do the tests same way.
Unless bug is confirmed please don't post anything. Can you provide a new set of data showing there is a bug, last time, it appeared there was no problem. Can you provide also full jmeter.log in INFO level of all injectors so that we check your hypothesis.
(In reply to UbikLoadPack support from comment #29) > Unless bug is confirmed please don't post anything. > I am not going to post anything myself. > Can you provide a new set of data showing there is a bug, last time, it > appeared there was no problem. I provided the JTL file to Felix. The Exception message says the line # 281268 did not have full columns. But when I checked the file, the line # 281268, I find this line and subsequent lines are completely written. I strongly believe that the dashboard generation thread is starting before the JTL file is completely written and the file is CLOSED. I would think the dashboard generation thread should obtain exclusive lock on JTL file before it starts reading. > Can you provide also full jmeter.log in INFO level of all injectors so that > we check your hypothesis. I am using five Windows Server 2016 VM instances to run this load test using Master/Slaves combination. The first host contains both Master & a slave. Other four hosts are slaves only. I have set 20 users for each Slave. So there are total simultaneous of 100 users (20 X 5 slaves). Hope this helps. Probably the Dashboard report generation thread is starting as soon as the test duration is completed. But the slaves might have started a test at the verge of completion of test duration. These tests may continue to execute for a few seconds/minutes even after the test duration is completed. So the dashboard generation thread has to start only after the JTL file is closed.
Also I would like to ask this question. Is there any property or properties combination that will ensure that the dashboard generation is started only after the JTL file is complete and is closed?
(In reply to vemular from comment #31) > Also I would like to ask this question. Is there any property or properties > combination that will ensure that the dashboard generation is started only > after the JTL file is complete and is closed? Hello, Generation is started at end of load test (including slaves ending) after completion of JTL file. If it's not happening this way it is a bug. So there is no property for that, as it is expected to work the right way.
The second file vermular posted to me is complete and contains no error.
Interesting would probably be to have the exact time, when JMeter thinks it ended the test -- and thus closes the jtl file -- and the last entries in the jtl file. Are there entries with timestamps older than the ending time. Most probably it is useful to attach a complete log and jtl file as noted already. It might even be helpful to know a bit more about the test you are trying to run. Why do you think that it is not ending at the scheduled time? Can you take threaddumps shortly after the test run should have ended, but did not?
(In reply to Felix Schumacher from comment #34) > Interesting would probably be to have the exact time, when JMeter thinks it > ended the test -- and thus closes the jtl file -- and the last entries in > the jtl file. Are there entries with timestamps older than the ending time. > > Most probably it is useful to attach a complete log and jtl file as noted > already. > > It might even be helpful to know a bit more about the test you are trying to > run. Why do you think that it is not ending at the scheduled time? Can you > take threaddumps shortly after the test run should have ended, but did not? Assume that the load test is set to run for 600 seconds. The threads continue to send requests until 599th second. The Requests that started at 599th second would continue to run until it gets a response. The threads do not abruptly end those reauests. After 600 seconds there will be no fresh requests but the already started requests continue to run. The server, to which the requests are sent, could take much time (the server maybe taking few seconds or minutes based on load, or based on nature of process, or based on the http timeouts). So the Master Jmeter can not guess when the slaves will be done. The Master JMeter has to wait for two things before it starts working for dashboard report.. First is to receive the message of "I am done with all requests" from all slaves. Second is the JTL file should be closed. I would think to obtain the exclusive lock on the file for reading & writing. I guess the thread(T1) that is writing into JTL File and the thread(T2) that is generating dashboard report are different threads. Both the threads may have to obtain exclusive locks before reading or writing thus ensuring no more than one thread deals with the JTL file at the same time. I am going to send the JTL file & Jmeter.log files(with INFO level) to email.
pmouawad@apache.org 10:02 PM (0 minutes ago) to commits@jmeter.apache.org This is an automated email from the ASF dual-hosted git repository. pmouawad pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/jmeter.git The following commit(s) were added to refs/heads/master by this push: new 105a399 This commit fixes bugs 63614 and 63723 Bug 63614 - Distributed testing: Unable to generate Dashboard report at end of load test 105a399 is described below commit 105a3999f44c73200fb08c249e120937840ee799 Author: pmouawad <p.mouawad@ubik-ingenierie.com> AuthorDate: Mon Sep 30 22:02:20 2019 +0200 This commit fixes bugs 63614 and 63723 Bug 63614 - Distributed testing: Unable to generate Dashboard report at end of load test Bug 63723 - JMeter master ends distributed test though some threads still are active --- .../src/main/java/org/apache/jmeter/JMeter.java | 136 +++++++++++---------- .../apache/jmeter/engine/ClientJMeterEngine.java | 5 + .../apache/jmeter/engine/DistributedRunner.java | 27 +++- xdocs/changes.xml | 2 + 4 files changed, 99 insertions(+), 71 deletions(-)
*** Bug 63673 has been marked as a duplicate of this bug. ***
This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/5125