Bug 63614 - Distributed testing: Unable to generate Dashboard report at end of load test
Summary: Distributed testing: Unable to generate Dashboard report at end of load test
Status: RESOLVED FIXED
Alias: None
Product: JMeter
Classification: Unclassified
Component: Main (show other bugs)
Version: 5.1.1
Hardware: All All
: P2 major (vote)
Target Milestone: JMETER_5.2
Assignee: JMeter issues mailing list
URL:
Keywords: FixedInTrunk
Depends on:
Blocks:
 
Reported: 2019-07-26 18:52 UTC by vemular
Modified: 2019-09-30 20:03 UTC (History)
1 user (show)



Attachments
writing what I think the problem is which may be helpful to you to fix the bug. (13.69 KB, application/vnd.openxmlformats-officedocument.wordprocessingml.document)
2019-07-26 18:52 UTC, vemular
Details
user.properties (11.09 KB, text/plain)
2019-07-30 16:59 UTC, vemular
Details
jmeter.properties (56.73 KB, text/x-matlab)
2019-07-30 16:59 UTC, vemular
Details
Please read this first (94.20 KB, application/vnd.openxmlformats-officedocument.wordprocessingml.document)
2019-07-31 21:47 UTC, vemular
Details
JTL report entries (8.99 KB, text/plain)
2019-07-31 21:47 UTC, vemular
Details
Jmeter log (1.80 KB, text/plain)
2019-07-31 21:48 UTC, vemular
Details
user.properties (19.24 KB, text/plain)
2019-07-31 21:48 UTC, vemular
Details
jmeter.properties (56.73 KB, text/x-matlab)
2019-07-31 21:49 UTC, vemular
Details

Note You need to log in before you can comment on or make changes to this bug.
Description vemular 2019-07-26 18:52:12 UTC
Created attachment 36685 [details]
writing what I think the problem is which may be helpful to you to fix the bug.

The product is not generating the HTML Dashboard report at the end of the test. I am pasting the Exception trace, Also writing what I think the problem is which may be helpful to you to fix the bug.

Please see the attached file.
Comment 1 UbikLoadPack support 2019-07-26 19:42:14 UTC
Hello,
Can you share your jmeter.properties and user.properties and any other properties file if you have used one
Also share the command line you used.

Thank you
Comment 2 vemular 2019-07-29 15:39:31 UTC
Today, 7/29/2019, Provided properties files and jmeter command line in the reply email.
Comment 3 UbikLoadPack support 2019-07-30 13:57:49 UTC
(In reply to vemular from comment #2)
> Today, 7/29/2019, Provided properties files and jmeter command line in the
> reply email.

Hello,
Can you please attach the files to this ticket ?
If you replied to the notification mail, it's not useful.

Regards
Comment 4 vemular 2019-07-30 16:59:02 UTC
Created attachment 36689 [details]
user.properties
Comment 5 vemular 2019-07-30 16:59:38 UTC
Created attachment 36690 [details]
jmeter.properties
Comment 6 vemular 2019-07-30 17:13:01 UTC
The Command Line I Used is

"C:\apache-jmeter-5.1.1\bin\jmeter.bat" --nongui  --remotestart  10.89.8.23,10.89.8.19,10.89.8.21,10.89.8.22,10.89.8.20   --testfile C:\JMeterLoadTests\SDIS_REST\SDIS_AND_QRS_REST_Sampler.jmx  --logfile "C:\JMeterLoadTests\SDIS_REST\D07292019_110334\SDIS\log\20KB\SDIS.log"  --reportatendofloadtests  --reportoutputfolder "C:\JMeterLoadTests\SDIS_REST\D07292019_110334\SDIS\report\20KB\SDIS_75vusers"    --remoteexit


@@@@@@@@@@@@@@@@@@@
I am copying code chunk taken from the class of org.apache.jmeter.save.CSVSaveService of Source of Jmeter master branch(from GIT code repo). I beleive this code is unable to parse the entries of errored samples correctly. This code parses each line into a String Array.

For example, if you compare a successful entry and a failed entry:

Failed Entry from the report file:

2019/07/29 17:25:46.807,21034,ECMaaSRESTCreateDocSampler,500,Request Failed,10.89.8.21-Thread Group 1-2,,false,Test failed: text expected to contain /Successfully performed action/,14,0,15,15,null,0,0,0

Success Entry from report file:

2019/07/29 17:26:07.523,303,ECMaaSRESTCreateDocSampler,200,Successfully performed action:  HOST_IPADDRESS =10.89.8.21,10.89.8.21-Thread Group 1-13,,true,,2539,0,15,15,null,0,0,0



The above both entries have same number of tokens. But the Jmeter code is wrongly tokenizing the failed entry. The column count and token count becoming unequal.


@@@@@@@@@@@@@@@@ code from line # 988 to 1075 @@@@@@@@@@@@@@@

   public static String[] csvReadFile(BufferedReader infile, char delim)
            throws IOException {
        int ch;
        ParserState state = ParserState.INITIAL;
        List<String> list = new ArrayList<>();
        CharArrayWriter baos = new CharArrayWriter(200);
        boolean push = false;
        while (-1 != (ch = infile.read())) {
            push = false;
            switch (state) {
            case INITIAL:
                if (ch == QUOTING_CHAR) {
                    state = ParserState.QUOTED;
                } else if (isDelimOrEOL(delim, ch)) {
                    push = true;
                } else {
                    baos.write(ch);
                    state = ParserState.PLAIN;
                }
                break;
            case PLAIN:
                if (ch == QUOTING_CHAR) {
                    baos.write(ch);
                    throw new IOException(
                            "Cannot have quote-char in plain field:["
                                    + baos.toString() + "]");
                } else if (isDelimOrEOL(delim, ch)) {
                    push = true;
                    state = ParserState.INITIAL;
                } else {
                    baos.write(ch);
                }
                break;
            case QUOTED:
                if (ch == QUOTING_CHAR) {
                    state = ParserState.EMBEDDEDQUOTE;
                } else {
                    baos.write(ch);
                }
                break;
            case EMBEDDEDQUOTE:
                if (ch == QUOTING_CHAR) {
                    baos.write(QUOTING_CHAR); // doubled quote => quote
                    state = ParserState.QUOTED;
                } else if (isDelimOrEOL(delim, ch)) {
                    push = true;
                    state = ParserState.INITIAL;
                } else {
                    baos.write(QUOTING_CHAR);
                    throw new IOException(
                            "Cannot have single quote-char in quoted field:["
                                    + baos.toString() + "]");
                }
                break;
            default:
                throw new IllegalStateException("Unexpected state " + state);
            } // switch(state)
            if (push) {
                if (ch == '\r') {// Remove following \n if present
                    infile.mark(1);
                    if (infile.read() != '\n') {
                        infile.reset(); // did not find \n, put the character
                                        // back
                    }
                }
                String s = baos.toString();
                list.add(s);
                baos.reset();
            }
            if ((ch == '\n' || ch == '\r') && state != ParserState.QUOTED) {
                break;
            }
        } // while not EOF
        if (ch == -1) {// EOF (or end of string) so collect any remaining data
            if (state == ParserState.QUOTED) {
                throw new IOException("Missing trailing quote-char in quoted field:[\""
                        + baos.toString() + "]");
            }
            // Do we have some data, or a trailing empty field?
            if (baos.size() > 0 // we have some data
                    || push // we've started a field
                    || state == ParserState.EMBEDDEDQUOTE // Just seen ""
            ) {
                list.add(baos.toString());
            }
        }
        return list.toArray(new String[list.size()]);
    }
Comment 7 Philippe Mouawad 2019-07-30 20:27:08 UTC
Hello,
your CSV file in word document uses | as separator which is not the default.
How is this file generated ?

You need to use the same config file for generation that was used for output.

1/ I see nowhere in user.properties or jmeter.properties that default_delimiter was set to |, so it looks to me regular that it fails.

2/ In last comment, you show other rows with ',' separator, did it fails for those data ?

3/ You say you generate report at end, what exactly do you mean ? As in your command line I don't see 
-e -o <report_folder>

https://jmeter.apache.org/usermanual/generating-dashboard.html#report_after_load_test
Comment 8 vemular 2019-07-30 21:29:39 UTC
Please see my notes inline.

1/ I see nowhere in user.properties or jmeter.properties that default_delimiter was set to |, so it looks to me regular that it fails.

My response:

[[Even with default delimiter which is comma (,) it gives the same error.
The problem is not with the delimiter. Yes I changed the delimiter to default then I sent you the user.properties.]]
##############
2/ In last comment, you show other rows with ',' separator, did it fails for those data ?

My response:

Yes a changed from | delimiter to , delimiter. But even then I see same error of 

2019-07-30 14:53:12,440 ERROR o.a.j.JMeter: Error generating dashboard: org.apache.jmeter.report.dashboard.GenerationException: Error while processing samples:Mismatch between expected number of columns:17 and columns in CSV file:4, check your jmeter.save.saveservice.* configuration or check line is complete.

Even with fresh install of Jmeter I would get the error.

#############

3/ You say you generate report at end, what exactly do you mean ? As in your command line I don't see 
-e -o <report_folder>

My response:

If you see the command line I pasted in earlier comment, I used the expanded flags rather than the short flags, the flags of  "--reportatendofloadtests  --reportoutputfolder "C:\JMeterLoadTests\SDIS_REST\D07292019_110334\SDIS\report\20KB\SDIS_75vusers"    --remoteexit"

The "--reportatendofloadtests" will generate the dashboard report at end. The "--reportoutputfolder" specifies the output folder. These are same as "-e -o <report_folder>"

############

Please do Google search. I am pasting the Google search URL. You will find users are getting these errors. But they are not reporting as bug.

https://www.google.com/search?safe=active&ei=V7ZAXbEcqbiCB8u4l3A&q=jmeter+%22error+while+processing+samples%3Amismatch+between+expected+number+of+columns%22&oq=jmeter+%22error+while+processing+samples%3Amismatch+between+expected+number+of+columns%22
Comment 9 Philippe Mouawad 2019-07-30 21:32:28 UTC
(In reply to vemular from comment #8)
> Please see my notes inline.
> 
> 1/ I see nowhere in user.properties or jmeter.properties that
> default_delimiter was set to |, so it looks to me regular that it fails.
> 
> My response:
> 
> [[Even with default delimiter which is comma (,) it gives the same error.
> The problem is not with the delimiter. Yes I changed the delimiter to
> default then I sent you the user.properties.]]
> ##############
> 2/ In last comment, you show other rows with ',' separator, did it fails for
> those data ?
> 
> My response:
> 
> Yes a changed from | delimiter to , delimiter. But even then I see same
> error of 
> 
> 2019-07-30 14:53:12,440 ERROR o.a.j.JMeter: Error generating dashboard:
> org.apache.jmeter.report.dashboard.GenerationException: Error while
> processing samples:Mismatch between expected number of columns:17 and
> columns in CSV file:4, check your jmeter.save.saveservice.* configuration or
> check line is complete.
> 
> Even with fresh install of Jmeter I would get the error.
> 
> #############
> 
> 3/ You say you generate report at end, what exactly do you mean ? As in your
> command line I don't see 
> -e -o <report_folder>
> 
> My response:
> 
> If you see the command line I pasted in earlier comment, I used the expanded
> flags rather than the short flags, the flags of  "--reportatendofloadtests 
> --reportoutputfolder
> "C:
> \JMeterLoadTests\SDIS_REST\D07292019_110334\SDIS\report\20KB\SDIS_75vusers" 
> --remoteexit"
> 
> The "--reportatendofloadtests" will generate the dashboard report at end.
> The "--reportoutputfolder" specifies the output folder. These are same as
> "-e -o <report_folder>"

Indeed, I overlooked it.
> 
> ############
> 
> Please do Google search. I am pasting the Google search URL. You will find
> users are getting these errors. But they are not reporting as bug.
> 
> https://www.google.com/
> search?safe=active&ei=V7ZAXbEcqbiCB8u4l3A&q=jmeter+%22error+while+processing+
> samples%3Amismatch+between+expected+number+of+columns%22&oq=jmeter+%22error+w
> hile+processing+samples%3Amismatch+between+expected+number+of+columns%22



Can you just provide with the user.properties/jmeter.properties that you attached a failing csv file ?

Thanks
Comment 10 Felix Schumacher 2019-07-31 05:06:00 UTC
(In reply to vemular from comment #6)
.
> 
> For example, if you compare a successful entry and a failed entry:
> 
> Failed Entry from the report file:
> 
> 2019/07/29 17:25:46.807,21034,ECMaaSRESTCreateDocSampler,500,Request
> Failed,10.89.8.21-Thread Group 1-2,,false,Test failed: text expected to
> contain /Successfully performed action/,14,0,15,15,null,0,0,0
> 
> Success Entry from report file:
> 
> 2019/07/29 17:26:07.523,303,ECMaaSRESTCreateDocSampler,200,Successfully
> performed action:  HOST_IPADDRESS =10.89.8.21,10.89.8.21-Thread Group
> 1-13,,true,,2539,0,15,15,null,0,0,0
> 
> 
> 
> The above both entries have same number of tokens. But the Jmeter code is
> wrongly tokenizing the failed entry. The column count and token count
> becoming unequal.

It looks to me as you are trying to send data inside the 'message' (Successfully
> performed action:  HOST_IPADDRESS =10.89.8.21,10.89.8.21-Thread Group
> 1-13,,true,). In the case of an exception those entries are MISSING (You could set an error message with the missing entries yourself though). As you are using the same delimiter for /your/ message as the data for JMeter, JMeters parser will fail.
Comment 11 vemular 2019-07-31 21:47:07 UTC
Created attachment 36693 [details]
Please read this first

Please read this first. attaching more files.
Comment 12 vemular 2019-07-31 21:47:45 UTC
Created attachment 36694 [details]
JTL report entries
Comment 13 vemular 2019-07-31 21:48:20 UTC
Created attachment 36695 [details]
Jmeter log
Comment 14 vemular 2019-07-31 21:48:52 UTC
Created attachment 36696 [details]
user.properties
Comment 15 vemular 2019-07-31 21:49:13 UTC
Created attachment 36697 [details]
jmeter.properties
Comment 16 Felix Schumacher 2019-08-01 04:25:50 UTC
(In reply to vemular from comment #11)
> Created attachment 36693 [details]
> Please read this first
> 
> Please read this first. attaching more files.

Why can't you post your answer as text inline? It is really annoying and maybe even dangerous to have to open a word doc.
Comment 17 Felix Schumacher 2019-08-01 06:30:49 UTC
And the best thing is, you haven't answered -- in the word doc -- my question about whether you mix your csv message with JMeter's one.
Comment 18 vemular 2019-08-01 13:17:17 UTC
(In reply to Felix Schumacher from comment #16)
> (In reply to vemular from comment #11)
> > Created attachment 36693 [details]
> > Please read this first
> > 
> > Please read this first. attaching more files.
> 
> Why can't you post your answer as text inline? It is really annoying and
> maybe even dangerous to have to open a word doc.

A word doc allows to have pictures, different fonts for different topics, etc... The plain text does not have that luxury.
Pictures make you understand well.
Comment 19 Felix Schumacher 2019-08-01 13:22:49 UTC
(In reply to vemular from comment #18)
> (In reply to Felix Schumacher from comment #16)
> > (In reply to vemular from comment #11)
> > > Created attachment 36693 [details]
> > > Please read this first
> > > 
> > > Please read this first. attaching more files.
> > 
> > Why can't you post your answer as text inline? It is really annoying and
> > maybe even dangerous to have to open a word doc.
> 
> A word doc allows to have pictures, different fonts for different topics,
> etc... The plain text does not have that luxury.
> Pictures make you understand well.

It didn't help me at all. 
 * I could not open it on my phone.
 * There was no new information available in the text (at least I did not find 
them).
 * the word document will probably not be indexed by search machines
 * bugzilla will not search inside it as it will do for text messages as this one
 * Pictures can be attached and shown directly by bugzilla
 * word documents can contain malicious content

So, please don't attach word documents, if they are not needed.
Comment 20 vemular 2019-08-01 13:38:38 UTC
(In reply to Felix Schumacher from comment #17)
> And the best thing is, you haven't answered -- in the word doc -- my
> question about whether you mix your csv message with JMeter's one.

I do not think I understood your question completely. 

When you develop a Java Sampler there is API to set certain things into result, for example sampleResult.setResponseMessage().

For failed tests the response message is different. The response assertion does not pass when the expected responseMessage is different than the actual responseMessage.

In case of assertion failure, the Jmeter itself putting this into the report file ->  "Test failed: text expected to contain /Test is OK/". This string is not from my code. This is output by Jmeter into the report.

Did you try to find why the error is -> "Mismatch between expected number of columns:17 and columns in CSV file:3,"

Every record has same 17 columns. Why does it say 3 columns?

Here I am understanding "CSV file" in the error message as the JTL report file.
Comment 21 Felix Schumacher 2019-08-01 15:15:37 UTC
(In reply to vemular from comment #20)
> (In reply to Felix Schumacher from comment #17)
> > And the best thing is, you haven't answered -- in the word doc -- my
> > question about whether you mix your csv message with JMeter's one.
> 
> I do not think I understood your question completely. 

That is probably since I misread the example lines you gave. I thought that you passed extra values in the successful messages that contained the separator used for the rest of the line. 

Now on closer inspection it turns out that I have misread the lines. The data is not problematic. 

> 
> When you develop a Java Sampler there is API to set certain things into
> result, for example sampleResult.setResponseMessage().
> 
> For failed tests the response message is different. The response assertion
> does not pass when the expected responseMessage is different than the actual
> responseMessage.
> 
> In case of assertion failure, the Jmeter itself putting this into the report
> file ->  "Test failed: text expected to contain /Test is OK/". This string
> is not from my code. This is output by Jmeter into the report.

Yes. No problem here. 

> 
> Did you try to find why the error is -> "Mismatch between expected number of
> columns:17 and columns in CSV file:3,"
> 
> Every record has same 17 columns. Why does it say 3 columns?
> 
> Here I am understanding "CSV file" in the error message as the JTL report
> file.

That might be caused by a corrupted jtl file. Have you inspected the jtl file? Is it complete? 

The jtl file you attached shows no problem in my test report generation.
Comment 22 vemular 2019-08-01 18:52:57 UTC
(In reply to Felix Schumacher from comment #21)
> (In reply to vemular from comment #20)
> > (In reply to Felix Schumacher from comment #17)
> > > And the best thing is, you haven't answered -- in the word doc -- my
> > > question about whether you mix your csv message with JMeter's one.
> > 
> > I do not think I understood your question completely. 
> 
> That is probably since I misread the example lines you gave. I thought that
> you passed extra values in the successful messages that contained the
> separator used for the rest of the line. 
> 
> Now on closer inspection it turns out that I have misread the lines. The
> data is not problematic. 
> 
> > 
> > When you develop a Java Sampler there is API to set certain things into
> > result, for example sampleResult.setResponseMessage().
> > 
> > For failed tests the response message is different. The response assertion
> > does not pass when the expected responseMessage is different than the actual
> > responseMessage.
> > 
> > In case of assertion failure, the Jmeter itself putting this into the report
> > file ->  "Test failed: text expected to contain /Test is OK/". This string
> > is not from my code. This is output by Jmeter into the report.
> 
> Yes. No problem here. 
> 
> > 
> > Did you try to find why the error is -> "Mismatch between expected number of
> > columns:17 and columns in CSV file:3,"
> > 
> > Every record has same 17 columns. Why does it say 3 columns?
> > 
> > Here I am understanding "CSV file" in the error message as the JTL report
> > file.
> 
> That might be caused by a corrupted jtl file. Have you inspected the jtl
> file? Is it complete? 

So far I ran the test at least 20 times. And each time the JTL is generated successfully. Every time it is the same error for dashboard generation - "Mismatch between expected number blah blah...."

There is no corruption of JTL file.

> 
> The jtl file you attached shows no problem in my test report generation.

Yes the JTL report is generated successfully. There is no issue in generating the JTL file. The next process after the generation of JTL process is generating the Dashboard report. The Dashboard report generation is failing.
Comment 23 Felix Schumacher 2019-08-01 18:59:15 UTC
in repository https://gitbox.apache.org/repos/asf/jmeter.git

commit 5f41f757ba6cfaee6d95892f1c15eff1d56d638b
AuthorDate: Thu Aug 1 20:49:56 2019 +0200

    Log more details on broken CSV files
    
    When corrupt CSV files are read by ReportGenerator it is often
    not clear enough where the broken data in the file is. Give
    more details in log messages and exceptions.
    
    Inspired by Bugzilla Id: 63614
---
 .../apache/jmeter/report/core/CsvSampleReader.java   | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

commit 2d8e630039afd439ca852cc705cc6f666184e899
AuthorDate: Thu Aug 1 20:53:45 2019 +0200

    Spacepolice
    
    Don't squash the exception cause directly to our own message.
---
 src/core/org/apache/jmeter/report/dashboard/ReportGenerator.java     | 2 +-
 test/src/org/apache/jmeter/gui/action/HtmlReportGeneratorSpec.groovy | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Comment 24 Felix Schumacher 2019-08-01 19:07:32 UTC
(In reply to vemular from comment #22)
> (In reply to Felix Schumacher from comment #21)
> > (In reply to vemular from comment #20)
> > > (In reply to Felix Schumacher from comment #17)
> > > > And the best thing is, you haven't answered -- in the word doc -- my
> > > > question about whether you mix your csv message with JMeter's one.
> > > 
> > > I do not think I understood your question completely. 
> > 
> > That is probably since I misread the example lines you gave. I thought that
> > you passed extra values in the successful messages that contained the
> > separator used for the rest of the line. 
> > 
> > Now on closer inspection it turns out that I have misread the lines. The
> > data is not problematic. 
> > 
> > > 
> > > When you develop a Java Sampler there is API to set certain things into
> > > result, for example sampleResult.setResponseMessage().
> > > 
> > > For failed tests the response message is different. The response assertion
> > > does not pass when the expected responseMessage is different than the actual
> > > responseMessage.
> > > 
> > > In case of assertion failure, the Jmeter itself putting this into the report
> > > file ->  "Test failed: text expected to contain /Test is OK/". This string
> > > is not from my code. This is output by Jmeter into the report.
> > 
> > Yes. No problem here. 
> > 
> > > 
> > > Did you try to find why the error is -> "Mismatch between expected number of
> > > columns:17 and columns in CSV file:3,"
> > > 
> > > Every record has same 17 columns. Why does it say 3 columns?
> > > 
> > > Here I am understanding "CSV file" in the error message as the JTL report
> > > file.
> > 
> > That might be caused by a corrupted jtl file. Have you inspected the jtl
> > file? Is it complete? 
> 
> So far I ran the test at least 20 times. And each time the JTL is generated
> successfully. Every time it is the same error for dashboard generation -
> "Mismatch between expected number blah blah...."

Have you checked the jtl file?

> 
> There is no corruption of JTL file.

Sure?

> 
> > 
> > The jtl file you attached shows no problem in my test report generation.
> 
> Yes the JTL report is generated successfully. There is no issue in
> generating the JTL file. The next process after the generation of JTL
> process is generating the Dashboard report. The Dashboard report generation
> is failing.

The attached jtl works. You might want to attach the jtl file that is not working or send it to me directly if there is private data in it, that you don't want to have posted publicly.

I have added more log detail to JMeter which should be available in the next nightly. It will give info about the line and the file with which JMeter has a problem.
Comment 25 vemular 2019-08-01 20:08:30 UTC
(In reply to Felix Schumacher from comment #24)
> (In reply to vemular from comment #22)
> > (In reply to Felix Schumacher from comment #21)
> > > (In reply to vemular from comment #20)
> > > > (In reply to Felix Schumacher from comment #17)
> > > > > And the best thing is, you haven't answered -- in the word doc -- my
> > > > > question about whether you mix your csv message with JMeter's one.
> > > > 
> > > > I do not think I understood your question completely. 
> > > 
> > > That is probably since I misread the example lines you gave. I thought that
> > > you passed extra values in the successful messages that contained the
> > > separator used for the rest of the line. 
> > > 
> > > Now on closer inspection it turns out that I have misread the lines. The
> > > data is not problematic. 
> > > 
> > > > 
> > > > When you develop a Java Sampler there is API to set certain things into
> > > > result, for example sampleResult.setResponseMessage().
> > > > 
> > > > For failed tests the response message is different. The response assertion
> > > > does not pass when the expected responseMessage is different than the actual
> > > > responseMessage.
> > > > 
> > > > In case of assertion failure, the Jmeter itself putting this into the report
> > > > file ->  "Test failed: text expected to contain /Test is OK/". This string
> > > > is not from my code. This is output by Jmeter into the report.
> > > 
> > > Yes. No problem here. 
> > > 
> > > > 
> > > > Did you try to find why the error is -> "Mismatch between expected number of
> > > > columns:17 and columns in CSV file:3,"
> > > > 
> > > > Every record has same 17 columns. Why does it say 3 columns?
> > > > 
> > > > Here I am understanding "CSV file" in the error message as the JTL report
> > > > file.
> > > 
> > > That might be caused by a corrupted jtl file. Have you inspected the jtl
> > > file? Is it complete? 
> > 
> > So far I ran the test at least 20 times. And each time the JTL is generated
> > successfully. Every time it is the same error for dashboard generation -
> > "Mismatch between expected number blah blah...."
> 
> Have you checked the jtl file?
> 
> > 
> > There is no corruption of JTL file.
> 
> Sure?
> 
> > 
> > > 
> > > The jtl file you attached shows no problem in my test report generation.
> > 
> > Yes the JTL report is generated successfully. There is no issue in
> > generating the JTL file. The next process after the generation of JTL
> > process is generating the Dashboard report. The Dashboard report generation
> > is failing.
> 
> The attached jtl works. You might want to attach the jtl file that is not
> working or send it to me directly if there is private data in it, that you
> don't want to have posted publicly.
> 
I have several JTL files. I zipped up all those. The path to them is D<MMDDYYYY_HHmmss>\SDIS\log\20KB. I am going to send to your email felix.schumacher@internetallee.de . The zipped up file is of 7MB. But each JTL file is around 30MB to 50MB.

> I have added more log detail to JMeter which should be available in the next
> nightly. It will give info about the line and the file with which JMeter has
> a problem.
Thank you for nightly. I will try to use it.
Comment 26 Felix Schumacher 2019-08-02 06:35:24 UTC
Only one of those seven jtl files shows a problem. With the more detailed error message you get:

An error occurred: Error while processing samples: Mismatch between expected number of columns:17 and columns in CSV file:3, check your jmeter.save.saveservice.* configuration or check if line 58376 in '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log' is complete

And in jmeter.log you can find:

 WARN o.a.j.r.c.CsvSampleReader: Short CSV read around line 58376 of file '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log'. Could only read 3 elements of 17 expected. Data is [2019/07/30 13:43:27.461, 121, ECMaaSRESTCreateDocSampl]

And if you have a look at line 58376 (which is the last line in that file), you see:

2019/07/30 13:43:27.461,121,ECMaaSRESTCreateDocSampl

Which seems to be a broken file to me.
Comment 27 vemular 2019-08-02 16:43:27 UTC
(In reply to Felix Schumacher from comment #26)
> Only one of those seven jtl files shows a problem. With the more detailed
> error message you get:
> 
> An error occurred: Error while processing samples: Mismatch between expected
> number of columns:17 and columns in CSV file:3, check your
> jmeter.save.saveservice.* configuration or check if line 58376 in
> '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log' is complete
> 
> And in jmeter.log you can find:
> 
>  WARN o.a.j.r.c.CsvSampleReader: Short CSV read around line 58376 of file
> '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log'. Could only read 3 elements
> of 17 expected. Data is [2019/07/30 13:43:27.461, 121,
> ECMaaSRESTCreateDocSampl]
> 
> And if you have a look at line 58376 (which is the last line in that file),
> you see:
> 
> 2019/07/30 13:43:27.461,121,ECMaaSRESTCreateDocSampl
> 
> Which seems to be a broken file to me.

Hello

Today(Aug 2, 2019) I ran another test for 15 minutes. I used the JMeter jars that are complied from the latest source from GIT as of yerterday.

The Jmeter log shows this below error. The error says there is a problem with line # 281268. I checked the line # 281268 in JTL file. The line # 281268 appears complete. All lines above & below this line also appear complete.
I beleive the Dashboard generation process may be triggering in before all clients completely send and data to Master (in distributed load test). I beleive before the JTL file is completely written the Dashbord generation process is triggering. The dashboard generation process should be started only after that all clients are sent the "test completed" message AND the JTL file is completely written and it is closed. This is my guess.
Also I beleive as the JTL file size increases Jmeter could not handle large JTL files. This is my assumption.
I think the communication between the main thread and the thread that generates the DashBoard generation lack something which is necessary.

One more thing is the timestamp in line #281268 is 2019/08/02 12:02:59.165,102. The timestamp of last line(# 281307) is 2019/08/02 12:02:59.644,3120. Probably the JTL file is being read before it is completely written and closed.


I am sending the JTL file (44MB) to your email felix.schumacher@internetallee.de. 

2019-08-02 12:03:30,530 ERROR o.a.j.JMeter: Error generating dashboard: org.apache.jmeter.report.dashboard.GenerationException: Error while processing samples: Mismatch between expected number of columns:17 and columns in CSV file:1, check your jmeter.save.saveservice.* configuration or check if line 281268 in 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is complete
org.apache.jmeter.report.dashboard.GenerationException: Error while processing samples: Mismatch between expected number of columns:17 and columns in CSV file:1, check your jmeter.save.saveservice.* configuration or check if line 281268 in 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is complete
	at org.apache.jmeter.report.dashboard.ReportGenerator.generate(ReportGenerator.java:247) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
	at org.apache.jmeter.JMeter$ListenToTest.generateReport(JMeter.java:1326) [ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
	at org.apache.jmeter.JMeter$ListenToTest.run(JMeter.java:1310) [ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.jmeter.report.core.SampleException: Mismatch between expected number of columns:17 and columns in CSV file:1, check your jmeter.save.saveservice.* configuration or check if line 281268 in 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is complete
	at org.apache.jmeter.report.core.CsvSampleReader.assertCorrectColumns(CsvSampleReader.java:205) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
	at org.apache.jmeter.report.core.CsvSampleReader.nextSample(CsvSampleReader.java:189) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
	at org.apache.jmeter.report.core.CsvSampleReader.readSample(CsvSampleReader.java:217) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
	at org.apache.jmeter.report.processor.CsvFileSampleSource.produce(CsvFileSampleSource.java:180) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
	at org.apache.jmeter.report.processor.CsvFileSampleSource.run(CsvFileSampleSource.java:238) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
	at org.apache.jmeter.report.dashboard.ReportGenerator.generate(ReportGenerator.java:245) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
	... 3 more
Comment 28 vemular 2019-08-06 14:16:12 UTC
(In reply to vemular from comment #27)
> (In reply to Felix Schumacher from comment #26)
> > Only one of those seven jtl files shows a problem. With the more detailed
> > error message you get:
> > 
> > An error occurred: Error while processing samples: Mismatch between expected
> > number of columns:17 and columns in CSV file:3, check your
> > jmeter.save.saveservice.* configuration or check if line 58376 in
> > '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log' is complete
> > 
> > And in jmeter.log you can find:
> > 
> >  WARN o.a.j.r.c.CsvSampleReader: Short CSV read around line 58376 of file
> > '/tmp/D07302019_133924/SDIS/log/20KB/SDIS.log'. Could only read 3 elements
> > of 17 expected. Data is [2019/07/30 13:43:27.461, 121,
> > ECMaaSRESTCreateDocSampl]
> > 
> > And if you have a look at line 58376 (which is the last line in that file),
> > you see:
> > 
> > 2019/07/30 13:43:27.461,121,ECMaaSRESTCreateDocSampl
> > 
> > Which seems to be a broken file to me.
> 
> Hello
> 
> Today(Aug 2, 2019) I ran another test for 15 minutes. I used the JMeter jars
> that are complied from the latest source from GIT as of yerterday.
> 
> The Jmeter log shows this below error. The error says there is a problem
> with line # 281268. I checked the line # 281268 in JTL file. The line #
> 281268 appears complete. All lines above & below this line also appear
> complete.
> I beleive the Dashboard generation process may be triggering in before all
> clients completely send and data to Master (in distributed load test). I
> beleive before the JTL file is completely written the Dashbord generation
> process is triggering. The dashboard generation process should be started
> only after that all clients are sent the "test completed" message AND the
> JTL file is completely written and it is closed. This is my guess.
> Also I beleive as the JTL file size increases Jmeter could not handle large
> JTL files. This is my assumption.
> I think the communication between the main thread and the thread that
> generates the DashBoard generation lack something which is necessary.
> 
> One more thing is the timestamp in line #281268 is 2019/08/02
> 12:02:59.165,102. The timestamp of last line(# 281307) is 2019/08/02
> 12:02:59.644,3120. Probably the JTL file is being read before it is
> completely written and closed.
> 
> 
> I am sending the JTL file (44MB) to your email
> felix.schumacher@internetallee.de. 
> 
> 2019-08-02 12:03:30,530 ERROR o.a.j.JMeter: Error generating dashboard:
> org.apache.jmeter.report.dashboard.GenerationException: Error while
> processing samples: Mismatch between expected number of columns:17 and
> columns in CSV file:1, check your jmeter.save.saveservice.* configuration or
> check if line 281268 in
> 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is
> complete
> org.apache.jmeter.report.dashboard.GenerationException: Error while
> processing samples: Mismatch between expected number of columns:17 and
> columns in CSV file:1, check your jmeter.save.saveservice.* configuration or
> check if line 281268 in
> 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is
> complete
> 	at
> org.apache.jmeter.report.dashboard.ReportGenerator.generate(ReportGenerator.
> java:247) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
> 	at org.apache.jmeter.JMeter$ListenToTest.generateReport(JMeter.java:1326)
> [ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
> 	at org.apache.jmeter.JMeter$ListenToTest.run(JMeter.java:1310)
> [ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> Caused by: org.apache.jmeter.report.core.SampleException: Mismatch between
> expected number of columns:17 and columns in CSV file:1, check your
> jmeter.save.saveservice.* configuration or check if line 281268 in
> 'C:\JMeterLoadTests\SDIS_REST\D08022019_114731\SDIS\log\20KB\SDIS.log' is
> complete
> 	at
> org.apache.jmeter.report.core.CsvSampleReader.
> assertCorrectColumns(CsvSampleReader.java:205)
> ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
> 	at
> org.apache.jmeter.report.core.CsvSampleReader.nextSample(CsvSampleReader.
> java:189) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
> 	at
> org.apache.jmeter.report.core.CsvSampleReader.readSample(CsvSampleReader.
> java:217) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
> 	at
> org.apache.jmeter.report.processor.CsvFileSampleSource.
> produce(CsvFileSampleSource.java:180)
> ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
> 	at
> org.apache.jmeter.report.processor.CsvFileSampleSource.
> run(CsvFileSampleSource.java:238)
> ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
> 	at
> org.apache.jmeter.report.dashboard.ReportGenerator.generate(ReportGenerator.
> java:245) ~[ApacheJMeter_core.jar:5.2-SNAPSHOT.20190802]
> 	... 3 more

I hope you noted my message in comment 27. Now I am no more generating the dashboard in the same command as the main test, but running the dashboard generation as a separate command. That is I am not going to use the flag of "--reportatendofloadtests" in the command. I would think to post this message in user forums who are Jmeter users so they also do the tests same way.
Comment 29 UbikLoadPack support 2019-08-06 14:28:30 UTC
Unless bug is confirmed please don't post anything.

Can you provide a new set of data showing there is a bug, last time, it appeared there was no problem.

Can you provide also full jmeter.log in INFO level of all injectors so that we check your hypothesis.
Comment 30 vemular 2019-08-06 15:00:16 UTC
(In reply to UbikLoadPack support from comment #29)
> Unless bug is confirmed please don't post anything.
> 

I am not going to post anything myself.

> Can you provide a new set of data showing there is a bug, last time, it
> appeared there was no problem.

I provided the JTL file to Felix. The Exception message says the line # 281268 did not have full columns. But when I checked the file, the line # 281268, I find this line and subsequent lines are completely written. I strongly believe that the dashboard generation thread is starting before the JTL file is completely written and the file is CLOSED. I would think the dashboard generation thread should obtain exclusive lock on JTL file before it starts reading.

> Can you provide also full jmeter.log in INFO level of all injectors so that
> we check your hypothesis.

I am using five Windows Server 2016 VM instances to run this load test using Master/Slaves combination. The first host contains both Master & a slave. Other four hosts are slaves only. I have set 20 users for each Slave. So there are total simultaneous of 100 users (20 X 5 slaves). Hope this helps. Probably the Dashboard report generation thread is starting as soon as the test duration is completed. But the slaves might have started a test at the verge of completion of test duration. These tests may continue to execute for a few seconds/minutes even after the test duration is completed. So the dashboard generation thread has to start only after the JTL file is closed.
Comment 31 vemular 2019-08-06 15:20:28 UTC
Also I would like to ask this question. Is there any property or properties combination that will ensure that the dashboard generation is started only after the JTL file is complete and is closed?
Comment 32 UbikLoadPack support 2019-08-06 15:24:46 UTC
(In reply to vemular from comment #31)
> Also I would like to ask this question. Is there any property or properties
> combination that will ensure that the dashboard generation is started only
> after the JTL file is complete and is closed?

Hello,
Generation is started at end of load test (including slaves ending) after completion of JTL file.
If it's not happening this way it is a bug.

So there is no property for that, as it is expected to work the right way.
Comment 33 Felix Schumacher 2019-08-06 16:35:13 UTC
The second file vermular posted to me is complete and contains no error.
Comment 34 Felix Schumacher 2019-08-06 16:47:01 UTC
Interesting would probably be to have the exact time, when JMeter thinks it ended the test -- and thus closes the jtl file -- and the last entries in the jtl file. Are there entries with timestamps older than the ending time.

Most probably it is useful to attach a complete log and jtl file as noted already.

It might even be helpful to know a bit more about the test you are trying to run. Why do you think that it is not ending at the scheduled time? Can you take threaddumps shortly after the test run should have ended, but did not?
Comment 35 vemular 2019-08-07 18:21:44 UTC
(In reply to Felix Schumacher from comment #34)
> Interesting would probably be to have the exact time, when JMeter thinks it
> ended the test -- and thus closes the jtl file -- and the last entries in
> the jtl file. Are there entries with timestamps older than the ending time.
> 
> Most probably it is useful to attach a complete log and jtl file as noted
> already.
> 
> It might even be helpful to know a bit more about the test you are trying to
> run. Why do you think that it is not ending at the scheduled time? Can you
> take threaddumps shortly after the test run should have ended, but did not?

Assume that the load test is set to run for 600 seconds. The threads continue to send requests until 599th second. The Requests that started at 599th second would continue to run until it gets a response. The threads do not abruptly end those reauests. After 600 seconds there will be no fresh requests but the already started requests continue to run. The server, to which the requests are sent, could take much time (the server maybe taking few seconds or minutes based on load, or based on nature of process, or based on the http timeouts). So the Master Jmeter can not guess when the slaves will be done. The Master JMeter has to wait for two things before it starts working for dashboard report.. First is to receive the message of  "I am done with all requests" from all slaves. Second is the JTL file should be closed.

I would think to obtain the exclusive lock on the file for reading & writing. I guess the thread(T1) that is writing into JTL File and the thread(T2) that is generating dashboard report are different threads. Both the threads may have to obtain exclusive locks before reading or writing thus ensuring no more than one thread deals with the JTL file at the same time. I am going to send the JTL file & Jmeter.log files(with INFO level) to email.
Comment 36 Philippe Mouawad 2019-09-30 20:03:57 UTC
pmouawad@apache.org
	
10:02 PM (0 minutes ago)
	
to commits@jmeter.apache.org
This is an automated email from the ASF dual-hosted git repository.

pmouawad pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/jmeter.git


The following commit(s) were added to refs/heads/master by this push:
     new 105a399  This commit fixes bugs 63614 and 63723 Bug 63614 - Distributed testing: Unable to generate Dashboard report at end of load test
105a399 is described below

commit 105a3999f44c73200fb08c249e120937840ee799
Author: pmouawad <p.mouawad@ubik-ingenierie.com>
AuthorDate: Mon Sep 30 22:02:20 2019 +0200

    This commit fixes bugs 63614 and 63723
    Bug 63614 - Distributed testing: Unable to generate Dashboard report at
    end of load test

    Bug 63723 - JMeter master ends distributed test though some threads
    still are active
---
 .../src/main/java/org/apache/jmeter/JMeter.java    | 136 +++++++++++----------
 .../apache/jmeter/engine/ClientJMeterEngine.java   |   5 +
 .../apache/jmeter/engine/DistributedRunner.java    |  27 +++-
 xdocs/changes.xml                                  |   2 +
 4 files changed, 99 insertions(+), 71 deletions(-)