This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 81465 - Test Run Result Comparator is too strict?
Summary: Test Run Result Comparator is too strict?
Status: VERIFIED FIXED
Alias: None
Product: soa
Classification: Unclassified
Component: JBI Test Driver (show other bugs)
Version: 5.x
Hardware: All All
: P1 blocker (vote)
Assignee: wchui
URL:
Keywords:
Depends on: 80777
Blocks: 81467 81554
  Show dependency tree
 
Reported: 2006-07-31 12:35 UTC by Michael Frisino
Modified: 2006-09-27 11:51 UTC (History)
0 users

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
screenhsot for the comparision (11.04 KB, application/octet-stream)
2006-09-26 08:33 UTC, pushpa
Details
diff screenshot (74.43 KB, application/octet-stream)
2006-09-26 09:12 UTC, pushpa
Details
White spaces considered for comparison (100.95 KB, image/jpeg)
2006-09-27 11:51 UTC, deepakhp
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Frisino 2006-07-31 12:35:15 UTC
See also bug 80777

The Test Run compares the actual output to the (expected output) as defined in a
output.xml file.

If the comparison returns false then the test fails.

The problem is that the comparison seems to be overly strict and does a strict
literal comparison of text. In xml the white space should not matter for
comparison of results, but the test driver is comparing white space.

This problem surfaces in the Ent Pack samples following the fix for bug 80777.

Prior to the fix for bug 80777, the sample test runs were succesful. But after
the fix for but 80777, the sample test runs fail. The reason is that the
original output.xml files in the samples match the previous style of actual
output from the runtime. But fix for bug 80777 changed the actual output by
adding carriage returns. 

So now, the strict comparator fails because the expected output no longer
matches the actual output.

So workaround is that we can (and should) change the sample output.xml to be the
more nicely formatted version.

However, I think the comparator is still a problem as customers may intrdoduce
whitespace on either side of the fence - runtime output or expected result
output.xml.
Comment 1 Michael Frisino 2006-07-31 14:54:34 UTC
I believe that the comparator used by the IDE Diff action is what you need to
use. It seems that the IDE diff action ignores the white space differences. 
Comment 2 _ hong_lin 2006-07-31 22:27:30 UTC
Since all sample tests are failed. I upgraded this bug to P1. 
Comment 3 Jun Qian 2006-08-01 00:44:27 UTC
Fixed now.
Comment 4 _ hong_lin 2006-08-01 22:50:06 UTC
Verified that bug is fixed in nightly build 060731.

All tests for the 3 sample apps are passed. 
Comment 5 Michael Frisino 2006-08-02 07:50:34 UTC
Is the current solution vulnerable to any known edge cases? That is to say, do
you know how to break the 81465 comparator today? It would help for users and
demo people to know what format edge cases to avoid.
Comment 6 wchui 2006-08-02 19:05:09 UTC
There is no known case to break the comparison. For the comparison, we will use
XML Diffs in the XML project - 81554 is filed.
Comment 7 lchang 2006-08-08 04:06:59 UTC
It seems to be forgiving in places that I would not expect.

For example the two messages below do not show any difference:
      <paramA>Hello World   </paramA>
      <paramA>Hello World</paramA>

However, I think that even though the spaces between tags can be ignored, the 
spaces within a "value" should be considered.
Comment 8 Michael Frisino 2006-08-08 11:08:45 UTC
This is probably related to simple algorithm for ignoring whitespace.
Comment 9 Michael Frisino 2006-08-08 11:10:23 UTC
I doubt this can be improved without switching to full blow XML diff comparator.
So question for bug council is whether to leave as is for beta or implement
switch to XML Diff comparator which is currently planned post-beta.
Comment 10 wchui 2006-08-08 19:37:58 UTC
The mentioned white space difference is always ignored even before the fix.
Comment 11 wchui 2006-09-22 21:23:00 UTC
Fixed. now using xdm diff and the issue mentioned below "the 
spaces within a "value" should be considered" is now fixed.
Comment 12 pushpa 2006-09-26 08:30:17 UTC
Reopening this issue.
1.The white space are within the values are considered during comparision, but 
white spaces between the stags are also considered.

These should be ignored.

attaching the screenshot
Comment 13 pushpa 2006-09-26 08:33:33 UTC
Created attachment 34492 [details]
screenhsot for the comparision
Comment 14 Jun Qian 2006-09-26 08:40:59 UTC
The screenshot you attached doesn't seem to have the comparison.
Comment 15 pushpa 2006-09-26 09:12:26 UTC
Created attachment 34495 [details]
diff screenshot
Comment 16 Jun Qian 2006-09-26 09:26:50 UTC
The complete text content inside an element should be compared. So in this case,
the comparison failed correctly.

Previously, the beginning and trailing white spaces of an element's text content
were ignored and that was a bug.
Comment 17 deepakhp 2006-09-27 11:50:05 UTC
Verified in 060925_1 build. complete text content inside an element is compared.
(As per jqian's comments)
Comment 18 deepakhp 2006-09-27 11:51:14 UTC
Created attachment 34571 [details]
White spaces considered for comparison