Bug 6893 - rule_tests.t fails if test_types is defined
Summary: rule_tests.t fails if test_types is defined
Status: RESOLVED FIXED
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Regression Tests (show other bugs)
Version: 3.4 SVN branch
Hardware: PC Windows 7
: P2 blocker
Target Milestone: Undefined
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-01-21 17:14 UTC by Kevin A. McGrail
Modified: 2013-01-22 02:56 UTC (History)
1 user (show)


Attachment Type Modified Status Actions Submitter/CLA Status
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Kevin A. McGrail 2013-01-21 17:14:33 UTC 
As seen on Jenkins, if Conf/Parser.pm leaves the test_types in the code (see bug 5503), rule_tests.t will fail.

This is because rule_tests.t was completely skipping every test previously because of this logic:

next unless defined($test_type);        # score, but no test

Now that the tests are running, they are bombing out on a few of them:

Jenkin's output:
Test for 'HIDE_WIN_STATUS' (type: 14) against 'attributes like href=foo onMouseOver="flashiness"'
ok 26
Use of uninitialized value in pattern match (m//) at t/rule_tests.t line 69.
Use of uninitialized value in substitution (s///) at t/rule_tests.t line 70.
Use of uninitialized value in concatenation (.) or string at t/rule_tests.t line 72.
Test for 'FROM_STARTS_WITH_NUMS' (type: 8) against '12345678matt@sergeant.org'
not ok 27
# Test 27 got: "0" (t/rule_tests.t at line 108 fail #26)
#    Expected: "1" (Test for 'FROM_STARTS_WITH_NUMS' (type: 8) against '12345678matt@sergeant.org')
Use of uninitialized value in pattern match (m//) at t/rule_tests.t line 69.
Use of uninitialized value in substitution (s///) at t/rule_tests.t line 70.
Use of uninitialized value in concatenation (.) or string at t/rule_tests.t line 72.
Test for 'FROM_STARTS_WITH_NUMS' (type: 8) against 'matt@sergeant.org'
ok 28
Test for 'FORGED_YAHOO_RCVD' (type: 9) against 'by mf1.lng.yahoo.com (8.11.1/8.11.1) id g3SDfPH19426'
ok 29
Use of uninitialized value in pattern match (m//) at t/rule_tests.t line 69.
Use of uninitialized value in substitution (s///) at t/rule_tests.t line 70.
Use of uninitialized value in concatenation (.) or string at t/rule_tests.t line 72.
Test for 'INVALID_DATE' (type: 8) against 'Sat, 31 Dec 2005 23:59:60 -0500'
ok 30
Use of uninitialized value in pattern match (m//) at t/rule_tests.t line 69.
Use of uninitialized value in substitution (s///) at t/rule_tests.t line 70.
Use of uninitialized value in concatenation (.) or string at t/rule_tests.t line 72.
Test for 'INVALID_DATE' (type: 8) against 'Wed, 31 Jul 2002 16:41:57 +0200'
ok 31
Use of uninitialized value in pattern match (m//) at t/rule_tests.t line 69.
Use of uninitialized value in substitution (s///) at t/rule_tests.t line 70.
Use of uninitialized value in concatenation (.) or string at t/rule_tests.t line 72.
Test for 'INVALID_DATE' (type: 8) against 'Sat, 31 Dec 2005 24:00:00 -0500'
not ok 32
# Test 32 got: "0" (t/rule_tests.t at line 108 fail #31)
#    Expected: "1" (Test for 'INVALID_DATE' (type: 8) against 'Sat, 31 Dec 2005 24:00:00 -0500')
Use of uninitialized value in pattern match (m//) at t/rule_tests.t line 69.
Use of uninitialized value in substitution (s///) at t/rule_tests.t line 70.
Use of uninitialized value in concatenation (.) or string at t/rule_tests.t line 72.
Test for 'INVALID_DATE' (type: 8) against 'Sat, 31 Dec 2005 23:00:00'
not ok 33
# Test 33 got: "0" (t/rule_tests.t at line 108 fail #32)
#    Expected: "1" (Test for 'INVALID_DATE' (type: 8) against 'Sat, 31 Dec 2005 23:00:00')
Use of uninitialized value in pattern match (m//) at t/rule_tests.t line 69.
Use of uninitialized value in substitution (s///) at t/rule_tests.t line 70.
Use of uninitialized value in concatenation (.) or string at t/rule_tests.t line 72.
Test for 'INVALID_DATE' (type: 8) against 'Thurs, 31 Jul 2002 16:41:57 +0200'
not ok 34
# Test 34 got: "0" (t/rule_tests.t at line 108 fail #33)
#    Expected: "1" (Test for 'INVALID_DATE' (type: 8) against 'Thurs, 31 Jul 2002 16:41:57 +0200')
 Failed 4/34 subtests
Comment 1 Kevin A. McGrail 2013-01-22 02:34:26 UTC 
I've been reviewing this code for some time now.

There is no working logic for header tests.  And even the header tests that pass shouldn't pass because the logic isn't actually working but it returns the expected answer.

Further, this test doesn't function correctly in SA 3.2.5 or 3.3 branch.

In short, no telling how long it's been broken but it's nothing recent.

I rewrote it from scratch because these regression tests could be quite helpful to add examples of what you want to pass and fail when you are writing a rule.  There's only 38 tests right now but I can see this being very useful.

Will commit after a make test completes and hopefully this will also get Jenkins back to stable.
Comment 2 Kevin A. McGrail 2013-01-22 02:56:03 UTC 
svn commit -m 'Added some functions to debug hashes and arrays for test debugging, rewrote rule_tests.t to work now that test_types is not cleared (bug 6893), fixed a few regression tests that were not up to date, commented one regression test that is not working (bug 6894).'
Sending        rules/20_head_tests.cf
Sending        rules/regression_tests.cf
Sending        t/SATest.pm
Sending        t/rule_tests.t
Transmitting file data ....
Committed revision 1436729.