On a linux-box, I'm testing with trunk and perl 5.14.0 using NetAddr:IP v4.044 and not having an issue.

I then upgraded to 4.050 and tested trunk with no issues like you describe.

We've made a lot of changes in sa-update recently.  Do you have the ability to try the trunk version?

Can you send sa-update -D? Perhaps something in your particular config files or a bug somewhere else manifesting here.

> This seems to happen for 4.049 and 4.050 I've tried them both.

Does the 'make test' of the NetAddr-IP module pass all its tests?
It fails on my machine, and there is a bug report on the CPAN
for this: https://rt.cpan.org/Public/Bug/Display.html?id=71916

Not sure if this is a cause of the SA failure.

Hi,

Sorry I havent run make test, unfortunately the update was being applied as part of an automatic cpanel script.

But I was also getting this issue with version 0.49

If I can get the time in a bit, I will rebuild the latest one and copy the output.

Duncan.

(In reply to comment #2)
> > This seems to happen for 4.049 and 4.050 I've tried them both.
> 
> Does the 'make test' of the NetAddr-IP module pass all its tests?
> It fails on my machine, and there is a bug report on the CPAN
> for this: https://rt.cpan.org/Public/Bug/Display.html?id=71916
> 
> Not sure if this is a cause of the SA failure.

FYI, make test passes on the box I am testing with.

Created attachment 4983 [details]
Output from installation script

I have copied and pasted the output from the earlier run installation script, not sure if this will be of any use.

> I have copied and pasted the output from the earlier run installation script,
> not sure if this will be of any use.

This doesn't seem to run a 'make test', you may need to run it
manually. Perhaps the most straightforward is to go the usual
CPAN route: download and unpack NetAddr-IP-4.050.tar.gz,
cd there, perl Makefile.PL; make; make test
(leaving out the final 'make install', so as not to clobber
your existing installation).

This came up on IRC.  I asked Duncan to post it here.  Seems likely to be a problem outside of SA, but I thought it would be worth having a record of the issue, whatever it ends up being.

I did a google search for the "svn917659" in his output, and it looks like that's SA v3.3.1.  

  Sep 21 14:28:12.450 [29737] dbg: generic: SpamAssassin version 3.3.1
  Sep 21 14:28:12.478 [29737] dbg: generic: sa-update version svn917659
    - http://www.directadmin.com/forum/archive/index.php/t-37777.html

Why doesn't sa-update -V say 3.3.1?

05:50AM < DuncanIII> Since the NetAddr::IP perl module has been upgrade to 4.049, it seems to stop SA from correctly marking spam
05:50AM < DuncanIII> also it brakes other various parts of SA like sa-update
05:50AM < DuncanIII> When i roll back to 4.048 it works fine
10:36AM < DuncanIII> Its installed on a CPanel machine so I have just been liaising with their tech support
10:36AM < DuncanIII> They seem to think its an identified issue that is being worked on
10:37AM < DuncanIII> there current fix is to stop the machine from auto updating to the new module
10:37AM < DuncanIII> But I maybe open a bug as I can't see anything that represents it on the bugtracker
10:48AM < DuncanIII> I'm not sure if its a NetAddrr::IP issue, as it could be the implementation of NetAddrr::IP within SA
10:52AM < Darxus> Any objection to me copying and pasting everything you've said here into the bug?
10:56AM < DuncanIII> no its fine copy and paste away
10:56AM < DuncanIII> ill see if i can track it down, i'm not sure if its relevant
10:58AM < DuncanIII> I'm not sure if they are having the same problem but it does relate to the same library, its starts of from a post from last year. then comes back again this year at the beginning of the week
10:58AM < DuncanIII> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=601601

Created attachment 4984 [details]
Make test

Yes it seems to error on mine to, attached is a copy of the transcript from make test.
Duncan.

> # sa-update
> netset: cannot include 0:0:0:0:0:0:0:1/128 as it has already been included
> netset: cannot include 0:0:0:0:0:0:0:1/128 as it has already been included

Btw, does a 'spamassassin --lint' spill any complaints?

Are you sure you do not have a '::1' listed
in internal_networks and trusted_networks ?

I will investigate and get back to you, I may have to wait till after it breaks it again on auto update, as each time I run a test SA lets spam through.

Duncan.

Hi,
trusted_networks is commented out and there is no reference to internal_networks
Duncan.

(In reply to comment #10)
> I will investigate and get back to you, I may have to wait till after it breaks
> it again on auto update, as each time I run a test SA lets spam through.
> 
> Duncan.

This bug is in NetAddr::IP from looking at make test.  Something in bigint.t is showing major issues.  This is not an SA problem.(In reply to comment #11)
> Hi,
> trusted_networks is commented out and there is no reference to
> internal_networks
> Duncan.


This bug is in NetAddr::IP or a related module from looking at make test.  Something in bigint.t is showing major issues.  This is not an SA problem.  

Based on the bigint issue and this report, you might try looking into the Math::BigInt module and see if you can fix it that way based on this info:

https://rt.cpan.org/Public/Bug/Display.html?id=71916

You might also try and open a bug about the issue/comment on the bug above that you are seeing the same issue in your environment.

Regards,
KAM

Btw, on our server the NetAddr::IP 4.050 does fail its self-test
as noted above, but it seems this bug has no impact on SpamAssassin
operations, which apparently still works normally after upgrading
4.044 to 4.050 on our FreeBSD system using perl 5.14.1.

We noticed the problem also today. I've been in contact with the author recently about bugs in 4.049 which he resolved, but at the same time, he did a re-factor of bigint handling for 4.050 (RT 68723?)

From the looks of it 4.051 is about to be uploaded to pause, so maybe this will be resolved shortly.

From what I can tell, I think it's an issue only on systems with 64bit perl.

> From the looks of it 4.051 is about to be uploaded to pause, so maybe
> this will be resolved shortly.
> From what I can tell, I think it's an issue only on systems with 64bit perl.

Great, thanks for the information and the push.

Anyway, it remains to be seen if this module's bug is just a red herring.
Like I said, the SpamAssassin (3.4/trunk) on our host seems to be unaffected
by this bug. I'm beginning to think that Duncan is affected by some other
problem.

Still waiting for Duncan to answer the question:
> does a 'spamassassin --lint' spill any complaints?

Btw, the message:
  netset: cannot include 0:0:0:0:0:0:0:1/128 as it has already been included
is just an informational warning. By itself it shouldn't be causing a failure.

Perhaps it's time to see a debugging output when it starts to
'prevent SA from marking emails as spam'.

> by this bug. I'm beginning to think that Duncan is affected by some other
> problem.

Possibly, but we have seen this on multiple systems which update from cpan nightly. Reverting to 4.048 is definitely fixing it. If this is not the issue, then something else that's new with NetAddr::IP is the culprit.

Hi,

Sorry for no response.

I have been away Ill be tied up this morning but maybe able to do some further investigation later

I Did recieve the following from Cpanel Support, they seem to think its something to do with the perl libraries.

Hi Duncan,

I just wanted to provide you with an update.  This ticket has been escalated past
the top tier of support, and is waiting for further input from the development
staff.  

The issue stems from an incompatibility between NetAddr::IP starting at version
4.049 and Socket6 prior to version 0.23.  A holdback is currently in place
preventing NetAddr::IP from updating past 4.048 with cPanel scripts.  

However, this will not cause automatic downgrading to 4.048 if a later version is
installed.  To do that, you can run /scripts/perlinstaller --force NetAddr::IP

Running "/scripts/perlinstaller Mail::SpamAssassin" after NetAddr::IP has been
reverted to the desired version should fix Spam Assassin.  

The holdback (which was removed briefly today when it was believed that NetAddr::IP
version 4.050 would resolve the issue) should prevent the issue from reoccurring
with nightly upcp's or other cPanel scripts.  We are, of course, very interested in
hearing if it does not.  :) 


This ticket will remain awaiting response from us until morning when the development
team can review the situation.  Thank you for your patience,

Thanks for the update.  I emailed this info to the NetAddr::IP maintainer and the Debian package maintainers just to make sure this work isn't unnecessarily duplicated.

> The issue stems from an incompatibility between NetAddr::IP starting at version
> 4.049 and Socket6 prior to version 0.23.  A holdback is currently in place
> preventing NetAddr::IP from updating past 4.048 with cPanel scripts.  
4.049 was a separate issue. In that case, NetAddr::IP wouldn't even install when Socket6 0.20 was installed.

> The holdback (which was removed briefly today when it was believed that
> NetAddr::IP. version 4.050 would resolve the issue) should prevent the issue from
4.050 created a new issue where Socket6 and NetAddr::IP would install but then saupdate and company would not run on some systems. It seems to be 64bit only systems as best I can tell. Mark reports he's having no problems on a64bit but I do note that Mark is running on BSD, not Linux so the problem may be tied to linux.

Seems the "cannot include 0:0:0:0:0:0:0:1/128" bug in NetAddr-IP
is documented in:

  https://rt.cpan.org/Ticket/Display.html?id=71925

and fixed 4.053:

  4.053 Wed Oct 26 08:52:34 PDT 2011
     In Lite.pm v1.36
      fix bug #71925. A a sub-varient of #62521 that showed up only for
      short notation for IPv4. i.e. 127/n, 127.0/n, 127.0.0/n but
      not 127.0.0.0/n

but the Math::BigInt incompatibility is fixed in 4.055.

A text in #71925 says:
| Please discard all NetAddr::IP versions 4.045 - 4.053


Suggesting the following change to the description
of a NetAddr::IP entry in DependencyInfo.pm :

--- lib/Mail/SpamAssassin/Util/DependencyInfo.pm        (revision 1196549)
+++ lib/Mail/SpamAssassin/Util/DependencyInfo.pm        (working copy)
@@ -80,7 +80,7 @@
   and used by DNSxL rules for assembling DNS queries out of IPv6 addresses.
   4.010 fixes an issue where NetAddr::IP::full6() causes a full6.al include
   error.
-  Avoid versions 4.034 and 4.035.",
+  Avoid versions 4.034 to 4.035 and 4.045 to 4.054",

(In reply to comment #20)
> Seems the "cannot include 0:0:0:0:0:0:0:1/128" bug in NetAddr-IP
> is documented in:
> 
>   https://rt.cpan.org/Ticket/Display.html?id=71925
> 
> and fixed 4.053:
> 
>   4.053 Wed Oct 26 08:52:34 PDT 2011
>      In Lite.pm v1.36
>       fix bug #71925. A a sub-varient of #62521 that showed up only for
>       short notation for IPv4. i.e. 127/n, 127.0/n, 127.0.0/n but
>       not 127.0.0.0/n
> 
> but the Math::BigInt incompatibility is fixed in 4.055.
> 
> A text in #71925 says:
> | Please discard all NetAddr::IP versions 4.045 - 4.053
> 
> 
> Suggesting the following change to the description
> of a NetAddr::IP entry in DependencyInfo.pm :
> 
> --- lib/Mail/SpamAssassin/Util/DependencyInfo.pm        (revision 1196549)
> +++ lib/Mail/SpamAssassin/Util/DependencyInfo.pm        (working copy)
> @@ -80,7 +80,7 @@
>    and used by DNSxL rules for assembling DNS queries out of IPv6 addresses.
>    4.010 fixes an issue where NetAddr::IP::full6() causes a full6.al include
>    error.
> -  Avoid versions 4.034 and 4.035.",
> +  Avoid versions 4.034 to 4.035 and 4.045 to 4.054",


Since it won't make test on 64 bit or ipv6 systems (not sure 100% of the scenario), should we consider just block those versions in Makefile.PL as well?

(In reply to comment #21)
> Since it won't make test on 64 bit or ipv6 systems (not sure 100% of the
> scenario), should we consider just block those versions in Makefile.PL as well?

Only 4.49 is still on CPAN. He's deleted the rest of the problematic versions. It's pretty rare for someone to install from backpan. Couldn't we just bump NetAddr::IP to 4.055 in Makefile.PL?

(In reply to comment #20)
> Seems the "cannot include 0:0:0:0:0:0:0:1/128" bug in NetAddr-IP
> is documented in:
> 
>   https://rt.cpan.org/Ticket/Display.html?id=71925

FYI, I'm a little nervous about additional changes due to a possible new incompatibility with some BSD network stacks. There's quite a few test failure reports in the testing system at the moment.

https://rt.cpan.org/Public/Bug/Display.html?id=72106

> Only 4.49 is still on CPAN. He's deleted the rest of the problematic versions.
> It's pretty rare for someone to install from backpan. Couldn't we just bump
> NetAddr::IP to 4.055 in Makefile.PL?

Please don't.  Existing systems running the older versions appear to work fine.

I'm +1 with the note in Dependency and then if we get more complaints, we modify to enforce better the exact versions.  I seem to remember we did this for Net::DNS but the logic to implement the check as discussed does elude me for makemaker.

> FYI, I'm a little nervous about additional changes due to a possible new
> incompatibility with some BSD network stacks.

I think the change to an info text only is the least obtrusive at the moment.
Probably not worth venturing into stronger restrictions - we just need to
keep this bug in mind to be able to point people in the right direction.

> There's quite a few test failure reports in the testing system at the moment.

I believe the trunk no longer uses a shorthand notation like 127.1
anywhere, which is probably why some of use who tried 4.050 were not
affected by a bug (provided this notation is also not used in a
local configuration like trusted_networks etc).

It is likely most people won't be affected by the 127.1 bug
re-introduction when switching to 3.4.0, nor by the Math::BigInt
incompatibility.

Just going for the simple text change:

trunk:
  Bug 6681: NetAddr-IP version above 4.048 break sa-update
  - recommend avoiding versions 4.045 to 4.054
Sending lib/Mail/SpamAssassin/Util/DependencyInfo.pm
Committed revision 1197179.

3.3.2 seems to be failing on CPAN a little: 

http://www.cpantesters.org/distro/M/Mail-SpamAssassin.html#Mail-SpamAssassin-3.3.2

One thing I noticed on my test box with perl 5.14.0 was similar to this error:

http://www.cpantesters.org/cpan/report/ccf897b2-a59f-11e0-96f9-a254a96f2e35

t/cidrs.t                       (Wstat: 0 Tests: 51 Failed: 4)
  Failed tests:  15-17, 41
t/recreate.t                    (Wstat: 0 Tests: 9 Failed: 1)
  Failed test:  9

with lots of Jul  3 14:04:26.276 [12501] warn: netset: cannot include 0:0:0:0:0:0:0:1/128 as it has already been included

So Based on https://rt.cpan.org/Public/Bug/Display.html?id=71925 "Please discard all NetAddr::IP versions 4.045 - 4.053", I think we should try and expand Makefile.PL to block those versions if we can.  The simple text change isn't effective for CPAN users.

With a fairly stock install of 5.14.0, I had no hope of getting Mail::SpamAssassin installed because I had NetAddr::IP 4.050.  A quick upgrade to 4.058 and voila, I was able to pass a make test.

Regards,
KAM

(In reply to comment #27)
> 3.3.2 seems to be failing on CPAN a little: 
> ...
> with lots of Jul  3 14:04:26.276 [12501] warn: netset: cannot include
> 0:0:0:0:0:0:0:1/128 as it has already been included

He messed with ip shortening alot (::1/28) in some of those versions and got it wrong multiple times. Even now, OSX does not shorten the same way as everyone else, so I'm a little frightened another release is pending one of these days.

I think I need to talk him into doing dev releases so broken versions don't accidentally get installed.

I'm tentatively closing this ticket. We have an advisory text now
in DependencyInfo.pm, and trunk is no longer using a shortened
network form like 128/8. If there are further bugs in current
or future versions of a NetAddr::IP module, this is mostly
out of SpamAssassin scope. Please re-open if further information
pops up.

This happens with Net-Addr::IP version 4.062 too:

zimbra@zre-ldap002:~$ /opt/zimbra/libexec/sa-learn -p /opt/zimbra/conf/sa/salocal.cf --dbpath /opt/zimbra/data/amavisd/.spamassassin -L --no-sync --siteconfigpath /opt/zimbra/conf/spamassassin --spam /tmp/q.spam
netset: cannot include 127.0.0.0/8 as it has already been included
netset: illegal network address given: '[::1]/128'
netset: illegal network address given: '[fc00:10:137:242::]/64'
netset: illegal network address given: '[fe80::%eth0]/64'
Learned tokens from 0 message(s) (0 message(s) examined)

Our sa-learn has:

use lib '/opt/zimbra/zimbramon/lib';                # substituted at 'make' time


where:
zimbra@zre-ldap002:~/zimbramon/lib/x86_64-linux-gnu-thread-multi/NetAddr$ vi IP.pm

$VERSION = do { sprintf " %d.%03d", (q$Revision: 4.62 $ =~ /\d+/g) };

So I don't think this is fully resolved.

This is my trusted_networks line from salocal.cf:

trusted_networks 127.0.0.0/8 10.137.242.0/24 [::1]/128 [fc00:10:137:242::]/64 [fe80::%eth0]/64

I've filed this with the NetAddr::IP author as https://rt.cpan.org/Ticket/Display.html?id=80429

> netset: cannot include 127.0.0.0/8 as it has already been included

This one is just an (unnecessary) warning.
Loopback addresses are already in both lists (internal and trusted).

> netset: illegal network address given: '[::1]/128'
> netset: illegal network address given: '[fc00:10:137:242::]/64'

Square brackets were not yet allowed in SpamAssassin 3.3,
and are stripped off in 3.4 (trunk) before passing to NetAddr.
No need to use them here, square brackets around IPv6 address
are only needed where an address is combined with a port number.

> netset: illegal network address given: '[fe80::%eth0]/64'

Scoped address (...%eth0) is currently not useful and I believe
it is not recognized by NetAddr::IP. The reason is that the
information about a peer address coming from a socket
does not currently carry scope information, so something
like %eth0 at the end of an address never reaches an application.
For the time being just use non-scoped fe80::/10 .

(In reply to comment #33)
> > netset: cannot include 127.0.0.0/8 as it has already been included
> 
> This one is just an (unnecessary) warning.
> Loopback addresses are already in both lists (internal and trusted).
> 
> > netset: illegal network address given: '[::1]/128'
> > netset: illegal network address given: '[fc00:10:137:242::]/64'
> 
> Square brackets were not yet allowed in SpamAssassin 3.3,
> and are stripped off in 3.4 (trunk) before passing to NetAddr.
> No need to use them here, square brackets around IPv6 address
> are only needed where an address is combined with a port number.

I'm using 3.4 from 4/27/2012, not 3.3.  Is that current enough for the bracket fix?

(In reply to comment #33)

> > netset: illegal network address given: '[fe80::%eth0]/64'
> 
> Scoped address (...%eth0) is currently not useful and I believe
> it is not recognized by NetAddr::IP. The reason is that the
> information about a peer address coming from a socket
> does not currently carry scope information, so something
> like %eth0 at the end of an address never reaches an application.
> For the time being just use non-scoped fe80::/10 .

This shouldn't have gotten generated in our installer bit that generates scope anyhow.  That I'll fix on my end. ;)

> > Square brackets were not yet allowed in SpamAssassin 3.3,
> > and are stripped off in 3.4 (trunk) before passing to NetAddr.
> > No need to use them here, square brackets around IPv6 address
> > are only needed where an address is combined with a port number.
> 
> I'm using 3.4 from 4/27/2012, not 3.3.  Is that current enough for the
> bracket fix?

I see - it only strips off square brackets and interface scope
when Net::Patricia is installed, but not when doing a sequential
search - which is why I haven't noticed a problem when trying
your example.

This should be (easily) improved before the release. Thanks
for stumbling across this. For the time being just leave out
square brackets around an IPv6 address in internal_networks
and trusted_networks.

Just to note, the %eth0 bit is from a bug in Postfix (that's where we pull the initial mynetworks bits from).

NetAddr::IP  4.066 which has just been pushed to CPAN may handle this situation better as well.

trunk:
  Bug 6681: add_cidr() to recognize and strip [] and %scope in IPv6 addresses
  - unifies parsing in non-Net::Patricia and Net::Patricia setups
  Sending lib/Mail/SpamAssassin/Message/Metadata/Received.pm
  Sending lib/Mail/SpamAssassin/NetSet.pm
Committed revision 1403578.