(In reply to comment #0)
> I may be having bad configuration ...

I guess you're right. ;)

Check you're debug output for the relevant metadata headers. There is no internal relay at all -- all of them are considered external.

dbg: metadata: X-Spam-Relays-Internal: 
dbg: metadata: X-Spam-Relays-External: [ ip=unknown rdns=
 helo=annwn23.rutgers.edu ...

The Received header:

Received: from unknown (HELO annwn23.rutgers.edu) (unknown)
  by unknown with SMTP; 21 Sep 2010 16:04:27 -0000

Two things to note.  (a) The receiving SMTP is in charge to add the sending hop's rDNS, SA doesn't do it itself, and relies on that trusted data. Moreover, even the sender's IP is missing.  (b) Once you got the SMTP fixed, be sure to correctly set up your trusted and internal networks settings.

(In reply to comment #1)
> (In reply to comment #0)
> > I may be having bad configuration ...
> 
> I guess you're right. ;)
> 

First Thank you for responding so quickly!
First, correction my typo. please assume  3.3.1, wherever you see 3.4.1.

 
I dont use the relay. I use "trusted_networks" and "internal_networks" variables. Even if I use  relays variables, I can not put in the ip for 'unknown' entry, can I? 

 
I realized that this is the problem with the server putting bad Received header. Unfortunately, I have no control over the SMTP server. Many times I send email to postmaster at certain SMTP server and they just refused to change it. My success rate is about 4 out of 10. 

Shouldn't SA handle this correctly especially when unknown is not last in the list? I did not see this issue with 3.2.4 and only see this in 3.3.1 

Thanks

Hanz

To make sure things are set, I do it twice so I can see the warning. 

Sep 21 17:13:21.466 [26116] warn: netset: cannot include 128.6.4.3/32 as it has already been included
Sep 21 17:13:21.467 [26116] warn: netset: cannot include 128.6.72.254/32 as it has already been included
Sep 21 17:13:21.468 [26116] warn: netset: cannot include 128.6.72.72/32 as it has already been included
Sep 21 17:13:21.468 [26116] warn: netset: cannot include 165.230.79.235/32 as it has already been included
Sep 21 17:13:21.469 [26116] warn: netset: cannot include 165.230.79.236/32 as it has already been included
Sep 21 17:13:21.470 [26116] warn: netset: cannot include 165.230.132.104/32 as it has already been included
Sep 21 17:13:21.471 [26116] warn: netset: cannot include 128.6.168.251/32 as it has already been included
Sep 21 17:13:21.472 [26116] warn: netset: cannot include 128.6.4.3/32 as it has already been included
Sep 21 17:13:21.473 [26116] warn: netset: cannot include 128.6.72.254/32 as it has already been included
Sep 21 17:13:21.473 [26116] warn: netset: cannot include 128.6.72.72/32 as it has already been included
Sep 21 17:13:21.474 [26116] warn: netset: cannot include 165.230.79.235/32 as it has already been included
Sep 21 17:13:21.475 [26116] warn: netset: cannot include 165.230.79.236/32 as it has already been included
Sep 21 17:13:21.476 [26116] warn: netset: cannot include 165.230.132.104/32 as it has already been included
Sep 21 17:13:21.477 [26116] warn: netset: cannot include 128.6.168.251/32 as it has already been included
Sep 21 17:13:21.478 [26116] warn: netset: cannot include 192.168.0.0/16 as it has already been included
Sep 21 17:13:21.479 [26116] warn: netset: cannot include 128.6.31.0/24 as it has already been included
Sep 21 17:13:21.480 [26116] warn: netset: cannot include 128.6.68.0/24 as it has already been included
Sep 21 17:13:21.481 [26116] warn: netset: cannot include 192.168.0.0/16 as it has already been included
Sep 21 17:13:21.482 [26116] warn: netset: cannot include 128.6.31.0/24 as it has already been included
Sep 21 17:13:21.483 [26116] warn: netset: cannot include 128.6.68.0/24 as it has already been included



However, it seems the debug still do not see  my trusted and internal network correctly. None seems to be internal. Could the 'unknown' IP in the header caused this to break?


Sep 21 17:13:23.333 [26116] dbg: received-header: parsed as [ ip=unknown rdns= helo=annwn23.rutgers.edu by=unknown ident= envfrom= intl=0 id= auth= msa=0 ]
Sep 21 17:13:23.335 [26116] dbg: received-header: relay unknown trusted? no internal? no msa? no
Sep 21 17:13:23.337 [26116] dbg: received-header: parsed as [ ip=127.0.0.1 rdns=localhost helo=localhost by=annwn23.rutgers.edu ident= envfrom= intl=0 id=34D7CABAC9 auth= msa=0 ]
Sep 21 17:13:23.337 [26116] dbg: received-header: relay 127.0.0.1 trusted? no internal? no msa? no
Sep 21 17:13:23.338 [26116] dbg: received-header: parsed as [ ip=127.0.0.1 rdns= helo=annwn23.rutgers.edu by=localhost ident= envfrom= intl=0 id=Fg0AoED3KKGU auth= msa=0 ]
Sep 21 17:13:23.338 [26116] dbg: received-header: relay 127.0.0.1 trusted? no internal? no msa? no
Sep 21 17:13:23.339 [26116] dbg: received-header: parsed as [ ip=128.6.31.86 rdns=rulink2.rutgers.edu helo=newsuit.rutgers.edu by=annwn23.rutgers.edu ident= envfrom= intl=0 id=E042CABAC5 auth= msa=0 ]
Sep 21 17:13:23.339 [26116] dbg: received-header: relay 128.6.31.86 trusted? no internal? no msa? no
Sep 21 17:13:23.340 [26116] dbg: received-header: parsed as [ ip=128.6.68.237 rdns= helo=annwn32.rutgers.edu by=rulink.rutgers.edu ident= envfrom= intl=0 id=0L9300A7KTZEDFD0@rulink.rutgers.edu auth= msa=0 ]
Sep 21 17:13:23.340 [26116] dbg: received-header: relay 128.6.68.237 trusted? no internal? no msa? no
Sep 21 17:13:23.341 [26116] dbg: received-header: parsed as [ ip=127.0.0.1 rdns=localhost helo=localhost by=annwn32.rutgers.edu ident= envfrom= intl=0 id=B192B3FE01B auth= msa=0 ]
Sep 21 17:13:23.341 [26116] dbg: received-header: relay 127.0.0.1 trusted? no internal? no msa? no
Sep 21 17:13:23.342 [26116] dbg: received-header: parsed as [ ip=127.0.0.1 rdns= helo=annwn32.rutgers.edu by=localhost ident= envfrom= intl=0 id=FUbSBwmT7zCC auth= msa=0 ]
Sep 21 17:13:23.342 [26116] dbg: received-header: relay 127.0.0.1 trusted? no internal? no msa? no
Sep 21 17:13:23.343 [26116] dbg: received-header: parsed as [ ip=128.6.31.85 rdns=rulink.rutgers.edu helo=newsuit.rutgers.edu by=annwn32.rutgers.edu ident= envfrom= intl=0 id=45BF83FE016 auth= msa=0 ]
Sep 21 17:13:23.343 [26116] dbg: received-header: relay 128.6.31.85 trusted? no internal? no msa? no
Sep 21 17:13:23.344 [26116] dbg: received-header: parsed as [ ip=188.186.137.158 rdns= helo=net137.186.188-158.dynamic.omsk.ertelecom.ru by=rulink.rutgers.edu ident= envfrom= intl=0 id=0L9300AAATZ9KMA0@rulink.rutgers.edu auth= msa=0 ]
Sep 21 17:13:23.344 [26116] dbg: received-header: relay 188.186.137.158 trusted? no internal? no msa? no
Sep 21 17:13:23.345 [26116] dbg: received-header: parsed as [ ip=66.231.92.249 rdns=elvnsvj.royalbowling.com helo=2xmc0a.royalbowling.com by=p.nsm.ctmail.com ident= envfrom= intl=0 id=BECF00CFF8 auth= msa=0 ]


Here is what I put:

#internal_networks !0/0
internal_networks 128.6.4.3 128.6.72.254 128.6.72.72  165.230.79.235 165.230.79.236  165.230.132.104  128.6.168.251

#local RU
trusted_networks  128.6.4.3 128.6.72.254 128.6.72.72  165.230.79.235 165.230.79.236  165.230.132.104  128.6.168.251

#RUlink machines

internal_networks   192.168/16 128.6.31/24  128.6.68/24
trusted_networks    192.168/16 128.6.31/24  128.6.68/24

(In reply to comment #3)
> However, it seems the debug still do not see  my trusted and internal network
> correctly. None seems to be internal. Could the 'unknown' IP in the header
> caused this to break?

Yes, that is exactly what I said. Or meant to say, anyway. ;)

The last relay does not record the sender's IP, nor rDNS. It even fails to add it's own rDNS. I believe this is precisely, where things break. SA cannot include that hop in its trustpath, and thus earlier relays will not be added either.

I believe this is not a bug and should be closed, but will keep it dangling just in case for a few days.

(For the record, non-bug configuration issues should better be brought up on the mailing lists. This is a bug-tracker, not a support source.)

> Sep 21 17:13:23.333 [26116] dbg: received-header: parsed as [ ip=unknown rdns=
> helo=annwn23.rutgers.edu by=unknown ident= envfrom= intl=0 id= auth= msa=0 ]
> Sep 21 17:13:23.335 [26116] dbg: received-header: relay unknown trusted? no
> internal? no msa? no


(In reply to comment #2)
> I realized that this is the problem with the server putting bad Received
> header. Unfortunately, I have no control over the SMTP server.

Uhm, that's the second top-most Received header, right before qmail invocation. You do not have control over that server?

And BTW, if you really do not have any control over that server and cannot have it fixed, you cannot really trust it to insert correct data, can you?

> I believe this is not a bug and should be closed, but will keep it dangling
> just in case for a few days.
> 
> (For the record, non-bug configuration issues should better be brought up on
> the mailing lists. This is a bug-tracker, not a support source.)

To clarify this -- I do not mean to rap your knuckles, and didn't even mention this in comment 1, because it was clear from your original report that you suspected to have found a regression bug. That's perfectly fine to bring to our attention immediately and file here as a bug report. :)

(In reply to comment #5)
> > I believe this is not a bug and should be closed, but will keep it dangling
> > just in case for a few days.
> > 
> > (For the record, non-bug configuration issues should better be brought up on
> > the mailing lists. This is a bug-tracker, not a support source.)
> 
> To clarify this -- I do not mean to rap your knuckles, and didn't even mention
> this in comment 1, because it was clear from your original report that you
> suspected to have found a regression bug. That's perfectly fine to bring to our
> attention immediately and file here as a bug report. :)


Understood! and I did not think it was a support question.

To answer your questions, No I do not have control/access over the other server(s) which generating the 'unknown' received header. Is this consider illegal header? Your comment may help me convince them to fix it.

I just wish that SA trusted_networks and internal_network are not that easy to break. I thought I would report them just incase you need to know.

If SA DevTeam is not going to fix it, I guest I will edit Received.pm file myself and add my own case.

Thanks.
Hanz

> And BTW, if you really do not have any control over that server and cannot have
> it fixed, you cannot really trust it to insert correct data, can you?


I finally get the other end to fix their servers. Now this is no longer an issue at least with this server. I am still wondering if such mistakes in the header 'should' cause a total break down in the trusted and internal network checks of spamassassin.

Thank you for your help.

Hanz

It would be nice if it wasn't so easy to break, but without manual configuration, the trust-path auto-guesser really just has to make educated guesses about your network. Over the years this has been discussed many, many, many times, and really it all boils down to there is no algorithm that works well in all cases. There's just too little information present to not be a fragile guess.

If you can think of a better algorithm, suggestions are welcome.

The best methods boil down to one of two algorithms:

Currently used:
Starting with the most recent (chronologically) relay:

1) If the most recent is a private IP, assume internal and assume all other private IPs are internal until you get to the first publicly routable address.

2) Assume the most recent publicly routable is your MX.

Pros: works well with properly configured sites and a un-natted MX
downside: breaks with broken sites, or with nated MX.

Alternate algorithm:

Same as above, but in step 2, assume the routable IP is external.

Pros: works well with properly configured sites and a NATed MX
downside: breaks with broken sites, or with un-nated MX.


Other popular proposals:

-Use RDNS and trust all the recent hosts that match the domain of the To: address
downside: breaks for multi-domain sites, Cc, BCC, and mailing list messages

-Use RDNS and trust all the recent hosts that are in the same domain as one another.
downside: breaks for anyone using a forwarder, or servers in multiple domains. Breaks in the same way as current for sites failing to add a received: header

-Consider nothing trusted/internal unless explicitly configured to do so
downside: This ensures SA is always broken by default. So rather than guess, we're making a hard-coded choice that is always wrong.. not so good. (yes, under-trust is also bad, in some cases worse than over-trust, see the wiki on trust path).

Looks like this should be closed.  The problem was a bad Received header due to MTA configuration.

Somebody please close this.  Not a problem in SA.

Configuration issue not a bug as noted in comments.