Bug 6501

Summary: RCVD_IN_PBL wrongly catching IP addresses in X-Originating-IP header
Product: Spamassassin Reporter: Andrew W <andrew.lists1>
Component: RulesAssignee: SpamAssassin Developer Mailing List <dev>
Status: RESOLVED INVALID    
Severity: normal CC: andrew.lists1, dominik, hannu.teulahti, jason, john, mss, noroutetohorst, tebruno99, tonster, ws.apache
Priority: P2    
Version: 3.3.1   
Target Milestone: Undefined   
Hardware: PC   
OS: Linux   
Whiteboard:

Description Andrew W 2010-10-19 19:42:56 UTC
The Spamhaus PBL is a list of IP addresses that shouldn't send unauthenticated SMTP mail - such as DSL subscribers.

Webmail applications add the X-Originating-IP header with the IP address that a webmail user is logged in from when sending mail.

Currently, if a webmail user sends a message through a host running SpamAssassin, the RCVD_IN_PBL rule is triggered, upping the spam score.

According to the documentation, X-Originating-IP addresses "are virtually appended into the Received: chain, so they are used in RBL checks where appropriate", however I don't believe PBL lists are appropriate for this.


Example: These headers (sanitised for privacy) are from a message sent using a Zimbra web client, from a user logged in from home, using an IP address his ISP has listed in the Spamhaus PBL at http://www.spamhaus.org/pbl/query/PBL255647 (the whole subnet 122.57.0.0/16). Zimbra appends the X-Originating-IP header (since switched off as a workaround).

Note that SpamAssassin used here is installed from CPAN on a host separate from the Zimbra server, running with qmail.

==============================================================================
Received: from zcs.(mydomain) (192.168.1.x)
  by mail.(mydomain) with SMTP; 17 Oct 2010 14:50:55 +1300
Received: from localhost (localhost [127.0.0.1])
        by zcs.(mydomain) (Postfix) with ESMTP id C8FF63C321
        for <bret@(recipientdomain)>; Sun, 17 Oct 2010 14:50:52 +1300 (NZDT)
Received: from zcs.(mydomain) ([127.0.0.1])
        by localhost (zcs.(mydomain) [127.0.0.1]) (amavisd-new, port 10024)
        with ESMTP id ouK02YHo0Yib for <bret@(recipientdomain)>;
        Sun, 17 Oct 2010 14:50:51 +1300 (NZDT)
Received: from zcs.(mydomain) (zcs.(mydomain) [127.0.0.2])
        by zcs.(mydomain) (Postfix) with ESMTP id 436C43C31E
        for <bret@(recipientdomain)>; Sun, 17 Oct 2010 14:50:51 +1300 (NZDT)
Date: Sun, 17 Oct 2010 14:50:50 +1300 (NZDT)
From: Hamish (surname) <hamish@(senderdomain)>
To: Bret (surname) <bret@(recipientdomain)>
Message-ID: <29966613.45.1287280250401.JavaMail.root@zcs>
In-Reply-To: <17276425.42.1287280152224.JavaMail.root@zcs>
Subject: (subject line was here)
MIME-Version: 1.0
Content-Type: multipart/mixed;
        boundary="----=_Part_44_23777274.1287280250388"
X-Originating-IP: [122.57.x.y]
X-Mailer: Zimbra 6.0.7_GA_2473.UBUNTU8 (ZimbraWebClient - IE8 (Win)/6.0.7_GA_2473.UBUNTU8)
==============================================================================

Spamassassin scores this message as follows (relevant lines only):
 pts rule name              description
---- ---------------------- --------------------------------------------------
 3.3 RCVD_IN_PBL            RBL: Received via a relay in Spamhaus PBL
                            [122.57.x.y listed in zen.spamhaus.org]
==============================================================================

Note that the originating IP address is not in any Received header.

I believe that scoring X-Originating-IP address in PBL (not RBL) checks is a bug - it is scoring authenticated HTTP originated mail as if it were unauthenticated SMTP mail.

Using "clear_originating_ip_headers" as a workaround seems to have no effect - could be Bug 6500.
Comment 1 Karsten Bräckelmann 2010-10-19 20:01:00 UTC
(In reply to comment #0)
> Currently, if a webmail user sends a message through a host running
> SpamAssassin, the RCVD_IN_PBL rule is triggered, upping the spam score.

This sounds like you are scanning outbound mail -- or inbound mail from your own users, to your users, which equals outbound for the sender.

SA does *not* do deep-header parsing for DNS BLs that are not meant for this. That includes PBL, which is only checked against the untrusted / external handing-over IP address.

Your sample indicates the message indeed has been sent from a dial-up IP directly to your network.

The common cure for this is to use authentication.

Note, that in your case, while having the X-Originating-IP header, there is no Received header even claiming the message was submitted via HTTP. But ESMTP (not authenticated).


I believe this to be a trusted / internal networks configuration issue, or missing authentication. Not a bug.
Comment 2 Andrew W 2010-10-19 20:41:07 UTC
(In reply to comment #1)
> (In reply to comment #0)
> > Currently, if a webmail user sends a message through a host running
> > SpamAssassin, the RCVD_IN_PBL rule is triggered, upping the spam score.
> 
> This sounds like you are scanning outbound mail -- or inbound mail from your
> own users, to your users, which equals outbound for the sender.

Yes, as extra insurance to ensure I do not want to fall foul of my upstream vendor's spam policy.

The path is:

client via the public internet uses HTTPS connection to Zimbra webmail on my network;
Zimbra then queues it up and delivers it to my qmail host with spamassassin;
qmail host scans then continues delivery via my ISP.

> SA does *not* do deep-header parsing for DNS BLs that are not meant for this.
> That includes PBL, which is only checked against the untrusted / external
> handing-over IP address.

> Your sample indicates the message indeed has been sent from a dial-up IP
> directly to your network.

A webmail user, logged in from home on the PBL'd address, sends a message. The webmail host is on my network, as is this mailserver, both on the same private subnet. It has not passed through any dial-up users via SMTP.
 
> The common cure for this is to use authentication.

It's webmail, therefore authenticated? SMTP is not authenticated between the webmail host and the qmail host it is via a trusted connection over my private subnet.

> Note, that in your case, while having the X-Originating-IP header, there is no
> Received header even claiming the message was submitted via HTTP. But ESMTP
> (not authenticated).

That's Zimbra queueing to itself before sending.

> I believe this to be a trusted / internal networks configuration issue, or
> missing authentication. Not a bug.

I have a non-spamassassin-scanned copy of the original e-mail saved as a file for the purpose of running spamassassin -t against it. It behaves as above.

If I make a copy of that file, and remove only the X-Originating-IP header, then the PBL rule doesn't get applied. I even get "ALL_TRUSTED" instead.

So the problem goes away if the X-Originating-IP header goes away, which is what led me to believe this is a bug. Should that happen if it were a authentication issue?
Comment 3 Karsten Bräckelmann 2010-10-19 22:13:25 UTC
(In reply to comment #2)

> A webmail user, logged in from home on the PBL'd address, sends a message. The
> webmail host is on my network, as is this mailserver, both on the same private
> subnet. It has not passed through any dial-up users via SMTP.

This is irrelevant. PBL is being checked against your perimeter, not whether it is passed through a PBL listed IP.

> > The common cure for this is to use authentication.
> 
> It's webmail, therefore authenticated? SMTP is not authenticated between the
> webmail host and the qmail host it is via a trusted connection over my private
> subnet.

There is no Received header indicating any kind of authentication.

> > I believe this to be a trusted / internal networks configuration issue, or
> > missing authentication. Not a bug.

> So the problem goes away if the X-Originating-IP header goes away, which is
> what led me to believe this is a bug. Should that happen if it were a
> authentication issue?

Yes. If it is an authenticated submission, as indicated in the headers, the IP would not be checked against PBL and you would indeed get an ALL_TRUSTED hit in your regular scanning.

There is no hint in the headers that the X-Original-IP is an authenticated or otherwise trusted source. That's why SA checks it.
Comment 4 Hannu Teulahti 2010-10-20 05:35:32 UTC
We have this same issue. Our M$ exchange web interface adds x-originating-ip header to emails, so those users who are in spamhaus pbl networks cannot get their emails trough. I think this changed at the last upgraqde to M$ exchange.
Clearly a bug.
Comment 5 Andrew W 2010-10-21 04:36:33 UTC
I can understand why an X-Originating-IP address is checked against RBL lists, but what is the rationale of checking them against PBLs if this is supposedly not a bug?

RBL = host known to be spammy.
PBL listing = host or whole subnet that shouldn't send unauthenticated SMTP.
A webmail client is not an unauthenticated SMTP client, so shouldn't this mean it should not be checked against PBLs?
Comment 6 Jason Bertoch 2010-10-21 14:40:40 UTC
I would expect the clear_originating_ip_headers configuration directive to fix your problem, but be careful of Bug 6500.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6500
Comment 7 John Wilcock 2010-10-22 02:15:04 UTC
(In reply to comment #6)
> I would expect the clear_originating_ip_headers configuration directive to fix
> your problem

Well, it would probably work around his problem, yes, at the expense of ignoring all X-Originating-IP headers in all incoming messages. 

AFAICT spamassassin is behaving exactly as expected, since nothing in the OP's headers indicates to SA that the webmail is authenticated. Fix that, and the problem will go away. 

I'd suggest setting this bug to INVALID (but I don't have the privs to do so myself).
Comment 8 mak 2010-11-22 07:30:11 UTC
I can confirm this problem and think this is a serious bug, because it significantly raises the number of false positives.

(In reply to comment #7)

> AFAICT spamassassin is behaving exactly as expected, since nothing in the OP's
> headers indicates to SA that the webmail is authenticated. Fix that, and the
> problem will go away. 

Is there any rule in the email related RFCs that says, that there has to be any header information about authentication?
If yes, please tell me where.
If no, this is a bug and has to be fixed, as the X-Originating-IP header is by no means relevant for checking against blacklists. Only the Received headers are.
Comment 9 Roman 2011-10-22 22:49:30 UTC
(In reply to comment #8)
> Is there any rule in the email related RFCs that says, that there has to be any
> header information about authentication?
> If yes, please tell me where.

Yes, it's called RFC3848, which I learned about from this post http://www.gossamer-threads.com/lists/spamassassin/users/162165#162165

I believe the only MTA that doesn't support this RFC and is still in widespread use is qmail. There is a patch for qmail but you'll have to apply it yourself.

Leaving aside rants at how rubbish this whole thing is, I'd argue that this is not a bug in SpamAssassin indeed.

P.S. I've been sending such "buggy" email for years and it seems that it never gets filtered as spam, which suggests that most people don't use rules like RCVD_IN_PBL when filtering spam. This might be why unpatched qmail remains usable to this day.
Comment 10 Karsten Bräckelmann 2011-10-24 20:04:53 UTC
Not a bug, closing RESOLVED INVALID, as per the discussion.

SA behaves exactly as expected. The problem is with scanning outbound mail of ones own users and local SMTP configuration, which fails to properly indicate the client was authenticated.

This is a local issue with outbound mail only. RCVD_IN_PBL will not match on the recipient SMTP, in fact the originating IP will not even be checked against PBL there.
Comment 11 Thomas 2012-01-26 17:35:32 UTC
Webmail servers typically sit behind their MTA servers and use trusted network to send mail via the MTA.  There isn't a way in services like Zimbra to configure webmail to produce smtp auth headers into webmail messages.  Nor should they because smtp auth did not occur.  The Webmail server is a Trusted sender of mail.

What do you recommend we do?
Comment 12 Hannu Teulahti 2012-01-26 18:26:31 UTC
We have this in our rules. seem to fix this problem:

# https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6501
# "it's not a bug"
#
score RCVD_IN_PBL 0
score RCVD_IN_RP_RNBL 0
Comment 13 Malte S. Stretz 2012-10-11 14:34:29 UTC
For the Zimbra case, I filed this bug (which suggests adding a faked LMTPA header):
https://bugzilla.zimbra.com/show_bug.cgi?id=77949