Subject: Re: [SAdev]  New: FORGED_MUA_AOL is wrong (and
  expensive)

Well you censored a LOT of headers, so it's hard to say what's going on. 
You also didn't include the results from a run of SA, although you indicate 
2.53 as the version.

As far as the 'AOL Communicator N2 Beta' goes, that shouldn't be it. Really 
the forged AOL rule do not look at the x-mailer header at all. It looks at 
hostnames and ESMTP IDs.

Looking at the rule in 2.52/2.53 It's expecting to see a mail relay 
starting with rly- that one starts with air-. The ESMTP ID in the message 
you got should work fine, it's just the air-* that's killing it.

 From Evaltests.pm	

   if (/ rly-[a-z][a-z]\d\d\./i) {
     return 0 if /\/AOL-\d+\.\d+\.\d+\)/;    # via AOL mail relay
     return 0 if /ESMTP id (?:RELAY|MAILRELAY|MAILIN)/; # AOLish
     return 1;
   }

and the header in question:

Received: from  XP_01 (dsl-105.132.212.155.conversent.net 
[155.212.132.105]) by
air-id12.mx.aol.com (v92.17) with ESMTP id MAILINID122-3eee3e91c63a167; Mon, 07

It's quite strange that you chose to censor the server that the mail was 
going from while still within AOL's network in this header:

Received: from [ deleted ]
	by imo-d03.mx.aol.com (mail_out_v34.21.) id u.d.dcb92f1 (16110)
	for <deleted>; Mon, 7 Apr 2003 14:40:58 -0400 (EDT)

Was that really anything other than air-id12.mx.aol.com??

Since this test is Received path oriented, could you possibly include a 
less-censored Received: trail? (ie: including all of the AOL server names 
and ESMTP id's uncensored?)

It's probably fine to censor email addresses and non-aol servers, but to 
evaluate this message the developers will probably at LEAST need all the 
aol servernames and ESMTP id's intact.

Subject: Re:  FORGED_MUA_AOL is wrong (and expensive)

Sorry, I wasn't trying to be unhelpful.  Full headers with only names/addrs 
removed are below.  But I don't think they are relevant.

The Message-ID header is "Message-ID: 
<3E91C621.3010903@aol.com>".  20_ratware.cf says

# AOL
header __AOL_MUA                X-Mailer =~ /\bAOL\b/
header __AOL_MSGID              MESSAGEID =~ 
/^<[0-9a-f]{1,3}\.[0-9a-f]{6,8}\.[0-9a-f]{8}\@aol.com>$/m
meta FORGED_MUA_AOL             (__AOL_MUA && !__UNUSABLE_MSGID && 
!__AOL_MSGID)
describe FORGED_MUA_AOL         Forged mail pretending to be from AOL

__AOL_MUA is correctly true.  The rule __AOL_MSGID is incorrectly false; it 
does not match the supplied Message-ID, even though I believe this message 
was not forged.  Therefore, the __AOL_MSGID rule is making an invalid 
assumption about the format of legal AOL Message-IDs.  As a result, the 
FORGED_MUA_AOL meta-rule is incorrectly matching the message.

Thanks,

Barry

--- snip ---



Return-Path: <[ deleted ]>
Received: from po12.mit.edu (po12.mit.edu [18.7.21.71])
	by po12.mit.edu (Cyrus v2.1.5) with LMTP; Mon, 07 Apr 2003 14:48:24 -0400
X-Sieve: CMU Sieve 2.2
Received: from pacific-carrier-annex.mit.edu by po12.mit.edu (8.12.4/4.7) 
id h37ImHhL025121; Mon, 7 Apr 2003 14:48:23 -0400 (EDT)
Received: from TheWorld.com (pcls2.std.com [199.172.62.104])
	by pacific-carrier-annex.mit.edu (8.12.4/8.9.2) with ESMTP id h37IhG5u028698
	for <[ deleted ]>; Mon, 7 Apr 2003 14:43:17 -0400 (EDT)
Received: from europe.std.com (europe.std.com [199.172.62.20])
	by TheWorld.com (8.12.8p1/8.12.8) with ESMTP id h37IgvdI010930;
	Mon, 7 Apr 2003 14:43:00 -0400
Received: (from daemon@localhost)
	by europe.std.com (8.9.3/8.9.3) id OAA03731
	for [ deleted ]; Mon, 7 Apr 2003 14:42:53 -0400 (EDT)
Received: from TheWorld.com (pcls4.std.com [199.172.62.106])
	by europe.std.com (8.9.3/8.9.3) with ESMTP id OAA03325
	for <[ deleted ]>; Mon, 7 Apr 2003 14:41:23 -0400 (EDT)
Received: from imo-d03.mx.aol.com (imo-d03.mx.aol.com [205.188.157.35])
	by TheWorld.com (8.12.8p1/8.12.8) with ESMTP id h37IfLxv032644
	for <[ deleted ]>; Mon, 7 Apr 2003 14:41:22 -0400
Received: from [ deleted ]
	by imo-d03.mx.aol.com (mail_out_v34.21.) id u.d.dcb92f1 (16110)
	 for <[ deleted ]>; Mon, 7 Apr 2003 14:40:58 -0400 (EDT)
Received: from  XP_01 (dsl-105.132.212.155.conversent.net 
[155.212.132.105]) by air-id12.mx.aol.com (v92.17) with ESMTP id 
MAILINID122-3eee3e91c63a167; Mon, 07 Apr 2003 14:40:58 -0500
Date: Mon, 7 Apr 2003 14:40:33 -0400
From: "[ deleted ]" <[ deleted ]>
Subject: Griffelkin
To: [ deleted ]
Message-ID: <3E91C621.3010903@aol.com>
X-Mailer: AOL Communicator N2 Beta (20030129.1)
MIME-Version: 1.0
Content-Type: TEXT/HTML; CHARSET=us-ascii
Sender: [ deleted ]
Precedence: list
Reply-To: "[ deleted ]" <[ deleted ]>

Subject: Re: [SAdev]  FORGED_MUA_AOL is wrong (and expensive)

At 01:44 PM 4/7/2003 -0700, you wrote:
>Sorry, I wasn't trying to be unhelpful.  Full headers with only names/addrs
>removed are below.  But I don't think they are relevant.

I'm sorry too, I misread that as FORGED_AOL_RCVD.. not MUA. You were 
correct in stripping them down in the first place.. Doh!


>The Message-ID header is "Message-ID:
><3E91C621.3010903@aol.com>".  20_ratware.cf says

<snip>

>  As a result, the
>FORGED_MUA_AOL meta-rule is incorrectly matching the message.

You are correct, it does appear there needs to be some strong revisting of 
the AOL message ID checks.

  That message has a message ID of the format:
<8 hex digits>.<7 decimal or hex digits>@aol.com

Also of note, I've got a legit looking AOL message in my mailboxes with 
this format of ID:

Message-ID: <7139AE4B.1C887510.48628178@aol.com>
X-Mailer: Atlas Mailer 2.0
Date: Thu, 06 Mar 2003 10:22:47 -0500

Which is
<8hex digits>.<8hex digits>.<8 decimal or hex digits>@aol.com

It was sent via an AOL mail relay through a sourceforge list (snort-users) 

Subject: Re: [SAdev]  New: FORGED_MUA_AOL is wrong (and expensive) 


> Received: [ many deleted ]
> Received: from [ deleted ]
> 	by imo-d03.mx.aol.com (mail_out_v34.21.) id u.d.dcb92f1 (16110)
> 	 for <deleted>; Mon, 7 Apr 2003 14:40:58 -0400 (EDT)
> Received: from  XP_01 (dsl-105.132.212.155.conversent.net [155.212.132.105]) 
> by > air-id12.mx.aol.com (v92.17) with ESMTP id MAILINID122-3eee3e91c63a167; Mon, 
> 07 > Apr 2003 14:40:58 -0500

Barry -- as a matter of interest, I'm working on better anti-AOL-forgery
rules.  Could you fwd on those deleted Received headers?

Subject: Re:  FORGED_MUA_AOL is wrong (and expensive)

Antony,

My previous message on this ticket dated 2003-04-07 13:44 contained the 
full headers with only addresses removed.

Thanks,

Barry

I'm getting hit by false positives on the AOL MUA rule all the time.

My guess is that the AOL MUA detection engine only works for people who
use the fat AOL client as MUA.

However, you can also access AOL Mailboxes using:
 - Netscape 7
 - AOL Communicator Preview Release "Photon"
 - AOL Anywhere webmail at http://www.aol.com/

Let me know what is needed to get the AOL MUA detection rule fixed.
I can provide complete headers for all AOL scenarios above +
traditional AOL "fat" client, and there may be useful information
on the AOL MUA at http://postmaster.info.aol.com/

Thanks & Cheers,
-- Nicolas

Nicolas, could you attach some sample messages for each type of AOL client, with
all headers included?  (as attachments btw).  thx.

BTW "Anthony Mawer" == me, bug in bugzilla ;)

Will do before Tuesday.

However, note that AOL customers can now access their mailboxes using *any*
Mail User Agent supporting IMAP4 + ESMTP AUTH (authenticated SMTP)
using the following settings:
  IMAP4 server: imap.cs.com (aka imap.fr.aol.com or imap.de.aol.com, CNAMEs)
  SMTP server:  smtp.cs.com (aka smtp.fr.aol.com or smtp.de.aol.com, CNAMEs)

and, if needed, going into advanced SMTP options and checking
"This SMTP server requires authentication" and providing the AOL account
username and password.

So basically if the Message-ID is created by the MUA (versus the MSA/MTA),
any approach based on detecting a pattern in the Message-ID is bound to
failure.

The only approach that could work would be based on the hostnames of the
AOL MSA/MTAs, and these depend on whether you use the fat AOL client,
Netscape 7 / AOL Communicator, the AOL WebMail system,
or any IMAP4/ESMTP AUTH-compatible MUA.

Let me know :-)

Created attachment 1015 [details]
from "fat" AOL 8 FR client, to an AOL recipient

Created attachment 1016 [details]
from "fat" AOL 8 FR client, to an Internet recipient

Created attachment 1017 [details]
from AOL WebMail to an AOL recipient

Created attachment 1018 [details]
from AOL WebMail to an Internet recipient

Created attachment 1019 [details]
from AOL Communicator "Photon" (en_us), to an AOL recipient

Created attachment 1020 [details]
from AOL Communicator "Photon" (en_us), to an Internet recipient

Created attachment 1021 [details]
from Netscape 7 (en_us) to an AOL recipient

Created attachment 1022 [details]
from Netscape 7 (en_us) to an Internet recipient

Created attachment 1023 [details]
from Outlook Express 6 via smtp.fr.aol.com to an AOL recipient

Created attachment 1024 [details]
from Outlook Express 6 via smtp.fr.aol.com to an Internet recipient

*** Bug 1989 has been marked as a duplicate of this bug. ***

Here is some additional information that may prove useful in verifying that an
email is really originating from an AOL member:

The ranges of IP addresses dynamically assigned to AOL members are listed at:
  http://webmaster.info.aol.com/proxyinfo.html

Look under the "AOL Client IP Addresses" section for the complete range list.

To combat AOL subscribers trying to use spamtools to open TCP connections to
remote ports 25 and "deliver to MX" directly, AOL has introduced a feature in
North America where outgoing connections (from client IP addresses) to remote
port 25/tcp are transparently redirected to an AOL SMTP server which will

 - accept the email and deliver it

 - enforce SMTP "rate control" -- ensuring that not more than one outgoing SMTP
connection is opened from a given IP address at a given time, to prevent
mass-parallelization of spam

 - add a Received: header listing the real screenname/email address of the aol
member opening the tcp connection, such as:

Received: from AOLSender@aol.com
	by imo-r05.mx.aol.com (mail_out_v36.3.) id 3.99.38d12fce (4230)

This port 25 filtering has been so successful to curb spam that it is about to
be implemented to all of AOL (not only North America).

Also note that AOL Mail servers are under the .mx.aol.com domain,
so maybe this information could be used to construct a more robust filter.

When the port 25 filtering is enabled world-wide (anytime now),
no SMTP connection from an AOL member can go out unless it goes through
some AOL mail server under .mx.aol.com first.

PS: another major source of AOL-originated spam recently was the AOL webmail,
used for both AOL anywhere at http://www.aol.com/
and the free Netscape webmail at http://webmail.netscape.com/
(netscape.net addresses)

This webmail adds a header: X-Mailer: Atlas Mailer 2.0
as user-agent, as can be read in the attached samples.

AOL WebMail servers are under the .webmail.aol.com domain I believe.

Unfortunately there are also legitimate users that use this webmail system :-/

PPS: more information about AOL Mail infrastructure is publicly available
at http://postmaster.info.aol.com/

hmm.
looks like we may have to drop the FORGED_AOL rules for 2.60 :(

yeah.  either at this point we need to drop it or just accept the fps.

I think I'd rather punt it to broken rules and fix it for 2.70 or something.

+1 for punting :-)

ok, punting.

not sure why this bug is mine, so ...

*** Bug 2125 has been marked as a duplicate of this bug. ***

marking LATER, since it's in broken_tests anyway.

Created attachment 1426 [details]
from AOL Communicator RTM (final, en_us) to an Internet recipient

This is an updated email sample sent from AOL Communicator final version
(August 2003 US-English release)

Localized versions of this product are currently being released too,
let me know if you'd like a sample email sent from a French AOL Communicator
for instance.

It looks like this latest message I have attached (attachment 1426 [details])
is triggering rule FORGED_AOL_HTML which is really expensive (+3.3 spam points)
in SpamAssassin 2.60 Final RTM

so even if FORGED_MUA_AOL has been disabled temporarily in 2.60,
AOL MUAs features and detection are still badly broken.

-> logging bug 2506 related to FORGED_AOL_HTML

This public web page lists the IP addresses of AOL Mail Servers,
and may prove very useful in finding out of an email comes from AOL or not:

http://postmaster.info.aol.com/servers.html

PS: shouldn't this bug be reopened as it was closed with a target fix version of
2.70 ?

reopening

ok, once 2506 is closed, this can be closed too.

the servers.html page has now moved to
http://postmaster.info.aol.com/info/servers.html .  Worth noting is that *all*
outgoing AOL mail will come from an IP with rDNS *.mx.aol.com; this could be a
good test.

Not sure if it'll catch all this spam, though -- since this rule also catches
cases where a spammer claims to be running the AOL client but sending through
mail.ru for example ;)

BTW, Nicolas -- can the AOL "fat" client ever mail through an ISP that is *not*
aol.com?  I would imagine not...

Nope... the "flagship" ( :-) ) AOL client still uses AOL's proprietary protocols
(FLAP/P3) inside a TCP connection to the AOL hosts to send or receive email, so
it doesn't use SMTP/IMAP/POP/whatever and has no choice nor any possible
configuration of mail servers. It's the "legacy" client.

FYI, Modern AOL MUAs such as AOL Communicator (or Netscape 7.1, or even AIM) use
an extended IMAP4 protocol, however they are still not "configurable" in the
traditional meaning: Members just go into the Mail Accounts Wizard, click on
"AOL Account", and simply enter their AOL login/password -- that's all.

The "modern" clients will use the DNS to locate the relevant AOL servers, login
on a "configuration" server and will retrieve all necessary parameters. There is
no way for the user to change or even view the configuration.

Side note: AOL has dramatically increased its spam fighting tools recently, and
is now refusing over 2.5 billion spams per day. This amount has increased
dramatically (linear increasing) over the last few months, while the amount of
"regular" email received from the Internet to AOL has remained stable (AOL
userbase has also remained stable over the last months, at roughly 30 million
accounts x 7 mailboxes per account max).

One of the latest AOL filtering rules I've discovered is that AOL will now
refuse accepting an email coming from the Internet if an envelope sender is set
to anything@aol.com

This means that if you intend to set the From: field to <anything>@aol.com, you
*must* use AOL servers to send it, otherwise you can't even write to other
@aol.com members (initially submitting your email via a third-party SMTP server
won't work).

Sorry for being that long...
I can provide updated emails sent from the following MUAs if needed:
 - "fat" AOL 9 US RTM client
 - "fat" AOL 9 FR beta7 client
 - more recent AOL Communicator builds, such as Final RTM FR or beta US

Hope this helps -- Cheers, N.

OK, that info will definitely provide a good rule; aol.com is frequently forged
as enveloper-from by spammers.

The samples from the other clients would be welcome too ;)

moving back to 2.70; the urgent bug is fixed, remaining stuff here is for new rules.

FYI, AOL Mail Operations people suggests checking for the IP addresses
of AOL Mail servers, as listed on
http://postmaster.info.aol.com/info/servers.html

They guarantee that these IP addresses will not change for a very long time.

Rationale is that anyone can place a reverse DNS on any IP address
that points at anything.mx.aol.com, even outside AOL.

Created attachment 1659 [details]
from "fat" AOL 9 US client, to an Internet recipient

mail sample from "fat" AOL 9 US client, to an Internet recipient

Created attachment 1660 [details]
from AOL Communicator US build 20030919.3 Win to internet recipient

from AOL Communicator US build 20030919.3 Win to internet recipient

http://postmaster.info.aol.com/info/servers.html

It would be nice if they captured this in an SPF DNS record for use by the new
SA SPF support. (See bug #2634.)

OK... if I understand SPF correctly, records should be:

aol.com. IN TXT "v=spf1 ip4:205.188.157.0/24 ip4:152.163.225.0/24
ip4:64.12.136.0/24 ip4:205.188.156.0/24 ip4:205.188.159.13/32 ip4:64.12.138.0/24
 -all"

According to
http://www.infinitepenguins.net/SPF/create.php
http://spf.pobox.com/

Will submit the request...

SPF records are in! (*rolls drum*)

% dig aol.com txt 

; <<>> DiG 9.2.2 <<>> aol.com txt
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51299
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 0

;; QUESTION SECTION:
;aol.com.                       IN      TXT

;; ANSWER SECTION:
aol.com.                300     IN      TXT     "v=spf1 ip4:152.163.225.0/24
ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/24
ip4:205.188.157.0/24 ip4:205.188.159.0/24 ip4:64.12.136.0/24 ip4:64.12.137.0/24
ip4:64.12.138.0/24 ptr:mx.aol.com -all"

;; AUTHORITY SECTION:
aol.com.                3600    IN      NS      dns-02.ns.aol.com.
aol.com.                3600    IN      NS      dns-06.ns.aol.com.
aol.com.                3600    IN      NS      dns-07.ns.aol.com.
aol.com.                3600    IN      NS      dns-01.ns.aol.com.


Anything else for your service ? :-)

Very nice!

We must do another mass-check using SPF to see what the results are like now...

AOL has removed the record for the weekend to avoid issues with legitimate
forwarding services, until those services can implement sender rewriting. See
the SPF mailing list.

FYI, the SPF record is back, albeit with a question mark, "?all" instead of "-all":

% dig aol.com txt

; <<>> DiG 9.2.2 <<>> aol.com txt
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5667
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 0

;; QUESTION SECTION:
;aol.com.                       IN      TXT

;; ANSWER SECTION:
aol.com.                269     IN      TXT     "v=spf1 ip4:152.163.225.0/24
ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/24
ip4:205.188.157.0/24 ip4:205.188.159.0/24 ip4:64.12.136.0/24 ip4:64.12.137.0/24
ip4:64.12.138.0/24 ptr:mx.aol.com ?all"

Quick suggestion:

since AOL has changed to "?all", meaning (I believe) undefined result if the
email doesn't come from one of the above IP ranges, I believe this could lessen
dramatically the effectiveness of this criteria to detect AOL-sender forgeries ?

Would it be possible to create a custom FORGET_AOL_SENDER rule with the AOL Mail
server IP address ranges hardcoded:

152.163.225.0/24
205.188.139.0/24
205.188.144.0/24
205.188.156.0/24
205.188.157.0/24
205.188.159.0/24
64.12.136.0/24
64.12.137.0/24
64.12.138.0/24

and use the genetic algorithm to compute the score of such a rule ?

I'd think it'd be much more effective than a SPF record with "?all" naïvely...
and it might be a "quick win" given the amount of forged spam that claims to
originate from aol.com...

Subject: Re:  useful data for AOL-forgery detection rules

> Would it be possible to create a custom FORGET_AOL_SENDER rule with
> the AOL Mail server IP address ranges hardcoded:

I don't think we want to start maintaining lists of IP addresses for
anything.

I'd rather have a custom rule for AOL that dinged the email if it wasn't
explicitly listed and fell-through to "?all".

Daniel

FYI, any AOL customer can now use any SMTP/IMAP4 compatible client as MUA
(supporting SMTP AUTH) -- complete instructions are posted at AOL keyword:
"IMAP" aol://1722:imap and the parameters are simply:

imap.aol.com
smtp.aol.com port 587, as per RFC 2476 paragraph 3.1

Thus, nothing should be assumed on the MUA used to compose messages issued from
a sender @aol.com.

Finally, a major update to http://postmaster.info.aol.com/ has just been
released, and the technical information related to AOL Mail infrastructure is
now detailed at http://postmaster.info.aol.com/servers/

Subject: Re:  useful data for AOL-forgery detection rules 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


bugzilla-daemon@bugzilla.spamassassin.org writes:
> FYI, any AOL customer can now use any SMTP/IMAP4 compatible client as
> MUA (supporting SMTP AUTH) -- complete instructions are posted at AOL
> keyword: "IMAP" aol://1722:imap and the parameters are simply:
> 
> imap.aol.com smtp.aol.com port 587, as per RFC 2476 paragraph 3.1
> 
> Thus, nothing should be assumed on the MUA used to compose messages
> issued from a sender @aol.com.

Yes.  that should be OK -- as long as the clients don't fake the
X-Mailer to appear to be AOL clients, we shouldn't misfire I think.

> Finally, a major update to http://postmaster.info.aol.com/ has just been
> released, and the technical information related to AOL Mail
> infrastructure is now detailed at
> http://postmaster.info.aol.com/servers/

excellent...

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFAiDOIQTcbUG5Y7woRAtRhAKDP/eIMU8OpZyODiDs2QVZw/Gz8zQCg0BYv
lGCbVQ8/DrM1AoLXy2l582Y=
=9XpZ
-----END PGP SIGNATURE-----

more accuracy and performance bugs going to 3.1.0 milestone

No longer an issue, we dont have a rule for AOL msgids