Bug 2759 - [review] Received parser can't deal with qmail lines with IDENT info
Summary: [review] Received parser can't deal with qmail lines with IDENT info
Status: RESOLVED FIXED
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Libraries (show other bugs)
Version: 2.60
Hardware: Other other
: P3 minor
Target Milestone: 2.61
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2003-11-14 19:35 UTC by Justin Mason
Modified: 2003-11-19 14:23 UTC (History)
0 users


Attachment Type Modified Status Actions Submitter/CLA Status
attempt at a fix patch None Justin Mason [HasCLA]
Patch for qmail Received: with IDENT patch None Michel Bouissou [HasCLA]
Comprehensive qmail Received: headers patch for Received.pm patch None Michel Bouissou [HasCLA]
Newer version of comprehensive qmail Received: headers patch for Received.pm patch None Michel Bouissou [HasCLA]
Show Obsolete (2) Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Justin Mason 2003-11-14 19:35:17 UTC 
split out from http://bugzilla.spamassassin.org/show_bug.cgi?id=2744#c12 --

Michel Bouissou 2003-11-14 00:02: The patch doesn't seem to work. I just got
another DYNABLOCK FP: 
 
X-Spam-RBL-Results: <dns:32.90.8.80.dynablock.easynet.nl?type=TXT>  
	["Dynamic/Residential IP range listed by easynet.nl DynaBlock -  
	http://dynablock.easynet.nl/errors.html"] 
X-Spam-Trusted-Relays: [ ip=80.67.174.41 rdns=samizdat.net  
	helo=slut.samizdat.net by=totor.bouissou.net ident= intl=0 ] [  
	ip=193.252.22.28 rdns=smtp3.wanadoo.fr helo=mwinf0304.wanadoo.fr  
	by=slut.samizdat.net ident= intl=0 ] 
X-Spam-DCC: Etherboy: totor.bouissou.net 1002; Body=1 Fuz1=1 Fuz2=1 
X-Spam-Status: No, hits=-103.4 required=5.0 tests=AWL,BAYES_00, 
	RCVD_IN_DYNABLOCK,USER_IN_WHITELIST autolearn=no version=2.60 
X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on  
	totor.bouissou.net 
X-Spam-Possible-Lang: fr 
X-Spam-Level:  
X-Spam-Untrusted-Relays: [ ip=80.8.90.32  
	rdns=ca-bordeaux-18-32.w80-8.abo.wanadoo.fr helo=something  
	by=mwinf0304.wanadoo.fr ident= intl=0 ] 
 

--

Another DYNABLOCK FP, and another issue as well: It seems that the last relay 
that sent the mail to my MX is ignored by SA, as it doesn't appear at all in 
SA relays list. 
Possibly because this machine gave an IDENT information that, formatted by the 
receiving qmail, cause SA could not understand the IP format...? 
 
Let's see: 
 
In local.cf, I have: 
 
trusted_networks 213.228.0/24 
 
The e-mail Received: headers are: 
 
Received: from postfix3-2.free.fr (foobar@213.228.0.169) 
  by totor.bouissou.net with SMTP; 14 Nov 2003 08:05:50 -0000 
Received: from asterix.laurier.org (lns-p19-8-82-65-66-244.adsl.proxad.net 
[82.65.66.244]) 
	by postfix3-2.free.fr (Postfix) with ESMTP id 7BACDC372 
	for <michel@bouissou.net>; Fri, 14 Nov 2003 09:05:49 +0100 (CET) 
 
SA diags are: 
 
X-Spam-RBL-Results: <dns:244.66.65.82.dynablock.easynet.nl?type=TXT>  
	["Dynamic/Residential IP range listed by easynet.nl DynaBlock -  
	http://dynablock.easynet.nl/errors.html"] 
X-Spam-Trusted-Relays:  
X-Spam-DCC: Etherboy: totor.bouissou.net 1002; Body=1 Fuz1=1 Fuz2=1 
X-Spam-Status: No, hits=-103.2 required=5.0 tests=AWL,BAYES_00, 
	RCVD_IN_DYNABLOCK,USER_IN_WHITELIST autolearn=no version=2.60 
X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on  
	totor.bouissou.net 
X-Spam-Possible-Lang: fr 
X-Spam-Level:  
X-Spam-Untrusted-Relays: [ ip=82.65.66.244  
	rdns=lns-p19-8-82-65-66-244.adsl.proxad.net helo=asterix.laurier.org  
	by=postfix3-2.free.fr ident= intl=0 ] 
 
As you can see, the "postfix3-2.free.fr (foobar@213.228.0.169)" relay, that 
should appear in the "trusted relays" section, doesn't show up. SA has missed 
it. 
 
(I had already seen this "missed relay" issue before applying today's patch, 
so it's not the patch that caused this) 
 
Please note that the "Received: from postfix3-2.free.fr 
(foobar@213.228.0.169)" format is the usual Received: line format used by 
qmail. "foobar@" appears when the remote SMTP server answered and "IDENT" 
request giving "foobar" as username. 
 
qmail Received format is: 
 
Received: from (<remote_machine_rDNS>|unknown) 
\((<IDENT_username@>)?<IP_address>\) by <receiving_hostname> with <protocol>; 
<timestamp> 
 


------- Additional Comment #14 From Michel Bouissou 2003-11-14 00:45 -------

Sorry, I was incomplete 
 
qmail Received format is:  
  
Received: from (<remote_machine_rDNS>|unknown) (\(HELO 
<helo_name_given_by_remote_if_different_from_rDNS>\))? 
\((<IDENT_username@>)?<IP_address>\) by <receiving_hostname> with <protocol>;  
<timestamp> 
 
This can make for example: 
 
Received: from unknown (HELO feux01a-isp) (213.199.4.210) 
  by totor.bouissou.net with SMTP; 1 Nov 2003 07:05:19 -0000 
 
or 
 
Received: from x1-6-00-04-bd-d2-e0-a3.k317.webspeed.dk 
(benelli@80.167.158.170) 
  by totor.bouissou.net with SMTP; 5 Nov 2003 23:18:42 -0000 
 
or 
 
Received: from adsl-207-213-27-129.dsl.lsan03.pacbell.net (HELO merlin.net.au) 
(Owner50@207.213.27.129) 
  by totor.bouissou.net with SMTP; 10 Nov 2003 06:30:34 -0000
Comment 1 Justin Mason 2003-11-14 19:37:56 UTC 
Created attachment 1567 [details]
attempt at a fix

ok, this may work -- could you try it Michel?
Comment 2 Justin Mason 2003-11-14 19:38:16 UTC 
milestone
Comment 3 Michel Bouissou 2003-11-15 02:39:19 UTC 
I wrote a little patch that works on my machine. I will attach it below. 
Then I will try yours.
Comment 4 Michel Bouissou 2003-11-15 02:40:14 UTC 
Created attachment 1568 [details]
Patch for qmail Received: with IDENT
Comment 5 Michel Bouissou 2003-11-15 02:46:21 UTC 
Justin Mason's patch seems to fix the problem on my machine's as well as my 
own patch does. 
 
Justin's patch looks to be more generic than mine at first sight, so maybe 
Justin's is better ?
Comment 6 Michel Bouissou 2003-11-15 03:11:36 UTC 
To be more complete about the qmail format, a complete qmail 1.03 Received 
line normally looks like this: 
 
Received: from postfix3-2.free.fr (HELO machine.domain.com) 
  (foobar@213.228.0.169) by totor.bouissou.net with SMTP; 
  14 Nov 2003 08:31:29 -0000 
 
- "from" is followed by remote's rDNS, or "unknown" if no rDNS available 
 
- "(HELO machine.domain.com)" appears ONLY if given HELO differs from rDNS or 
no rDNS was available 
 
- "(foobar@213.228.0.169)" is composed with IDENT info, "@" remote IP 
- If remote gave no IDENT info, then "foobar@" doesn't appear, and we just 
have the remote's IP between parentheses: "(213.228.0.169)" 
 
- qmail normally doesn't put remote IP between square brackets []. If it does 
(I saw some examples in Received.pm), then somebody probably patched it for 
getting those. 
 
So some variants are possible depending upon what the remote gave, i.e.: 
 
The shortest possible: 
Received: from postfix3-2.free.fr (213.228.0.169) by totor.bouissou.net 
  with SMTP; 14 Nov 2003 08:31:29 -0000 
 
Received: from postfix3-2.free.fr (foobar@213.228.0.169) by totor.bouissou.net 
  with SMTP; 14 Nov 2003 08:31:29 -0000 
 
Received: from unknown (HELO machine.domain.com) (foobar@213.228.0.169) 
  by totor.bouissou.net with SMTP; 14 Nov 2003 08:31:29 -0000 
 
Received: from unknown (HELO machine.domain.com) (213.228.0.169) 
  by totor.bouissou.net with SMTP; 14 Nov 2003 08:31:29 -0000 
 
etc...
Comment 7 Michel Bouissou 2003-11-16 01:16:13 UTC 
I just saw Justin's Received: parsing patch misses some Received: lines on my 
machine. 
 
This one was missed: 
 
Received: from loki.komtel.net (212.7.146.145) 
  by totor.bouissou.net with SMTP; 16 Nov 2003 04:53:54 -0000 
 
And this one as well: 
 
Received: from c66.169.197.134.ts46v-19.pkcty.ftwrth.tx.charter.com 
  (66.169.197.134) by totor.bouissou.net with SMTP; 
  16 Nov 2003 05:59:32 -0000
Comment 8 Michel Bouissou 2003-11-16 01:23:44 UTC 
...and I just checked that the simpler patch that I have proposed works 
allright with these.
Comment 9 Michel Bouissou 2003-11-16 07:39:06 UTC 
Taking another look at the parse_received_line routine in Received.pm, I 
noticed that this routine included a whole lot of qmail headers tests 
disseminated everywhere inside it. 
 
So I rewrote a couple of tests that should fit for every qmail-smtpd 
"Received: from ..." format I know about, and replaces all the qmailish tests 
that were previously disseminated in the parse_received_line routine. 
 
I will attach my proposed patch, that removes 9 old tests and replaces them 
with more compact ones, and allows to have "all qmail at the same place". 
 
Tests I've performed on my system so far shows it seems to be working 
allright, but I'd appreciate if other qmail users could test it on their 
systems as well. 
 
Cheers.
Comment 10 Michel Bouissou 2003-11-16 07:41:05 UTC 
Created attachment 1569 [details]
Comprehensive qmail Received: headers patch for Received.pm
Comment 11 Michel Bouissou 2003-11-16 23:51:00 UTC 
Created attachment 1571 [details]
Newer version of comprehensive qmail Received: headers patch for Received.pm

A little rework on this patch to make sure that qmail-tests won't be triggered
by non-qmail Received: entries (that may put the HELO where qmail puts the
rDNS, but that may be very similar besides this).

This patch seems completely satisfactory on my machine, but I'd like to get
some feedback from others.
Cheers.
Comment 12 Justin Mason 2003-11-17 14:40:28 UTC 
That patch looks good to me; it cleans up a lot of the formats and looks like
it'll handle pretty much everything.

+1
Comment 13 Michel Bouissou 2003-11-18 00:17:23 UTC 
Still works good here after several days of testing and checks made against a 
lot of received mail. 
 
+1
Comment 14 Justin Mason 2003-11-19 23:23:09 UTC 
ok, applied