Oh sorry, i meant not into learning part.
Reporting should be a seperate feature, not combined with learning.
It's time to go to bed :-)

Nico, how would you get around the FP problem? As I understand it, reporting
spam to various lists/services should always be done manually, to avoid
accidentally sending them non-spam. Since spamc/spamd is automated, I don't see
how to avoid these errors. 

Hello Bob,

I think you've misunderstood my request.

My aim is to introduce an reporting capability usable by spamc/spamd, so 
anybody who runs only spamc/spamd could be able to report spam to various 
lists/services.

This idea has it's origin by the bug #1201 there we introduced the spamc/spamd 
learning capability.

Pherhaps I've misunderstood you. Any thoughts?

NicoP.

(In reply to comment #2)
> Nico, how would you get around the FP problem? As I understand it, reporting
> spam to various lists/services should always be done manually, to avoid
> accidentally sending them non-spam. Since spamc/spamd is automated, I don't 
see
> how to avoid these errors. 

Also not every environment uses spamc/spamd (always) automated. Due to the C-API 
of spamc it would be a nice feature to introduce.

I would like to start the spamc-part. But before I want to discuss the usage.
I think it would make sense to make reporting and learning combineable, so only 
one time the message has to be transfered to spamd. But too I want to make it an 
single useable feature.
Or should it be only configureable at the spamd site. So spamd would decide to 
report the message to razor...? So it could be user-dependet (sql database). 

What do you think?

Subject: Re:  RFE: add reporting support to spamd/spamc

> 
> I would like to start the spamc-part. But before I want to discuss the usage.
> I think it would make sense to make reporting and learning combineable, so only 
> one time the message has to be transfered to spamd. But too I want to make it an 
> single useable feature.
> Or should it be only configureable at the spamd site. So spamd would decide to 
> report the message to razor...? So it could be user-dependet (sql database). 

Please feel free to get going on the spamc part, please note that I
had to backtrack on some of the spamc portions for learning to keep it
binary compatible, so please keep that that in mind.

You'll want to support reporting and revoking, unfortunately -r is
already taken as an option (it is time to move to long options for
spamc?), so do your best to come up with something sane.

No need to somehow combine the efforts, since reporting automatically
triggers a learn as spam and revoking automatically triggers a learn
as ham.

Lets not overthink the user dependent stuff, and just implement the
same way as we did with learn, if someone wants to add the ability to
turn things off later then we'll add it then.

I don't think the spamd portion will be too hard, but it might be a
bit too ambitious for 3.1, but we'll see.

Michael

Subject: Re:  RFE: add reporting support to spamd/spamc

Also, the REPORT command for the spamd protocol is taken so we'll have
to come up with something else.

Hello Michael,

the spamc part is nearly complete!

The protocol is:
LISTSREPORT SPAMC/1.3
ListsReport-type: 0
User: testuser

ListsReport-type: 0 defines report message as spam
ListsReport-type: 1 defines revoke message (if ham)

What would be a suitable return (code) from spamd?

Do I've missed something?

Subject: Re:  RFE: add reporting support to spamd/spamc

> ------- Additional Comments From nico.prenzel@pn-systeme.de  2005-05-05 10:37 -------
> 
> the spamc part is nearly complete!
> 
> The protocol is:
> LISTSREPORT SPAMC/1.3
> ListsReport-type: 0
> User: testuser
> 
> ListsReport-type: 0 defines report message as spam
> ListsReport-type: 1 defines revoke message (if ham)
> 

To tell you the truth, I don't much like the name, but I honestly
can't think of anything better.  Otherwise it looks good.

> What would be a suitable return (code) from spamd?
> 

We could go simple and do something like:
Reported: Yes
Reported: No

or slightly more complicated:
Reported: Yes
Reported: No

Revoked: Yes
Revoked: No


Standard return codes for the first line of the response.

> Do I've missed something?

Did you pick out command line options?

Michael

(In reply to comment #5)
> Subject: Re:  RFE: add reporting support to spamd/spamc
> 
> > 
> > I would like to start the spamc-part. But before I want to discuss the 
usage.
> > I think it would make sense to make reporting and learning combineable, so 
only 
> > one time the message has to be transfered to spamd. But too I want to make 
it an 
> > single useable feature.
> > Or should it be only configureable at the spamd site. So spamd would decide 
to 
> > report the message to razor...? So it could be user-dependet (sql database). 
> 
> Please feel free to get going on the spamc part, please note that I
> had to backtrack on some of the spamc portions for learning to keep it
> binary compatible, so please keep that that in mind.
> 
> You'll want to support reporting and revoking, unfortunately -r is
> already taken as an option (it is time to move to long options for
> spamc?), so do your best to come up with something sane.
> 
> No need to somehow combine the efforts, since reporting automatically
> triggers a learn as spam and revoking automatically triggers a learn
> as ham.
> 
> Lets not overthink the user dependent stuff, and just implement the
> same way as we did with learn, if someone wants to add the ability to
> turn things off later then we'll add it then.
> 
> I don't think the spamd portion will be too hard, but it might be a
> bit too ambitious for 3.1, but we'll see.
> 
> Michael
> 

As I consider this again. I think it won't be a good idea to learn automatically 
if triggered for report/revoke.
- report/revoke would produce some (network) overhead if done everytime a 
message is learned (especially no one remembers that report/revoke happens when 
learning is done)
- no need for a seperate spamc switch (-L already implemented)

I'am not amused not to overthink the user part.
Please introduce to query the 'use_Razor2' (and other) settings from SQL/
userpref and use that for reporting!? I think it should be!
Sorry, but I've another opinion about that :-)

>We could go simple and do something like:
>Reported: Yes
>Reported: No

I vote for that. +1 :-)

>Did you pick out command line options?

Not as you considered. I've simply add an switch (-W) to spamc! Too, there are 
some others free. Not perfectly but it works.

Subject: Re:  RFE: add reporting support to spamd/spamc

> ------- Additional Comments From nico.prenzel@pn-systeme.de  2005-05-06 03:26 -------
> 
> As I consider this again. I think it won't be a good idea to learn automatically 
> if triggered for report/revoke.
> - report/revoke would produce some (network) overhead if done everytime a 
> message is learned (especially no one remembers that report/revoke happens when 
> learning is done)
> - no need for a seperate spamc switch (-L already implemented)

> I'am not amused not to overthink the user part.
> Please introduce to query the 'use_Razor2' (and other) settings from SQL/
> userpref and use that for reporting!? I think it should be!
> Sorry, but I've another opinion about that :-)

The code will use all of the same mechanisms that it does now, so
there is nothing to think about here.  If -L is specified then it
won't be available, period.  If use_razor2 is set to 0 then we won't
report/revoke fomr razor, so on and so forth.

Learning either as spam for reporting or as ham for revoking is just
how that code works.

Like I said, I think you are overthinking.


> 
> >We could go simple and do something like:
> >Reported: Yes
> >Reported: No
> 
> I vote for that. +1 :-)
> 
> >Did you pick out command line options?
> 
> Not as you considered. I've simply add an switch (-W) to spamc! Too, there are 
> some others free. Not perfectly but it works.

Works for me, it is easily changed if for some reason it doesn't work
out.

FYI, here is the output I just got from spamd:
2329] dbg: markup: removing markup
[2329] dbg: plugin: Mail::SpamAssassin::Plugin::Razor2=HASH(0x8dadbd4) implements 'plugin_revoke'
[2329] dbg: info: entering helper-app run mode
[2329] dbg: info: leaving helper-app run mode
[2329] dbg: reporter: spam revoked from Razor
[2329] info: spamd: revoked ham message for parker:1000 in 0.7 seconds, 2569 bytes

Michael

Created attachment 2849 [details]
First throw of ListsReport for spamc part

>>Learning either as spam for reporting or as ham for revoking is just how that 
code works.

Okay, for me learning is always a learn to bayesian. So, I think we've 
misunderstood each other. I thought you wanted to learn each reported/revoked 
message automatically to the bayesian database. If that isn't your intention all 
is right!

Sorry for the confusion, but a question should be legitimately.

Greetings NicoP.

Created attachment 2850 [details]
Second throw

forgot to implement exclusion!

Subject: Re:  RFE: add reporting support to spamd/spamc

> 
> ------- Additional Comments From nico.prenzel@pn-systeme.de  2005-05-06 06:54 -------
> >>Learning either as spam for reporting or as ham for revoking is just how that 
> code works.
> 
> Okay, for me learning is always a learn to bayesian. So, I think we've 
> misunderstood each other. I thought you wanted to learn each reported/revoked 
> message automatically to the bayesian database. If that isn't your intention all 
> is right!
> 
> Sorry for the confusion, but a question should be legitimately.

I think we're having a communication problem.  Please check out the
options in spamassassin for -r,--report and -k,--revoke:

--report:
...
           The message will also be submitted to SpamAssassin's
           learning systems; currently this is the internal Bayesian
           statistical-filtering system (the BAYES rules).  (Note that
           if you only want to perform statistical learning, and do
           not want to report mail to third-parties, you should use
           the "sa-learn" command directly instead.)

and -revoke:
...
           The message will also be submitted as 'ham' (non-spam) to
           SpamAssassin's learning systems; currently this is the
           internal Bayesian statistical-filtering system (the BAYES
           rules).  (Note that if you only want to perform statistical
           learning, and do not want to report mail to third-parties,
           you should use the "sa-learn" command directly instead.)


So, report and revoke always learn, unless bayes_learn_during_report
is set to 0.

Michael

Thanks Michael,

now I know why you want the learning combined with reporting. I didn't ever use 
the spamassassin command. So it wasn't clear to me that a 
bayes_learn_during_report parameter exists.

Ok, this doesn't need anything changed to previous posted patch.

If a patch to spamd is posted I'll check that new functionality.

Until then, NicoP.

Created attachment 2851 [details]
Patch File

Here is a combined version of the patch.  You'll notice several things that
have changed:

1) The spamd protocol command is now COLLABREPORT (for collaborative filtering
databases).  Other things have changed to follow, the header, the method name,
vars, etc.

2) Bunch of cleanup in the spamc C code.

3) There is a perl based client method for report and revoke

4) There is a start of a test but I don't have it like I want it yet

One problem is that even if you have -L turned on or all reporting methods
returned off spamd will still return a Reported: Yes header to the client, this
isn't right so I need to fix that.

However, in the interest of getting the code out there for eyeballs and the
fact that I might not be able to work on it again for a few days, here is my
latest patch.  Feel free to make improvements, and if you do be sure to
document where those improvements were so I can fold them into anything else
I'm working on.

Michael

I've been making heavy use of the learn via spamd feature, I'm sure this will
also come in handy and since it's 90% of the way there it should be good for 3.1.

There are a couple of issues with the tests, one spamc doesn't return anything
in the failure case, fixed in an upcoming patch, Reporter.pm defaults to
everything being ok for report, hopefully fixed in an upcoming patch, and the
default init.pre doesn't have Razor turned on, so need to come up with a
solution for this, I'm guessing this is also why the razor2.t tests have been
failing for me as of late.

Much better idea for the t/spamc_optC.t test, I'll just create a quick reporter
plugin, then we'll turn everything off and just use that plugin to test the
capabilities.  Have to do something fancy to test the failure case but it
shouldn't be too hard.

*** Bug 4302 has been marked as a duplicate of this bug. ***

Committed revision 169089.