headers
Matt Simerson | 3 Apr 2012 08:09
Gravatar

A basket of changes


This last weekend I finally replaced my smtpd daemon with qpsmtpd. I encountered a number of little issues
that I have written changes for. All changes are published on my github fork.

1. Issue #1: auth/vpopmaild authentications were failing. Apparently the format of vpopmaild responses
has been expanded and the responses are conditional. I replaced the 'sometimes works' eq comparison with
a 'works every time' regexp. I also added some test files for the 3 vpopmail plugins.

2. 50MB of RAM isn't enough on my FreeBSD 8.2 amd64 system. Increased the default to 75MB. 

3. Backscatter spam!  I'm not fond of the "periodically build a list" approach, and found
Qmail::Deliverable, which includes the check_qmail_deliverable qpsmtpd plugin. I imported the
plugin and removed the reference to the no longer available check_delivery plugin from the STATUS file. 
So far, the plugin seems to work perfectly.

4. updated the github repo URL (ask -> smtpd) in docs.

5. applied the greylisting patch posted as Issue #1 on Google issue tracker. The patch was 'accepted' by Ask
in 2007, but never applied. 

6. spamassassin plugin: added support for per-user SA config settings, rewrote header processing logic
in spamassassin plugin so it adds all the SA generated X-Spam-* headers to the message. Refactored the
code for better maintainability.

I have all of these changes applied to my production running qpsmtpd install. 

I have one other issue I have yet to resolve. SpamAssassin generates a folded (nicely formatted)
X-Spam-Report header. It's a thing of beauty, and unfortunately, if I add it to the message, the
Mail::Header->add function unfolds and truncates it. Yuck.
(Continue reading)

Permalink | Reply | 8 comments |
headers
Jared Johnson | 3 Apr 2012 16:09
Favicon

Re: A basket of changes

> 6. spamassassin plugin: added support for per-user SA config settings,
> rewrote header processing logic in spamassassin plugin so it adds all the
> SA generated X-Spam-* headers to the message. Refactored the code for
> better maintainability.

Adding all the X-Spam-* headers to all recipients' messages could
technically be thought of as too transparent in some situations.  In
practical terms the worst thing I can think of would be that one recipient
would be able to see that some other recipient whose preferences they are
not allowed to look at has whitelisted or blacklisted some sender address.
 That's assuming you have added information to each header to indicate
which recipient it is for -- but if you haven't, how would each recipient
know which X-Spam-* header applies to them?

Anyway this isn't exactly an objection, just pointing it out in case
anyone can think of a more serious problem with such transparency.  The
only other option is to queue multiple messages, one for each recipient. 
This is quite a lot of trouble to go to just to avoid exposing X-Spam-*
headers to other recipients... although our organization's fork does go to
this trouble in order to deal with other problems, like Subject header to
use when different recipients have differing spam scores and subject
tagging is on :)

-Jared
Permalink | Reply | 4 comments |
headers
Matt Simerson | 3 Apr 2012 17:51
Gravatar

Re: A basket of changes

We could easily add a "suppress headers..." config option.

I would suggest instead editing the SpamAssassin config and not have those headers added in the first place...

Sent from my iPad

On Apr 3, 2012, at 7:09 AM, "Jared Johnson" <jjohnson <at> efolder.net> wrote:

>> 6. spamassassin plugin: added support for per-user SA config settings,
>> rewrote header processing logic in spamassassin plugin so it adds all the
>> SA generated X-Spam-* headers to the message. Refactored the code for
>> better maintainability.
> 
> Adding all the X-Spam-* headers to all recipients' messages could
> technically be thought of as too transparent in some situations.  In
> practical terms the worst thing I can think of would be that one recipient
> would be able to see that some other recipient whose preferences they are
> not allowed to look at has whitelisted or blacklisted some sender address.
> That's assuming you have added information to each header to indicate
> which recipient it is for -- but if you haven't, how would each recipient
> know which X-Spam-* header applies to them?
> 
> Anyway this isn't exactly an objection, just pointing it out in case
> anyone can think of a more serious problem with such transparency.  The
> only other option is to queue multiple messages, one for each recipient. 
> This is quite a lot of trouble to go to just to avoid exposing X-Spam-*
> headers to other recipients... although our organization's fork does go to
> this trouble in order to deal with other problems, like Subject header to
> use when different recipients have differing spam scores and subject
> tagging is on :)
(Continue reading)

Permalink | Reply | 0 comments |
headers
Matt Simerson | 3 Apr 2012 21:13
Gravatar

Re: A basket of changes


On Apr 3, 2012, at 7:09 AM, Jared Johnson wrote:

>>> 6. spamassassin plugin: added support for per-user SA config settings,
>>> rewrote header processing logic in spamassassin plugin so it adds all the
>>> SA generated X-Spam-* headers to the message. Refactored the code for
>>> better maintainability.
>> 
>> Adding all the X-Spam-* headers to all recipients' messages could
>> technically be thought of as too transparent in some situations. <snip>
>> 
>> Anyway this isn't exactly an objection, just pointing it out in case
>> anyone can think of a more serious problem with such transparency.  The
>> only other option is to queue multiple messages, one for each recipient. 
>> This is quite a lot of trouble to go to just to avoid exposing X-Spam-*
>> headers to other recipients... although our organization's fork does go to
>> this trouble in order to deal with other problems, like Subject header to
>> use when different recipients have differing spam scores and subject
>> tagging is on :)
> 
> We could add a "suppress headers..." config option.
> 
> I would suggest instead editing the SpamAssassin config and not have those headers added in the first place...

This seems like the sort of topic that deserves a mention in the spamassassin plugin POD. After pondering
it, I looked around and found this:

http://www.qmailwiki.org/Simscan/Guide#Multiple_Recipient_Problem_with_Per_User_simscan_option

If there is a single recipient in the email then simscan adds the -u user <at> domain option to spamc. This tells
(Continue reading)

Permalink | Reply | 2 comments |
headers
Matt Simerson | 6 Apr 2012 09:08
Gravatar

Re: A basket of changes


On Apr 3, 2012, at 12:13 PM, Matt Simerson wrote:

> On Apr 3, 2012, at 7:09 AM, Jared Johnson wrote:
> 
>>>> 6. spamassassin plugin: added support for per-user SA config settings,
>>>> rewrote header processing logic in spamassassin plugin so it adds all the
>>>> SA generated X-Spam-* headers to the message. Refactored the code for
>>>> better maintainability.
>>> 
>>> Adding all the X-Spam-* headers to all recipients' messages could
>>> technically be thought of as too transparent in some situations. <snip>
>>> 
>>> Anyway this isn't exactly an objection, just pointing it out in case
>>> anyone can think of a more serious problem with such transparency.  The
>>> only other option is to queue multiple messages, one for each recipient. 
>>> This is quite a lot of trouble to go to just to avoid exposing X-Spam-*
>>> headers to other recipients... although our organization's fork does go to
>>> this trouble in order to deal with other problems, like Subject header to
>>> use when different recipients have differing spam scores and subject
>>> tagging is on :)
>> 
>> We could add a "suppress headers..." config option.
>> 
>> I would suggest instead editing the SpamAssassin config and not have those headers added in the first place...
> 
> <snip>We could adopt a similar approach. If the message has multiple recipients, then leave the username
as qpsmtpd typically sets:
> 
> 	my $username = $self->{_args}->{spamd_user} || getpwuid($>);
(Continue reading)

Permalink | Reply | 1 comment |
headers
Jared Johnson | 6 Apr 2012 18:10
Favicon

Re: A basket of changes

> I also changed the per-user feature to only take effect if the
> message has a single recipient.

Perhaps it would be useful to give people the option of only accepting one
recipient per SMTP transaction?  This seems a bit messy/inefficient, we
don't do it ourselves, but I've heard from at least one QP user with a big
installation that has had success with this; anything after the first
(accepted) recipient is simply deferred, so the sending MTA is forced to
send messages to multiple recipients with one transaction per recipient. 
This would allow people to always take advantage of per-user spam rules,
if they wished.  I should note that our installation does institute a
higher recipient limit (since we queue once per recipient we were running
into problems with client timeouts being reached while waiting for our
response to the DATA command...) and sending MTA's have generally obeyed
this, but it has generated some questions from sending organizations as
their queued take a little longer to empty out, especially in the case
where our clients are doing outbound filtering.  ...Actually, if it was
also possible to turn of SA scanning on relay-allowed transactions, then
it would perhaps be reasonable to make it a one-inbound-recipient limit
rather than simply a one-recipient limit.  For inbound mail it's much less
common to have a whole lot of recipients in a single transactions.

-Jared
Permalink | Reply | 0 comments |
headers
Michael Papet | 7 Apr 2012 18:43
Picon
Favicon

Queue Routing?

Hi,

I want to write a queue plugin that routes mail depending on the user-side of the address.  The decision to
deliver locally or forward is no problem. It's just a look up in a table.  If the look up returns the forward
result, then I'd like to put the email into the smtp-forwarding queue.  If the lookup returns local
delivery, then it goes into the MailDir queue.

Is there an elegant way to chain queue plugins together this way?

My first guess is to modify the needed queue plugins and turn them into subroutines or maybe even functions. 
If anyone has any more elegant ideas where I don't have to edit the existing queue plugins, please let me know.

I'm an amateur Perl hacker, so messing around with code written by a very likely better programmer seems
suboptimal to me.

Thanks in advance,

mpapet
Permalink | Reply | 1 comment |
headers
Steve Kemp | 7 Apr 2012 18:49
Picon
Favicon
Gravatar

Re: Queue Routing?

On Sat Apr 07, 2012 at 09:43:32 -0700, Michael Papet wrote:

> Is there an elegant way to chain queue plugins together this way?

  Use notes.  At the point you make a decision you would set a
 note to record which option you wish to go for, something like:

        ...
         $transaction->notes( "delivery",  "forward" );
       or
         $transaction->notes( "delivery",  "locally" );

  Then in the two queue handlers you'd update their hook routines
 as follows:

   queue/smtp-forward:

sub hook_queue {
  my ($self, $transaction) =  <at> _;

    # add something like this to terminate this plugin if the
    # mail is not to be forwarded
    my $delivery  =$transaction->notes("delivery") || "unknown";
    return DECLINED unless( $delivery =~ /forward/i );
    ...

  }

  Ditto for queue/maildir:
(Continue reading)

Permalink | Reply | 0 comments |
headers
Matt Simerson | 8 Apr 2012 02:14
Gravatar

[PATCH] fixed failing auths to auth/vpopmaild, added tests

Available on github: https://github.com/msimerson/qpsmtpd/commit/ed8eca0a630be3a100840a247f7a8305bfae15c4

Apparently the format of vpopmaild responses has been expanded and the responses
are conditional.

* Replaced the 'sometimes works' eq comparison with a regexp that always works.
* added tests for all 3 vpopmail plugins
* added cram-md5 auth support to auth_vpopmaild.
---
MANIFEST                              |    3 +++
plugins/auth/auth_vpopmail            |    6 ++++-
plugins/auth/auth_vpopmaild           |   45 ++++++++++++++++++---------------
t/plugin_tests/auth/auth_vpopmail     |   27 ++++++++++++++++++++
t/plugin_tests/auth/auth_vpopmail_sql |   27 ++++++++++++++++++++
t/plugin_tests/auth/auth_vpopmaild    |   27 ++++++++++++++++++++
6 files changed, 113 insertions(+), 22 deletions(-)
create mode 100644 t/plugin_tests/auth/auth_vpopmail
create mode 100644 t/plugin_tests/auth/auth_vpopmail_sql
create mode 100644 t/plugin_tests/auth/auth_vpopmaild

diff --git a/MANIFEST b/MANIFEST
index 59c9260..e4a9e0b 100644
--- a/MANIFEST
+++ b/MANIFEST
 <at>  <at>  -145,6 +145,9  <at>  <at>  t/helo.t
t/misc.t
t/plugin_tests.t
t/plugin_tests/auth/auth_flat_file
+t/plugin_tests/auth/auth_vpopmail
+t/plugin_tests/auth/auth_vpopmail_sql
(Continue reading)

Permalink | Reply | 1 comment |
headers
Matt Simerson | 8 Apr 2012 02:28
Gravatar

[PATCH] doc fix: changed $TRACE_LEVEL to $TraceLevel

---
README |    4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/README b/README
index 05569da..2b68d0e 100644
--- a/README
+++ b/README
 <at>  <at>  -59,7 +59,7  <at>  <at>  directory.
Put the files there.  If you install from git you can just do
run the following command in the /home/smtpd/ directory.

-   git clone git://github.com/abh/qpsmtpd.git
+   git clone git://github.com/smtpd/qpsmtpd.git

Beware that the master branch might be unstable and unsuitable for anything
but development, so you might want to get a specific release, for
 <at>  <at>  -192,7 +192,7  <at>  <at>  with the normal name of the server).
In case of problems always first check the logfile.

As default it goes into log/main/current.  Qpsmtpd can log a lot of
-debug information.  You can get more or less by adjusting $TRACE_LEVEL
+debug information.  You can get more or less by adjusting $TraceLevel
in lib/Qpsmtpd.pm (sorry, no easy switch for that yet).  Something
between 1 and 3 should give you just a little bit.  If you set it to
10 or higher you will get lots of information in the logs.
--

-- 
1.7.9.4
(Continue reading)

Permalink | Reply | 0 comments |

Gmane