headers
Adam Nielsen | 1 Jul 2008 07:32
Picon
Picon
Favicon

How do you get multiple messages sent via the same smarthost SMTP connection?

Hi all,

I've got a Mailman installation on a server that feeds its messages in 
to Exim.  Since this machine is behind a firewall, all mail is 
smarthosted out to another mailserver.

The problem is that this other mailserver (beyond my control) has a rate 
limit enforced.  This means that after 20 connections in a row further 
connections are dropped until traffic stops for a few minutes.

Once Mailman has delivered all the messages to Exim, Exim seems to make 
a separate SMTP connection to the smarthost for each message.  After 20 
messages (20 connections) the IP gets rate limited and Exim can't 
deliver any more messages (the rest of the queued messages fail with 
"defer (-53): all host address lookups failed temporarily")  15 minutes 
later when the next queue runner starts another 20 messages go but 
that's it.

 From reading the docs it seems that Exim is supposed to reuse SMTP 
connections, but for some reason this isn't happening for me.  Perhaps 
it's because of the "mailman_verp_smtp" transport I have set up? 
There's an option in there called "max_rcpt" which limits one recipient 
per message which would seem to be causing this problem, but as far as I 
understand I can't remove that otherwise the wrong envelope addresses 
etc. would be used.

Any ideas would be much appreciated, because at the moment a mailout 
takes almost an entire day to complete!

Many thanks,
(Continue reading)

Permalink | Reply | 11 comments |
headers
Phil Pennock | 1 Jul 2008 08:28

Re: How do you get multiple messages sent via the same smarthost SMTP connection?

On 2008-07-01 at 15:32 +1000, Adam Nielsen wrote:
> I've got a Mailman installation on a server that feeds its messages in 
> to Exim.  Since this machine is behind a firewall, all mail is 
> smarthosted out to another mailserver.
> 
> The problem is that this other mailserver (beyond my control) has a rate 
> limit enforced.  This means that after 20 connections in a row further 
> connections are dropped until traffic stops for a few minutes.

You haven't set connection_max_messages = 1  ?

Look at remote_max_parallel and queue_run_max.

What's probably happening is that Exim is receiving the mail and
immediately trying to send it on.  If you can tolerate a short delay in
sending email, you might try configuring Exim to queue all mail instead
of delivering immediately and running with more frequent queue-runners.

Set "queue_only" true in the main configuration.  Or, if it's not *all*
mail sent out via the smarthost, look at instead:
  queue_smtp_domains = *

Start exim with "exim -bd -q2m" to be a listening daemon which forks
queue-runners every 2 minutes.  Or -q1m if you want.

Actually, "exim -bd -odqs -q2m" would be equivalent if you want to
change the cmdline invocation and not change the config.

Basically, you're arranging for Exim to figure out where all the mail
has to go, but only actually sending it out periodically, when there's
(Continue reading)

Permalink | Reply | 9 comments |
headers
Adam Nielsen | 1 Jul 2008 08:37
Picon
Picon
Favicon

Re: How do you get multiple messages sent via the same smarthost SMTP connection?

>> The problem is that this other mailserver (beyond my control) has a rate 
>> limit enforced.  This means that after 20 connections in a row further 
>> connections are dropped until traffic stops for a few minutes.
> 
> You haven't set connection_max_messages = 1  ?

Nope, I've tried setting it to 200 messages as well but with no difference.

> Look at remote_max_parallel and queue_run_max.
> 
> What's probably happening is that Exim is receiving the mail and
> immediately trying to send it on.  If you can tolerate a short delay in
> sending email, you might try configuring Exim to queue all mail instead
> of delivering immediately and running with more frequent queue-runners.

That's what I thought, but after the first run through there are about 
1000 messages stuck in the queue (because they couldn't contact the 
server.)  Every time I start a queue runner another 20 messages go but 
the rest are stuck, so I assume pre-queueing all the messages wouldn't 
make any difference?

> Basically, you're arranging for Exim to figure out where all the mail
> has to go, but only actually sending it out periodically, when there's
> likely to be multiple items that can be delivered down a connection.

Are "figuring out where the mail has to go" and "sending it out" 
actually two different steps?  Because like I say the mail is being 
queued, and flushing the queue still only sends one message per SMTP 
connection - but if it's not "figuring out where the mail has to go" 
until the last minute perhaps this could still explain why it's only
(Continue reading)

Permalink | Reply | 8 comments |
headers
Sascha Schmidt | 1 Jul 2008 11:36
Picon

Exim, Vexim, Sieve and Spamassassin

Hi *,

I've got problems tagging spam-mails with "X-Spam-Flag: YES". I'm using 
vexim where users can define the value when their emails should be 
tagged. The following code exists within vexim-acl-check-content.conf
that will add the Spam-Score and Report to all emails (globally defined):

---
    warn  message         = X-Spam-Score: $spam_score ($spam_bar)
          spam            = vmail:true
    warn  message         = X-Spam-Report: $spam_report
          spam            = vmail:true
---

The following code works and adds the X-Spam-Flag correctly but does not 
use the userdefined values from the vexim webinterface. The global value 
is used:

---
     warn message         = X-Spam-Flag: YES
          spam            = vmail
---

I thought about a mysql-clause which is used within the condition part. 
The problem I have is that I'm not knowing the correct syntax to create 
the part correctly; missing curely brackets etc... :-)

My first thought was the following one:
---
warn message = X-Spam-Flag: YES
(Continue reading)

Permalink | Reply | 1 comment |
headers
Graeme Fowler | 1 Jul 2008 11:56

Re: How do you get multiple messages sent via the same smarthost SMTP connection?

On Tue, 2008-07-01 at 15:32 +1000, Adam Nielsen wrote:
> From reading the docs it seems that Exim is supposed to reuse SMTP 
> connections, but for some reason this isn't happening for me.  Perhaps 
> it's because of the "mailman_verp_smtp" transport I have set up? 

Yes, that's it. Mailman and the VERP router ensures that every message
is sent to a single recipient (for bounce handling automation).

You can change the way exim is running to get around this. At the moment
you're probably calling it with options like "-bd -q20m" or similar.

If you change the daemon options to "-bd" only, to run the listening
daemon, you can then run the queue runners from cron instead with
different options.

http://www.exim.org/exim-html-current/doc/html/spec_html/ch05.html#SECID39

"-qq...

        An option starting with -qq requests a two-stage queue run. In
        the first stage, the queue is scanned as if the
        queue_smtp_domains option matched every domain. Addresses are
        routed, local deliveries happen, but no remote transports are
        run.

         The hints database that remembers which messages are waiting
        for specific hosts is updated, as if delivery to those hosts had
        been deferred. After this is complete, a second, normal queue
        scan happens, with routing and delivery taking place as normal.
        Messages that are routed to the same host should mostly be
(Continue reading)

Permalink | Reply | 0 comments |
headers
Schramm, Dominik | 1 Jul 2008 12:48
Picon
Favicon

Re: Deferred messages disappearing, message "Completed" prematurely

Now, to prevent double logging, I've set up the filter like this:

# Exim filter

if error_message
then
   finish
endif

logfile /var/log/exim4-prod/mainlog

if foranyaddress $recipients ( $thisaddress is order <at> otherdomain.biz )
then
   if first_delivery
   then
      logwrite "$tod_log $message_id unseen deliver: ..."
      headers add "X-test-header: test"
   endif

   unseen deliver copyaddress <at> outside.com errors_to it-operations <at> mydomain.de
endif

This seems almost fine. However, yesterday I encountered a very 
strange error (next-to-last line): 

2008-06-30 16:21:33 1KDKG5-000205-68 <= edi <at> ... [4.5.6.7] for order <at> otherdomain.biz
2008-06-30 16:21:34 1KDKG5-000205-68 unseen deliver: copyaddress <at> outside.com errors_to it-operations <at> mydomain.de
2008-06-30 16:21:34 1KDKG5-000205-68 => order <at> otherdomain.biz
2008-06-30 16:21:34 1KDKG5-000205-68 mailserver-2.outside.com [9.10.11.12] Connection timed out**
copyaddress <at> outside.com F=<edi <at> ...> P=<it-operations <at> mydomain.de> R=dnslookup T=remote_smtp:
(Continue reading)

Permalink | Reply | 0 comments |
headers
Peter Velan | 1 Jul 2008 14:00

Re: Problem with vexim upgrade

am 30.06.2008 20:01 schrieb Rob Connon:
> I updated my vexim install from 1.x to 2.x, following the instruction 
> and running the create_db script [...]

Hi Rob,

this is the wrong list; try <http://silverwraith.com/vexim/>, especially
<http://silverwraith.com/mailman/listinfo/vexim> instead.

Cheers,
Peter

--

-- 
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Permalink | Reply | 0 comments |
headers
Lena | 1 Jul 2008 18:51
Picon
Favicon

Re: How do you get multiple messages sent via the same smarthost SMTP connection?

> From: Adam Nielsen

> Exim seems to make 
> a separate SMTP connection to the smarthost for each message.

> From reading the docs it seems that Exim is supposed to reuse SMTP 
> connections, but for some reason this isn't happening for me.

I'm not sure, but you can try:

begin transports
remote_smtp:
  driver = smtp
  serialize_hosts = *

This solution is better (if it works) because messages will not wait for
next queue run.

--

-- 
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Permalink | Reply | 1 comment |
headers
Eduardo M KALINOWSKI | 1 Jul 2008 23:30
Picon

maildirsize not being created with correct permissions

I have this transport set:

virtual_user_maildir:
  debug_print = "T: virtual_user_maildir for $local_part <at> $domain"
  driver = appendfile
  directory = /home/vmail/$domain/$local_part/INBOX
  create_directory
  delivery_date_add
  envelope_to_add
  return_path_add
  maildir_format
  directory_mode = 0750
  mode = 0640
  mode_fail_narrower = false
  user = vmail
  group = mail
# Store quota information
  maildir_use_size_file = true
  maildir_quota_directory_regex = ^(?:cur|new|\.(?!Trash|Spam).*)$
# Store message sizes in filename
  maildir_tag = ,S=$message_size
  quota_size_regex = ,S=(\d+)
# Use maildirfolder files
  maildirfolder_create_regex = /\.[^/]+/?$

The messages and folders are correctly created with the specified modes,
but the maildirsize file is created with mode 600 (user and group are
OK, though). I'd like it to have group read permission to ease backups,
but exim does not seem to be creating the file with the requested mode.
Exim version is 4.68. Any suggestions?
(Continue reading)

Permalink | Reply | 1 comment |
headers
Phil Pennock | 2 Jul 2008 01:03

Re: How do you get multiple messages sent via the same smarthost SMTP connection?

On 2008-07-01 at 16:37 +1000, Adam Nielsen wrote:
> Are "figuring out where the mail has to go" and "sending it out" 
> actually two different steps?  Because like I say the mail is being 
> queued, and flushing the queue still only sends one message per SMTP 
> connection - but if it's not "figuring out where the mail has to go" 
> until the last minute perhaps this could still explain why it's only 
> sending one message per connection.

Figuring out: Routers
Sending it out: Transports

You need more than just "it's left around"; the routing needs to have
happened, so that a given SMTP connection has a whole bunch of messages
which can go down, even if split up into one recipient each.

You want to ensure that Routing is done ahead of time; -odqs will do
this, or two-stage queueing with -qq (thanks Graeme for the reminder)
or using queue_smtp_domains.

Just using queue_only isn't sufficient, my mistake.  The memory of
managing heavily loaded Exim is fading.  :^/

-Phil

--

-- 
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
(Continue reading)

Permalink | Reply | 7 comments |

Gmane