Paul Hoffman / IMC | 6 Feb 19:23 2003
Picon

New draft on internationalizing email addresses


Greetings. This message is Bcc'd to many IETF-related lists that are 
discussing internationalization, Internet mail and Usenet. With the 
imminent release of the RFCs for IDNA, Adam Costello and I have a 
proposal for allowing internationalized characters in email addresses 
(on both sides of the  <at> ). Instead of having it discussed on a bunch 
of different lists, there is a new list for discussing the draft and 
other ideas related to internationalizing email address.

If you are interested, please see <http://www.imc.org/ietf-imaa/> for 
information on the Internet Draft and the mailing list.

--Paul Hoffman, Director
--Internet Mail Consortium

Adonis El Fakih | 10 Feb 05:21 2003

Introduction and query

Dear ietf-smtp members,

My name is Adonis El Fakih. I have been working on a new mail delivery process to be used as a replacement for the process specified in SMTP documents. The document is a result of many months of design and development of a mail delivery process that is able to withstand current and future trends of mail abuse. The design includes a modular design that allows future designs to be integrated into the mail delivery process.

I have submitted an Internet draft http://www.ietf.org/internet-drafts/draft-fakih-amdp-00.txt to ietf discussing the protocol along with its pros and cons. I also outlined an implementation guideline to enable developers to implement the protocol ASAP . I was also able to develop a demo implementation and tested it and the basic delivery system seems to work as described in the document.

I would like to start a dialogue with people interested in a solution to the problems associated with spam, and not just a band-aid as it is proposed today by many implementations out there. I do not want to shoot down SMTP, since AMDP relies on SMTP style mail messages, the major difference between the two protocols is the order by which mail is delivered.

Again I am open to all comments, and if this is NOT the proper channel for this discussion, please excuse me and do recommend the proper channel for doing so.

Best regards,
Adonis


_______________________________________________________________
Ayna.com the Arabic web starts right here.

Valdis.Kletnieks | 10 Feb 06:52 2003
Picon

Re: Introduction and query

On Sun, 09 Feb 2003 20:21:42 PST, Adonis El Fakih <adonis <at> aynacorp.com>  said:

> I have submitted an Internet draft
> http://www.ietf.org/internet-drafts/draft-fakih-amdp-00.txt to ietf
> discussing the protocol along with its pros and cons. I also outlined an
> implementation guideline to enable developers to implement the protocol
> ASAP . I was also able to develop a demo implementation and tested it
> and the basic delivery system seems to work as described in the
> document. 

I've read through the draft, and find nothing that can't either already be done
in the current context of ESMTP and already-existing error codes and SMTP
extensions, or isn't a generally bad idea engineering or security wise.

A whole class of things addressed here (authentication and confidentiality)
were addressed in RFC1847:

1847 Security Multiparts for MIME: Multipart/Signed and
     Multipart/Encrypted. J. Galvin, S. Murphy, S. Crocker, N. Freed.
     October 1995. (Format: TXT=23679 bytes) (Status: PROPOSED STANDARD)

Feel free to use either S/MIME (RFC2623-2634) or OpenPGP (RFC3156) on top
of 1847, depending on your personal preference in PKI trust schemes.

There is a general failure throughout the draft to distinguish between
the concepts of "authentication" (proving who the sender of an e-mail is)
and "authorization" (whether I want to accept mail from this source).

In addition, it seems to make a general assumption that the same spammers
who currently lie through their teeth about such basic things as a From:
address will magically tell the truth in this scheme.  The draft would
benefit greatly from a careful re-reading, and at *each* point where the
sending system has to provide information, ask 2 questions:

1) Is this information that a spammer would feel motivated to lie about?
2) Does the protocol accept a source-provided value for the information?

A *very* basic precept in security is to not trust user-supplied data, and
this draft fails to do so in a huge number of ways.

I've not bothered thinking about scaling issues, as there are lots of basic
problems with the draft - the fact that it won't work on my laptop that only
handles 400-500 pieces of mail a day means that it's not worth asking how it
will fare on our central mail server that's seeing 200K POP3 connections an
hour, or how it will scale to the billions that Hotmail/MSN is seeing...

Now on to specifics... on page 15/16 we have:

           Use a trusted connection between [10] and [20].  This can be 
           achieved by enforcing the use of assigned internal IPÆs, 
           firewall, encryption, etc.  It is also recommended that the 
           connection does not use a clear text mechanism when possible.

This is already done by using SMTP AUTH and/or STARTTLS on port 587.

Page 20:

    4. The AMDP recipient server [70] will accept envelopes that meet 
        its business requirements, do not violate the public mail 
        policy [60], and can authenticate themselves.

The first two points are a purely internal matter (business requirements
and mail policy), and RFC2821, section 4.2.3 already lists this error code:

     550 Requested action not taken: mailbox unavailable
         (e.g., mailbox not found, no access, or command rejected 
         for policy reasons)

so an SMTP server is totally within its rights to return '550 Policy rejection'
for mail that it doesn't like.

"Can authenticate themselves" is also already doable already - simply reject
connections that don't do STARTTLS with a verified certificate.

The devil is in the details - the problem here isn't authentication, it's
authorization.  On the one hand, it is *certainly* doable to just say "I refuse
e-mail that I haven't previously authorized the sender". To see how broken this
is, consider that this policy would prevent your receipt of this e-mail, since
I'm fairly sure that I haven't sent you mail before.

On the other hand, simply saying "make them have a valid X.509 cert" isn't
workable either, for all the usual "whack-a-mole" reasons - unless you have
some way to make sure that spammers cannot get a cert, you can't use the
cert as a "I am not a spammer" test.

            serve the mail messages. However in both instances these 
            servers are authenticated as MHFs in the DNS entries of 
            Yahoo.com. 

And *OF COURSE*, the MHF for spammers-r-us.com is authenticated as such in
it's DNS.  And the whole idea of a callback is broken as well - you're
introducing a whole new TCP 3-way handshake, which doesn't prove anything.
If I want to prove that a business is legitimate, I don't call the phone
number they gave me - whoever answers the phone there will say of course they
are legit.  You need to call the Better Business Bureau or some similar
trusted third party and see what they say.  Similarly, if I get spam from
some site, I'm certainly *not* seriously expecting that I'll contact the
server that *THEY* tell me to contact, and have that server say "Don't
accept the mail, it's spam".  Again, an authentication/authorization issue.

        The MHF also keeps track of the email topics also known by 
        threads, by maintaining an active list of threads.  The 
        originating MHF will maintain the master copy of the thread 
        index.  When negotiating message ids, the servers can send the 
        updated thread keys to the receiving server if it requires 
        having the thread tree which is used to reference back the 
        thread.  This is useful to reduce the size of a message if it 
        is a thread so you do not need to send the original message 
        back and forth. A thread is also related to the to the 
        classification scheme, where the originator or sender can 
        classify the message.    

This makes the very broken assumption that there exists one server that
sees all of the thread *and is authoritative* for that thread.  If your
mail had also been posted to ietf-822 and that list was on a different server,
and then different subthreads which were sometimes cross-posted and sometimes
not, would horribly confuse the concept of "thread".

   Rejected Classifications 

      This defines the classifications that are not accepted by the 
      network administrator i.e. a government agency, or an elementary 
      school do not want to accept any mail from marketing firms. And 
      any MHF contacting them with such messages will be reported back 
      to the external incident reporting service [120]. 

Hello?  www.mail-abuse.org?  I'd like to report a spammer....

There's already a *LOT* of organizations that provide blacklists of
sites - if you can't name at least 6, you haven't been in the anti-spam
business long.  You probably want to think very long and hard about the
reasons for the existence of *so many* blacklists, and why there is still
spam increasing even with the existence of said lists....

      Government - This means that the emails generated are mainly 
      official in nature, and mass mailing is not expected from these. 

      Educational - This means that the emails generated are mainly 
      educational, and not to be confused with messages from .edu 
      domains.  A university may be classified as a business if its 
      emails are to prospective students, while domains or MHFÆs used to 
      server students email, can set this key to "Personal" or "Bulk" 
      depending on the volume of messages generated.  

Hmm... that puts vt.edu in quite the pickle.  We're a university, but we're
also an agency of the Commonwealth of Virginia.  For some things, we can also
be considered a $740M/year company...

And I would like to point out that such things as "government official mass
mailings" *do* exist (want to guess how the Governor gets information to as
many people as possible about things like changes to the pension plan? ;)

If "educational" isn't "mail from .edus", you're opening yourself up to a *BIG*
abuse hole - a spammer can just claim "he was just educating the public about
a new offer".

And there isn't much here to prevent a spammer from lying about the nature of
their "personal, not used for spam" domain, is there?

      This is the sub category of mail types and it is defined by the 
      user.  It lets the final recipient know what type of message it 
      is, and it used for informational purposes only, since there is no 
      way to guarantee that the field is used correctly.  However it is 

"No way to guarantee".  Exactly - that's the problem here.
--

-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

Adonis El Fakih | 10 Feb 08:08 2003

Re: Introduction and query

Hi Valdis,

I welcome your comment, and thank you for being the first to bring great points to the table for discussion. I will definetly look into the refences pointed out in your message.

Before I answer some of the points presented here I would like to make the distinction about the major differences between the two appraches of mail delivery, and I hope as I make these points the readers will start seeing the differences that may not be apparent from initial glance of the document.

In SMTP you actually have to receive the message in its entirety before you can apply any of spam filters, unless you have a filter on email. Most of spam messages hurt in the pocket becuase you have to receive them before you figure if they are welcome or not. You have to store them, process them, and figure out what to do next.. Sometimes I use the term "band-aid the problem," and what I meant is that the damage is done by the spammer once you receive the message to decide if it is good or bad.

In AMDP you do not have to do that at all, becuase the sender MUST keep the mail message on their OWN server, and send you an envelope describing its contents. Then they have to autheticate themselves so you know that a mail message is actually residing where the spammer/or non spammer says. Most of the spam today uses fake FROM, so this will stop this kind of abuse. Today you can send mail to anyone and claim to be from yahoo, ietf or any one else. In AMDP you can not do that, I agree that we can do this in extenstions of SMTP and I do not really mind where it is implemeted as long as it is done somewhere.. The point is to start clean, use the good points of SMTP but put something better out there..

> I've read through the draft, and find nothing that can't either already be done
> in the current context of ESMTP and already-existing error codes and SMTP
> extensions, or isn't a generally bad idea engineering or security wise.
>
> A whole class of things addressed here (authentication and confidentiality)
> were addressed in RFC1847:
>
> 1847 Security Multiparts for MIME: Multipart/Signed and
> Multipart/Encrypted. J. Galvin, S. Murphy, S. Crocker, N. Freed.
> October 1995. (Format: TXT=23679 bytes) (Status: PROPOSED STANDARD)
>
> Feel free to use either S/MIME (RFC2623-2634) or OpenPGP (RFC3156) on top
> of 1847, depending on your personal preference in PKI trust schemes.
>
> There is a general failure throughout the draft to distinguish between
> the concepts of "authentication" (proving who the sender of an e-mail is)
> and "authorization" (whether I want to accept mail from this source).
Ok I should take than into consideraion when wording changes to document. The
MIME standards will all stand true as today within AMDP, it is the process of
deliverying mail that is the subject of the proposal, not the encryption, or packaging
since many system rely on these mail formats.

> In addition, it seems to make a general assumption that the same spammers
> who currently lie through their teeth about such basic things as a From:
> address will magically tell the truth in this scheme. The draft would
> benefit greatly from a careful re-reading, and at *each* point where the
> sending system has to provide information, ask 2 questions:

AMDP will enforce that mail received has to be from an explicitly assigned host
by the domain admin. This is not available in SMTP anyone can do it, and if they
do lie it will not accept the mail. They can make domains for that purpose, which is
fine becuase at this point the source of spam is known, which can not be traced at
all in smtp. (there are other measures in the document to make it wrothless for
a spammer to do what they do today)

> 1) Is this information that a spammer would feel motivated to lie about?
In my opinion that spammer will lie at every chance they get :) so the less
areas are left to lie about the less confusing info we receive..

> 2) Does the protocol accept a source-provided value for the information?
>
> A *very* basic precept in security is to not trust user-supplied data, and
> this draft fails to do so in a huge number of ways.
>
> I've not bothered thinking about scaling issues, as there are lots of basic
> problems with the draft - the fact that it won't work on my laptop that only
> handles 400-500 pieces of mail a day means that it's not worth asking how it
> will fare on our central mail server that's seeing 200K POP3 connections an
> hour, or how it will scale to the billions that Hotmail/MSN is seeing...
It depends on which piece of the puzzle we are talking about.

The idea behind it is in the details of the protocol. If you have a small site
then you can set your public mail policy in such a way that you will not be flooded
by mesages, and you will only handle what you can. In SMTP your server
crashes and there is nothing you can do to stop the spam unless you pull the plug :(

Small sites will also have to use the services of the post office, since they will
offer MHF services on behalf of your domain. It will be like shipping a package today
if you can not handle it yourself, then give it to experts.

> Now on to specifics... on page 15/16 we have:
>
> Use a trusted connection between [10] and [20]. This can be
> achieved by enforcing the use of assigned internal IPÆs,
> firewall, encryption, etc. It is also recommended that the
> connection does not use a clear text mechanism when possible.
>
> This is already done by using SMTP AUTH and/or STARTTLS on port 587.
Sure why not. there is not need to reinvent the wheel. the difference here
is that 20 is not used to email the outside world but to enforce outgoing mail
rules. you can not do this in SMTP today. You can not enforce outgoing mail
size, language, etc.. that is what [20] is there for..

> Page 20:
>
> 4. The AMDP recipient server [70] will accept envelopes that meet
> its business requirements, do not violate the public mail
> policy [60], and can authenticate themselves.
>
> The first two points are a purely internal matter (business requirements
> and mail policy), and RFC2821, section 4.2.3 already lists this error code:
Yes and no. What is proposed here that a truley public mail policy is to be
used. something similar to robots.txt where any person wishing to email
the domain must query the server first. This reduce wasted time and resources
for both sender and receipient, this is also not available today. I can live
with today's rules but should not we tell the person trying to email us publicly
hey! we will not take mail from you, unless...??

> 550 Requested action not taken: mailbox unavailable
> (e.g., mailbox not found, no access, or command rejected
> for policy reasons)
>
> so an SMTP server is totally within its rights to return '550 Policy rejection'
> for mail that it doesn't like.
>
> "Can authenticate themselves" is also already doable already - simply reject
> connections that don't do STARTTLS with a verified certificate.
>
> The devil is in the details - the problem here isn't authentication, it's
> authorization. On the one hand, it is *certainly* doable to just say "I refuse
> e-mail that I haven't previously authorized the sender". To see how broken this
> is, consider that this policy would prevent your receipt of this e-mail, since
> I'm fairly sure that I haven't sent you mail before.
It is not about rejecting mail. If you are dealing with a domain dealing with few
million messages a day. The space and computing power needed to deal with
AMDP envelopes (not the content of eth mesage) is much more acceptable than
dealing with the storage requierment for the actual message. The sender MUST
keep the message on their server until it is picked up by the end user


>
> On the other hand, simply saying "make them have a valid X.509 cert" isn't
> workable either, for all the usual "whack-a-mole" reasons - unless you have
> some way to make sure that spammers cannot get a cert, you can't use the
> cert as a "I am not a spammer" test.
The certification is just one of the methods to use to autheticate domains in good
standing. I did not cover this in the document, since I thought it will be too much
for a first draft, however the general idea goes like this.

Once you know that an email is coming from domain A and no one else, then we
can go to a third party (that is paid by domain A to be their certificate manager) and
check if they are within the category they claim to be. So if domain A claims to be
XY category and ends up being ZZ using some smart filters, then we can report
the abuse to the manager of the certificate and they update the category based on
feedback not only from me, but based on reports received from other AMDP sources.
Domain A can not deny that mail is not from his domain, since the design gaurantees
that the host must be explicitly authroized to mail. All mail from A to other AMDP
servers will autmoatically be converted to the new classification since the third party
job is to provide the realtime classification of the domain.

One option that I looked at is to charge small fees for incoming mail messages, so even
if a spammer fakes everything, by imposing a pay-per-message model, you can control
the flow of messages.

>
> serve the mail messages. However in both instances these
> servers are authenticated as MHFs in the DNS entries of
> Yahoo.com.
>
> And *OF COURSE*, the MHF for spammers-r-us.com is authenticated as such in
> it's DNS. And the whole idea of a callback is broken as well - you're
> introducing a whole new TCP 3-way handshake, which doesn't prove anything.
> If I want to prove that a business is legitimate, I don't call the phone
> number they gave me - whoever answers the phone there will say of course they
> are legit. You need to call the Better Business Bureau or some similar
> trusted third party and see what they say. Similarly, if I get spam from
> some site, I'm certainly *not* seriously expecting that I'll contact the
> server that *THEY* tell me to contact, and have that server say "Don't
> accept the mail, it's spam". Again, an authentication/authorization issue.
yes I agree see above, the three way handshake is just one of many conditions
that play together to close the wholes available in SMTP.

>
> The MHF also keeps track of the email topics also known by
> threads, by maintaining an active list of threads. The
> originating MHF will maintain the master copy of the thread
> index. When negotiating message ids, the servers can send the
> updated thread keys to the receiving server if it requires
> having the thread tree which is used to reference back the
> thread. This is useful to reduce the size of a message if it
> is a thread so you do not need to send the original message
> back and forth. A thread is also related to the to the
> classification scheme, where the originator or sender can
> classify the message.
This is open to discussion, if it can not be implemented than it can be removed
nothing written in the proposed document is written in stone :) I agree with you
we will find things that are already there that can be integrated..

> This makes the very broken assumption that there exists one server that
> sees all of the thread *and is authoritative* for that thread. If your
> mail had also been posted to ietf-822 and that list was on a different server,
> and then different subthreads which were sometimes cross-posted and sometimes
> not, would horribly confuse the concept of "thread".
I agree again, this may be difficuly to implement (at the least the thread synching)
but the idea here is to create an automatic schema that allows certain email classification
to be autmoatically saved to tape in antipicpation to some ruling by SEC where companies
must backup all communication relating to certain business practices.

>
> Rejected Classifications
>
> This defines the classifications that are not accepted by the
> network administrator i.e. a government agency, or an elementary
> school do not want to accept any mail from marketing firms. And
> any MHF contacting them with such messages will be reported back
> to the external incident reporting service [120].
>
> Hello? www.mail-abuse.org? I'd like to report a spammer....
Sure the [120] is an open end, i.e. AMDP will have to define an API that is used by any external
incident reporting service, not like today, you have to draft an email, check what happened to it.
The idea is to automate the system so reports are sent and received in real-time, so if you have
a virus in your network it will automatically alert the external network, and other AMDP servers
will know of this before the mail admin does!!

> There's already a *LOT* of organizations that provide blacklists of
> sites - if you can't name at least 6, you haven't been in the anti-spam
> business long. You probably want to think very long and hard about the
> reasons for the existence of *so many* blacklists, and why there is still
> spam increasing even with the existence of said lists....
:) I do know a few. Working on ayna.com we had unforunatly been on few lists
and I had to go and explain what happen to be taken off the list. Spammers
put a return address of ayna.com, and how can I explain to few thousand angry
people it is not us!!!

> Government - This means that the emails generated are mainly
> official in nature, and mass mailing is not expected from these.
>
> Educational - This means that the emails generated are mainly
> educational, and not to be confused with messages from .edu
> domains. A university may be classified as a business if its
> emails are to prospective students, while domains or MHFÆs use
>
> === message truncated ===
Sorry it seems that my mail client truncated the message. We can disuss them later, but the examples are just that, they can be anything you want, and they can flexible to meet tha needs of real business out there..

However the discussion of the categories is also an extensive one and will need a group of dedicated people to figure out somtheing that would fit.
_______________________________________________________________
Ayna.com the Arabic web starts right here.

Valdis.Kletnieks | 10 Feb 10:21 2003
Picon

Re: Introduction and query

On Sun, 09 Feb 2003 23:08:10 PST, Adonis El Fakih said:

(OK, so it's insomnia time again..;)

> In SMTP you actually have to receive the message in its entirety before
> you can apply any of spam filters, unless you have a filter on email.

Quite correct.. However...

> In AMDP you do not have to do that at all, becuase the sender MUST keep
> the mail message on their OWN server, and send you an envelope
> describing its contents.

Notice that you still have to download the entire message to tell if the
other end is telling the truth regarding its contents.  And I've seen
enough Murkowski-compliant "Under S.1618, this message isn't spam if it
includes a remove link" spam to not believe that spammers will tell the
truth in the envelope.  As you yourself note - a big problem is that they
lie in the MAIL FROM - why would APMD-MAIL-CLASS contain truth?

If anything, this is actually doing the spammer a favor - it means that he
can save bandwidth.  Sites that have blacklisted them won't call back to
pick up the spam.

On the other hand, sites that have blacklisted the spammer already are free
to issue a 550 on the MAIL FROM or RCPT TO and thus skip the DATA phase, so
you're not actually providing any benefit here.

>                          Then they have to autheticate themselves so you
> know that a mail message is actually residing where the spammer/or non
> spammer says. Most of the spam today uses fake FROM, so this will stop
> this kind of abuse.

Actually, if you think really hard about it, you'll realize that this doesn't
*really* stop fake FROM - all it does is make the spammer use a throw-away
FROM address that happens to point to a server he controls.

>> There is a general failure throughout the draft to distinguish between 
>> the concepts of "authentication" (proving who the sender of an e-mail is) 
>> and "authorization" (whether I want to accept mail from this source). 
> Ok I should take than into consideraion when wording changes to document. 

It's more than just wording - it's a way of thinking.  It's even possible
to conceive and design authorization systems that don't involve any actual
authentication at all.  In this class fall proposals such as the "I don't
care who you are, but if you send me e-mail you first have to perform
such-and-such complex computation that will chew several seconds of CPU - this
won't matter to any legitimate one-off mail, but will matter to a spammer".

Another example of anonymous authorization would be a rate-limiting system,
where a mail server would say "I don't care WHO you are, you're only allowed X
msgs/hour per /24 of source address space without prior arrangement" - this is
already implemented in some systems, and deals nicely with the "one-off
anonymous personal mail" problem while drastically limiting what a spammer can
do.

> AMDP will enforce that mail received has to be from an explicitly assigned host
> by the domain admin. This is not available in SMTP anyone can do it, and if they
> do lie it will not accept the mail.

No - they merely can't use an existing domain.  All this forces is that the
spammer also has to get a DNS entry updated at the same time he buys his
network connectivity.

And if an ISP will sell bandwidth, they will likely sell DNS on the same
whack-a-mole contract.

>                                      They can make domains for that purpose, which
> becuase at this point the source of spam is known, which can not be traced at 
> all in smtp.

Umm.. It's traceable.

Received: from npsmtp02la.mail2world.com (mw27.mail2world.com [66.28.189.27])
  by zidane.cc.vt.edu (Mirapoint Messaging Server MOS 3.3.2-CR)  with ESMTP id
  BAS07392; Mon, 10 Feb 2003 02:07:59 -0500 (EST)

Interesting that your mailserver said 'npsmtp02la' but the PTR says 'mw27'.
Reverse DNS for the 66.28.189/24 is provided by cogentco.com, and the IP address
block is owned by:

route:      66.28.189.0/24
descr:      Mail2World Network
origin:     AS26254
remarks:    this is non-portable space, no exceptions
notify:     wkim <at> mail2world.net
mnt-by:     MAINT-MAIL2WORLD
changed:    wkim <at> mail2world.net 20030110
source:     VERIO

I'm too lazy to go poke a BGP looking-glass to see who AS26254 is getting
transit from, but I'd start by asking Verio. ;)

A bigger problem here is that although open SMTP relays are fast becoming
rarer (I've seen one reliable statistic that open SMTP relays have fallen from
60% down to about 1% of the problem), there are signs that spammers are
starting to abuse open proxy servers (many older HTTP proxies would quite
happily accept 'CONNECT destination.com 25').

> Sure why not. there is not need to reinvent the wheel. the difference here
> is that 20 is not used to email the outside world but to enforce outgoing mail 
> rules. you can not do this in SMTP today. You can not enforce outgoing mail
> size, language, etc.. that is what [20] is there for..

Given the number of ISPs that currently block outbound port 25, this seems
to be an "already done".  All you need is a firewall that blocks outbound
SYN packets on port 25 from everything from the mail server, and filtering
software on the mail server.  Given the number of e-mail a day I receive with
silly "This e-mail is proprietary" banners, I have to assume that most sites
who wish to do this already know how to do so.

> Once you know that an email is coming from domain A and no one else, then we 
> can go to a third party (that is paid by domain A to be their certificate manager) and 
> check if they are within the category they claim to be. So if domain A claims to be 
> XY category and ends up being ZZ using some smart filters, then we can report 
> the abuse to the manager of the certificate and they update the category based on 
> feedback not only from me, but based on reports received from other AMDP sources. 
> Domain A can not deny that mail is not from his domain, since the design gaurantees 
> that the host must be explicitly authroized to mail. All mail from A to other AMDP 
> servers will autmoatically be converted to the new classification since the third party
> job is to provide the realtime classification of the domain.

Nothing here that ORBS and MAPS haven't been doing for years already.

> yes I agree see above, the three way handshake is just one of many conditions
> that play together to close the wholes available in SMTP.

You missed the point - if you don't trust the spammer to tell the truth
about "this is not spam" when he contacts you, why do you expect a truthful
answer when you spend the extra effort to contact a server *the spammer runs*?

> I agree again, this may be difficuly to implement (at the least the thread synching)
> but the idea here is to create an automatic schema that allows certain email classification
> to be autmoatically saved to tape in antipicpation to some ruling by SEC where companies
> must backup all communication relating to certain business practices. 

This is so totally outside of the scope of SMTP that I'm not going to delve
further into it, other than to comment that every current MTA that I'm aware
of already has the ability to either archive a copy of everything, or can be
modified to do so.  Also, most of the most incriminating documents will be
intra-office memos - remember that Oliver North was convicted in the Iran/Contra
scandal largely on the contents of backed-up memos of the PROFS message system
that was in use at the time.  These likely never hit SMTP *at all*.

> The idea is to automate the system so reports are sent and received in real-time, so if you have
> a virus in your network it will automatically alert the external network, and other AMDP servers
> will know of this before the mail admin does!!

Security-wise, this is a Very Risky Idea.  Consider what happens to your
network e-mail connectivity if a number of sites submit "we're getting virus
spam from Mail2World".  All it takes is one script kiddy with 10K zombie
hosts (and there's lots of them out there), and you're suddenly off the air.

> :) I do know a few. Working on ayna.com we had unforunatly been on few lists
> and I had to go and explain what happen to be taken off the list. Spammers
> put a return address of ayna.com, and how can I explain to few thousand angry 
> people it is not us!!!

Your proposal may deal with preventing a "joe-job" (at the price of vastly
complicating things) while doing absolutely nothing new to stop spam from
mail.only-valid-for-12-hours.com whack-a-moles or entrenched spamhauses
that have been using the same domains and IP addresses for years, and who
thus have been in blacklists for years....

/Valdis
Adonis El Fakih | 10 Feb 19:41 2003

Re: Introduction and query

Good morning Valdis :)

AMDP is about fairness and distributing the processing of a message between the
sender and receiver. SMTP does not outline a "fair" process for deliverying messsages.
The recipient takes the short end of the stick. The recipient has to process, store, and deliver
the message, in AMDP I am trying to shift the store and deliver parts to the sender,
while leaving the process, and some parts of the store/deliver in the hands of the recpient.

In our day-to-day business you do not pay or facilitate for the sender to send you what they
want, and SMTP we do just that. We pay for the network connectivity, and have to be
purdened by unsolicited messages. The way we curtail this in our lives by reversing the
process where the sender has to take care of paying for the services to deliver (i.e. storing, and
delivering) and we just open the package. The reversed process and imposed fees are the
best control mechanism for free-for-all spamming. It is not a magic bullet, but it is a well
proven concept that when adapted to the mail process on the net, a fundamental change
of how spam is handled will occur.

> In SMTP you actually have to receive the message in its entirety before
> you can apply any of spam filters, unless you have a filter on email.
>
> Quite correct.. However...
>
> In AMDP you do not have to do that at all, becuase the sender MUST keep
> the mail message on their OWN server, and send you an envelope
> describing its contents.
>
> Notice that you still have to download the entire message to tell if the
> other end is telling the truth regarding its contents. And I've seen
> enough Murkowski-compliant "Under S.1618, this message isn't spam if it
> includes a remove link" spam to not believe that spammers will tell the
> truth in the envelope. As you yourself note - a big problem is that they
> lie in the MAIL FROM - why would APMD-MAIL-CLASS contain truth?

No you do not have to download the message in its entierty yo assert that.
There are two cases for AMDP mail to accept a message.
1. in the public policy is states what is allowed to come in based on size limitations
if the message being read from the connection exceeds +/- some bytes the transfer is
terminated. i.e. i can set my public policy to say

Max email size 30k which means message >= 30 will pass to be delivered (including
content, if as we receive it the counting algorithm hits 35k, the message is dropped)
this is the difference. We asked the mailer to be honest, if they do not, we drop the
message. So if they want a message to pass they have to be bound by the terms
of the public policy.

Again back to the APMD-MAIL-CLASS. I propsed three part for it
1 is the one set up by the domain admin is basically does not
telll us anyting about the type of email, instead it tells us if this
domain is managed internally a thirda party (agent), or if it is a public
gateway like yahoo, hotmail etc..

2. the second part is set by the user, which is the spammer, but it is
status is validated and maintained through a third party. This service
will provide certain key information regarding the classification
a) the classification claimed by the sender
b) classification reported by receiveing users
c) data since inception of certificate
d)... anything else we need to validate the claims

You keep saying that spammer wil lit in their classification, and of course
they will do that. The certificate will tell us what the have been doing
on other sites, and give statistics about the report ratio and when a
cenrtificate was issued used. there could be a statsitical honesty meter
applied to the stats coming in from AMDP server at each query which are
used in building the case for or against a domain.

Now back to the public policy. I can set up a policy such as this
a) we accept mail messages that do not exceed 30k (which is fine be
our network)
b) we will accept category DIRECT::PERSONAL for a price of $0.001
c) we will accept category *::* for a price of $0.25
d) (this is not in document but came out of our disccsuion)
there could be a section in the public policy that points out the
rates for the "honesty rating" such as
- no rating or poor one : 50 cents / message
- 2-4 rating 30 cents
- 4-6 rating 20 cents
- etc.. the better the rating the lowest, it could even have a
- 10 rating (excellent) 0 cents

What you are creating here is a mechanism that is self correcting becuase
you are using other non-legal or tehcnical solutions to overcome the
descrepencies in the classification. The prize for the unsolicited mail
business is lower rates for better reporting of their classifications.

Also I proposed the addition of a subsription engine that is linked to domains
that have heavy subscription made on behalf of their users, which will provide
these businesses with email messages that want to receive messages to them
and that will give them the lowest rates to deliver the messages.

There is also the rate (not money but time in between connections) at which we
accept envelopes from an source is specified by the AMDP server when it
negotiates connection withe connecting server.

So in the event it is a spammer, only one message is received until
1. we can contact them (eliminates using other people's domains)
2. and payment in the event that things did not check using
3. the time lag we informed him to wait has elapsed

I want to reiterate, you will not stop spam, but you will be open in
your communication and be able to set rules. Failure to meet those
rules will drop a message, and the utlimate goal is for them to get the
message to the users, and as long as you use "post" processing
you will fail in controling the problem.

> If anything, this is actually doing the spammer a favor - it means that he
> can save bandwidth. Sites that have blacklisted them won't call back to
> pick up the spam.
I do not get you comment here. You as a receipient save money too, becuase
you did not receive or process the spam, teh spammer is loosing becuase
he did not achieve his goal of deliverying the message, and now he has to
maintain the messages on his servers for delivery by people who want to
receive those messages.

> On the other hand, sites that have blacklisted the spammer already are free
> to issue a 550 on the MAIL FROM or RCPT TO and thus skip the DATA phase, so
> you're not actually providing any benefit here.
Yes I agree, but spammer empersonate mail messages from yahoo, msn, aol, rcn,
etc.. as a business many of these crucial domains can not be blacklisted. Sometimes
we need a gray list :) and that is only possible with a realtime mechanism that is part
of the mail delivery cycle that can be used free by the recpient, and paid by the sender.

Again back to tha fairness model. As a recipient, who is beinf spammed, I am being forced
to pay for more services, and subscribe to solutions, etc.. while the spammers figure new
ways to scam me from more money, while they are breaking the law and my piggy bank :)

By reversing these actions the spammer will have no other choice but play by the rules, like
they do today in the air mail business. Yes we still receive junk mail, but that is because the
post office sets the rates. If you really do not want to get junk mail, just up your rates at the
domain level, and you will be left alone for others that want the junk.

>
> Then they have to autheticate themselves so you
> know that a mail message is actually residing where the spammer/or non
> spammer says. Most of the spam today uses fake FROM, so this will stop
> this kind of abuse.
>
> Actually, if you think really hard about it, you'll realize that this doesn't
> *really* stop fake FROM - all it does is make the spammer use a throw-away
> FROM address that happens to point to a server he controls.
>
Actually it does, since that host has to be explicitly on the outgoing mail list
for the domain. Let us assume we have 100 servers within out network, as
a system admin i can assign 4 of them to be used for outgoing mail handleing
i.e. MHFs. The chances of somone hacking onto other non-attended hosts
is much higher that the ones the admin controls. Also in the case of abuse
it can be pin pointed to the specific host within the domain.

Today I get millions a message a month claiming to be from yahoo, and domains
we do business with, and sometimes spammers use ISP so you can not block that
ISP since you block everyone on that host. The AMDP model will allow teh ISP
to set the host in question to be an MHF for his customer, and emails outgoing
carrying a different FROM are corrected in [20] or ignored by the AMDP.

> >> There is a general failure throughout the draft to distinguish between
> >> the concepts of "authentication" (proving who the sender of an e-mail is)
> >> and "authorization" (whether I want to accept mail from this source).
> Ok I should take than into consideraion when wording changes to document.
>
> It's more than just wording - it's a way of thinking. It's even possible
> to conceive and design authorization systems that don't involve any actual
> authentication at all. In this class fall proposals such as the "I don't
> care who you are, but if you send me e-mail you first have to perform
> such-and-such complex computation that will chew several seconds of CPU - this
> won't matter to any legitimate one-off mail, but will matter to a spammer".
We use this technique today in one of the products we use, where it slows down the
spammer incrementaly, but still does not work. Many proffesional spammers will have
a whole C bock and use them randomly to overcome settings made based on host

>
> Another example of anonymous authorization would be a rate-limiting system,
> where a mail server would say "I don't care WHO you are, you're only allowed X
> msgs/hour per /24 of source address space without prior arrangement" - this is
> already implemented in some systems, and deals nicely with the "one-off
> anonymous personal mail" problem while drastically limiting what a spammer can
> do.
>
> AMDP will enforce that mail received has to be from an explicitly assigned host
> by the domain admin. This is not available in SMTP anyone can do it, and if they
> do lie it will not accept the mail.
>
> No - they merely can't use an existing domain. All this forces is that the
> spammer also has to get a DNS entry updated at the same time he buys his
> network connectivity.
Which is fine, but his category listing and "honesty" ratings are also low, so they will
have to pay higher rates for mail delivery.


> And if an ISP will sell bandwidth, they will likely sell DNS on the same
> whack-a-mole contract.
Which is also fine, becuase the MHF is where the money is going to be for an ISP.
not is DNS.

> They can make domains for that purpose, which
> becuase at this point the source of spam is known, which can not be traced at
> all in smtp.
>
> Umm.. It's traceable.
:) Yes this message is traceable because I am using a reputable company and it is
not trying to hide who they are. I have seen verey interesting "fake" SMTP conversations
in headers that go nowhere.

- One time IP, used to post the millions of messages and then goes offline. In AMDP the domai has to be within the outgoing mail scheme of the domain, and must stay online so we retrieve the message. Spweing out a million message gonly sends out a million envelopes, not the messages. if they want to make some cash they will need to stay online to server those messages.

- If it is a hijacked MHF once notifed the admin can drop the message from queue, and the damage is controled at his end.


>
> Received: from npsmtp02la.mail2world.com (mw27.mail2world.com [66.28.189.27])
> by zidane.cc.vt.edu (Mirapoint Messaging Server MOS 3.3.2-CR) with ESMTP id
> BAS07392; Mon, 10 Feb 2003 02:07:59 -0500 (EST)
>
> Interesting that your mailserver said 'npsmtp02la' but the PTR says 'mw27'.
> Reverse DNS for the 66.28.189/24 is provided by cogentco.com, and the IP address
> block is owned by:
>
> route: 66.28.189.0/24
> descr: Mail2World Network
> origin: AS26254
> remarks: this is non-portable space, no exceptions
> notify: wkim <at> mail2world.net
> mnt-by: MAINT-MAIL2WORLD
> changed: wkim <at> mail2world.net 20030110
> source: VERIO
>
> I'm too lazy to go poke a BGP looking-glass to see who AS26254 is getting
> transit from, but I'd start by asking Verio. ;)
>
> A bigger problem here is that although open SMTP relays are fast becoming
> rarer (I've seen one reliable statistic that open SMTP relays have fallen from
> 60% down to about 1% of the problem), there are signs that spammers are
> starting to abuse open proxy servers (many older HTTP proxies would quite
> happily accept 'CONNECT destination.com 25').
I disagree about these figures, Most of the mail I receive today uses fake host
name, fake mail froms, open relays, fresh relays setup by the spammers all
over the world. They use cheap ISP accounts in brazil, russia, south africa,
middle east, canada, japan, korea, etc..

Put in place a public policy that is clear, make it easy for spammers to spam
you on certain topics, within your computing power. Make it pricey to email
you unsolicited mail, and you got yourself a good start.. SMTP does not
do that, and believe me if we can add these functinalities in SMTP I will be the
first one to say let us go for it. It is not about the name, or the packaging is
about giving domain admins control. There is no way ion SMTP that I can manage
my domain, people often use it in spam, and there is nothing I can do about it..

>
> Sure why not. there is not need to reinvent the wheel. the difference here
> is that 20 is not used to email the outside world but to enforce outgoing mail
> rules. you can not do this in SMTP today. You can not enforce outgoing mail
> size, language, etc.. that is what [20] is there for..
>
> Given the number of ISPs that currently block outbound port 25, this seems
> to be an "already done". All you need is a firewall that blocks outbound
> SYN packets on port 25 from everything from the mail server, and filtering
> software on the mail server. Given the number of e-mail a day I receive with
> silly "This e-mail is proprietary" banners, I have to assume that most sites
> who wish to do this already know how to do so.
You are relying on a third party to do the blocking here. Why should it not be part
of the design?? not everyone has a firwall, and if someone want to go around than
they can.

> Once you know that an email is coming from domain A and no one else, then we
> can go to a third party (that is paid by domain A to be their certificate manager) and
> check if they are within the category they claim to be. So if domain A claims to be
> XY category and ends up being ZZ using some smart filters, then we can report
> the abuse to the manager of the certificate and they update the category based on
> feedback not only from me, but based on reports received from other AMDP sources.
> Domain A can not deny that mail is not from his domain, since the design gaurantees
> that the host must be explicitly authroized to mail. All mail from A to other AMDP
> servers will autmoatically be converted to the new classification since the third party
> job is to provide the realtime classification of the domain.
>
> Nothing here that ORBS and MAPS haven't been doing for years already.
ORBS and MAPS only check the host, and do not give you stats that can be used in
between. I agree that there service can be upgraded to support a more sophistcated
report.
>
> yes I agree see above, the three way handshake is just one of many conditions
> that play together to close the wholes available in SMTP.
>
> You missed the point - if you don't trust the spammer to tell the truth
> about "this is not spam" when he contacts you, why do you expect a truthful
> answer when you spend the extra effort to contact a server *the spammer runs*?
I asked this to myself. In SMTP the message goes one way, regardless if the
message is good or bad, however in most cases spam over shadows the good
mail. By doing the extra handshake i achieve the following.
1. I am forcing the sender to be online to tell me that indeed they sent me a message
2. I am asking the sender to have that message ready for later pickup

If they do not do that, teh message is not delivered, so you are making changes on the
other side. Spammers now have to invest in a system that can serve those message.
Also now they can truley know which message was read, and by whom, and concetrate
on creating a business geared on the users, and not random mailings to no end..

Thanks in adance for your feedback..

Adonis
_______________________________________________________________
Ayna.com the Arabic web starts right here.

Adonis El Fakih | 10 Feb 20:24 2003

Re: Introduction and query

Hi John,

Thanks for the email, enclosed are my comments

> If "educational" isn't "mail from .edus", you're opening
> yourself up to a *BIG* abuse hole - a spammer can just claim
> "he was just educating the public about a new offer".

Ideally .edu and educationally can be linked, however they do not have to be. I could have an education site that mails out exams and reports, and still be educational to the ones who ask for it.

And you are correct to assume that anyone can send a message saying it is education, but the educational category is not a per message classification i.e. it defines the business of the domain. only the third level of the category is used to classify per message topics and could be ignored in determining a message for acceptance. (informational purposes only)

A good example goes back to public email gateway. Hotmail will set its classification to PUBLIC::ACCOUNT_TYPE::USER_CLASS. This tells that Hotmail tells the recpient that this is a public mail gateway. and hence they are not controling the message content, ACCOUNT_TYPE can be used to give an indication of the message is. The USER_CLASS could be set the by the sender by it is for informational purposes only. Receiving servers will require payment from these mail messages (paid by the sender, not hotmail) and the enevelope is accepted. A spammer can not claim to be from hotmail becuase they do not have access to the MHF so that drops over half of the spammers using this approach.

If they do setup account on hotmail.
1) they will have an outgoing mailbox quota so a "PERSONAL" classification can be given to someone who maintains an outgoing mailbox size of 2 megs
2) while a customer who pays for an outgoing mailbox of 50M will be classified as "UNSOLICITED" or equivalent.

What this does is put much more control in the hands of hotmail. So no one can just slap the From: <at> hotmail.com and get away with it. The message will be DOA.


Going back to the classification topic. One domain can have multiple uses for their domain. and they can maintain those classifications with the third party and properly set them in outgoing mail, as to not affect their ratings within each category of mail used.

If a business defines its mail as being educational, then it will be coraborated by the recipients, and it will affect the "honesty" level maintained by the third party.

Receiving AMDP servers will look at the reported classifications, and based on the level reported by the third party, they can ask for accept reject or ask for more money to be paid for delivery..


> ICANN and EDUCAUSE have quietly reclassified EDU to be US
> institutions only, so universities outside the US _cannot_ use
> .EDU names. Of course, that doesn't change your main points: a
> domain-based system won't work, and a system based on the
> spammer's assertions about content won't work either.

I do not want to confuse the classification of .edu with the email classification they are totally seperate and need to be looked at further. It is a mechanism to identiy the type of business reardless of the domain name. i.e. .gov can have a domain used for send unsolicited messages so they will tend to use a domain for that purpose, or properly classify their messages that way, so it will not affect their rating in that classification.

This system does a much better job that SMTP, it can not solve everything, but it does fix many problems, and opens the doors many new implementaions you can only dream of in SMTP. We played out many test cases involving varios spam scenarios using this algorithm, and the results we positive.

However the more questions you have the better for me to explain and find problems with the design. For example Valdis questions raised the bar by making the third part classification service report an honesty ranking. I did not consider this in detail earlier, but it is a good approach. Usage of mail is dynamic, and hence the reporting system has to be dynamic as well. If a business starts spamming, their ranking will immediatly start to change (ala ebay ranking system), which will affect their postage rates. They can ignore centificates completely and have to pay full price for not being classified, or rejected from domains that will not accept non-classified domains.

What I do ask from everyone is to please play the devils advocate to truley understand what the capabilities are, find the shortcoming so I can have the chance to correct them, but not to take a position on deciding that it works or not before I prove or disprove my case :)

Best regards,
Adonis
_______________________________________________________________
Ayna.com the Arabic web starts right here.

Valdis.Kletnieks | 11 Feb 00:27 2003
Picon

Re: Introduction and query

On Mon, 10 Feb 2003 10:41:05 PST, Adonis El Fakih said:

> Max email size 30k which means message >= 30 will pass to be delivered
> (including
> content, if as we receive it the counting algorithm hits 35k, the
> message is dropped)
> this is the difference. We asked the mailer to be honest, if they do
> not, we drop the 
> message. So if they want a message to pass they have to be bound by the
> terms
> of the public policy.

This is already provided by the ESMTP SIZE extension.

> You keep saying that spammer wil lit in their classification, and of
> course
> they will do that. The certificate will tell us what the have been doing
> on other sites,

There's privacy issues and traffic-analysis threats here.

> > Actually, if you think really hard about it, you'll realize that this
> doesn't 
> > *really* stop fake FROM - all it does is make the spammer use a
> throw-away 
> > FROM address that happens to point to a server he controls. 
> > 
> Actually it does, since that host has to be explicitly on the outgoing
> mail list
> for the domain. Let us assume we have 100 servers within out network, as
> a system admin i can assign 4 of them to be used for outgoing mail
> handleing
> i.e. MHFs. The chances of somone hacking onto other non-attended hosts
> is much higher that the ones the admin controls. Also in the case of
> abuse
> it can be pin pointed to the specific host within the domain.

And in your model, it doesn't matter, because the other 96 hosts will be
configured to forward all their mail to the 4 that are MHFs.  So all a spammer
needs to do is find an open formail or proxy, and use that to inject
mail with a forged MAIL FROM:<whatever <at> victim.dom>, and let its MHFs do the
work for it.

> We use this technique today in one of the products we use, where it
> slows down the
> spammer incrementaly, but still does not work. Many proffesional
> spammers will have
> a whole C bock and use them randomly to overcome settings made based on
> host

Which is why the next paragraph said "from a given /24".. ;)

> Which is fine, but his category listing and "honesty" ratings are also
> low, so they will
> have to pay higher rates for mail delivery.

Be careful here that you don't make e-mail too expensive for small sites.

> - One time IP, used to post the millions of messages and then goes
> offline. In AMDP the domai has to be within the outgoing mail scheme of
> the domain, and must stay online so we retrieve the message. Spweing out
> a million message gonly sends out a million envelopes, not the messages.
> if they want to make some cash they will need to stay online to server
> those messages.

And the majority of recipients will try to fetch it right away or fairly soon,
so all they need is another (or even the same IP) address online to catch the
inbound connections, and a DNS entry to point them there.

> - If it is a hijacked MHF once notifed the admin can drop the message
> from queue, and the damage is controled at his end.

Oh, so you're expecting the MHF admin to fix the problem, when these are
probably going to be the same servers that have been open SMTP relays for
years?

> I disagree about these figures, Most of the mail I receive today uses
> fake host
> name, fake mail froms, open relays, fresh relays setup by the spammers
> all 
> over the world. They use cheap ISP accounts in brazil, russia, south
> africa,
> middle east, canada, japan, korea, etc.. 

OK... open relays?  See above comment about MHF servers not being closed.

Fresh relays set up by spammers?  Hmm.. if they can set up a relay,
they can set up a DNS pointer.

Cheap ISP accounts? The ISP will provide a MHF for you.

You're not really addressing the real problems here.

> You are relying on a third party to do the blocking here. Why should it
> not be part 
> of the design?? not everyone has a firwall, and if someone want to go
> around than 
> they can.

Umm.. No.  You're talking about *OUTBOUND* mail here - and if *YOUR* firewall
isn't able to stop outbound SMTP that you don't filter, why do you expect it
to stop outbound AMDP?

> > Once you know that an email is coming from domain A and no one else,
> then we 
> > can go to a third party (that is paid by domain A to be their
> certificate manager) and 
> > check if they are within the category they claim to be. So if domain A
> claims to be 
> > XY category and ends up being ZZ using some smart filters, then we can
> report 
> > the abuse to the manager of the certificate and they update the
> category based on 
> > feedback not only from me, but based on reports received from other
> AMDP sources. 

*BZZT* Wrong, but thank you for playing.  This is called a "paid endorsement"
and is usually about as valid as a sports figure endorsing a brand of life
insurance.   There's no reason to assume that a registrar hired by the
spammer will tell you the truth....

And remember that even Verisign has been caught spamming.  Think about that. ;)

You want to be asking a third party that *YOU* pay to give you good advice,
or at least some third party you can reasonably trust to be neutral...

> I asked this to myself. In SMTP the message goes one way, regardless if
> the
> message is good or bad, however in most cases spam over shadows the good
> mail. By doing the extra handshake i achieve the following.
> 1. I am forcing the sender to be online to tell me that indeed they sent
> me a message

If the spammer can be online to send the message, they can have a second
server online as well.

> 2. I am asking the sender to have that message ready for later pickup

Be careful what you ask for, you may receive it. ;)

Do you think many users will actually *ASK* for "don't bother picking it
up for at least 2 hours"?

Also, note that if the spammer is making a very large run, they'll be online
for a few days doing it, so the chances of them being gone when you call back
aren't that high.

> Also now they can truley know which message was read, and by whom, and
> concetrate
> on creating a business geared on the users, and not random mailings to
> no end..

Read section 6.2 and 6.3 of RFC2298 (Message Disposition Notifications), which
discusses the security implications of this sort of service.

You may also want to ponder the statistics produced by the guys over at
http://cluelessmailers.org - they believe that 90% of the spam in the US is
the result of a small group of 100 to 150 individuals.

That's all the commentary for now, it's been a long and ugly day (nothing
like a print server misbehaving to make your day ;)

/Valdis
Matti Aarnio | 18 Feb 01:43 2003
Picon
Picon

MS Exchange 2000 broke RFC 3030 badly...


I write at this forum, as apparently MS people read this, and
me being a non-customer to them, I have no direct access routes
to send bug reports.

I have implemented ESMTP  CHUNKING / BDAT since 1997,  and now
finally some major implementations do appear -- but sadly first
of them is broken.   The end result is that even if Microsoft
fixes things this week, bug containing versions will be running
for years...     (Shall RFC 3030 be thus considered failure due
to widely installed buggy implementation ?)

In RFC 3030 two ESMTP extensions are defined:
 - CHUNKING
 - BINARYMIME

The buggy implementation is of CHUNKING's BDAT verb processing.

Said verb runs like this:

  BDAT nnnn [LAST] CRLF
  < nnnn bytes of message header+content data with CRLF line ends >

The server shall always consume the message content associated
with BDAT verb.  (Ok, RFC 3030 text isn't explicitely clear at this,
which has lead to a number of implementation bugs, including
the one presently being discussed.)

With simultaneous PIPELINING facility/compability declared by the
remote MTA server, a smtp-client sending in optimized pipeline mode
sends MAIL FROM, one or more of RCPT TO, and also BDAT+message content
all in one TCP data push.

How   MS Exchange 2000   as deployed at Hotmail.COM now does work is:

 - If   MAIL FROM   and   RCPT TO (one or more) are received
   successfully,  BDAT will be accepted successfully.

 - If for some reason all RCPT TOs are rejected, treatment of BDAT
   is broken.  Associated data is not consumed (discarded), and
   the result is unpredictable number of "500 Unrecognized command"
   messages, as the message content ended up in SMTP command processing.

   Following UNIX bash-script shows what happens.

(echo "EHLO foo";sleep 1;
echo -e "MAIL FROM:<>\nRCPT TO:<lklkzzh <at> hotmail.com>\nBDAT 9 LAST\n1\n2\n3\n";
sleep 2;
echo -e "MAIL FROM:<>\nRCPT TO:<lklkzzh <at> hotmail.com>\nBDAT 9 LAST\n1\n2\n3\n";
sleep 3) | telnet mx1.hotmail.com smtp

   Doing network protocol dump, one can see that from MAIL FROM to
   BDAT content data all are sent with single TCP data push, as is
   the nature of smtp-client in fully developed PIPELINING mode.

   Here are the received responses:

220 mc1-f8.law16.hotmail.com Microsoft ESMTP MAIL Service, Version: 5.0.2195.5600 ready at  Mon, 17 Feb
2003 16:23:42 -0800 
   <<-  EHLO foo
250-mc1-f8.law16.hotmail.com (02.01.00.0007) Hello [62.240.94.4]
250-SIZE 4278190
250-PIPELINING
250-8bitmime
250-BINARYMIME
250-CHUNKING
250-VRFY
250-AUTH LOGIN
250-AUTH=LOGIN
250-X-HMAUTH
250 OK
   <<- MAIL FROM:<>
250 <>....Sender OK
   <<- RCPT TO:<lklkzzh <at> hotmail.com>
550 Requested action not taken: mailbox unavailable
   <<- BDAT 9 LAST (+ data)
503 Need Rcpt command.
500 Unrecognized command
500 Unrecognized command
500 Unrecognized command
   <<- MAIL FROM:<>
503 Sender already specified
   <<- RCPT TO:<lklkzzh <at> hotmail.com>
550 Requested action not taken: mailbox unavailable
   <<- BDAT 9 LAST (+ data)
503 Need Rcpt command.
500 Unrecognized command
500 Unrecognized command
500 Unrecognized command

   Fully pipelined smtp client code does not expect
   any of those "500 Unrecognized command" lines, and
   goes out of protocol sync.

--

-- 
/Matti Aarnio	<mea <at> nic.funet.fi>


Gmane