ok..I'll target submitting the updated draft by monday.

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
vivk <at> us.ibm.com
kashyapv <at> us.ibm.com
Ph: 503 578 3422 T/L: 775 3422

"Dror Goldenberg" <gdror <at> mellanox.co.il> Sent by: ipoverib-bounces <at> ietf.org 11/15/2005 03:04 PM

To:        "H.K. Jerry Chu" <Jerry.Chu <at> eng.sun.com>, <gdror <at> mellanox.co.il>, kashyapv <at> us.ltcfwd.linux.ibm.com         cc:        ipoverib <at> ietf.org, vandana.rao <at> hp.com         Subject:        RE: [Ipoverib] ipoib-cm multiple connection model

I think it's OK. We can move forward from my perspective.

> -----Original Message-----
> From: H.K. Jerry Chu [mailto:Jerry.Chu <at> eng.sun.com]
> Sent: Monday, November 14, 2005 9:34 PM
> To: gdror <at> mellanox.co.il; kashyapv <at> us.ibm.com
> Cc: ipoverib <at> ietf.org; vandana.rao <at> hp.com
> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
>
>
> Hi folks,
>
> Have we reached a concensus the connection setup procedure in
> the current draft is "good enough", or folks still think it
> needs to be improved?
>
> I'd like to see any remaining issue resolved asap so we can
> start a WG last call (after Vivke revs the draft based on all
> the comments so far).
>
> Jerry
>
> >Date: Thu, 27 Oct 2005 22:47:16 -0700 (PDT)
> >From: Vivek Kashyap <kashyapv <at> us.ibm.com>
> >X-X-Sender: kashyapv <at> localhost.localdomain
> >To: Dror Goldenberg <gdror <at> mellanox.co.il>
> >Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >MIME-Version: 1.0
> >X-Spam-Score: 0.0 (/)
> >X-Scan-Signature: f8ee348dcc4be4a59bc395f7cd6343ad
> >Cc: "'ipoverib <at> ietf.org'" <ipoverib <at> ietf.org>, Vandana Rao
> ><vandana.rao <at> hp.com>
> >X-BeenThere: ipoverib <at> ietf.org
> >X-Mailman-Version: 2.1.5
> >List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
> >List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
> <mailto:ipoverib-request <at> ietf.org?subject=unsubscribe>
> >List-Post: <mailto:ipoverib <at> ietf.org>
> >List-Help: <mailto:ipoverib-request <at> ietf.org?subject=help>
> >List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
> <mailto:ipoverib-request <at> ietf.org?subject=subscribe>
> >
> >Hi Dror,
> >
> >On Thu, 27 Oct 2005, Dror Goldenberg wrote:
> >
> >> Hi Vivek,
> >>
> >> I agree with the flows that you describe, and with the
> fact that the
> >> original requester has always the "second chance" to withdraw its
> >> request by rejecting the REP. However, in my opinion,
> giving all this
> >> freedom to the implementation can lead to interoperability
> issues and
> >> poor implementations. If an implementation tries to open n
> >> connections with a certain peer, it may end up opening
> more or less
> >> than n and that might take few iterations and that all
> depends on the
> >> CM timing and some other "unrelated" parameters - how can you even
> >> test it ? I therefore feel that we should provide a robust and
> >> deterministic specification of how to establish more than one
> >> connection in the RFC or live with a single connection.
> >
> >I'd like to delve into this a bit more to be sure. As per
> >your/Vandana's comments, the left peer will cancel the
> request. Isn't
> >this is a 'conscious' decision by the ULP? The ULP would then remove
> >the its internal object that it would associate with this IB
> >connection. If it was successful in cancelling the REQ then there is
> >only one connection.
> >
> >If, however, as you pointed out it is unable to, it might
> receive a REP
> >*if* the remote peer accepts multiple connections. The left
> peer's ulp
> >however will not find any object associated with the private
> data and
> >the connection request (there will be another one already existing
> >though) and can use that to indicate rejection.
> >
> >Does the above sound right?
> >
> >Vivek
> >
> >>
> >> BTW, this is my opinion in this case, and if folks feel OK
> with the
> >> current draft, then maybe we should move on ?
> >>
> >> Thanks
> >> Dror
> >>
> >>> -----Original Message-----
> >>> From: Vivek Kashyap [mailto:kashyapv <at> us.ibm.com]
> >>> Sent: Wednesday, October 26, 2005 10:33 PM
> >>> To: Dror Goldenberg
> >>> Cc: Vandana Rao; 'ipoverib <at> ietf.org'
> >>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>>
> >>>
> >>> On Wed, 26 Oct 2005, Dror Goldenberg wrote:
> >>>
> >>>> Vivek,
> >>>>
> >>>> I agree that the two scenarios you mention here are the
> interesting
> >>>> ones. I don't agree about the solution. Let's look at the two
> >>>> cases:
> >>>>
> >>>> 1) Left peer accepts
> >>>>                  I disagree. You have an assumption that left peer can
> >>>>                  revoke its original connection request - BEFORE transmitting
> >>>>                  the REP that accepts the peer request. I am not sure that
> >>>>                  you can do it with every CM implementation. Even if you can,
> >>>>                  then the HCA can reorder those packets if they are on
> >>>>                  different QPs, and the fabric can reorder them as well if
> >>>>                  they are on different VLs.
> >>>
> >>> Yes, that has the potential for extra connections..however, the
> >>> following will occur if CM checks don't work.
> >>>
> >>> Left peer accepted. Sent a REP. And of course the REQ was
> also sent.
> >>> The right receiver can receive :
> >>>
> >>>                   a. REP first
> >>>
> >>>                   It sets up the connection. Then receives a REQ.
> >>>
> >>>                   i) If it accepts multiple connections from the same peer:
> >>>                                     it will  REP with a new QP (connection).
> >>>
> >>>                                    Now, if the left peer decides that it didn't
> >>> need another
> >>>                                    connection it terminates the connection.
> >>>
> >>>                                    Otherwise, it is legitimate to setup multiple
> connections
> >>>                                    since it is equivalent to a second valid request.
> >>>
> >>>                   ii) If the right peer does not accept multiple connections from
> >>> the
> >>>                   same peer then it will reject the request.
> >>>
> >>>                   Note that these decisions in part depend on the ipoib component
> >>> peeking
> >>>                   into the local link address and not on CM.
> >>>
> >>>
> >>>                   b. REQ first
> >>>
> >>>                   Since its own REQ is outstanding, and as per link
> >>> address comparison,
> >>>                   the left peer accepts then this will be rejected.
> >>>
> >>> Vivek
> >>>>
> >>>> 2) Left peer rejects
> >>>>                  I think that this scenario works.
> >>>>
> >>>> Thanks
> >>>> Dror
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Vivek Kashyap [mailto:kashyapv <at> us.ibm.com]
> >>>>> Sent: Tuesday, October 25, 2005 11:38 PM
> >>>>> To: Vandana Rao
> >>>>> Cc: Dror Goldenberg; 'ipoverib <at> ietf.org'
> >>>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>>>>
> >>>>>
> >>>>> Hi Vandana,
> >>>>>
> >>>>> I too agree there is no problem here.
> >>>>>
> >>>>> I think you are right if the left peer accepts the connection.
> >>>>> However, if the reverse was the case, that is the left
> >>> peer rejected
> >>>>> the connection since it is found that the local address is
> >>>>> numerically larger. It will do this comparison since its
> >>> REQ is also
> >>>>> outstanding.  The right peer will not receive any REQ  (it
> >>> is delayed
> >>>>> or yet to be retransmitted). If it receives the reject it
> >>> will cancel
> >>>>> its request. At a later time it will receive the
> >>>>> delayed/retransmitted request and one connection will be
> >>>>> established.
> >>>>>
> >>>>> Dror, do these two scenarios look right to you?
> >>>>>
> >>>>> Vivek
> >>>>>
> >>>>> On Thu, 20 Oct 2005, Vandana Rao wrote:
> >>>>>
> >>>>>> Hi Dror,
> >>>>>> I think you are confusing the 2 IDs that come into play in
> >>>>> this setup.
> >>>>>> The
> >>>>>> communication ID is something that the CM entity on a node
> >>>>> understands but
> >>>>>> the transaction ID is something that the MAD handling
> >>>>> entity uses (for
> >>>>>> retransmission, req/resp matchup etc.).
> >>>>>> So, in the second slide, there should not be a
> >>>>> retransmission of the left
> >>>>>> peer REQ MAD since when the left peer accepts the
> >>>>> connection, it will cancel
> >>>>>> its outstanding REQ request and hence the retransmission
> >>>>> should not take
> >>>>>> place.
> >>>>>> -Vandana
> >>>>>>
> >>>>>> At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
> >>>>>>> After giving it some more thought, I think that there
> is still a
> >>>>>>> problem. Please look at the attached pdf.
> >>>>>>>
> >>>>>>> In the first slide, everything happens smoothly, the left
> >>>>> peer knows
> >>>>>>> that
> >>>>>>> it has to accept the connection and the right peer knows
> >>>>> that it has to
> >>>>>>> decline.
> >>>>>>> On the 2nd slide, the left peer REQ transmission is
> >>>>> dropped in the fabric.
> >>>>>>> It is retransmitted independently by the CM way after a
> >>>>> channel has already
> >>>>>>> been established. Now the right side can not really tell
> >>>>> whether this is a
> >>>>>>> new channel request or an old "simultaneous connection
> >>>>> channel". This may
> >>>>>>> lead to two channels being opened mistakenly.
> >>>>>>>
> >>>>>>> I don't believe it can be solved by looking at the
> >>>>> communication ID.
> >>>>>>> This
> >>>>>>> number is just being fabricated by the sender of the REQ
> >>>>> and is just a
> >>>>>>> "random number" with respect to what we're trying to
> determine.
> >>>>>>>
> >>>>>>> -Dror
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Dror Goldenberg
> >>>>>>> Sent: Wednesday, October 19, 2005 9:37 PM
> >>>>>>> To: 'Vandana Rao'; Dror Goldenberg; ipoverib <at> ietf.org
> >>>>>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>>>>>>
> >>>>>>> Hi Vandana,
> >>>>>>>
> >>>>>>> Thanks for clarifying this. I agree that if IPoiB-CM
> >>>>> implementation
> >>>>>>> looks
> >>>>>>> at the Communication ID it can make those decisions
> >>>>> correctly. The question
> >>>>>>> is whether this value is exposed through common CM APIs.
> >>>>> For all other
> >>>>>>> purposes, the CM doesn't need to expose the Communication
> >>>>> ID to the ULP.
> >>>>>>> However, seems like it's not a big deal to expose this
> >>>>> value to the ULP.
> >>>>>>> Anyway, it worth noting this explanation in the RFC so
> >>>>> that it's clear that
> >>>>>>> a proper implementation of IPoIB-CM should monitor
> >>>>> Communication ID as part
> >>>>>>> of the simultaneous connection decision.
> >>>>>>>
> >>>>>>> -Dror
> >>>>>>> -----Original Message-----
> >>>>>>> From: Vandana Rao [mailto:vandana.rao <at> hp.com]
> >>>>>>> Sent: Wednesday, October 19, 2005 6:40 PM
> >>>>>>> To: Dror Goldenberg; ipoverib <at> ietf.org
> >>>>>>> Subject: Re: [Ipoverib] ipoib-cm multiple connection model
> >>>>>>>
> >>>>>>> Hi Dror,
> >>>>>>> The purpose of the MAD transaction ID is to allow one to
> >>>>> do what you
> >>>>>>> say
> >>>>>>> below, i.e. have multiple connection establishment MADs
> >>>>> between 2 peers. It
> >>>>>>> is essentially the same as the "channel ID" that you talk
> >>>>> about below. Its
> >>>>>>> just that it is a generic channel ID for any MAD
> >>>>> transactions, not just CM
> >>>>>>> MADs.
> >>>>>>> I do not believe there is any issue with existing IB CM in
> >>>>> this regard.
> >>>>>>> -Vandana
> >>>>>>>
> >>>>>>> At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
> >>>>>>>> I think that the current model for multiple connections
> >>>>>>>> establishment between two peers works well for 1
> connection but
> >>>>>>>> is broken for
> >>>>> more than one
> >>>>>>>> connection.
> >>>>>>>> Because of cases where CM packets are dropped or travel
> >>>>> on different VLs,
> >>>>>>>> they
> >>>>>>>> might get reordered on the way and observed in a
> >>>>> different order than were
> >>>>>>>> originally
> >>>>>>>> sent. This breaks the current model which assumes that
> >>>>> the REQs are sent
> >>>>>>>> and
> >>>>>>>> processed in order.
> >>>>>>>>
> >>>>>>>> I think that we need to think of a model which is
> robust to CM
> >>>>>>>> packets reordering which can be one of:
> >>>>>>>> - Adding a "channel ID" in the private data. The field
> >>>>> goes 0,1,2 and on.
> >>>>>>>> It helps
> >>>>>>>>    synchronizing both peers on which is the channel at
> >>>>> question now. The
> >>>>>>>> rest
> >>>>>>>>    of the decision (per channel) of whether to accept or
> >>>>> reject can be the
> >>>>>>>> same
> >>>>>>>>    as defined in the draft.
> >>>>>>>> - Not supporting multiple connections between the same peers.
> >>>>>>>>
> >>>>>>>> -Dror
> >>>>>>>> _______________________________________________
> >>>>>>>> IPoverIB mailing list
> >>>>>>>> IPoverIB <at> ietf.org
> >>> https://www1.ietf.org/mailman/listinfo/ipoverib
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
> >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB <at> ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib
>

_______________________________________________
IPoverIB mailing list
IPoverIB <at> ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
IPoverIB <at> ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib