Benny Halevy | 11 Oct 22:49 2014

Re: Fwd: New Version Notification for draft-haynes-nfsv4-versioning-00.txt

Glad to see progress on this and I like the overall direction.
From first reading I have a couple editorial level comments.

1) Did you avoid capitalization of "may" and "must" on purpose?

For example:
Minor versions may append attributes
Minor versions must not modify the structure of an existing operation's arguments or results.


A client and server that support minor version X SHOULD support minor versions 0 through X-1 as well.
2 There seems to be an extra sentence pasted here:
Such feature definition documents would contain a number of Addition
 of features to an existing minor version will take advantage of the
3) The following paragraph need to refer to servers supporting v4.2 and up.
o Existing server implementations that do not recognize the new arm of a switched union will return will return NFS4ERR_INVAL or NFS4ERR_UNION_NOTSUPP, enabling the client to determine that the the new union arm is not supported by the server.
4) Security and IANA considerations
Rather than stating that there are no considerations I'd basically repeat what was said above:

o Minor version and feature specification will include the "Security Considerations" / "IANA considerations" (respectively) section
as required for RFC publication.

5) IANA considerations
So who's assigning values to added bits / enums for new features and when in the process?
I envision multiple features being worked on in parallel by individuals so they may clash.
When accepted as a W/G document we can serialize the enumerations, yet if one a the features
lingers behind and may eventually be aborted we might end up with a "hole" in the enumeration.
Hence I think we (W/G with the help of the RFC Editor) must assign the final values upon becoming an RFC.

This could be documented as a non-consideration for IANA in this section.

Does this make sense?


On Mon, Oct 6, 2014 at 2:54 PM, Tom Haynes <thomas.haynes <at>> wrote:
And here it is, I ended up having to resubmit as draft-haynes-nfsv4-versioning. ;-)

Begin forwarded message:

Subject: New Version Notification for draft-haynes-nfsv4-versioning-00.txt
Date: October 6, 2014 at 2:54:14 PM EDT

A new version of I-D, draft-haynes-nfsv4-versioning-00.txt
has been successfully submitted by Thomas Haynes and posted to the
IETF repository.

Name: draft-haynes-nfsv4-versioning
Revision: 00
Title: Minor versioning Rules for NFSv4
Document date: 2014-10-06
Group: Individual Submission
Pages: 12

  This document specifies the minor versioning rules for NFSv4.  It
  also specifies how those minor versioning rules may be modified.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at

The IETF Secretariat

nfsv4 mailing list
nfsv4 <at>

nfsv4 mailing list
nfsv4 <at>
David Noveck | 9 Oct 14:30 2014

Re: proposed text for v4.2-based conversion to a more incremental protocol extension model

>  I’m going to push out an update with Dave’s micro versioning text.

Just to clarify, I'm not proposing anything like 4.2.1.   A more incremental versioning model would address the same issues that micro-versioning was intended to, but it does so in a different way.

On Mon, Oct 6, 2014 at 2:49 PM, Tom Haynes <thomas.haynes <at>> wrote:

On Oct 5, 2014, at 8:51 AM, David Noveck <davenoveck <at>> wrote:

> I'd actually like to see us pull the minorversioning out of 4.2 with a statement like:

Good idea.  Not only is it the proper response to the fact that minor versioning rules
do not belong in individual minor version documents, it also makes the v4.2 spec shorter.

It puts me under the three-page limit.  How does minus-one pages sound? :-)

> At the time this document was specified, the minorversioning are detailed in
> RFC5661. The implementer of the NFSv4.2 specification is expected to track
> the evolution of the minorversioning standard.

I think the problem may be that someone (for example, in the IESG) may ask "what minor versioning standard?".    I expect we would have to convert draft-haynes-nfsv4-minorversioning to a working group standards-track document and have our co-chairs explain this to the IESG.   I hope we are covered on charter-related issues.

So, speaking of that draft, I’m going to push out an update with Dave’s micro versioning text.

nfsv4 mailing list
nfsv4 <at>
Matt W. Benjamin | 8 Oct 16:08 2014


----- "David Quigley" <dpquigl <at>> wrote:

> xattr for its internal access control but anything beyond that I don't
> think is ok. Trond in the past has expressed concerns about people 
> establishing arbitrary sub protocols by using xattrs. I agree that
> this 
> is a problem because it opens up the system to interoperability issues
> by encouraging people to bypass any sort of standardization effort and
> just using the excuse that its just opaque data.

I find this argument a bit strange.  A general-purpose tool like xattr allows
application developers and users to evolve subprotocols to satisfy real use
case scenarios.  The need to interoperate arises naturally to reduce/remove
impedence amongst competing approaches/implementations, so that what is standardized
arises from existing practice and experience.



Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

nfsv4 mailing list
nfsv4 <at>

Dave Quigley | 8 Oct 03:08 2014

Re: request for xattr as a WG item

On 10/6/2014 6:11 PM, Marc Eshel wrote:
> We added a section to the draft based on your request to explain why it is
> needed, but you are still not convinced. I think that you are looking for
> a killer application that will prove the need for xattr and I don't know
> of one. all I know is what we added to the draft and what customers tell
> us, but it is not enough for you. So I don't know how to convince you,
> others on the WG seem to agree that it is needed, I assume based on their
> knowledge and experience. I can not give you a name of an application that
> a customer is running because I don't know the name and even if I did it
> would mean nothing, all I can say that for example one application is used
> to tag files using xattr for future migration, it is like a policy for
> file movement among the different storage levels. Maybe you should talk to
> your customers and see if they needed it.

The thing is that just having a section justifying the feature is not 
enough. I learned that when I started Labeled NFS. I started with a 
requirements document. I then was asked to a use case document, an 
impact study, a requirements document, and finally the actual 
specification which got merged into 4.2. Compared to the varied 
semantics of xattr implementations on different systems security labels 
were easy. We punted on syntax and semantics and abstracted that out 
into a separate document. We punted on establishing a security model and 
said it was out of scope since its really a platform specific 
implementation detail.

If you want a way of laying out your case for xattrs you can try 
following what I started for Labeled NFS and what Tom continued and 
finally shepherded through the 4.2 adoption process. We have specific 
use cases in there for security labels. I'm sure you can find real 
concrete examples for the equivalent of the user namespace style xattrs 
from Linux.


nfsv4 mailing list
nfsv4 <at>

Saul Tamari | 8 Oct 00:13 2014

Re: request for xattr as a WG item


Maybe it is possible to add a new and simplified named-attribute model
for NFSv4.0 while keeping the NFSv4.0 XDR?
This simplified model will not support GETATTR/SETATTR for the
attributes and remove the filesystem inode requirement.

With regard to the OPEN/CLOSE over the wire overhead, these are only
required when creating/modifying an attribute, while READ (which I
assume is the most common call) can take an all 1's stateid without
OPENing the file.


On Tue, Oct 7, 2014 at 6:00 AM, Christoph Hellwig <hch <at>> wrote:
> On Tue, Oct 07, 2014 at 12:05:14AM -0400, Tom Haynes wrote:
>> Why don’t named attributes meet these needs?
> They are much higher overhead, both on disk and on the wire.  For example
> an xattr on XFS consumes just the size of the key + attr in the best
> case, and some additional btree linkage in the worse case.  A named attribute
> (which is just a horrible name for a file that is hidden from the normal
> namespaces) needs a full inode for the stat data, a directory entry, and
> often a block/extent to store the actual data.
> Similarly on the wire you need CTO semantics, explicit OPEN / CLOSE calls,
> GETATTRS and so on.
> _______________________________________________
> nfsv4 mailing list
> nfsv4 <at>

nfsv4 mailing list
nfsv4 <at>
Manoj Naik | 7 Oct 21:00 2014

Re: request for xattr as a WG item

Christoph Hellwig <hch <at>> wrote on 10/07/2014 02:44:09 AM:

> From: Christoph Hellwig <hch <at>>
> To: Manoj Naik/Almaden/IBM <at> IBMUS
> Cc: Trond Myklebust <trondmy <at>>, "nfsv4 <at>" <nfsv4 <at>>
> Date: 10/07/2014 02:44 AM
> Subject: Re: [nfsv4] request for xattr as a WG item
> On Mon, Oct 06, 2014 at 03:20:49PM -0700, Manoj Naik wrote:
> > No they shouldn't. Extended attributes is a file system feature that is,
> > for all practical purposes, well defined for the platform/file system - if
> > not interoperable. (Which is why I agree we need to state _requirements_ in
> > the protocol). Customers are free to use them as they wish, they have no
> > obligation to tell you what they put in there and why.
> >
> > Today, I can cp (or rsync) a file from one filesystem (say XFS) to another
> > (say ext4) across two different Linux systems and it'll preserve xattrs.
> > But I can't do the same over NFS. This should be sufficient to establish a
> > "need" IMO. Obviously, you disagree.
> Two Linux filesystems aren't too interesting.  You need to demonstrate
> interoperability between say XFS on Linux and FFS on FreeBSD exported over
> NFS.

I don't have access to FreeBSD, but I can preserve xattrs with rsync (-aX) between Linux/ext4 and OSX/hfs. rsync will correctly strip off the Linux "user." prefix and copy all user xattrs.


nfsv4 mailing list
nfsv4 <at>
Christoph Hellwig | 4 Oct 18:17 2014

xattr model in draft-naik-nfsv4-xattrs-01?

Hi Manoj,

can you explain what xattr model your draft is based on?  It seems to
have extremly complicated semantics that include a lot of semantics
not present in Linux/IRIX/FreeBSD xattrs.

On the GETXATTR side you conflate the LISTXATTR operation with GETXATTR,
as well as adding a new bulk get one that isn't present in any OS API
I know.  Both the argument and return structures are basically entirely
different for the different cases.

I think we would be much better of starting with:


struct GETXATTR4args {
	/* CURRENT_FH: file */
	xattrname4	ga_name;

struct GETXATTR4res {
	xattrvalue4	gr_value;


struct LISTXATTR4args {
	/* CURRENT_FH: file */

struct LISTXATTR4res {
	xattrname4	gr_names<>

Also how is the client supposed to cope with sizes of both each
xattr and the list?  Don't we need a maxcount/mincount scheme similar

In the SETXATTR I would suggest to remove the _ALL types, and the
multiplexing not present in any OS API.  Additionally we need an
anything goes type that can create or replace to properly support
the existing Linux/IRIX/FreeBSD semantics.  The remove case could
be roled into this - while none of the OS APIs do this, the Linux
kernel internal API does and it works fairly well, although I still
think a separate operation would be cleaner protocol wise:

enum setxattr_type4 {

struct SETXATTR4args {
	/* CURRENT_FH: file */
	setxattr_type4	sa_type;
	xattrname4	sa_xattrname;

struct SETXATTR4res {
	nfsstat4 sr_res;

struct REMOVEXATTR4args {
	/* CURRENT_FH: file */
	xattrname4	sa_xattrname;

struct REMOVEXATTR4res {
	nfsstat4 sr_res;

A few more comments on the standard outside of this high level model:

In Section 2. mentioning Solaris is wrong - Solaris never exposed xattr-like
APIs, just name attributes which are different and supported in NFSv4.

Instead of mentioning NTFS you should mention the API, preferably with
an informative reference.

Asking about not interpreting the name is a desaster waiting to happen
given that Linux differs from the other implementations in how it handles
the name.  I sent a separate mail about this earlier today, but I think
trying to handle anything but the "user" xattrs in this protocol would
be a grave mistake, so a Linux client would have to strip the "user."
and a Linux server would have to add this "user."

Section 5.1:  a very useful attribute missing here is the maximum size
of each individual attribute.

nfsv4 mailing list
nfsv4 <at>

David Noveck | 3 Oct 14:53 2014

proposed text for v4.2-based conversion to a more incremental protocol extension model

I've attached a new section to be added, if people are OK with it, to the NFSv4.2 spec.  The idea is that, if we want to move to a less monolithic development model, we should do it as soon as possible.  We had  been thinking that "as soon as possible" meant on the next minor version to start out (i.e. v4.3), but i think we could do this in the next minor version to go to IETF last call, namely NFSv4.2.  If that seems too abrupt for people we could still use the same sort of scheme in v4.3, but if we do, we should do that with the  understanding that v4.3 would be a micro-version, consisting of extensibility and not much else (I think xattrs are far enough along to merit inclusion but others might not agree).  That might enable us to have a small but extensible v4.3 out within a year.

This is not within Tom's three-page limit.  It is about four pages and would allow the v4.2 spec to easily stay under 100 pages, as Tom thinks important (and so do I).

As Tom has pointed out, there is a lot of interest in a less monolithic, more running-code-oriented protocol development process.   I think this is easier than many people believe because the NFSv4 XDR extension model gives us the infrastructure we need.   I think the issue is not the level of working group energy, but a general sense of uncertainty about what is possible here, plus the press of other business (i.e. this is not anybody's full-time job).

I think that if people realize that this can be done and the working group decides that this is important, we can effect the necessary transition.    

I appreciate people''s comments and suggestions.  However,  I will be reluctant to try Tom's patience by adding much more material when I'm already over three pages.   In the context of a v4.3-based proposal, we have more flexibility, but I'd hope we can  keep any spec on a new extension model down to 20-25 pages.

Section 1.6: NFSv4.2 as an Extensible Protocol

An additional goal of NFSv4.2 is to enable extensions to be made to the
features included in this document, without the overhead attendant upon 
the creation of an entirely new minor version.

Each such extension will be in the form of a working-group standards-
track document which defines a new optional feature.   The definition
of the new feature may include one or more "feature elements" which
add to the existing XDR in ways already used in creating new minor
versions.  Other sorts of XDR modification are not allowed.  Feature
elements include new operations, callbacks,  attributes, and enumeration
values.  The functionality of some existing operations may be extended by 
the addition of new flags bits in existing flag words and new cases in 
existing switched unions.  New error codes may be added but the set of 
valid error codes to be returned by an operation is fixed, except that 
existing operations may return new errors to respond to situations that 
only arise when previously unused flag bits are set or when extensions 
to a switched union are used.

Such an additional feature will become, for all intents and purposes, 
part of NFSv4.2 upon publication of the description as a Proposed 
Standard, enabling such extensions to be used by new client and server
implementations without, as previously required, a change in the value
of the minorversion field with the COMPOUND operation.  

The working group has two occasions to make sure that such features are 
appropriate ones:

   o At the time the feature definition document becomes a working group
     document, the working group needs to determine, in addition to the
     feature's general compatibility with NFSv4, that the XDR 
     assignments (i.e. additional values for operation callback and 
     attribute numbers, and for new flags and switch values to be added 
     to existing operations) associated with the new feature are 
     complete and do not conflict with those in the existing protocol or 
     those currently under development.

   o At the time the working group document is complete, the working 
     group, in addition to, normal document review, can look at 
     what prototype implementations of the feature have been done and 
     use that information to determine the workability of the

Such feature definition documents would contain a number of items, 
following the pattern of this specification.  The only difference 
would be that while this specification defines a number of features to
be incorporated in NFSv4.2, the feature definition documents would each 
define a single feature.  

In addition to a general explanation of the feature in question, the 
items to be included in such feature definition documents would be:

  o Description of new operations (corresponding to sections 16 and 17
    of the current document).
  o Description of any modified operations (corresponding to section 
    15 of the current document).
  o Description of new attributes (corresponding to section 13 of the 
    current document). 

  o Description of any added error codes (corresponding to section 12.1 
    of the current document). 

  o A summary description all changes made by this feature to the xdr
    defunion of the protocol, including operation codes, attribute numbers, 
    added flag bits and enumeration values, and request and response 
    structures for new iperation together with the other xdr extensions 
    needed to support them.
  o A listing giving the valid errors for each new operation and callback
    (corresponds to sections 12.2 and 12.3 of the current document). 

  o A table giving for each new feature element its status (always 
    OPTIONAL), and its relationship to the feature being described 
    (i.e. REQUIRED for every implementation of the feature, or OPTIONAL
    in the presence of the feature).  This would be similar to the 
    material in section 14 of the current document but it is restricted
    to a single feature and expanded in scope to include all feature 

  o All of the sections required for RFC publication, such as "Security
    Considerations", "IANA considerations", etc.

Addition of features to NFSv4.2 will take advantage of the existing 
NFSv4 infrastructure that allows optional features to be added to minor 
versions, but without in this case requiring any change the version 
number.  This will enable compatibility with existing clients and 
servers.  In particular:

  o Existing server implementations will return NFS4ERR_NOTSUPP in 
    response to any use of the new operation, allowing the client to 
    determine that the requested (and potentially the feature in 
    question) is not supported by the server.

  o Clients can determine whether particular new attributes are 
    supported by a given server by examining the value returned as the
    value of the supported_attr attribute.

  o New callbacks will only be sent to clients that have used the
    new features associated with them, allowing existing clients to be
    unaware of their existence.

  o Existing server implementations that do not recognize new flag
    bits will return NFS4ERR_INVAL, enabling the client to determine 
    that the new flag value is not supported by the server.

  o Existing server implementations that do not recognize the new
    arm of a switched union will return will return NFS4ERR_INVAL
    or NFS4ERR_UNION_NOTSUPP, enabling the client to determine that the
    the new union arm is not supported by the server.

  o Error values returned to the client for all requests that do not use
    new features will only be those previously allowed.  Only when the 
    client uses a new feature will a previously invalid error value be 

Given that some existing servers may have XDR parsing implementations
that cannot easily accommodate previously unknown operations or
switched union arms, clients should carefully determine whether
particular features are supported by the server before proceeding to use 
them and need to be prepared to treat NFS4ERR_BADXDR as indicating 
non-support of a new operation or switched union arm where server
support for a particular feature is being tested.

Additional documents will be required from time to time.   The purpose
of these documents will be to organize existing material so that an
implementer will not have to scan a large set of feature definition
to find information being sought.  The frequency of updates for such
documents will be affected by implementer needs and the ability to 
easily generate such documents, preferably by automated means.  These
documents will be informational documents whose purpose is to simplify 
use of the standards-track documents.  Some desirable elements would 

  o An updated XDR for the protocol as a whole including feature
    elements from all features accepted as Proposed Standards.

  o A consolidated list of XDR assigments of values (e.g operation 
    codes, attribute numbers, error codes, new flag bits, enumeration 
    extensions) for all features under developmenT (i.e. accepted as 
    working-group standards-track documents byut not yet published or

  o A list of all feature definition documents that have been approved 
    as working group documents but have not yet been approved as
    proposed standards.

  o A table mapping operations and callbacks to the most recent 
    document containing a description of that operation.

  o A table mapping attributes to the most recent document containing 
    a description of that attribute.

  o A table giving, for each operation in the protocol, the errors
    that may validly be returned for that operation.  If possible, it
    would be desirable to give, as does RFC5661, the operations which
    may validly return each particular error.

  o A table giving for each operation, callback, and attribute and for 
    each feature element in a published extension giving its status  
    and its relationship to the feature which allows its inclusion
    (i.e. REQUIRED for every implementation of the feature, or OPTIONAL
    in the presence of the feature).  This would be similar to the 
    material in section 14 expanded in scope to include all feature 
    elements, and updated to include all published features that are 
    part of NFSv4.2, at the date of publication.

While making NFSv4.2 extensible will decrease the need for new minor 
versions, it will not eliminate them.  In particular,

  o A new minor version will be required for any change in the status of
    a feature element (i.e. an operation, callback, attribute, added flag 
    or switch case).  For example, changes which make operations
    Recommended, Required or Mandatory to Not Implement will require 
    a minor version.

  o Any incompatible semantic change in the required or allowed processing 
    of an existing operation or attribute will require a minor version.

  o Any change that extends the set of operations that an existing
    operation, with the exception noted above.  New errors may be 
    added when the conditions that give rise to these new errors cannot
    arise as long as new flag bits or switched union arms are not used
    i.e. when it s clear that existing client cannot receive these
  o Any change in the mapping of feature elements to features will
    require a minor version.  For example, if a feature is to be split 
    into two separate features clients would no longer be able to infer
    support for one operation from support for the other, in the same
    way that had been done previously, invalidating logic in existing 
nfsv4 mailing list
nfsv4 <at>
Tom Haynes | 29 Sep 17:48 2014

Section the delegation. See Section 18.7.4 of RFC 5661 for details [2]. Consequently, if a client needs to verify the list of extended attributes with the server, it must also query the change attribute of the file with GETATTR. This handling is similar to how a client
Is that a must or a MUST?

The SETXATTR operation changes one or more of the extended attributes of a file system object. The change desired is specified by sr_type. SETXATTR4_CREATE is used to associate the specified values with the extended attribute keys for the file system object specified by the current filehandle. The server MUST return an error if the attribute key already exists. SETXATTR4_REPLACE is also used to set an xattr, but the server MUST return an error if the attribute key does not exist. An application can delete all existing xattrs for a file and replace them with a new set by using SETXATTR4_REPLACE_ALL. SETXATTR4_DELETE can be used to remove the specified xattr keys, if they exist. SETXATTR4_DELETE_ALL removes all the xattr keys for the file.

Use a hanging list for these…

While the SETXATTR request MAY contain multiple attribute keys and/or values to be changed for a file, this does not impose any atomicity considerations on the server implementation. If the server cannot update all the attributes for the file atomically, it MUST set them in the order specified. In such cases, it is possible that some keys are changed successfully while others encounter errors. To handle this, contained within the SETXATTR results is a "status" field. If any of the change operations incur an error, then the "status" value MUST NOT be NFS4_OK. In this case, the status of the individual change operations is returned in sr_array. If the xattr keys and/or values contained in the client request exceeds the channel's negotiated maximum request size, then the server MUST return NFS4ERR_REQ_TOO_BIG in sr_status.

I thought in earlier drafts that the atomicity requirements were what kept us from using named attributes.
If there are no longer any atomicity requirements, does that open the door back for using named attributes?

A successful SETXATTR SHOULD change the file time_modify and change attributes. However, these attributes SHOULD NOT be changed unless the xattrs are changed.

And an unsuccessful SETXATTR can do what? I.e., what if some keys are changed successfully and some are not?

Also, I find the two sentences to be contradictory. Under what circumstances could the SETXATTR be successful and not change the xattrs?


6 Security Considerations The additions to the NFS protocol for supporting extended attributes do not alter the security considerations of the NFSv4.1 protocol [2].

I’m not sure this is as simple as suggested here.  You should talk about what could happen if the NFS client interprets the value.

Also, wasn’t an earlier concern that these payloads could be executable code? I thought we agreed to disallow this?

Finally, can these values be used as attack vectors?  I.e., the client may already be exposed to them from local xattrs, but doesn’t having the xattrs being stored away from the client introduce some risk?
nfsv4 mailing list
nfsv4 <at>
David Noveck | 27 Sep 15:11 2014

My review of draft-ietf-nfsv4-minorversion2-27

Overall, this document is good shape.


Saying that the document will "focus mainly" on the 4.2 extensions, seems to imply there is something else in the document.  As there isn't, I suggest replacing the first sentence by:

This Internet-Draft describes NFS version 4 minor version two, presenting the protocol extensions and other changes made from NFS version 4 minor one .

Section 1.[12]:

Not sure if this goes in 1.1 or 1.2, but in one if those places we need to state the basic rule that all operations, attributes , etc. not mentioned in this document are to be as described in NFSv4.1.

Section 1.3:

Not clear that all of the 4.2 features can be subsumed under this goal. Suggest replacing "The goal" by "A major goal".

Also this section consists two sentences. The second of these has three interpolated bullet points followed by a clause beginning "but".

Suggest replacing the material after the bullets by the following:

NFSv4.2 provides means for clients to leverage these features on the server in cases in which that had previously not been possible within the confines of the NFS protocol.

Section 1..1:

The intraserver case of server side copy is not mentioned but should be. Suggest replacing "from one server to another" by "from a location on one server to another location on the same server or a different one". Also replace "the two servers'" by "source and destination servers".

Section 1.4.2:

Suggest replacing the last sentence by This operation can be used to support the posix_fadvise function as well as other applications such as databases and video editors.

nfsv4 mailing list
nfsv4 <at>
Trond Myklebust | 27 Sep 02:08 2014

Proposal for errata to RFC5661

Section 8.4.3 says:
   o  a boolean that indicates whether the client may have locks that it
      believes to be reclaimable in situations in which the grace period
      was terminated, making the server's view of lock reclaimability
      suspect.  The server will set this for any client record in stable
      storage where the client has not done a suitable RECLAIM_COMPLETE
      (global or file system-specific depending on the target of the
      lock request) before it grants any new (i.e., not reclaimed) lock
      to any client.

It should say:

   o  a timestamp that is updated the first time after a server boot or reboot
      when the client acquires byte-range locking, share reservation, or
      delegation state on the server, or upon receiving a suitable
      RECLAIM_COMPLETE (global or file system-specific depending on the
      target of the lock request).

Section 8.4.3 says:

   For the second edge condition, after the server restarts for a second
   time, the indication that the client had not completed its reclaims
   at the time at which the grace period ended means that the server
   must reject a reclaim from client A with the error NFS4ERR_NO_GRACE.

   When either edge condition occurs, the client's attempt to reclaim
   locks will result in the error NFS4ERR_NO_GRACE.  When this is
   received, or after the client restarts with no lock state, the client
   will send a global RECLAIM_COMPLETE.  When the RECLAIM_COMPLETE is
   received, the server and client are again in agreement regarding
   reclaimable locks and both booleans in persistent storage can be
   reset, to be set again only when there is a subsequent event that
   causes lock reclaim operations to be questionable.

It should say:

   For the second edge condition, after the server restarts for a second
   time, the lack of a timestamp indication that the client did indeed establish
   reclaimable state in the previous boot instance means that the server must
   reject a reclaim from client A with the error NFS4ERR_NO_GRACE.

   When either edge condition occurs, the client's attempt to reclaim
   locks will result in the error NFS4ERR_NO_GRACE.  When this is
   received, or after the client restarts with no lock state, the client
   will send a global RECLAIM_COMPLETE.  When the RECLAIM_COMPLETE is
   received, the server and client are again in agreement regarding
   reclaimable locks. The server can then set the timestamp, and reset
   the boolean in persistent storage.

As it reads now, the spec appears to state that in the second edge
condition case, the locks that were reclaimed by client A after the
initial server reboot should not be reclaimed after the second reboot.
There is no reason why they shouldn't, since those locks are not
contested by any other clients.

The real issue in the second edge condition is that both client A and
the server need to agree whether or not lock state was held by client
A during the period between the first and second reboot. Once that
agreement has been established, then client A has full knowledge of
which locks were held, and it can therefore attempt to recover only
those locks.

nfsv4 mailing list
nfsv4 <at>