Ged Murphy | 8 Oct 2009 15:47
Picon

Building the ntfs-3g core into an NT driver

Hi all,

 

I’m from the reactos project and I’m doing a bit of forward planning for a future NTFS driver for our OS.

I see your project is available for most unix based OS’s, but do you know if anyone has ever looked into building an NT compatible version?

 

I’m unsure how your code works at the moment as I’m not too familiar with programming for the unix kernel, but if it’s possible for me to consolidate your code into a library, I’d be interested in writing a wrapper to get it working in Windows.

 

Any comments are much appreciated.

 

Regards,

Ged Murphy

 

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
ntfs-3g-devel mailing list
ntfs-3g-devel@...
https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
Erik Larsson | 8 Oct 2009 20:51
Picon
Picon
Favicon

Re: Building the ntfs-3g core into an NT driver

Ged Murphy wrote 2009-10-08 15.47:
I’m from the reactos project and I’m doing a bit of forward planning for a future NTFS driver for our OS.

I see your project is available for most unix based OS’s, but do you know if anyone has ever looked into building an NT compatible version?


I have heard rumours that NTFS-3G has been shown to work on Windows using Dokan and the fuse4win compatibility layer. If Dokan can be used with ReactOS, then you might already have a solution (or part of it).

I’m unsure how your code works at the moment as I’m not too familiar with programming for the unix kernel, but if it’s possible for me to consolidate your code into a library, I’d be interested in writing a wrapper to get it working in Windows.


It's really a library already, libntfs-3g, and the fuse driver simply uses it to access the NTFS volume. Take a look in the sources... they won't bite. :)

Regards,

- Erik


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
ntfs-3g-devel mailing list
ntfs-3g-devel@...
https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
Ged Murphy | 8 Oct 2009 21:42
Picon

Re: Building the ntfs-3g core into an NT driver

Erik Larsson wrote:

> I have heard rumours that NTFS-3G has been shown to work on Windows using 
> Dokan and the fuse4win compatibility layer. If Dokan can be used with 
> ReactOS, then you might already have a solution (or part of it).

Yeah, I've heard about people using Dokan too.
However, as this will be our primary FSD it needs to be the kernel and be
optimized.

> It's really a library already, libntfs-3g, and the fuse driver simply uses

> it to access the NTFS volume. Take a look in the sources... they won't
bite. :)

Ah great, I'll start to look over the code and get it built within the
coming weeks.
I'm sure I'll be back with questions at some point in the future :)

Thanks,
Ged.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Szabolcs Szakacsits | 12 Oct 2009 16:58
Favicon

NTFS-3G 2009.10.5-RC Release Candidate


Welcome,

A new stable NTFS-3G release candidate is available for testing. It is 
based on the latest Advanced NTFS-3G release which is already used by 
several distributions as the default NTFS driver. The changes include:

    * New: Full file ownership and permissions support. The ownership 
	and permissions of files are interoperable with Windows and 
	conforms to the POSIX rules.
    * New: POSIX ACL support. The ACLs are an extension to ownership 
	and permissions as defined in POSIX.
    * New: Optional Windows-type inheritance of ACLs.
    * New: Full extended attributes support for all namespaces: user, 
	system, security and trusted.
    * New: Junction points support. The junction points and symbolic 
	links created by Windows are seen as symbolic links.
    * New: Access is possible to NTFS internal data like Windows style 
	archive bit and file attributes. They are mapped to extended 
	attributes for easier access.
    * New: Compression write support: sequential writing to compressed 
	files is supported.
    * New: Support copying and restoring encrypted files without 
	decrypting them.
    * New: Support getting and setting high-resolution file timestamps.
    * New: Support to get, set and remove DOS filenames.
    * New: Tools for managing ownership, permissions and ACLs: secaudit 
	checks the integrity of ownership, permissions and ACL, usermap 
	gives help to configure ownership.
    * Change: Performance optimizations. Frequently used data are cached 
	to improve speed. Space allocation for fragmented and sparse 
	files has been improved.

We are also very happy to announce the new NTFS-3G lead developer, 
Jean-Pierre Andre.

The source code of the latest stable driver is available at 

	http://www.tuxera.com/community/ntfs-3g-download/

Enjoy!

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
James Harper | 28 Oct 2009 10:18
Picon

Invalid ACLS on windows file

(I posted this to the forum before I'd done much investigation)

I have a windows file that has invalid ACL's (as far as NTFS-3G
concerned). I came across this because I am using the data generated by
the Windows BackupRead API call in Bacula to restore the ACL's (and
junctions etc) to an ntfs-3g filesystem mounted under Linux.

Under Windows:
C:\temp>secaudit -b index.html
secaudit 1.3.8 : NTFS security data auditing
*** Could not get user mapping data
#
# Recursive ACL collection on Wed Oct 28 19:13:54 2009
#
File index.html
** Descriptor for index.html is not valid
** 1 warning was signalled
** 1 error was found

The hex ACL string from that file I am trying to apply on ntfs-3g:
"0x010004841400000024000000000000004000000001020000000000052000000020020
000010500000000000515000000AA8B635446D7000586C8A697010200000400340002000
00000101800FF011F000102000000000005200000002002000000101400FF011F0001010
0000000000512000000"

By adding lots of debug statements to security.c I tracked it down to
"pdacl->revision == ACL_REVISION" in the big if clause in
ntfs_valid_descr. pdacl->revision is 4 while ACL_REVISION is 2.

If I 'xcopy /O' the file, the 'bad' ACL copies with it with no
comlaints. Windows shows the permissions as being purely inherited from
the parent.

A chkdsk /f followed by a reboot does not resolve anything.

The system is a Windows 2003 machine that has been around for a number
of years.

ntfs-3g is ntfs-3g-2009.4.4AR.21.tgz

Ultimately I'd like for ntfs-3g to apply any ACL I give it without any
validation at all...

James

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Jean-Pierre André | 28 Oct 2009 15:00
Picon

Re: Invalid ACLS on windows file

Hi James

James Harper wrote:
> (I posted this to the forum before I'd done much investigation)
>   

... with an example which led to no problem.

> I have a windows file that has invalid ACL's (as far as NTFS-3G
> concerned). I came across this because I am using the data generated by
> the Windows BackupRead API call in Bacula to restore the ACL's (and
> junctions etc) to an ntfs-3g filesystem mounted under Linux.
>
> Under Windows:
> C:\temp>secaudit -b index.html
> secaudit 1.3.8 : NTFS security data auditing
> *** Could not get user mapping data
> #
> # Recursive ACL collection on Wed Oct 28 19:13:54 2009
> #
> File index.html
> ** Descriptor for index.html is not valid
> ** 1 warning was signalled
> ** 1 error was found
>
> The hex ACL string from that file I am trying to apply on ntfs-3g:
> "0x010004841400000024000000000000004000000001020000000000052000000020020
> 000010500000000000515000000AA8B635446D7000586C8A697010200000400340002000
> 00000101800FF011F000102000000000005200000002002000000101400FF011F0001010
> 0000000000512000000"
>
> By adding lots of debug statements to security.c I tracked it down to
> "pdacl->revision == ACL_REVISION" in the big if clause in
> ntfs_valid_descr. pdacl->revision is 4 while ACL_REVISION is 2.
>   

True.

The value 4 is for ACL_REVISION_DS, which is another
value which should be accepted. I found a mention about it in
http://msdn.microsoft.com/en-us/library/aa374970(VS.85).aspx
with the comment "This value can be ACL_REVISION or
ACL_REVISION_DS. Use ACL_REVISION_DS if the ACL
contains object-specific ACEs."

This will be fixed in the next release.

> If I 'xcopy /O' the file, the 'bad' ACL copies with it with no
> comlaints. Windows shows the permissions as being purely inherited from
> the parent.
>
> A chkdsk /f followed by a reboot does not resolve anything.
>
> The system is a Windows 2003 machine that has been around for a number
> of years.
>
> ntfs-3g is ntfs-3g-2009.4.4AR.21.tgz
>
> Ultimately I'd like for ntfs-3g to apply any ACL I give it without any
> validation at all...
>   

And you obviously know how to do that...

PS I will put a comment about it on the forum, as
your example was misleading.

Regards

Jean-Pierre

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
James Harper | 29 Oct 2009 01:10
Picon

Re: Invalid ACLS on windows file

> Hi James
> 
> James Harper wrote:
> > (I posted this to the forum before I'd done much investigation)
> >
> 
> ... with an example which led to no problem.

Yes. Oops. I posted the wrong one. Sorry.

> 
> This will be fixed in the next release.
> 

Excellent!

> >
> > Ultimately I'd like for ntfs-3g to apply any ACL I give it without any
> > validation at all...
> >
> 
> And you obviously know how to do that...
> 

Yes, I think so.

Would you accept a patch which allowed the setting of raw data without any validation? The data stream
generated by the BackupRead API under Windows contains DATA, ALTERNATE_DATA, EA_DATA, LINK,
OBJECT_ID, PROPERTY_DATA, REPARSE_DATA, SECURITY_DATA, and SPARSE_BLOCK streams. You handle most of
them already, but not OBJECT_ID which is one that I need. I can't see that it would add much value to ntfs-3g
for anything other than backup and restore.

Do the xattr namespaces have to be pre-registered with Linux, or can I create something like
'system.ntfs_raw_X' where X is the hex value that corresponds to the data stream?

Thanks

James

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Jean-Pierre André | 29 Oct 2009 11:48
Picon

Re: Invalid ACLS on windows file

Hi James,

James Harper wrote:
> Would you accept a patch which allowed the setting of raw data without any validation? The data stream
generated by the BackupRead API under Windows contains DATA, ALTERNATE_DATA, EA_DATA, LINK,
OBJECT_ID, PROPERTY_DATA, REPARSE_DATA, SECURITY_DATA, and SPARSE_BLOCK streams. You handle most of
them already, but not OBJECT_ID which is one that I need. I can't see that it would add much value to ntfs-3g
for anything other than backup and restore.
>   

Doing that blindly is calling for problems.

Examples :

I guess "LINK" is defining alternate names for the file.
Assuming the name space information is provided, setting
a name is not just setting an attribute, the main purpose
is to create an entry in the index of some directory. At
least the information about the directory is needed.
You must have solved that one, where did you get the
directory from ?
Do you have an example of backup data associated to
a LINK attribute ?

The same problem goes with "OBJECT_ID". Its purpose
is to define a unique id is some domain, so that the
file can be located in the domain. For the OBJECT_ID
to be meaningful, it has to be indexed in a special file,
which means special processing for this attribute.

Also with "SECURITY_DATA", as each ACL is indexed
twice in a special file. This situation a currently dealt
with, which gives indications to what has to be done
for OBJECT_ID.

As far as I understand, "SPARSE_BLOCK" is not an
attribute of the file, but rather an information needed
to decode DATA or ALTERNATE_DATA when the
stream is sparse (to avoid backing up holes).

I do not know what PROPERTY_DATA is. It must be
the "$PROPERTY_SET" mentioned in
http://msdn.microsoft.com/en-us/library/bb470038(VS.85).aspx
with the (user) comment "The associated data stream
is undocumented. Its usage is not clear."
Do you have an example ?

I would add that "EA_DATA" is considered obsolete,
as the only known reference to it in Win32 are the
backup and restore function. The EA_DATA were the
extended attributes on OS2, and the ADS is now
used for this purpose.

And in your list there is nothing about encrypted files
which use the attribute "$LOGGED_UTILITY_STREAM"
and have significant data beyond the advertised
stream size.

Finally, unless I get more information, only OBJECT_ID
appears as eligible for extra support, with a specific
processing.

> Do the xattr namespaces have to be pre-registered with Linux, or can I create something like
'system.ntfs_raw_X' where X is the hex value that corresponds to the data stream?
>   

I do not know of any registration process for the
extended attributes in system name space. I consider
that prefixing by "ntfs_" is enough to avoid conflicts.
The attributes in system name space are not returned
by listxattr(), consequently they cannot be wrongly
processed by applications which do not know about
them.

For the OBJECT_ID, I acknowledge they can be
needed in a networked environment and may have to
be backed up. I can add support for them if you provide
me with examples... (should take a few weeks).
I can also merge the patches you would send me !

Regards

Jean-Pierre

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
James Harper | 31 Oct 2009 03:38
Picon

Re: Invalid ACLS on windows file

> 
> James Harper wrote:
> > Would you accept a patch which allowed the setting of raw data without any
> validation? The data stream generated by the BackupRead API under Windows
> contains DATA, ALTERNATE_DATA, EA_DATA, LINK, OBJECT_ID, PROPERTY_DATA,
> REPARSE_DATA, SECURITY_DATA, and SPARSE_BLOCK streams. You handle most of them
> already, but not OBJECT_ID which is one that I need. I can't see that it would
> add much value to ntfs-3g for anything other than backup and restore.
> >
> 
> Doing that blindly is calling for problems.
> 
> Examples :
> 
> I guess "LINK" is defining alternate names for the file.
> Assuming the name space information is provided, setting
> a name is not just setting an attribute, the main purpose
> is to create an entry in the index of some directory. At
> least the information about the directory is needed.
> You must have solved that one, where did you get the
> directory from ?
> Do you have an example of backup data associated to
> a LINK attribute ?

No, I don't think I do. For the use I need this for I think that hardlinks are provided another way.

> 
> The same problem goes with "OBJECT_ID". Its purpose
> is to define a unique id is some domain, so that the
> file can be located in the domain. For the OBJECT_ID
> to be meaningful, it has to be indexed in a special file,
> which means special processing for this attribute.
> 
> Also with "SECURITY_DATA", as each ACL is indexed
> twice in a special file. This situation a currently dealt
> with, which gives indications to what has to be done
> for OBJECT_ID.

Hmmm... so, more complicated than I thought.

> 
> As far as I understand, "SPARSE_BLOCK" is not an
> attribute of the file, but rather an information needed
> to decode DATA or ALTERNATE_DATA when the
> stream is sparse (to avoid backing up holes).
> 

OK. I haven't seen one of these yet either.

> I do not know what PROPERTY_DATA is. It must be
> the "$PROPERTY_SET" mentioned in
> http://msdn.microsoft.com/en-us/library/bb470038(VS.85).aspx
> with the (user) comment "The associated data stream
> is undocumented. Its usage is not clear."
> Do you have an example ?

I might have seen one go flying by the restore logs. I'll know more once I get OBJECT_ID's working and the logs
are reduced to a more sensible size :)

> 
> I would add that "EA_DATA" is considered obsolete,
> as the only known reference to it in Win32 are the
> backup and restore function. The EA_DATA were the
> extended attributes on OS2, and the ADS is now
> used for this purpose.

OK

> And in your list there is nothing about encrypted files
> which use the attribute "$LOGGED_UTILITY_STREAM"
> and have significant data beyond the advertised
> stream size.

I haven't yet looked into how BackupRead deals with encrypted streams. 

> 
> Finally, unless I get more information, only OBJECT_ID
> appears as eligible for extra support, with a specific
> processing.
> 
> > Do the xattr namespaces have to be pre-registered with Linux, or can I
> create something like 'system.ntfs_raw_X' where X is the hex value that
> corresponds to the data stream?
> >
> 
> I do not know of any registration process for the
> extended attributes in system name space. I consider
> that prefixing by "ntfs_" is enough to avoid conflicts.
> The attributes in system name space are not returned
> by listxattr(), consequently they cannot be wrongly
> processed by applications which do not know about
> them.
> 
> For the OBJECT_ID, I acknowledge they can be
> needed in a networked environment and may have to
> be backed up. I can add support for them if you provide
> me with examples... (should take a few weeks).
> I can also merge the patches you would send me !
> 

I haven't written any code yet, but in looking through your code I think I mostly just need to replicate a
similar code path as for existing attributes, eg:

. Create a ntfs_object_id entry
. Create a routine to read/write - call ntfs_attr_readall etc with AT_OBJECT_ID
. Create some validation for write's - your assistance is probably required here, although if I can find
some documentation on the structure I can probably put something together myself

Thanks for considering this.

James

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
James Harper | 31 Oct 2009 04:04
Picon

Re: Invalid ACLS on windows file

> 
> I haven't written any code yet, but in looking through your code I think I
> mostly just need to replicate a similar code path as for existing attributes,
> eg:
> 
> . Create a ntfs_object_id entry
> . Create a routine to read/write - call ntfs_attr_readall etc with
> AT_OBJECT_ID
> . Create some validation for write's - your assistance is probably required
> here, although if I can find some documentation on the structure I can
> probably put something together myself
> 

Having re-read what you wrote, and then having another look at the code, I better understand what you said
about the 'special file'. I'll have to do some more thinking :)

Thanks again

James

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

Gmane