headers
didier | 1 Mar 2003 10:57
Picon
Favicon

Re: CNID database - better stability/recoverability

Joerg Lenneis wrote:
> Dear list members,
> 
> I have looked at the source code of netatalk for the last couple of
> weeks to find out more about its current features. I am especially
> interested in the various approaches with respect to CNIDs as well as
> interoperability with Samba, also with respect to CNIDs and locking
> issues. So far, these are my findings and I would be thankful for any
> corrections and additional input:
> 
> - The only "safe" CNID scheme available at the moment is CNID_DB using
>   Berkeley DB. All other schemes can potentially lead to data
>   corruption, mainly because there is no guarantee that a CNID is not
>   reused at a later point in time. 
Rather duplicate than reused,  we can get reused ID with CNID too.
> 
> - Storing CNIDs in a Berkeley DB is in itself not foolproof. A crash
>   at the wrong time can lead to database corruption and possibly to
>   the loss of all CNIDs on a volume (since the last backup,
>   obviously) unless precautions are taken (see below).
It's a pain but IMO not that much, but not be able to access files is 
worse. It seems that opening the db doesn't always return an error if 
there's locks held by deaded processes.
> 
> 

> Extending the AppleDouple1 file to AppleDouple2 is more complicated but
> fortunately much of the work has been done already. There is a
> function ad_v1tov2 in ad_open.c which does all of the work of
> converting existing AppleDouble1 files to AppleDouble2. It does not
(Continue reading)

Permalink | Reply | 4 comments |
headers
Joerg Lenneis | 3 Mar 2003 02:00
Picon

Re: CNID database - better stability/recoverability

didier :

> Joerg Lenneis wrote:
>> Dear list members,
>> 

[...]

> It's a pain but IMO not that much, but not be able to access files is 
> worse. It seems that opening the db doesn't always return an error if 
> there's locks held by deaded processes.
>> 
>> 

[...]

> What netatalk version are you working on?

1.7cvs, from the CVS tree.

>> 
>> The other part is to actually record CNIDs in the new AppleDouble2
>> files. That works for newly created directories already and I have
>> begun to implement the same for new files. It works pretty well so far
>> and I think the remaining changes are not that many. One thing that is
>> a bit more difficult to deal with is what to do about CNIDs that are
>> recorded in the database but not in the AppleDouble file for
>> the transition period.  
> Update the AppleDouble header with the cnid.
> If we can put device and inode in the header then it can be
(Continue reading)

Permalink | Reply | 3 comments |
headers
MenaceChgo | 3 Mar 2003 04:31
Picon
Favicon

Re: CNID database - better stability/recoverability

In a message dated 03/02/03 7:08:34 PM Central Standard Time, 
lenneis <at> wu-wien.ac.at writes:

<< OK, that should be doable. Unfortunately, there is no space right now
 for device/inode in the ADEID_DID field, only 4 bytes. It is not very
 difficult to extend it, but that would break the Apple specification
 for that field.  >>

Does the netatalk code already use this field???

If not, you don't have to use that field... 
put the CNID hints into a Netatalk specific ID field.
from the AppleSingle/AppleDouble V2 Specs page 6 (in my pdf copy)
"Apple reserves the range of entry IDs from 1 to $7FFFFFFF. The rest of the 
range
is available for applications to define their own entries."
so for instance $80434944 could stand for   custom ID 'CND'  and define (at 
the offset location of course) what ever number of bytes that would be needed.

It doesn't break specification and since that ID field would either only be 
seen by the server, or ignored as an unrecognized ID type.

"Applications reading an AppleSingle or AppleDouble file interpret the entry 
IDs
that are relevant to their home file system and treat the other entries as 
opaque
data."   AppleSingle/AppleDouble V2 Specs page 7 (in my pdf copy)
Permalink | Reply | 2 comments |
headers
Nir Soffer | 3 Mar 2003 10:21
Favicon

Netatalk Developer Requested for Contracting Work

Hi!

We at Exanet are seeking an experienced developer with intimate knowledge of the Netatalk source code and
the Apple Filing Protocol.

The person we are seeking must:

* Have used Netatalk in various settings.
* Have developed in netatalk for over a year.
* Can work on tight schedules and stressing timelines.
* Is available for contracting work immediately.

If you are interested, please mail a resume to nirs <at> exanet.com and we can get talking.

Thanks,
Nir.

--
Nir Soffer -=- Software Engineer, Exanet Inc. -=-
Permalink | Reply | 0 comments |
headers
didier | 3 Mar 2003 10:59
Picon
Favicon

Re: CNID database - better stability/recoverability

Joerg Lenneis wrote:
> 
> 1.7cvs, from the CVS tree.
> 
Do you have a patch?
> 
> You mean the user could add a cnid, but then would not be able to
> create a file? If the sequence was the other way round, it would not
> be a problem.
I don't use quota myself but from messages on the list, quota is not 
always correctly reported and when it's reached it's a DB corruption.

>>- when a user can't open dbs he gets an empty volume.
> 
> OK, one has to make sure that the DB is in a "openable" state then,
> otherwise it has to be rebuilt.
In most case it's not a corrupted DB but a missing write perm. For now 
if you don't have rw access you can't mount a volume.

>- performance under load, sleep too much.
> 
> I have been thinking about that. One possible solution would be to use
> a global lock on the database for write updates (fcntl on a lock file
> that has nothing to do with Berkeley DB).
I'm not sure it'd be enough, you need read lock too and then you have a 
live lock risk. If such a solution is needed then Berkeley DB is not the 
right lib for afpd.
>  another to put the database into its own daemon.
It would solve the permission, quota, corrupted db and to some extend 
speed issue, I have no idea how single thread will slow things down,
(Continue reading)

Permalink | Reply | 2 comments |
headers
didier | 3 Mar 2003 11:00
Picon
Favicon

Re: CNID database - better stability/recoverability

MenaceChgo <at> aol.com wrote:
> from the AppleSingle/AppleDouble V2 Specs page 6 (in my pdf copy)
> "Apple reserves the range of entry IDs from 1 to $7FFFFFFF. The rest of the 
> range
> is available for applications to define their own entries."
> so for instance $80434944 could stand for   custom ID 'CND'  and define (at 
> the offset location of course) what ever number of bytes that would be needed.
> 
> It doesn't break specification and since that ID field would either only be 
> seen by the server, or ignored as an unrecognized ID type.
Yes, I was thinking of something like that, we need dev/inode for 'cp -a 
folder1 folder2' like case. When afpd needs a CNID, in getparams, the 
AppleDouble header is opened and afpd has the data fork dev/inode too.

Didier
Permalink | Reply | 0 comments |
headers
Joerg Lenneis | 3 Mar 2003 20:31
Picon

Re: CNID database - better stability/recoverability

didier :

[...]

>> - performance under load, sleep too much.
>> 
>> I have been thinking about that. One possible solution would be to use
>> a global lock on the database for write updates (fcntl on a lock file
>> that has nothing to do with Berkeley DB).
> I'm not sure it'd be enough, you need read lock too and then you have a 
> live lock risk. If such a solution is needed then Berkeley DB is not the 
> right lib for afpd.

Maybe that is correct. It is just strange that something so obvious
should happen in an library like DB, which is a rather well tested
tool otherwise.

>> another to put the database into its own daemon.
> It would solve the permission, quota, corrupted db and to some extend 
> speed issue, I have no idea how single thread will slow things down, 
> not much if AppleDouble V2 is used too.

I have done some testing on that aproach (single daemon) and it looks
like there would be no slowdown as opposed to CDB reads/writes. I
think, though, that some further testing wold be a good idea, as well
as finding out for sure that CDB is somehow broken. I will do that in
the next couple of days and report back on the result.

>>> - assume 32 bits for device/inode.
>>
(Continue reading)

Permalink | Reply | 1 comment |
headers
didier | 3 Mar 2003 20:58
Picon
Favicon

Re: CNID database - better stability/recoverability

Joerg Lenneis wrote:
> didier :
> 
> [...]
> 
> 
>>>- performance under load, sleep too much.
> Maybe that is correct. It is just strange that something so obvious
> should happen in an library like DB, which is a rather well tested
> tool otherwise.
Or we are doing something stupid.
> 
> 
> I have done some testing on that aproach (single daemon) and it looks
> like there would be no slowdown as opposed to CDB reads/writes. I
> think, though, that some further testing wold be a good idea, as well
> as finding out for sure that CDB is somehow broken. I will do that in
> the next couple of days and report back on the result.
There's a patch for a bug with CDB and concurrent access, it's not in 
CVS yet, check the list.

Didier
Permalink | Reply | 0 comments |
headers
Joerg Lenneis | 3 Mar 2003 21:20
Picon

Re: CNID database - better stability/recoverability

MenaceChgo :

> In a message dated 03/02/03 7:08:34 PM Central Standard Time, 
> lenneis <at> wu-wien.ac.at writes:

[...]

> Does the netatalk code already use this field???

> If not, you don't have to use that field... 
> put the CNID hints into a Netatalk specific ID field.
> from the AppleSingle/AppleDouble V2 Specs page 6 (in my pdf copy)
> "Apple reserves the range of entry IDs from 1 to $7FFFFFFF. The rest of the 
> range
> is available for applications to define their own entries."
> so for instance $80434944 could stand for   custom ID 'CND'  and define (at 
> the offset location of course) what ever number of bytes that would be needed.

> It doesn't break specification and since that ID field would either only be 
> seen by the server, or ignored as an unrecognized ID type.

> "Applications reading an AppleSingle or AppleDouble file interpret the entry 
> IDs
> that are relevant to their home file system and treat the other entries as 
> opaque
> data."   AppleSingle/AppleDouble V2 Specs page 7 (in my pdf copy)

That would have been my preference as well. Unfortunately, the current
layout of AppleDouble information on disk and in memory makes that
much harder to implement. These are the definitions of various
(Continue reading)

Permalink | Reply | 0 comments |
headers
Joerg Lenneis | 4 Mar 2003 01:12
Picon

Re: CNID database - better stability/recoverability

didier :

> Bjorn Fernhomberg wrote:
>> 
>> Several weeks ago, Didier Gautheron stated that AD2 was currently
>> incompatible with CNID. I'm not sure what he ment in detail.
>> Didier, maybe you can shed some more light on this?
> With AD2 if there's an ID in the appledouble header then it's used as 
> hint for cnid_add but it doesn't work , we had some logic for dealing 
> with it but it didn't work well so it was removed and I'm not sure it 
> was the right thing to do.

> Didier

You are probably refering to this piece of code from getmetadate:

        case FILPBIT_FNUM :
            aint = 0;
#if AD_VERSION > AD_VERSION1
            /* look in AD v2 header */
            if (adp)
                memcpy(&aint, ad_entry(adp, ADEID_DID), sizeof(aint));
#endif /* AD_VERSION > AD_VERSION1 */

#ifdef CNID_DB
            aint = cnid_add(vol->v_db, st, dir->d_did, upath,
                            strlen(upath), aint);
            /* Throw errors if cnid_add fails. */
            if (aint == CNID_INVALID) {
                switch (errno) {
(Continue reading)

Permalink | Reply | 5 comments |

Gmane