headers
Jeff Stearns | 23 Jan 03:27
Picon

Re: Database Locking

Olly Betts <olly <at> survex.com> writes:

> Locking of Quartz databases is currently done with a lockfile.  This
> works well from the actual locking side, but the downside is that
> the lock isn't released if a process doesn't destroy the
> Xapian::WritableDatabase object.  This is made worse because some
> of the bindings don't call destructors (or don't do it reliably).
> 
> The obvious alternative is to use actual locking APIs.  On Unix, this
> mean fcntl.  Windows has a suitable API too.
> 
> But the problem is that on Unix, fcntl locks are per process.  So if
> we used fcntl locking, then if a process tried to open the same database
> twice as a WritableDatabase, it would succeed.  This is probably more of
> a worry in a threaded application, but threads aren't actually needed to
> hit this problem.

Olly -

I know nothing about the internals of Xapian, but I wonder whether this
cunning plan is more complex  and expensive than necessary.

I wonder why Xapian doesn't apply the flock, then set a flag indicating that
the database is locked.  Now whenever Xapian goes to open a database,
it would first check whether the flag is set.  If so, Xapian knows that that
the database is already open within this process.  If the flag is not set,
Xapian continues onward, probing the file with flock to see whether it's
open within some other process.

This should work in a threaded environment so long as the customary
(Continue reading)

Permalink | Reply | 2 comments |
headers
Kevin Duraj | 23 Jan 19:17
Picon

Re: Database Locking

We should try to avoid to do anykind of locking because it affects  
performance for everybody. Search engine is not database where user is  
allow to lock tables and make updates.

You can easily implement your own locking localy on your machine. To  
the rest of the users who are using Xapian index read only mode would  
be great overhead, and slow down in performance.

- Kevin Duraj

On Jan 22, 2010, at 6:27 PM, Jeff Stearns <jeff.stearns 
+gmane <at> gmail.com> wrote:

> Olly Betts <olly <at> survex.com> writes:
>
>> Locking of Quartz databases is currently done with a lockfile.  This
>> works well from the actual locking side, but the downside is that
>> the lock isn't released if a process doesn't destroy the
>> Xapian::WritableDatabase object.  This is made worse because some
>> of the bindings don't call destructors (or don't do it reliably).
>>
>> The obvious alternative is to use actual locking APIs.  On Unix, this
>> mean fcntl.  Windows has a suitable API too.
>>
>> But the problem is that on Unix, fcntl locks are per process.  So if
>> we used fcntl locking, then if a process tried to open the same  
>> database
>> twice as a WritableDatabase, it would succeed.  This is probably  
>> more of
>> a worry in a threaded application, but threads aren't actually
(Continue reading)

Permalink | Reply | 0 comments |
headers
Olly Betts | 24 Jan 07:48
Favicon
Gravatar

Re: Database Locking

On Sat, Jan 23, 2010 at 02:27:42AM +0000, Jeff Stearns wrote:
> I know nothing about the internals of Xapian, but I wonder whether this
> cunning plan is more complex  and expensive than necessary.
> 
> I wonder why Xapian doesn't apply the flock, then set a flag indicating that
> the database is locked.

How do you track such a flag?

You can't use the database path, because symlinks (and also bind mounts on
Linux) mean that the same database could be opened via entirely different
paths.

You could try to use the device and inode number of the lock file.  Not all
filing systems have the concept of inodes, so you couldn't support those FSes
though maybe all those that support fcntl() locking have inodes.

There also seems to be a race condition here - you have to use stat() to get
the device and inode (you can't open the lock file and call fstat() because
then if it is locked, you will unlock it when you close it), but then between
you checking the file with stat, another process could remove and recreate the
database, changing the inode of the lock file.  I guess you could have a second
file which is the "inode of the database" and open that then use fstat() on the
fd.

I have also wondered about just locking based on the uuid of the database
(now that databases have uuids).  This means if you copy a database, you
couldn't update the old and new version at once (without forcing a new uuid) 
which is probably enough to sink the idea, but at least it fails in a safe
way (can't update when it would be safe, rather than allowing update when it
(Continue reading)

Permalink | Reply | 0 comments |

Gmane