headers
shad | 1 Jun 15:45
Picon

Xapian over NFS (again...)

Hi everyone,

I've read some of documentation about xapian over NFS and I have
probably found a bug.

I use several readers on a xapian index mounted over NFS. I use a
try/catch mechanism to reopen database only on modifications
(DatabaseModifiedError exception). My software works great on a local
filesystem. But I noticed that there is no DatabaseModifiedError
exception raised on NFS. So all readers never reopen the database on
modifications. I tried some different configurations. For example set
NFS mount with sync, noactime. But nothing changed.

I ask myself if it's a good thing to reopen the database for each search query.

Thanks for any comment / help.

--

-- 
Samuel Alba
Permalink | Reply | 7 comments |
headers
Miki Tebeka | 1 Jun 20:28
Favicon

Re: Xapian over NFS (again...)

Hello shad,

> I've read some of documentation about xapian over NFS and I have
> probably found a bug.
> 
> I use several readers on a xapian index mounted over NFS. I use a
> try/catch mechanism to reopen database only on modifications
> (DatabaseModifiedError exception). My software works great on a local
> filesystem. But I noticed that there is no DatabaseModifiedError
> exception raised on NFS. So all readers never reopen the database on
> modifications. I tried some different configurations. For example set
> NFS mount with sync, noactime. But nothing changed.
> 
> I ask myself if it's a good thing to reopen the database for each search query.
I had the same problem, one of the things I found that on linux you should
have rpc.statd running on the client machine.

Another thing was to batch the writes (I have about 120000 a day), this
also helped a lot.

Last I switched to running xapian-tcpsrv on the local storage.

All the best,
--
Miki Tebeka
miki <at> fattoc.com
Permalink | Reply | 2 comments |
headers
Jim Spath | 1 Jun 22:16
Favicon

Xapian::DatabaseModifiedError

Hi,

We recently started running into the Xapian::DatabaseModifiedError 
problem under Perl (ticket #284), due to a change in the way we update 
our database.

I noticed that newer versions of Xapian are supposed to deal with this 
problem (ticket #284), so we upgraded to 1.0.12.

I now am able to properly trap the error, most of the time, by using 
eval and checking $@ to see if it matches "revision being read has been 
discarded".

The problem is that every once in a while I still see a segfault:

terminate called after throwing an instance of 
'Xapian::DatabaseModifiedError'
Aborted

I have been unable to determine why the segfaults still sometimes occur 
... any suggestions?

Thanks!
Jim
Permalink | Reply | 3 comments |
headers
shad | 1 Jun 22:39
Picon

Re: Xapian over NFS (again...)

Hi Miki,

thanks for your answer. I will follow your advice about rpc.statd.
What do you think about NFS mount options ? Did you use NFS mounted as
sync ?

About the writes batching, what happened without it ?

Thanks a lot for your help.

On Mon, Jun 1, 2009 at 8:28 PM, Miki Tebeka <miki <at> fattoc.com> wrote:
> Hello shad,
>
>> I've read some of documentation about xapian over NFS and I have
>> probably found a bug.
>>
>> I use several readers on a xapian index mounted over NFS. I use a
>> try/catch mechanism to reopen database only on modifications
>> (DatabaseModifiedError exception). My software works great on a local
>> filesystem. But I noticed that there is no DatabaseModifiedError
>> exception raised on NFS. So all readers never reopen the database on
>> modifications. I tried some different configurations. For example set
>> NFS mount with sync, noactime. But nothing changed.
>>
>> I ask myself if it's a good thing to reopen the database for each search query.
> I had the same problem, one of the things I found that on linux you should
> have rpc.statd running on the client machine.
>
> Another thing was to batch the writes (I have about 120000 a day), this
> also helped a lot.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Jim Spath | 1 Jun 23:58
Favicon

Re: Xapian::DatabaseModifiedError

Jim Spath wrote:
> Hi,
> 
> We recently started running into the Xapian::DatabaseModifiedError 
> problem under Perl (ticket #284), due to a change in the way we update 
> our database.
> 
> I noticed that newer versions of Xapian are supposed to deal with this 
> problem (ticket #284), so we upgraded to 1.0.12.
> 
> I now am able to properly trap the error, most of the time, by using 
> eval and checking $@ to see if it matches "revision being read has been 
> discarded".
> 
> The problem is that every once in a while I still see a segfault:
> 
> terminate called after throwing an instance of 
> 'Xapian::DatabaseModifiedError'
> Aborted
> 
> I have been unable to determine why the segfaults still sometimes occur 
> ... any suggestions?
> 
> Thanks!
> Jim

If it matters, this is under Ubuntu 6.06 LTS (Dapper Drake).
Permalink | Reply | 0 comments |
headers
Richard Heycock | 2 Jun 06:20
Picon

search without flush.

Hi,

Is it possible to perform a search without flushing the index? I've got
an application that updates the index every 4 hours but I need to be
able to search the new data fairly quickly after the index is updated.
The problem revolves around the fact that the update is often much less
than 10 000 documents so it isn't being flushed until quite a bit
latter. I realise I can do a flush after the documents have been updated
but I was trying to avoid the flush.

Is there a way of running a search after I've called add_document but
before flush is called?

rgh
Permalink | Reply | 6 comments |
headers
Charlie Hull | 2 Jun 12:58

Xapian's 10th Birthday

Hi all,

It's the Xapian project's 10th birthday this year - so how should we 
mark it? We've had meetups and gatherings over the past couple of years:
http://trac.xapian.org/wiki/MeetingsAndGatherings
Perhaps we should have a 'sprint' of development, or a hackday, or 
simply raise a glass?

Any ideas?

Cheers

Charlie
Permalink | Reply | 6 comments |
headers
Charlie Hull | 2 Jun 18:12

Re: Xapian 1.0.13 released

Olly Betts wrote:
> I've uploaded Xapian 1.0.13 (including Search::Xapian 1.0.13.0), which
> as usual you can download from:
> 
> http://xapian.org/download

Hi all,

Build files and prebuilt binaries for Windows are now available at
http://www.flax.co.uk/xapian_windows.shtml

Sorry for the delay,

Cheers

Charlie
Permalink | Reply | 0 comments |
headers
Miki Tebeka | 2 Jun 20:36
Favicon

Re: search without flush.

Hello Richard,

> Is it possible to perform a search without flushing the index? I've got
> an application that updates the index every 4 hours but I need to be
> able to search the new data fairly quickly after the index is updated.
> The problem revolves around the fact that the update is often much less
> than 10 000 documents so it isn't being flushed until quite a bit
> latter. I realise I can do a flush after the documents have been updated
> but I was trying to avoid the flush.
You can set the XAPIAN_FLUSH_THRESHOLD to a number lower than that.

All the best,
--
Miki Tebeka
miki <at> fattoc.com
Permalink | Reply | 1 comment |
headers
Miki Tebeka | 2 Jun 20:39
Favicon

Re: Xapian over NFS (again...)

Hello shad,

> What do you think about NFS mount options ? Did you use NFS mounted as
> sync ?
Hmm, IT did that. The output of the mount just shows rw and the address.

> About the writes batching, what happened without it ?
I got "DatabaseModifiedError" a lot.

All the best,
--
Miki Tebeka
miki <at> fattoc.com
Permalink | Reply | 0 comments |

Gmane