headers
Olly Betts | 1 Dec 2004 08:10
Favicon
Gravatar

constructors vs factory functions (was Re: TCL Binding working, but db_lock doesn't get removed?)

On Sat, Nov 06, 2004 at 10:37:58AM -0800, Eric Parusel wrote:
> [SWIG Tcl8 not calling WritableDatabase destructor bug]
> I joined, and sent an email, but it never made it to the list...
> I emailed the list maintainer, and never got a reply.

Well, I reduced it down to a really simple testcase and sent it to the
swig list (took me 3 attempts to post - it seems to just throw away any
mail with attachments):

http://article.gmane.org/gmane.comp.programming.swig/5422

But I've not had a single reply, either on or off the list.  I guess
that this isn't going to get fixed in SWIG any time soon...

So I thought about how we could work around this.  The problem is that
Tcl8 doesn't cope with a factory function which returns an object.  But
we could give WritableDatabase a normal constructor instead of using a
factory function.

And in fact thinking about this, it's probably a more natural design
for the API user.

I'm trying to remember quite why I chose factory functions originally.
The change is here:

http://cvs.xapian.org/xapian/xapian-core/api/Attic/omdatabaseinternal.cc.diff?r1=1.27&r2=1.28

http://cvs.xapian.org/xapian/xapian-core/include/om/Attic/omdatabase.h.diff?r1=1.51&r2=1.52

Previously the database type and parameters were encapsulated in an
(Continue reading)

Permalink | Reply | 1 comment |
headers
Olly Betts | 4 Dec 2004 19:52
Favicon
Gravatar

Re: constructors vs factory functions (was Re: TCL Binding working, but db_lock doesn't get removed?)

I wrote:
> So I thought about how we could work around this.  The problem is that
> Tcl8 doesn't cope with a factory function which returns an object.  But
> we could give WritableDatabase a normal constructor instead of using a
> factory function.
>
> And in fact thinking about this, it's probably a more natural design
> for the API user.

OK, I've now made this change - it'll be in 0.8.4 (coming soon).

And it fixes the tcl problem - provided you remember to use "db -delete"
the database is closed.

Cheers,
    Olly
Permalink | Reply | 0 comments |
headers
Eric Parusel | 6 Dec 2004 22:49

Search::Xapian::WritableDatabase SIGABRT?

Hello,

	On certain xapian dbs (my two largest ones, hrm) I cannot
write data to them anymore, without my import perl script aborting...

Searches on both these dbs seemingly work fine.

Size of files in each of those db's: (z-2004, and t-2004 fails, the rest 
work fine)
844M    i-2004
877M    g-2004
890M    a-2004
1.3G    z-2004
4.5G    t-2004

t-2004 file listing:
          0 Dec  6 13:13 db_lock
         10 Oct 27 14:40 meta
          0 Oct 27 14:40 position_DB
         17 Nov 29 23:13 position_baseA
         17 Nov 29 22:58 position_baseB
2465808384 Nov 29 23:29 postlist_DB
      37643 Nov 29 23:13 postlist_baseA
   15122432 Nov 29 23:28 record_DB
        223 Nov 29 23:13 record_baseA
2304335872 Nov 29 23:28 termlist_DB
      35186 Nov 29 23:13 termlist_baseA
          0 Oct 27 14:40 value_DB
         17 Nov 29 23:13 value_baseA
         17 Nov 29 22:58 value_baseB
(Continue reading)

Permalink | Reply | 5 comments |
headers
Olly Betts | 7 Dec 2004 02:15
Favicon
Gravatar

Re: Search::Xapian::WritableDatabase SIGABRT?

On 2004-12-06, Eric Parusel <eparusel <at> creativens.com> wrote:
> Searches on both these dbs seemingly work fine.
>
> [snip]
>
> I can email the straces I took for both dbs, without db_lock and with 
> db_lock already present, if requested...
>
> both states result in an abort.

I suspect it's this bug:

http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id=55

If it is, it looks like the Perl bindings don't handle C++ exceptions as
they should, at least in this case - both the "with db_lock" and
"without db_lock" cases will throw exceptions, just different ones.

Try applying the patch in that bug, or try a CVS snapshot (the ones
currently available are pretty close to what 0.8.4 will be - I'd have
made the release by now, but I'm presently unable to ssh to James'
colo box which hosts most of xapian.org...)

Cheers,
    Olly
Permalink | Reply | 4 comments |
headers
Eric Parusel | 7 Dec 2004 19:50

Re: Re: Search::Xapian::WritableDatabase SIGABRT?

Olly Betts wrote:
> On 2004-12-06, Eric Parusel <eparusel <at> creativens.com> wrote:
> 
>>Searches on both these dbs seemingly work fine.
>>
>>[snip]
>>
>>I can email the straces I took for both dbs, without db_lock and with 
>>db_lock already present, if requested...
>>
>>both states result in an abort.
> 
> 
> I suspect it's this bug:
> 
> http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id=55
> 
> If it is, it looks like the Perl bindings don't handle C++ exceptions as
> they should, at least in this case - both the "with db_lock" and
> "without db_lock" cases will throw exceptions, just different ones.
> 
> Try applying the patch in that bug, or try a CVS snapshot (the ones
> currently available are pretty close to what 0.8.4 will be - I'd have
> made the release by now, but I'm presently unable to ssh to James'
> colo box which hosts most of xapian.org...)

Ok, I will try out the patch, or CVS...  I would prefer to wait for a 
debian package, but I suppose my updates need to resume soon :)

As for the Perl bindings not returning the exception, what can I do to
(Continue reading)

Permalink | Reply | 3 comments |
headers
Olly Betts | 8 Dec 2004 04:41
Favicon
Gravatar

Re: Re: Search::Xapian::WritableDatabase SIGABRT?

On Tue, Dec 07, 2004 at 10:50:26AM -0800, Eric Parusel wrote:
> Ok, I will try out the patch, or CVS...  I would prefer to wait for a 
> debian package, but I suppose my updates need to resume soon :)

It'll be at least a few days before I sort out the release and Richard
sorts out debian packages for it.  Perhaps longer if Richard is busy.

If you want a quick fix, you should be able to use copydatabase (from
xapian-examples) to make a copies of the problem databases and use the
copies instead.

Or you could use quartzcompact to copy them.  That's much faster than
copydatabase (it copies at the Btree level rather than document by
document), but you end up with a database with full compaction which
means that updates will be slower for a while as every updated block
will need to be split.

> As for the Perl bindings not returning the exception, what can I do to 
> get Perl to not kick the bucket in these cases?

I've had a quick look.  I added a new testcase which tries to open
the same database for writing twice, and checks it got an exception
the second time.  And this works for me!

So I don't know why you're getting SIGABRT.  I've attached the testcase
- just stick it in Search-Xapian-0.8.3.1/t with the other ".t" files and
run "make test" in Search-Xapian-0.8.3.1 to run all the tests.

Cheers,
    Olly
(Continue reading)

Permalink | Reply | 2 comments |
headers
Eric Parusel | 8 Dec 2004 09:48

Re: Re: Search::Xapian::WritableDatabase SIGABRT?

Olly Betts wrote:
> Or you could use quartzcompact to copy them.  That's much faster than
> copydatabase (it copies at the Btree level rather than document by
> document), but you end up with a database with full compaction which
> means that updates will be slower for a while as every updated block
> will need to be split.

Hmm, I tried quartzcompact twice, and both times it seemingly is paused 
or stopped at the same point? (destination db at 612M of content)

# du -sh t-2004*
4.5G    t-2004
612M    t-2004-compact2
612M    t-2004-compact

root     25128 75.4  0.2  4436 2080 pts/2    RN   00:00  23:48 
quartzcompact t-2004 t-compact2

It's been running for 20 minutes so far (99% cpu) without a change in 
destination db size...  is this normal?   I have killed it, but would
it have continued writing if I had left it for longer? (Xeon 2.8Ghz)

Uh oh...
------snip------
#dmesg
...
megaraid: aborting-1770320103 cmd=2a <c=0 t=0 l=0>
megaraid: abort sequence successfully completed.
megaraid: aborting-1770320104 cmd=2a <c=0 t=0 l=0>
megaraid: abort sequence successfully completed.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Olly Betts | 8 Dec 2004 16:02
Favicon
Gravatar

Re: Re: Search::Xapian::WritableDatabase SIGABRT?

On Wed, Dec 08, 2004 at 12:48:58AM -0800, Eric Parusel wrote:
> Hmm, I tried quartzcompact twice, and both times it seemingly is paused 
> or stopped at the same point? (destination db at 612M of content)
> [snip]
> It's been running for 20 minutes so far (99% cpu) without a change in 
> destination db size...  is this normal?

No, I'd expect whichever of the 5 output _DB files is currently being
worked on to grow pretty much continuously.

Did you try quartzcheck on the problem databases?

> [RAID errors]  This may be a source of corruption potentially.

I guess data corruption could create an infinite (or long) loop somehow.

> Output:
> # make test
> PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib 
> -I/usr/lib/perl/5.6.1 -I/usr/share/perl/5.6.1 -e 'use Test::Harness 
> qw(&runtests $verbose); $verbose=0; runtests  <at> ARGV;' t/*.t
> t/create.......ok
> t/document.....ok
> t/exception....dubious
>         Test returned status 0 (wstat 6, 0x6)
> DIED. FAILED test 4
>         Failed 1/4 tests, 75.00% okay
> t/index........dubious
>         Test returned status 0 (wstat 6, 0x6)
> DIED. FAILED tests 21-40
(Continue reading)

Permalink | Reply | 0 comments |
headers
Olly Betts | 8 Dec 2004 19:48
Favicon
Gravatar

Xapian 0.8.4 released

I've uploaded Xapian 0.8.4:

http://www.xapian.org/download.php

There are three important changes.  The first is a change to the
preferred way of opening a database - the existing way will still work;
the second and third are incompatible changes to features we believe
aren't widely used.  The second should be caught at compile time, but
the third won't:

* Added constructors to Database and WritableDatabase which fulfil the role
  that the Auto::open() factory functions currently do.  Auto::open() is
  now deprecated.

* Removed the ability to write a Xapian object to an ostream directly, as
  it's little used and potentially dangerous ('cout << mset[i];' will
  compile, but you almost certainly meant 'cout << *mset[i];').  You can
  get the old effect by writing 'cout << obj->get_description();' instead
  of 'cout << obj;'.  Note that including xapian.h no longer pulls in
  fstream, which code may have been implicitly relying on - if this is
  a problem add '#include <fstream>' after '#include <xapian.h>'.

* Renamed BM25 parameters to match standard naming in papers and elsewhere
  (A->k3, B->k1, C->k2, D->b), eliminated the extra factor of 2 which our C
  had, and reordered the parameters to k1, k2, k3.  This is an incompatible API
  change for BM25Weight(), so if you are using custom parameters for BM25
  you'll need to update your code.

A few other highlights:
(Continue reading)

Permalink | Reply | 3 comments |
headers
Olly Betts | 9 Dec 2004 02:05
Favicon
Gravatar

Compressed Btrees

I've been working on allowing Btree tags to be compressed using zlib,
and it's now working sufficiently well that some external testing would
be useful (a patch to xapian-core 0.8.4.0 is attached).

What I've done is to find a bit in the Btree key header which is always
0 at present.  Now it is set to 1 if the tag value has been compressed
with zlib.  So the patched code can read existing databases (but new
databases won't be readable with the new code if any tags have been
compressed).

I originally intended this to be used for compressing the document data
(which is stored in the record Btree).  But my experiments so far suggest
it's also worth compressing at least the termlist.

So you can experiment, the patch looks for two files per Btree to decide
whether to compress and how to compress.  For the record btree (which
currently uses record_DB, record_baseA, and record_baseB) these files
are record_compress and record_compress_strategy.

If record_compress exists, then quartz will use zlib to try to compress
any tag added to the record table which is more than 4 bytes long.

If record_compress isn't empty, then the contents are used as a dictionary
to seed compression.  So for the record table, putting in a typical record
will improve the compression achieved.

If record_compress_strategy exists, the first byte is read to determine
the compression strategy which zlib will be told to use:

* "F" or "f" : Z_FILTERED
(Continue reading)

Permalink | Reply | 15 comments |

Gmane