General discussion about Xapian, an Open Source Probabilistic Information Retrieval library

Jesper Krogh | 1 Feb 06:23

Re: Failure trying to update document.

Olly Betts wrote:
> On Sun, Jan 31, 2010 at 08:00:02PM +0100, Jesper Krogh wrote:
>> Jesper Krogh wrote:
>>> Which is kind of odd, it matches the documents terms gets the docid but
>>> fails to get the termslist?
> 
> The postlist is used to find the documents matching particular terms.  The
> termlist holds the reverse mapping - for each document, it stores the list
> of terms which index it.
> 
>> $ xapian-check termlist.DB
>> baseA blocksize=8K items=2980957 lastblock=2357252 revision=2452
>> levels=3 root=108
>> B-tree error 90
>> xapian-check: btree error
>>
>> Can I somehow "fix" it?
> 
> The error here is that the keys in the Btree don't satisfy an expected
> invariant, so it's hard to know quite how corrupted things are.
> 
> If you have the original data, rebuilding is the simplest and safest
> option.
> 
> You may be able to use copydatabase to generate a good copy.  If the
> issue is just a missing termlist entry for one (or several) documents,
> that would skip those, but if things are more messed up it may fail.

I tried to run xapian-compact over night. It has processed the termlist
now and xapian-check is ok on it. Now I'm just waiting for the

(Continue reading)

Olly Betts | 1 Feb 06:51

Re: Failure trying to update document.

On Mon, Feb 01, 2010 at 06:23:47AM +0100, Jesper Krogh wrote:
> Olly Betts wrote:
> > If you have the original data, rebuilding is the simplest and safest
> > option.
> > 
> > You may be able to use copydatabase to generate a good copy.  If the
> > issue is just a missing termlist entry for one (or several) documents,
> > that would skip those, but if things are more messed up it may fail.
> 
> I tried to run xapian-compact over night. It has processed the termlist
> now and xapian-check is ok on it. Now I'm just waiting for the
> position.DB to finish.

That would fix the problem if it is just with a dividing key in a non-leaf
block.  If an entry is actually missing, you'll still get DocNotFoundError,
and copydatabase may still help.

Cheers,
    Olly

Henry C. | 1 Feb 09:38

Re: postlist: Tag containing meta information is corrupt.


Thanks to Richard and Olly for sorting out this bug.

Just checking - wrt http://trac.xapian.org/ticket/427:

When running xapian-check, the previous problem has been resolved, but I
still get "Tag containing meta information is corrupt." on the postlist,
resulting in retcode 1.

Is this something which can be safely ignored for now?

Thanks
Henry

Tom | 1 Feb 10:48

Solaris core dump

Hi everyone,

I'm having a problem with xapian (the matchspy branch, with Python
bindings) on Solaris 10 / SPARC. Users can run queries for a few hours
with no problems, then it returns "inflate failed (invalid code
lengths set)"  to Python and dumps core. gdb reports:

#0  0xfe6dc910 in inflate_table () from /usr/lib/libz.so.1
#1  0xfe6d9d6c in inflate () from /usr/lib/libz.so.1
#2  0xfe2ebe20 in FlintTable::read_tag (this=0x75ed70, C_=0x75ede0,
   tag=0xfdcf8568, keep_compressed=false)
   at backends/flint/flint_table.cc:1254
#3  0xfe2eff70 in FlintTable::get_exact_entry (this=0x75ed70, key=@0xfdcf8578,
   tag=@0xfdcf8568) at backends/flint/flint_table.cc:1190
#4  0xfe2ddc04 in FlintSpellingTable::open_termlist (this=0x75ed70,
   word=@0xfdcf8998) at backends/flint/flint_spelling.h:46
#5  0xfe20dcb0 in Xapian::Database::get_spelling_suggestion (this=0x75b5a8,
   word=@0xfdcf8998, max_edit_distance=2) at include/xapian/base.h:476
#6  0xfe39ba90 in Xapian::QueryParser::Internal::parse_query (this=0x75b590,
   qs=@0x798288, flags=128, default_prefix=@0xfdcf8a44)
   at queryparser/queryparser.lemony:948
#7  0xfe38fa0c in Xapian::QueryParser::parse_query (this=0x798218,
   query_string=@0x798288, flags=128, default_prefix=@0xfdcf8be0)
   at include/xapian/base.h:154
#8  0xfe5551f0 in _wrap_QueryParser_parse_query (self=0x0, args=0x712648)
   at /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../include/c++/3.4.3/ext/new_allocator.h:69
#9  0x00117290 in PyCFunction_Call (func=0x60d9e0, arg=0x712648, kw=0x0)
   at Objects/methodobject.c:116

This doesn't seem to be related to any particular query, and I haven't

(Continue reading)

Tom | 1 Feb 10:41

Xapian core dumping on Solaris

Hi everyone,

I'm having a problem with xapian (the matchspy branch, with Python
bindings) on Solaris 10 / SPARC. Users can run queries for a few hours
with no problems, then it returns "inflate failed (invalid code
lengths set)"  to Python and dumps core. gdb reports:

#0  0xfe6dc910 in inflate_table () from /usr/lib/libz.so.1
#1  0xfe6d9d6c in inflate () from /usr/lib/libz.so.1
#2  0xfe2ebe20 in FlintTable::read_tag (this=0x75ed70, C_=0x75ede0,
    tag=0xfdcf8568, keep_compressed=false)
    at backends/flint/flint_table.cc:1254
#3  0xfe2eff70 in FlintTable::get_exact_entry (this=0x75ed70, key=@0xfdcf8578,
    tag=@0xfdcf8568) at backends/flint/flint_table.cc:1190
#4  0xfe2ddc04 in FlintSpellingTable::open_termlist (this=0x75ed70,
    word=@0xfdcf8998) at backends/flint/flint_spelling.h:46
#5  0xfe20dcb0 in Xapian::Database::get_spelling_suggestion (this=0x75b5a8,
    word=@0xfdcf8998, max_edit_distance=2) at include/xapian/base.h:476
#6  0xfe39ba90 in Xapian::QueryParser::Internal::parse_query (this=0x75b590,
    qs=@0x798288, flags=128, default_prefix=@0xfdcf8a44)
    at queryparser/queryparser.lemony:948
#7  0xfe38fa0c in Xapian::QueryParser::parse_query (this=0x798218,
    query_string=@0x798288, flags=128, default_prefix=@0xfdcf8be0)
    at include/xapian/base.h:154
#8  0xfe5551f0 in _wrap_QueryParser_parse_query (self=0x0, args=0x712648)
    at /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../include/c++/3.4.3/ext/new_allocator.h:69
#9  0x00117290 in PyCFunction_Call (func=0x60d9e0, arg=0x712648, kw=0x0)
    at Objects/methodobject.c:116

This doesn't seem to be related to any particular query, and I haven't

(Continue reading)

Richard Boulton | 1 Feb 10:57

Re: Xapian core dumping on Solaris

On 1 February 2010 09:41, Tom <banoffi <at> gmail.com> wrote:
> I'm having a problem with xapian (the matchspy branch, with Python
> bindings) on Solaris 10 / SPARC. Users can run queries for a few hours
> with no problems, then it returns "inflate failed (invalid code
> lengths set)"  to Python and dumps core. gdb reports:

To be clear - when you say "returns "inflate failed ..." to Python",
do you mean that it prints that message out on the console?  Or does
it return a value to python, and then dump core shortly afterwards?

> This doesn't seem to be related to any particular query, and I haven't
> worked out a simple way to reproduce the error. There is no database
> writer while all this happens.

Is this in a single threaded environment?  The most likely cause of
this kind of error is accidentally calling a xapian object from two
threads simultaneously.

> Any ideas? I might try rebuilding with the latest version of zlib -
> does that sound sensible?

Sounds worth a try - what zlib version are you using currently?

> Could it even be a low-level I/O error?

Could be, but it doesn't look particularly likely to me, from that traceback.

--

-- 
Richard

(Continue reading)

Tom | 1 Feb 11:06

Re: Xapian core dumping on Solaris

On 1 February 2010 09:57, Richard Boulton <richard <at> tartarus.org> wrote:
> On 1 February 2010 09:41, Tom <banoffi <at> gmail.com> wrote:
>> I'm having a problem with xapian (the matchspy branch, with Python
>> bindings) on Solaris 10 / SPARC. Users can run queries for a few hours
>> with no problems, then it returns "inflate failed (invalid code
>> lengths set)"  to Python and dumps core. gdb reports:
>
> To be clear - when you say "returns "inflate failed ..." to Python",
> do you mean that it prints that message out on the console?  Or does
> it return a value to python, and then dump core shortly afterwards?

Hi Richard,

The latter. The error message is caught and logged before the process dies.

>> This doesn't seem to be related to any particular query, and I haven't
>> worked out a simple way to reproduce the error. There is no database
>> writer while all this happens.
>
> Is this in a single threaded environment?  The most likely cause of
> this kind of error is accidentally calling a xapian object from two
> threads simultaneously.

It's a multithreaded web app, using web.py. The only shared object,
though, is the Database. That's safe, isn't it?

>> Any ideas? I might try rebuilding with the latest version of zlib -
>> does that sound sensible?
>
> Sounds worth a try - what zlib version are you using currently?

(Continue reading)

Richard Boulton | 1 Feb 11:09

Re: Xapian core dumping on Solaris

On 1 February 2010 10:06, Tom <banoffi <at> gmail.com> wrote:
> The latter. The error message is caught and logged before the process dies.

Weird - the backtrace makes it look like the process has died in the
middle of a call to zlib, so I can't see how it could return the error
to Python before dumping core.

> It's a multithreaded web app, using web.py. The only shared object,
> though, is the Database. That's safe, isn't it?

No (assuming you mean the Xapian Database).  You can't have any Xapian
objects accessed concurrently, including the Database objects (or any
objects derived from them, like iterator objects, since those will
access the Database internally).

--

-- 
Richard

Tom | 1 Feb 12:01

Re: Xapian core dumping on Solaris

On 1 February 2010 10:09, Richard Boulton <richard <at> tartarus.org> wrote:
> On 1 February 2010 10:06, Tom <banoffi <at> gmail.com> wrote:
>> The latter. The error message is caught and logged before the process dies.
>
> No (assuming you mean the Xapian Database).  You can't have any Xapian
> objects accessed concurrently, including the Database objects (or any
> objects derived from them, like iterator objects, since those will
> access the Database internally).

OK, I've fixed that - there are no shared Xapian objects at all. But
it's just crashed again, with a different backtrace:

#0  0xfe1196d0 in std::basic_string<char, std::char_traits<char>,
std::allocator<char> >::basic_string () from
/usr/sfw/lib/libstdc++.so.6
#1  0xfe22e9a4 in Internal (this=0x3a1680, copyme=@0x7b4e00)
    at api/omqueryinternal.cc:654
#2  0xfe22e9f4 in Internal (this=0x5788ac, copyme=@0x7b0cc8)
    at /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../include/c++/3.4.3/bits/stl_iterator.h:614
#3  0xfe34b180 in LocalSubMatch (this=0x5788a0, db_=0x795f10, query=0x7b0cc8,
    qlen_=4, omrset=@0x56e4c0, wt_factory_=0x7e3608)
    at matcher/localmatch.cc:49
#4  0xfe360424 in MultiMatch (this=0xfdaf8770, db_=@0x57df5c,
    query_=0x7b0cc8, qlen=4, omrset=0x0, collapse_max_=0,
    collapse_key_=4256138712, percent_cutoff_=5694656, weight_cutoff_=0,
    order_=Xapian::Enquire::ASCENDING, sort_key_=1,
    sort_by_=Xapian::Enquire::Internal::REL, sort_value_forward_=true,
    sorter_=0x0, errorhandler_=0x0, stats=@0xfdaf8808, weight_=0xffffff00,
    matchspies_=@0xffffff00)
    at /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../include/c++/3.4.3/bits/stl_vector.h:462

(Continue reading)

Tom | 1 Feb 12:18

Re: Xapian core dumping on Solaris

And another -

#0  0xff056328 in _smalloc () from /usr/lib/libc.so.1
#1  0xff05639c in malloc () from /usr/lib/libc.so.1
#2  0xfe137ae4 in operator new () from /usr/sfw/lib/libstdc++.so.6
#3  0xfe119598 in std::string::_Rep::_S_create ()
   from /usr/sfw/lib/libstdc++.so.6
#4  0xfe3aa730 in std::string::_S_construct<char*> (
    __beg=0x7ac6dc "and, 1, 2b8]", __end=0x7ac6e5 "b8]", __a=@0xfdef81f0)
    at /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../include/c++/3.4.3/bits/basic_string.tcc:150
#5  0xfe3ad86c in basic_string<char*> (this=0xfdef81f8,
    __beg=0x7ac6dc "and, 1, 2b8]", __end=0x7ac6e5 "b8]", __a=@0xfdef81f0)
    at /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../include/c++/3.4.3/bits/basic_string.h:1386
#6  0xfe2296e8 in Query (this=0xfdef84b8, tname_=@0xfe43d740, wqf_=1, pos_=2)
    at /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../include/c++/3.4.3/ext/new_allocator.h:62
#7  0xfe393380 in Term::get_query (this=0x1d9630)
    at /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../include/c++/3.4.3/bits/stl_list.h:210
#8  0xfe393db4 in Term::get_query_with_auto_synonyms (this=0x1d9630)
    at queryparser/queryparser.lemony:303
#9  0xfe394628 in TermGroup::as_group (this=0x6f1ed8, state=0xfdef8a60)
    at /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../include/c++/3.4.3/bits/stl_iterator.h:614
#10 0xfe3982c4 in yy_reduce (yypParser=0x74e608, yyruleno=36)
    at queryparser/queryparser.lemony:1708
#11 0xfe3996c4 in Parse (yypParser=0x74e608, yymajor=0, yyminor=0x0,
    state=0x0) at queryparser/queryparser_internal.cc:2674
#12 0xfe39a8ec in Xapian::QueryParser::Internal::parse_query (this=0x340c20,
    qs=@0x6f6ee0, flags=128, default_prefix=@0xfdef8a44)
    at queryparser/queryparser.lemony:1045
#13 0xfe38fa0c in Xapian::QueryParser::parse_query (this=0x6f6e90,
    query_string=@0x6f6ee0, flags=128, default_prefix=@0xfdef8be0)

(Continue reading)

Group

gmane.comp.search.xapian.general.

Project Web Page

General discussion about Xapian, an Open Source Probabilistic Information Retrieval library

Search Archive

Page

1 | 2 | 3 | 4 | 5 | 6

Articles in period: 57

February 2010

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane

Recent entries

get_docid??? (23 Jan 11:23)
Python Xapian tutorial (20 Jan 18:16)
get_docid??? (20 Jan 17:00)
Perl version of sortable_serialize missing? (20 Jan 12:02)
I'm trying to relate what I know about Omega/Scriptindex with the actu (15 Jan 17:49)
I'm trying to relate what I know about Omega/Scriptindex with the actu (15 Jan 14:55)
Another broken link (15 Jan 14:07)
Wiki broken link (15 Jan 12:57)
Testing document size preallocation. (8 Jan 22:32)
Enhance synonyms feature of the query parser (patch included) (5 Jan 06:28)
[issue] The difference between QueryParser::FLAG_AUTO_SYNONYMS and Que (4 Jan 08:53)
Xapian php module doesn't load. (27 Dec 14:01)
How to choose the proximity between search words (22 Dec 15:36)
Xapian 1.2.8 released (16 Dec 10:29)
Xapian 1.2.8 released (14 Dec 11:32)
How to choose the proximity between search words (7 Dec 15:14)
pbs with stdin and scriptindex (5 Dec 14:54)
Problems with scriptindex (2 Dec 11:42)
Apache conf.d file for omega? (1 Dec 10:11)
not found on this server. (29 Nov 12:40)
TermGenerators not respecting Stopper class in Python (28 Nov 20:16)

Archives

2	February 2012
35	January 2012
22	December 2011
67	November 2011
86	October 2011
49	September 2011
52	August 2011
55	July 2011
53	June 2011
40	May 2011
58	April 2011
43	March 2011
28	February 2011
50	January 2011
45	December 2010
33	November 2010
63	October 2010
64	September 2010
50	August 2010
34	July 2010
67	June 2010
21	May 2010
41	April 2010
54	March 2010
58	February 2010
73	January 2010
50	December 2009
97	November 2009
63	October 2009
150	September 2009
72	August 2009
94	July 2009
83	June 2009
119	May 2009
109	April 2009
116	March 2009
92	February 2009
68	January 2009
114	December 2008
161	November 2008
117	October 2008
92	September 2008
97	August 2008
92	July 2008
55	June 2008
62	May 2008
130	April 2008
104	March 2008
95	February 2008
191	January 2008
84	December 2007
155	November 2007
186	October 2007
143	September 2007
147	August 2007
137	July 2007
213	June 2007
180	May 2007
150	April 2007
128	March 2007
174	February 2007
96	January 2007
142	December 2006
220	November 2006
101	October 2006
88	September 2006
116	August 2006
88	July 2006
98	June 2006
154	May 2006
136	April 2006
229	March 2006
196	February 2006
123	January 2006
75	December 2005
38	November 2005
61	October 2005
80	September 2005
78	August 2005
29	July 2005
78	June 2005
59	May 2005
44	April 2005
79	March 2005
37	February 2005
33	January 2005
78	December 2004
34	November 2004
108	October 2004
141	September 2004
49	August 2004
34	July 2004
107	June 2004
112	May 2004
45	April 2004
202	March 2004
47	February 2004
63	January 2004
39	December 2003
8	November 2003
28	October 2003
40	September 2003
16	August 2003
66	July 2003
19	June 2003
41	May 2003
43	April 2003
22	March 2003
15	February 2003
14	January 2003
20	December 2002

Design

Zawodny | Right Menu | Left Menu

Your Own Design

Paste the URL of your CSS below. Download CSS template