Xapian Indexer problem.

_______________________________________________
Xapian-devel mailing list
Xapian-devel <at> lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Xapian Indexer problem.

On Fri, Nov 14, 2008 at 11:06:55AM +0800, liminghit wrote:
> Only ??0 001a heathrow taxis?? can have 100% matching.
> 
> Shorter or longer query, should less than 100% matching, right?

A longer query would since (unless you repeat terms) it must have
words which aren't in the document.

But otherwise no, and this behaviour is as intended.  It's not
"percentage of document text matched" it's a measure of "how well your
query matches this document".

If all the query terms match the highest scoring document, we give it
100%.  If not all the terms match the highest scoring document, we give
it a proportion of 100% based on the term weights

And then we calculate percentage scores for all other documents based on
this assigned percentage value.

Your definition seems unhelpful to me - in most uses the query is quite
a lot shorter than the document, and a 3 word query would score at most
0.3% for a 1000 word document.

> If I want to archive this, how to do indexing?

You might be able to achieve something like what you describe at search
time by writing your own weighting scheme and making get_sumpart()
return 1/(unnormalised document length)

Cheers,
    Olly

Re: Xapian Indexer problem.

_______________________________________________
Xapian-devel mailing list
Xapian-devel <at> lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Trying to patch xapian perl add/remove_spelling

Greets,

I'm giving a stab at patching the CPAN module to add the missing  
WritableDatabase::add_spelling and remove_spelling, but need a bit of  
guidance since I'm coming in cold, and pressed for time (aren't we all).

I've modified XS/WritableDatabase.xs and added the two necessary  
functions, and also added the two basic tests in t/index.t.

Compilation completes cleanly, but running

perl index.t

fails with

Exception: This backend doesn't implement spelling correction at  
index.t line 48.

I've obviously missed some file or something that needs to be set.   
The patch is attached.

I'd appreciate some comments wrt what I'm missing.

Thanks
Henry

_______________________________________________
Xapian-devel mailing list
Xapian-devel <at> lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Trying to patch xapian perl add/remove_spelling

On Wed, Nov 26, 2008 at 12:08:40PM +0200, Henry wrote:
> Exception: This backend doesn't implement spelling correction at  
> index.t line 48.
> 
> I've obviously missed some file or something that needs to be set.   
> The patch is attached.
> 
> I'd appreciate some comments wrt what I'm missing.

Much of the code in index.t is run for both inmemory and "auto"
backends, but the inmemory backend doesn't implement spelling
correction.

Also, you can't use "ok(...)" around methods which don't return
anything (if you do call "ok(...)" more, you need to increase the
expected number of calls at the top of the file).

I've tweaked the patch to compile and pass "make check".  I haven't
applied it yet as it doesn't seem to be useful without
QueryParser::get_corrected_query_string() and/or
Database::get_spelling_suggestion() also being wrapped...

http://oligarchy.co.uk/xapian/patches/xapian-perl-spelling-updated.patch

Cheers,
    Olly

Re: Trying to patch xapian perl add/remove_spelling

Quoting "Olly Betts" <olly <at> survex.com>:
> Much of the code in index.t is run for both inmemory and "auto"
> backends, but the inmemory backend doesn't implement spelling
> correction.

Gotcha.

> Also, you can't use "ok(...)" around methods which don't return
> anything (if you do call "ok(...)" more, you need to increase the
> expected number of calls at the top of the file).

Thanks - 'fraid I've never got around to using Test::More...  /makes  
mental note to do so.

> I've tweaked the patch to compile and pass "make check".  I haven't
> applied it yet as it doesn't seem to be useful without
> QueryParser::get_corrected_query_string() and/or
> Database::get_spelling_suggestion() also being wrapped...

Hm, will take a few stabs there too...

Cheers
Henry