Development discussion about Xapian, an Open Source Probabilistic Information Retrieval library

headers Radovan Garabik | 9 Dec 2005 11:50 · /local/

Hi all,
I am rather new to xapian, I just recently tried to include it in
my application, so bear with me if this has already been discussed.

I was playing with QueryParser and noticed that it expects
input to be in ISO8859_1 encoding - characters above 0x80 are
transliterated, and are not considered letters. For example,
using single word (in utf-8 encoding) "bože" as input for 
parse_query, the resulting query is something like:
Xapian::Query((boaa:(pos=1) OR e:(pos=2)))
which makes the parse_query quite unusable for UTF-8 strings (or 
indeed, for any encoding other than ISO8859_1).

I tried to disable the transliteration in 
accentnormalisingitor.h and modified common/utils.h to contain:

inline bool C_isalpha(char ch) {
    using namespace Xapian::Internal;
    return (static_cast<unsigned char>(ch)>=0x80) ||
(is_tab[static_cast<unsigned char>(ch)] & (IS_UPPER|IS_LOWER));
}

inline bool C_isalnum(char ch) {
    using namespace Xapian::Internal;
    return (static_cast<unsigned char>(ch)>=0x80) ||
(is_tab[static_cast<unsigned char>(ch)] & (IS_UPPER|IS_LOWER|IS_DIGIT));
}

since most of the characters above 0x80 are meant as letters, only with 
very few exceptions (non breaking spaces and punctuation, and 

Mon	Tue	Wed	Thu	Fri	Sat	Sun

			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30	31

2	May 2013
15	April 2013
81	March 2013
18	February 2013
37	January 2013
3	December 2012
2	November 2012
31	October 2012
9	September 2012
4	August 2012
22	July 2012
12	June 2012
84	May 2012
103	April 2012
96	March 2012
8	February 2012
2	January 2012
19	December 2011
4	September 2011
11	August 2011
14	July 2011
15	June 2011
37	May 2011
57	April 2011
93	March 2011
11	February 2011
9	January 2011
6	December 2010
2	October 2010
9	September 2010
5	August 2010
7	June 2010
4	April 2010
9	March 2010
5	February 2010
3	January 2010
3	November 2009
11	September 2009
6	August 2009
20	July 2009
4	June 2009
14	May 2009
22	April 2009
24	January 2009
4	December 2008
6	November 2008
16	October 2008
4	September 2008
8	August 2008
11	July 2008
8	June 2008
1	May 2008
21	April 2008
5	March 2008
9	February 2008
3	January 2008
11	December 2007
6	November 2007
19	October 2007
24	September 2007
2	August 2007
5	July 2007
30	June 2007
26	May 2007
146	April 2007
57	March 2007
44	February 2007
12	January 2007
32	December 2006
44	November 2006
23	October 2006
6	September 2006
20	August 2006
11	July 2006
34	June 2006
22	May 2006
70	April 2006
13	March 2006
4	February 2006
8	January 2006
17	December 2005
11	November 2005
2	October 2005
4	September 2005
22	August 2005
19	July 2005
19	June 2005
24	May 2005
9	April 2005
8	March 2005
11	February 2005
1	January 2005
41	December 2004
4	October 2004
17	September 2004
16	August 2004
17	June 2004
41	May 2004
4	April 2004
3	March 2004
16	January 2004
2	December 2003
27	November 2003
1	October 2003
15	August 2003
65	July 2003
29	June 2003
32	May 2003
25	April 2003
29	March 2003
37	February 2003
20	January 2003
25	December 2002

QueryParser and utf-8 strings

Re: QueryParser and utf-8 strings

Xapian Binding compile error in Windows XP using CygWin

Re: Xapian Binding compile error in Windows XP using CygWin

Re: Xapian Binding compile error in Windows XP using CygWin

Re: Xapian Binding compile error in Windows XP using CygWin

Re: Re: Xapian Binding compile error in Windows XP using CygWin

Re: Xapian Binding compile error in Windows XP using CygWin

Re: Re: Xapian Binding compile error in Windows XP using CygWin

Re: Xapian Binding compile error in Windows XP using CygWin