Development discussion about Xapian, an Open Source Probabilistic Information Retrieval library

Olly Betts | 1 Mar 23:37

Re: [Xapian-commits] 7817: trunk/xapian-core/ trunk/xapian-core/common/

On Thu, Mar 01, 2007 at 03:29:26PM +0000, richard wrote:
> * common/utils.c,common/utils.h: For windows, add an om_tostring()function for
>   64 bit integers: time() and GetProcessId() return these, and without this,
>   backends/quartz/quartz_log.cc reports an error.

This change generates a new warning in the mingw cross-build:

common/utils.cc:82: warning: int format, different type arg (arg 3)

Presumably this warning is bogus, since mingw uses the MSVC C runtime
DLL?

Cheers,
    Olly

Richard Boulton | 2 Mar 10:14

Re: Re: [Xapian-commits] 7817: trunk/xapian-core/ trunk/xapian-core/common/

Olly Betts wrote:
> On Thu, Mar 01, 2007 at 03:29:26PM +0000, richard wrote:
>> * common/utils.c,common/utils.h: For windows, add an om_tostring()function for
>>   64 bit integers: time() and GetProcessId() return these, and without this,
>>   backends/quartz/quartz_log.cc reports an error.
> 
> This change generates a new warning in the mingw cross-build:
> 
> common/utils.cc:82: warning: int format, different type arg (arg 3)
> 
> Presumably this warning is bogus, since mingw uses the MSVC C runtime
> DLL?

It sounds pretty bogus - perhaps the best plan would be to put together 
a test (presumably in internaltest.cc) which checked that it was working 
correctly.  I'll look into that shortly.

--

-- 
Richard

Olly Betts | 2 Mar 13:56

Re: Re: [Xapian-commits] 7817: trunk/xapian-core/ trunk/xapian-core/common/

On Fri, Mar 02, 2007 at 09:14:56AM +0000, Richard Boulton wrote:
> Olly Betts wrote:
> >On Thu, Mar 01, 2007 at 03:29:26PM +0000, richard wrote:
> >>* common/utils.c,common/utils.h: For windows, add an 
> >>om_tostring()function for
> >>  64 bit integers: time() and GetProcessId() return these, and without 
> >>  this,
> >>  backends/quartz/quartz_log.cc reports an error.

If this is only required by the quartz log stuff, I wonder if we should
just drop that code.

It doesn't get used unless you touch "log" in a quartz database by hand,
but that's not documented anywhere so I doubt anybody except you and me
knows to even try it!  We could document it, but I never actually found
it a useful debugging aid, and haven't used it at all in years.  Plus
quartz will be deprecated for new systems as of 1.0 anyway.

But I suspect support for 64 bit integers is also required for the debug
logging code.

Hmm, and I don't understand how this worked for MSVC before this patch
if it gives an error now.  Or why I haven't seen an error building for
mingw.

> >This change generates a new warning in the mingw cross-build:
> >
> >common/utils.cc:82: warning: int format, different type arg (arg 3)
> >
> >Presumably this warning is bogus, since mingw uses the MSVC C runtime

(Continue reading)

Richard Boulton | 2 Mar 14:10

Re: Re: [Xapian-commits] 7817: trunk/xapian-core/ trunk/xapian-core/common/

Olly Betts wrote:
> If this is only required by the quartz log stuff, I wonder if we should
> just drop that code.
> 
> It doesn't get used unless you touch "log" in a quartz database by hand,
> but that's not documented anywhere so I doubt anybody except you and me
> knows to even try it!  We could document it, but I never actually found
> it a useful debugging aid, and haven't used it at all in years.  Plus
> quartz will be deprecated for new systems as of 1.0 anyway.

All true, and good points - I haven't ever really used it either.

> But I suspect support for 64 bit integers is also required for the debug
> logging code.

Quite possibly - we've not got round to building with debug logging 
turned on yet.

> Hmm, and I don't understand how this worked for MSVC before this patch
> if it gives an error now.  Or why I haven't seen an error building for
> mingw.

I think it's only needed for the newer compilers, because time_t only 
recently became a 64 bit quantity.  And GetProcessId similarly, I 
suppose.  Charlie will know more detail.

> A testcase is a good idea, but it's going to be tricky for me to
> exercise it in a cross-build...

Fair point.  I've written one anyway, but I can't test it either.  I'll

(Continue reading)

Olly Betts | 2 Mar 15:02

Re: Re: [Xapian-commits] 7817: trunk/xapian-core/ trunk/xapian-core/common/

On Fri, Mar 02, 2007 at 01:10:07PM +0000, Richard Boulton wrote:
> Olly Betts wrote:
> >Hmm, and I don't understand how this worked for MSVC before this patch
> >if it gives an error now.  Or why I haven't seen an error building for
> >mingw.
> 
> I think it's only needed for the newer compilers, because time_t only 
> recently became a 64 bit quantity.  And GetProcessId similarly, I 
> suppose.  Charlie will know more detail.

Ah, and I bet there's some C preprocessor magic or other trickery going
on so that programs can use the same runtime DLLs but just call a
different version of the functions concerned.

> >A testcase is a good idea, but it's going to be tricky for me to
> >exercise it in a cross-build...
> 
> Fair point.  I've written one anyway, but I can't test it either.  I'll 
> add it to a bug report, so we don't forget to look into this.

I suggest you commit it to HEAD anyway.  It can be tested with MSVC, and
if there's a bug in the test, the worst that can happen is it might stop
"make check" from working under native mingw when someone tries it,
until the problem is reported and fixed.

Cheers,
    Olly

Olly Betts | 2 Mar 20:47

svn-ci script (was: Re: [Xapian-commits] 7830: trunk/xapian-core/)

On Fri, Mar 02, 2007 at 12:20:44PM +0000, richard wrote:
> * HACKING: Add note about how to generate ChangeLog timestamps
>   using the unix date command - and I've started generating them in
>   the same format as Olly is. (I hope.)

I actually use a perl script which does:

    print CO strftime("%a %h %d %H:%M:%S %Z %Y", localtime);

But %b and %h are just aliases and %T is %H:%M:%S, so your date command
should produce identical results.

I guess my "svn ci" wrapper script is probably useful to other people
with commit access, so I've just committed it to
xapian-maintainer-tools.  Feel free to use it as is, hack it about, or
just ignore it:

http://svn.xapian.org/trunk/xapian-maintainer-tools/svn-ci?view=markup

It writes the tedious parts of the ChangeLog entry for you, and pastes
in "svn diff" for the files/directories you are committing, which I find
helps avoid me checking in other changes that might be in my tree by
mistake, and helps make sure I document all changes in the ChangeLog
entry.  It also massages the ChangeLog entry into a suitable SVN commit
message.  If you fail to delete all the diff fragments or conflict
markers, it won't let you commit.

It has a few foibles, but generally you get to see them before it
commits at least.  The only exception is that if ChangeLog is modified
and doesn't appear to have diff fragments or conflict markers, it will

(Continue reading)

Richard Boulton | 3 Mar 02:05

Error handling in the bindings

Currently, the SWIG generated bindings catch exceptions raised by Xapian 
and rethrow them using the standard SWIG error types: for example, if 
any Xapian::DatabaseError exception is raised, SWIG will use its
standard SWIG_IOError error type to report the error.

In Python, this leads to code like the following:

try:
     db.get_document(1)
except RuntimeError, e:
     if str(e).startswith('DocNotFoundError:'):
         # Handle a DocNotFoundError

(Incidentally, the error raised in python for this actually is a 
RuntimeError, not an IOError as I would have expected from reading the 
SWIG code, but that's not a major issue.)

For Python, it would be much nicer to have the full exception heirarchy 
of Xapian mirrored as Python classes, with the appropriate inheritance 
structure.  This is probably true for other languages too.  Looking at 
the code, I don't think it would take terribly long to sort this out for 
Python, but I haven't looked at what's needed for the other bindings we 
support.  It would certainly be much nicer to be able to write the above 
Python snippet as:

try:
     db.get_document(1)
except xapian.DocNotFoundError, e:
     # Handle a DocNotFoundError

(Continue reading)

Olly Betts | 3 Mar 02:46

Re: Error handling in the bindings

On Sat, Mar 03, 2007 at 01:05:05AM +0000, Richard Boulton wrote:
> try:
>     db.get_document(1)
> except RuntimeError, e:
>     if str(e).startswith('DocNotFoundError:'):
>         # Handle a DocNotFoundError

This is probably the worst thing about the bindings at present.

> For Python, it would be much nicer to have the full exception heirarchy 
> of Xapian mirrored as Python classes, with the appropriate inheritance 
> structure.  This is probably true for other languages too.  Looking at 
> the code, I don't think it would take terribly long to sort this out for 
> Python, but I haven't looked at what's needed for the other bindings we 
> support.

For all those that support classes with inheritance and throwing classes
as exceptions, I think it should be fairly easy.

Of the SWIG bindings, Python and C# do.  Ruby almost certainly does, but
I don't know it very well.  PHP5 does, but PHP4 doesn't support
exceptions (the current approach is clumsy, but probably about as good
as we're likely to get there).  Tcl8 essentially only supports throwing
a string from what I recall (but I could be wrong).

And of the non-SWIG bindings, Java does, and Perl effectively does (the
mechanism is a bit different though).

> In theory, the good thing about using the built in SWIG error handling 
> system is that we don't have to write code for each language we support

(Continue reading)

James Aylett | 3 Mar 04:22

Re: Error handling in the bindings

On Sat, Mar 03, 2007 at 01:46:41AM +0000, Olly Betts wrote:

> For all those that support classes with inheritance and throwing classes
> as exceptions, I think it should be fairly easy.
> 
> Of the SWIG bindings, Python and C# do.  Ruby almost certainly does, but
> I don't know it very well.  PHP5 does, but PHP4 doesn't support
> exceptions (the current approach is clumsy, but probably about as good
> as we're likely to get there).  Tcl8 essentially only supports throwing
> a string from what I recall (but I could be wrong).

Given Xapian needs to be installed -- ie there are going to be very
few "standard" hosting providers that support it -- we don't have to
worry about PHP4 too much. Most of the PHP extensions and useful bits
of code have started dropping PHP4 support now. (At least, most of the
ones I find.)

I was always intending to add proper exception support to the Python
bindings, but I never found the time.

> > and it would be nice to get this done before we release 1.0 if
> > possible, so unless someone shouts up with a reason that it's not a
> > good idea to try this, I'll try making a patch to fix up the Python
> > exception handling next week, to see how feasible it is.
> 
> I think it would be very useful.  And 1.0 would be a natural point to
> make such a change.
> 
> I am starting to wonder at the number of things that have been noted
> as "ought to be done for 1.0" though.  It would be good to get it

(Continue reading)

Olly Betts | 6 Mar 02:49

Merging stats from multiple databases for expand

In matcher/expandweight.cc we have:

OmExpandBits
operator+(const OmExpandBits &bits1, const OmExpandBits &bits2)
{
    OmExpandBits sum(bits1); 
    sum.multiplier += bits2.multiplier;
    sum.rtermfreq += bits2.rtermfreq;

    // FIXME - try to share this information rather than pick half of it
    if (bits2.dbsize > sum.dbsize) {
        DEBUGLINE(WTCALC, "OmExpandBits::operator+ using second operand: " <<
                  bits2.termfreq << "/" << bits2.dbsize << " instead of " <<
                  bits1.termfreq << "/" << bits1.dbsize);
        sum.termfreq = bits2.termfreq;
        sum.dbsize = bits2.dbsize;
    } else {
        DEBUGLINE(WTCALC, "OmExpandBits::operator+ using first operand: " <<
                  bits1.termfreq << "/" << bits1.dbsize << " instead of " <<
                  bits2.termfreq << "/" << bits2.dbsize);
        // sum already contains the parts of the first operand
    }
    return sum;
}

Why don't we "share this information" by just replacing the "if" by:

    sum.termfreq += bits2.termfreq;
    sum.dbsize += bits2.dbsize;

(Continue reading)

Group

gmane.comp.search.xapian.devel.

Project Web Page

Development discussion about Xapian, an Open Source Probabilistic Information Retrieval library

Search Archive

Page

1 | 2 | 3 | 4 | 5 | 6

Articles in period: 57

March 2007

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane

Recent entries

How to count I/O consuming in OProfile while profiling xapian (30 Dec 10:09)
Xapian 1.2.8 core for Windows (23 Dec 10:17)
Looking for a project (23 Dec 08:39)
How to enable stemming with default_op set to OP_NEAR (15 Dec 00:16)
Bindings for a Scheme implementation? (1 Dec 13:30)
apply for attend xapian develope ! (25 Aug 16:55)
Compilation error (24 Aug 12:23)
Hi, (23 Aug 15:18)
GSoC Final Evaluations (19 Aug 03:49)
Interest in developement as a student (27 Jul 11:15)
How to add support of Chinese & Japanese (14 Jul 10:39)
GSoC Admin (12 Jul 17:51)
Compiling on Mac OS X (6 Jul 03:21)
xapian csharp bindings crash (25 Jun 17:37)
1.2.6 PHP bindings not building (22 Jun 17:45)
GNU GPL, really!? (9 Jun 03:15)
Introduction and Discussion for Learning to Rank Framework (7 Jun 08:54)
Conversion of all docs to RST (30 May 02:32)
Does OP_NEAR works with stemming? (27 May 21:59)
Xapian GSoC 2011 - Official Start of Coding (23 May 07:04)
Fwd: FLAG_PARTIAL and subset words (15 May 22:00)

Archives

2	January 2012
19	December 2011
4	September 2011
11	August 2011
14	July 2011
15	June 2011
37	May 2011
57	April 2011
93	March 2011
11	February 2011
9	January 2011
6	December 2010
2	October 2010
9	September 2010
5	August 2010
7	June 2010
4	April 2010
9	March 2010
5	February 2010
3	January 2010
3	November 2009
11	September 2009
6	August 2009
20	July 2009
4	June 2009
14	May 2009
22	April 2009
24	January 2009
4	December 2008
6	November 2008
16	October 2008
4	September 2008
8	August 2008
11	July 2008
8	June 2008
1	May 2008
21	April 2008
5	March 2008
9	February 2008
3	January 2008
11	December 2007
6	November 2007
19	October 2007
24	September 2007
2	August 2007
5	July 2007
30	June 2007
26	May 2007
146	April 2007
57	March 2007
44	February 2007
12	January 2007
32	December 2006
44	November 2006
23	October 2006
6	September 2006
20	August 2006
11	July 2006
34	June 2006
22	May 2006
70	April 2006
13	March 2006
4	February 2006
8	January 2006
17	December 2005
11	November 2005
2	October 2005
4	September 2005
22	August 2005
19	July 2005
19	June 2005
24	May 2005
9	April 2005
8	March 2005
11	February 2005
1	January 2005
41	December 2004
4	October 2004
17	September 2004
16	August 2004
17	June 2004
41	May 2004
4	April 2004
3	March 2004
16	January 2004
2	December 2003
27	November 2003
1	October 2003
15	August 2003
65	July 2003
29	June 2003
32	May 2003
25	April 2003
29	March 2003
37	February 2003
20	January 2003
25	December 2002

Design

Zawodny | Right Menu | Left Menu

Your Own Design

Paste the URL of your CSS below. Download CSS template