Monet DB (relational database & XML database) development discussinos ()

Stefan Manegold | 2 Apr 2009 01:19

Picon

Picon

Re: [Monetdb-checkins] MonetDB5/src/modules/mal mat.mx, , 1.25, 1.26

On Wed, Apr 01, 2009 at 07:59:56PM +0000, Martin Kersten wrote:
> Update of /cvsroot/monetdb/MonetDB5/src/modules/mal
> In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv25709
> 
> Modified Files:
> 	mat.mx 
> Log Message:
> Packing of BATs should avoid falling into the BATextend trap.
> Preclaim enough space. Don't bother about the properties of the
> result. That should be decided later.
> 

This checking removes the only initialization of the pointer variable
	int *ret;
which is then later dereferenced for assginment --- this might/will result
in a segfault or other non-deterministic behavior. See details below.

Stefan 

> 
> U mat.mx
> Index: mat.mx
> ===================================================================
> RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/mat.mx,v
> retrieving revision 1.25
> retrieving revision 1.26
> diff -u -d -r1.25 -r1.26
> --- mat.mx	7 Jan 2009 14:15:13 -0000	1.25
> +++ mat.mx	1 Apr 2009 19:59:50 -0000	1.26
>  <at>  <at>  -134,22 +134,31  <at>  <at>

(Continue reading)

Martin Kersten | 3 Apr 2009 12:41

Picon

Picon

Re: [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.278, 1.279 gdk_bat.mx, , 1.213, 1.214 gdk_posix.mx, , 1.168, 1.169 gdk_relop.mx, , 1.161, 1.162 gdk_utils.mx, , 1.241, 1.242

Peter Boncz wrote:
> Update of /cvsroot/monetdb/MonetDB/src/gdk
> In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv31767/gdk
> 
> Modified Files:
> 	gdk.mx gdk_bat.mx gdk_posix.mx gdk_relop.mx gdk_utils.mx 
> Log Message:
> various tweaks to the memory mangement
> 
> - tune the GDK_mmap_minsize to vary between full RAM size and 128MB
>   (lowest when memory pressure is extreme)
> 
> - in fetchjoin, when the optimization of string heap copying is applied
>   (instead of creating a new string heap and inserting into it), try
>   to share the heaps in virtual memory (rather than copying it)
>   This is possible with the logical view mechanism.
> 
>   now fetchjoin, batcopy and remap use this VM heap sharing.
If we do a fetchjoin(L,R) to a table R that does not fit in memory,
and the operand L has more oids then the number of pages in R
then we should also sort L on the tail oids first

This under the constraint that sorting L also cost some IO

If you are tweaking in that area anyway, such a patch is
appreciated.
> 
> - disable vmalloc on Linux -- well actually on platforms
>   that do not have posix_fadvise. A bit of a hack.
>

(Continue reading)

Stefan Manegold | 3 Apr 2009 13:18

Picon

Picon

Re: [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.278, 1.279 gdk_bat.mx, , 1.213, 1.214 gdk_posix.mx, , 1.168, 1.169 gdk_relop.mx, , 1.161, 1.162 gdk_utils.mx, , 1.241, 1.242

Peter,

these changes lead to

$ Mserver 
# MonetDB Server v4.29.0
# based on GDK   v1.29.0
# Copyright (c) 1993-July 2008, CWI. All rights reserved.
# Copyright (c) August 2008-2009, MonetDB B.V.. All rights reserved.
# Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs; dynamically linked.
# Visit http://monetdb.cwi.nl/ for further information.
!ERROR: CMDfind: value not found.
!ERROR: CMDfind: operation failed.
!ERROR: interpret_params: int(param 1): evaluation error.
MonetDB>

In fact, Sjoerd fixed this yesterday in 

===================================================================
2009/04/02 - sjoerd: MonetDB/src/gdk/gdk_utils.mx,1.241
Reintroduced monet_pid, gdk_mem_pagebits, and gdk_mem_bigsize into the
MonetDB environment, but not through advertizing them as options.
This should fix bug 2726337.
===================================================================

Your checkin apparently removed this fix, again.

Why?

Stefan

(Continue reading)

Peter Boncz | 3 Apr 2009 13:58

Re: [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.278, 1.279 gdk_bat.mx, , 1.213, 1.214 gdk_posix.mx, , 1.168, 1.169 gdk_relop.mx, , 1.161, 1.162 gdk_utils.mx, , 1.241, 1.242

sorry, I was helping Stefan de Konink, but he had a read-only version of the MonetDB sources.

> -----Original Message-----
> From: Stefan Manegold [mailto:Stefan.Manegold@...]
> Sent: Friday, April 03, 2009 1:18 PM
> To: monetdb-developers@...; Peter Boncz
> Subject: Re: [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.278, 1.279
> gdk_bat.mx, , 1.213, 1.214 gdk_posix.mx, , 1.168, 1.169 gdk_relop.mx, ,
> 1.161, 1.162 gdk_utils.mx, , 1.241, 1.242
> 
> Peter,
> 
> these changes lead to
> 
> $ Mserver
> # MonetDB Server v4.29.0
> # based on GDK   v1.29.0
> # Copyright (c) 1993-July 2008, CWI. All rights reserved.
> # Copyright (c) August 2008-2009, MonetDB B.V.. All rights reserved.
> # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs; dynamically
> linked.
> # Visit http://monetdb.cwi.nl/ for further information.
> !ERROR: CMDfind: value not found.
> !ERROR: CMDfind: operation failed.
> !ERROR: interpret_params: int(param 1): evaluation error.
> MonetDB>
> 
> In fact, Sjoerd fixed this yesterday in
> 
> ===================================================================

(Continue reading)

Stefan Manegold | 3 Apr 2009 15:18

Picon

Picon

Re: [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.278, 1.279 gdk_bat.mx, , 1.213, 1.214 gdk_posix.mx, , 1.168, 1.169 gdk_relop.mx, , 1.161, 1.162 gdk_utils.mx, , 1.241, 1.242

On Fri, Apr 03, 2009 at 04:58:45AM -0700, Peter Boncz wrote:
> sorry, I was helping Stefan de Konink, but he had a read-only version of the MonetDB sources.

Tip for next time:
- use `cvs diff | patch` instead of `cp`
- use `cvs diff` to double-check your changes before check-in
- check diffs in checkin emails to see whether your check-in went as
  intended

Stefan

> > -----Original Message-----
> > From: Stefan Manegold [mailto:Stefan.Manegold@...]
> > Sent: Friday, April 03, 2009 1:18 PM
> > To: monetdb-developers@...; Peter Boncz
> > Subject: Re: [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.278, 1.279
> > gdk_bat.mx, , 1.213, 1.214 gdk_posix.mx, , 1.168, 1.169 gdk_relop.mx, ,
> > 1.161, 1.162 gdk_utils.mx, , 1.241, 1.242
> > 
> > Peter,
> > 
> > these changes lead to
> > 
> > $ Mserver
> > # MonetDB Server v4.29.0
> > # based on GDK   v1.29.0
> > # Copyright (c) 1993-July 2008, CWI. All rights reserved.
> > # Copyright (c) August 2008-2009, MonetDB B.V.. All rights reserved.
> > # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs; dynamically
> > linked.

(Continue reading)

Peter Boncz | 7 Apr 2009 10:37

Picon

Picon

sorting fetchjoin

Martin,

you suggest:

> If we do a fetchjoin(L,R) to a table R that does not fit in memory,
> and the operand L has more oids then the number of pages in R
> then we should also sort L on the tail oids first

The problem is that many MonetDB use fetchjoin as an order-aware operator and
expect the output to be in left input order. This is then exploited in tuple
reconstruction. It will certainly break pathfinder, for instance. Applications
like Proximity would also break.

In my thesis, I had suggested to create an accelerator that attaches a
radix-clustered version to the left input. Fetchjoin would then use this
radix-clustered version, and call decluster on the result, restoring the
original order (using a cache and dick-friendly pattern).

However, I have come to dislike the idea of low-level accelerators. I now think
this is better handled on a higher layer, specifically the relational algebra
layer.

Peter

> -----Original Message-----
> From: Martin Kersten [mailto:Martin.Kersten@...]
> Sent: Friday, April 03, 2009 12:42 PM
> To: monetdb-developers@...; Peter Boncz
> Subject: Re: [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.278, 1.279
> gdk_bat.mx, , 1.213, 1.214 gdk_posix.mx, , 1.168, 1.169 gdk_relop.mx, ,

(Continue reading)

Stefan Manegold | 7 Apr 2009 11:06

Picon

Picon

monetdb-install.sh & MAKEOPTS=-j

Fabian,

`make -j[N] install` is know not to work (in particular with MonetDB5) due
to race conditions and/or dependency issues, and there is hardly any way to
fix this.

Hence, setting MAKEOPTS=-j[N] before calling monetdb-install.sh might (will)
result in an error during make install.

I propose the following (or similar) patch to omit -j[N] when calling make
install (of course, the "simpler" but maybe less desireable option is to
omit MAKEOPTS completely with make install ...):

Index: monetweb/Assets/monetdb-install.sh
===================================================================
RCS file: /ufs/monet/repository/monetweb/Assets/monetdb-install.sh,v
retrieving revision 1.42
diff -u -r1.42 monetdb-install.sh
--- monetweb/Assets/monetdb-install.sh	16 Mar 2009 20:27:42 -0000	1.42
+++ monetweb/Assets/monetdb-install.sh	7 Apr 2009 09:00:33 -0000
 <at>  <at>  -434,7 +434,7  <at>  <at> 
 			MAKE=make
 		fi
 	fi
-	${MAKE} ${MAKEOPTS} "$ <at> " > $QUIET
+	${MAKE} "$ <at> " > $QUIET
 	return $?
 }

 <at>  <at>  -502,8 +502,8  <at>  <at>

(Continue reading)

Fabian Groffen | 7 Apr 2009 11:30

Picon

Picon

Re: monetdb-install.sh & MAKEOPTS=-j

On 07-04-2009 11:06:56 +0200, Stefan Manegold wrote:
> Fabian,
> 
> `make -j[N] install` is know not to work (in particular with MonetDB5) due
> to race conditions and/or dependency issues, and there is hardly any way to
> fix this.
> 
> Hence, setting MAKEOPTS=-j[N] before calling monetdb-install.sh might (will)
> result in an error during make install.
> 
> I propose the following (or similar) patch to omit -j[N] when calling make
> install (of course, the "simpler" but maybe less desireable option is to
> omit MAKEOPTS completely with make install ...):

I prefer removing the -j argument from the script, and leaving it a
"advanced" developers' option instead then.  

We should not try to needlessly complicate the monetdb-install.sh script
for a flaw in our makefiles.  Ultimately we should concentrate on fixing
them, IMO.

------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com

Sjoerd Mullender | 9 Apr 2009 17:41

Picon

Re: MonetDB/src/gdk gdk_bbp.mx, , 1.251, 1.252 gdk_utils.mx, , 1.244, 1.245

Peter Boncz wrote:
> Update of /cvsroot/monetdb/MonetDB/src/gdk
> In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv27450
> 
> Modified Files:
> 	gdk_bbp.mx gdk_utils.mx 
> Log Message:
> gdk_utils.mx:
> - performance fix: do not use malloc() on Linux either
>   its malloc() implementation appears to give sizable private arenas to
>   each new thread that performs significant malloc work; this leads
>   to huge memory fragmentation (especially when used with mitosis)
> 
> gdk_bbp.mx:
> - BBPtrim() should not be allowed to trim logical views
>   this may lead to BBPdecref errors!!

Is this a bug fix that should also be applied to the stable branch?

> 
> 
> U gdk_bbp.mx
> Index: gdk_bbp.mx
> ===================================================================
> RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_bbp.mx,v
> retrieving revision 1.251
> retrieving revision 1.252
> diff -u -d -r1.251 -r1.252
> --- gdk_bbp.mx	31 Mar 2009 13:42:54 -0000	1.251
> +++ gdk_bbp.mx	9 Apr 2009 15:29:18 -0000	1.252

(Continue reading)

Peter Boncz | 9 Apr 2009 19:31

Re: [Monetdb-checkins] MonetDB/src/gdk gdk_bbp.mx, , 1.251, 1.252 gdk_utils.mx, , 1.244, 1.245

Yes, it actually should.

Sent from my iPhone

On Apr 9, 2009, at 5:42 PM, "Sjoerd Mullender" <sjoerd@...> wrote:

> Peter Boncz wrote:
>> Update of /cvsroot/monetdb/MonetDB/src/gdk
>> In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv27450
>>
>> Modified Files:
>>    gdk_bbp.mx gdk_utils.mx
>> Log Message:
>> gdk_utils.mx:
>> - performance fix: do not use malloc() on Linux either
>>  its malloc() implementation appears to give sizable private arenas  
>> to
>>  each new thread that performs significant malloc work; this leads
>>  to huge memory fragmentation (especially when used with mitosis)
>>
>> gdk_bbp.mx:
>> - BBPtrim() should not be allowed to trim logical views
>>  this may lead to BBPdecref errors!!
>
> Is this a bug fix that should also be applied to the stable branch?
>
>>
>>
>> U gdk_bbp.mx
>> Index: gdk_bbp.mx

(Continue reading)

Group

gmane.comp.db.monetdb.devel.

Advertisement

Project Web Page

Monet DB (relational database & XML database) development discussinos ()

Search Archive

Page

1 | 2 | 3 | 4 | 5 | 6 | 7

Articles in period: 65

April 2009

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane

Recent entries

This mailing list has been discontinued (8 Jan 2013)
developers-list Digest, Vol 4, Issue 18 (26 Dec 2012)
developers-list Digest, Vol 4, Issue 17 (24 Dec 2012)
developers-list Digest, Vol 4, Issue 16 (22 Dec 2012)
developers-list Digest, Vol 4, Issue 6 (10 Dec 2012)
developers-list Digest, Vol 3, Issue 9 (21 Nov 2012)
developers-list Digest, Vol 3, Issue 7 (14 Nov 2012)
developers-list Digest, Vol 2, Issue 11 (26 Oct 2012)
developers-list Digest, Vol 2, Issue 3 (11 Oct 2012)
developers-list Digest, Vol 1, Issue 1 (27 Sep 2012)
MonetDB project will move to monetdb.org (26 Sep 2012)
MonetDB project will move to monetdb.org (26 Sep 2012)
Announcement: New Jul2012-SP2 Bugfix release of MonetDB suite (21 Sep 2012)
MonetDB on windows(visual studio 2010) (12 Sep 2012)
some questions (12 Sep 2012)
error: not protecting function: no buffer at least 8 bytes long (29 Aug 2012)
Migration DW to MonetDB (27 Aug 2012)
MonetDB: Jul2012 - Only print runtimeTiming information when in ... (25 Aug 2012)
MonetDB manager for Python (23 Aug 2012)
Problems compiling MonetDB-11.11.5 on SuSE SLES-11 SP2 (22 Aug 2012)
New Oct2012 stable branch created (13 Aug 2012)

Archives

3	January 2013
21	December 2012
12	November 2012
12	October 2012
8	September 2012
22	August 2012
21	July 2012
6	June 2012
8	May 2012
6	April 2012
7	March 2012
15	February 2012
15	January 2012
20	December 2011
20	November 2011
19	October 2011
8	September 2011
7	July 2011
3	June 2011
9	May 2011
1	April 2011
5	March 2011
16	February 2011
10	January 2011
13	November 2010
23	October 2010
18	September 2010
29	August 2010
50	July 2010
22	June 2010
40	May 2010
60	April 2010
73	March 2010
62	February 2010
31	January 2010
34	December 2009
29	November 2009
18	October 2009
16	September 2009
47	August 2009
36	July 2009
27	June 2009
104	May 2009
63	April 2009
28	March 2009
50	February 2009
32	January 2009
36	December 2008
77	November 2008
50	October 2008
60	September 2008
42	August 2008
66	July 2008
127	June 2008
54	May 2008
34	April 2008
39	March 2008
77	February 2008
40	January 2008
88	December 2007
43	November 2007
162	October 2007
85	September 2007
48	August 2007
69	July 2007
57	June 2007
63	May 2007
97	April 2007
30	March 2007
78	February 2007
87	January 2007

Design

Zawodny | Right Menu | Left Menu

Your Own Design

Paste the URL of your CSS below. Download CSS template