Discussion of Python object-relational mapper SQLObject

BLOBCol and MySQLdb 1.2.1+

Hi all,

I'm having problems with BLOBCol on MySQLdb 1.2.1+. I use UTF-8
as the charset for connection, and consequently I cannot save binary
data to the BLOB column. I tracked this down to sqlobject/mysql/mysqlconnection.py,
line 78:

                    # For MysqlDB 1.2.1 and later, we go
                    # encoding->unicode->charset (in the mysql db)
                    myquery = unicode(query, self.encoding)
                    return cursor.execute(myquery)

However, the query (the generated SQL query) contain's binary data, so unicode
function fails.

It seems that this conversion to unicode should be done while still building
the query, i.e. convert only string column values, not blobs or column names.

While I could possibly dig into this and fix it, I wonder maybe anyone else has
solution for this?

Cheers,
Adomas

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss

headers

Toke Høiland-Jørgensen | 2 Jan 22:43

Transactions leaking memory?

I am having problems with an application of mine leaking memory. The
application reads in some files, parses the data, and does a few
inserts and/or updates depending on the data in the files. My problem
is that the applications memory usage (as reported by top) is growing
by roughly 20-40 bytes every time a file is read and processed.

By writing a couple of test cases, I have identified the problem to
have something to do with the use of transactions. This is illustrated
by a simple test case. If the following insert is put into an infinite
loop the memory usage stays constant (Metadata is a class that
inherits from SQLObject):

while True:
	m = Metadata(
		orderflow = None,
		transaction = None,
		time = DateTime.now(),
		fieldName = "Comment",
		fieldContents = "duh"
		)

However, if the loop is modified to look like this, the memory usage
will keep rising:

while True:
	trans = connection.transaction()
	m = Metadata(
		connection = trans,
		orderflow = None,
		transaction = None,
		time = DateTime.now(),
		fieldName = "Comment",
		fieldContents = "duh"
		)
	trans.commit()

Reusing the same transaction object (with trans.begin()) yields the
same results.
This is using a trunk version of sqlobject (rev. 2171) and psycopg
2.0.5.1 connecting to postgresql 8.1.5. Python is 2.4.3 running on
x86-64.

How do I keep the transactions from leaking memory?

Thanks in advance,
-Toke

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

headers

Oleg Broytmann | 3 Jan 12:54

Re: Transactions leaking memory?

On Tue, Jan 02, 2007 at 10:43:41PM +0100, Toke H??iland-J??rgensen wrote:
> How do I keep the transactions from leaking memory?

   trans.cache.clear() (I suppose the "problem" which is not really a bug
is in the cache. If not you have to investigate further.)

Oleg.
--

-- 
     Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

headers

Toke Høiland-Jørgensen | 3 Jan 22:51

Re: Transactions leaking memory?

trans.cache.clear() made the problem disappear from the test case from
my last mail (why are objects cached when sqlmeta.cacheValues is set
to False?). However, the application is still leaking memory, although
not as much as before. As far as I can see by stepping through with
the debugger it happens when using SelectResults.

Do SelectResults objects need to be expired, and if so, how?

I am pretty sure that i .expire() all SQLObject objects and run
.cache.clear() on both the connection and transaction objects when i'm
done with them. Is there anything else I need to do to clear caches?

Thanks again,
-Toke

On 1/3/07, Oleg Broytmann <phd <at> phd.pp.ru> wrote:
> On Tue, Jan 02, 2007 at 10:43:41PM +0100, Toke H??iland-J??rgensen wrote:
> > How do I keep the transactions from leaking memory?
>
>    trans.cache.clear() (I suppose the "problem" which is not really a bug
> is in the cache. If not you have to investigate further.)
>
> Oleg.
> --
>      Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
>            Programmers don't die, they just GOSUB without RETURN.
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> sqlobject-discuss mailing list
> sqlobject-discuss <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

headers

David Turner | 4 Jan 19:52

Re: Patch: versioning

On Sat, 2006-12-30 at 08:13 +0200, Dan Pascu wrote:
> On Thursday 28 December 2006 21:40, David Turner wrote:
> > I should probably also have a method which returns a dict of only items
> > that have changed between this version and a specified version (by
> > default, the current version).
> 
> Such a method would be nice.

OK, it's in svn.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

headers

Dan Pascu | 4 Jan 21:29

Re: Patch: versioning

On Thursday 04 January 2007 20:52, David Turner wrote:
> On Sat, 2006-12-30 at 08:13 +0200, Dan Pascu wrote:
> > On Thursday 28 December 2006 21:40, David Turner wrote:
> > > I should probably also have a method which returns a dict of only
> > > items that have changed between this version and a specified
> > > version (by default, the current version).
> >
> > Such a method would be nice.
>
> OK, it's in svn.

Thank you

--

-- 
Dan

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

headers

Christoph Berendes | 7 Jan 05:07

macosx: 12 tests fail/186 pass,incl. SQLMultipleJoin

I'm trying to install SQLObject on my Ibook running MACOS 10.4.8, python 2.3.5 and SQLite version 3.1.3

I've installed and test pysqlite 2.3.2 successfully, then checked out SQLObject from the trunk and ran setup.py build and setup.py install with no error messages.

Running py.test with no arguments then generates 186 tests passed but also 12 tests failed:

sqlobject/tests/test_SQLMultipleJoin.py[1] F

sqlobject/tests/test_auto.py[5] .FFF.

sqlobject/tests/test_auto_old.py[4] .FF.

sqlobject/tests/test_foreignKey.py[2] .F

sqlobject/tests/test_select.py[14] ............F.

sqlobject/tests/test_sqlobject_admin.py - FAILED TO LOAD MODULE

sqlobject/tests/test_transactions.py[4] ..FF

sqlobject/versioning/test/test_version.py[6] ....F.

I assume that all tests should pass, so I'd be grateful for pointers as to how to begin troubleshooting.

Thanks.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss

headers

Diez B. Roggisch | 8 Jan 12:02

DecimalCol

Hi, 

I simply don't get the decimal column support to work - it will always return 
floats.

I'm using turbogears, sqlobject 7.2, and I tried psycopg as well as psycopg2.

Any suggestions?

-- 
>> Diez B. Roggisch
>> Developer

T  +49 (30) 443 50 99 - 27
F  +49 (30) 443 50 99 - 99
M  +49 (179) 11 75 303
E  diez.roggisch <at> artnology.com

>> artnology GmbH

A  Milastraße 4 / D-10437 Berlin
T  +49 (30) 443 50 99 - 0
F  +49 (30) 443 50 99 - 99
E  info <at> artnology.com
I  http://www.artnology.com

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss

headers

Diez B. Roggisch | 8 Jan 12:14

Re: DecimalCol

On Monday 08 January 2007 12:02, Diez B. Roggisch wrote:
> Hi,
>
> I simply don't get the decimal column support to work - it will always
> return floats.
>
> I'm using turbogears, sqlobject 7.2, and I tried psycopg as well as
> psycopg2.
>
> Any suggestions?

False alarm. The problem was that I tested in TG using sqlite - but with 
a "real" database, things worked out as expected.

Diez

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

headers

Diez B. Roggisch | 9 Jan 14:17

event callbacks somwhere?

Hi,

the docs don't mention anything - so I ask: is there a way to have events 
thrown  when there are actual updates/inserts on certain objects?

I've got some rather complex to compute data I'd like to cache until the base 
of that data- some persistent objects - change.

Regards,

-- 
>> Diez B. Roggisch
>> Developer

T  +49 (30) 443 50 99 - 27
F  +49 (30) 443 50 99 - 99
M  +49 (179) 11 75 303
E  diez.roggisch <at> artnology.com

>> artnology GmbH

A  Milastraße 4 / D-10437 Berlin
T  +49 (30) 443 50 99 - 0
F  +49 (30) 443 50 99 - 99
E  info <at> artnology.com
I  http://www.artnology.com

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss

5	February 2012
12	January 2012
16	December 2011
17	November 2011
12	August 2011
29	July 2011
9	June 2011
27	May 2011
26	April 2011
36	March 2011
44	February 2011
15	January 2011
30	December 2010
11	November 2010
26	October 2010
12	September 2010
23	August 2010
37	July 2010
13	June 2010
14	May 2010
71	April 2010
23	March 2010
63	February 2010
24	January 2010
11	December 2009
10	November 2009
20	October 2009
38	September 2009
57	August 2009
36	July 2009
30	June 2009
38	May 2009
19	April 2009
22	March 2009
46	February 2009
69	January 2009
65	December 2008
41	November 2008
37	October 2008
21	September 2008
30	August 2008
66	July 2008
31	June 2008
82	May 2008
69	April 2008
55	March 2008
139	February 2008
82	January 2008
54	December 2007
137	November 2007
101	October 2007
31	September 2007
81	August 2007
118	July 2007
82	June 2007
129	May 2007
120	April 2007
115	March 2007
102	February 2007
152	January 2007
135	December 2006
128	November 2006
330	October 2006
142	September 2006
161	August 2006
94	July 2006
160	June 2006
254	May 2006
218	April 2006
228	March 2006
157	February 2006
235	January 2006
235	December 2005
315	November 2005
466	October 2005
303	September 2005
336	August 2005
265	July 2005
166	June 2005
193	May 2005
152	April 2005
145	March 2005
258	February 2005
264	January 2005
177	December 2004
169	November 2004
60	October 2004
78	September 2004
65	August 2004
52	July 2004
67	June 2004
65	May 2004
88	April 2004
123	March 2004
92	February 2004
38	January 2004
61	December 2003
95	November 2003
85	October 2003
88	September 2003
64	August 2003
129	July 2003
107	June 2003
206	May 2003
204	April 2003
43	March 2003
2	February 2003

Mon	Tue	Wed	Thu	Fri	Sat	Sun

1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31